TimeEval: Evaluation Tool for Anomaly Detection Algorithms on Time Series¶
Overview¶
TimeEval is an evaluation tool for time series anomaly detection algorithms. It defines common interfaces for datasets and algorithms to allow the efficient comparison of the algorithms’ quality and runtime performance. TimeEval can be configured using a simple Python API and comes with a large collection of compatible datasets and algorithms.
TimeEval takes your input and automatically creates experiment configurations by taking the cross-product of your inputs. It executes all experiment configurations one after the other or — when distributed — in parallel and records the anomaly detection quality and the runtime of the algorithms.
TimeEval takes four inputs for the experiment creation:
Algorithms
Datasets
Algorithm hyperparameter specifications
A repetition number
The following code snippet shows a simple example experiment evaluating COF and a simple baseline algorithm on some test data:
1#!/usr/bin/env python3
2from pathlib import Path
3from typing import Any, Dict
4
5import numpy as np
6
7from timeeval import TimeEval, DatasetManager, DefaultMetrics, Algorithm, TrainingType, InputDimensionality
8from timeeval.adapters import DockerAdapter, FunctionAdapter
9from timeeval.algorithms import cof
10from timeeval.data_types import AlgorithmParameter
11from timeeval.params import FixedParameters
12
13
14def your_algorithm_function(data: AlgorithmParameter, args: Dict[str, Any]) -> np.ndarray:
15 if isinstance(data, np.ndarray):
16 return np.zeros_like(data)
17 else: # isinstance(data, pathlib.Path)
18 return np.genfromtxt(data, delimiter=",", skip_header=1)[:, 1]
19
20
21def main():
22 dm = DatasetManager(Path("tests/example_data"), create_if_missing=False)
23 datasets = dm.select()
24
25 algorithms = [
26 Algorithm(
27 name="COF",
28 main=DockerAdapter(image_name="ghcr.io/timeeval/cof", skip_pull=True),
29 param_config=FixedParameters({
30 "n_neighbors": 20,
31 "random_state": 42
32 }),
33 data_as_file=True,
34 training_type=TrainingType.UNSUPERVISED,
35 input_dimensionality=InputDimensionality.MULTIVARIATE
36 ),
37 # or:
38 # cof(params=FixedParameters({"n_neighbors": 20, "random_state": 42})),
39 Algorithm(
40 name="MyPythonFunctionAlgorithm",
41 main=FunctionAdapter(your_algorithm_function),
42 data_as_file=False
43 )
44 ]
45
46 timeeval = TimeEval(dm, datasets, algorithms, metrics=[DefaultMetrics.ROC_AUC, DefaultMetrics.RANGE_PR_AUC])
47 timeeval.run()
48 results = timeeval.get_results(aggregated=False)
49 print(results)
50
51
52if __name__ == "__main__":
53 main()
Features¶
Listing Example usage of TimeEval illustrates some main features of TimeEval:
- Dataset API:
Interface to available dataset collection to select datasets easily (L21-22).
- Algorithm Adapter Architecture:
TimeEval supports different algorithm adapters to execute simple Python functions or whole pipelines and applications (L27, L38).
- Hyperparameter Specification:
Algorithm hyperparameters can be specified using different search grids (L28-31).
- Metrics:
TimeEval provides various evaluation metrics (such as
timeeval.utils.metrics.DefaultMetrics.ROC_AUC
,timeeval.utils.metrics.DefaultMetrics.RANGE_PR_AUC
, ortimeeval.utils.metrics.FScoreAtK
) and measures algorithm runtimes automatically (L43).- Distributed Execution:
TimeEval can be deployed on a compute cluster to execute evaluation tasks distributedly.
Installation¶
Prerequisites:
TimeEval is published to PyPI and you can install it using:
pip install TimeEval
Attention
Currently, TimeEval is tested only on Linux systems and relies on unix-oid capabilities.
License¶
The project is licensed under the MIT license.
If you use TimeEval in your project or research, please cite our demonstration paper:
Phillip Wenig, Sebastian Schmidl, and Thorsten Papenbrock. TimeEval: A Benchmarking Toolkit for Time Series Anomaly Detection Algorithms. PVLDB, 15(12): 3678 - 3681, 2022. doi:10.14778/3554821.3554873
You can use the following BibTeX entry:
@article{WenigEtAl2022TimeEval,
title = {TimeEval: {{A}} Benchmarking Toolkit for Time Series Anomaly Detection Algorithms},
author = {Wenig, Phillip and Schmidl, Sebastian and Papenbrock, Thorsten},
date = {2022},
journaltitle = {Proceedings of the {{VLDB Endowment}} ({{PVLDB}})},
volume = {15},
number = {12},
pages = {3678--3681},
doi = {10.14778/3554821.3554873}
}
User Guide¶
New to TimeEval? Check out our User Guides to get started with TimeEval. The user guides explain TimeEval’s API and how to use it to achieve your goal.
TimeEval Concepts¶
Background information and in-depth explanations about how TimeEval works can be found in the TimeEval concepts reference.
API Reference¶
The API reference guide contains a detailed description of the functions, modules, and objects included in TimeEval. The API reference describes how the methods work and which parameters can be used.
Contributor’s Guide¶
Want to add to the codebase? You can help with the documentation? The contributing guidelines will guide you through the process of improving TimeEval and its ecosystem.
Additional Links¶
Datasets for TimeEval
TimeEval GUI (prototype)
Time series anomaly dataset generator GutenTAG