TimeEval: Evaluation Tool for Anomaly Detection Algorithms on Time Series¶
Overview¶
TimeEval is an evaluation tool for time series anomaly detection algorithms. It defines common interfaces for datasets and algorithms to allow the efficient comparison of the algorithms’ quality and runtime performance. TimeEval can be configured using a simple Python API and comes with a large collection of compatible datasets and algorithms.
TimeEval takes your input and automatically creates experiment configurations by taking the cross-product of your inputs. It executes all experiment configurations one after the other or — when distributed — in parallel and records the anomaly detection quality and the runtime of the algorithms.
TimeEval takes four inputs for the experiment creation:
Algorithms
Datasets
Algorithm hyperparameter specifications
A repetition number
The following code snippet shows a simple example experiment evaluating COF and a simple baseline algorithm on some test data:
1#!/usr/bin/env python3
2from pathlib import Path
3from typing import Any, Dict
4import numpy as np
5
6from timeeval import TimeEval, DatasetManager, DefaultMetrics, Algorithm, TrainingType, InputDimensionality
7from timeeval.adapters import FunctionAdapter # for defining customized algorithm
8from timeeval.algorithms import cof
9from timeeval.data_types import AlgorithmParameter
10from timeeval.params import FixedParameters
11
12def your_algorithm_function(data: AlgorithmParameter, args: Dict[str, Any]) -> np.ndarray:
13 if isinstance(data, np.ndarray):
14 return np.zeros_like(data)
15 else: # isinstance(data, pathlib.Path)
16 return np.genfromtxt(data, delimiter=",", skip_header=1)[:, 1]
17
18def main():
19 dm = DatasetManager(Path("tests/example_data"), create_if_missing=False)
20 datasets = dm.select()
21 algorithms = [
22 # list of algorithms which will be executed on the selected dataset(s)
23 cof(
24 params=FixedParameters({
25 "n_neighbors": 20,
26 "random_state": 42})
27 ),
28 # calling customized algorithm
29 Algorithm(
30 name="MyPythonFunctionAlgorithm",
31 main=FunctionAdapter(your_algorithm_function),
32 data_as_file=False)
33 ]
34
35 timeeval = TimeEval(dm, datasets, algorithms, metrics=[DefaultMetrics.ROC_AUC, DefaultMetrics.RANGE_PR_AUC])
36 timeeval.run()
37 results = timeeval.get_results(aggregated=False)
38 print(results)
39
40
41if __name__ == "__main__":
42 main()
Features¶
Listing Example usage of TimeEval illustrates some main features of TimeEval:
- Dataset API:
Interface to available dataset collection to select datasets easily (L21-22).
- Algorithm Adapter Architecture:
TimeEval supports different algorithm adapters to execute simple Python functions or whole pipelines and applications (L27, L38).
- Hyperparameter Specification:
Algorithm hyperparameters can be specified using different search grids (L28-31).
- Metrics:
TimeEval provides various evaluation metrics (such as
timeeval.utils.metrics.DefaultMetrics.ROC_AUC
,timeeval.utils.metrics.DefaultMetrics.RANGE_PR_AUC
, ortimeeval.utils.metrics.FScoreAtK
) and measures algorithm runtimes automatically (L43).- Distributed Execution:
TimeEval can be deployed on a compute cluster to execute evaluation tasks distributedly.
Installation¶
Prerequisites:
TimeEval is published to PyPI and you can install it using:
pip install TimeEval
Attention
Currently, TimeEval is tested only on Linux systems and relies on unix-oid capabilities.
License¶
The project is licensed under the MIT license.
If you use TimeEval in your project or research, please cite our demonstration paper:
Phillip Wenig, Sebastian Schmidl, and Thorsten Papenbrock. TimeEval: A Benchmarking Toolkit for Time Series Anomaly Detection Algorithms. PVLDB, 15(12): 3678 - 3681, 2022. doi:10.14778/3554821.3554873
You can use the following BibTeX entry:
@article{WenigEtAl2022TimeEval,
title = {TimeEval: {{A}} Benchmarking Toolkit for Time Series Anomaly Detection Algorithms},
author = {Wenig, Phillip and Schmidl, Sebastian and Papenbrock, Thorsten},
date = {2022},
journaltitle = {Proceedings of the {{VLDB Endowment}} ({{PVLDB}})},
volume = {15},
number = {12},
pages = {3678--3681},
doi = {10.14778/3554821.3554873}
}
User Guide¶
New to TimeEval? Check out our User Guides to get started with TimeEval. The user guides explain TimeEval’s API and how to use it to achieve your goal.
TimeEval Concepts¶
Background information and in-depth explanations about how TimeEval works can be found in the TimeEval concepts reference.
API Reference¶
The API reference guide contains a detailed description of the functions, modules, and objects included in TimeEval. The API reference describes how the methods work and which parameters can be used.
Contributor’s Guide¶
Want to add to the codebase? You can help with the documentation? The contributing guidelines will guide you through the process of improving TimeEval and its ecosystem.
Additional Links¶
Datasets for TimeEval
TimeEval GUI (prototype)
Time series anomaly dataset generator GutenTAG