TimeEval: Evaluation Tool for Anomaly Detection Algorithms on Time Series

Overview

TimeEval is an evaluation tool for time series anomaly detection algorithms. It defines common interfaces for datasets and algorithms to allow the efficient comparison of the algorithms’ quality and runtime performance. TimeEval can be configured using a simple Python API and comes with a large collection of compatible datasets and algorithms.

TimeEval takes your input and automatically creates experiment configurations by taking the cross-product of your inputs. It executes all experiment configurations one after the other or — when distributed — in parallel and records the anomaly detection quality and the runtime of the algorithms.

TimeEval takes four inputs for the experiment creation:

  1. Algorithms

  2. Datasets

  3. Algorithm hyperparameter specifications

  4. A repetition number

The following code snippet shows a simple example experiment evaluating COF and a simple baseline algorithm on some test data:

Example usage of TimeEval
 1#!/usr/bin/env python3
 2from pathlib import Path
 3from typing import Any, Dict
 4import numpy as np
 5
 6from timeeval import TimeEval, DatasetManager, DefaultMetrics, Algorithm, TrainingType, InputDimensionality
 7from timeeval.adapters import FunctionAdapter # for defining customized algorithm
 8from timeeval.algorithms import cof 
 9from timeeval.data_types import AlgorithmParameter
10from timeeval.params import FixedParameters
11
12def your_algorithm_function(data: AlgorithmParameter, args: Dict[str, Any]) -> np.ndarray:
13    if isinstance(data, np.ndarray):
14        return np.zeros_like(data)
15    else: # isinstance(data, pathlib.Path)
16        return np.genfromtxt(data, delimiter=",", skip_header=1)[:, 1]
17
18def main():
19    dm = DatasetManager(Path("tests/example_data"), create_if_missing=False)
20    datasets = dm.select()
21    algorithms = [
22    # list of algorithms which will be executed on the selected dataset(s)
23        cof(
24            params=FixedParameters({
25                "n_neighbors": 20, 
26                "random_state": 42})
27        ),
28        # calling customized algorithm
29        Algorithm(
30            name="MyPythonFunctionAlgorithm",
31            main=FunctionAdapter(your_algorithm_function),
32            data_as_file=False)
33    ]
34
35    timeeval = TimeEval(dm, datasets, algorithms, metrics=[DefaultMetrics.ROC_AUC, DefaultMetrics.RANGE_PR_AUC])
36    timeeval.run()
37    results = timeeval.get_results(aggregated=False)
38    print(results)
39
40
41if __name__ == "__main__":
42    main()

Features

Listing Example usage of TimeEval illustrates some main features of TimeEval:

Dataset API:

Interface to available dataset collection to select datasets easily (L21-22).

Algorithm Adapter Architecture:

TimeEval supports different algorithm adapters to execute simple Python functions or whole pipelines and applications (L27, L38).

Hyperparameter Specification:

Algorithm hyperparameters can be specified using different search grids (L28-31).

Metrics:

TimeEval provides various evaluation metrics (such as timeeval.utils.metrics.DefaultMetrics.ROC_AUC, timeeval.utils.metrics.DefaultMetrics.RANGE_PR_AUC, or timeeval.utils.metrics.FScoreAtK) and measures algorithm runtimes automatically (L43).

Distributed Execution:

TimeEval can be deployed on a compute cluster to execute evaluation tasks distributedly.

Installation

Prerequisites:

  • Python 3.7, 3.8, or 3.9

  • Docker

  • rsync (if you want to use distributed TimeEval)

TimeEval is published to PyPI and you can install it using:

pip install TimeEval

Attention

Currently, TimeEval is tested only on Linux systems and relies on unix-oid capabilities.

License

The project is licensed under the MIT license.

If you use TimeEval in your project or research, please cite our demonstration paper:

Phillip Wenig, Sebastian Schmidl, and Thorsten Papenbrock. TimeEval: A Benchmarking Toolkit for Time Series Anomaly Detection Algorithms. PVLDB, 15(12): 3678 - 3681, 2022. doi:10.14778/3554821.3554873

You can use the following BibTeX entry:

@article{WenigEtAl2022TimeEval,
  title = {TimeEval: {{A}} Benchmarking Toolkit for Time Series Anomaly Detection Algorithms},
  author = {Wenig, Phillip and Schmidl, Sebastian and Papenbrock, Thorsten},
  date = {2022},
  journaltitle = {Proceedings of the {{VLDB Endowment}} ({{PVLDB}})},
  volume = {15},
  number = {12},
  pages = {3678--3681},
  doi = {10.14778/3554821.3554873}
}

User Guide

New to TimeEval? Check out our User Guides to get started with TimeEval. The user guides explain TimeEval’s API and how to use it to achieve your goal.

To the user guide

TimeEval Concepts

Background information and in-depth explanations about how TimeEval works can be found in the TimeEval concepts reference.

To the concepts

API Reference

The API reference guide contains a detailed description of the functions, modules, and objects included in TimeEval. The API reference describes how the methods work and which parameters can be used.

To the API reference

Contributor’s Guide

Want to add to the codebase? You can help with the documentation? The contributing guidelines will guide you through the process of improving TimeEval and its ecosystem.

To the contributor’s guide