Optuna integration¶

Optuna is an automatic hyperparameter optimization framework and this integration allows you to use it within TimeEval. TimeEval will load the OptunaModule automatically if at least one algorithm uses BayesianParameterSearch as its parameter search strategy. Please make sure that you install the required dependencies for Optuna before using this integration (we also recommend to install psycopg2 to use the PostgreSQL storage backend):

pip install 'optuna>=3.1.0' psycopg2

The following Optuna features are supported:

Definition of search spaces using Optuna distributions for each algorithm (one study per algorithm): BayesianParameterSearch.
Configurable samplers.
Configurable storage backends (in-memory, RDB, Journal, etc.).
Resuming of existing studies (via RDB storage backend).
Parallel and distributed parameter search of a single or multiple studies (synchronized via RDB storage backend).

Warning

Parameter search using the Optuna integration is non-deterministic. The results may vary between different runs, even if the same seed is used (e.g., for the Optuna sampler or pruner). This is because TimeEval needs to re-seed the Optuna samplers for every trial in distributed mode. This is necessary to ensure that initial random samples are different over all workers.

TimeEval will automatically manage an RDB storage backend if you use the default configuration. This allows you to start TimeEval in distributed mode and perform the parameter search in parallel and distributed.

timeeval.integration.optuna.OptunaModule¶

class timeeval.integration.optuna.OptunaModule(config: OptunaConfiguration)¶

Bases: TimeEvalModule

This module is automatically loaded when at least one algorithm uses timeeval.params.BayesianParameterSearch as parameter config.

TimeEval provides the option to use an automatically managed PostgreSQL database as the storage backend for the Optuna studies. The database is started as an additional Docker container either on the local machine or on the scheduler node in distributed execution mode. The database is automatically stopped when TimeEval is finished. The database storage backend allows you to monitor the studies using the Optuna dashboard (that can also be started automatically using another Docker container) and the distributed execution of the studies. This is the default behavior if no storage backend is specified in the configuration.

Parameters: config (OptunaConfiguration) – The configuration for the Optuna module.

finalize(timeeval: TimeEval) → None¶

Called during the FINALIZE-phase of TimeEval and after the individual algorithms’ finalize-functions were executed.

Parameters: timeeval (TimeEval) – The TimeEval instance that is currently running.

load_studies() → List[StudySummary]¶

Load all studies from the default storage. This does not include studies, which were stored in a different storage backend (i.a. where the storage backend was changed using the timeeval.integration.optuna.OptunaStudyConfiguration).

Returns: study_summaries – A list of study summaries.
Return type: List[StudySummary]

timeeval.integration.optuna.OptunaConfiguration¶

class timeeval.integration.optuna.OptunaConfiguration(default_storage: Union[str, Callable[[], BaseStorage]], default_sampler: Optional[BaseSampler] = None, default_pruner: Optional[BasePruner] = None, continue_existing_studies: bool = False, dashboard: bool = False, remove_managed_containers: bool = False, use_default_logging: bool = False, log_level: Union[int, str] = 20)¶

Bases: object

Configuration options for the Optuna module. This includes default options for all Optuna studies.

Parameters

default_storage (str or Lambda returning instance of optuna.storages.BaseStorage) – Storage to store and synchronize the results of the studies. Per default, TimeEval will use a journal file in local execution mode and a PostgreSQL database in distributed execution mode. The database is automatically started and stopped by TimeEval using the latest postgres-Docker image. Use "postgresql" to let TimeEval handle starting and stopping a PostgreSQL database using Docker. Use "journal-file" to let TimeEval create a local file as the storage backend. This only works in non-distributed mode.
default_sampler (optuna.samplers.BaseSampler, optional) – Sampler to use for the study. If not provided, the default sampler is used.
default_pruner (optuna.pruners.BasePruner, optional) – Pruner to use for the study. If not provided, the default pruner is used.
continue_existing_studies (bool, optional) – If True, continue a study with the given name if it already exists in the storage backend. If False, raise an error if a study with the same name already exists.
dashboard (bool, optional) – If True, start the Optuna dashboard (within its own Docker container) to monitor the studies. In distributed execution mode, the dashboard is started on the scheduler node.
remove_managed_containers (bool, optional) – If True, remove the containers managed by TimeEval (e.g., the PostgreSQL database) when TimeEval is finished.
use_default_logging (bool, optional) – If True, use the default logging configuration of the Optuna library. This will log the progress of the studies to stderr. If False, use the logging configuration of TimeEval and propagates the Optuna log messages.
log_level (int or str, optional) – The log level to use for the Optuna logger. The default is info = logging.INFO = 20.

timeeval.integration.optuna.OptunaStudyConfiguration¶

class timeeval.integration.optuna.OptunaStudyConfiguration(n_trials: int, metric: Metric, storage: Optional[Union[str, Callable[[], BaseStorage]]] = None, sampler: Optional[BaseSampler] = None, pruner: Optional[BasePruner] = None, direction: Optional[Union[str, StudyDirection]] = 'maximize', continue_existing_study: bool = False)¶

Bases: object

Configuration for BayesianParameterSearch.

The parameters n_trials and metric are required. All other parameters are optional and will be filled with the default values from the global Optuna configuration if not provided.

Parameters

n_trials (int) – Number of trials to perform.
metric (Metric) – TimeEval metric to use as the studies objective function.
storage (str or Lambda returning instance of optuna.storages.BaseStorage, optional) – Storage to store the results of the study.
sampler (optuna.samplers.BaseSampler, optional) – Sampler to use for the study. If not provided, the default sampler is used.
pruner (optuna.pruners.BasePruner, optional) – Pruner to use for the study. If not provided, the default pruner is used.
direction (str or optuna.study.StudyDirection, optional) – Direction of the optimization (minimize or maximize). If None, the Optuna default direction is used.
continue_existing_study (bool, optional) – If True, continue a study with the given name if it already exists in the storage backend. If False, raise an error if a study with the same name already exists.