Professional Tutorial on Hydra: A
Python Framework for Configuration Management
This
tutorial provides an in-depth, professional guide to Hydra, an open-source
Python framework developed by Meta (formerly Facebook AI) for managing complex configurations
in applications, especially in machine learning, research, and software
development. Hydra enables hierarchical configuration composition, command-line
overrides, multi-run executions, and integration with various tools, promoting
reproducibility and flexibility.
The
content is compiled from official documentation and examples, covering
installation, core concepts, tutorials, patterns, advanced features, plugins,
and best practices. By the end, you'll have a complete understanding to
implement Hydra in your projects.
For
the latest updates (as of September 2025), refer to the official site at
hydra.cc.
1. Introduction to Hydra
Hydra
simplifies the development of research and complex applications by handling
hierarchical configurations composable from multiple sources. Key features
include:
- Hierarchical Composition: Combine configs from YAML files, code, or
command-line inputs.
- Command-Line Overrides: Modify configs at runtime without altering files.
- Dynamic Tab Completion: Interactive support for config options in the command
line.
- Multi-Run Support: Launch multiple jobs with varying parameters using a
single command.
- Remote Launching: Integrate with launchers for local or cluster
environments.
- Type Safety: Use structured configs for validation and
autocompletion.
Benefits:
- Enhances experiment management
in ML by allowing hyperparameter sweeps and dataset switches without code
changes.
- Improves reproducibility by
logging configs and outputs automatically.
- Reduces boilerplate in
configuring applications.
Hydra
builds on OmegaConf for flexible config handling and supports plugins for
extensibility.
Use
Cases:
- Machine learning experiments
(e.g., with PyTorch Lightning).
- Simulations and data pipelines
in research.
- Deployment configurations in
production software.
Citation
for academic use:
text
@Misc{Yadan2019Hydra,
author = {Omry Yadan},
title = {Hydra - A framework for elegantly
configuring complex applications},
howpublished = {Github},
year = {2019},
url = {https://github.com/facebookresearch/hydra}
}
```<grok-card
data-id="98710a"
data-type="citation_card"></grok-card>
## 2. Installation and
Setup
### Requirements
- Python 3.6+ (stable
version 1.3 supports up to 3.11; older versions down to 2.7).
- No core dependencies
beyond OmegaConf (installed automatically).
### Installation
Install the core
package:
pip
install hydra-core --upgrade
text
For plugins (e.g.,
launchers):
pip
install hydra-submitit-launcher --upgrade # Example for SLURM
text
Hydra supports Linux,
Mac, and Windows.<grok-card data-id="2ab15f"
data-type="citation_card"></grok-card>
### Project Structure
A typical Hydra
project:
project/ ├── conf/ # Config
directory │ ├── config.yaml # Main config │ ├── db/ # Config group │ │ ├── mysql.yaml
│ │ └── postgresql.yaml │ └── init.py # For Python module configs
(optional) ├── my_app.py # Entry point └── outputs/ # Auto-generated for logs
(do not create)
text
Organize configs into
groups (subdirectories) for modularity.<grok-card data-id="c4159d"
data-type="citation_card"></grok-card>
## 3. Getting Started:
Your First Hydra App
### Basic Example
Create
`conf/config.yaml`:
```yaml
db:
driver: mysql
user: omry
pass: secret
Create
my_app.py:
python
import hydra
from omegaconf import DictConfig, OmegaConf
@hydra.main(version_base=None, config_path="conf", config_name="config")
def my_app(cfg: DictConfig) -> None:
print(OmegaConf.to_yaml(cfg))
if __name__ == "__main__":
my_app()
Run:
text
python my_app.py
Output:
yaml
db:
driver:
mysql
pass:
secret
user:
omry
Hydra
creates an output directory (e.g., outputs/YYYY-MM-DD/HH-MM-SS) with logs and a
config copy.
Command-Line Overrides
Override
values:
text
python my_app.py
db.user=root db.pass=1234
Output
reflects changes. Add new fields:
text
python my_app.py
+db.port=3306
```<grok-card
data-id="bda630"
data-type="citation_card"></grok-card>
### Config Composition
Update
`conf/config.yaml`:
```yaml
defaults:
- db: mysql
Create
conf/db/mysql.yaml:
yaml
driver: mysql
user: omry
pass: secret
Create
conf/db/postgresql.yaml:
yaml
driver: postgresql
user: postgres_user
pass: drowssap
timeout: 10
Run
with override:
text
python my_app.py
db=postgresql db.timeout=20
Output:
yaml
db:
driver:
postgresql
pass:
drowssap
timeout:
20
user:
postgres_user
```<grok-card data-id="213488"
data-type="citation_card"></grok-card>
### Multi-Run
Executions
Run multiple configs:
python
my_app.py --multirun db=mysql,postgresql
text
Launches two jobs,
creating separate output directories.<grok-card data-id="9cb3b0"
data-type="citation_card"></grok-card>
## 4. Structured
Configs
Structured configs use
Python dataclasses for type-safe, validated configurations.
### Config Store API
Use `ConfigStore` to
register configs in code:
```python
from dataclasses
import dataclass
from
hydra.core.config_store import ConfigStore
@dataclass
class
PostgresSQLConfig:
driver: str = "postgresql"
user: str = "jieru"
password: str = "secret"
cs =
ConfigStore.instance()
cs.store(name="postgresql",
group="db", node=PostgresSQLConfig)
Run:
text
python my_app.py
+db=postgresql
Output:
yaml
db:
driver:
postgresql
user:
jieru
password:
secret
Supports
types, instances, or dicts for nodes.
Structured Config Schema
Use
dataclasses as schemas for validation.
Example
schemas:
python
from dataclasses import dataclass, MISSING
from hydra.core.config_store import ConfigStore
@dataclass
class DBConfig:
driver: str = MISSING
host: str = "localhost"
port: int = MISSING
@dataclass
class MySQLConfig(DBConfig):
driver: str = "mysql"
port: int = 3306
user: str = MISSING
password: str = MISSING
@dataclass
class PostGreSQLConfig(DBConfig):
driver: str =
"postgresql"
user: str = MISSING
port: int = 5432
password: str = MISSING
timeout: int = 10
@dataclass
class Config:
db: DBConfig = MISSING
debug: bool = False
cs = ConfigStore.instance()
cs.store(name="base_config",
node=Config)
cs.store(group="db", name="base_mysql",
node=MySQLConfig)
cs.store(group="db", name="base_postgresql",
node=PostGreSQLConfig)
Config
files extend bases:
- db/mysql.yaml:
yaml
defaults:
- base_mysql
user: omry
password: secret
Validation
catches errors, e.g., invalid types.
Instantiating Objects with Structured Configs
Define
classes and configs:
python
class DBConnection:
def
__init__(self, driver: str, host: str = "localhost",
port:
int
= MISSING):
self.driver
=
driver
self.host
=
host
self.port
=
port
def
connect(self):
print(f"{self.driver.capitalize()} connecting to {self.host}:{self.port}")
class MySQLConnection(DBConnection):
def
__init__(self, **kwargs):
super().__init__(driver="mysql", port=1234, **kwargs)
class PostgreSQLConnection(DBConnection):
def
__init__(self, timeout: int = 10, **kwargs):
super().__init__(driver="postgresql", port=5678, **kwargs)
self.timeout
=
timeout
@dataclass
class DBConfig:
_target_: str = MISSING
@dataclass
class MySQLConfig(DBConfig):
_target_: str = "my_app.MySQLConnection"
@dataclass
class PostGreSQLConfig(DBConfig):
_target_: str = "my_app.PostgreSQLConnection"
timeout: int = 10
@dataclass
class Config:
db: DBConfig = MISSING
# Register
cs =
ConfigStore.instance()
cs.store(name="config",
node=Config)
cs.store(group="db", name="mysql",
node=MySQLConfig)
cs.store(group="db", name="postgresql",
node=PostGreSQLConfig)
@hydra.main(version_base=None, config_path="conf", config_name="config")
def my_app(cfg: Config) -> None:
db = hydra.utils.instantiate(cfg.db)
db.connect()
Run
python my_app.py db=postgresql outputs: "Postgresql connecting to
localhost:5678"
5. Common Patterns
Extending Configs
Extend
bases in the same or different groups.
Same
group example:
- db/base_mysql.yaml:
yaml
host: localhost
port: 3306
user: ???
password: ???
- db/mysql.yaml:
yaml
defaults:
- base_mysql
user: omry
password: secret
port: 3307
encoding: utf8
Output
merges and overrides.
Different
group: Use /group@package.
Specializing Configuration
Depend
configs on choices using interpolation.
Example:
- config.yaml:
yaml
defaults:
- dataset: imagenet
- model: alexnet
- optional
dataset_model: ${dataset}_${model}
- dataset_model/cifar10_alexnet.yaml:
yaml
# @package _global_
model:
num_layers:
5
Specializes
for specific combinations.
6. Advanced Topics
Packages and Overriding
Packages
define config placement in the hierarchy. Default from group path.
Override
in defaults list (relative) or directive (absolute).
Example:
- Relocate server/db to
admin/backup.
Use
multiple times:
yaml
defaults:
- server/db@src: mysql
- server/db@dst: mysql
```<grok-card data-id="78f0e7"
data-type="citation_card"></grok-card>
### Compose API
For programmatic use
(e.g., Jupyter):
```python
from hydra import initialize,
compose
from omegaconf import
OmegaConf
with
initialize(version_base=None, config_path="conf",
job_name="test_app"):
cfg =
compose(config_name="config", overrides=["db=mysql",
"db.user=me"])
print(OmegaConf.to_yaml(cfg))
Supports
global or context init.
Config Search Path
List
of paths (file:// or pkg://) to find configs. Customize via hydra.searchpath or
plugins.
Job Configuration
Under
hydra.job:
|
Field |
Description |
Default/Example |
|
name |
Job name |
File name |
|
chdir |
Change to output dir |
True |
|
override_dirname |
Dir from overrides |
Auto |
|
env_set |
Env vars to set |
{} |
|
env_copy |
Env vars to copy |
[] |
7. Configuring Hydra Itself
Hydra's
config is customizable:
- Aspects: Launcher, sweeper,
logging, output dirs, help.
- Use snippets, composition, or
CLI overrides.
- Resolvers: ${hydra:...},
${now:...}, ${python_version:...}.
Example:
Set hydra.searchpath in primary config.
8. Plugins
Overview
- Sweeper: Generates job lists from args (e.g., for sweeps).
- Launcher: Executes jobs (e.g., local, SLURM).
- SearchPathPlugin: Modifies search path.
- ConfigSource: Accesses configs from custom sources.
Examples:
Basic launcher/sweeper (built-in); Submitit for SLURM.
Configuring Plugins
Use
defaults or override via config/CLI.
9. Best Practices and Troubleshooting
- Use defaults lists for
modularity.
- Enable strict mode for
validation.
- Log configs with
OmegaConf.to_yaml(cfg).
- Handle paths with
get_original_cwd().
- Integrate with ML tools like
WandB.
- Debug: --cfg hydra, --info
searchpath.
- Community: GitHub,
StackOverflow (#fb-hydra).
For
full examples, see hydra.cc/docs/tutorials/ and GitHub repo.
This
tutorial covers all aspects of Hydra. Experiment with examples for hands-on
learning. If saving as a .docx, copy this Markdown into a converter tool like
Pandoc.