Config System

How pyPhases loads, merges, and uses configuration

pyPhases uses a single flat config dict (project.config) that is accessible to every phase, plugin, and data object throughout a run. This page describes how that dict is built from YAML files, how it is read and written at runtime, how additional files are merged in, and how config values are used to produce deterministic cache hashes.

---

Basic get / set

Every phase exposes getConfig and setConfig as thin wrappers around project.getConfig / project.setConfig.

getConfig(key, default=None)

# Flat key
loaderString = self.getConfig("useLoader")

# Dot-notation for nested keys
lr = self.getConfig("trainingParameter.learningRate")

# With a default value
fold = self.getConfig("dataversion.folds", 1)

Keys are resolved with dot notation: "a.b.c" is equivalent to config["a"]["b"]["c"]. A ConfigNotFoundException is raised when a key is missing and no default is provided:

val = self.getConfig("nonExistent")            # raises ConfigNotFoundException
val = self.getConfig("nonExistent", "fallback") # returns "fallback"

setConfig(key, value)

self.setConfig("datasetSplit", "training")

# Dot notation — every parent key must already exist
self.setConfig("trainingParameter.learningRate", 0.001)
# Raises KeyError if "trainingParameter" doesn't exist yet

Warning

setConfig with dot notation does not create intermediate dicts at the moment. Use project.addConfig(bigDict) to add multiple values at the top-level.

Environment variable overrides

Any config value can be overridden via PHASE_CONFIG_<KEY> (underscores replace dots, uppercase):

PHASE_CONFIG_preprocessing_dtype=float16 phases run

CLI --set overrides

phases run --set preprocessing.dtype=float16 --set trainingParameter.learningRate=1e-4

Both env and --set overrides are applied after all YAML files have been loaded, so every downstream cache hash reflects them.

---

Config scopes

project.config is a pdict (a dict subclass with dot-notation support and deep-merge). All phases share the same instance. Use a config scope when a phase needs to temporarily change a key without affecting other phases:

with self.project:
    self.setConfig("datasetSplit", "validation")
    result = self.getData("dataset-val", list)
# config is fully restored to its pre-enter state here

__enter__ deep-copies the config onto a stack; __exit__ pops it. Scopes can be nested. BuildDataset uses this pattern to iterate over all splits without permanently changing datasetSplit.

---

Project file (project.yaml)

The root configuration file. It is always loaded as a full config — its keys map to the top-level project dict directly.

name: my-sleep-study          # required

phases:                       # required
  - name: Extract
  - name: BuildDataset
  - name: Training

plugins:
  - SleePyPhases

data:
  - name: data-processed
    dependsOn: [preprocessing]

config:                       # everything here goes into project.config
  useLoader: sleepedf
  sleepedf-path: /data/sleepedf
  dataversion:
    version: v1
  preprocessing:
    dtype: float32

Keys under config: are merged into project.config. All other top-level keys (name, phases, data, plugins, exporter, importBefore, importAfter) are consumed by the framework and do not appear in the config dict.

---

Additional config files (-c)

Extra YAML files are passed with -c at runtime:

phases run -c local.yml                # single file
phases run -c local.yml,hpc.yml        # comma-separated list, merged left-to-right

Files are loaded after project.yaml and merged with deep-update (later files win).

Partial config (default)

A -c file is a partial config by default: its root keys are wrapped inside config: before merging.

# local.yml
preprocessing:
  dtype: float16
trainingParameter:
  learningRate: 0.0001

This is equivalent to placing this in the project.yaml:

config:
  preprocessing:
    dtype: float16
  trainingParameter:
    learningRate: 0.0001

Full config (isFullConfig: true)

When a -c file must touch top-level project keys (add phases, change exporters, etc.), declare it as a full config:

# cluster.yml
isFullConfig: true

config:
  preprocessing:
    dtype: float16

exporter:
  - name: HDF5RecordExporter
    system: false

isFullConfig: true disables the automatic config: wrapping.

---

importBefore / importAfter

Any YAML file can pull in sibling files relative to its own path:

# project.yaml
name: my-study

importBefore:
  - defaults/base.yml      # merged first — can be overridden by this file
  - defaults/channels.yml

importAfter:
  - local.yml              # merged last — wins over this file
  - secrets/paths.yml

config:
  dataversion:
    version: v1

Merge order for one file:

importBefore[0] → importBefore[1] → … → this file → importAfter[0] → importAfter[1] → …

• importBefore / importAfter keys are stripped from the parsed dict before merging and never appear in project.config.
• Imports are recursive — each imported file may itself declare importBefore / importAfter.
• Paths are always resolved relative to the importing file’s directory.

Tip

It is adivisable to only have import statements in the project.yaml and avoid them in imported files, to prevent complex recursive loading patterns.

---

Full loading sequence

When running phases run -c local.yml,hpc.yml, the config is assembled in this order:

1. project.yaml  (root / full config)
   ├─ importBefore files
   └─ importAfter files

2. local.yml  (partial → wrapped in config:)
   ├─ importBefore files
   └─ importAfter files

3. hpc.yml  (partial → wrapped in config:)
   ├─ importBefore files
   └─ importAfter files

4. Plugin defaultConfig files  (loaded with setdefaults → never overwrite)

5. Environment variables  (PHASE_CONFIG_* → always overwrite)

6. --set values  (applied last → always overwrite)

---

How config drives cache hashes

Artifacts are stored under paths that encode a config hash derived from the keys the artifact depends on. Changing any of those keys invalidates the old artifact and triggers re-generation.

data.dependsOn

Defined in the project.yaml under each data artifact, dependsOn lists config keys or other data artifact names that affect the output of that artifact:

data:
  - name: data-processed
    dependsOn: [recordIds, preprocessing, labelChannels, dataversion.version]
  - name: modelState
    dependsOn:
      - data-processed # depends on all config values from data-processed
      - modelName
      - trainingParameter.learningRate

Tag string rules

Config value type	Representation in tag string
Missing / None	omitted
Scalar (str, int, float, bool)	special chars stripped
dict or list	sha1(str(value))[0:8] — 8-char hex
Another Data	recursively resolved to its own tag string

Tags are joined with - to form the tag string, which is appended to the artifact name:

data-processed<tagString>--<version>

Inspecting a data ID

phases explain data-processed    # full breakdown of every contributing key

Or from Python:

project.explainDataString("data-processed")

Example output:

Explaining tag string for Data 'data-processed':
  dataTags = ['allDBRecordIds', 'preprocessing', 'labelChannels', 'dataversion.version']
  -----------------------------------------
  • allDBRecordIds  →  Data dependency  (resolves to its own tags: ['useLoader', ...])
  • preprocessing   →  config value
      raw      = {'dtype': 'float32', 'steps': [...]}
      flatten  = 'a3f9b12c'  [dict/list → sha1[0:8]]
  • labelChannels   →  config value
      raw      = ['EEG Fpz-Cz', 'EOG horizontal']
      flatten  = 'b7c2d3e4'  [dict/list → sha1[0:8]]
  • dataversion.version  →  config value
      raw      = 'v1'
      flatten  = 'v1'
  -----------------------------------------
  Final tag string : 'sleepedf-v1-a3f9b12c-b7c2d3e4-v1'
  Data ID          : 'data-processedsleepedf-v1-a3f9b12c-b7c2d3e4-v1--current'

Saving the config snapshot for an artifact

# Writes every config key that contributed to modelState as JSON
project.saveConfig("model-config.json", dataName="modelState")

# Writes the entire project.config as JSON (default when dataName=None)
project.saveConfig("model-config.json", dataName=None)

For example the Training phase does this automatically for generateing the log-folder via modelStateConfig, ensuring the exact hyperparameter set saved for a given checkpoint is always stored alongside it.