Load Data

SleePyPhases uses a plugin system to load PSG recordings from any supported dataset. You configure which loader to use in project.yaml. Each loader plugin is designed for a specific dataset or vendor. The loader outputs harmonized RecordSignal objects that are passed to the rest of the pipeline in a consistent format, regardless of the original dataset.

Configure a loader

Add the loader plugin to the plugins list and set the path:

# project.yaml
name: simplecnn
plugins:
  - pyPhasesRecordloaderSleepEDF   # dataset-specific loader
  - pyPhasesRecordloader           # base loader support
  - SleepHarmonizer                # channel harmonization
  - pyPhasesML
  - SleePyPhases

config:
  useLoader: sleepedf              # which loader to activate
  sleepedf-path: /data/sleepedf   # path to dataset root

The useLoader value must match the loader’s registered name. For SleepEDF it is sleepedf.

Dataset split

Control how records are divided into train/val/test folds using the dataversion key:

config:
  dataversion:
    version: Default-split   # arbitrary label — part of cache hash
    seed: null             # random seed for shuffling (null = deterministic)
    folds: 5               # number of cross-validation folds
    split:
      trainval: "0:170"    # record indices for training + validation
      test: "170:197"      # record indices held out for testing

The version string is included in data cache filenames, and just exist for easier identification. Even so it is not consumed, change the name will still trigger a full cache rebuild, so only change it when you want to invalidate the cache.

Available loaders

Plugin package	`useLoader` value	Dataset
`pyPhasesRecordloaderSleepEDF`	`sleepedf`	Sleep-EDF (PhysioNet)
`pyPhasesRecordloadershhs`	`shhs`	SHHS
`pyPhasesRecordloadermesa`	`mesa`	MESA
`pyPhasesRecordloadermros`	`mros`	MrOS
`pyPhasesRecordloadercfs`	`cfs`	CFS
`pyPhasesRecordloaderCAP`	`cap`	CAP (PhysioNet)
`pyPhasesRecordloaderDOD`	`dod`	DOD
`pyPhasesRecordloaderphysionet2018`	`physionet2018`	PhysioNet 2018
`pyPhasesRecordloaderphysionet2023`	`physionet2023`	PhysioNet 2023
`pyPhasesRecordloaderPhysio2026`	`physio2026`	PhysioNet 2026
`pyPhasesRecordloaderAlice`	`alice`	Alice 6 (vendor)
`pyPhasesRecordloaderDomino`	`domino`	Domino (vendor)
`pyPhasesRecordloaderNox`	`nox`	Noxturnal (vendor)
`pyPhasesRecordloaderProfusion`	`profusion`	Profusion (vendor)
`pyPhasesRecordloaderSonata`	`sonata`	Sonata (vendor)

→ For vendor loaders that need per-device channel mappings see Vendor Config YAML.

Which channels to use

Two config keys control which channels are available at different stages:

useSourceChannels (optimization) — tells the record loader which raw signals to read from disk. Signals not listed are never loaded and cannot be accessed later. Derivations computed from multiple source channels (e.g. F4-M1) require every contributing channel to appear in useSourceChannels.

config:
  useSourceChannels:
    - EEG Fpz-Cz           # channel names as stored in the raw recordings
  labelChannels:
    - SleepStagesAASM      # annotation channel to use as ground truth

preprocessing.targetChannels — defines which channels are retained and stored in the extracted dataset cache. Channels not listed are discarded after preprocessing. Each entry is a list of alternative names — the first name found in the record is used, enabling harmonization across datasets:

preprocessing:
  targetChannels:
    - [EEG Fpz-Cz, EEG1]   # SleepEDF name first, MESA fallback
    - [EOG horizontal, EOG-L]

Tip

Best practice: keep a clean, dataset-specific targetChannels list per dataset entry rather than long cross-dataset alias lists.

Multi-dataset training

Use datafold.datasets to list each dataset as its own entry. Each entry overrides any parts of the config (common are: loader, dataversion, channel config and custom segment manipulation).

datafold:
  shuffle: true
  seed: 2026
  datasets:
    - config:                           # SleepEDF
        useLoader: sleepedf
        sleepedf-path: /data/sleepedf
        dataversion:
          version: sleepedf-split
          split: {trainval: "0:170", test: "170:197"}
        useSourceChannels:
          - EEG Fpz-Cz
        preprocessing:
          targetChannels:
            - [EEG Fpz-Cz]
        segmentManipulation:
          - name: selectChannel
            channel: 0
    - config:                           # SHHS (different recording site)
        useLoader: shhs
        shhs-path: /data/shhs
        dataversion:
          version: shhs-split
          split: {trainval: "0:3000", test: "3000:3500"}
        useSourceChannels:
          - EEG
          - EEG(sec)
        preprocessing:
          targetChannels:
            - [EEG]
            - [EEG(sec)]
        segmentManipulation:
          - name: selectChannel
            channel: 1

Tip

To use a dataset purely as a hold-out test set within datafold, omit trainval and set only a test range:

dataversion:
  split:
    test: "0:3500"   # all records go to test — no training data from this dataset

External evaluation with `evalOn`

To evaluate a trained model on a completely separate cohort, use evalOn in a dedicated config file. This is separate from the training datafold setup and does not interfere with the training cache. This allows extending the test set with new records without invalidating the training cache, and also allows evaluating on datasets that are not part of the training datafold at all.

Create a file per external dataset, e.g. configs/datasets/mesa/evalOn.yaml:

useSourceChannels: null    # load all channels (or list specific ones)

evalOn:
  useLoader: mesa
  datasetSplit: test       # which split to evaluate on
  dataversion:
    version: Default
    seed: 2
    split:
      test: "0:2500"

  preprocessing:
    targetChannels:
      - [EEG1]             # harmonized name for this dataset
      - [EOG-L]
      - [EOG-R]
      - [EMG]

  segmentManipulationEval:
    - name: addBatchDimension
    - name: changeType
      dtype: float32
    - name: selectChannels
      channels: [0, 1, 2]
    - name: znorm

Run evaluation with:

phases run -c configs/datasets/mesa/evalOn.yaml EvalReport

Building a new loader

If your dataset is not in the list above, see New Record Loader to create a loader package.

Next step

→ Preprocess signals — resample, filter, and normalize raw channels

Configure a loader

Dataset split

Available loaders

Which channels to use

Multi-dataset training

External evaluation with evalOn

Building a new loader

Next step

External evaluation with `evalOn`