Dataset Splits

The dataversion config block controls how a dataset is divided into train, validation, and test sets.

Basic configuration

dataversion:
  version: my-split-v1   # label for identification, part of cache hash
  seed: null             # random seed for shuffling records before splitting (null = no shuffle)
  folds: 5               # number of cross-validation folds
  split:
    trainval: "0:170"    # record index range for training + validation
    test: "170:197"      # record indices held out for testing

The split ranges use Python slice notation over the ordered list of record IDs returned by the loader. If using explicit validation, training and test splits, multiple ranges per split can be defined.

Available split keys

The splitting is dynamic and flexible and you can use your own split keys and names, if you have other requirements. For example adding a split for pre-training. While it is possible you are responsible for handling any custom split logic in your code. The framework only recognizes the following keys for special handling:

Split name	Description
training	records used for training only
test	records held out for testing (not used training and thresholdoptimization, only for final evaluation)
validation	records reserved for validation during training and thresholdoptimization
trainval	records used for training and validation (subject to further splitting if `validationSplit` or `fold` is set)
trainvaltest	records used for training, validation, and testing (subject to further splitting if `validationSplit`/``or`fold` is set)

Advanced split options

Key	Type	Description
`folds`	`int`	Number of CV folds (default: 1 — single train/val split)
`seed`	`int \\| null`	Seed for shuffling the record list before splitting
`groupBy`	`str \\| null`	Metadata field to group records by before splitting (e.g. `patient` or `site`) — prevents data leakage when a patient appears in multiple recordings
`recordIds`	`list[str] \\| null`	Explicit list of record IDs to include (overrides the full dataset list)
`validationSplit`	`float \\| null`	Fraction of `trainval` to reserve as a fixed validation set (e.g. `0.1` = 10%). When set, the rest of `trainval` is used for training only, and folds are not applied.
`testSplit`	`float \\| null`	Fraction of `test` to reserve as a fixed validation set (e.g. `0.2` = 20%).
`filterQuery`	`str \\| null`	Pandas query string applied to the record metadata DataFrame before splitting (e.g. `recordId.str.start == 'shhs1' and ahi > 15`)

Grouping to prevent leakage

When a dataset contains multiple recordings per subject, a naive random split can place the same subject in both training and test sets. Use groupBy to split at the subject level:

dataversion:
  version: subject-split
  groupBy: patient          # any field name in the record metadata
  folds: 5
  split:
    trainval: "0:100"       # 100 subjects in trainval
    test: "100:110"         # 10 subjects held out

Fixed validation split

For studies that use a fixed validation set rather than k-fold CV:

dataversion:
  version: fixed-val
  validationSplit: 0.111    # 11.1% of trainval → validation
  split:
    trainval: "0:900"
    test: "900:1000"

Selecting a subset of records

To work with a curated list of record IDs.

dataversion:
  version: curated-v2
  recordIds:
    - shhs1-200001
    - shhs1-200002
    - shhs1-200003
    # ...
  split:
    trainval: "0:80"
    test: "80:100"

Filtering by metadata

Apply a pandas query string to include only records matching a condition:

dataversion:
  version: adults-only
  filterQuery: "age >= 18"
  split:
    trainval: "0:500"
    test: "500:600"

The query is evaluated against the metadata DataFrame returned by the loader.

Named splits and custom logic

For more complex splitting logic (e.g. uneven folds or specific records reserved for testing), you can define any named splits and handle them in your code. For example:


dataversion:
  version: Default
  seed: 29
  groupBy: null
  recordIds: null
  namedFolds:
    Auxiliary1:
      test: "0:100"
      validation: "100:200"
      training: "200:994"
    Auxiliary2:
      test: "0:100"
      validation: "300:400"
      training: ["100:300", "400:994"]
    Auxiliary3:
      test: "0:100"
      validation: "600:700"
      training: ["100:600", "700:994"]
    Auxiliary4:
      test: "0:100"
      validation: "894:994"
      training: "100:894"

Validating splits and preventing data leakage

By default, pyphases validates your split configuration at dataset construction time. If any record appears in more than one split, or if any record is missing from all splits, an error is raised with the list of offending record IDs. When groupBy is set, the RecordLoader groups records by the specified metadata field (e.g. patient ID), which prevents data leakage by ensuring that all records from the same group are assigned to the same split.

validateDataset: true    # default true — runs split validation BuildDataset
dataversion:
  version: my-split-v1
  split:
    trainval: "0:170"
    test: "170:197"

The validation checks three properties:

Check	Description
Completeness	Every record in the dataset appears in at least one split. No records are silently dropped.
Uniqueness	No record appears in more than one split at the same time (e.g. a record cannot be in both `training` and `test`).
No data leakage	When `groupBy` is set, no group (e.g. patient) spans multiple splits.

If any check fails an error is raised with the list of offending record IDs or groups before any training starts. To disable this validation, set validateDataset: false in the root config.