Evaluate

The Eval phase involves loading the test fold, running model evaluation, and computing metric scores for segments and events. Additionally, clinical metrics are calculated during event evaluation, depending on the values of the classification.predictionSignals and classification.predictionFrequencies parameters. The EvalReport function generates evaluation results, stores them in CSV files and outputs a summary including the project configuration. All data is stored in an evaluation folder (eval-path, default ./eval), to ensure reproducibility and ease of later analysis.

Running evaluation

phases run EvalReport

Config keys

eval:
  batchSize: 1              # inference batch size (often 1 for record-wise)
  metrics:                  # optional override of metrics to compute
    - kappa
    - accuracy
    - f1

fixedThreshold:
  - 0.5                     # per-label thresholds for converting logits to labels
                            # set to null to use ThresholdOptimization phase

manipulationAfterPredict:
  - name: toNumpy           # steps applied to model output before metric computation
Key Type Default Description
eval.batchSize int 1 Batch size used during inference
eval.metrics list[str] inherits from training Metrics to compute on test fold
manipulationAfterPredict list[{name}] [] Data manipulation steps applied to model predictions
eventEval.manipulationAfterPredict list[{name}] [] Additional manipulation steps applied for event prediction
fixedThreshold list[float] or null null Fixed per-label classification thresholds. Set null to use ThresholdOptimization
optimizeThresholdFor list[str] or null null Classification label names to optimize thresholds (default all label channels)
thresholdMetric list[str] or str f1 The metric to be optimised, can be defined as an array of values for each label.
optimizeOn str validation Specifies the data to optimize the thresholds. Possible values: validation, training, trainval
startThresholdsAt list[int] [0.5, ...] The start value for optimisation for each label.
eventEval.augmentationCountThatShouldNotBeOptimized int 0 Only steps bigger this number are optimized, previous steps are cached

Threshold optimization

Instead of fixed thresholds, run the ThresholdOptimization phase to find optimal thresholds on the validation fold, this phase is run automatically if fixedThreshold is set to null and a threshold is required:

phases run ThresholdOptimization

Metrics

Standard metrics computed per class and averaged:

Metric Description
kappa Cohen’s kappa
accuracy Accuracy
f1 F1 score (macro)
micro_f1 F1 score (micro)
auroc Area under the ROC curve
auprc Area under precision-recall curve
eventCountDiff Difference in counting events

To define you own metric, see the custom metrics guide.

Clinical metrics

SleePyPhases also computes sleep-medicine–relevant summary metrics when eval.clinicalMetrics is set:

clinicalMetrics: [tst, waso, ahi, arI ]
Metric Description
tst Total Sleep Time
sLatency Sleep Latency
waso Wake After Sleep Onset
rLatency REM latency
sEfficiency Sleep Efficiency
ahi Apnea-Hypopnea index
indexArousal Arousal index
indexPLMS Periodicl leg movement index

The metrics are computed depending on classification.predictionSignals (see classification for more details on prediction signals). The SleepMetaData class contains a full list of clinical metrics and the channels that can calculate them.

Event-based evaluation

Event-level metrics run on default, to skip this step set enableEventEval: false for example for single-label sleep staging. The event evaluation converts the predicted and true label sequences into onset/offset events, then computes event-based metrics such as eventCountDiff and clinical metrics that require event-level data (e.g. AHI, WASO). Classical machine learning metrics (e.g. f1, kappa) are also computed on the event-level data.

enableEventEval: false

EvalReport

EvalReport reads the outputs of Eval and generates output for the evaluation.

phases run EvalReport

Output is written to the eval-path (default ./eval/) where a subfolder is generated that depends on the configuration.

Next steps