Manipulate Data

Data manipulation runs after signal preprocessing (see Preprocessing). It operates on segmented data arrays (pairs of (X, Y)) and prepares them for the model input. These segments are either whole recordings (when recordWise: true) or fixed-length windows (when recordWise: false). There are two manipulation stages: segment manipulation (applied per segment) and batch manipulation (applied when forming a mini-batch for model input).

Shapes

If not otherwise modified by preprocessing or manipulation, the shapes of X and Y are:

• X has shape (samples, channels)
• Y has shape (samples, labels)

Tip

It is advisable to transform the shape to (batch, channels, samples) as the first step (e.g. via the built-in addBatchDimension) so the segment manipulation and batch manipulation are homogeneous in their expected input shapes.

How it works

Like SignalPreprocessing, manipulation is dispatched to methods on a DataManipulation class (see Swappable Classes for how to connect it to the framework):

pyPhasesML.DataManipulation   (base)
    └── SleePyPhases.DataManipulation   (adds hotEncode, znorm, MagScale, …)
            └── yourproject.DataManipulation   (your custom steps)

Configuring manipulation steps

recordWise: true              # process one full recording at a time (vs. per-segment)

segmentManipulation:
- name: fixedSize           # pad/trim to fixed (X, Y) dimensions
  sizeX: 806400
  sizeY: 840
- name: addBatchDimension   # insert batch dim → (1, samples, channels)

batchManipulation:
- name: toTensor            # convert segmetns of numpy arrays to PyTorch tensors

Steps are applied in order. Each step is a method on your DataManipulation class.

Config keys

Key	Type	Default	Description
recordWise	bool	false	If true, the dataset loader yields one full record per sample instead of fixed-length windows
segmentManipulation	list[{name, ...}]	[]	Steps applied to each (X, Y) segment before batching
batchManipulation	list[{name, ...}]	[]	Steps applied to assembled batches
segmentLength	int	—	Window length in samples (used when recordWise: false)
segmentLengthLabel	int	—	Label length in samples (used when recordWise: false)
segmentPaddingValue	list[float]	[0, 0]	Padding before and after the segment, for asymmetric padding
manipulationAfterPredict	list[{name, ...}]	[]	Steps applied to model output after inference (e.g. toNumpy)

Whole records and fixed-length segments

The recordWise config key controls whether the data manipulation yields whole recordings or fixed-length segments as input.

recordWise: true

Value	Behaviour
true	The dataset yields one full recording per sample. The model receives the entire night. Use with segmentManipulation to pad/trim to a fixed size.
false	The dataset yields fixed-length windows. Use with segmentLength.

For windowed models, add:

recordWise: false
segmentLength: 3000 # window length in samples
segmentLengthLabel: 1 # label length in samples (e.g. 1 for one sleep stage per segment)
segmentPaddingValue: [19, 0] # add 19 segments before and 0 after each segment, so the current sleep stage has a context of 10 minutes

Writing manipulation methods

Create yourproject/DataManipulation.py:

import numpy as np
import torch
from SleePyPhases import DataManipulation as SPPDataManipulation

class DataManipulation(SPPDataManipulation):
    def fixedSize(self, X, Y, sizeX, sizeY,
                  fillValue=0, fillValueY=-1):
        """Pad or truncate (X, Y) to fixed dimensions."""
        # Pad X along time axis
        if X.shape[0] < sizeX:
            pad = np.full((sizeX - X.shape[0], *X.shape[1:]),
                          fillValue, dtype=X.dtype)
            X = np.concatenate([X, pad])
        else:
            X = X[:sizeX]

        # Pad Y
        if Y.shape[0] < sizeY:
            pad = np.full((sizeY - Y.shape[0], *Y.shape[1:]),
                          fillValueY, dtype=Y.dtype)
            Y = np.concatenate([Y, pad])
        else:
            Y = Y[:sizeY]

        return X, Y

    def toTensor(self, X, Y):
        """Convert numpy arrays to PyTorch tensors on the active device."""
        X = torch.tensor(X, dtype=torch.float32).permute(0, 2, 1)
        Y = torch.tensor(Y, dtype=torch.float32)
        return X, Y

    def toNumpy(self, X, Y):
        """Convert tensors back to numpy (used after inference)."""
        return X.cpu().numpy(), Y.cpu().numpy()

Method signature: (self, X, Y, **kwargs) where extra kwargs come from the name: ... dict in config.

Return the modified (X, Y) tuple.

Built-in steps (from SleePyPhases.DataManipulation)

Method	Description
addBatchDimension	Add leading batch dimension to X

Next step

→ Define a model — write a model class and configure training