Preprocess Signals

Signal preprocessing runs during the Extract phase. It operates on raw RecordSignal objects. The main preprocessing stage is per-channel, where you can apply steps like filtering and resampling. There is also a whole-record manipulation stage for operations that require the full signal (e.g. trimming based on sleep events or discarding records on specific criteria). After preprocessing, all recordings of a single dataset are saved as a single file to data-processed and data-features, either by a MemMap or H5 writer (see Big File Extractor).

How it works

SleePyPhases dispatches preprocessing steps to methods on a SignalPreprocessing class. You extend the base class provided by SleePyPhases (which itself extends pyPhasesRecordloader’s base) and add or override methods. For more details on how to connect Signal Processing to SleePyPhases, see Swappable Classes.

pyPhasesRecordloader.SignalPreprocessing   (base)
        └── SleePyPhases.SignalPreprocessing   (adds rls, iir, fftConvolution)
                └── yourproject.SignalPreprocessing   (your custom steps)

Configuring preprocessing steps

All preprocessing is driven by the preprocessing config block:

preprocessing:
  targetFrequency: 32          # Hz — target sample rate for all channels
  labelFrequency: 0.03333      # Hz — 1/30 for 30-second epoch labels
  stepsPerType:
    eog: [resample, bandpass]  # list of method names to call, in order
    eeg:                       # applied to all EEG channels
      - name: bandpass         # different annotation style: dict with name and method params
        low: 0.3               # parameters passed to the method
        high: 35.0
      - name: resample               
    emg: []
    EEG Fpz-Cz: [resample]     # applied to specific channel by name
  manipulationSteps:
    - name: trimSPT            # post-resample whole-record manipulations
      before: 30               # seconds to keep before first sleep event
      after: 30                # seconds to keep after last sleep event
  targetChannels:
    - [EEG Fpz-Cz]            # which channels survive to the dataset

Each entry in stepsPerType maps a channel type or channel name to a list of method names. The method is looked up on your SignalPreprocessing instance and called with (signal, recordSignal, **kwargs).

Config keys

Key	Type	Default	Description
`preprocessing.targetFrequency`	`float`	—	Target sample rate in Hz after resampling
`preprocessing.labelFrequency`	`float`	—	Label frequency in Hz (e.g. `0.03333` = 1 epoch/30 s)
`preprocessing.stepsPerType`	`dict[str, list[str]]`	`{}`	Per-channel-type ordered list of preprocessing method names
`preprocessing.manipulationSteps`	`list[{name}]`	`[]`	Whole-record manipulation steps applied after per-channel steps
`preprocessing.targetChannels`	`list[list[str]]`	—	Channels that are kept; others are discarded

Warning

Target frequency only set the values for the recordSignal.targetFrequency, but does not trigger any resampling by itself. You must add a resampling method to the stepsPerType for it to be applied.

Writing preprocessing methods

Create yourproject/SignalPreprocessing.py:

from scipy.signal import resample
from SleePyPhases import SignalPreprocessing as SPPSignalPreprocessing

class SignalPreprocessing(SPPSignalPreprocessing):
    def resample(self, signal, recordSignal):
        """Resample signal to recordSignal.targetFrequency."""
        factor = recordSignal.targetFrequency / signal.frequency
        signal.signal = resample(signal.signal,
                                 int(len(signal.signal) * factor))
        signal.frequency = recordSignal.targetFrequency

    def bandpass(self, signal, recordSignal, low=0.3, high=35.0):
        """Example: apply a bandpass filter."""
        from scipy.signal import butter, filtfilt
        nyq = signal.frequency / 2
        b, a = butter(4, [low / nyq, high / nyq], btype="band")
        signal.signal = filtfilt(b, a, signal.signal)

Method signature: (self, signal, recordSignal) where:

signal.signal — numpy array of raw samples
signal.frequency — original sample frequency (Hz)
recordSignal.targetFrequency — target frequency from config

Modify signal.signal and signal.frequency in place.

Built-in steps (from `SleePyPhases.SignalPreprocessing`)

Method	Description
`rls`	Recursive Least Squares adaptive filter
`iir`	IIR filter
`fftConvolution`	Frequency-domain convolution

Event Manipulation and whole record manipulation (PreManipulation)

Preprocessing steps that operate on the whole record (e.g. trimming, merging events, etc.) can be added as methods to a PreManipulation class and configured in the preprocessing.manipulationSteps list. See Swappable Classes for details on how to connect this to SleePyPhases.


from pyPhasesRecordloader import RecordSignal, Event, ChannelsNotPresent

from SleePyPhases import PreManipulation as sppPreManipulation

class PreManipulation(SPPPreManipulation):
    def trimSPT(self, recordSignal: RecordSignal, events: List[Event], before = 0, after = 0):
        allSleepStages = ["R", "N1", "N2", "N3", "W"]
        sleepEvents = [e for e in events if e.name in allSleepStages]
        
        offsetStart = sleepEvents[0].end()-before if sleepEvents[0].name == "W" else 0
        offsetEnd = sleepEvents[-1].start+ after if sleepEvents[-1].name == "W" else sleepEvents[-1].end()

        offsetStart = max(offsetStart, 0)
        epochModMax = recordSignal.getShape()[1] // recordSignal.targetFrequency // 30 * 30 - offsetStart
        offsetEnd = min(offsetEnd, sleepEvents[-1].end(), epochModMax)

        recordSignal.signalOffset(offsetStart, offsetEnd, offsetFrequency=1)
        events = self._trimEventsByOffset(events, offsetStart, offsetEnd)

        return recordSignal, events

Combine channels

The combineChannels config key allows you to specify methods that combine multiple channels into a single one, which is part of the data harmonization process. For example, you can derive a bipolar EEG channel from two monopolar channels:

combineChannels:
  - name: F3-M2
    combineType: derived
    type: eeg
    channel: [F3, M2]

Feature channels

featureChannels allows ML-based or transform-based feature extraction inside preprocessing. The methodname is specified using the name field. The method receives the full RecordSignal object, so it can access, add or manipulate any channel. By default the channels field specifies the name(s) of the new channel(s) to be created which should be returend by the method.

featureChannels:
  - name: spectogram
    epoch_second: 30
    win_size: 2
    overlap: 1
    channel: [EEG, EEG Fpz-Cz]
    channels: [spectogram-eeg]

This example config extracts spectrogram features from the EEG and EEG Fpz-Cz channels using a sliding window approach with 2-second windows and 1-second overlap. The resulting spectrogram features are saved as new channels named spectogram-eeg.

from pyPhasesML import FeatureExtraction as pyPhasesFeatureExtraction
from pyPhasesRecordloader import RecordSignal, Signal

from SleePyPhases import FeatureExtraction as pyPhasesFeatureExtraction

class FeatureExtraction(pyPhasesFeatureExtraction):
    def spectogram(self, recordSignal: RecordSignal, channel, epoch_second, win_size, overlap):
        import numpy as np

        signal = recordSignal.getFirstSignalByName(channel)
        fs = signal.frequency
        X = signal.signal
        # see sleeptransformer for full implementation:
        # https://gitlab.com/sleep-is-all-you-need/reproduce/spp-sleeptranformer/-/blob/main/src/FeatureExtraction.py
        # ...
        spectrograms = np.transpose(spectrograms, (0, 1, 2))

        return spectrograms

The Feature channels can be used to generate compute heavy features such as machine learning features that are stored before the training.

Tip

When using GPU for feature extraction use following one of following configurations to avoid GPU memory issues:

Extract:
  useMultiThreading: false # no multi-threading depending on gpu memory

# or 

Extract:
  useMultiThreading: true
  spawnMultiprocessing: true
  threads: 4 # set to a number small enough to fit in gpu memory

Wiring the classes

The Init phase wires your class into the framework (already generated by phases create):

from SleePyPhases import SignalPreprocessing as SP, PreManipulation as PM
from myproject.SignalPreprocessing import SignalPreprocessing
from myproject.PreManipulation import PreManipulation

class Init(Phase):
    def prepareConfig(self):
        SP.setClass(SignalPreprocessing)
        PM.setClass(PreManipulation)

Next step

→ Manipulate data — segment, normalize, and format tensors for ML

How it works

Configuring preprocessing steps

Config keys

Writing preprocessing methods

Built-in steps (from SleePyPhases.SignalPreprocessing)

Event Manipulation and whole record manipulation (PreManipulation)

Combine channels

Feature channels

Wiring the classes

Next step

Built-in steps (from `SleePyPhases.SignalPreprocessing`)