Signal Exporter

How pyPhases maps Python types to persistent storage

An exporter is the persistence layer for a specific Python type. When a phase calls registerData, pyPhases picks the right exporter based on the type of the value and writes it to disk. When a phase calls getData, the same exporter reads it back.

This keeps your phase code type-agnostic: you work with plain Python objects, NumPy arrays, or DataFrames; the framework handles serialisation.

Registration in project.yaml

Exporters are declared in the top-level exporter: list. The simplest form is just a class name (resolved from the project’s own exporter/ package or from a plugin):

exporter:
  - PickleExporter           # short form — no extra config
  - PandasExporter
  - name: MemmapRecordExporter
    basePath: data/          # flat key-value config

When the phases CLI loads the project it instantiates each entry in order and calls project.registerExporter(obj). Exporters that ship inside a plugin are usually registered programmatically inside the plugin’s Plugin.py instead (see Writing a Plugin).

The exporter interface

All exporters subclass DataExporter and must implement three methods.

checkType(type) → bool

Called by project.getExporterForType(theType) to find the right exporter for a given Python type. Return True if this exporter handles that type:

def checkType(self, type):
    return type in [str, int, float, list, dict]

If multiple exporters match, the one with which is called (registerExporter(exporter, priority=200)) with the highest priority wins (default 100).

write(dataId, data, options={})

Persist data to dataId (a config-based string identifier). Usually the dataId is used directly for file names or database keys. This method is called when registerData is ready to save:

def write(self, dataId, data, options={}):
    with open(self.getPath(dataId), "wb") as f:
        pickle.dump(data, f)

read(dataId, options={}) → any

Reload from the persistent storage. Raise DataNotFound if the file is absent, so the framework can trigger regeneration if needed.

def read(self, dataId, options={}):
    path = self.getPath(dataId)
    try:
        with open(path, "rb") as f:
            return pickle.load(f)
    except FileNotFoundError:
        raise DataNotFound(f"Data not found: {path}")

How registerData and getData trigger exporters

phase.registerData("my-result", value)
  └─ project.getExporterForInstance(value)   # finds exporter via checkType(type(value))
       └─ exporter.write(dataId, value)       # dataId = name + config-hash

phase.getData("my-result", MyType)
  └─ project.getExporterForType(MyType)      # same lookup, by type
       └─ exporter.read(dataId)              # returns the persisted value

If the data is already in the in-memory cache, getData returns it directly without hitting the exporter. The exporter is only reached on a cache miss.

Implementing a custom exporter

Subclass DataExporter and implement the three required methods:

from pyPhases.exporter.DataExporter import DataExporter
from pyPhases.Data import DataNotFound
import json, pathlib

class JsonExporter(DataExporter):

    def checkType(self, type):
        return type in [dict, list]

    def write(self, dataId, data, options={}):
        path = self.getPath(dataId)
        path.parent.mkdir(parents=True, exist_ok=True)
        path.write_text(json.dumps(data))

    def read(self, dataId, options={}):
        path = self.getPath(dataId)
        if not path.exists():
            raise DataNotFound(f"Data not found: {path}")
        return json.loads(path.read_text())

self.getPath(dataId) resolves to basePath / dataId. The basePath is passed via the options dict when the exporter is instantiated (e.g. JsonExporter({"basePath": "./data"})).

To make the exporter available to a project, it needs to be registered before usage:

project.registerExporter(JsonExporter({"basePath": dataPath}))

To make the exporter available to all projects that use the current projecct as plugin, register it inside the Plugin.py.

For a complete walkthrough of creating and registering a plugin — including how Plugin.py is structured and loaded — see Writing a Plugin.