Writing a Plugin

How to package extensions — exporters, phases, and loaders — as a reusable plugin

A plugin is a Python package that extends a pyPhases project at load time. It can register custom exporters, add phase classes, configure record loaders, and set default config values. Plugins are declared in project.yaml and loaded automatically when the project starts.

Anatomy of a plugin package

A minimal plugin package has this structure:

my-plugin/
  my_plugin/
    Plugin.py          # entry point — must be named Plugin.py
    exporter/
      MyExporter.py
    config.yaml        # optional default config
  setup.py / pyproject.toml

The package name and the importable module name must match the string used in project.yaml.

The Plugin.py entry point

Plugin.py must define a class named Plugin that subclasses PluginAdapter:

# my_plugin/Plugin.py
from pyPhases import PluginAdapter, Project

class Plugin(PluginAdapter):
    defaultConfig = "config.yaml"   # optional — loaded with setdefaults

    def __init__(self, project: Project, options=None):
        super().__init__(project, options)
        # Register phases, exporters, listeners here
        # self.project is available immediately

    def initPlugin(self):
        # Called after all plugins are loaded and config is fully merged.
        # Safe to read config values here.
        pass

__init__ is called while the project is being constructed. It is the right place to register exporters and phases because those need to be in place before any phase runs.

initPlugin is called after the full config is assembled. Use it for setup that depends on config values (e.g. reading a file path from config and registering a record loader).

Registering an exporter

Call project.registerExporter(instance) inside __init__. The exporter with the highest priority (default 100) is tried first when getExporterForType is called. Pass the exporter instance with its options (typically just basePath):

from pyPhases import PluginAdapter, Project
from my_plugin.exporter.JsonExporter import JsonExporter

class Plugin(PluginAdapter):

    def initPlugin(self):
        data_path = self.project.getConfig("data-path", "./data")
        jsonExporter = JsonExporter({"basePath": data_path})
        # priority=200 ensures this exporter is tried before any default ones
        self.project.registerExporter(jsonExporter, priority=200)

Declaring plugin in project.yaml

Add the module name to the plugins list. The name must be the importable Python package name. The easiest way to do this is to install it using pip (e.g. pip install -e . from the plugin directory), or publish it to PyPI and install it. If the module already exists, only the name needs to be added to the project.yaml file.

plugins:
  - my_plugin          # loads my_plugin.Plugin.Plugin

Options can be passed as a dict:

plugins:
  - name: my_plugin
    options:
      someKey: someValue

Options are forwarded as the second argument to Plugin.__init__ and are accessible via self.getOption("someKey").

Default config (defaultConfig)

Set defaultConfig to a YAML file path (relative to Plugin.py) to supply default values. These are merged with setdefaults — they never overwrite values already set by project.yaml or additional config files:

# my_plugin/config.yaml
my-plugin-threshold: 0.5
my-plugin-path: ./data/my-plugin
class Plugin(PluginAdapter):
    defaultConfig = "config.yaml"

The framework resolves the path relative to the plugin package directory and loads it automatically before initPlugin is called.

Registering phases

Add phase instances directly to the project inside __init__:

from pyPhases import Data
from my_plugin.phases.Analyse import Analyse

class Plugin(PluginAdapter):

    def initPlugin(self):
      dependsOn = ["data-processed"]  # dataId that this phase depends on
      export_data = [Data("analysis-result", self.project, dependsOn)]
      phase = Analyse(export_data)
      self.project.addPhase(phase)

If the name of the phase class (e.g. Analyse) is already taken an error is raised.

Full example

The SleePyPhases plugin registers all its phases and exporters in one Plugin.__init__:

# SleePyPhases/SleePyPhases/Plugin.py  (simplified)
from pyPhases import Data, PluginAdapter, Project
from pyPhases.exporter.PickleExporter import PickleExporter
from pyPhases.exporter.PandasExporter import PandasExporter
from pyPhasesML.exporter.MemmapRecordExporter import MemmapRecordExporter
from SleePyPhases.phases.Extract import Extract
from SleePyPhases.phases.BuildDataset import BuildDataset
# ...

class Plugin(PluginAdapter):
    defaultConfig = "config.yaml"

    def __init__(self, project: Project, options=None):
        super().__init__(project, options)

        exports = [Data("data-processed", project, ["allDBRecordIds", "preprocessing"])]
        project.addPhase(Extract(exports))
        # ...

        data_path = project.getConfig("data-path", "./data")
        project.registerExporter(PickleExporter({"basePath": data_path}))
        project.registerExporter(PandasExporter({"basePath": data_path}))
        project.registerExporter(MemmapRecordExporter({"basePath": data_path}))