Testing an Analyzer

An analyzer declares what it takes and gives in its schemas (see The contract), so much of it can be checked without real data: that the schemas are valid, that analyze() accepts exactly the declared inputs, and that its outputs match what it declares. AnalyzerPipeline, in plesty.lib.test.analyzer_pipeline, runs these checks on inputs it generates from the schema.

Test	Checks
`test_analyzer_subclass`	the module has a public `Analyzer` subclass that can be created with the given settings
`test_schema_integrity`	both schemas are non-empty, their types are known, their shapes are well-formed
`test_analyze_signature`	`analyze()` takes exactly the inputs `input_schema` declares
`test_mock_roundtrip`	generated inputs produce outputs that match `output_schema`
`test_provenance_stamped`	the outputs carry their provenance

An analyzer module exposes one pytest test per check:

# tests/test_pipeline.py
import numpy as np
from plesty.lib.test.analyzer_pipeline import AnalyzerPipeline

from plesty.power_peak import PowerPeak

PIPELINE = AnalyzerPipeline(PowerPeak, mock_inputs={"power": np.array([0.002, 0.008, 0.004])})

def test_analyzer_subclass():
    PIPELINE.test_analyzer_subclass()

def test_schema_integrity():
    PIPELINE.test_schema_integrity()

def test_analyze_signature():
    PIPELINE.test_analyze_signature()

def test_mock_roundtrip():
    PIPELINE.test_mock_roundtrip()

def test_provenance_stamped():
    PIPELINE.test_provenance_stamped()

Inputs the analyzer needs to be realistic, such as an image with actual features or a power curve with a peak, are given in mock_inputs. Every other input is generated: zeros of the declared shape, or a neutral number. Arguments after the class, other than mock_inputs, are passed to its constructor. PIPELINE.run_mock_pipeline() runs all five checks at once.

plesty-sdk, PLESTY’s developer tool, looks for these five tests and runs them as gate a1 of plesty check before the module may be published (see Which tests each module runs).