Data & Metadata Schemas

plesty-lib defines one schema contract per data domain so that every HUB module acquires, stores, and exchanges data in the same shape. Three domains are covered (issue lib#4):

Domain	Schema	Module
Device configuration	ConfigParameter / ConfigGroup	plesty.lib.device.params
Operational telemetry	DeviceStatus / TelemetryEvent	plesty.lib.device.telemetry
Experimental output	ResultDocument	plesty.lib.data.io
Experiment data	ExperimentData / SpectrumData	plesty.lib.data.experiment_data, plesty.lib.data.spectrum

Device configuration

Structured representation of device parameters and settings — see Parameter System. Every parameter is a ConfigParameter (name, dtype, unit, constraints, command binding) organised into ConfigGroups and managed by the ConfigSystem that every device model composes. Device modules declare their parameters in a YAML/JSON schema which ConfigSystem materialises at init.

Operational telemetry

Real-time status, state, and diagnostic data, defined in plesty.lib.device.telemetry. Two shapes cover pull and push:

• DeviceStatus — a point-in-time snapshot. Every device inherits status() via the TelemetrySystem mixin; the default implementation is assembled from the connection check and the cached configuration values (no device I/O), so it is always cheap to call:

  status = device.status()
  status.connection   # "connected" | "disconnected" | "connecting" | "error"
  status.state        # "ready", "busy", ... (override device_state() to refine)
  status.parameters   # last-known parameter values
  status.last_error   # most recent error message, if any
  

• TelemetryEvent — a timestamped record of kind reading, state_change, error, or warning. Consumers register a hook; the base device emits state_change/reading events automatically on every successful write/query, and devices can emit custom events with emit_telemetry():

  events = []
  device.register_telemetry_hook(events.append)
  device.write("wavelength", 850.0)   # emits a state_change event
  
  from plesty.lib.device.telemetry import append_events
  append_events(run_dir / "telemetry.jsonl", events)
  

Hooks run synchronously and never disturb device operation (a failing hook is logged and skipped). When no hook is registered, no event is constructed. append_events/read_events persist events in the same crash-tolerant append-only JSONL format as the experiment journal. Polling loops, threads, and transports are deliberately out of scope — they belong to monitors and services, not to the schema.

Experimental output

Acquired data plus its metadata, defined in plesty.lib.data.io. The storage contract keeps blobs in their original raw format (.npy for PlestyArray, native encoding for image bytes) described by a JSON metadata document — the typed ResultDocument model — holding the Plesty meta (name, unit, range, description), timestamps, provenance, and a reference link to the blob:

from plesty.lib.data import save_result, load_result, load_document

path = save_result(array, run_dir / "step_0001", provenance={"step_id": "s1"})
document = load_document(path)   # metadata only — no blob I/O
result = load_result(path)       # reconstructed PlestyArray

The JSON document is the commit record: a result exists once its .json file does, and writes are crash-safe (blob first, document last, both atomic). convert_to_hdf5 optionally packs a set of documents into a single HDF5 archive for exchange; it needs the hdf5 extra (uv add "plesty-lib[hdf5]") and raises an ImportError naming it otherwise.

An ExperimentData result (below) is stored as one .npz blob; its document has type: "experiment_data" and names the kind and schema_version in meta, so a step returning a SpectrumData is committed and restored as one record.

Experiment data

A measurement is usually more than one array — a spectrum is an intensity and the wavelength axis it was taken on, plus the exposure and the file it came from. ExperimentData is the contract for such typed data, so a device can hand it to an experiment, the experiment can persist it as one record, and an analyzer can declare it as one input.

A subclass declares a unique kind and implements to_arrays, from_arrays and example. Every instance carries metadata (acquisition context) and provenance (origin), and inherits:

Native formats	save / load by suffix: .npz (lossless), .json, .csv, .txt — text tables start with a # plesty: JSON header line
Registry	ExperimentData.load(path) rebuilds the subclass that wrote the file; a file of a newer schema_version is refused
Schema dtype	the kind resolves in resolve_dtype, e.g. {"dtype": "spectrum"}
Foreign formats	a subclass’s readers table, e.g. .spe for spectra
Report columns	a subclass’s columns: array name → label, unit, description, result, error; an analyzer’s report (see Reports) tabulates the data from it

Example: SpectrumData

A 1-D wavelength axis (default nm), an intensity of one spectrum (pixels,) or a stack (frames, pixels) (default counts), and optional coordinates — the sweep value of each frame, by name.

from plesty.lib.data import SpectrumData

# From an experiment run: each record's raw device file, with its sweep values
spectrum = SpectrumData.from_run(run_dir, coordinates=("hwp_deg", "power_w"))
spectrum.coordinates["hwp_deg"]                  # one angle per frame, unit deg

prepared = spectrum.crop(770, 780).subtract_background("median")
prepared.find_peaks(prominence=100)
prepared.save("qd1.npz")

• From a run — from_run(run, file_key="data_file", coordinates=(...), where={...}) stacks, in record order, the raw file each record names: a .spe file, a camera .npy frame (rows summed per pixel; pass wavelength=, the file carries no axis) or any native file. A record whose value is a SpectrumData is used as is. Coordinates take their unit from the name’s suffix (_deg, _w, _nm, _s, …).

• Reads files — LightField .spe 3.0 (regions summed per pixel when multi-row; acquisition settings into metadata), LightField’s CSV export (Frame,Wavelength,Intensity), legacy exports that open each frame with a Frame line (whitespace columns, decimal commas allowed), and plain wide tables (wavelength, then one column per frame). WinSpec SPE 2.x files are refused; .spe is never written. SpeFile.read exposes every region and the raw XML footer.

• Processing returns a new spectrum and appends the step to metadata["processing"]: frame, combine, crop, subtract_background, normalize; energy() gives the axis in eV. Coordinates follow the frames — frame keeps its one value, combine drops them.

• Peaks: find_peaks (NumPy only) locates lines by height, prominence and distance; peak_prominences measures found lines, and match_peaks relocates known positions to the strongest nearby maximum. Fitting a line is analysis, not data: the SpectrumFit analyzer does it (see SpectrumFit).