# Data & Metadata Schemas `plesty-lib` defines one schema contract per data domain so that every HUB module acquires, stores, and exchanges data in the same shape. Three domains are covered (issue lib#4): | Domain | Schema | Module | |---|---|---| | Device configuration | `ConfigParameter` / `ConfigGroup` | `plesty.lib.device.params` | | Operational telemetry | `DeviceStatus` / `TelemetryEvent` | `plesty.lib.device.telemetry` | | Experimental output | `ResultDocument` | `plesty.lib.data.io` | | Experiment data | `ExperimentData` / `SpectrumData` | `plesty.lib.data.experiment_data`, `plesty.lib.data.spectrum` | ## Device configuration Structured representation of device parameters and settings — see [Parameter System](../device/param_system.md). Every parameter is a `ConfigParameter` (name, dtype, unit, constraints, command binding) organised into `ConfigGroup`s and managed by the `ConfigSystem` that every device model composes. Device modules declare their parameters in a YAML/JSON schema which `ConfigSystem` materialises at init. ## Operational telemetry Real-time status, state, and diagnostic data, defined in `plesty.lib.device.telemetry`. Two shapes cover pull and push: * **`DeviceStatus`** — a point-in-time snapshot. Every device inherits `status()` via the `TelemetrySystem` mixin; the default implementation is assembled from the connection check and the cached configuration values (no device I/O), so it is always cheap to call: ```python status = device.status() status.connection # "connected" | "disconnected" | "connecting" | "error" status.state # "ready", "busy", ... (override device_state() to refine) status.parameters # last-known parameter values status.last_error # most recent error message, if any ``` * **`TelemetryEvent`** — a timestamped record of kind `reading`, `state_change`, `error`, or `warning`. Consumers register a hook; the base device emits `state_change`/`reading` events automatically on every successful `write`/`query`, and devices can emit custom events with `emit_telemetry()`: ```python events = [] device.register_telemetry_hook(events.append) device.write("wavelength", 850.0) # emits a state_change event from plesty.lib.device.telemetry import append_events append_events(run_dir / "telemetry.jsonl", events) ``` Hooks run synchronously and never disturb device operation (a failing hook is logged and skipped). When no hook is registered, no event is constructed. `append_events`/`read_events` persist events in the same crash-tolerant append-only JSONL format as the experiment journal. Polling loops, threads, and transports are deliberately out of scope — they belong to monitors and services, not to the schema. ## Experimental output Acquired data plus its metadata, defined in `plesty.lib.data.io`. The storage contract keeps blobs in their **original raw format** (`.npy` for `PlestyArray`, native encoding for image bytes) described by a JSON metadata document — the typed `ResultDocument` model — holding the Plesty meta (name, unit, range, description), timestamps, provenance, and a reference link to the blob: ```python from plesty.lib.data import save_result, load_result, load_document path = save_result(array, run_dir / "step_0001", provenance={"step_id": "s1"}) document = load_document(path) # metadata only — no blob I/O result = load_result(path) # reconstructed PlestyArray ``` The JSON document is the commit record: a result exists once its `.json` file does, and writes are crash-safe (blob first, document last, both atomic). `convert_to_hdf5` optionally packs a set of documents into a single HDF5 archive for exchange; it needs the `hdf5` extra (`uv add "plesty-lib[hdf5]"`) and raises an `ImportError` naming it otherwise. An `ExperimentData` result (below) is stored as one `.npz` blob; its document has `type: "experiment_data"` and names the `kind` and `schema_version` in `meta`, so a step returning a `SpectrumData` is committed and restored as one record. ## Experiment data A measurement is usually more than one array — a spectrum is an intensity *and* the wavelength axis it was taken on, plus the exposure and the file it came from. `ExperimentData` is the contract for such typed data, so a device can hand it to an experiment, the experiment can persist it as one record, and an analyzer can declare it as one input. A subclass declares a unique `kind` and implements `to_arrays`, `from_arrays` and `example`. Every instance carries `metadata` (acquisition context) and `provenance` (origin), and inherits: | | | |---|---| | Native formats | `save` / `load` by suffix: `.npz` (lossless), `.json`, `.csv`, `.txt` — text tables start with a `# plesty:` JSON header line | | Registry | `ExperimentData.load(path)` rebuilds the subclass that wrote the file; a file of a newer `schema_version` is refused | | Schema dtype | the `kind` resolves in `resolve_dtype`, e.g. `{"dtype": "spectrum"}` | | Foreign formats | a subclass's `readers` table, e.g. `.spe` for spectra | | Report columns | a subclass's `columns`: array name → `label`, `unit`, `description`, `result`, `error`; an analyzer's report (see [Reports](../analysis/reports.md)) tabulates the data from it | ### Example: `SpectrumData` A 1-D `wavelength` axis (default `nm`), an `intensity` of one spectrum `(pixels,)` or a stack `(frames, pixels)` (default `counts`), and optional `coordinates` — the sweep value of each frame, by name. ```python from plesty.lib.data import SpectrumData # From an experiment run: each record's raw device file, with its sweep values spectrum = SpectrumData.from_run(run_dir, coordinates=("hwp_deg", "power_w")) spectrum.coordinates["hwp_deg"] # one angle per frame, unit deg prepared = spectrum.crop(770, 780).subtract_background("median") prepared.find_peaks(prominence=100) prepared.save("qd1.npz") ``` - **From a run** — `from_run(run, file_key="data_file", coordinates=(...), where={...})` stacks, in record order, the raw file each record names: a `.spe` file, a camera `.npy` frame (rows summed per pixel; pass `wavelength=`, the file carries no axis) or any native file. A record whose value is a `SpectrumData` is used as is. Coordinates take their unit from the name's suffix (`_deg`, `_w`, `_nm`, `_s`, …). - **Reads files** — LightField `.spe` 3.0 (regions summed per pixel when multi-row; acquisition settings into `metadata`), LightField's CSV export (`Frame,Wavelength,Intensity`), legacy exports that open each frame with a `Frame` line (whitespace columns, decimal commas allowed), and plain wide tables (wavelength, then one column per frame). WinSpec SPE 2.x files are refused; `.spe` is never written. `SpeFile.read` exposes every region and the raw XML footer. - **Processing** returns a new spectrum and appends the step to `metadata["processing"]`: `frame`, `combine`, `crop`, `subtract_background`, `normalize`; `energy()` gives the axis in eV. Coordinates follow the frames — `frame` keeps its one value, `combine` drops them. - **Peaks**: `find_peaks` (NumPy only) locates lines by height, prominence and distance; `peak_prominences` measures found lines, and `match_peaks` relocates known positions to the strongest nearby maximum. Fitting a line is analysis, not data: the `SpectrumFit` analyzer does it (see [SpectrumFit](../analysis/spectrum_fit.md)).