# Architecture

This is the living architecture document for `plesty-lib`. It records the
module structure, the public contracts between the layers, and the reasoning
behind the design choices — so that future changes are made *with* the
design, or change this document first. Detailed usage of each part lives in
the guide pages ([Device](device/index.md), [Experiment](experiment.md),
[Data & Metadata Schemas](data_schemas.md)); this page stays at the level of
boundaries and decisions.

## High-level module view

Everything ships under the `plesty.lib` namespace package:

| Package | Role |
|---|---|
| `plesty.lib.utils` | Foundation utilities: logging setup, error handling, the process-wide `ResourceRegistry`, `EnvSettings`, the module/run configuration loader (`module_config`, `load_yaml`), DLL helpers. |
| `plesty.lib.data` | Data containers and persistence: `PlestyArray`, tables, units, dtype helpers, and the result store (`save_result` / `ResultDocument`, optional HDF5 export). |
| `plesty.lib.traffic` | Transport backends (traffic managers): Serial, VISA, USB utilities, raw TCP/IP, and Thorlabs APT (binary framing, `apt_protocol`). Moves bytes; knows nothing about parameters or operations. |
| `plesty.lib.solver` | Protocol translation: the `CmdSolver` / `OpSolver` ABCs and concrete solvers (`SCPISolver`, `ICEBLOCKSolver`) that turn standardized calls into vendor commands. |
| `plesty.lib.device` | The core device tier: `BaseDeviceSyncModel` with its three system mixins (`ConfigSystem`, `FunctionSystem`, `TelemetrySystem`), the render-agnostic `doc_model()`, async wrappers, ready-made SCPI base devices, and `CompositeDevice`. |
| `plesty.lib.service` | Remote access: the ZMQ-based device TCP/IP server and client, plus the per-device `ResourceManager` for multi-client access control. |
| `plesty.lib.analyzer` | Data reduction: the synchronous `Analyzer` ABC — schema-declared transforms from raw device data to meaningful data, with input/output validation and per-result provenance stamps. |
| `plesty.lib.experiment` | Reproducible orchestration: the async `Experiment` ABC with `Plan`/`Step` schedules and the crash-safe `Journal`. |
| `plesty.lib.monitor` | Live views: the `Monitor` ABC — schema-declared projections of arriving data onto declared traces — and the sources that feed it (`RunSource`, `TelemetrySource`, `PushSource`). Read-only and toolkit-free. |
| `plesty.lib.ui` | GUI framework: the `Panel` contract, the dockable `Shell` home window, the `Theme` carrying the plesty.net design tokens, and the PySide6/pyqtgraph backend behind the `gui` extra. |
| `plesty.lib.sim` | Stand-ins for hardware: `DataGenerator` (schema-driven synthetic data for mock solvers and tests), `DemoDevice` (the standard demo device tests build on instead of hand-writing one), and `AptMotorSimulator` (a Thorlabs APT motor answering real protocol frames). |
| `plesty.lib.test` | Shipped test helpers and the standard contract pipelines that hub modules import: `DevicePipeline`, `ExperimentPipeline`, `AnalyzerPipeline` (mock, hardware-free), plus `FieldTestPipeline`, `client_field_test` and `field_test_concurrency` for what only an instrument can prove. |

### Layering

The library is layered bottom-up; each tier may depend on the tiers below
it, never above:

| Tier | Packages | May import |
|---|---|---|
| 5 — presentation | `ui` | `monitor`, `data`, `utils` — and, only inside `ui.qt`, a GUI toolkit. Nothing imports `ui`. |
| 4 — orchestration | `experiment`, `test`, `monitor` | everything below (`test` additionally uses `sim` and `service`; `monitor` uses `data` and reads what `experiment` persisted, but never runs one) |
| 3 — remote access | `service` | `device`, `utils` |
| 2 — device core | `device` | `data`, `solver`, `traffic`, `service`, `utils` |
| 1 — protocol & transport | `solver`, `traffic`, `sim`, `analyzer` | device metadata dataclasses (`ConfigParameter`, `FuncMeta`, …), `data`, `utils` (`analyzer` imports `data` only — it never touches transport or devices) |
| 0 — foundation | `utils`, `data` | standard library (+ `numpy` for `data`); nothing else in the library |

Two edges deserve a note. `device` and `service` are a deliberate pair
rather than a strict layer: the TCP/IP server wraps a device (via the async
wrapper), while `CompositeDevice` consumes TCP/IP clients — the packages
reference each other, and this coupling is contained to
`composite_device.py` on the device side. Similarly, `solver` sits *beside*
`device` rather than below it: solvers depend only on the metadata
dataclasses (`ConfigParameter`, `FuncMeta`) that the device systems define,
not on the device model itself.

The device tier itself is split into six layers above the vendor hardware —
traffic manager, command solver, synchronized device, async wrapper, TCP/IP
server, TCP/IP client — of which only the bottom two are written per device:

![Device layer architecture: hosting machine, hardware, clients and network](imgs/device_layers.svg)

The [landing page](index.md) walks through this figure and the structure of
the synchronized device (`imgs/device_standard.svg`).

## Interface contracts

The contracts below are the load-bearing public API. Each entry says what
the contract guarantees and where the detailed guide lives — the guides, not
this page, document parameters and usage.

**`BaseDeviceSyncModel` + system mixins** —
`plesty.lib.device.base_device_sync`. Every device inherits the abstract
model, which composes three orthogonal systems: `ConfigSystem` (schema-declared,
validated parameters — [Parameter system](device/param_system.md)),
`FunctionSystem` (schema-registered operations plus `@expose_to_api`
methods — [Function system](device/func_system.md)), and `TelemetrySystem`
(cheap `status()` snapshots and hookable events — see
[Data & Metadata Schemas](data_schemas.md)). Subclasses implement
`connect`/`disconnect`/`_write_`/`_query_`/`check_errors`/`check_operatability`;
`write`/`query` validate, log, parse, cache, and emit telemetry around them.
The context manager (`with device:`) owns the resource-lock lifecycle. See
[Base device](device/base_device.md).

**`doc_model()` / `DeviceDocModel`** — `plesty.lib.device.doc`. The single,
render-agnostic source of truth for a device's user-facing documentation:
`parameters` grouped by config group, `functions` (schema-registered and
custom operations unified as `FuncDoc`), and `standard_methods` (the common
device API). Renderers — the SDK docs generator, `summary()`, the server
`describe` endpoint — consume it instead of re-deriving metadata. See
[Base device](device/base_device.md).

**Solver contracts** — `plesty.lib.solver`. `CmdSolver` builds write/query
command strings from a `ConfigParameter`; `OpSolver.solve_request` resolves
an operation request dict into a response dict. `SCPISolver.solve_func`
additionally dispatches any operation whose whole exchange is one SCPI
command and a scalar response, driven by a `FuncMeta` built from the
`"command"` field of the operation schema — no hand-written dispatch for
simple operations. See [Command solver](device/cmd_solver.md).

**`CompositeDevice` remote-call contract** —
`plesty.lib.device.composite_device`. Groups local devices and remote
TCP/IP clients behind one object. `call(dev, func, ...)` guarantees that a
request timeout on a remote sub-device triggers a client rebuild (flushing
the stale ZMQ reply) and a bounded retry before the final `TimeoutError`
propagates; `reconnect(dev)` and `connect_client()` back it. See
[Composite device](device/composite_device.md).

**`Experiment` / `Plan` / `Journal` lifecycle** — `plesty.lib.experiment`.
`build_plan()` yields a frozen, content-hashed `Plan` of `Step`s; `run()`
validates every `Step.op` against the experiment's own public methods before
any device is touched, journals each step to an append-only JSONL log, and
`run(resume=<run_id>)` skips journaled-complete steps while refusing a plan
whose hash changed. `teardown()` is always awaited. See
[Experiment framework](experiment.md).

**`Analyzer` transform contract** — `plesty.lib.analyzer`. Subclasses
declare `input_schema`/`output_schema` (device-schema dtype vocabulary) and
implement one synchronous `analyze()`; the validating `__call__` checks
inputs, fills output metadata from the schema, and stamps every array/table
result with a `provenance` dict (analyzer class, package version,
constructor params — calibration included). Analyzers consume data-layer
objects and never import device drivers. See
[Analyzer framework](analyzer.md).

**`Monitor` live-view contract** — `plesty.lib.monitor`. Subclasses declare
`input_schema` (device-schema dtype vocabulary) and implement `traces()`
(what is drawn, once) plus `update()` (what to draw per frame); `submit()`
validates the frame, checks the returned trace keys, and fans out to every
attached `Renderer`. Sources expose one non-blocking `poll()`, so the
framework owns no thread and no clock. Monitors are strictly read-only —
`RunSource` follows the result documents an experiment already committed,
which is why watching cannot disturb measuring. See
[Monitor framework](monitor.md).

**Shell and panels** — `plesty.lib.ui`. `Panel` is the dockable unit
(anything worth a sub-window, not only a view); `Shell` is the home window
that docks, floats, tabs, and persists their arrangement, ticking each panel
on one timer. `Theme` reads the packaged palette and stylesheet assets, so no
colour or markup is written in Python. The Qt backend lives behind the `gui`
extra and nothing else in the library imports it. See
[Shell framework](ui.md).

**Result persistence** — `plesty.lib.data.io`. `save_result` writes the
blob first (atomically) and the typed `ResultDocument` JSON last — the JSON
file is the commit record. `load_document` reads metadata without blob I/O;
`convert_to_hdf5` optionally packs documents into one archive. See
[Data & Metadata Schemas](data_schemas.md).

**Contract test pipelines** — `plesty.lib.test`. `DevicePipeline` (eight
mock gates, SDK gate d1), `ExperimentPipeline` (five hardware-free gates,
SDK gate E1), and `AnalyzerPipeline` (five gates on schema-generated
synthetic inputs, groundwork for SDK gate a1) define what every hub module
must prove: schema integrity, mock round-trips, lifecycle (device tier);
deterministic plans, resolvable ops, serializability (experiment tier);
analyze-signature match, round-trip validation, provenance (analyzer tier).
A device module exposes one pytest function calling `run_mock_pipeline()`;
experiment and analyzer modules expose one function per gate. What a mock
cannot reach — the command-format contract, transport pathologies, resource
contention — belongs to `FieldTestPipeline` on the instrument,
`client_field_test` over the wire, and `field_test_concurrency` for devices
sharing a host. See [Test helpers](device/test_helper.md).

## Design Decisions

Decisions are recorded with the alternative that lost and why. Revisit a
decision by adding a dated amendment, not by silently diverging.

### System mixins instead of a deep inheritance chain

`BaseDeviceSyncModel` composes `ConfigSystem`, `FunctionSystem`, and
`TelemetrySystem` as cooperating mixins with explicitly chained `__init__`s,
rather than stacking them in an inheritance ladder
(`ConfigurableDevice → FunctionDevice → …`). The systems are orthogonal
concerns, so a ladder would impose an arbitrary order and force every
subclass to take all rungs; with mixins, each system is independently
testable (the test helpers exercise `ConfigSystem` alone) and adding
`TelemetrySystem` in this release required no change to existing device
subclasses. The cost is a wider attribute surface on one class and the risk
of name collisions — real enough that the system solver attributes were
renamed to `_cmd_solver`/`_op_solver` after a clash with user-defined
device methods ([lib#1](https://gitlab.com/plesty/core/plesty-lib/-/issues/1)).

### A structured `doc_model()` instead of generating docs directly

Device documentation used to be whatever `summary()` rendered. This release
introduces `DeviceDocModel` — data, not markup — and keeps rendering out of
the library. The alternative (each renderer walking the parameter and
function registries itself, or the device emitting Markdown) had already
produced drift: the server `describe` endpoint, `summary()`, and the SDK
docs generator each re-derived metadata slightly differently. One extraction
path also let framework plumbing be separated from device operations
(`standard_methods` vs `functions`) once, for all renderers. Tradeoff: an
extra indirection, and `summary()` has not yet been migrated onto the model,
so two extraction paths coexist until it is.

### Resource lock acquired on connect, not on construction

`ResourceRegistry.acquire()` runs in `__enter__` (released in `__exit__` and
on a failed connect), no longer in `__init__`. Locking at construction made
*instantiating* a device a side effect — documentation generators, offline
tooling, and tests could not create a device object without claiming the
hardware key, and a crashed process could leave a stale claim (now also
mitigated: a dead weak reference is reclaimed). The tradeoff is honest: the
conflict between two instances of the same device id now surfaces later, at
connect time, and code that calls `connect()` directly instead of using the
context manager bypasses the lock entirely — the guarantee is tied to the
`with` lifecycle.

### Resilient `CompositeDevice` remote calls instead of failing fast

A ZMQ REQ/REP client that times out leaves a stale reply in the socket; the
naive client was permanently broken afterwards
([lib#10](https://gitlab.com/plesty/core/plesty-lib/-/issues/10)).
`CompositeDevice.call` therefore rebuilds the client and retries on
`TimeoutError` ([lib#15](https://gitlab.com/plesty/core/plesty-lib/-/issues/15))
rather than propagating immediately: in a multi-hour scan, a transient
network hiccup on one sub-device should not abort the whole run, and before
this every hub composite re-implemented its own ad-hoc retry loop. Failing
fast remains available — retries are bounded and the final timeout
propagates. The real tradeoff is that a retried call is not guaranteed
idempotent: if the server received the request but the reply timed out, the
operation may execute twice. Callers with non-idempotent operations should
pass `retries=1`.

### Consumer-owned telemetry hooks instead of a background telemetry service

`TelemetrySystem` defines the *schema* (`DeviceStatus`, `TelemetryEvent`)
and a synchronous hook mechanism — no polling threads, no transports, no
service process ([lib#4](https://gitlab.com/plesty/core/plesty-lib/-/issues/4)).
A built-in telemetry daemon was rejected because it would impose lifecycle,
threading, and transport choices on every device, and background polling
can disturb timing-sensitive instrument I/O. Instead the cost model is
strictly pay-per-use: with no hook registered, no event object is even
constructed. The tradeoff: consumers who want periodic monitoring must run
their own loop, and a slow hook slows the device call that emits it
(failing hooks are logged and skipped, slow ones are the consumer's
problem).

### JSONL journal + content-hashed plan instead of a database

Experiment state is an append-only JSONL journal next to a frozen
`plan.json` ([lib#3](https://gitlab.com/plesty/core/plesty-lib/-/issues/3)).
A database would give queries and concurrent writers, but a lab experiment
has one writer, and a database server (or even SQLite file locking on
network shares) is exactly the kind of dependency that fails at 2 a.m. in
an instrument room. Append-only JSONL is crash-safe by construction (a torn
final line is detectable and discardable), diffable, and greppable; the
SHA-256 content hash of the plan makes "the schedule silently changed"
a refusable error on resume rather than a corrupted run. Replay cost is
linear in journal length — acceptable at measurement scale, and a real
limit if runs ever reach millions of steps.

### Raw blob + JSON metadata, HDF5 as optional export — not mandatory HDF5

`save_result` keeps each result in its original raw format (`.npy`, native
image bytes, or values inlined in JSON) described by a typed
`ResultDocument`, with `convert_to_hdf5` as an opt-in packing step. An
HDF5-first store was rejected: it would make `h5py` a hard dependency of
every device module, re-encode camera output that is already a valid PNG,
and concentrate crash risk in one archive file — whereas the two-file
blob-then-document protocol is atomic per result. The tradeoff is many
small files per run instead of one archive, and consumers who want HDF5
must run the conversion step.

## Allowed dependencies

The sanctioned third-party **runtime** dependencies (from
`pyproject.toml`):

| Dependency | Why it is allowed |
|---|---|
| `numpy` | The array baseline of scientific Python; backs `PlestyArray` and the data layer. |
| `pyserial` | Serial-port transport for the serial traffic manager. |
| `pyvisa` | VISA instrument transport (GPIB/USB-TMC/…); the de-facto standard for SCPI instruments. |
| `pyusb` | Raw USB access and device-reset utilities in the traffic layer. |
| `pyzmq` | Messaging for the device TCP/IP server/client; battle-tested REQ/REP with timeouts. |

`pytest` and `plesty-sdk` are development-group dependencies only and must
never become runtime requirements.

**Policy:** a new runtime library may be introduced only if it is actively
maintained — visible upstream activity (release or commits) within the last
two years — and it must be added to the table above in the same merge
request. Every dependency also passes `pip-audit` on each push (`plesty
check` gate 10), so a library that stops receiving security fixes will be
flagged and must be replaced or vendored.

### Optional dependencies

Every module in the hub depends on `plesty.lib`, so a package added to the
table above is installed on every lab machine running Plesty. A library that
only one feature needs — or that is heavy, compiled, or platform-specific —
is therefore declared as an **extra** in `[project.optional-dependencies]`
instead, imported lazily inside the function that needs it, and raises an
`ImportError` naming the extra when it is missing. The base install stays
small and the dependency is still visible to resolvers, lockfiles, and
audits.

| Extra | Adds | Needed by |
|---|---|---|
| `hdf5` | `h5py` | `convert_to_hdf5` — an export step; the primary store is raw blob + JSON. |

The packages behind these extras are mirrored in the `optional` dependency
group, which `[tool.uv] default-groups` installs by default. CI runs a bare
`uv sync --frozen`, which installs no extras — without the mirror, every
optional code path would go untested there.

## Technical debt

Technical debt lives in the issue tracker, not in this document: every debt item is filed
as a GitLab issue labeled
[`type::debt`](https://gitlab.com/plesty/core/plesty-lib/-/issues/?label_name%5B%5D=type%3A%3Adebt)
— per the
[governance DoD](https://gitlab.com/plesty/governance/-/blob/main/CONTRIBUTING.md),
permanent architectural debt is filed before the shortcut lands. One standing caveat
matters when reading this document: the `DevicePipeline` gate design reaches `main` for
the first time with v0.3.0 after an earlier revert — treat its gate list as provisional
until the redesign pass
([lib#16](https://gitlab.com/plesty/core/plesty-lib/-/issues/16)).