Architecture

This is the living architecture document for plesty-lib. It records the module structure, the public contracts between the layers, and the reasoning behind the design choices — so that future changes are made with the design, or change this document first. Detailed usage of each part lives in the guide pages (Device, Experiment, Data & Metadata Schemas); this page stays at the level of boundaries and decisions.

High-level module view

Everything ships under the plesty.lib namespace package:

Package	Role
plesty.lib.utils	Foundation utilities: logging setup, error handling, the process-wide ResourceRegistry, EnvSettings, the module/run configuration loader (module_config, load_yaml), DLL helpers.
plesty.lib.data	Data containers and persistence: PlestyArray, tables, units, dtype helpers, and the result store (save_result / ResultDocument, optional HDF5 export).
plesty.lib.traffic	Transport backends (traffic managers): Serial, VISA, USB utilities, raw TCP/IP, and Thorlabs APT (binary framing, apt_protocol). Moves bytes; knows nothing about parameters or operations.
plesty.lib.solver	Protocol translation: the CmdSolver / OpSolver ABCs and concrete solvers (SCPISolver, ICEBLOCKSolver) that turn standardized calls into vendor commands.
plesty.lib.device	The core device tier: BaseDeviceSyncModel with its three system mixins (ConfigSystem, FunctionSystem, TelemetrySystem), the render-agnostic doc_model(), async wrappers, ready-made SCPI base devices, and CompositeDevice.
plesty.lib.service	Remote access: the ZMQ-based device TCP/IP server and client, plus the per-device ResourceManager for multi-client access control.
plesty.lib.analyzer	Data reduction: the synchronous Analyzer ABC — schema-declared transforms from raw device data to meaningful data, with input/output validation and per-result provenance stamps.
plesty.lib.experiment	Reproducible orchestration: the async Experiment ABC with Plan/Step schedules and the crash-safe Journal.
plesty.lib.monitor	Live views: the Monitor ABC — schema-declared projections of arriving data onto declared traces — and the sources that feed it (RunSource, TelemetrySource, PushSource). Read-only and toolkit-free.
plesty.lib.ui	GUI framework: the Panel contract, the dockable Shell home window, the Theme carrying the plesty.net design tokens, and the PySide6/pyqtgraph backend behind the gui extra.
plesty.lib.sim	Stand-ins for hardware: DataGenerator (schema-driven synthetic data for mock solvers and tests), DemoDevice (the standard demo device tests build on instead of hand-writing one), and AptMotorSimulator (a Thorlabs APT motor answering real protocol frames).
plesty.lib.test	Shipped test helpers and the standard contract pipelines that hub modules import: DevicePipeline, ExperimentPipeline, AnalyzerPipeline (mock, hardware-free), plus FieldTestPipeline, client_field_test and field_test_concurrency for what only an instrument can prove.

Layering

The library is layered bottom-up; each tier may depend on the tiers below it, never above:

Tier	Packages	May import
5 — presentation	ui	monitor, data, utils — and, only inside ui.qt, a GUI toolkit. Nothing imports ui.
4 — orchestration	experiment, test, monitor	everything below (test additionally uses sim and service; monitor uses data and reads what experiment persisted, but never runs one)
3 — remote access	service	device, utils
2 — device core	device	data, solver, traffic, service, utils
1 — protocol & transport	solver, traffic, sim, analyzer	device metadata dataclasses (ConfigParameter, FuncMeta, …), data, utils (analyzer imports data only — it never touches transport or devices)
0 — foundation	utils, data	standard library (+ numpy for data); nothing else in the library

Two edges deserve a note. device and service are a deliberate pair rather than a strict layer: the TCP/IP server wraps a device (via the async wrapper), while CompositeDevice consumes TCP/IP clients — the packages reference each other, and this coupling is contained to composite_device.py on the device side. Similarly, solver sits beside device rather than below it: solvers depend only on the metadata dataclasses (ConfigParameter, FuncMeta) that the device systems define, not on the device model itself.

The device tier itself is split into six layers above the vendor hardware — traffic manager, command solver, synchronized device, async wrapper, TCP/IP server, TCP/IP client — of which only the bottom two are written per device:

The landing page walks through this figure and the structure of the synchronized device (imgs/device_standard.svg).

Interface contracts

The contracts below are the load-bearing public API. Each entry says what the contract guarantees and where the detailed guide lives — the guides, not this page, document parameters and usage.

BaseDeviceSyncModel + system mixins — plesty.lib.device.base_device_sync. Every device inherits the abstract model, which composes three orthogonal systems: ConfigSystem (schema-declared, validated parameters — Parameter system), FunctionSystem (schema-registered operations plus @expose_to_api methods — Function system), and TelemetrySystem (cheap status() snapshots and hookable events — see Data & Metadata Schemas). Subclasses implement connect/disconnect/_write_/_query_/check_errors/check_operatability; write/query validate, log, parse, cache, and emit telemetry around them. The context manager (with device:) owns the resource-lock lifecycle. See Base device.

doc_model() / DeviceDocModel — plesty.lib.device.doc. The single, render-agnostic source of truth for a device’s user-facing documentation: parameters grouped by config group, functions (schema-registered and custom operations unified as FuncDoc), and standard_methods (the common device API). Renderers — the SDK docs generator, summary(), the server describe endpoint — consume it instead of re-deriving metadata. See Base device.

Solver contracts — plesty.lib.solver. CmdSolver builds write/query command strings from a ConfigParameter; OpSolver.solve_request resolves an operation request dict into a response dict. SCPISolver.solve_func additionally dispatches any operation whose whole exchange is one SCPI command and a scalar response, driven by a FuncMeta built from the "command" field of the operation schema — no hand-written dispatch for simple operations. See Command solver.

CompositeDevice remote-call contract — plesty.lib.device.composite_device. Groups local devices and remote TCP/IP clients behind one object. call(dev, func, ...) guarantees that a request timeout on a remote sub-device triggers a client rebuild (flushing the stale ZMQ reply) and a bounded retry before the final TimeoutError propagates; reconnect(dev) and connect_client() back it. See Composite device.

Experiment / Plan / Journal lifecycle — plesty.lib.experiment. build_plan() yields a frozen, content-hashed Plan of Steps; run() validates every Step.op against the experiment’s own public methods before any device is touched, journals each step to an append-only JSONL log, and run(resume=<run_id>) skips journaled-complete steps while refusing a plan whose hash changed. teardown() is always awaited. See Experiment framework.

Analyzer transform contract — plesty.lib.analyzer. Subclasses declare input_schema/output_schema (device-schema dtype vocabulary) and implement one synchronous analyze(); the validating __call__ checks inputs, fills output metadata from the schema, and stamps every array/table result with a provenance dict (analyzer class, package version, constructor params — calibration included). Analyzers consume data-layer objects and never import device drivers. See Analyzer framework.

Monitor live-view contract — plesty.lib.monitor. Subclasses declare input_schema (device-schema dtype vocabulary) and implement traces() (what is drawn, once) plus update() (what to draw per frame); submit() validates the frame, checks the returned trace keys, and fans out to every attached Renderer. Sources expose one non-blocking poll(), so the framework owns no thread and no clock. Monitors are strictly read-only — RunSource follows the result documents an experiment already committed, which is why watching cannot disturb measuring. See Monitor framework.

Shell and panels — plesty.lib.ui. Panel is the dockable unit (anything worth a sub-window, not only a view); Shell is the home window that docks, floats, tabs, and persists their arrangement, ticking each panel on one timer. Theme reads the packaged palette and stylesheet assets, so no colour or markup is written in Python. The Qt backend lives behind the gui extra and nothing else in the library imports it. See Shell framework.

Result persistence — plesty.lib.data.io. save_result writes the blob first (atomically) and the typed ResultDocument JSON last — the JSON file is the commit record. load_document reads metadata without blob I/O; convert_to_hdf5 optionally packs documents into one archive. See Data & Metadata Schemas.

Contract test pipelines — plesty.lib.test. DevicePipeline (eight mock gates, SDK gate d1), ExperimentPipeline (five hardware-free gates, SDK gate E1), and AnalyzerPipeline (five gates on schema-generated synthetic inputs, groundwork for SDK gate a1) define what every hub module must prove: schema integrity, mock round-trips, lifecycle (device tier); deterministic plans, resolvable ops, serializability (experiment tier); analyze-signature match, round-trip validation, provenance (analyzer tier). A device module exposes one pytest function calling run_mock_pipeline(); experiment and analyzer modules expose one function per gate. What a mock cannot reach — the command-format contract, transport pathologies, resource contention — belongs to FieldTestPipeline on the instrument, client_field_test over the wire, and field_test_concurrency for devices sharing a host. See Test helpers.

Design Decisions

Decisions are recorded with the alternative that lost and why. Revisit a decision by adding a dated amendment, not by silently diverging.

System mixins instead of a deep inheritance chain

BaseDeviceSyncModel composes ConfigSystem, FunctionSystem, and TelemetrySystem as cooperating mixins with explicitly chained __init__s, rather than stacking them in an inheritance ladder (ConfigurableDevice → FunctionDevice → …). The systems are orthogonal concerns, so a ladder would impose an arbitrary order and force every subclass to take all rungs; with mixins, each system is independently testable (the test helpers exercise ConfigSystem alone) and adding TelemetrySystem in this release required no change to existing device subclasses. The cost is a wider attribute surface on one class and the risk of name collisions — real enough that the system solver attributes were renamed to _cmd_solver/_op_solver after a clash with user-defined device methods (lib#1).

A structured doc_model() instead of generating docs directly

Device documentation used to be whatever summary() rendered. This release introduces DeviceDocModel — data, not markup — and keeps rendering out of the library. The alternative (each renderer walking the parameter and function registries itself, or the device emitting Markdown) had already produced drift: the server describe endpoint, summary(), and the SDK docs generator each re-derived metadata slightly differently. One extraction path also let framework plumbing be separated from device operations (standard_methods vs functions) once, for all renderers. Tradeoff: an extra indirection, and summary() has not yet been migrated onto the model, so two extraction paths coexist until it is.

Resource lock acquired on connect, not on construction

ResourceRegistry.acquire() runs in __enter__ (released in __exit__ and on a failed connect), no longer in __init__. Locking at construction made instantiating a device a side effect — documentation generators, offline tooling, and tests could not create a device object without claiming the hardware key, and a crashed process could leave a stale claim (now also mitigated: a dead weak reference is reclaimed). The tradeoff is honest: the conflict between two instances of the same device id now surfaces later, at connect time, and code that calls connect() directly instead of using the context manager bypasses the lock entirely — the guarantee is tied to the with lifecycle.

Resilient CompositeDevice remote calls instead of failing fast

A ZMQ REQ/REP client that times out leaves a stale reply in the socket; the naive client was permanently broken afterwards (lib#10). CompositeDevice.call therefore rebuilds the client and retries on TimeoutError (lib#15) rather than propagating immediately: in a multi-hour scan, a transient network hiccup on one sub-device should not abort the whole run, and before this every hub composite re-implemented its own ad-hoc retry loop. Failing fast remains available — retries are bounded and the final timeout propagates. The real tradeoff is that a retried call is not guaranteed idempotent: if the server received the request but the reply timed out, the operation may execute twice. Callers with non-idempotent operations should pass retries=1.

Consumer-owned telemetry hooks instead of a background telemetry service

TelemetrySystem defines the schema (DeviceStatus, TelemetryEvent) and a synchronous hook mechanism — no polling threads, no transports, no service process (lib#4). A built-in telemetry daemon was rejected because it would impose lifecycle, threading, and transport choices on every device, and background polling can disturb timing-sensitive instrument I/O. Instead the cost model is strictly pay-per-use: with no hook registered, no event object is even constructed. The tradeoff: consumers who want periodic monitoring must run their own loop, and a slow hook slows the device call that emits it (failing hooks are logged and skipped, slow ones are the consumer’s problem).

JSONL journal + content-hashed plan instead of a database

Experiment state is an append-only JSONL journal next to a frozen plan.json (lib#3). A database would give queries and concurrent writers, but a lab experiment has one writer, and a database server (or even SQLite file locking on network shares) is exactly the kind of dependency that fails at 2 a.m. in an instrument room. Append-only JSONL is crash-safe by construction (a torn final line is detectable and discardable), diffable, and greppable; the SHA-256 content hash of the plan makes “the schedule silently changed” a refusable error on resume rather than a corrupted run. Replay cost is linear in journal length — acceptable at measurement scale, and a real limit if runs ever reach millions of steps.

Raw blob + JSON metadata, HDF5 as optional export — not mandatory HDF5

save_result keeps each result in its original raw format (.npy, native image bytes, or values inlined in JSON) described by a typed ResultDocument, with convert_to_hdf5 as an opt-in packing step. An HDF5-first store was rejected: it would make h5py a hard dependency of every device module, re-encode camera output that is already a valid PNG, and concentrate crash risk in one archive file — whereas the two-file blob-then-document protocol is atomic per result. The tradeoff is many small files per run instead of one archive, and consumers who want HDF5 must run the conversion step.

Allowed dependencies

The sanctioned third-party runtime dependencies (from pyproject.toml):

Dependency	Why it is allowed
numpy	The array baseline of scientific Python; backs PlestyArray and the data layer.
pyserial	Serial-port transport for the serial traffic manager.
pyvisa	VISA instrument transport (GPIB/USB-TMC/…); the de-facto standard for SCPI instruments.
pyusb	Raw USB access and device-reset utilities in the traffic layer.
pyzmq	Messaging for the device TCP/IP server/client; battle-tested REQ/REP with timeouts.

pytest and plesty-sdk are development-group dependencies only and must never become runtime requirements.

Policy: a new runtime library may be introduced only if it is actively maintained — visible upstream activity (release or commits) within the last two years — and it must be added to the table above in the same merge request. Every dependency also passes pip-audit on each push (plesty check gate 10), so a library that stops receiving security fixes will be flagged and must be replaced or vendored.

Optional dependencies

Every module in the hub depends on plesty.lib, so a package added to the table above is installed on every lab machine running Plesty. A library that only one feature needs — or that is heavy, compiled, or platform-specific — is therefore declared as an extra in [project.optional-dependencies] instead, imported lazily inside the function that needs it, and raises an ImportError naming the extra when it is missing. The base install stays small and the dependency is still visible to resolvers, lockfiles, and audits.

Extra	Adds	Needed by
hdf5	h5py	convert_to_hdf5 — an export step; the primary store is raw blob + JSON.

The packages behind these extras are mirrored in the optional dependency group, which [tool.uv] default-groups installs by default. CI runs a bare uv sync --frozen, which installs no extras — without the mirror, every optional code path would go untested there.

Technical debt

Technical debt lives in the issue tracker, not in this document: every debt item is filed as a GitLab issue labeled type::debt — per the governance DoD, permanent architectural debt is filed before the shortcut lands. One standing caveat matters when reading this document: the DevicePipeline gate design reaches main for the first time with v0.3.0 after an earlier revert — treat its gate list as provisional until the redesign pass (lib#16).