The field test every device runs

A device module can pass every mock gate and still fail the first time it meets its instrument. August closed that gap: one shared on-hardware gate sequence in plesty-lib, generated per module by plesty init field-test, and a sanitised report committed beside the module saying what was verified and how fast it ran — with nothing that says where it ran.

Three things arrived together. The gates themselves (plesty.lib.test.field_test) — eight of them, run against a connected instrument, producing a timing profile and per-command failure statistics rather than a pass/fail. A second tier (plesty.lib.test.client_field_test) that drives the device the way an experiment does, over the wire. And the generator that writes both tests for your module from one interview, because the gates were never the hard part — the configuration was.

Two tiers, and the difference is not a detail

▸For users: running one

You have a device module and the hardware in front of you.

1 · Generate the test — the interview

# in the module root, with the device importable
uv run plesty init field-test

Six to eight questions, depending on what your device offers. The schemas supply the candidates; you supply the judgement no inspection can make. Arrow keys move, space selects, Enter confirms.

2 · Run it

# host tier — address comes from .env, the same file the server reads
uv run python tests/field_test.py
uv run python tests/field_test.py --address mock       # the simulator
uv run python tests/field_test.py --gates param_roundtrip stability
uv run python tests/field_test.py --keep-going         # don't stop at the first failure

# client tier — run the host tier first; this one reads its report
uv run python tests/field_test_client.py                     # attached, from .env
uv run python tests/field_test_client.py --address tcp://<instrument-pc>:5555
uv run python tests/field_test_client.py --spawn             # loopback, no hardware

3 · What the run writes

The markdown and JSONL are for the person who just ran the thing: they carry the host, the address, the serial and the lab's paths, because that reader needs to know which rig produced them. They never enter git — a hub repository is world-readable, and a field-test report is a small inventory of the lab. Set PLESTY_REPORT_ARCHIVE in .env to also copy every run to a shared location, each in its own timestamped folder; an unmounted share warns and does not fail the run.

field-test.json is the published projection, and sanitisation is the library's job rather than yours: every field is either derived from the module's own source and schemas — already public — or is a measurement. The host name is deliberately not among the environment keys it may carry; which Windows, which Python and which library version explain a difference between two runs, while which machine locates a lab. A document that fails validation is refused rather than published.

4 · Reading the result

A real host-tier run of a powermeter module, as published:

outcome   { passed: 8, failed: 0, skipped: 0, ok: true }
gates     discovery pass · connect_lifecycle pass · param_roundtrip pass
          param_constraints pass · functions pass · buffer_drain pass
          stability pass · error_recovery pass
coverage  7 parameters, 10 operations — each with attempts, failures,
          failure_rate and latency {min, median, p95, max}
lifecycle connect 34.2 ms · identity · disconnect · reconnect
findings  []

Three things to read, in order:

• Gate status. pass fail skip — and a skip is a question left unanswered, never a pass. Each skip carries the reason: no allow-listed operation, no slow call to abandon, a connect too expensive to repeat.
• Coverage. Every declared parameter and operation appears with its verification state — declared (offered, never touched), queried, roundtrip, called. A module declaring 262 parameters of which 11 were verified says exactly that.
• Findings. What the run found wrong with the module, at error / warning / info. An error means the module declares an API the instrument does not have.

Failures are classified rather than merely counted — timeout, wrong_response, device_error, exception — per command, so a flaky query becomes a rate you can point at instead of an anecdote. The stability gate never retries: measuring how often a command fails is its entire purpose. The run stops at the first failed gate by default, because on real hardware a failure usually leaves the instrument in a state where the remaining gates measure nothing.

▸For contributors: making a module field-testable

You are writing or maintaining a device module and want the gates to drive it correctly.

Declare what each operation does

This is the August addition that made unattended runs possible (plesty-lib#39): an operation now records its kind, and the gates act on the kind instead of asking about every operation individually.

from plesty.lib.device.device_utils import expose_to_api

@expose_to_api(kind="motion", position_key="MO.Position")
def move_absolute(self, position: float) -> float: ...

@expose_to_api(kind="motion", position_key="MO.Position", relative=True, step=1.0)
def move_relative(self, step: float) -> float: ...

@expose_to_api(kind="lifecycle")
def home_stage(self) -> bool: ...

A schema-registered operation declares the same fields in its schema entry — kind, position_key, target, relative, step — validated at registration, so a typo fails when the module loads rather than during a test on hardware. The declaration travels: it reaches FuncDoc.kind, function_docs() and the describe payload, so the generated documentation and every tool reading the API see it too.

A motion declared without a position_key, a target and a step fails its gate rather than guessing. The client tier is handed the same declarations explicitly, because describe() flattens the served methods and the client cannot read the module's own registration — one set of kinds, not two that can drift apart.

Give the gates what they need

• A distinguishable pair for the drain probe. Two parameters whose values could never be confused — velocity in °/s against acceleration in °/s² will not do. The gate abandons a query on the first and reads the second; if the answer to the first comes back, the transport left a reply behind.
• Something genuinely slow to provoke. An averaged measurement is ideal: it takes as long as you ask it to. Where shortening the timeout is not a lever — some drivers simply do not enforce short ones — a module can supply a callable that transmits a query and never reads its answer, creating the orphaned reply deliberately.
• Restore keys. Parameters read before the provoking gates and written back after. The write-back is verified by reading the value again, and a parameter that will not go back fails the gate — a warning was not enough, because a green report is a reason not to look at the bench.
• Name what to test, not what to skip. only_keys inverts the old exclusion list. Exclusion stops scaling somewhere around a few dozen parameters, and the one that matters is whichever was forgotten.
• Resolve the unsafe list from the connected device when what the instrument supports depends on what is plugged into it — answered from the open session, since every extra connect cycle is another chance to strand a lock.

The gate it satisfies

Gate f1 — Field Test Findings now runs inside plesty check. It reads every published reports/<instrument>/field-test.json and fails the module if one holds an error finding: the module is telling every caller that something exists which does not, and must not be published in that state.

✓ Field Test Findings (f1 — 1 report(s): S130C)

A module that has never been field-tested publishes no report and the gate is skipped as not applicable. Absence is the honest rendering — "never run" and "ran and found nothing" are different facts, and treating them alike is how an untested module reads as a verified one.

Regenerating

uv run plesty init field-test           # resumes from .plesty/field-test.json
uv run plesty init field-test --force   # replace an existing tests/field_test.py

An existing field test is never replaced silently: it holds answers somebody thought about, and a hardware run may since have corrected them. One interview writes both tiers.