Testing on the Instrument

Testing a Device shows that a module is consistent; FieldTestPipeline, in plesty.lib.test.field_test, shows that it works on the real instrument. A module can pass every test against a simulated instrument and still fail on the real one: only the real one checks that the commands are what the instrument expects, and only a real connection can hang, or leave an old reply behind for the next question.

Each check records whether it passed and why, and every call to the instrument is timed and classified. The report therefore also has a timing profile (fastest, median, 95th percentile and slowest time per command) and failure statistics (attempts and failures per command), so timeouts can be set from measurements instead of guesses.

This test talks to hardware, so it is not run in CI. A device module keeps it in tests/field_test.py and runs it next to the instrument:

# tests/field_test.py
from plesty.lib.test.field_test import FieldTestPipeline

from plesty.mydevice import Device

PIPELINE = FieldTestPipeline(
    Device,
    "USB0::0x1313::0x8078::P0000001::INSTR",
    repetitions=20,
    safe_ops=["measure_power"],
    report_path="reports/field-test-mydevice",
)

if __name__ == "__main__":
    report = PIPELINE.run()
    raise SystemExit(0 if report.ok else 1)

plesty init field-test, a command of plesty-sdk, generates this file after asking about the instrument.

What it checks

In this order:

Test	Checks
test_discovery	the address leads to a connected instrument
test_connect_lifecycle	connect, identity, check_errors, disconnect and reconnect, each timed; the identity is the same after reconnecting
test_param_roundtrip	every writable parameter can be read, written back and read again; the instrument is left as it was
test_param_constraints	the instrument’s own limits and options against the declared ones; declared limits wider than the instrument’s fail
test_functions	every operation allowed to run (see below) can be called, in the way its kind says
test_buffer_drain	a question abandoned halfway does not leave its answer for the next question
test_error_recovery	the device recovers from a forced timeout, and the first command after reconnecting works
test_stability	every readable parameter and allowed operation, repeated repetitions times, for the statistics

By default a failed check is recorded and the rest still run, so one failure does not cost the timings of the others. With stop_on_failure=True the run stops at the first failure, and the checks not run are recorded as skipped: useful when a failure leaves the instrument in a state where the rest would measure nothing. run(gates=[...]) runs only some checks; reset() clears the results, to run again.

Which operations may run

An operation is called only if it is listed in safe_ops, or declared with safe=True (a "safe": true entry in the module’s operation file counts). Without any, the operations check is skipped: nothing that moves or emits ever runs unattended.

How an allowed operation is called depends on its kind, which the module declares with @expose_to_api(kind=...) or a "kind" entry in its operation file:

Kind	How the tests call it
read (the default)	with its declared arguments, as often as needed
acquire	the same, but recorded as acting on the instrument
motion	moved a step away from where position_key says it is, and back; the check fails if it does not come back
lifecycle (home, reset)	once; never repeated
control (stop, abort)	once, after the motions
configure	never: the parameter checks cover settings

@expose_to_api(kind="motion", position_key="MO.Position", step=1.0)
def move_absolute(self, position: float) -> float: ...

op_kinds={"move_absolute": OpKind("motion", position_key="MO.Position", step=1.0)} on the pipeline replaces what the module declares. A motion without a position key or a step fails the check rather than guessing how far a part may move.

Arguments

Argument	Default	Meaning
arguments after the class		passed to its constructor, typically the address
repetitions	20	how often each command is repeated in test_stability
report_path	None	where to write the report: <path>.jsonl (added to) and <path>.md
safe_ops, unsafe_ops	None	operations allowed to run, and operations never to run
op_kinds	None	the kind of each operation, replacing what the module declares
op_payloads	None	the arguments to call an operation with, instead of generated ones
ignore_keys	None	parameters left out of every check
only_keys	None	the only parameters to check; easier than ignore_keys for an instrument with hundreds
drain_probe	chosen automatically	the two parameters test_buffer_drain uses
slow_op, slow_op_payload	None	an operation slow enough to be abandoned halfway, for test_buffer_drain and test_error_recovery, when reading a parameter is too fast to interrupt
drain_timeout_ms	1	the timeout used to abandon a question on purpose
settle_s	0.0	pause between repetitions in test_stability
value_tolerance	0.001	relative difference allowed when a number is read back
max_failure_rate	0.0	the share of failures per command test_stability accepts
max_retries	0	attempts per call in total; 0 means one attempt, no retry
expensive_connect	False	set it when connecting starts an application and takes long: the checks that need a connection of their own are skipped, saying why, and the rest share one connection
stop_on_failure	False	stop at the first failed check
seed	123	makes generated arguments repeatable

Reports

The report at report_path includes the computer’s name and the instrument’s address, so keep it within the lab. For a report that can be shared, give instrument=Instrument(model="PM100D", sensor="S155C") (from plesty.lib.test.report_artifact) and reports_dir="reports": the run then also writes reports/<sensor or model>/field-test.json, without the address or serial numbers, which a module can publish. archive_dir names a shared folder, such as a lab network drive, that receives a copy of every full report.

propose_schema_update() compares the module’s parameter file with what the instrument reported, and returns a proposed change. Nothing is written until the proposal’s apply() is called, so every change is reviewed.