plesty init field-test

Adds a field test — the gate sequence run against the real instrument — to an existing device module. Unlike plesty init mock-test, it interviews you.

cd plesty-pm100d
plesty init field-test          # interactive
plesty init field-test --force  # replace an existing tests/field_test.py

plesty.lib.test.field_test.FieldTestPipeline gives every module the same gate sequence. The gates are the easy part — they are generic and already shared. What takes the time, and what three modules each got wrong at least once, is the configuration, because every line of it is a judgement about the instrument that no schema inspection can make:

  • which operations are safe to call unattended — the wrong answer moved a stage, fired an exposure, and rewrote a sensor’s zero calibration

  • which parameters survive being written back their own value — one crashed the controlling application outright, killing every call after it

  • what is slow enough to be timed out on demand, since a call returning in two milliseconds cannot be abandoned and its gate silently skips

  • which two parameters are distinguishable enough that a stale reply is unambiguous

  • whether connecting is cheap, or starts an application and costs half a minute

Candidates from the device, judgement from you

So the command asks the device for the candidates, and you for the judgement. It constructs the module hardware-free — exactly as plesty docs does, which gate d3 already requires — and reads doc_model().

Not the JSON schema files, and the difference is not cosmetic. A module registers parameters and operations in Python too (register_config in __init__, an @expose_to_api method), and none of that is in a file: plesty-pm100d alone has two such operations, which a file-reading generator would never offer. And for a grouped schema the keys differ — get_config_list() answers group_prefix.param_key while the file is keyed by the bare name, so an only_keys list built from the file would name parameters the pipeline cannot resolve. That is the group-name against command-prefix mismatch three modules hit independently.

There is no fallback to the files. A device module that cannot build its doc model already fails gate d3, and generating from a partial picture would produce a file that looks complete and tests less than it claims.

The interview

Each question states what the answer is used for before offering the choices, and each answer is followed by “why?” — the reason is written into the generated file as a comment, so the next reader sees why and not only what.

Choices are a checkable list: arrow keys move, space selects, a selects all, n clears, Enter confirms, Escape backs out. Each entry shows what it is — a parameter’s type and range, an operation’s description — because choosing which operations may run unsupervised from a list of bare names is how the wrong one gets picked.

Nothing is ticked to begin with, and Enter confirms whatever is ticked, so the cheapest keypress gives the most conservative answer. a exists but has to be typed. While a list is empty it says what empty means for that question, which is not the same everywhere: no operation selected means none is called, while no parameter selected means every one is written back.

A run with no keyboard — piped input, a script, CI — falls back to typing comma-separated numbers rather than waiting for a keypress that never comes.

Nothing unclassified is treated as safe. An operation you do not select is in neither list: it is never called, and the gate that would have called it skips saying so. A generated allow-list that quietly included a motion command would be worse than no generator at all.

What the module declares is not asked again. An operation exposed with a kind — @expose_to_api(kind="motion", position_key="MO.Position", step=1.0), kind="lifecycle", kind="control", or a "kind" block in schema_func.json (plesty-lib#39) — is shown as [motion], [lifecycle], [control] in the chooser, and the “acts on the world” question is asked only about operations that declared nothing. A configure operation is not offered at all: it is recorded as excluded, because a setting that outlives the call is verified by the round-trip gate. For an allow-listed motion the interview confirms how far the test may move it (step; the declared value is the default) and, if the module left it open, which parameter reports the position — a motion with neither is dropped, since the pipeline drives a motion one step out from that parameter and back and will not guess how far. A motion chosen as the slow operation is asked only how far the long move goes: the pipeline starts it from wherever the part is and moves it back afterwards. Both generated files carry the result as OP_KINDS.

An operation can be given the arguments it needs. For each allow-listed operation that declares inputs (motions aside), the interview asks for arguments as name=value pairs — device=LED1, voltage=0.5. Each value is read as YAML, so 0.5 is a number and LED1 a string. Enter gives none, and the gate generates every argument from its declared type and range, which suits a count or a voltage but not an argument that has to name something the device registered. Both generated files carry the answer as OP_PAYLOADS and pass it to their tier.

The generated files

The result is two files from one interview. tests/field_test.py runs on the bench against the instrument: hardware-gated (it defines no pytest test functions, so CI collects nothing), wired to publish reports/<instrument>/field-test.json, and runnable against the simulator first:

uv run python tests/field_test.py --address mock
uv run python tests/field_test.py --address <address>

tests/field_test_client.py runs the same checks over the wire, against the served module. It inherits the answers rather than asking again: an operation unsafe on the bench is unsafe over the network too, and a second allow-list would be a second chance to disagree.

Neither file is replaced without --force: they hold answers somebody thought about, and a hardware run may since have corrected them.

The answers are kept

Each answer is written to .plesty/field-test.json as it is given, and every run of the command starts from what that file holds — so Ctrl-C loses nothing, and running plesty init field-test again carries on from where the interview stopped. A question that was displayed but not answered is asked again; it is marked done only once an answer is in hand. The file is the source of truth: edit it to change an answer without repeating the interview, because an answer hand-edited in the generated .py is overwritten by the next run. Answers left by an older SDK in .plesty-field-test.partial.json are moved into .plesty/ on the first run.

The run ends by saying which constant holds which answer.

The interview text ships in the wheel (plesty/sdk/assets/init/field-test-questions.yaml) and the generated file’s skeleton alongside it (field-test.py.template).