Field tests

A hub device module scaffolded with plesty init field-test carries its own field test — the hardware-gated run that verifies the module against the real instrument and publishes reports/<instrument>/field-test.json, the artifact hub.plesty.net and manager.plesty.net read. plesty-server runs it on the bench, from the developer’s machine, without anyone walking over.

Tiers

Tier

Script (in the checkout)

Needs

Publishes

host

tests/field_test.py

the instrument itself — the device server must be stopped (unless --address mock)

field-test.json

client

tests/field_test_client.py

a running server (tcp://…)

field-test-client.json

Both scripts exist only in a git checkout (see Installing) and run with the checkout’s interpreter. The markdown/JSONL beside the JSON carry the address and the serial and stay on the bench.

The pieces

  • FieldTestRun (model) — the request: tier, address, gates, repetitions, keep_going, keep_server, extra args. argv() builds the script’s arguments (the host tier gets --report reports/field-test-run for its internal pair); command(python, checkout) builds the full command line and fails with the “install from git and scaffold” hint when the script is missing; needs_instrument says whether the server has to be stopped.

  • JobRunner (service) — runs that command as a job of kind field-test, in the checkout, with the instance’s env from the fleet, so the address and the report archive come from the same place as for the server. The developer tails jobs.log(id, offset) and waits for the exit code.

  • FieldTestReport (model) — reads what was published: ok, passed/failed/ skipped, per-gate results, findings by severity. find_reports(checkout, since=job.started_utc) lists what this run produced.

The choreography — Bench.field_test_now

run = FieldTestRun(tier="host", address=None, gates=("connect", "drain"))
inst = manager.resolve(spec)                     # must be a git install
record = supervisor.record(name)
if run.needs_instrument and not run.keep_server and supervisor.alive(name):
    supervisor.stop(name)                        # the instrument is single-owner
if run.tier == "client" and not supervisor.alive(name):
    supervisor.start(spec, installer.python(inst))
job = jobs.start(name, JOB_KIND, run.command(installer.python(inst), checkout),
                 cwd=checkout, env=spec.env, detail=run.to_dict())
job = jobs.wait(job.id)                          # or stream jobs.store.log(...)
if record is not None:
    supervisor.relaunch(record)                  # back exactly as it was
reports = find_reports(checkout, since_utc=job.started_utc)

relaunch(record) is why ProcessRecord keeps the interpreter, module, args, env and port: the server comes back with the same arguments and on the same port a client connected to before the run.

From the command line:

plesty-server field-test pm100d --gates connect,drain        # host tier, stops/relaunches the server
plesty-server field-test pm100d --address mock               # simulator: server left alone
plesty-server field-test pm100d --tier client                # against the running server
plesty-server field-test pm100d --no-wait                    # returns the job id; `jobs log <id> --follow`
plesty-server reports pm100d                                 # what was published