# Testing on the Instrument

[Testing a Device](device.md) shows that a module is consistent; `FieldTestPipeline`,
in `plesty.lib.test.field_test`, shows that it works on the real instrument. A module
can pass every test against a simulated instrument and still fail on the real one:
only the real one checks that the commands are what the instrument expects, and only
a real connection can hang, or leave an old reply behind for the next question.

Each check records whether it passed and why, and every call to the instrument is
timed and classified. The report therefore also has a **timing profile** (fastest,
median, 95th percentile and slowest time per command) and **failure statistics**
(attempts and failures per command), so timeouts can be set from measurements
instead of guesses.

This test talks to hardware, so it is not run in CI. A device module keeps it in
`tests/field_test.py` and runs it next to the instrument:

```python
# tests/field_test.py
from plesty.lib.test.field_test import FieldTestPipeline

from plesty.mydevice import Device

PIPELINE = FieldTestPipeline(
    Device,
    "USB0::0x1313::0x8078::P0000001::INSTR",
    repetitions=20,
    safe_ops=["measure_power"],
    report_path="reports/field-test-mydevice",
)

if __name__ == "__main__":
    report = PIPELINE.run()
    raise SystemExit(0 if report.ok else 1)
```

`plesty init field-test`, a command of plesty-sdk, generates this file after asking
about the instrument.

## What it checks

In this order:

| Test | Checks |
|---|---|
| `test_discovery` | the address leads to a connected instrument |
| `test_connect_lifecycle` | connect, identity, `check_errors`, disconnect and reconnect, each timed; the identity is the same after reconnecting |
| `test_param_roundtrip` | every writable parameter can be read, written back and read again; the instrument is left as it was |
| `test_param_constraints` | the instrument's own limits and options against the declared ones; declared limits wider than the instrument's fail |
| `test_functions` | every operation allowed to run (see below) can be called, in the way its kind says |
| `test_buffer_drain` | a question abandoned halfway does not leave its answer for the next question |
| `test_error_recovery` | the device recovers from a forced timeout, and the first command after reconnecting works |
| `test_stability` | every readable parameter and allowed operation, repeated `repetitions` times, for the statistics |

By default a failed check is recorded and the rest still run, so one failure does not
cost the timings of the others. With `stop_on_failure=True` the run stops at the first
failure, and the checks not run are recorded as skipped: useful when a failure leaves
the instrument in a state where the rest would measure nothing. `run(gates=[...])`
runs only some checks; `reset()` clears the results, to run again.

## Which operations may run

An operation is called only if it is listed in `safe_ops`, or declared with
`safe=True` (a `"safe": true` entry in the module's operation file counts). Without
any, the operations check is skipped: nothing that moves or emits ever runs
unattended.

*How* an allowed operation is called depends on its **kind**, which the module
declares with `@expose_to_api(kind=...)` or a `"kind"` entry in its operation file:

| Kind | How the tests call it |
|---|---|
| `read` (the default) | with its declared arguments, as often as needed |
| `acquire` | the same, but recorded as acting on the instrument |
| `motion` | moved a `step` away from where `position_key` says it is, and back; the check fails if it does not come back |
| `lifecycle` (home, reset) | once; never repeated |
| `control` (stop, abort) | once, after the motions |
| `configure` | never: the parameter checks cover settings |

```python
@expose_to_api(kind="motion", position_key="MO.Position", step=1.0)
def move_absolute(self, position: float) -> float: ...
```

`op_kinds={"move_absolute": OpKind("motion", position_key="MO.Position", step=1.0)}`
on the pipeline replaces what the module declares. A motion without a position key or
a step fails the check rather than guessing how far a part may move.

## Arguments

| Argument | Default | Meaning |
|---|---|---|
| arguments after the class | | passed to its constructor, typically the address |
| `repetitions` | `20` | how often each command is repeated in `test_stability` |
| `report_path` | `None` | where to write the report: `<path>.jsonl` (added to) and `<path>.md` |
| `safe_ops`, `unsafe_ops` | `None` | operations allowed to run, and operations never to run |
| `op_kinds` | `None` | the kind of each operation, replacing what the module declares |
| `op_payloads` | `None` | the arguments to call an operation with, instead of generated ones |
| `ignore_keys` | `None` | parameters left out of every check |
| `only_keys` | `None` | the only parameters to check; easier than `ignore_keys` for an instrument with hundreds |
| `drain_probe` | chosen automatically | the two parameters `test_buffer_drain` uses |
| `slow_op`, `slow_op_payload` | `None` | an operation slow enough to be abandoned halfway, for `test_buffer_drain` and `test_error_recovery`, when reading a parameter is too fast to interrupt |
| `drain_timeout_ms` | `1` | the timeout used to abandon a question on purpose |
| `settle_s` | `0.0` | pause between repetitions in `test_stability` |
| `value_tolerance` | `0.001` | relative difference allowed when a number is read back |
| `max_failure_rate` | `0.0` | the share of failures per command `test_stability` accepts |
| `max_retries` | `0` | attempts per call in total; `0` means one attempt, no retry |
| `expensive_connect` | `False` | set it when connecting starts an application and takes long: the checks that need a connection of their own are skipped, saying why, and the rest share one connection |
| `stop_on_failure` | `False` | stop at the first failed check |
| `seed` | `123` | makes generated arguments repeatable |

## Reports

The report at `report_path` includes the computer's name and the instrument's
address, so keep it within the lab. For a report that can be shared, give
`instrument=Instrument(model="PM100D", sensor="S155C")` (from
`plesty.lib.test.report_artifact`) and `reports_dir="reports"`:
the run then also writes `reports/<sensor or model>/field-test.json`, without the
address or serial numbers, which a module can publish. `archive_dir` names a shared
folder, such as a lab network drive, that receives a copy of every full report.

`propose_schema_update()` compares the module's parameter file with what the
instrument reported, and returns a proposed change. Nothing is written until the
proposal's `apply()` is called, so every change is reviewed.