Experiment runs

How the bench runs an experiment module: what it asks the module, how a run is started across several benches, how it is watched and stopped, and how its report is written. The user’s view of the same thing is Run an experiment and The Experiments tab.

The code is plesty/server/services/experiments.py (the run directory), run_report.py (the report), and the experiment methods of Bench in plesty/server/presenter/bench.py (the choreography).

No per-experiment code

The console holds no per-experiment code, for the same reason it holds no per-device code. A device answers describe and its panel is built from the reply; an experiment is not a server and answers nothing, so it publishes the same knowledge another way — python -m <module> --manifest, run in the module’s own environment. The document it prints carries its configuration schema in the shape a device’s describe carries parameters, so the settings form is built by the mapping that already exists and a module cannot be offered a setting it does not take.

The manifest is read from the module on every use rather than cached in the fleet. A declaration carrying a copy of the contract would be a second thing to keep in step, and an updated module would leave it stale. The read spawns a process, so it must not sit on the console’s refresh timer uncached.

An experiment is declared like a device and run like nothing else. fleet.yaml grows experiments:, and the installer is untouched: it is keyed by the package and knows nothing of devices. But an experiment binds no port, nothing probes it, fleet up does not start it, and it is installed from its repository rather than from a release — its config/ and .env.example are in the checkout and not in the wheel, so a release install would leave nothing to configure.

Starting a run

A run is a job like any other — a detached plesty-server _job experiment-run whose output lands in logs/jobs/<id>.log — so nothing here re-implements launching or log tailing. What is particular to an experiment happens before and after the module runs.

Before, Bench.start_experiment:

0. experiment_devices — when the module’s manifest says chooses_devices, asks python -m <module> --devices --config <the run's config> which devices of setup.yaml this configuration uses (plesty-lib#62). Steps 1 and 2 then cover only those devices and the benches serving them, and a missing <DEVICE>_ADDRESS of another device is not asked for. A module that does not choose gets every device, as before.

1. check_shared_disk — writes a probe under this bench’s shared-data folder (in .plesty-share-check/), asks every other bench of the module’s setup.yaml to run share-check through its agent, and reads each reply back. Equal content at both ends proves both see one folder and can write to it. The files are removed afterwards. A bench whose setup names no shared folder is not asked, and the whole check is skipped when this machine has none.

2. serve_experiment_devices — for each device of the setup, has its bench run it: Bench.serve here, plesty-server serve NAME --entry JSON --json through the agent elsewhere. serve declares from the entry when the bench lacks the device, re-declares (keeping its port) and restarts it when the entry differs, and refuses a name declared for another package. It returns the port and whether this call started the server. The entry is the setup’s device with two things of this experiment’s added (_served_entry): a ${VARIABLE} in its args is the value from the checkout’s .env or the declaration, as a bench’s fields in setup.yaml are, and a declaration variable <STEM>__<NAME> (SPEC__DEVICE_DATA_PATH) is NAME in that device’s environment only — <STEM> as in SPEC_ADDRESS. Given there, a value reaches the device while this experiment serves it, and not every later experiment that serves a device of that name, as a device .env would (#43).

3. Builds the run’s environment: PLESTY_DATA, and <DEVICE>_ADDRESS from each bench’s host and the port it said. A device whose address is already set — in the checkout’s .env or the declaration’s env: — is skipped in step 2: whoever set it said where the device is.

The devices step 2 started are passed to the job as --release. A device that was already running is not: it was someone else’s.

After, however the module ends — completed, stopped, aborted, crashed — the job’s finally stops the released devices, each on its bench (release_experiment_devices), and then writes the report. Devices first: a report is worth having and a device left holding the hardware is not.

When the job dies — killed from the task manager, a crash — there is no finally. Two things cover it. The module is bound to the job: on Windows it is held in a job object that closes with the job (host.bind_to_me), on Linux it gets SIGTERM when its parent dies (PR_SET_PDEATHSIG), so it does not measure on with nobody to stop its devices. And the console reads a run whose journal says running or paused while no job of it is alive as aborted (lost), so it can be resumed or deleted, and stops the devices that run started (release_lost_run_devices) — each only while the server running is the one that run started, judged by its start time, so a device started since is left alone (#46).

Watching a run

Status is read, never asked. The run directory is the whole interface: journal.jsonl gives the state and the step counts, plan.json the plan’s length, progress.json whatever the step in flight last reported. One directory read per refresh, nothing on the wire, and a console that crashes leaves a measurement that carries on.

The journal is replayed here rather than through plesty.lib.experiment.Journal, though it computes the same thing. The run was written by an experiment module, whose plesty-lib has nothing to do with the one this console happens to have installed: delegating would make the same run read differently on two benches, and a run written by a newer module would lose its paused state on an older console.

Stopping a run

Stopping is not cancelling the job. JobRunner.cancel kills a process tree, which for a measurement means no run_canceled in the journal and so no point to resume from. A run is stopped by writing what it should do next into control.json, which it reads at a step boundary — where the journal has just committed and no instrument is mid-operation. The kill stays, as kill_experiment, for a run that has stopped reading that file; it stops the run’s devices itself, since the job’s finally never runs. Nothing in the console or the command line calls it yet. The kill a user can reach is plesty-server jobs cancel, which kills the tree and leaves the run’s devices running.

Runs and their configuration

create_run names a run before it starts: it writes created.json and copies the chosen configuration file into the run directory as config.yaml. Start and resume read that copy, so configuring one run cannot change another. reconfigure_run rewrites the copy and refuses once the run has started — a run with journal.jsonl or plan.json has frozen its configuration into its plan. For the same reason a run is renamed only before it starts (rename_experiment_run, which moves its folder), and a name an earlier run had is refused: its job and log stay behind, and would read as the new run’s (#27).

Who measured a run and why is kept beside its data, not in the bench: created.json carries the operator, and notes.txt the notes — plain text, one entry per note with who and when, added while the run is open or after it ended (#40). The bench keeps only last-operator.txt, to propose the next operator.

The welcome demo

The first start of an empty bench offers a demo that needs no hardware (Bench.install_demo, set_up_demo, #43). It is an ordinary experiment: plesty-pol-pl installed from git at its release tag (DEMO in the presenter), declared, a first run created from its config/demo.yaml and started. Its setup.yaml names the two devices — the K10CR1 stage in its simulator, and a spectrometer replaying spectra recorded on E098-SIL — and pins them, so the demo does not change when their repositories do. Nothing in the bench knows it is a demo except where it points: the declaration gives LOCAL_DATA and PLESTY_DATA as <home>/demo, and the spectrometer its replay and data folders there through SPEC__… variables, so runs and replay share one folder of the bench’s own.

The report

run_report.write compiles the run directory into one self-contained HTML file — the template and stylesheet are plesty/server/assets/report.html and report.css. The job writes it as report.html into the run directory when the run ends; Export HTML writes the same thing elsewhere.

A report that cannot be written is a line in the job log and never fails the job: the measurement is over, its data is on disk, and failing the job would report a successful run as a failed one.

The viewer

The Visualization pane loads the module’s viewer in the console’s process. When the viewer is not importable there, the module’s environment is added behind the console’s own on the import path, so the module and what only it depends on come from there while plesty-lib, Qt and numpy stay the console’s. An environment of another Python version is not used. When loading fails, Open in its own window starts the module’s viewer as a separate process in its own environment instead.