Stopping and Continuing a Run

A run can stop before its last step in four ways: it crashes, someone presses Ctrl-C, someone asks it to stop through its run folder, or a step fails every attempt. In each case the finished steps are already saved, and the run can be continued later from where it stopped.

Continuing a run

run() returns the run’s id. Pass it back as resume to continue:

run_id = asyncio.run(LineScan(run_root="runs").run())
# ... stopped?
asyncio.run(LineScan(run_root="runs").run(resume=run_id))

From The Command Line, the same is --resume <run id>.

When continuing, the framework:

1. builds the plan again and compares its fingerprint with the saved plan’s. If they differ, because the code or the settings changed, it refuses to continue (PlanMismatchError), so a changed plan cannot mix into an old run;

2. reads the journal to find the finished steps, and skips them;

3. notes in the journal that the run continued, and runs the remaining steps.

teardown() runs however a run ends, so the devices are always released.

Stopping a run on purpose

From the terminal it was started in, press Ctrl-C. The run stops, notes the stop in the journal, and releases the devices. Do not close the terminal or kill the process instead: without the note, the journal does not say the run was stopped.

From anywhere else, write a request into the run folder with plesty.lib.experiment.control:

from plesty.lib.experiment import control

control.pause(run_dir)                      # wait before the next step
control.resume(run_dir)                     # go on
control.cancel(run_dir, "sample drifted")   # finish this step, then stop
control.abort(run_dir, "laser off")         # stop now, even in the middle of a step

A pause or a cancel takes effect between two steps. An abort interrupts the current step, which is then not saved and is repeated when the run continues. Asking a running measurement to pause or stop explains how the run notices a request.

What the journal records

The journal, journal.jsonl in the run folder, has one line per event. Reading it is how a run is continued, and how a viewer or a console knows the run’s state:

Event	Written when
run_started	the run starts, and again each time it is continued
step_started	a step begins
step_completed	a step finished and its record is saved
step_failed	an attempt at a step raised an exception
step_aborted	a step was interrupted by an abort
run_paused, run_resumed	the run waited on a pause request, and went on
report	a note was sent with control.report(run_dir, text)
run_completed	every step finished
run_canceled	the run was stopped by Ctrl-C or a cancel request
run_aborted	the run was stopped by an abort request, or a step failed every attempt

Reading a Run shows how to read the state from a program.