Process management

How plesty-server starts, watches and stops other programs — device servers that should stay up for days, and jobs (an install, a field test) that run for minutes. It assumes no prior background; the terms are introduced as they come up.

The problem

Several plesty-server processes look at the same bench at the same time: a CLI command you type, the GUI, the headless fleet run service, the remote agent. Each of them must be able to answer is the pm100d server running?, what did the install job print?, why did the server die? — about processes another plesty-server process started, possibly yesterday.

Two facts about operating systems make that harder than it sounds.

1. A child belongs to its parent. When a program starts another one (its child), the parent is told when the child exits and must collect that exit (wait). Until it does, the child lingers as a zombie: it is dead, but the process table still lists it, and a liveness check such as kill(pid, 0) says “exists”. If the CLI that started a server exits, the server’s exit code is lost for everyone else; if the GUI started it and the GUI is busy, the dead server still looks alive to the CLI. The parent/child relationship is exactly what we do not want between plesty-server and the processes it manages.

2. Signals are the only way to ask nicely. To stop a program from the outside you send it a signal: SIGINT is “you were interrupted” (what Ctrl-C sends), SIGTERM is “please terminate”, SIGKILL ends it without asking. A plesty-lib device server handles the interrupt by closing the instrument session cleanly ([SERVER] Shutting down...); a kill leaves the instrument in whatever state it was in. So the order is always ask, wait, insist, then force.

The design: nothing is a child

Every server and every job is started through the launcher (plesty.server.services.launch), a tiny program run as python -m plesty.server.services.launch — plesty-server _launch inside a bundle, see reexec — with a JSON spec on stdin. It

detaches itself so that nobody has to reap it (host.detach — on POSIX a fork + new session, after which the intermediate parent exits and the waiter is adopted by init; on Windows the launcher process simply is the detached process),
prints the pid of the detached waiter — that is the pid everybody tracks,
starts the real command with stdout and stderr appended to the log file, forwards SIGINT/SIGTERM to it, waits,
writes the exit code to the exit file (<name>.exit, written atomically via a temp file) and appends === exit N at <time> to the log.

plesty-server (CLI/GUI)                 waiter (detached)            command
   │ Popen launcher, spec on stdin  ──►  │
   │ ◄── "pid\n"                          │ log << banner
   │ write record {pid, port, ...}        │ Popen command ─────────► │  stdout+stderr → log
   │ (returns at once)                    │ wait                      │
                                          │ ◄── exit code ───────────┘
                                          │ write <name>.exit

Consequences, all of them intended:

Liveness is a file + a pid. A process “runs” while its pid is alive and no exit file exists. Any plesty-server process can evaluate that.
The exit code survives. The waiter writes it; whoever reads the record later finds it. LOST_EXIT = -1 is recorded when the pid is gone but no exit file appeared (the bench rebooted, someone killed the waiter).
Logs are append-only files, tailed by byte offset (JobStore.log) or by lines (tail_file). Streaming to the GUI is “read from where you stopped”.
No zombies: the waiter is never anyone’s un-reaped child.

Jobs versus servers

Both go through the launcher; they differ in what they mean.

	Job (`JobRunner` + `JobStore`)	Device server (`Supervisor`)
Identity	`<name>-<kind>-<utc-stamp>`	the instance name
Lifetime	minutes; meant to end	days; meant to stay up
Record	`jobs/<id>.json`	`state/<name>.json`
Log	`logs/jobs/<id>.log`, the child’s own stdout	`logs/<name>-<stamp>.log`, a new one per start and per Clear log; `logs/<name>.current` names it. The waiter relays the output into it and switches files on a `.switch` request beside the stop file
Done when	exit file exists	never — `exited` is a failure state
Alive means	pid alive, no exit file	pid alive, no exit file, and the port answers `describe`
Stop	kill the tree (a cancelled install has nothing to save)	interrupt → terminate → kill, with timeouts

A job that runs a child of its own — an experiment run launches the module — binds that child to itself, so the child cannot outlive the job: a Windows job object that kills its members when the job’s handle closes, PR_SET_PDEATHSIG on Linux (host.bind_to_me, host.die_with_me_kwargs). Killed from the task manager, a run’s job used to leave the module measuring on alone; what the console then does about the run is in Experiment runs.

Server states

Supervisor.status(name) combines three observations — record present, pid alive, port answering — into one of four states:

State	Record	Pid alive	Probe answers	Meaning
`stopped`	no	–	–	never started, or stopped cleanly
`starting`	yes	yes	no	launched, device session still opening
`running`	yes	yes	yes	ready for clients
`exited`	yes	no	–	it died — `returncode` and the log tail say why

The probe is a real describe request on the plesty-lib wire (ZMQ DEALER → ROUTER, JSON). A pid says the interpreter is running; only the probe says the server is. start(..., wait=True) polls the probe until ready_timeout (30 s by default; a slow instrument session can take that).

Starting safely

Supervisor.start holds a per-device lock — two concurrent starts of one device (a double click) cannot both pass the “not running” check and leave one server unmanaged — and refuses with PortBusy when the port already answers: another device, or a server from an earlier start whose record was lost, holds it. status says so too: an exited record whose port still answers carries the hint in its detail.

PortBusy is the last word only for a port the declaration names, because that number is passed to the device itself and the bench cannot change it. A port the bench allocated never reaches this check as a failure: the caller re-allocates it first, records where it went, and starts there. See Ports.

Stopping, in order

request_stop (SIGINT / stop file)  ──wait stop_timeout──►  terminate (SIGTERM)  ──3 s──►  kill tree

The clean request differs by platform. On POSIX it is SIGINT to the waiter, which forwards it. On Windows no console event can reach the waiter from the GUI, so request_stop writes a stop file the waiter watches for, and the waiter sends its server CTRL_BREAK_EVENT. Each step is a host function; on Windows terminate is a no-op and kill tree is taskkill /T /F. The record is dropped afterwards, which is what frees the name and the port for the next start.

The first step only works if the server catches what it is sent. On Windows CTRL_BREAK_EVENT arrives as SIGBREAK, whose default action is to end the process outright — a server that does not handle it is killed with its device still open, and whatever that device holds (a vendor subprocess, a camera, a serial handle) is leaked and blocks the next start. Servers built on a current plesty.lib.service handle it: DeviceTCPIPServer.run serves inside stop_on_break. The bench recognises the aftermath either way — a server that exits 0xC000013A did not shut down, and host.exit_note says so in the status detail, since the log of such a server stops mid-run and otherwise reads like an unexplained crash.

Seeing the same thing from two processes

test_another_supervisor_sees_the_record is the contract: a second Supervisor built on the same home — a new CLI invocation — finds the server running and can stop it, using only the record, the pid and the probe. That is why every piece of state is a file in the home.

Where the platform differences went

Fork, sessions, SIGINT vs CTRL_BREAK_EVENT, killpg vs taskkill, kill(pid, 0) vs OpenProcess, console windows — none of it appears in launch.py, jobs.py or supervisor.py. They call host.detach, host.request_stop, host.kill_tree, host.pid_alive, host.spawn_kwargs. See Cross-platform development.