Process management

This page explains how plesty-server starts, watches and stops other programs — device servers that should stay up for days, and jobs (an install, a field test) that run for minutes. It assumes no prior background; the terms are introduced as they come up.

The problem

Several plesty-server processes look at the same bench at the same time: a CLI command you type, the GUI, the headless fleet run service, the remote agent. Each of them must be able to answer is the pm100d server running?, what did the install job print?, why did the server die? — about processes another plesty-server process started, possibly yesterday.

Two facts about operating systems make that harder than it sounds.

1. A child belongs to its parent. When a program starts another one (its child), the parent is told when the child exits and must collect that exit (wait). Until it does, the child lingers as a zombie: it is dead, but the process table still lists it, and a liveness check such as kill(pid, 0) says “exists”. If the CLI that started a server exits, the server’s exit code is lost for everyone else; if the GUI started it and the GUI is busy, the dead server still looks alive to the CLI. The parent/child relationship is exactly what we do not want between plesty-server and the processes it manages.

2. Signals are the only way to ask nicely. To stop a program from the outside you send it a signal: SIGINT is “you were interrupted” (what Ctrl-C sends), SIGTERM is “please terminate”, SIGKILL ends it without asking. A plesty-lib device server handles the interrupt by closing the instrument session cleanly ([SERVER] Shutting down...); a kill leaves the instrument in whatever state it was in. So the order is always ask, wait, insist, then force.

The design: nothing is a child

Every server and every job is started through the launcher (plesty.server.services.launch), a tiny program run as python -m plesty.server.services.launch — plesty-server _launch inside a bundle, see reexec — with a JSON spec on stdin. It

detaches itself so that nobody has to reap it (host.detach — on POSIX a fork + new session, after which the intermediate parent exits and the waiter is adopted by init; on Windows the launcher process simply is the detached process),
prints the pid of the detached waiter — that is the pid everybody tracks,
starts the real command with stdout and stderr appended to the log file, forwards SIGINT/SIGTERM to it, waits,
writes the exit code to the exit file (<name>.exit, written atomically via a temp file) and appends === exit N at <time> to the log.

plesty-server (CLI/GUI)                 waiter (detached)            command
   │ Popen launcher, spec on stdin  ──►  │
   │ ◄── "pid\n"                          │ log << banner
   │ write record {pid, port, ...}        │ Popen command ─────────► │  stdout+stderr → log
   │ (returns at once)                    │ wait                      │
                                          │ ◄── exit code ───────────┘
                                          │ write <name>.exit

Consequences, all of them intended:

Liveness is a file + a pid. A process “runs” while its pid is alive and no exit file exists. Any plesty-server process can evaluate that.
The exit code survives. The waiter writes it; whoever reads the record later finds it. LOST_EXIT = -1 is recorded when the pid is gone but no exit file appeared (the bench rebooted, someone killed the waiter).
Logs are append-only files, tailed by byte offset (JobStore.log) or by lines (tail_file). Streaming to the GUI is “read from where you stopped”.
No zombies: the waiter is never anyone’s un-reaped child.

Jobs vs servers

Both go through the launcher; they differ in what they mean.

	Job (`JobRunner` + `JobStore`)	Device server (`Supervisor`)
Identity	`<name>-<kind>-<utc-stamp>`	the instance name
Lifetime	minutes; meant to end	days; meant to stay up
Record	`jobs/<id>.json`	`state/<name>.json`
Log	`logs/jobs/<id>.log`	`logs/<name>.log` (appended per start)
Done when	exit file exists	never — `exited` is a failure state
Alive means	pid alive, no exit file	pid alive, no exit file, and the port answers `describe`
Stop	kill the tree (a cancelled install has nothing to save)	interrupt → terminate → kill, with timeouts

Server states

Supervisor.status(name) combines three observations — record present, pid alive, port answering — into one of four states:

State	Record	Pid alive	Probe answers	Meaning
`stopped`	no	–	–	never started, or stopped cleanly
`starting`	yes	yes	no	launched, device session still opening
`running`	yes	yes	yes	ready for clients
`exited`	yes	no	–	it died — `returncode` and the log tail say why

The probe is a real describe request on the plesty-lib wire (ZMQ DEALER → ROUTER, JSON). A pid says the interpreter is running; only the probe says the server is. start(..., wait=True) polls the probe until ready_timeout (30 s by default; a slow instrument session can take that).

Starting safely

Supervisor.start holds a per-device lock (two concurrent starts of one device — a double click — cannot both pass the “not running” check and leave one server unmanaged), and refuses with PortBusy when the port already answers: another device, or a server from an earlier start whose record was lost, holds it. status says so too — an exited record whose port still answers carries the hint in its detail.

PortBusy is the last word only for a port the declaration names, because that number is passed to the device itself and the bench cannot change it. A port the bench allocated never reaches this check as a failure: the caller re-allocates it first, records where it went, and starts there — two benches on one machine allocate from the same range, and a server left from an earlier start holds what it holds. See Command line for what declare and start report when a port moves.

Stopping, in order

interrupt (SIGINT / CTRL_BREAK)  ──wait stop_timeout──►  terminate (SIGTERM)  ──3 s──►  kill tree

Each step is a host function; on Windows terminate is a no-op and kill tree is taskkill /T /F. The record is dropped afterwards, which is what frees the name and the port for the next start.

Seeing the same thing from two processes

test_another_supervisor_sees_the_record is the contract: a second Supervisor built on the same home — a new CLI invocation — finds the server running and can stop it, using only the record, the pid and the probe. That is why every piece of state is a file in the home.

Where the platform differences went

Fork, sessions, SIGINT vs CTRL_BREAK_EVENT, killpg vs taskkill, kill(pid, 0) vs OpenProcess, console windows — none of it appears in launch.py, jobs.py or supervisor.py. They call host.detach, host.interrupt, host.kill_tree, host.pid_alive, host.spawn_kwargs. See Cross-platform development.