# Process management

This page explains how `plesty-server` starts, watches and stops other
programs — device servers that should stay up for days, and jobs (an
install, a field test) that run for minutes. It assumes no prior background;
the terms are introduced as they come up.

## The problem

Several `plesty-server` processes look at the same bench at the same time:
a CLI command you type, the GUI, the headless `fleet run` service, the
remote agent. Each of them must be able to answer *is the pm100d server
running?*, *what did the install job print?*, *why did the server die?* —
about processes **another** `plesty-server` process started, possibly
yesterday.

Two facts about operating systems make that harder than it sounds.

**1. A child belongs to its parent.** When a program starts another one
(its *child*), the parent is told when the child exits and must *collect*
that exit (`wait`). Until it does, the child lingers as a **zombie**: it is
dead, but the process table still lists it, and a liveness check such as
`kill(pid, 0)` says "exists". If the CLI that started a server exits, the
server's exit code is lost for everyone else; if the GUI started it and the
GUI is busy, the dead server still looks alive to the CLI. The parent/child
relationship is exactly what we do *not* want between `plesty-server` and
the processes it manages.

**2. Signals are the only way to ask nicely.** To stop a program from the
outside you send it a *signal*: `SIGINT` is "you were interrupted" (what
Ctrl-C sends), `SIGTERM` is "please terminate", `SIGKILL` ends it without
asking. A plesty-lib device server handles the interrupt by closing the
instrument session cleanly (`[SERVER] Shutting down...`); a kill leaves the
instrument in whatever state it was in. So the order is always *ask, wait,
insist, then force*.

## The design: nothing is a child

Every server and every job is started through the **launcher**
(`plesty.server.services.launch`), a tiny program run as
`python -m plesty.server.services.launch` — `plesty-server _launch` inside
a bundle, see `reexec` — with a JSON spec on stdin. It

1. **detaches** itself so that nobody has to reap it
   (`host.detach` — on POSIX a fork + new session, after which the
   intermediate parent exits and the waiter is adopted by `init`; on
   Windows the launcher process simply *is* the detached process),
2. prints the pid of the detached **waiter** — that is the pid everybody
   tracks,
3. starts the real command with stdout and stderr appended to the **log
   file**, forwards `SIGINT`/`SIGTERM` to it, waits,
4. writes the exit code to the **exit file** (`<name>.exit`, written
   atomically via a temp file) and appends `=== exit N at <time>` to the log.

```text
plesty-server (CLI/GUI)                 waiter (detached)            command
   │ Popen launcher, spec on stdin  ──►  │
   │ ◄── "pid\n"                          │ log << banner
   │ write record {pid, port, ...}        │ Popen command ─────────► │  stdout+stderr → log
   │ (returns at once)                    │ wait                      │
                                          │ ◄── exit code ───────────┘
                                          │ write <name>.exit
```

Consequences, all of them intended:

- **Liveness is a file + a pid.** A process "runs" while its pid is alive
  *and* no exit file exists. Any `plesty-server` process can evaluate that.
- **The exit code survives.** The waiter writes it; whoever reads the record
  later finds it. `LOST_EXIT = -1` is recorded when the pid is gone but no
  exit file appeared (the bench rebooted, someone killed the waiter).
- **Logs are append-only files**, tailed by byte offset (`JobStore.log`)
  or by lines (`tail_file`). Streaming to the GUI is "read from where you
  stopped".
- **No zombies**: the waiter is never anyone's un-reaped child.

## Jobs vs servers

Both go through the launcher; they differ in what they mean.

| | Job (`JobRunner` + `JobStore`) | Device server (`Supervisor`) |
|---|---|---|
| Identity | `<name>-<kind>-<utc-stamp>` | the instance name |
| Lifetime | minutes; *meant* to end | days; *meant* to stay up |
| Record | `jobs/<id>.json` | `state/<name>.json` |
| Log | `logs/jobs/<id>.log` | `logs/<name>.log` (appended per start) |
| Done when | exit file exists | never — `exited` is a failure state |
| Alive means | pid alive, no exit file | pid alive, no exit file, **and** the port answers `describe` |
| Stop | kill the tree (a cancelled install has nothing to save) | interrupt → terminate → kill, with timeouts |

### Server states

`Supervisor.status(name)` combines three observations — record present, pid
alive, port answering — into one of four states:

| State | Record | Pid alive | Probe answers | Meaning |
|---|---|---|---|---|
| `stopped` | no | – | – | never started, or stopped cleanly |
| `starting` | yes | yes | no | launched, device session still opening |
| `running` | yes | yes | yes | ready for clients |
| `exited` | yes | no | – | it died — `returncode` and the log tail say why |

The probe is a real `describe` request on the plesty-lib wire (ZMQ DEALER
→ ROUTER, JSON). A pid says the interpreter is running; only the probe says
the *server* is. `start(..., wait=True)` polls the probe until
`ready_timeout` (30 s by default; a slow instrument session can take that).

### Starting safely

`Supervisor.start` holds a per-device lock (two concurrent starts of one
device — a double click — cannot both pass the "not running" check and leave
one server unmanaged), and refuses with `PortBusy` when the port already
answers: another device, or a server from an earlier start whose record was
lost, holds it. `status` says so too — an `exited` record whose port still
answers carries the hint in its detail.

`PortBusy` is the last word only for a port the declaration *names*, because
that number is passed to the device itself and the bench cannot change it.
A port the **bench allocated** never reaches this check as a failure: the
caller re-allocates it first, records where it went, and starts there — two
benches on one machine allocate from the same range, and a server left from
an earlier start holds what it holds. See [Command line](cli.md) for what
`declare` and `start` report when a port moves.

### Stopping, in order

```text
interrupt (SIGINT / CTRL_BREAK)  ──wait stop_timeout──►  terminate (SIGTERM)  ──3 s──►  kill tree
```

Each step is a `host` function; on Windows `terminate` is a no-op and
`kill tree` is `taskkill /T /F`. The record is dropped afterwards, which is
what frees the name and the port for the next start.

### Seeing the same thing from two processes

`test_another_supervisor_sees_the_record` is the contract: a second
`Supervisor` built on the same home — a new CLI invocation — finds the
server `running` and can stop it, using only the record, the pid and the
probe. That is why every piece of state is a file in the home.

## Where the platform differences went

Fork, sessions, `SIGINT` vs `CTRL_BREAK_EVENT`, `killpg` vs `taskkill`,
`kill(pid, 0)` vs `OpenProcess`, console windows — none of it appears in
`launch.py`, `jobs.py` or `supervisor.py`. They call `host.detach`,
`host.interrupt`, `host.kill_tree`, `host.pid_alive`, `host.spawn_kwargs`.
See [Cross-platform development](cross_platform.md).