plesty.server.services.experiments ================================== .. py:module:: plesty.server.services.experiments .. autoapi-nested-parse:: Running experiments on this bench, and reading what they are doing. An experiment run is a job like any other — a detached process whose output lands in ``logs/jobs/.log`` — so nothing here re-implements launching or log tailing. What is particular to an experiment is on either side of that: * **Before**, the module is asked what it takes. ``python -m --manifest`` runs in the module's own environment and prints the declaration; the console builds its settings from that and holds no per-experiment code. The answer is read fresh rather than cached in the fleet, so an updated module cannot leave a stale copy of its own contract behind. * **After**, the run is watched and driven through its run directory. ``journal.jsonl``, ``records.jsonl`` and ``progress.json`` say what it is doing, and ``control.json`` says what it should do next. Both are files in a directory this bench owns: one read per refresh, nothing on the wire, and a console that crashes leaves a run that carries on. **Stopping is not cancelling the job.** ``JobRunner.cancel`` kills a process tree, which for a measurement means no ``run_canceled`` in the journal and so no point to resume from. A run is stopped by asking it to stop — which it notices at a step boundary, where the journal has just committed — and the kill stays for a process that has stopped answering. Attributes ---------- .. autoapisummary:: plesty.server.services.experiments.MANIFEST_TIMEOUT plesty.server.services.experiments.JOB_KIND plesty.server.services.experiments.VIEWER_KIND plesty.server.services.experiments.CONTROL_FILE plesty.server.services.experiments.CANCEL plesty.server.services.experiments.ABORT plesty.server.services.experiments.CREATED_FILE plesty.server.services.experiments.RUN_CONFIG_FILE plesty.server.services.experiments.RUN_REPORT_FILE plesty.server.services.experiments.RUN_LOGS_DIR plesty.server.services.experiments.NOTES_FILE plesty.server.services.experiments.NOTE_MARK plesty.server.services.experiments.LOCAL_HOST_NAME Exceptions ---------- .. autoapisummary:: plesty.server.services.experiments.ExperimentError Functions --------- .. autoapisummary:: plesty.server.services.experiments.read_manifest plesty.server.services.experiments.read_devices plesty.server.services.experiments._ask_module plesty.server.services.experiments.run_environment plesty.server.services.experiments.runs_of plesty.server.services.experiments.check_run_id plesty.server.services.experiments.created_of plesty.server.services.experiments.add_note plesty.server.services.experiments.notes_of plesty.server.services.experiments.create_run plesty.server.services.experiments.has_started plesty.server.services.experiments.rename_run plesty.server.services.experiments.reconfigure_run plesty.server.services.experiments.report_of plesty.server.services.experiments.keep_log plesty.server.services.experiments.kept_log plesty.server.services.experiments.latest plesty.server.services.experiments.find_run plesty.server.services.experiments.resumable plesty.server.services.experiments.status_of plesty.server.services.experiments._replay_status plesty.server.services.experiments.progress_of plesty.server.services.experiments.steps_of plesty.server.services.experiments.records_of plesty.server.services.experiments.ask plesty.server.services.experiments.asked plesty.server.services.experiments.clear plesty.server.services.experiments.reports_in Module Contents --------------- .. py:data:: MANIFEST_TIMEOUT :value: 60.0 .. py:data:: JOB_KIND :value: 'experiment' .. py:data:: VIEWER_KIND :value: 'viewer' .. py:data:: CONTROL_FILE :value: 'control.json' .. py:data:: CANCEL :value: 'cancel' .. py:data:: ABORT :value: 'abort' .. py:data:: CREATED_FILE :value: 'created.json' .. py:data:: RUN_CONFIG_FILE :value: 'config.yaml' .. py:data:: RUN_REPORT_FILE :value: 'report.html' .. py:data:: RUN_LOGS_DIR :value: 'job-logs' .. py:data:: NOTES_FILE :value: 'notes.txt' .. py:data:: NOTE_MARK :value: '--- ' .. py:exception:: ExperimentError Bases: :py:obj:`RuntimeError` Raised when an experiment cannot be read, started or driven. Initialize self. See help(type(self)) for accurate signature. .. py:function:: read_manifest(spec: plesty.server.model.experiment.ExperimentSpec, python: str | pathlib.Path, cwd: str | pathlib.Path, env: Mapping[str, str] | None = None, timeout: float = MANIFEST_TIMEOUT) -> plesty.server.model.experiment.ExperimentManifest Ask the module what it takes, and return what it said. :param spec: The declared experiment. :param python: Interpreter of its installed environment. :param cwd: Where to run it — the checkout, so a relative configuration path in the answer means what it says. :param env: Environment for the call; the manifest needs none, but a module that reads its own package data may. :param timeout: Seconds to wait. :returns: The parsed manifest. :raises ExperimentError: If the command fails, times out, or prints something that is not a manifest. .. py:function:: read_devices(spec: plesty.server.model.experiment.ExperimentSpec, python: str | pathlib.Path, cwd: str | pathlib.Path, config: str, env: Mapping[str, str] | None = None, timeout: float = MANIFEST_TIMEOUT) -> list[str] | None Ask the module which devices of its ``setup.yaml`` a run of *config* uses. Only for a module whose manifest says ``chooses_devices`` (plesty-lib#62). :param spec: The declared experiment. :param python: Interpreter of its installed environment. :param cwd: The checkout. :param config: The configuration file the run will read. :param env: Environment for the call. :param timeout: Seconds to wait. :returns: The device names, or ``None`` for all of them. :raises ExperimentError: If the command fails, times out, or prints no list. .. py:function:: _ask_module(spec: plesty.server.model.experiment.ExperimentSpec, command: list[str], cwd: str | pathlib.Path, env: Mapping[str, str] | None, timeout: float) -> subprocess.CompletedProcess[str] Run a question the module answers without connecting, and capture the answer. :raises ExperimentError: If it could not be run, or did not answer in time. .. py:data:: LOCAL_HOST_NAME :value: 'local' .. py:function:: run_environment(spec: plesty.server.model.experiment.ExperimentSpec, base: Mapping[str, str], dotenv: Mapping[str, str]) -> dict[str, str] Build the environment a run gets. The bench's own environment first, then the module's ``.env``, then what the fleet declares — so the declaration is what an operator edits to point one experiment at a different stage, and the ``.env`` stays the module's own deployment file. :param spec: The declared experiment. :param base: The bench's environment. :param dotenv: The module's ``.env``, already parsed. :returns: The environment for the run. .. py:function:: runs_of(run_root: str | pathlib.Path) -> list[pathlib.Path] Return this experiment's run directories, newest first. A run directory is one holding a ``journal.jsonl`` — or a ``created.json``, for a run named and configured but not yet started. The run root may also hold whatever the module wrote beside its runs, and a directory that is neither is not a run anything can report on. :param run_root: The experiment's run root. :returns: The run directories, newest first by name — run ids carry a date, so name order is close enough to time order to read as it. .. py:function:: check_run_id(text: str) -> str Return *text* if it is usable as a run directory name, else raise. Deliberately the console's own check rather than plesty-lib's, for the same reason the journal is replayed here: the rule is about what a directory name may be, and the console must apply it on any plesty-lib — including one too old to carry the check at all. It is the same rule, and a short one. :param text: The proposed run id. :returns: The same string, trimmed. :raises ExperimentError: If it is empty, a path, or holds a character that does not belong in a directory name. .. py:function:: created_of(run_dir: str | pathlib.Path) -> dict[str, Any] Return what the console recorded when the run was created, if anything. :param run_dir: The run directory. :returns: The document, or ``{}`` for a run that was started directly. .. py:function:: add_note(run_dir: str | pathlib.Path, text: str, by: str, when: Any = None) -> None Append a note to the run's :data:`NOTES_FILE`, while it runs or after it ended. :param run_dir: The run directory. :param text: The note; blank lines inside it are kept. :param by: Who wrote it. :param when: A ``datetime``; now by default (tests pass one). :raises ExperimentError: When the note is empty or cannot be written. .. py:function:: notes_of(run_dir: str | pathlib.Path) -> list[dict[str, str]] The run's notes, oldest first: each ``{"at", "by", "text"}``; none without a file. .. py:function:: create_run(run_dir: str | pathlib.Path, config_text: str | None = None, **fields: Any) -> dict[str, Any] Name a run before it is started, recording what it will be run with. :param run_dir: The directory the run will use — its name is the run id. :param config_text: The run's own configuration, written as its :data:`RUN_CONFIG_FILE`; none is written when ``None``. :param \*\*fields: What to record: the experiment, the configuration file the run's copy came from, and whatever else the run needs to be started later. :returns: The document written. :raises ExperimentError: If the directory already holds a run, or cannot be written. Naming a second run over a started one would interleave two journals into one directory. .. py:function:: has_started(run_dir: str | pathlib.Path) -> bool Whether the run has started: its configuration is frozen into its plan then. .. py:function:: rename_run(run_dir: str | pathlib.Path, new_name: str) -> pathlib.Path Give a run that has not started another name, which is its directory's. The directory moves with everything in it — the run's configuration and its :data:`CREATED_FILE` — and a duplicate recorded as a copy of it follows the new name, so it stays beside it in the table (#27). :param run_dir: The run directory. :param new_name: What to call it; checked as :func:`check_run_id` checks. :returns: The run's directory under its new name. :raises ExperimentError: If the name is unusable or taken, the run has started — its journal and data are written under the name it has — or the directory cannot be moved. .. py:function:: reconfigure_run(run_dir: str | pathlib.Path, config_text: str, config: str) -> pathlib.Path Replace the configuration of a run that has not started. :param run_dir: The run directory. :param config_text: The run's configuration, as it is to be run. :param config: The module's file it was taken from (``config/demo.yaml``), recorded in :data:`CREATED_FILE` so the run table names it. :returns: The run's configuration file. :raises ExperimentError: If the run has started — its plan was built from the configuration it had, and resuming checks it against that — or the files cannot be written. .. py:function:: report_of(run_dir: str | pathlib.Path) -> dict[str, Any] | None The run's own report, or ``None`` when it has none. :param run_dir: The run directory. :returns: Its ``name``, ``path``, ``bytes`` and ``written`` (epoch seconds), or ``None`` when no report has been written — a run that has not ended, or one that ended before this bench wrote them. .. py:function:: keep_log(run_dir: str | pathlib.Path, log: str | pathlib.Path) -> pathlib.Path Copy a job's log into the run it ran, under :data:`RUN_LOGS_DIR`. Copied when the job ends, not written there from the start: a run the module names itself has no directory until it has begun, and the job's log is already being written by then. :param run_dir: The run directory. :param log: The job's log, named after the job. :returns: The copy. :raises OSError: When the log cannot be read or the copy written. .. py:function:: kept_log(run_dir: str | pathlib.Path) -> pathlib.Path | None The newest job log *run_dir* keeps, or ``None`` when it keeps none. Job ids end in their start time, so the newest is the last by name — the one a resumed run was finished by. .. py:function:: latest(run_root: str | pathlib.Path) -> pathlib.Path | None Return the newest run of *run_root*, or ``None`` when there is none. .. py:function:: find_run(run_root: str | pathlib.Path, run_id: str | None) -> pathlib.Path | None Return the run directory *run_id* names, or the newest one. :param run_root: The experiment's run root. :param run_id: The run to find, or ``None`` for the newest. :returns: The directory, or ``None`` when it does not exist. A run created but not yet started counts: it is the one an operator is about to run. .. py:function:: resumable(run_root: str | pathlib.Path) -> str | None Return the id of the newest run that could be continued, if any. A run is resumable when it stopped without finishing: canceled or aborted, or left running by a process that is no longer there. A completed run is not, and neither is one that never wrote a journal. :param run_root: The experiment's run root. :returns: The run id, or ``None``. .. py:function:: status_of(run_dir: str | pathlib.Path) -> dict[str, Any] Return what a run's journal says about it, by replaying the journal here. Deliberately not ``plesty.lib.experiment.Journal.status()``, though it computes the same thing. The run was written by an experiment module, whose plesty-lib has nothing to do with the one this console happens to have installed: delegating would make the same run read differently on two benches, and a run written by a newer module would lose its paused state on an older console. The journal is an append-only document with five event names, and reading it is cheaper than the coupling. :param run_dir: The run directory. :returns: ``state``, ``completed``, ``failed``, ``reports``, ``started``, ``ended``, ``provenance`` and the last event. A run that was named but not yet started is ``created``, which is distinct from ``not_started``: one is a run waiting to go, the other an experiment that has never had one. :rtype: The status mapping .. py:function:: _replay_status(journal: pathlib.Path) -> dict[str, Any] Derive a run's status from its journal. :param journal: The journal file. :returns: The status mapping. .. py:function:: progress_of(run_dir: str | pathlib.Path) -> dict[str, Any] | None Return what the step in flight last said about itself. :param run_dir: The run directory. :returns: The reported progress, or ``None`` when the step reports none. A step that says nothing is not an error — most are short enough not to. .. py:function:: steps_of(run_dir: str | pathlib.Path) -> list[dict[str, Any]] Return the plan's steps, so a status can name the one in flight. :param run_dir: The run directory. :returns: One ``{id, op, params}`` per step; empty when the plan is unreadable. .. py:function:: records_of(run_dir: str | pathlib.Path, limit: int = 200) -> list[dict[str, Any]] Return the newest committed step records. :param run_dir: The run directory. :param limit: How many to return, newest last. :returns: The records in write order. A record is the durable result of its step, so this is the raw data a run has produced so far. .. py:function:: ask(run_dir: str | pathlib.Path, **changes: Any) -> dict[str, Any] Amend a run's control file and return what it now asks for. The file is replaced rather than rewritten, so a run reading it sees either the old document or the new one and never half of each. :param run_dir: The run directory. :param \*\*changes: Fields of the control document to set. :returns: The control document as it now stands. :raises ExperimentError: If the file cannot be written. .. py:function:: asked(run_dir: str | pathlib.Path) -> dict[str, Any] Return what a run is currently being asked for. :param run_dir: The run directory. :returns: The control document; empty when nothing is being asked. .. py:function:: clear(run_dir: str | pathlib.Path) -> None Remove a run's control file, so a resumed run is asked for nothing. A run resumed while its control file still says ``cancel`` would stop at its first step, which reads as a resume that did nothing. :param run_dir: The run directory. .. py:function:: reports_in(*documents: Mapping[str, Any]) -> list[str] Collect the operator's reports out of run statuses, in order. :param \*documents: Status mappings. :returns: Every report line.