plesty.lib.experiment.setup =========================== .. py:module:: plesty.lib.experiment.setup .. autoapi-nested-parse:: An experiment's setup: the benches it runs across, and the instruments on them. Every experiment module ships one ``setup.yaml`` at its root: .. code-block:: yaml version: 1 benches: bench-local: # no address: the machine the run is on data: ${LOCAL_DATA} # where it sees the lab's shared data disk bench-xxx: data: ${BENCH_XXX_DATA} # where it sees the same disk address: ${BENCH_XXX_HOST} # where its bench agent listens token: ${BENCH_XXX_TOKEN} # the secret that agent checks devices: spec: package: plesty-lightfield-spectrometer host_name: bench-xxx # the bench it runs on; this machine's when left out The file is committed with the module and says what the setup is made of, nothing about one lab: where a bench is, its token and where it mounts the shared disk are written as the variable holding each, and the values are set in the checkout's ``.env`` beside it. A value written into the file itself is refused. The one bench without an ``address`` is the machine the run is on, whatever it is called. Every bench mounts the one shared data disk under its own path — its ``data``, the shared point — and every process is told only its own: :meth:`Setup.environment` gives the run ``PLESTY_DATA``. Where a device may write is its bench's to say — its folder on the share — never the setup's: a run routes each device to ``//raw/`` under that folder, and the device answers with the folder relative to the share, which any reader joins to its own ``PLESTY_DATA``. A device entry is its bench's to install, start and give a port; this module reads only its ``host_name``. Attributes ---------- .. autoapisummary:: plesty.lib.experiment.setup.SETUP_FILE plesty.lib.experiment.setup.ENV_FILE plesty.lib.experiment.setup.ENV_EXAMPLE_FILE plesty.lib.experiment.setup.LOCAL_BENCH plesty.lib.experiment.setup.BENCH_FIELDS plesty.lib.experiment.setup._REFERENCE Exceptions ---------- .. autoapisummary:: plesty.lib.experiment.setup.SetupError plesty.lib.experiment.setup.UnsetVariables Classes ------- .. autoapisummary:: plesty.lib.experiment.setup.Bench plesty.lib.experiment.setup.SetupDevice plesty.lib.experiment.setup.Setup Functions --------- .. autoapisummary:: plesty.lib.experiment.setup.reference plesty.lib.experiment.setup.variable_of plesty.lib.experiment.setup.variable_for plesty.lib.experiment.setup.env_stem plesty.lib.experiment.setup.split_benches plesty.lib.experiment.setup.seed_env plesty.lib.experiment.setup.apply_setup Module Contents --------------- .. py:data:: SETUP_FILE :value: 'setup.yaml' .. py:data:: ENV_FILE :value: '.env' .. py:data:: ENV_EXAMPLE_FILE :value: '.env.example' .. py:data:: LOCAL_BENCH :value: 'local' .. py:data:: BENCH_FIELDS :value: ('address', 'token', 'data') .. py:data:: _REFERENCE .. py:exception:: SetupError Bases: :py:obj:`ValueError` Raised when a setup cannot be read, or writes a value where a variable belongs. Initialize self. See help(type(self)) for accurate signature. .. py:function:: reference(variable: str) -> str How a setup names *variable*: ``${SPEC_PC_ADDRESS}``. .. py:function:: variable_of(written: str) -> str The variable a setup's *written* field names — ``SPEC_PC_TOKEN`` for ``${SPEC_PC_TOKEN}``. :raises SetupError: When *written* is a value rather than a variable. .. py:function:: variable_for(bench: str, field_name: str) -> str The variable a bench's field is held in when nobody named one: ``SPEC_PC_ADDRESS``. .. py:function:: env_stem(name: str) -> str A bench or device *name* as the start of its variables: ``spec-pc`` → ``SPEC_PC``. .. py:class:: Bench One bench of a setup. As a setup writes it, each field is the ``${VARIABLE}`` holding the value; as :meth:`Setup.resolve` gives it, the value itself. :ivar name: What devices call it in ``host_name``. :ivar address: Where its bench agent listens: ``host``, ``host:port`` or a ZMQ endpoint. Empty for the machine the run is on. :ivar token: The secret that agent checks; empty when it checks none. :ivar data: Where it sees the lab's shared data disk; empty when unset. .. py:attribute:: name :type: str .. py:attribute:: address :type: str :value: '' .. py:attribute:: token :type: str :value: '' .. py:attribute:: data :type: str :value: '' .. py:property:: is_local :type: bool the bench with no address. :type: Whether it is the machine the run is on .. py:property:: host :type: str The machine, without the agent's port or scheme; ``127.0.0.1`` for the local one. .. py:method:: to_dict() -> dict[str, str] The bench's entry, without its name; empty fields left out. .. py:method:: from_dict(name: str, raw: Any) -> Bench :classmethod: Read the bench *name* of a setup. :raises SetupError: When the entry is not a mapping, has a key a bench does not take, or writes a value where a variable belongs. .. py:class:: SetupDevice What a setup says about one device, as far as a run is concerned. :ivar name: The device's name; the run reads its address from ``_ADDRESS``, upper-cased, which its bench provides. :ivar host_name: The bench it runs on; empty for the machine the run is on. .. py:attribute:: name :type: str .. py:attribute:: host_name :type: str :value: '' .. py:method:: from_dict(name: str, raw: Any) -> SetupDevice :classmethod: Read what a run needs of the device entry *name*; its other keys are the bench's. :raises SetupError: When the entry is not a mapping. .. py:class:: Setup The benches and devices of a module's ``setup.yaml``. :ivar benches: By name, as written — each field the variable holding it. :ivar devices: By name, in file order. .. py:attribute:: benches :type: collections.abc.Mapping[str, Bench] .. py:attribute:: devices :type: collections.abc.Mapping[str, SetupDevice] .. py:method:: from_dict(raw: Any) -> Setup :classmethod: Read a parsed ``setup.yaml``. :raises SetupError: When it is not a mapping of ``benches`` and ``devices``, a bench or device entry is malformed, more than one bench has no address, or a device names a bench the setup does not declare. .. py:method:: load(root: str | os.PathLike[str] = '.') -> Setup :classmethod: Read the ``setup.yaml`` of the module checkout at *root*. :raises SetupError: When there is none, or it does not parse. .. py:method:: only(devices: collections.abc.Iterable[str]) -> Setup This setup reduced to *devices* and the benches they run on (#62). A run that uses some of a module's devices needs nothing of the benches serving only the others: their addresses, tokens and data need not be set. The bench without an address — this machine, where the run reads ``PLESTY_DATA`` — is always kept. :raises SetupError: When *devices* names a device the setup does not have. .. py:method:: variables() -> list[str] Every variable the benches name, in file order. .. py:method:: unset(env: collections.abc.Mapping[str, str]) -> list[str] The variables the benches name that *env* leaves empty. .. py:method:: resolve(env: collections.abc.Mapping[str, str]) -> dict[str, Bench] The benches with each field's value read from *env*; empty where unset. .. py:method:: local_name() -> str The bench the file writes without an address; ``local`` when it declares none. Read from the file, not from the values: a bench whose address variable is unset has an empty address once resolved, and is still another machine. .. py:method:: bench_of(device: str, env: collections.abc.Mapping[str, str]) -> Bench The bench *device* runs on, with its values read from *env*. A device naming no bench — or ``local`` when the setup declares no bench for this machine — runs on the machine the run is on: the setup's bench without an address, or one called ``local`` when it declares none. :raises KeyError: When the setup has no such device. :raises SetupError: When the device's bench is another machine whose address *env* does not set. .. py:method:: environment(env: collections.abc.Mapping[str, str]) -> dict[str, str] What a run on this machine is given, built from the setup and the values in *env*. A device's address is not among it: its bench starts it, picks its port, and gives the run ``_ADDRESS``. :param env: The run's environment, ``.env`` included — where the values the benches name are read. :returns: ``PLESTY_DATA`` — where this machine sees the shared data disk — when its bench says so; else nothing. .. py:method:: record(env: collections.abc.Mapping[str, str]) -> dict[str, dict[str, str]] Where each device was for a run: what the run journals when it starts. The setup says which bench a device runs on, and only the run's environment says the rest — the address its bench served it on, which changes from start to start, and where that bench sees the shared disk. Nothing is refused here: a run that got this far connects with what it was given, and the record says what that was. A bench's token is never recorded. :param env: The run's environment, ``.env`` included. :returns: ``bench``, the bench it runs on; ``address``, the run's ``_ADDRESS``; ``data``, that bench's shared point in its own spelling. A value nobody set is empty. :rtype: By device, in file order .. py:function:: split_benches(benches: collections.abc.Sequence[Bench], written: Optional[collections.abc.Mapping[str, Bench]] = None) -> tuple[dict[str, dict[str, str]], dict[str, str]] Split benches given by value into what ``setup.yaml`` says and what ``.env`` holds. A field keeps the variable the setup already names for it; a new one gets :func:`variable_for`. An empty field is left out of both. :param benches: The benches, each field its value. :param written: The benches as the setup writes them now. :returns: ``benches:`` for the setup — name → field → ``${VARIABLE}`` — and variable → value for ``.env``. .. py:function:: seed_env(root: str | os.PathLike[str] = '.') -> list[str] Give the checkout at *root* a ``.env`` line for every variable its setup names. A variable already there is left as it is; a new one takes the value the module's ``.env.example`` suggests, else none — to be filled in. A module with no setup gets nothing. :returns: The variables added. :raises SetupError: When the setup does not parse. .. py:exception:: UnsetVariables(names: collections.abc.Sequence[str], env: pathlib.Path) Bases: :py:obj:`SetupError` Raised when the setup names variables nothing gives a value. :ivar names: The variables, in the order the setup names them. :ivar env: The ``.env`` file they are set in. Say which variables to set, and where. .. py:attribute:: names .. py:attribute:: env .. py:function:: apply_setup(root: str | os.PathLike[str] = '.', require: bool = True, devices: Optional[collections.abc.Iterable[str]] = None) -> dict[str, str] Put what the setup at *root* builds into ``os.environ``, under what is set already. The checkout's ``.env`` is where the values the setup names are set: a fresh checkout has none, so it is made first (:func:`seed_env`), with a line for each. A variable already in the environment — set by hand, in ``.env``, or by a bench that started the run — is left as it is. :param root: The module checkout. :param require: Refuse when a variable the setup names has no value, rather than build what can be built. A run that connects to nothing — printing its plan — needs none of them. :param devices: The setup's devices this run uses (:meth:`Setup.only`); ``None`` for all of them. :returns: The variables this set. :raises UnsetVariables: When *require* and a variable the setup names is empty — before anything connects, naming them and the file. :raises SetupError: When the setup does not parse.