plesty.lib.experiment.setup

An experiment’s setup: the benches it runs across, and the instruments on them.

Every experiment module ships one setup.yaml at its root:

version: 1
benches:
  bench-local:                    # no address: the machine the run is on
    data: ${LOCAL_DATA}           # where it sees the lab's shared data disk
  bench-xxx:
    data: ${BENCH_XXX_DATA}       # where it sees the same disk
    address: ${BENCH_XXX_HOST}    # where its bench agent listens
    token: ${BENCH_XXX_TOKEN}     # the secret that agent checks
devices:
  spec:
    package: plesty-lightfield-spectrometer
    host_name: bench-xxx          # the bench it runs on; this machine's when left out

The file is committed with the module and says what the setup is made of, nothing about one lab: where a bench is, its token and where it mounts the shared disk are written as the variable holding each, and the values are set in the checkout’s .env beside it. A value written into the file itself is refused. The one bench without an address is the machine the run is on, whatever it is called.

Every bench mounts the one shared data disk under its own path — its data, the shared point — and every process is told only its own: Setup.environment() gives the run PLESTY_DATA. Where a device may write is its bench’s to say — its folder on the share — never the setup’s: a run routes each device to <experiment root>/<run id>/raw/<device> under that folder, and the device answers with the folder relative to the share, which any reader joins to its own PLESTY_DATA. A device entry is its bench’s to install, start and give a port; this module reads only its host_name.

Attributes

`SETUP_FILE`
`ENV_FILE`
`ENV_EXAMPLE_FILE`
`LOCAL_BENCH`
`BENCH_FIELDS`
`_REFERENCE`

Exceptions

`SetupError`	Raised when a setup cannot be read, or writes a value where a variable belongs.
`UnsetVariables`	Raised when the setup names variables nothing gives a value.

Classes

`Bench`	One bench of a setup.
`SetupDevice`	What a setup says about one device, as far as a run is concerned.
`Setup`	The benches and devices of a module's `setup.yaml`.

Functions

`reference`(→ str)	How a setup names variable: `${SPEC_PC_ADDRESS}`.
`variable_of`(→ str)	The variable a setup's written field names — `SPEC_PC_TOKEN` for `${SPEC_PC_TOKEN}`.
`variable_for`(→ str)	The variable a bench's field is held in when nobody named one: `SPEC_PC_ADDRESS`.
`env_stem`(→ str)	A bench or device name as the start of its variables: `spec-pc` → `SPEC_PC`.
`split_benches`(→ tuple[dict[str, dict[str, str]], ...)	Split benches given by value into what `setup.yaml` says and what `.env` holds.
`seed_env`(→ list[str])	Give the checkout at root a `.env` line for every variable its setup names.
`apply_setup`(→ dict[str, str])	Put what the setup at root builds into `os.environ`, under what is set already.

Module Contents

plesty.lib.experiment.setup.SETUP_FILE = 'setup.yaml'

plesty.lib.experiment.setup.ENV_FILE = '.env'

plesty.lib.experiment.setup.ENV_EXAMPLE_FILE = '.env.example'

plesty.lib.experiment.setup.LOCAL_BENCH = 'local'

plesty.lib.experiment.setup.BENCH_FIELDS = ('address', 'token', 'data')

plesty.lib.experiment.setup._REFERENCE

exception plesty.lib.experiment.setup.SetupError

Bases: ValueError

Raised when a setup cannot be read, or writes a value where a variable belongs.

Initialize self. See help(type(self)) for accurate signature.

plesty.lib.experiment.setup.reference(variable: str) → str

How a setup names variable: ${SPEC_PC_ADDRESS}.

Parameters:: variable (str)
Return type:: str

plesty.lib.experiment.setup.variable_of(written: str) → str

The variable a setup’s written field names — SPEC_PC_TOKEN for ${SPEC_PC_TOKEN}.

Raises:: SetupError – When written is a value rather than a variable.
Parameters:: written (str)
Return type:: str

plesty.lib.experiment.setup.variable_for(bench: str, field_name: str) → str

The variable a bench’s field is held in when nobody named one: SPEC_PC_ADDRESS.

Parameters:

bench (str)
field_name (str)

Return type:

str

plesty.lib.experiment.setup.env_stem(name: str) → str

A bench or device name as the start of its variables: spec-pc → SPEC_PC.

Parameters:: name (str)
Return type:: str

class plesty.lib.experiment.setup.Bench

One bench of a setup.

As a setup writes it, each field is the ${VARIABLE} holding the value; as Setup.resolve() gives it, the value itself.

Variables:

name – What devices call it in host_name.
address – Where its bench agent listens: host, host:port or a ZMQ endpoint. Empty for the machine the run is on.
token – The secret that agent checks; empty when it checks none.
data – Where it sees the lab’s shared data disk; empty when unset.

name: str

address: str = ''

token: str = ''

data: str = ''

property is_local: bool

the bench with no address.

Type:: Whether it is the machine the run is on
Return type:: bool

property host: str

The machine, without the agent’s port or scheme; 127.0.0.1 for the local one.

Return type:: str

to_dict() → dict[str, str]

The bench’s entry, without its name; empty fields left out.

Return type:: dict[str, str]

classmethod from_dict(name: str, raw: Any) → Bench

Read the bench name of a setup.

Raises:

SetupError – When the entry is not a mapping, has a key a bench does not take, or writes a value where a variable belongs.

Parameters:

name (str)
raw (Any)

Return type:

Bench

class plesty.lib.experiment.setup.SetupDevice

What a setup says about one device, as far as a run is concerned.

Variables:

name – The device’s name; the run reads its address from <NAME>_ADDRESS, upper-cased, which its bench provides.
host_name – The bench it runs on; empty for the machine the run is on.

name: str

host_name: str = ''

classmethod from_dict(name: str, raw: Any) → SetupDevice

Read what a run needs of the device entry name; its other keys are the bench’s.

Raises:

SetupError – When the entry is not a mapping.

Parameters:

name (str)
raw (Any)

Return type:

SetupDevice

class plesty.lib.experiment.setup.Setup

The benches and devices of a module’s setup.yaml.

Variables:

benches – By name, as written — each field the variable holding it.
devices – By name, in file order.

benches: collections.abc.Mapping[str, Bench]

devices: collections.abc.Mapping[str, SetupDevice]

classmethod from_dict(raw: Any) → Setup

Read a parsed setup.yaml.

Raises:: SetupError – When it is not a mapping of benches and devices, a bench or device entry is malformed, more than one bench has no address, or a device names a bench the setup does not declare.
Parameters:: raw (Any)
Return type:: Setup

classmethod load(root: str | os.PathLike[str] = '.') → Setup

Read the setup.yaml of the module checkout at root.

Raises:: SetupError – When there is none, or it does not parse.
Parameters:: root (str | os.PathLike[str])
Return type:: Setup

only(devices: collections.abc.Iterable[str]) → Setup

This setup reduced to devices and the benches they run on (#62).

A run that uses some of a module’s devices needs nothing of the benches serving only the others: their addresses, tokens and data need not be set. The bench without an address — this machine, where the run reads PLESTY_DATA — is always kept.

Raises:: SetupError – When devices names a device the setup does not have.
Parameters:: devices (collections.abc.Iterable[str])
Return type:: Setup

variables() → list[str]

Every variable the benches name, in file order.

Return type:: list[str]

unset(env: collections.abc.Mapping[str, str]) → list[str]

The variables the benches name that env leaves empty.

Parameters:: env (collections.abc.Mapping[str, str])
Return type:: list[str]

resolve(env: collections.abc.Mapping[str, str]) → dict[str, Bench]

The benches with each field’s value read from env; empty where unset.

Parameters:: env (collections.abc.Mapping[str, str])
Return type:: dict[str, Bench]

local_name() → str

The bench the file writes without an address; local when it declares none.

Read from the file, not from the values: a bench whose address variable is unset has an empty address once resolved, and is still another machine.

Return type:: str

bench_of(device: str, env: collections.abc.Mapping[str, str]) → Bench

The bench device runs on, with its values read from env.

A device naming no bench — or local when the setup declares no bench for this machine — runs on the machine the run is on: the setup’s bench without an address, or one called local when it declares none.

Raises:

KeyError – When the setup has no such device.
SetupError – When the device’s bench is another machine whose address env does not set.

Parameters:

device (str)
env (collections.abc.Mapping[str, str])

Return type:

Bench

environment(env: collections.abc.Mapping[str, str]) → dict[str, str]

What a run on this machine is given, built from the setup and the values in env.

A device’s address is not among it: its bench starts it, picks its port, and gives the run <DEVICE>_ADDRESS.

Parameters:: env (collections.abc.Mapping[str, str]) – The run’s environment, .env included — where the values the benches name are read.
Returns:: PLESTY_DATA — where this machine sees the shared data disk — when its bench says so; else nothing.
Return type:: dict[str, str]

record(env: collections.abc.Mapping[str, str]) → dict[str, dict[str, str]]

Where each device was for a run: what the run journals when it starts.

The setup says which bench a device runs on, and only the run’s environment says the rest — the address its bench served it on, which changes from start to start, and where that bench sees the shared disk. Nothing is refused here: a run that got this far connects with what it was given, and the record says what that was. A bench’s token is never recorded.

Parameters:: env (collections.abc.Mapping[str, str]) – The run’s environment, .env included.
Returns:: bench, the bench it runs on; address, the run’s <DEVICE>_ADDRESS; data, that bench’s shared point in its own spelling. A value nobody set is empty.
Return type:: By device, in file order

plesty.lib.experiment.setup.split_benches(benches: collections.abc.Sequence[Bench], written: collections.abc.Mapping[str, Bench] | None = None) → tuple[dict[str, dict[str, str]], dict[str, str]]

Split benches given by value into what setup.yaml says and what .env holds.

A field keeps the variable the setup already names for it; a new one gets variable_for(). An empty field is left out of both.

Parameters:

benches (collections.abc.Sequence[Bench]) – The benches, each field its value.
written (Optional[collections.abc.Mapping[str, Bench]]) – The benches as the setup writes them now.

Returns:

benches: for the setup — name → field → ${VARIABLE} — and variable → value for .env.

Return type:

tuple[dict[str, dict[str, str]], dict[str, str]]

plesty.lib.experiment.setup.seed_env(root: str | os.PathLike[str] = '.') → list[str]

Give the checkout at root a .env line for every variable its setup names.

A variable already there is left as it is; a new one takes the value the module’s .env.example suggests, else none — to be filled in. A module with no setup gets nothing.

Returns:: The variables added.
Raises:: SetupError – When the setup does not parse.
Parameters:: root (str | os.PathLike[str])
Return type:: list[str]

exception plesty.lib.experiment.setup.UnsetVariables(names: collections.abc.Sequence[str], env: pathlib.Path)

Bases: SetupError

Raised when the setup names variables nothing gives a value.

Variables:

names – The variables, in the order the setup names them.
env – The .env file they are set in.

Parameters:

names (collections.abc.Sequence[str])
env (pathlib.Path)

Say which variables to set, and where.

names

env

plesty.lib.experiment.setup.apply_setup(root: str | os.PathLike[str] = '.', require: bool = True, devices: collections.abc.Iterable[str] | None = None) → dict[str, str]

Put what the setup at root builds into os.environ, under what is set already.

The checkout’s .env is where the values the setup names are set: a fresh checkout has none, so it is made first (seed_env()), with a line for each. A variable already in the environment — set by hand, in .env, or by a bench that started the run — is left as it is.

Parameters:

root (str | os.PathLike[str]) – The module checkout.
require (bool) – Refuse when a variable the setup names has no value, rather than build what can be built. A run that connects to nothing — printing its plan — needs none of them.
devices (Optional[collections.abc.Iterable[str]]) – The setup’s devices this run uses (Setup.only()); None for all of them.

Returns:

The variables this set.

Raises:

UnsetVariables – When require and a variable the setup names is empty — before anything connects, naming them and the file.
SetupError – When the setup does not parse.

Return type:

dict[str, str]