plesty.lib.test.report_artifact

The published projection of a field test report.

A field test produces two artifacts, for two readers.

The markdown and JSONL reports are for the person who just ran the thing. They carry the host name, the instrument address, its serial number and the lab’s file paths, because that reader needs to know which rig produced them. They are git-ignored: a hub repository is world-readable, and a field test report is a small inventory of the lab.

field-test.json is the published artifact: what was verified, how well it performed, and what the module offers — with nothing that says where. It is committed, and read by hub.plesty.net and manager.plesty.net.

Sanitisation is this module’s job, not the module author’s. Every field in the document is either derived from the module’s own source and schemas — already public — or is a measurement. There is no free-text field a host name can reach by accident, and write_document() refuses a document that fails validate_document() rather than publishing it.

Layout, relative to the module root:

reports/
  check.json                    plesty check            committed
  <instrument>/
    field-test.json             field test              committed
  *.md                          every report            git-ignored
  *.jsonl                       raw per-run records     git-ignored

check.json sits at the top because the standard gates are a property of the module and have no instrument. A field test report belongs to a module and the hardware it ran against, so it lives in a directory named for that hardware: a powermeter with a silicon photodiode reports different limits and different timings than the same module with a thermopile head, and merging them would publish whichever ran last as though it were the module’s behaviour.

Attributes

`SCHEMA_VERSION`
`SEVERITIES`
`VERIFICATION_STATES`
`PUBLIC_ENVIRONMENT_KEYS`
`INSTRUMENT_SLUG_PATTERN`
`_SLUG_PATTERN`
`_MODULE_TYPES`

Exceptions

ReportSchemaError

A document does not satisfy the published artifact contract.

Classes

`Finding`	Something wrong with the module, and what to do about it.
`Instrument`	The hardware a field test ran against, as far as it may be published.
`ModuleInfo`	Identification of the module a report belongs to.
`Coverage`	How much of what the module declares the run actually exercised.

Functions

`build_document`(→ dict[str, Any])	Assemble a `field-test` document.
`validate_document`(→ None)	Check a document against the published artifact contract.
`write_document`(→ pathlib.Path)	Validate a document and write it under `<reports_dir>/<slug>/`.
`_validate_coverage`(→ None)	Check the `coverage` block, which the docs generator walks by name.
`_require_keys`(→ None)	Require value to be a mapping with exactly the required keys.
`_require_text`(→ None)	Require value to be a non-empty string.
`_require_version`(→ None)	Require value to be a `major.minor` version this writer produces.
`_call_record`(→ dict[str, Any])	Build one publishable call record from the run's own statistics.
`latency_ms`(→ dict[str, float])	Convert a `LatencyStats` mapping to milliseconds.
`_distribution_version`(→ str)	Resolve the module's version, preferring the installed distribution.
`_repository_path`(→ str \| None)	Return the origin remote's path without its host, or `None`.
`_git`(→ str)	Run a read-only git command in root, returning its output or `""`.

Module Contents

plesty.lib.test.report_artifact.SCHEMA_VERSION = '1.0'

plesty.lib.test.report_artifact.SEVERITIES: tuple[str, Ellipsis] = ('error', 'warning', 'info')

plesty.lib.test.report_artifact.VERIFICATION_STATES: tuple[str, Ellipsis] = ('declared', 'queried', 'roundtrip', 'called')

plesty.lib.test.report_artifact.PUBLIC_ENVIRONMENT_KEYS: tuple[str, Ellipsis] = ('platform', 'python', 'plesty_lib')

plesty.lib.test.report_artifact.INSTRUMENT_SLUG_PATTERN

plesty.lib.test.report_artifact._SLUG_PATTERN

plesty.lib.test.report_artifact._MODULE_TYPES: tuple[str, Ellipsis] = ('device', 'analyzer', 'experiment', 'default')

exception plesty.lib.test.report_artifact.ReportSchemaError

Bases: ValueError

A document does not satisfy the published artifact contract.

Raised instead of writing, because a malformed artifact published to a world-readable repository is not something a later fix takes back.

Initialize self. See help(type(self)) for accurate signature.

class plesty.lib.test.report_artifact.Finding

Something wrong with the module, and what to do about it.

A gate that failed says which gate; a finding says what is wrong with the module, and carries the fix. The two are not the same: a schema declaring a wider range than the hardware accepts is a finding even when the run passes.

Variables:

severity – One of SEVERITIES.
kind – Machine-readable finding class, e.g. "range_wider_than_hardware". Lowercase, underscore-separated.
subject – The parameter, operation or gate the finding is about.
summary – One sentence, for a table row.
detail – What was observed, in the module’s own terms.
action – What the developer should do. A finding without an action is an observation, and observations belong in the markdown report.

severity: str

kind: str

subject: str

summary: str

detail: str = ''

action: str = ''

to_dict() → dict[str, Any]

Return a JSON-serialisable mapping of the finding.

Return type:: dict[str, Any]

class plesty.lib.test.report_artifact.Instrument

The hardware a field test ran against, as far as it may be published.

The model is a property of the module — the repository name already implies it. The serial number and the address are not here, and there is no field they could reach.

The transport is deliberately absent. No gate reads it, nothing branches on it, and the module’s own repository already says which one it uses; it was a grouping hint for a comparison a reader can make without it, and it cost a naming scheme across five classes to produce.

Variables:

model – Instrument model, e.g. "PM100D". For a module driving several at once, the models joined by +.
sensor – The interchangeable part fitted, when the module has one — a sensor head, a stage, a camera. It changes the measured limits and timings, so it is what distinguishes two runs of one module.

model: str

sensor: str | None = None

property slug: str

Directory name for this instrument’s reports.

The fitted part when there is one, because that is what distinguishes two runs of the same module; the model otherwise.

Return type:: str

to_dict() → dict[str, Any]

Return a JSON-serialisable mapping of the instrument.

Return type:: dict[str, Any]

class plesty.lib.test.report_artifact.ModuleInfo

Identification of the module a report belongs to.

Variables:

name – Distribution name, e.g. "plesty-pm100d".
version – Resolved version, including the versioningit suffix.
module_type – One of device, analyzer, experiment, default.
standard – Compliance standard the module is held to.
repository – Repository path without the host, e.g. "plesty/hub/devices/thorlab/plesty-pm100d". None when the project has no remote — a local checkout publishes no path.

name: str

version: str

module_type: str

standard: str

repository: str | None = None

classmethod from_project(project_root: str | pathlib.Path = '.') → ModuleInfo

Read the module’s identification from its own project.

Every field comes from pyproject.toml, the installed distribution metadata or the git remote — all of them already public.

Parameters:: project_root (str | pathlib.Path) – Directory holding pyproject.toml.
Returns:: The module identification.
Raises:: ReportSchemaError – If pyproject.toml is missing or names no project. A report cannot say which module it belongs to.
Return type:: ModuleInfo

to_dict() → dict[str, Any]

Return a JSON-serialisable mapping of the module identification.

Return type:: dict[str, Any]

class plesty.lib.test.report_artifact.Coverage

How much of what the module declares the run actually exercised.

Keyed by what the module declares, not by what the run happened to call. A parameter nobody exercised appears with verified: "declared" and no calls; the alternative is a schema presented as though every entry were equally trustworthy, which is how a setting that crashes the controlling application stays undiscovered.

Variables:

parameters – Declared configuration keys, in schema order.
operations – Declared operation names, in schema order.
verified – Per name, one of VERIFICATION_STATES. Names absent from the mapping are reported as declared.

parameters: list[str] = []

operations: list[str] = []

verified: dict[str, str]

to_dict(timing_profile: dict[str, Any], failure_stats: dict[str, Any]) → dict[str, Any]

Build the published coverage block.

Parameters:

timing_profile (dict[str, Any]) – Per command latency statistics, as {"query:wavelength": {...}}.
failure_stats (dict[str, Any]) – Per command attempts and failures, same keys.

Returns:

The coverage mapping, with one entry per declared parameter and operation.

Return type:

dict[str, Any]

_entry(label: str, name: str, commands: dict[str, str], timing_profile: dict[str, Any], failure_stats: dict[str, Any]) → dict[str, Any]

Build one coverage entry, folding its measurements in.

Parameters:

label (str) – Key naming the entry — "key" or "name".
name (str) – Parameter key or operation name.
commands (dict[str, str]) – Call role mapped to the report key it was recorded under.
timing_profile (dict[str, Any]) – Per command latency statistics.
failure_stats (dict[str, Any]) – Per command attempts and failures.

Return type:

dict[str, Any]

plesty.lib.test.report_artifact.build_document(*, module: ModuleInfo, instrument: Instrument, tier: str = 'host', generated_utc: str, environment: dict[str, Any], outcome: dict[str, int | bool], gates: list[dict[str, Any]], coverage: dict[str, Any], lifecycle: list[dict[str, Any]], findings: list[Finding]) → dict[str, Any]

Assemble a field-test document.

Parameters:

module (ModuleInfo) – Module identification.
instrument (Instrument) – The hardware the run was pointed at.
tier (str) – Which path the run exercised — “host” in-process, or “client” through the device server. A consumer comparing two runs of one module keys on this rather than on the file name.
generated_utc (str) – ISO-8601 UTC timestamp of the run.
environment (dict[str, Any]) – Environment mapping; filtered to PUBLIC_ENVIRONMENT_KEYS, so passing the full one is safe.
outcome (dict[str, int | bool]) – passed / failed / skipped counts and ok.
gates (list[dict[str, Any]]) – Per gate name, status, duration and reason.
coverage (dict[str, Any]) – The mapping built by Coverage.to_dict().
lifecycle (list[dict[str, Any]]) – Connect, identity and disconnect timings.
findings (list[Finding]) – What the run found wrong with the module.

Returns:

The document, ready for validate_document().

Return type:

dict[str, Any]

plesty.lib.test.report_artifact.validate_document(document: dict[str, Any]) → None

Check a document against the published artifact contract.

Deliberately strict about keys as well as values: an unknown key is how a host name, a serial number or a file path would arrive in a published document, and every legitimate field is enumerated here.

Parameters:: document (dict[str, Any]) – The document to check.
Raises:: ReportSchemaError – On the first violation, naming the field.
Return type:: None

plesty.lib.test.report_artifact.write_document(reports_dir: str | pathlib.Path, document: dict[str, Any], instrument_slug: str, filename: str = 'field-test.json') → pathlib.Path

Validate a document and write it under <reports_dir>/<slug>/.

Parameters:

reports_dir (str | pathlib.Path) – The module’s reports/ directory.
document (dict[str, Any]) – The document to publish.
instrument_slug (str) – Directory name for the hardware this run used — Instrument.slug.
filename (str) – What to call it. The client tier publishes alongside the host tier rather than over it: the two answer different questions about the same instrument, and a run that overwrote the other would leave the module looking tested at one layer only.

Returns:

The path written.

Raises:

ReportSchemaError – If the document or the slug fails validation. Nothing is written in that case.

Return type:

pathlib.Path

plesty.lib.test.report_artifact._validate_coverage(coverage: Any) → None

Check the coverage block, which the docs generator walks by name.

Parameters:: coverage (Any)
Return type:: None

plesty.lib.test.report_artifact._require_keys(value: Any, where: str, required: tuple[str, Ellipsis]) → None

Require value to be a mapping with exactly the required keys.

Parameters:

value (Any) – The value to check.
where (str) – Path of the value, for the error message.
required (tuple[str, Ellipsis]) – The complete key set.

Raises:

ReportSchemaError – If the value is not a mapping, or its keys differ.

Return type:

None

plesty.lib.test.report_artifact._require_text(value: Any, where: str) → None

Require value to be a non-empty string.

Parameters:

value (Any)
where (str)

Return type:

None

plesty.lib.test.report_artifact._require_version(value: Any) → None

Require value to be a major.minor version this writer produces.

Parameters:: value (Any)
Return type:: None

plesty.lib.test.report_artifact._call_record(stats: dict[str, Any] | None, latency: dict[str, Any] | None) → dict[str, Any]

Build one publishable call record from the run’s own statistics.

FailureStats.to_dict carries last_error, the string form of the most recent exception. That is the one field in the whole report where a file path or an instrument address arrives as free text, so it is dropped here rather than filtered later — the counts say a command failed and how, which is what a published artifact is for; the message stays in the git-ignored markdown for whoever has to fix it.

Parameters:

stats (dict[str, Any] | None) – The FailureStats.to_dict() form, or None.
latency (dict[str, Any] | None) – The LatencyStats.to_dict() form, or None.

Returns:

Attempts, failures, failure rate, the per-class breakdown, and latency in milliseconds.

Return type:

dict[str, Any]

plesty.lib.test.report_artifact.latency_ms(latency: dict[str, Any] | None) → dict[str, float]

Convert a LatencyStats mapping to milliseconds.

Parameters:: latency (dict[str, Any] | None) – The to_dict() form, whose values are seconds, or None when the command was never timed.
Returns:: min/median/p95/max in milliseconds; zeros when the command produced no successful call, which is itself the measurement.
Return type:: dict[str, float]

plesty.lib.test.report_artifact._distribution_version(name: str, root: pathlib.Path) → str

Resolve the module’s version, preferring the installed distribution.

Parameters:

name (str) – Distribution name from pyproject.toml.
root (pathlib.Path) – Project root, used for the git fallback.

Returns:

The version string, or "unknown" when neither source answers.

Return type:

str

plesty.lib.test.report_artifact._repository_path(root: pathlib.Path) → str | None

Return the origin remote’s path without its host, or None.

The path is what a consumer links to and is public by construction; the host is dropped because a module’s repository is identified by its path in every PLESTY surface that renders one.

Parameters:: root (pathlib.Path) – Project root.
Return type:: str | None

plesty.lib.test.report_artifact._git(root: pathlib.Path, *args: str) → str

Run a read-only git command in root, returning its output or "".

Parameters:

root (pathlib.Path)
args (str)

Return type:

str