plesty.lib.test.report_artifact =============================== .. py:module:: plesty.lib.test.report_artifact .. autoapi-nested-parse:: The published projection of a field test report. A field test produces two artifacts, for two readers. The markdown and JSONL reports are for the person who just ran the thing. They carry the host name, the instrument address, its serial number and the lab's file paths, because that reader needs to know which rig produced them. They are git-ignored: a hub repository is world-readable, and a field test report is a small inventory of the lab. ``field-test.json`` is the published artifact: what was verified, how well it performed, and what the module offers — with nothing that says *where*. It is committed, and read by ``hub.plesty.net`` and ``manager.plesty.net``. Sanitisation is this module's job, not the module author's. Every field in the document is either derived from the module's own source and schemas — already public — or is a measurement. There is no free-text field a host name can reach by accident, and :func:`write_document` refuses a document that fails :func:`validate_document` rather than publishing it. Layout, relative to the module root:: reports/ check.json plesty check committed / field-test.json field test committed *.md every report git-ignored *.jsonl raw per-run records git-ignored ``check.json`` sits at the top because the standard gates are a property of the module and have no instrument. A field test report belongs to a module *and the hardware it ran against*, so it lives in a directory named for that hardware: a powermeter with a silicon photodiode reports different limits and different timings than the same module with a thermopile head, and merging them would publish whichever ran last as though it were the module's behaviour. Attributes ---------- .. autoapisummary:: plesty.lib.test.report_artifact.SCHEMA_VERSION plesty.lib.test.report_artifact.SEVERITIES plesty.lib.test.report_artifact.VERIFICATION_STATES plesty.lib.test.report_artifact.PUBLIC_ENVIRONMENT_KEYS plesty.lib.test.report_artifact.INSTRUMENT_SLUG_PATTERN plesty.lib.test.report_artifact._SLUG_PATTERN plesty.lib.test.report_artifact._MODULE_TYPES Exceptions ---------- .. autoapisummary:: plesty.lib.test.report_artifact.ReportSchemaError Classes ------- .. autoapisummary:: plesty.lib.test.report_artifact.Finding plesty.lib.test.report_artifact.Instrument plesty.lib.test.report_artifact.ModuleInfo plesty.lib.test.report_artifact.Coverage Functions --------- .. autoapisummary:: plesty.lib.test.report_artifact.build_document plesty.lib.test.report_artifact.validate_document plesty.lib.test.report_artifact.write_document plesty.lib.test.report_artifact._validate_coverage plesty.lib.test.report_artifact._require_keys plesty.lib.test.report_artifact._require_text plesty.lib.test.report_artifact._require_version plesty.lib.test.report_artifact._call_record plesty.lib.test.report_artifact.latency_ms plesty.lib.test.report_artifact._distribution_version plesty.lib.test.report_artifact._repository_path plesty.lib.test.report_artifact._git Module Contents --------------- .. py:data:: SCHEMA_VERSION :value: '1.0' .. py:data:: SEVERITIES :type: tuple[str, Ellipsis] :value: ('error', 'warning', 'info') .. py:data:: VERIFICATION_STATES :type: tuple[str, Ellipsis] :value: ('declared', 'queried', 'roundtrip', 'called') .. py:data:: PUBLIC_ENVIRONMENT_KEYS :type: tuple[str, Ellipsis] :value: ('platform', 'python', 'plesty_lib') .. py:data:: INSTRUMENT_SLUG_PATTERN .. py:data:: _SLUG_PATTERN .. py:data:: _MODULE_TYPES :type: tuple[str, Ellipsis] :value: ('device', 'analyzer', 'experiment', 'default') .. py:exception:: ReportSchemaError Bases: :py:obj:`ValueError` A document does not satisfy the published artifact contract. Raised instead of writing, because a malformed artifact published to a world-readable repository is not something a later fix takes back. Initialize self. See help(type(self)) for accurate signature. .. py:class:: Finding Something wrong with the module, and what to do about it. A gate that failed says which gate; a finding says what is wrong with the *module*, and carries the fix. The two are not the same: a schema declaring a wider range than the hardware accepts is a finding even when the run passes. :ivar severity: One of :data:`SEVERITIES`. :ivar kind: Machine-readable finding class, e.g. ``"range_wider_than_hardware"``. Lowercase, underscore-separated. :ivar subject: The parameter, operation or gate the finding is about. :ivar summary: One sentence, for a table row. :ivar detail: What was observed, in the module's own terms. :ivar action: What the developer should do. A finding without an action is an observation, and observations belong in the markdown report. .. py:attribute:: severity :type: str .. py:attribute:: kind :type: str .. py:attribute:: subject :type: str .. py:attribute:: summary :type: str .. py:attribute:: detail :type: str :value: '' .. py:attribute:: action :type: str :value: '' .. py:method:: to_dict() -> dict[str, Any] Return a JSON-serialisable mapping of the finding. .. py:class:: Instrument The hardware a field test ran against, as far as it may be published. The model is a property of the module — the repository name already implies it. The serial number and the address are not here, and there is no field they could reach. The transport is deliberately absent. No gate reads it, nothing branches on it, and the module's own repository already says which one it uses; it was a grouping hint for a comparison a reader can make without it, and it cost a naming scheme across five classes to produce. :ivar model: Instrument model, e.g. ``"PM100D"``. For a module driving several at once, the models joined by ``+``. :ivar sensor: The interchangeable part fitted, when the module has one — a sensor head, a stage, a camera. It changes the measured limits and timings, so it is what distinguishes two runs of one module. .. py:attribute:: model :type: str .. py:attribute:: sensor :type: str | None :value: None .. py:property:: slug :type: str Directory name for this instrument's reports. The fitted part when there is one, because that is what distinguishes two runs of the same module; the model otherwise. .. py:method:: to_dict() -> dict[str, Any] Return a JSON-serialisable mapping of the instrument. .. py:class:: ModuleInfo Identification of the module a report belongs to. :ivar name: Distribution name, e.g. ``"plesty-pm100d"``. :ivar version: Resolved version, including the versioningit suffix. :ivar module_type: One of ``device``, ``analyzer``, ``experiment``, ``default``. :ivar standard: Compliance standard the module is held to. :ivar repository: Repository path without the host, e.g. ``"plesty/hub/devices/thorlab/plesty-pm100d"``. ``None`` when the project has no remote — a local checkout publishes no path. .. py:attribute:: name :type: str .. py:attribute:: version :type: str .. py:attribute:: module_type :type: str .. py:attribute:: standard :type: str .. py:attribute:: repository :type: str | None :value: None .. py:method:: from_project(project_root: str | pathlib.Path = '.') -> ModuleInfo :classmethod: Read the module's identification from its own project. Every field comes from ``pyproject.toml``, the installed distribution metadata or the git remote — all of them already public. :param project_root: Directory holding ``pyproject.toml``. :returns: The module identification. :raises ReportSchemaError: If ``pyproject.toml`` is missing or names no project. A report cannot say which module it belongs to. .. py:method:: to_dict() -> dict[str, Any] Return a JSON-serialisable mapping of the module identification. .. py:class:: Coverage How much of what the module declares the run actually exercised. Keyed by what the module *declares*, not by what the run happened to call. A parameter nobody exercised appears with ``verified: "declared"`` and no calls; the alternative is a schema presented as though every entry were equally trustworthy, which is how a setting that crashes the controlling application stays undiscovered. :ivar parameters: Declared configuration keys, in schema order. :ivar operations: Declared operation names, in schema order. :ivar verified: Per name, one of :data:`VERIFICATION_STATES`. Names absent from the mapping are reported as ``declared``. .. py:attribute:: parameters :type: list[str] :value: [] .. py:attribute:: operations :type: list[str] :value: [] .. py:attribute:: verified :type: dict[str, str] .. py:method:: to_dict(timing_profile: dict[str, Any], failure_stats: dict[str, Any]) -> dict[str, Any] Build the published ``coverage`` block. :param timing_profile: Per command latency statistics, as ``{"query:wavelength": {...}}``. :param failure_stats: Per command attempts and failures, same keys. :returns: The ``coverage`` mapping, with one entry per declared parameter and operation. .. py:method:: _entry(label: str, name: str, commands: dict[str, str], timing_profile: dict[str, Any], failure_stats: dict[str, Any]) -> dict[str, Any] Build one coverage entry, folding its measurements in. :param label: Key naming the entry — ``"key"`` or ``"name"``. :param name: Parameter key or operation name. :param commands: Call role mapped to the report key it was recorded under. :param timing_profile: Per command latency statistics. :param failure_stats: Per command attempts and failures. .. py:function:: build_document(*, module: ModuleInfo, instrument: Instrument, tier: str = 'host', generated_utc: str, environment: dict[str, Any], outcome: dict[str, int | bool], gates: list[dict[str, Any]], coverage: dict[str, Any], lifecycle: list[dict[str, Any]], findings: list[Finding]) -> dict[str, Any] Assemble a ``field-test`` document. :param module: Module identification. :param instrument: The hardware the run was pointed at. :param tier: Which path the run exercised — "host" in-process, or "client" through the device server. A consumer comparing two runs of one module keys on this rather than on the file name. :param generated_utc: ISO-8601 UTC timestamp of the run. :param environment: Environment mapping; filtered to :data:`PUBLIC_ENVIRONMENT_KEYS`, so passing the full one is safe. :param outcome: ``passed`` / ``failed`` / ``skipped`` counts and ``ok``. :param gates: Per gate name, status, duration and reason. :param coverage: The mapping built by :meth:`Coverage.to_dict`. :param lifecycle: Connect, identity and disconnect timings. :param findings: What the run found wrong with the module. :returns: The document, ready for :func:`validate_document`. .. py:function:: validate_document(document: dict[str, Any]) -> None Check a document against the published artifact contract. Deliberately strict about *keys* as well as values: an unknown key is how a host name, a serial number or a file path would arrive in a published document, and every legitimate field is enumerated here. :param document: The document to check. :raises ReportSchemaError: On the first violation, naming the field. .. py:function:: write_document(reports_dir: str | pathlib.Path, document: dict[str, Any], instrument_slug: str, filename: str = 'field-test.json') -> pathlib.Path Validate a document and write it under ``//``. :param reports_dir: The module's ``reports/`` directory. :param document: The document to publish. :param instrument_slug: Directory name for the hardware this run used — :attr:`Instrument.slug`. :param filename: What to call it. The client tier publishes alongside the host tier rather than over it: the two answer different questions about the same instrument, and a run that overwrote the other would leave the module looking tested at one layer only. :returns: The path written. :raises ReportSchemaError: If the document or the slug fails validation. Nothing is written in that case. .. py:function:: _validate_coverage(coverage: Any) -> None Check the ``coverage`` block, which the docs generator walks by name. .. py:function:: _require_keys(value: Any, where: str, required: tuple[str, Ellipsis]) -> None Require *value* to be a mapping with exactly the *required* keys. :param value: The value to check. :param where: Path of the value, for the error message. :param required: The complete key set. :raises ReportSchemaError: If the value is not a mapping, or its keys differ. .. py:function:: _require_text(value: Any, where: str) -> None Require *value* to be a non-empty string. .. py:function:: _require_version(value: Any) -> None Require *value* to be a ``major.minor`` version this writer produces. .. py:function:: _call_record(stats: dict[str, Any] | None, latency: dict[str, Any] | None) -> dict[str, Any] Build one publishable call record from the run's own statistics. ``FailureStats.to_dict`` carries ``last_error``, the string form of the most recent exception. That is the one field in the whole report where a file path or an instrument address arrives as free text, so it is dropped here rather than filtered later — the counts say a command failed and how, which is what a published artifact is for; the message stays in the git-ignored markdown for whoever has to fix it. :param stats: The ``FailureStats.to_dict()`` form, or ``None``. :param latency: The ``LatencyStats.to_dict()`` form, or ``None``. :returns: Attempts, failures, failure rate, the per-class breakdown, and latency in milliseconds. .. py:function:: latency_ms(latency: dict[str, Any] | None) -> dict[str, float] Convert a :class:`~plesty.lib.test.field_test.LatencyStats` mapping to milliseconds. :param latency: The ``to_dict()`` form, whose values are seconds, or ``None`` when the command was never timed. :returns: ``min``/``median``/``p95``/``max`` in milliseconds; zeros when the command produced no successful call, which is itself the measurement. .. py:function:: _distribution_version(name: str, root: pathlib.Path) -> str Resolve the module's version, preferring the installed distribution. :param name: Distribution name from ``pyproject.toml``. :param root: Project root, used for the git fallback. :returns: The version string, or ``"unknown"`` when neither source answers. .. py:function:: _repository_path(root: pathlib.Path) -> str | None Return the origin remote's path without its host, or ``None``. The path is what a consumer links to and is public by construction; the host is dropped because a module's repository is identified by its path in every PLESTY surface that renders one. :param root: Project root. .. py:function:: _git(root: pathlib.Path, *args: str) -> str Run a read-only git command in *root*, returning its output or ``""``.