plesty.lib.data.experiment_data =============================== .. py:module:: plesty.lib.data.experiment_data .. autoapi-nested-parse:: The experimental-data contract (issue plesty-lib#51). A measurement is more than one array: a spectrum is an intensity *and* the wavelength axis it was taken on, plus the exposure, the grating and the file it came from. :class:`ExperimentData` is the contract every such type fulfils, so a device can hand one to an experiment, an experiment can persist it as one record, and an analyzer can declare it as one input. A subclass declares a unique :attr:`~ExperimentData.kind` and implements three methods — :meth:`~ExperimentData.to_arrays`, :meth:`~ExperimentData.from_arrays` and :meth:`~ExperimentData.example`. Everything else is inherited: * :meth:`~ExperimentData.save` / :meth:`~ExperimentData.load` in every native format — ``.npz`` (lossless, the persistence format), ``.json``, and the tables ``.csv`` / ``.txt``. * A registry keyed by ``kind``: ``ExperimentData.load(path)`` rebuilds the subclass that wrote the file, and :func:`plesty.lib.data.types.resolve_dtype` resolves the ``kind`` as a schema dtype. * Foreign formats a subclass can read (e.g. ``.spe`` for spectra) through its :attr:`~ExperimentData.readers` table. .. code-block:: python from plesty.lib.data import ExperimentData, SpectrumData spectrum.save(run_dir / "qd1.npz") same = ExperimentData.load(run_dir / "qd1.npz") # -> SpectrumData Attributes ---------- .. autoapisummary:: plesty.lib.data.experiment_data.NATIVE_FORMATS plesty.lib.data.experiment_data.HEADER_KEY plesty.lib.data.experiment_data.TEXT_HEADER_PREFIX plesty.lib.data.experiment_data._KIND_PATTERN plesty.lib.data.experiment_data._REGISTRY plesty.lib.data.experiment_data._COLUMN_KEYS Classes ------- .. autoapisummary:: plesty.lib.data.experiment_data.ExperimentData Functions --------- .. autoapisummary:: plesty.lib.data.experiment_data.experiment_data_type plesty.lib.data.experiment_data.registered_kinds plesty.lib.data.experiment_data._check_columns plesty.lib.data.experiment_data._normalize_suffix plesty.lib.data.experiment_data._format_cell plesty.lib.data.experiment_data._parse_columns plesty.lib.data.experiment_data._split_row plesty.lib.data.experiment_data._is_numeric_row Module Contents --------------- .. py:data:: NATIVE_FORMATS :type: tuple[str, Ellipsis] :value: ('.npz', '.json', '.csv', '.txt') .. py:data:: HEADER_KEY :value: '__plesty__' .. py:data:: TEXT_HEADER_PREFIX :value: '# plesty: ' .. py:data:: _KIND_PATTERN .. py:data:: _REGISTRY :type: dict[str, type[ExperimentData]] .. py:function:: experiment_data_type(kind: str) -> type[ExperimentData] | None Return the :class:`ExperimentData` subclass registered under *kind*. :param kind: The ``kind`` a subclass declares, e.g. ``"spectrum"``. :returns: The registered subclass, or ``None`` when no subclass declares it. .. py:function:: registered_kinds() -> tuple[str, Ellipsis] Return every registered ``kind``, sorted. .. py:data:: _COLUMN_KEYS .. py:function:: _check_columns(cls: type) -> None Check a class's own ``columns`` declaration when the class is defined. :raises TypeError: If an entry is not a dict, has an unknown key, a non-boolean ``result``, or an ``error`` naming itself. .. py:class:: ExperimentData(metadata: dict[str, Any] | None = None, provenance: dict[str, Any] | None = None) Bases: :py:obj:`abc.ABC` Base contract of a typed piece of experimental data. Class attributes a subclass sets: * :attr:`kind` — **mandatory**, unique, lower-case identifier; the registry key, the schema dtype token, and what a saved file names. * :attr:`schema_version` — bumped when the arrays or their meaning change; a file written by a newer version is refused, not misread. * :attr:`readers` — suffix → callable reading a foreign file format. * :attr:`columns` — what each array means to a reader, and which are the result; a report tabulates the data from it (:func:`plesty.lib.report.data_table`). Instance attributes every piece of data carries: * :attr:`metadata` — acquisition context (exposure, grating, source file, processing steps, …); JSON-serializable. * :attr:`provenance` — who produced it (device, analyzer, parameters); JSON-serializable. A subclass that does not set ``kind`` itself is not registered, so an intermediate base class can share code without claiming a kind. Store the metadata and provenance, copied so callers can reuse theirs. :param metadata: Acquisition context. :param provenance: Origin of the data. .. py:attribute:: kind :type: ClassVar[str] Registry key and schema dtype token, e.g. ``"spectrum"``. .. py:attribute:: schema_version :type: ClassVar[int] :value: 1 Version of the array layout this class writes and reads. .. py:attribute:: readers :type: ClassVar[dict[str, Callable[[pathlib.Path], Any]]] lower-case suffix → ``reader(path)``. :type: Foreign formats this class reads .. py:attribute:: columns :type: ClassVar[dict[str, dict[str, Any]]] array name → entry, in reading order. An entry's keys, all optional: * ``label`` — the column heading, inline markup and ``$math$`` allowed; the array name when left out. * ``unit`` — used when the array carries none of its own. * ``description`` — one line of meaning. * ``result`` — ``False`` for an array the analysis kept but a reader does not need (a fit quality, a point count); default ``True``. * ``error`` — name of the array holding this one's one-sigma uncertainty, so the two are shown as one ``value ± error``. Arrays left out are not tabulated. :type: What each array means to a reader .. py:attribute:: metadata :type: dict[str, Any] Acquisition context; JSON-serializable. .. py:attribute:: provenance :type: dict[str, Any] Origin of the data (device, analyzer, parameters); JSON-serializable. .. py:method:: __init_subclass__(**kwargs: Any) -> None :classmethod: Register a subclass that declares its own ``kind``. :raises TypeError: If the ``kind`` is malformed or already taken by a different class. .. py:method:: to_arrays() -> dict[str, plesty.lib.data.array.PlestyArray] :abstractmethod: Return the named arrays that fully describe this data. The names are the keys :meth:`from_arrays` receives back; each array keeps its Plesty metadata (name, unit, description, …). .. py:method:: from_arrays(arrays: dict[str, plesty.lib.data.array.PlestyArray], metadata: dict[str, Any], provenance: dict[str, Any]) -> Self :classmethod: :abstractmethod: Rebuild an instance from the output of :meth:`to_arrays`. :param arrays: Name → array, as written by :meth:`to_arrays`. :param metadata: The saved :attr:`metadata`. :param provenance: The saved :attr:`provenance`. .. py:method:: example() -> Self :classmethod: :abstractmethod: Return a small valid instance — for docs, mocks and pipeline tests. .. py:method:: header() -> dict[str, Any] Return the JSON header describing this data without its values. :returns: ``kind``, ``schema_version``, ``metadata``, ``provenance`` and, per array, its Plesty metadata plus ``shape`` and ``dtype``. .. py:method:: _checked_arrays() -> dict[str, plesty.lib.data.array.PlestyArray] Return :meth:`to_arrays`, refusing names that collide with the header. .. py:method:: save(path: str | pathlib.Path) -> pathlib.Path Write this data to *path*, in the format its suffix names. The write is atomic: a temporary sibling is written and renamed, so a crash never leaves a half-written file under the final name. :param path: Destination; the suffix picks the format, one of :data:`NATIVE_FORMATS`. :returns: The written path. :raises ValueError: If the suffix is not a native format, or the arrays cannot be laid out as a table (``.csv`` / ``.txt``). .. py:method:: to_bytes(fmt: str) -> bytes Serialize this data in a native format. :param fmt: A suffix from :data:`NATIVE_FORMATS` (the dot is optional). :returns: The encoded file content. :raises ValueError: If *fmt* is not native, or the arrays cannot be laid out as a table. .. py:method:: _to_npz() -> bytes Encode as ``.npz``: one entry per array plus the JSON header. .. py:method:: _to_json() -> bytes Encode as ``.json``: the header with each array's values inlined. .. py:method:: _to_table(delimiter: str) -> bytes Encode as a delimited table: one column per 1-D array or per 2-D row. .. py:method:: load(path: str | pathlib.Path) -> Self :classmethod: Read data from *path* — a native file or a format a reader handles. Called on :class:`ExperimentData` itself, the class is taken from the file's header. Called on a subclass, the file must hold that subclass (or one derived from it); a plain table without a Plesty header is then handed to the subclass's :meth:`from_table`. :param path: The file to read. :returns: The reconstructed data. :raises ValueError: If the format is unknown, the ``kind`` is not registered, or the file was written by a newer schema version. :raises TypeError: If the file holds a different kind than requested. .. py:method:: _reader_for(suffix: str) -> Callable[[pathlib.Path], Any] | None :classmethod: Find a reader for *suffix* on this class, or on any registered one. .. py:method:: from_bytes(payload: bytes, fmt: str) -> Self :classmethod: Deserialize data written by :meth:`to_bytes`. :param payload: The encoded file content. :param fmt: The native format it is in. :returns: The reconstructed data. :raises ValueError: As for :meth:`load`. :raises TypeError: As for :meth:`load`. .. py:method:: _from_table_bytes(payload: bytes, delimiter: str | None) -> Self :classmethod: Parse a delimited table, with or without a Plesty header line. .. py:method:: from_text(lines: list[str], delimiter: str | None) -> Self :classmethod: Build an instance from a text file without a Plesty header. The default parses one delimited table — an optional row of column names, then numbers — and hands it to :meth:`from_table`. Subclasses that read a text layout other than one table override this. :param lines: The file's non-empty lines, ``#`` comments removed. :param delimiter: ``","`` for ``.csv``; ``None`` (whitespace) for ``.txt``. .. py:method:: from_table(columns: list[numpy.ndarray], names: list[str] | None) -> Self :classmethod: Build an instance from a plain table without a Plesty header. The base class cannot interpret foreign columns; subclasses that know a common layout (e.g. wavelength + intensity) override this. :param columns: The table's columns, left to right. :param names: The column header row, when the file had one. :raises ValueError: Always, for types that do not define a layout. .. py:method:: _rebuild(header: dict[str, Any], raw: dict[str, numpy.ndarray]) -> Self :classmethod: Resolve the header's kind, check its version, and call :meth:`from_arrays`. .. py:method:: __repr__() -> str Return the kind and the shape of every array. .. py:function:: _normalize_suffix(fmt: str) -> str Return *fmt* as a lower-case suffix with a leading dot. .. py:function:: _format_cell(value: Any) -> str Format one table cell losslessly: integers exactly, floats round-trippable. .. py:function:: _parse_columns(lines: list[str], delimiter: str | None) -> tuple[list[numpy.ndarray], list[str] | None] Parse table lines into columns and the optional header row of names. .. py:function:: _split_row(line: str, delimiter: str | None) -> list[str] Split a table row on *delimiter*, or on whitespace when it is ``None``. .. py:function:: _is_numeric_row(line: str, delimiter: str | None) -> bool Return whether every cell of *line* parses as a number.