plesty.lib.data.experiment_data

The experimental-data contract (issue plesty-lib#51).

A measurement is more than one array: a spectrum is an intensity and the wavelength axis it was taken on, plus the exposure, the grating and the file it came from. ExperimentData is the contract every such type fulfils, so a device can hand one to an experiment, an experiment can persist it as one record, and an analyzer can declare it as one input.

A subclass declares a unique kind and implements three methods — to_arrays(), from_arrays() and example(). Everything else is inherited:

• save() / load() in every native format — .npz (lossless, the persistence format), .json, and the tables .csv / .txt.

• A registry keyed by kind: ExperimentData.load(path) rebuilds the subclass that wrote the file, and plesty.lib.data.types.resolve_dtype() resolves the kind as a schema dtype.

• Foreign formats a subclass can read (e.g. .spe for spectra) through its readers table.

from plesty.lib.data import ExperimentData, SpectrumData

spectrum.save(run_dir / "qd1.npz")
same = ExperimentData.load(run_dir / "qd1.npz")   # -> SpectrumData

Attributes

NATIVE_FORMATS
HEADER_KEY
TEXT_HEADER_PREFIX
_KIND_PATTERN
_REGISTRY
_COLUMN_KEYS

Classes

ExperimentData

Base contract of a typed piece of experimental data.

Functions

experiment_data_type(→ type[ExperimentData] | None)	Return the ExperimentData subclass registered under kind.
registered_kinds(→ tuple[str, Ellipsis])	Return every registered kind, sorted.
_check_columns(→ None)	Check a class's own columns declaration when the class is defined.
_normalize_suffix(→ str)	Return fmt as a lower-case suffix with a leading dot.
_format_cell(→ str)	Format one table cell losslessly: integers exactly, floats round-trippable.
_parse_columns(→ tuple[list[numpy.ndarray], ...)	Parse table lines into columns and the optional header row of names.
_split_row(→ list[str])	Split a table row on delimiter, or on whitespace when it is None.
_is_numeric_row(→ bool)	Return whether every cell of line parses as a number.

Module Contents

plesty.lib.data.experiment_data.NATIVE_FORMATS: tuple[str, Ellipsis] = ('.npz', '.json', '.csv', '.txt')

plesty.lib.data.experiment_data.HEADER_KEY = '__plesty__'

plesty.lib.data.experiment_data.TEXT_HEADER_PREFIX = '# plesty: '

plesty.lib.data.experiment_data._KIND_PATTERN

plesty.lib.data.experiment_data._REGISTRY: dict[str, type[ExperimentData]]

plesty.lib.data.experiment_data.experiment_data_type(kind: str) → type[ExperimentData] | None

Return the ExperimentData subclass registered under kind.

Parameters:

kind (str) – The kind a subclass declares, e.g. "spectrum".

Returns:

The registered subclass, or None when no subclass declares it.

Return type:

type[ExperimentData] | None

plesty.lib.data.experiment_data.registered_kinds() → tuple[str, Ellipsis]

Return every registered kind, sorted.

Return type:

tuple[str, Ellipsis]

plesty.lib.data.experiment_data._COLUMN_KEYS

plesty.lib.data.experiment_data._check_columns(cls: type) → None

Check a class’s own columns declaration when the class is defined.

Raises:

TypeError – If an entry is not a dict, has an unknown key, a non-boolean result, or an error naming itself.

Parameters:

cls (type)

Return type:

None

class plesty.lib.data.experiment_data.ExperimentData(metadata: dict[str, Any] | None = None, provenance: dict[str, Any] | None = None)

Bases: abc.ABC

Base contract of a typed piece of experimental data.

Class attributes a subclass sets:

• kind — mandatory, unique, lower-case identifier; the registry key, the schema dtype token, and what a saved file names.

• schema_version — bumped when the arrays or their meaning change; a file written by a newer version is refused, not misread.

• readers — suffix → callable reading a foreign file format.

• columns — what each array means to a reader, and which are the result; a report tabulates the data from it (plesty.lib.report.data_table()).

Instance attributes every piece of data carries:

• metadata — acquisition context (exposure, grating, source file, processing steps, …); JSON-serializable.

• provenance — who produced it (device, analyzer, parameters); JSON-serializable.

A subclass that does not set kind itself is not registered, so an intermediate base class can share code without claiming a kind.

Store the metadata and provenance, copied so callers can reuse theirs.

Parameters:

• metadata (dict[str, Any] | None) – Acquisition context.

• provenance (dict[str, Any] | None) – Origin of the data.

kind: ClassVar[str]

Registry key and schema dtype token, e.g. "spectrum".

schema_version: ClassVar[int] = 1

Version of the array layout this class writes and reads.

readers: ClassVar[dict[str, Callable[[pathlib.Path], Any]]]

lower-case suffix → reader(path).

Type:

Foreign formats this class reads

columns: ClassVar[dict[str, dict[str, Any]]]

array name → entry, in reading order.

An entry’s keys, all optional:

• label — the column heading, inline markup and $math$ allowed; the array name when left out.

• unit — used when the array carries none of its own.

• description — one line of meaning.

• result — False for an array the analysis kept but a reader does not need (a fit quality, a point count); default True.

• error — name of the array holding this one’s one-sigma uncertainty, so the two are shown as one value ± error.

Arrays left out are not tabulated.

Type:

What each array means to a reader

metadata: dict[str, Any]

Acquisition context; JSON-serializable.

provenance: dict[str, Any]

Origin of the data (device, analyzer, parameters); JSON-serializable.

classmethod __init_subclass__(**kwargs: Any) → None

Register a subclass that declares its own kind.

Raises:

TypeError – If the kind is malformed or already taken by a different class.

Parameters:

kwargs (Any)

Return type:

None

abstractmethod to_arrays() → dict[str, plesty.lib.data.array.PlestyArray]

Return the named arrays that fully describe this data.

The names are the keys from_arrays() receives back; each array keeps its Plesty metadata (name, unit, description, …).

Return type:

dict[str, plesty.lib.data.array.PlestyArray]

classmethod from_arrays(arrays: dict[str, plesty.lib.data.array.PlestyArray], metadata: dict[str, Any], provenance: dict[str, Any]) → Self

Abstractmethod:

Parameters:

• arrays (dict[str, plesty.lib.data.array.PlestyArray])

• metadata (dict[str, Any])

• provenance (dict[str, Any])

Return type:

Self

Rebuild an instance from the output of to_arrays().

Parameters:

• arrays (dict[str, plesty.lib.data.array.PlestyArray]) – Name → array, as written by to_arrays().

• metadata (dict[str, Any]) – The saved metadata.

• provenance (dict[str, Any]) – The saved provenance.

Return type:

Self

classmethod example() → Self

Abstractmethod:

Return type:

Self

Return a small valid instance — for docs, mocks and pipeline tests.

header() → dict[str, Any]

Return the JSON header describing this data without its values.

Returns:

kind, schema_version, metadata, provenance and, per array, its Plesty metadata plus shape and dtype.

Return type:

dict[str, Any]

_checked_arrays() → dict[str, plesty.lib.data.array.PlestyArray]

Return to_arrays(), refusing names that collide with the header.

Return type:

dict[str, plesty.lib.data.array.PlestyArray]

save(path: str | pathlib.Path) → pathlib.Path

Write this data to path, in the format its suffix names.

The write is atomic: a temporary sibling is written and renamed, so a crash never leaves a half-written file under the final name.

Parameters:

path (str | pathlib.Path) – Destination; the suffix picks the format, one of NATIVE_FORMATS.

Returns:

The written path.

Raises:

ValueError – If the suffix is not a native format, or the arrays cannot be laid out as a table (.csv / .txt).

Return type:

pathlib.Path

to_bytes(fmt: str) → bytes

Serialize this data in a native format.

Parameters:

fmt (str) – A suffix from NATIVE_FORMATS (the dot is optional).

Returns:

The encoded file content.

Raises:

ValueError – If fmt is not native, or the arrays cannot be laid out as a table.

Return type:

bytes

_to_npz() → bytes

Encode as .npz: one entry per array plus the JSON header.

Return type:

bytes

_to_json() → bytes

Encode as .json: the header with each array’s values inlined.

Return type:

bytes

_to_table(delimiter: str) → bytes

Encode as a delimited table: one column per 1-D array or per 2-D row.

Parameters:

delimiter (str)

Return type:

bytes

classmethod load(path: str | pathlib.Path) → Self

Read data from path — a native file or a format a reader handles.

Called on ExperimentData itself, the class is taken from the file’s header. Called on a subclass, the file must hold that subclass (or one derived from it); a plain table without a Plesty header is then handed to the subclass’s from_table().

Parameters:

path (str | pathlib.Path) – The file to read.

Returns:

The reconstructed data.

Raises:

• ValueError – If the format is unknown, the kind is not registered, or the file was written by a newer schema version.

• TypeError – If the file holds a different kind than requested.

Return type:

Self

classmethod _reader_for(suffix: str) → Callable[[pathlib.Path], Any] | None

Find a reader for suffix on this class, or on any registered one.

Parameters:

suffix (str)

Return type:

Callable[[pathlib.Path], Any] | None

classmethod from_bytes(payload: bytes, fmt: str) → Self

Deserialize data written by to_bytes().

Parameters:

• payload (bytes) – The encoded file content.

• fmt (str) – The native format it is in.

Returns:

The reconstructed data.

Raises:

• ValueError – As for load().

• TypeError – As for load().

Return type:

Self

classmethod _from_table_bytes(payload: bytes, delimiter: str | None) → Self

Parse a delimited table, with or without a Plesty header line.

Parameters:

• payload (bytes)

• delimiter (str | None)

Return type:

Self

classmethod from_text(lines: list[str], delimiter: str | None) → Self

Build an instance from a text file without a Plesty header.

The default parses one delimited table — an optional row of column names, then numbers — and hands it to from_table(). Subclasses that read a text layout other than one table override this.

Parameters:

• lines (list[str]) – The file’s non-empty lines, # comments removed.

• delimiter (str | None) – "," for .csv; None (whitespace) for .txt.

Return type:

Self

classmethod from_table(columns: list[numpy.ndarray], names: list[str] | None) → Self

Build an instance from a plain table without a Plesty header.

The base class cannot interpret foreign columns; subclasses that know a common layout (e.g. wavelength + intensity) override this.

Parameters:

• columns (list[numpy.ndarray]) – The table’s columns, left to right.

• names (list[str] | None) – The column header row, when the file had one.

Raises:

ValueError – Always, for types that do not define a layout.

Return type:

Self

classmethod _rebuild(header: dict[str, Any], raw: dict[str, numpy.ndarray]) → Self

Resolve the header’s kind, check its version, and call from_arrays().

Parameters:

• header (dict[str, Any])

• raw (dict[str, numpy.ndarray])

Return type:

Self

__repr__() → str

Return the kind and the shape of every array.

Return type:

str

plesty.lib.data.experiment_data._normalize_suffix(fmt: str) → str

Return fmt as a lower-case suffix with a leading dot.

Parameters:

fmt (str)

Return type:

str

plesty.lib.data.experiment_data._format_cell(value: Any) → str

Format one table cell losslessly: integers exactly, floats round-trippable.

Parameters:

value (Any)

Return type:

str

plesty.lib.data.experiment_data._parse_columns(lines: list[str], delimiter: str | None) → tuple[list[numpy.ndarray], list[str] | None]

Parse table lines into columns and the optional header row of names.

Parameters:

• lines (list[str])

• delimiter (str | None)

Return type:

tuple[list[numpy.ndarray], list[str] | None]

plesty.lib.data.experiment_data._split_row(line: str, delimiter: str | None) → list[str]

Split a table row on delimiter, or on whitespace when it is None.

Parameters:

• line (str)

• delimiter (str | None)

Return type:

list[str]

plesty.lib.data.experiment_data._is_numeric_row(line: str, delimiter: str | None) → bool

Return whether every cell of line parses as a number.

Parameters:

• line (str)

• delimiter (str | None)

Return type:

bool