plesty.lib.data.io

Persistence helpers for Plesty data (issue plesty-lib#4).

Measured data must survive process crashes so experiments can resume from a checkpoint. The storage contract keeps blob data in its original raw format and describes it in a JSON metadata document that carries a reference link to the blob:

PlestyArray — raw .npy blob; the JSON document stores the Plesty metadata (name, range, options, unit, description) plus shape/dtype and the blob reference.
bytes (e.g. an image already encoded by a camera) — written untouched; common image formats (PNG, JPEG, TIFF) are recognised so the blob gets its native suffix.
JSON-serializable values (dicts, lists, scalars) — inlined in the JSON document itself; no blob is written.

Writes are crash-safe: the blob is written first (atomically, via a temporary sibling file and os.replace()), then the JSON document. The JSON document is the commit record — a result exists once its document does.

Two layouts carry the documents. save_result() writes one .json file per result — the right shape for standalone results. An experiment run writes all of its step documents to one append-only records.jsonl (append_record() / read_records()): opening, syncing, and later listing thousands of small files on a network share is what made a run slow to write and slow to follow, whereas one line appended and fsynced costs one write and is read back by offset. Blobs still go to data/ beside it.

Attributes

`RESULT_TYPES`
`_IMAGE_MAGIC`
`RECORDS_FILE`
`BLOB_SUBDIR`

Classes

ResultDocument

Typed model of the JSON metadata document written by save_result().

Functions

`_atomic_write_bytes`(→ None)	Write payload to path atomically via a temporary sibling file.
`_sniff_suffix`(→ str)	Return the native file suffix for an encoded blob (default `bin`).
`_persist`(→ ResultDocument)	Write result's blob (if it has one) next to stem; return its document.
`save_result`(→ pathlib.Path)	Persist a measurement result as raw blob + JSON metadata document.
`append_record`(→ dict[str, Any])	Append one step result to the run's `records.jsonl`.
`read_records`(→ tuple[list[dict[str, Any]], int])	Read the record lines appended to `records.jsonl` since offset.
`record_value`(→ Any)	Return the value a `records.jsonl` line stands for.
`load_document`(→ ResultDocument)	Load only the metadata document of a persisted result — no blob I/O.
`load_result`(→ Any)	Load a result previously written by `save_result()`.
`convert_to_hdf5`(→ pathlib.Path)	Pack results written by `save_result()` into one HDF5 archive.

Module Contents

plesty.lib.data.io.RESULT_TYPES: tuple[str, Ellipsis] = ('array', 'bytes', 'value')

class plesty.lib.data.io.ResultDocument

Typed model of the JSON metadata document written by save_result().

Formalises the experimental-output metadata schema (issue plesty-lib#4): every persisted result is described by one such document, which is the commit record of the write. Blob-backed results (array, bytes) reference their raw data file via blob; plain values are inlined in value.

Variables:

type – Result category — one of RESULT_TYPES.
saved_at – UTC ISO-8601 write time.
provenance – Context recorded at save time (step id, operation, parameters, device identity, …).
blob – File name of the raw data blob next to the document (array/bytes results only).
format – Blob encoding ("npy", "png", "bin", …).
shape – Array shape (array results only).
dtype – Array dtype string (array results only).
meta – Plesty metadata of the array (name, range, options, unit, description; array results only).
value – The inlined JSON-serializable value (value results only).

type: str

saved_at: str = ''

provenance: dict[str, Any]

blob: str | None = None

format: str | None = None

shape: list[int] | None = None

dtype: str | None = None

meta: dict[str, Any] | None = None

value: Any = None

__post_init__() → None

Validate the result type and stamp the write time when missing.

Return type:: None

to_dict() → dict[str, Any]

Return the document as a JSON-serializable dictionary.

Blob-related fields that do not apply to the result type are omitted, matching the on-disk layout written since the schema’s introduction.

Return type:: dict[str, Any]

classmethod from_dict(document: dict[str, Any]) → ResultDocument

Reconstruct a document from a dictionary produced by to_dict().

Parameters:: document (dict[str, Any]) – The parsed JSON document.
Returns:: The reconstructed ResultDocument.
Raises:: ValueError – If the document declares an unknown result type.
Return type:: ResultDocument

plesty.lib.data.io._IMAGE_MAGIC: dict[bytes, str]

plesty.lib.data.io._atomic_write_bytes(path: pathlib.Path, payload: bytes) → None

Write payload to path atomically via a temporary sibling file.

Parameters:

path (pathlib.Path)
payload (bytes)

Return type:

None

plesty.lib.data.io._sniff_suffix(payload: bytes) → str

Return the native file suffix for an encoded blob (default bin).

Parameters:: payload (bytes)
Return type:: str

plesty.lib.data.io.RECORDS_FILE = 'records.jsonl'

plesty.lib.data.io.BLOB_SUBDIR = 'data'

plesty.lib.data.io._persist(result: Any, stem: pathlib.Path, provenance: dict[str, Any] | None) → ResultDocument

Write result’s blob (if it has one) next to stem; return its document.

The document’s blob is the bare file name; callers writing the document elsewhere than beside the blob re-base it.

Parameters:

result (Any)
stem (pathlib.Path)
provenance (Optional[dict[str, Any]])

Return type:

ResultDocument

plesty.lib.data.io.save_result(result: Any, path_stem: str | pathlib.Path, provenance: dict[str, Any] | None = None) → pathlib.Path

Persist a measurement result as raw blob + JSON metadata document.

Parameters:

result (Any) – The value to persist — a PlestyArray, encoded bytes (e.g. an image), or any JSON-serializable value.
path_stem (str | pathlib.Path) – Destination path without suffix; the JSON document is written to <stem>.json and any blob next to it.
provenance (Optional[dict[str, Any]]) – Optional context (step id, operation, parameters, device identity) recorded verbatim in the document.

Returns:

The path of the JSON metadata document.

Raises:

TypeError – If the result is neither a PlestyArray, bytes, nor JSON-serializable.

Return type:

pathlib.Path

plesty.lib.data.io.append_record(run_dir: str | pathlib.Path, index: int, result: Any, provenance: dict[str, Any] | None = None) → dict[str, Any]

Append one step result to the run’s records.jsonl.

A blob-backed result (array, bytes) is written to <run_dir>/data/step_<index>.<fmt> first, atomically; the document line — the commit record — is appended and fsynced afterwards, so a reader never sees a document whose blob is missing.

Parameters:

run_dir (str | pathlib.Path) – The run directory.
index (int) – Position of the step in the plan; stamped onto the line and used for the blob file name.
result (Any) – The value to persist (see save_result()).
provenance (Optional[dict[str, Any]]) – Context recorded verbatim in the document.

Returns:

The line that was written, as a dictionary — the document fields plus index; blob (when present) is relative to run_dir.

Raises:

TypeError – If the result is neither a PlestyArray, bytes, nor JSON-serializable.

Return type:

dict[str, Any]

plesty.lib.data.io.read_records(run_dir: str | pathlib.Path, offset: int = 0) → tuple[list[dict[str, Any]], int]

Read the record lines appended to records.jsonl since offset.

Only complete lines are returned; a line still being written (no trailing newline yet) is left for the next call, which is what makes tailing a run in progress safe. Lines that fail to parse are skipped.

Parameters:

run_dir (str | pathlib.Path) – The run directory.
offset (int) – Byte position to read from — pass back the returned offset to read only what appeared since.

Returns:

The parsed lines (see append_record()) and the offset just after the last complete line; ([], offset) while the file does not exist yet.

Return type:

tuple[list[dict[str, Any]], int]

plesty.lib.data.io.record_value(line: dict[str, Any], run_dir: str | pathlib.Path) → Any

Return the value a records.jsonl line stands for.

Parameters:

line (dict[str, Any]) – A line as returned by read_records().
run_dir (str | pathlib.Path) – The run directory the line’s blob is relative to.

Returns:

The inlined value, or the PlestyArray / raw bytes loaded from the referenced blob.

Raises:

ValueError – If the line declares an unknown result type.

Return type:

Any

plesty.lib.data.io.load_document(path: str | pathlib.Path) → ResultDocument

Load only the metadata document of a persisted result — no blob I/O.

Useful for browsing run directories (checking provenance, shapes, or timestamps) without paying the cost of decoding the referenced blobs.

Parameters:: path (str | pathlib.Path) – Path to the JSON metadata document (<stem>.json).
Returns:: The parsed ResultDocument.
Raises:: ValueError – If the document declares an unknown result type.
Return type:: ResultDocument

plesty.lib.data.io.load_result(path: str | pathlib.Path) → Any

Load a result previously written by save_result().

Parameters:: path (str | pathlib.Path) – Path to the JSON metadata document (<stem>.json).
Returns:: a PlestyArray with its metadata restored, raw bytes for encoded blobs, or the inlined value.
Return type:: The reconstructed value
Raises:: ValueError – If the document declares an unknown result type.

plesty.lib.data.io.convert_to_hdf5(documents: Iterable[str | pathlib.Path], target: str | pathlib.Path) → pathlib.Path

Pack results written by save_result() into one HDF5 archive.

The raw blob + JSON layout stays the primary on-disk format; this is an optional post-hoc export to the unified HDF5 format preferred by issue plesty-lib#4. Each JSON document becomes one HDF5 group (named after the document stem) holding a data dataset — the decoded array, the raw encoded bytes, or the JSON-encoded inline value — with the document’s metadata and provenance stored as group attributes.

Parameters:

documents (Iterable[str | pathlib.Path]) – Paths of the JSON metadata documents to include.
target (str | pathlib.Path) – Destination .h5 file; parent directories are created.

Returns:

The resolved path of the written HDF5 file.

Raises:

ImportError – If the hdf5 extra is not installed.
ValueError – If a document declares an unknown result type.

Return type:

pathlib.Path