plesty.lib.data.io ================== .. py:module:: plesty.lib.data.io .. autoapi-nested-parse:: Persistence helpers for Plesty data (issue plesty-lib#4). Measured data must survive process crashes so experiments can resume from a checkpoint. The storage contract keeps blob data in its **original raw format** and describes it in a JSON metadata document that carries a reference link to the blob: * :class:`~plesty.lib.data.array.PlestyArray` — raw ``.npy`` blob; the JSON document stores the Plesty metadata (name, range, options, unit, description) plus shape/dtype and the blob reference. * ``bytes`` (e.g. an image already encoded by a camera) — written untouched; common image formats (PNG, JPEG, TIFF) are recognised so the blob gets its native suffix. * JSON-serializable values (dicts, lists, scalars) — inlined in the JSON document itself; no blob is written. Writes are crash-safe: the blob is written first (atomically, via a temporary sibling file and :func:`os.replace`), then the JSON document. The JSON document is the commit record — a result exists once its document does. Two layouts carry the documents. :func:`save_result` writes one ``.json`` file per result — the right shape for standalone results. An experiment run writes all of its step documents to **one** append-only ``records.jsonl`` (:func:`append_record` / :func:`read_records`): opening, syncing, and later listing thousands of small files on a network share is what made a run slow to write and slow to follow, whereas one line appended and fsynced costs one write and is read back by offset. Blobs still go to ``data/`` beside it. Attributes ---------- .. autoapisummary:: plesty.lib.data.io.RESULT_TYPES plesty.lib.data.io._IMAGE_MAGIC plesty.lib.data.io.RECORDS_FILE plesty.lib.data.io.BLOB_SUBDIR Classes ------- .. autoapisummary:: plesty.lib.data.io.ResultDocument Functions --------- .. autoapisummary:: plesty.lib.data.io._atomic_write_bytes plesty.lib.data.io._sniff_suffix plesty.lib.data.io._persist plesty.lib.data.io.save_result plesty.lib.data.io.append_record plesty.lib.data.io.read_records plesty.lib.data.io.record_value plesty.lib.data.io.load_document plesty.lib.data.io.load_result plesty.lib.data.io.convert_to_hdf5 Module Contents --------------- .. py:data:: RESULT_TYPES :type: tuple[str, Ellipsis] :value: ('array', 'bytes', 'value') .. py:class:: ResultDocument Typed model of the JSON metadata document written by :func:`save_result`. Formalises the experimental-output metadata schema (issue plesty-lib#4): every persisted result is described by one such document, which is the commit record of the write. Blob-backed results (``array``, ``bytes``) reference their raw data file via :attr:`blob`; plain values are inlined in :attr:`value`. :ivar type: Result category — one of :data:`RESULT_TYPES`. :ivar saved_at: UTC ISO-8601 write time. :ivar provenance: Context recorded at save time (step id, operation, parameters, device identity, …). :ivar blob: File name of the raw data blob next to the document (``array``/``bytes`` results only). :ivar format: Blob encoding (``"npy"``, ``"png"``, ``"bin"``, …). :ivar shape: Array shape (``array`` results only). :ivar dtype: Array dtype string (``array`` results only). :ivar meta: Plesty metadata of the array (name, range, options, unit, description; ``array`` results only). :ivar value: The inlined JSON-serializable value (``value`` results only). .. py:attribute:: type :type: str .. py:attribute:: saved_at :type: str :value: '' .. py:attribute:: provenance :type: dict[str, Any] .. py:attribute:: blob :type: str | None :value: None .. py:attribute:: format :type: str | None :value: None .. py:attribute:: shape :type: list[int] | None :value: None .. py:attribute:: dtype :type: str | None :value: None .. py:attribute:: meta :type: dict[str, Any] | None :value: None .. py:attribute:: value :type: Any :value: None .. py:method:: __post_init__() -> None Validate the result type and stamp the write time when missing. .. py:method:: to_dict() -> dict[str, Any] Return the document as a JSON-serializable dictionary. Blob-related fields that do not apply to the result type are omitted, matching the on-disk layout written since the schema's introduction. .. py:method:: from_dict(document: dict[str, Any]) -> ResultDocument :classmethod: Reconstruct a document from a dictionary produced by :meth:`to_dict`. :param document: The parsed JSON document. :returns: The reconstructed :class:`ResultDocument`. :raises ValueError: If the document declares an unknown result type. .. py:data:: _IMAGE_MAGIC :type: dict[bytes, str] .. py:function:: _atomic_write_bytes(path: pathlib.Path, payload: bytes) -> None Write *payload* to *path* atomically via a temporary sibling file. .. py:function:: _sniff_suffix(payload: bytes) -> str Return the native file suffix for an encoded blob (default ``bin``). .. py:data:: RECORDS_FILE :value: 'records.jsonl' .. py:data:: BLOB_SUBDIR :value: 'data' .. py:function:: _persist(result: Any, stem: pathlib.Path, provenance: Optional[dict[str, Any]]) -> ResultDocument Write *result*'s blob (if it has one) next to *stem*; return its document. The document's ``blob`` is the bare file name; callers writing the document elsewhere than beside the blob re-base it. .. py:function:: save_result(result: Any, path_stem: str | pathlib.Path, provenance: Optional[dict[str, Any]] = None) -> pathlib.Path Persist a measurement result as raw blob + JSON metadata document. :param result: The value to persist — a :class:`PlestyArray`, encoded ``bytes`` (e.g. an image), or any JSON-serializable value. :param path_stem: Destination path without suffix; the JSON document is written to ``.json`` and any blob next to it. :param provenance: Optional context (step id, operation, parameters, device identity) recorded verbatim in the document. :returns: The path of the JSON metadata document. :raises TypeError: If the result is neither a PlestyArray, bytes, nor JSON-serializable. .. py:function:: append_record(run_dir: str | pathlib.Path, index: int, result: Any, provenance: Optional[dict[str, Any]] = None) -> dict[str, Any] Append one step result to the run's ``records.jsonl``. A blob-backed result (array, bytes) is written to ``/data/step_.`` first, atomically; the document line — the commit record — is appended and fsynced afterwards, so a reader never sees a document whose blob is missing. :param run_dir: The run directory. :param index: Position of the step in the plan; stamped onto the line and used for the blob file name. :param result: The value to persist (see :func:`save_result`). :param provenance: Context recorded verbatim in the document. :returns: The line that was written, as a dictionary — the document fields plus ``index``; ``blob`` (when present) is relative to *run_dir*. :raises TypeError: If the result is neither a PlestyArray, bytes, nor JSON-serializable. .. py:function:: read_records(run_dir: str | pathlib.Path, offset: int = 0) -> tuple[list[dict[str, Any]], int] Read the record lines appended to ``records.jsonl`` since *offset*. Only complete lines are returned; a line still being written (no trailing newline yet) is left for the next call, which is what makes tailing a run in progress safe. Lines that fail to parse are skipped. :param run_dir: The run directory. :param offset: Byte position to read from — pass back the returned offset to read only what appeared since. :returns: The parsed lines (see :func:`append_record`) and the offset just after the last complete line; ``([], offset)`` while the file does not exist yet. .. py:function:: record_value(line: dict[str, Any], run_dir: str | pathlib.Path) -> Any Return the value a ``records.jsonl`` line stands for. :param line: A line as returned by :func:`read_records`. :param run_dir: The run directory the line's ``blob`` is relative to. :returns: The inlined value, or the :class:`PlestyArray` / raw ``bytes`` loaded from the referenced blob. :raises ValueError: If the line declares an unknown result type. .. py:function:: load_document(path: str | pathlib.Path) -> ResultDocument Load only the metadata document of a persisted result — no blob I/O. Useful for browsing run directories (checking provenance, shapes, or timestamps) without paying the cost of decoding the referenced blobs. :param path: Path to the JSON metadata document (``.json``). :returns: The parsed :class:`ResultDocument`. :raises ValueError: If the document declares an unknown result type. .. py:function:: load_result(path: str | pathlib.Path) -> Any Load a result previously written by :func:`save_result`. :param path: Path to the JSON metadata document (``.json``). :returns: a :class:`PlestyArray` with its metadata restored, raw ``bytes`` for encoded blobs, or the inlined value. :rtype: The reconstructed value :raises ValueError: If the document declares an unknown result type. .. py:function:: convert_to_hdf5(documents: Iterable[str | pathlib.Path], target: str | pathlib.Path) -> pathlib.Path Pack results written by :func:`save_result` into one HDF5 archive. The raw blob + JSON layout stays the primary on-disk format; this is an optional post-hoc export to the unified HDF5 format preferred by issue plesty-lib#4. Each JSON document becomes one HDF5 group (named after the document stem) holding a ``data`` dataset — the decoded array, the raw encoded bytes, or the JSON-encoded inline value — with the document's metadata and provenance stored as group attributes. :param documents: Paths of the JSON metadata documents to include. :param target: Destination ``.h5`` file; parent directories are created. :returns: The resolved path of the written HDF5 file. :raises ImportError: If the ``hdf5`` extra is not installed. :raises ValueError: If a document declares an unknown result type.