plesty.lib.data =============== .. py:module:: plesty.lib.data .. autoapi-nested-parse:: Data types and utilities for Plesty. Submodules ---------- .. toctree:: :maxdepth: 1 /reference/plesty/lib/data/array/index /reference/plesty/lib/data/ctype_manager/index /reference/plesty/lib/data/io/index /reference/plesty/lib/data/table/index /reference/plesty/lib/data/types/index /reference/plesty/lib/data/units/index Classes ------- .. autoapisummary:: plesty.lib.data.PlestyArray plesty.lib.data.Units plesty.lib.data.TableHeader plesty.lib.data.PlestyTable2D plesty.lib.data.PlestyTable3D plesty.lib.data.ResultDocument Functions --------- .. autoapisummary:: plesty.lib.data.resolve_dtype plesty.lib.data.resolve_array_item_dtype plesty.lib.data.normalize_shape plesty.lib.data.istype plesty.lib.data.cast_basic_type plesty.lib.data.append_record plesty.lib.data.convert_to_hdf5 plesty.lib.data.load_document plesty.lib.data.load_result plesty.lib.data.read_records plesty.lib.data.record_value plesty.lib.data.save_result Package Contents ---------------- .. py:class:: PlestyArray(shape, dtype=float, buffer=None, offset=0, strides=None, order=None) Bases: :py:obj:`numpy.ndarray` Data definition for a numerical array with meta information. The array inherits from numpy.ndarray, so it can be used as a regular numpy array, but it also has additional attributes for meta information and additional methods for data manipulation. :ivar name: Optional name for the data. :ivar range: Optional range for the data values, can be a tuple of (min, max). :ivar options: Optional list of possible values for the data, if applicable. :ivar unit: Optional unit for the data, e.g., "nm", "s", "m/s", etc. :ivar description: Optional description for the data, providing more context and information. Usage: .. code-block:: python from plesty.lib.data import PlestyArray as PArray data = PArray([1.0, 2.0, 3.0], name="Example Data", unit="nm", description="This is an example data array.") np_arr = np.random.rand(3) plesty_arr = PArray(np_arr, name="Random Data", unit="s", description="This is a random data array.") # NumPy-like constructors on PlestyArray z = PArray.zeros((10, 2), unit="a.u.", description="Zero-filled matrix") o = PArray.ones(5, name="weights") x = PArray.arange(0, 1, 0.1, unit="s") grid = PArray.linspace(400, 700, 5, unit="nm", name="wavelength") # Unit-aware operations with metadata updates speed = PArray([10.0, 12.0], name="speed", unit="m/s", description="Speed of Rocket") time = PArray([2.0, 3.0], name="time", unit="s", description="Time") distance = speed * time # distance.unit -> "m" # distance.name -> "speed(m/s) * time(s)" # distance.description -> "speed: Speed of Rocket; time: Time" Create a new PlestyArray instance with optional metadata. .. py:attribute:: _META_KEYS .. py:attribute:: __array_priority__ :value: 1000 .. py:attribute:: name :type: Optional[str] .. py:attribute:: range :type: Optional[Tuple[float, float] | Tuple[int, int]] .. py:attribute:: options :type: Optional[List[Any]] .. py:attribute:: unit :type: Optional[str] .. py:attribute:: description :type: Optional[str] .. py:method:: update_meta(**kwargs) -> None Update meta information for the PlestyArray. Example: .. code-block:: python arr = PlestyArray([1.0, 2.0, 3.0], name="Example Data", unit="nm") arr.update_meta(description="Updated description", range=(0.0, 5.0)) .. py:method:: _meta_kwargs() -> Dict[str, Any] .. py:method:: _find_meta_source(items) -> Optional[PlestyArray] :classmethod: .. py:method:: _unit_text(unit: Optional[str]) -> str :staticmethod: .. py:method:: _build_binary_meta(left: PlestyArray, right: PlestyArray, op_symbol: str) -> Dict[str, Any] :classmethod: .. py:method:: _split_meta_kwargs(kwargs: Dict[str, Any]) -> Tuple[Dict[str, Any], Dict[str, Any]] :classmethod: .. py:method:: _wrap_numpy_result(result: Any, meta_kwargs: Dict[str, Any]) :classmethod: .. py:method:: _call_numpy(np_func, *args, **kwargs) :classmethod: .. py:method:: __array_finalize__(obj) Finalize the PlestyArray after creation, copying metadata from source. .. py:method:: __array_ufunc__(ufunc, method, *inputs, **kwargs) Handle NumPy ufuncs with unit-aware metadata propagation. .. py:method:: __array_function__(func, types, args, kwargs) Dispatch NumPy functions while preserving PlestyArray metadata. .. py:method:: __repr__() Return a developer-friendly representation of the PlestyArray. .. py:method:: __str__() Return a human-readable string representation of the PlestyArray. .. py:class:: Units(dims: Optional[Dict[str, int]] = None, factor: float = 1.0) Standardized unit expression parser and composer. The internal representation tracks: - ``dims``: dimension exponents, e.g. {"L": 1, "T": -1} for speed - ``factor``: numeric scaling factor to SI base units Initialize a Units instance with dimension exponents and a scaling factor. .. py:attribute:: SYMBOLS :type: dict[str, dict[str, Any]] .. py:attribute:: BASE_SYMBOLS .. py:attribute:: dims .. py:attribute:: factor .. py:method:: scalar() -> Units :classmethod: Return a dimensionless scalar Units instance. .. py:method:: parse(unit_str: Optional[str]) -> Optional[Units] :classmethod: Parse a unit string into a Units instance, returning None if unparseable. .. py:method:: standardize(unit_str: Optional[str]) -> Optional[str] :classmethod: Return a normalized whitespace-free unit string, or None for empty input. .. py:method:: compatible(other: Units) -> bool Return True if this unit has the same dimensions as other. .. py:method:: conversion_scale_to(target: Units) -> float Return the numeric factor to convert this unit to the target unit. .. py:method:: multiply(other: Units) -> Units Return the unit resulting from multiplying this unit by other. .. py:method:: divide(other: Units) -> Units Return the unit resulting from dividing this unit by other. .. py:method:: power(exponent: int) -> Units Return this unit raised to an integer exponent. .. py:method:: to_unit_string() -> Optional[str] Render the unit dimensions as a human-readable string. .. py:function:: resolve_dtype(dtype: Any) -> type[Any] | plesty.lib.data.ctype_manager.CtypeParam | None Resolve schema dtype strings into Python types. Supports scalar aliases and array types in the form array_, e.g. array_float, array_int32, array_float64. .. py:function:: resolve_array_item_dtype(dtype: Any) -> Any Resolve the NumPy item dtype from an array dtype token like ``array_float32``. .. py:function:: normalize_shape(shape: Any) -> tuple[Any, Ellipsis] | None Normalize a shape specification into a tuple, or return None. .. py:function:: istype(value: Any, dtype: Any) -> bool Check if a value is of a specified data type, including basic and iterable types. .. py:function:: cast_basic_type(value: Any, dtype: Any) -> Any Cast a value to the given basic type, raising ValueError on failure. .. py:class:: TableHeader Represents the header of a PlestyTable, defining column names and types. .. py:attribute:: name :type: str .. py:attribute:: dtype :type: Any :value: None .. py:attribute:: unit :type: str | None :value: None .. py:attribute:: description :type: str | None :value: None .. py:class:: PlestyTable2D A simple 2D table structure that holds a list of 1d PlestyArrays, each representing a row/column of the table. .. py:attribute:: name :type: str .. py:attribute:: data :type: list[plesty.lib.data.array.PlestyArray] .. py:attribute:: header :type: list[TableHeader] | None :value: None .. py:attribute:: description :type: str | None :value: None .. py:class:: PlestyTable3D A simple 3D table structure that holds a list of 2d PlestyTables, each representing a layer of the table. .. py:attribute:: name :type: str .. py:attribute:: shape :type: tuple[int, int] .. py:attribute:: data :type: list[list[plesty.lib.data.array.PlestyArray]] .. py:attribute:: rheader :type: list[TableHeader] | None :value: None .. py:attribute:: cheader :type: list[TableHeader] | None :value: None .. py:attribute:: description :type: str | None :value: None .. py:method:: get_item_at(row: int, col: int) -> plesty.lib.data.array.PlestyArray Get the item at the specified row and column. .. py:class:: ResultDocument Typed model of the JSON metadata document written by :func:`save_result`. Formalises the experimental-output metadata schema (issue plesty-lib#4): every persisted result is described by one such document, which is the commit record of the write. Blob-backed results (``array``, ``bytes``) reference their raw data file via :attr:`blob`; plain values are inlined in :attr:`value`. :ivar type: Result category — one of :data:`RESULT_TYPES`. :ivar saved_at: UTC ISO-8601 write time. :ivar provenance: Context recorded at save time (step id, operation, parameters, device identity, …). :ivar blob: File name of the raw data blob next to the document (``array``/``bytes`` results only). :ivar format: Blob encoding (``"npy"``, ``"png"``, ``"bin"``, …). :ivar shape: Array shape (``array`` results only). :ivar dtype: Array dtype string (``array`` results only). :ivar meta: Plesty metadata of the array (name, range, options, unit, description; ``array`` results only). :ivar value: The inlined JSON-serializable value (``value`` results only). .. py:attribute:: type :type: str .. py:attribute:: saved_at :type: str :value: '' .. py:attribute:: provenance :type: dict[str, Any] .. py:attribute:: blob :type: str | None :value: None .. py:attribute:: format :type: str | None :value: None .. py:attribute:: shape :type: list[int] | None :value: None .. py:attribute:: dtype :type: str | None :value: None .. py:attribute:: meta :type: dict[str, Any] | None :value: None .. py:attribute:: value :type: Any :value: None .. py:method:: __post_init__() -> None Validate the result type and stamp the write time when missing. .. py:method:: to_dict() -> dict[str, Any] Return the document as a JSON-serializable dictionary. Blob-related fields that do not apply to the result type are omitted, matching the on-disk layout written since the schema's introduction. .. py:method:: from_dict(document: dict[str, Any]) -> ResultDocument :classmethod: Reconstruct a document from a dictionary produced by :meth:`to_dict`. :param document: The parsed JSON document. :returns: The reconstructed :class:`ResultDocument`. :raises ValueError: If the document declares an unknown result type. .. py:function:: append_record(run_dir: str | pathlib.Path, index: int, result: Any, provenance: Optional[dict[str, Any]] = None) -> dict[str, Any] Append one step result to the run's ``records.jsonl``. A blob-backed result (array, bytes) is written to ``/data/step_.`` first, atomically; the document line — the commit record — is appended and fsynced afterwards, so a reader never sees a document whose blob is missing. :param run_dir: The run directory. :param index: Position of the step in the plan; stamped onto the line and used for the blob file name. :param result: The value to persist (see :func:`save_result`). :param provenance: Context recorded verbatim in the document. :returns: The line that was written, as a dictionary — the document fields plus ``index``; ``blob`` (when present) is relative to *run_dir*. :raises TypeError: If the result is neither a PlestyArray, bytes, nor JSON-serializable. .. py:function:: convert_to_hdf5(documents: Iterable[str | pathlib.Path], target: str | pathlib.Path) -> pathlib.Path Pack results written by :func:`save_result` into one HDF5 archive. The raw blob + JSON layout stays the primary on-disk format; this is an optional post-hoc export to the unified HDF5 format preferred by issue plesty-lib#4. Each JSON document becomes one HDF5 group (named after the document stem) holding a ``data`` dataset — the decoded array, the raw encoded bytes, or the JSON-encoded inline value — with the document's metadata and provenance stored as group attributes. :param documents: Paths of the JSON metadata documents to include. :param target: Destination ``.h5`` file; parent directories are created. :returns: The resolved path of the written HDF5 file. :raises ImportError: If the ``hdf5`` extra is not installed. :raises ValueError: If a document declares an unknown result type. .. py:function:: load_document(path: str | pathlib.Path) -> ResultDocument Load only the metadata document of a persisted result — no blob I/O. Useful for browsing run directories (checking provenance, shapes, or timestamps) without paying the cost of decoding the referenced blobs. :param path: Path to the JSON metadata document (``.json``). :returns: The parsed :class:`ResultDocument`. :raises ValueError: If the document declares an unknown result type. .. py:function:: load_result(path: str | pathlib.Path) -> Any Load a result previously written by :func:`save_result`. :param path: Path to the JSON metadata document (``.json``). :returns: a :class:`PlestyArray` with its metadata restored, raw ``bytes`` for encoded blobs, or the inlined value. :rtype: The reconstructed value :raises ValueError: If the document declares an unknown result type. .. py:function:: read_records(run_dir: str | pathlib.Path, offset: int = 0) -> tuple[list[dict[str, Any]], int] Read the record lines appended to ``records.jsonl`` since *offset*. Only complete lines are returned; a line still being written (no trailing newline yet) is left for the next call, which is what makes tailing a run in progress safe. Lines that fail to parse are skipped. :param run_dir: The run directory. :param offset: Byte position to read from — pass back the returned offset to read only what appeared since. :returns: The parsed lines (see :func:`append_record`) and the offset just after the last complete line; ``([], offset)`` while the file does not exist yet. .. py:function:: record_value(line: dict[str, Any], run_dir: str | pathlib.Path) -> Any Return the value a ``records.jsonl`` line stands for. :param line: A line as returned by :func:`read_records`. :param run_dir: The run directory the line's ``blob`` is relative to. :returns: The inlined value, or the :class:`PlestyArray` / raw ``bytes`` loaded from the referenced blob. :raises ValueError: If the line declares an unknown result type. .. py:function:: save_result(result: Any, path_stem: str | pathlib.Path, provenance: Optional[dict[str, Any]] = None) -> pathlib.Path Persist a measurement result as raw blob + JSON metadata document. :param result: The value to persist — a :class:`PlestyArray`, encoded ``bytes`` (e.g. an image), or any JSON-serializable value. :param path_stem: Destination path without suffix; the JSON document is written to ``.json`` and any blob next to it. :param provenance: Optional context (step id, operation, parameters, device identity) recorded verbatim in the document. :returns: The path of the JSON metadata document. :raises TypeError: If the result is neither a PlestyArray, bytes, nor JSON-serializable.