plesty.lib.data

Data types and utilities for Plesty.

Submodules

Classes

`PlestyArray`	Data definition for a numerical array with meta information.
`Units`	Standardized unit expression parser and composer.
`TableHeader`	Represents the header of a PlestyTable, defining column names and types.
`PlestyTable2D`	A simple 2D table structure that holds a list of 1d PlestyArrays,
`PlestyTable3D`	A simple 3D table structure that holds a list of 2d PlestyTables,
`ResultDocument`	Typed model of the JSON metadata document written by `save_result()`.

Functions

`resolve_dtype`(...)	Resolve schema dtype strings into Python types.
`resolve_array_item_dtype`(→ Any)	Resolve the NumPy item dtype from an array dtype token like `array_float32`.
`normalize_shape`(→ tuple[Any, Ellipsis] \| None)	Normalize a shape specification into a tuple, or return None.
`istype`(→ bool)	Check if a value is of a specified data type, including basic and iterable types.
`cast_basic_type`(→ Any)	Cast a value to the given basic type, raising ValueError on failure.
`append_record`(→ dict[str, Any])	Append one step result to the run's `records.jsonl`.
`convert_to_hdf5`(→ pathlib.Path)	Pack results written by `save_result()` into one HDF5 archive.
`load_document`(→ ResultDocument)	Load only the metadata document of a persisted result — no blob I/O.
`load_result`(→ Any)	Load a result previously written by `save_result()`.
`read_records`(→ tuple[list[dict[str, Any]], int])	Read the record lines appended to `records.jsonl` since offset.
`record_value`(→ Any)	Return the value a `records.jsonl` line stands for.
`save_result`(→ pathlib.Path)	Persist a measurement result as raw blob + JSON metadata document.

Package Contents

class plesty.lib.data.PlestyArray(shape, dtype=float, buffer=None, offset=0, strides=None, order=None)

Bases: numpy.ndarray

Data definition for a numerical array with meta information. The array inherits from numpy.ndarray, so it can be used as a regular numpy array, but it also has additional attributes for meta information and additional methods for data manipulation.

Variables:

name – Optional name for the data.
range – Optional range for the data values, can be a tuple of (min, max).
options – Optional list of possible values for the data, if applicable.
unit – Optional unit for the data, e.g., “nm”, “s”, “m/s”, etc.
description – Optional description for the data, providing more context and information.

Usage:

from plesty.lib.data import PlestyArray as PArray
data = PArray([1.0, 2.0, 3.0], name="Example Data", unit="nm",
              description="This is an example data array.")
np_arr = np.random.rand(3)
plesty_arr = PArray(np_arr, name="Random Data", unit="s",
                    description="This is a random data array.")

# NumPy-like constructors on PlestyArray
z = PArray.zeros((10, 2), unit="a.u.", description="Zero-filled matrix")
o = PArray.ones(5, name="weights")
x = PArray.arange(0, 1, 0.1, unit="s")
grid = PArray.linspace(400, 700, 5, unit="nm", name="wavelength")

# Unit-aware operations with metadata updates
speed = PArray([10.0, 12.0], name="speed", unit="m/s", description="Speed of Rocket")
time = PArray([2.0, 3.0], name="time", unit="s", description="Time")
distance = speed * time
# distance.unit -> "m"
# distance.name -> "speed(m/s) * time(s)"
# distance.description -> "speed: Speed of Rocket; time: Time"

Create a new PlestyArray instance with optional metadata.

_META_KEYS

__array_priority__ = 1000

name: str | None

range: Tuple[float, float] | Tuple[int, int] | None

options: List[Any] | None

unit: str | None

description: str | None

update_meta(**kwargs) → None

Update meta information for the PlestyArray.

Example:

arr = PlestyArray([1.0, 2.0, 3.0], name="Example Data", unit="nm")
arr.update_meta(description="Updated description", range=(0.0, 5.0))

Return type:: None

_meta_kwargs() → Dict[str, Any]

Return type:: Dict[str, Any]

classmethod _find_meta_source(items) → PlestyArray | None

Return type:: Optional[PlestyArray]

static _unit_text(unit: str | None) → str

Parameters:: unit (Optional[str])
Return type:: str

classmethod _build_binary_meta(left: PlestyArray, right: PlestyArray, op_symbol: str) → Dict[str, Any]

Parameters:

left (PlestyArray)
right (PlestyArray)
op_symbol (str)

Return type:

Dict[str, Any]

classmethod _split_meta_kwargs(kwargs: Dict[str, Any]) → Tuple[Dict[str, Any], Dict[str, Any]]

Parameters:: kwargs (Dict[str, Any])
Return type:: Tuple[Dict[str, Any], Dict[str, Any]]

classmethod _wrap_numpy_result(result: Any, meta_kwargs: Dict[str, Any])

Parameters:

result (Any)
meta_kwargs (Dict[str, Any])

classmethod _call_numpy(np_func, *args, **kwargs)

__array_finalize__(obj): Finalize the PlestyArray after creation, copying metadata from source.

__array_ufunc__(ufunc, method, *inputs, **kwargs): Handle NumPy ufuncs with unit-aware metadata propagation.

__array_function__(func, types, args, kwargs): Dispatch NumPy functions while preserving PlestyArray metadata.

__repr__(): Return a developer-friendly representation of the PlestyArray.

__str__(): Return a human-readable string representation of the PlestyArray.

class plesty.lib.data.Units(dims: Dict[str, int] | None = None, factor: float = 1.0)

Standardized unit expression parser and composer.

The internal representation tracks: - dims: dimension exponents, e.g. {“L”: 1, “T”: -1} for speed - factor: numeric scaling factor to SI base units

Initialize a Units instance with dimension exponents and a scaling factor.

Parameters:

dims (Optional[Dict[str, int]])
factor (float)

SYMBOLS: dict[str, dict[str, Any]]

BASE_SYMBOLS

dims

factor

classmethod scalar() → Units

Return a dimensionless scalar Units instance.

Return type:: Units

classmethod parse(unit_str: str | None) → Units | None

Parse a unit string into a Units instance, returning None if unparseable.

Parameters:: unit_str (Optional[str])
Return type:: Optional[Units]

classmethod standardize(unit_str: str | None) → str | None

Return a normalized whitespace-free unit string, or None for empty input.

Parameters:: unit_str (Optional[str])
Return type:: Optional[str]

compatible(other: Units) → bool

Return True if this unit has the same dimensions as other.

Parameters:: other (Units)
Return type:: bool

conversion_scale_to(target: Units) → float

Return the numeric factor to convert this unit to the target unit.

Parameters:: target (Units)
Return type:: float

multiply(other: Units) → Units

Return the unit resulting from multiplying this unit by other.

Parameters:: other (Units)
Return type:: Units

divide(other: Units) → Units

Return the unit resulting from dividing this unit by other.

Parameters:: other (Units)
Return type:: Units

power(exponent: int) → Units

Return this unit raised to an integer exponent.

Parameters:: exponent (int)
Return type:: Units

to_unit_string() → str | None

Render the unit dimensions as a human-readable string.

Return type:: Optional[str]

plesty.lib.data.resolve_dtype(dtype: Any) → type[Any] | plesty.lib.data.ctype_manager.CtypeParam | None

Resolve schema dtype strings into Python types.

Supports scalar aliases and array types in the form array_<numpy_dtype>, e.g. array_float, array_int32, array_float64.

Parameters:: dtype (Any)
Return type:: type[Any] | plesty.lib.data.ctype_manager.CtypeParam | None

plesty.lib.data.resolve_array_item_dtype(dtype: Any) → Any

Resolve the NumPy item dtype from an array dtype token like array_float32.

Parameters:: dtype (Any)
Return type:: Any

plesty.lib.data.normalize_shape(shape: Any) → tuple[Any, Ellipsis] | None

Normalize a shape specification into a tuple, or return None.

Parameters:: shape (Any)
Return type:: tuple[Any, Ellipsis] | None

plesty.lib.data.istype(value: Any, dtype: Any) → bool

Check if a value is of a specified data type, including basic and iterable types.

Parameters:

value (Any)
dtype (Any)

Return type:

bool

plesty.lib.data.cast_basic_type(value: Any, dtype: Any) → Any

Cast a value to the given basic type, raising ValueError on failure.

Parameters:

value (Any)
dtype (Any)

Return type:

Any

class plesty.lib.data.TableHeader

Represents the header of a PlestyTable, defining column names and types.

name: str

dtype: Any = None

unit: str | None = None

description: str | None = None

class plesty.lib.data.PlestyTable2D

A simple 2D table structure that holds a list of 1d PlestyArrays, each representing a row/column of the table.

name: str

data: list[plesty.lib.data.array.PlestyArray]

header: list[TableHeader] | None = None

description: str | None = None

class plesty.lib.data.PlestyTable3D

A simple 3D table structure that holds a list of 2d PlestyTables, each representing a layer of the table.

name: str

shape: tuple[int, int]

data: list[list[plesty.lib.data.array.PlestyArray]]

rheader: list[TableHeader] | None = None

cheader: list[TableHeader] | None = None

description: str | None = None

get_item_at(row: int, col: int) → plesty.lib.data.array.PlestyArray

Get the item at the specified row and column.

Parameters:

row (int)
col (int)

Return type:

plesty.lib.data.array.PlestyArray

class plesty.lib.data.ResultDocument

Typed model of the JSON metadata document written by save_result().

Formalises the experimental-output metadata schema (issue plesty-lib#4): every persisted result is described by one such document, which is the commit record of the write. Blob-backed results (array, bytes) reference their raw data file via blob; plain values are inlined in value.

Variables:

type – Result category — one of RESULT_TYPES.
saved_at – UTC ISO-8601 write time.
provenance – Context recorded at save time (step id, operation, parameters, device identity, …).
blob – File name of the raw data blob next to the document (array/bytes results only).
format – Blob encoding ("npy", "png", "bin", …).
shape – Array shape (array results only).
dtype – Array dtype string (array results only).
meta – Plesty metadata of the array (name, range, options, unit, description; array results only).
value – The inlined JSON-serializable value (value results only).

type: str

saved_at: str = ''

provenance: dict[str, Any]

blob: str | None = None

format: str | None = None

shape: list[int] | None = None

dtype: str | None = None

meta: dict[str, Any] | None = None

value: Any = None

__post_init__() → None

Validate the result type and stamp the write time when missing.

Return type:: None

to_dict() → dict[str, Any]

Return the document as a JSON-serializable dictionary.

Blob-related fields that do not apply to the result type are omitted, matching the on-disk layout written since the schema’s introduction.

Return type:: dict[str, Any]

classmethod from_dict(document: dict[str, Any]) → ResultDocument

Reconstruct a document from a dictionary produced by to_dict().

Parameters:: document (dict[str, Any]) – The parsed JSON document.
Returns:: The reconstructed ResultDocument.
Raises:: ValueError – If the document declares an unknown result type.
Return type:: ResultDocument

plesty.lib.data.append_record(run_dir: str | pathlib.Path, index: int, result: Any, provenance: dict[str, Any] | None = None) → dict[str, Any]

Append one step result to the run’s records.jsonl.

A blob-backed result (array, bytes) is written to <run_dir>/data/step_<index>.<fmt> first, atomically; the document line — the commit record — is appended and fsynced afterwards, so a reader never sees a document whose blob is missing.

Parameters:

run_dir (str | pathlib.Path) – The run directory.
index (int) – Position of the step in the plan; stamped onto the line and used for the blob file name.
result (Any) – The value to persist (see save_result()).
provenance (Optional[dict[str, Any]]) – Context recorded verbatim in the document.

Returns:

The line that was written, as a dictionary — the document fields plus index; blob (when present) is relative to run_dir.

Raises:

TypeError – If the result is neither a PlestyArray, bytes, nor JSON-serializable.

Return type:

dict[str, Any]

plesty.lib.data.convert_to_hdf5(documents: Iterable[str | pathlib.Path], target: str | pathlib.Path) → pathlib.Path

Pack results written by save_result() into one HDF5 archive.

The raw blob + JSON layout stays the primary on-disk format; this is an optional post-hoc export to the unified HDF5 format preferred by issue plesty-lib#4. Each JSON document becomes one HDF5 group (named after the document stem) holding a data dataset — the decoded array, the raw encoded bytes, or the JSON-encoded inline value — with the document’s metadata and provenance stored as group attributes.

Parameters:

documents (Iterable[str | pathlib.Path]) – Paths of the JSON metadata documents to include.
target (str | pathlib.Path) – Destination .h5 file; parent directories are created.

Returns:

The resolved path of the written HDF5 file.

Raises:

ImportError – If the hdf5 extra is not installed.
ValueError – If a document declares an unknown result type.

Return type:

pathlib.Path

plesty.lib.data.load_document(path: str | pathlib.Path) → ResultDocument

Load only the metadata document of a persisted result — no blob I/O.

Useful for browsing run directories (checking provenance, shapes, or timestamps) without paying the cost of decoding the referenced blobs.

Parameters:: path (str | pathlib.Path) – Path to the JSON metadata document (<stem>.json).
Returns:: The parsed ResultDocument.
Raises:: ValueError – If the document declares an unknown result type.
Return type:: ResultDocument

plesty.lib.data.load_result(path: str | pathlib.Path) → Any

Load a result previously written by save_result().

Parameters:: path (str | pathlib.Path) – Path to the JSON metadata document (<stem>.json).
Returns:: a PlestyArray with its metadata restored, raw bytes for encoded blobs, or the inlined value.
Return type:: The reconstructed value
Raises:: ValueError – If the document declares an unknown result type.

plesty.lib.data.read_records(run_dir: str | pathlib.Path, offset: int = 0) → tuple[list[dict[str, Any]], int]

Read the record lines appended to records.jsonl since offset.

Only complete lines are returned; a line still being written (no trailing newline yet) is left for the next call, which is what makes tailing a run in progress safe. Lines that fail to parse are skipped.

Parameters:

run_dir (str | pathlib.Path) – The run directory.
offset (int) – Byte position to read from — pass back the returned offset to read only what appeared since.

Returns:

The parsed lines (see append_record()) and the offset just after the last complete line; ([], offset) while the file does not exist yet.

Return type:

tuple[list[dict[str, Any]], int]

plesty.lib.data.record_value(line: dict[str, Any], run_dir: str | pathlib.Path) → Any

Return the value a records.jsonl line stands for.

Parameters:

line (dict[str, Any]) – A line as returned by read_records().
run_dir (str | pathlib.Path) – The run directory the line’s blob is relative to.

Returns:

The inlined value, or the PlestyArray / raw bytes loaded from the referenced blob.

Raises:

ValueError – If the line declares an unknown result type.

Return type:

Any

plesty.lib.data.save_result(result: Any, path_stem: str | pathlib.Path, provenance: dict[str, Any] | None = None) → pathlib.Path

Persist a measurement result as raw blob + JSON metadata document.

Parameters:

result (Any) – The value to persist — a PlestyArray, encoded bytes (e.g. an image), or any JSON-serializable value.
path_stem (str | pathlib.Path) – Destination path without suffix; the JSON document is written to <stem>.json and any blob next to it.
provenance (Optional[dict[str, Any]]) – Optional context (step id, operation, parameters, device identity) recorded verbatim in the document.

Returns:

The path of the JSON metadata document.

Raises:

TypeError – If the result is neither a PlestyArray, bytes, nor JSON-serializable.

Return type:

pathlib.Path