Hdf5File

Hdf5File is a File subclass that points at an HDF5 file and provides methods for inspecting its groups and datasets and reading dataset data.

Install the optional dependency with pip install 'datachain[hdf5]'.

An HDF5 file is a single byte stream, so Hdf5File rows are created by read_storage with type="hdf5". Data is read through the regular streaming file handle, so a single dataset - or a single slice of one - is fetched without pulling the whole file:

import datachain as dc

chain = dc.read_storage("s3://bucket-name/trajectories/", type="hdf5")
for (file,) in chain.limit(1).to_iter("file"):
    print(file.get_info())

Paths follow the HDF5 convention and are absolute within the file (e.g. /robot/joint_positions); the reader also accepts them without the leading slash. A File obtained some other way can be converted with file.as_hdf5_file().

The models are not re-exported from the top-level datachain namespace, so that import datachain never loads h5py. Import them directly when annotating a UDF or building a model by hand:

from datachain.lib.hdf5 import Hdf5Dataset, Hdf5File, Hdf5Selection

There are additional models for working with HDF5 files:

• Hdf5Info - summary metadata for a file (attributes, dataset paths, group paths).
• Hdf5Dataset - a single dataset within a file; exposes shape, chunks, dtype, and attrs, and reads data via read() or select().
• Hdf5Selection - a lazy, bounded region inside a dataset (e.g. one image frame) that can travel through a chain as a column and is materialized on demand via read() or rendered to image bytes via read_bytes().

Only the generic HDF5 group/dataset model is handled here. Conventions layered on top of HDF5 - NetCDF4 dimensions and coordinates, LeRobot episode layouts - are not interpreted, though such files still load as ordinary HDF5.

Hdf5File

Hdf5File(**kwargs)

Bases: File

A data model for handling HDF5 files.

This model inherits from the File model and provides additional functionality for inspecting an HDF5 file's groups and datasets and reading dataset data.

Paths follow the HDF5 convention and are absolute within the file (e.g. /robot/joint_positions); the reader also accepts them without the leading slash.

Source code in datachain/lib/file.py

def __init__(self, **kwargs):
    super().__init__(**kwargs)
    self._catalog = None
    self._caching_enabled: bool = False
    self._download_cb: Callback = DEFAULT_CALLBACK
    self._fs_path_cache: tuple[str, str, str] | None = None

get_dataset

get_dataset(path: str) -> Hdf5Dataset

Return a single dataset by its path within the file.

Source code in datachain/lib/hdf5.py

def get_dataset(self, path: str) -> "Hdf5Dataset":
    """Return a single dataset by its path within the file."""
    h5py = _require_h5py()
    with self._open_h5() as f:
        node = f[path]
        if not isinstance(node, h5py.Dataset):
            raise ValueError(  # noqa: TRY004
                f"'{path}' is not an HDF5 dataset in file {self.path!r}"
            )
        return self._to_dataset(node)

get_datasets

get_datasets(group: str = '/') -> Iterator[Hdf5Dataset]

Yield every dataset under group (recursively).

Source code in datachain/lib/hdf5.py

def get_datasets(self, group: str = "/") -> Iterator["Hdf5Dataset"]:
    """Yield every dataset under ``group`` (recursively)."""
    h5py = _require_h5py()
    found: list[Hdf5Dataset] = []

    def collect(_name: str, node: Any) -> None:
        if isinstance(node, h5py.Dataset):
            found.append(self._to_dataset(node))

    with self._open_h5() as f:
        node = f[group]
        if isinstance(node, h5py.Dataset):
            found.append(self._to_dataset(node))
        else:
            # visititems tracks visited objects, so a group reachable
            # through more than one hard link is walked only once.
            node.visititems(collect)

    yield from found

get_info

get_info() -> Hdf5Info

Return summary metadata for the file.

Source code in datachain/lib/hdf5.py

def get_info(self) -> Hdf5Info:
    """Return summary metadata for the file."""
    h5py = _require_h5py()
    datasets: list[str] = []
    groups: list[str] = []

    def collect(_name: str, node: Any) -> None:
        target = datasets if isinstance(node, h5py.Dataset) else groups
        target.append(node.name)

    with self._open_h5() as f:
        f.visititems(collect)
        return Hdf5Info(attrs=_attrs(f), datasets=datasets, groups=groups)

Hdf5Dataset

Bases: DataModel

A single dataset within an :class:Hdf5File.

shape is the HDF5 shape as a list, so a scalar dataset has an empty shape while a zero-length one-dimensional dataset has [0].

read

read(selection: Any = None) -> Any

Read dataset data, optionally restricted to a NumPy-style selection.

Source code in datachain/lib/hdf5.py

def read(self, selection: Any = None) -> Any:
    """Read dataset data, optionally restricted to a NumPy-style selection."""
    with self.file._open_h5() as f:
        node = f[self.path]
        if selection is None:
            return node[...]
        return node[selection]

select

select(
    index: int | list[int],
    media: Literal["image", "audio", "video"] | None = None,
) -> Hdf5Selection

Return a lazy :class:Hdf5Selection pointing at an item in this dataset.

index addresses the leading axes (e.g. i or [i] for one frame of an (N, H, W, C) dataset). The region is read on demand via :meth:Hdf5Selection.read, so the item can travel through a DataChain as a column without materializing its bytes.

Source code in datachain/lib/hdf5.py

def select(
    self,
    index: "int | list[int]",
    media: "Literal['image', 'audio', 'video'] | None" = None,
) -> "Hdf5Selection":
    """Return a lazy :class:`Hdf5Selection` pointing at an item in this dataset.

    ``index`` addresses the leading axes (e.g. ``i`` or ``[i]`` for one
    frame of an ``(N, H, W, C)`` dataset).  The region is read on demand via
    :meth:`Hdf5Selection.read`, so the item can travel through a DataChain
    as a column without materializing its bytes.
    """
    idx = [index] if isinstance(index, int) else list(index)
    return Hdf5Selection(dataset=self, index=idx, media=media)

Hdf5Selection

Bases: DataModel

A lazy, bounded region inside an :class:Hdf5Dataset.

Points at a single item (or block) inside a dataset without reading it, analogous to how :class:~datachain.lib.file.File points at a byte stream. index addresses the leading axes; :meth:read materializes the region.

read

read() -> Any

Read and return the selected region.

Source code in datachain/lib/hdf5.py

def read(self) -> Any:
    """Read and return the selected region."""
    return self.dataset.read(tuple(self.index))

read_bytes

read_bytes(format: str = 'PNG') -> bytes

Render the selected region to encoded media bytes.

Only media="image" is supported for now: the region is read and encoded with Pillow (e.g. PNG), so callers such as Studio can stream a preview without materializing the image into the row.

Source code in datachain/lib/hdf5.py

def read_bytes(self, format: str = "PNG") -> bytes:
    """Render the selected region to encoded media bytes.

    Only ``media="image"`` is supported for now: the region is read and
    encoded with Pillow (e.g. PNG), so callers such as Studio can stream a
    preview without materializing the image into the row.
    """
    if self.media not in (None, "image"):
        raise ValueError(f"read_bytes() supports image media, not {self.media!r}")
    import io

    import numpy as np
    from PIL import Image

    # Normalize e.g. "jpg"/".png" to a registered Pillow format name, with a
    # plain upper-cased fallback.
    ext = format if format.startswith(".") else f".{format}"
    pil_format = Image.registered_extensions().get(ext.lower(), format.upper())

    arr = np.asarray(self.read())
    if arr.dtype != np.uint8:
        arr = arr.astype("uint8")
    buf = io.BytesIO()
    Image.fromarray(arr).save(buf, format=pil_format)
    return buf.getvalue()

Hdf5Info

Bases: DataModel

Summary metadata for an HDF5 file.