Unified Zarr v3 convention: align hdmf-zarr and LINDI storage formats

[bendichter]

Context

Problem: hdmf-zarr and LINDI both represent data types that Zarr doesn't natively support (links, references, compound types, scalars) but use incompatible conventions. This prevents using LINDI's Zarr output directly with hdmf-zarr's NWBZarrIO, forcing LINDI to maintain an h5py shim layer that causes edge-case bugs.

Opportunity: Both libraries are breaking backwards compatibility for Zarr v2 → v3 migration. This is the ideal time to align on a single standard.

• hdmf-zarr (zarr-v3-migration branch): already on zarr>=3.1.3, using v3 APIs
• LINDI: pinned to zarr>=2.16.1,<3, still fully v2

Goal: Define a shared convention so that LINDI can produce Zarr v3 files that NWBZarrIO reads natively, eliminating the h5py shim.

Key finding: object_id / source_object_id are HDMF attributes on groups/datasets — they are never used for reference resolution (only source + path are used). They are dropped from the unified ref format.

Cross-reference: NeurodataWithoutBorders/lindi — a corresponding issue will be filed there.

Reference implementation: zindi implements this convention for generating Zarr v3 reference file systems from HDF5 files.

---

Side-by-Side Comparison

Feature	hdmf-zarr (current)	LINDI (current)
Attribute prefix	zarr_ (zarr_dtype, zarr_link)	_ prefix (_SCALAR, _COMPOUND_DTYPE, _REFERENCE, _SOFT_LINK)
Links	Parent-centric: zarr_link array on parent group with {"source", "path", "name", "object_id", "source_object_id"}	Child-centric: empty group with _SOFT_LINK = {"path": "..."} attribute
Refs in attrs	{"zarr_dtype": "object", "value": {"source", "path", "object_id", "source_object_id"}}	{"_REFERENCE": {"source", "path", "object_id", "source_object_id"}}
Ref resolution	Only uses source + path (object_id/source_object_id written but never read)	Same — only source + path used
Refs in datasets	StringDType array, zarr_dtype = "object", each element is JSON string of ref dict	Object array with numcodecs.JSON(), refs are {"_REFERENCE": {...}} dicts
Compound types	zarr_dtype = [{"name": "x", "dtype": "uint32"}, ...], structured numpy arrays	_COMPOUND_DTYPE = [["x", "uint32"], ...], JSON-encoded object arrays
Scalars	zarr_dtype = "scalar", shape=(1,) StringDType	_SCALAR = True, shape=(1,), native dtype preserved
Strings	StringDType() (Zarr v3 native)	dtype='object' + numcodecs.JSON() (Zarr v2 pattern)
Region refs	Not supported	Not supported (placeholder comment)
External array links	Not supported	_EXTERNAL_ARRAY_LINK = {"link_type", "url", "name"}
NaN/Inf in attrs	Not handled	Encoded as "NaN", "Infinity", "-Infinity" strings

---

Unified Convention Design

Naming Convention

Decision: Use LINDI-style _ALL_CAPS attribute names (e.g., _SCALAR, _LINKS, _REFERENCE).

Rationale:

• HDF5-agnostic — these are generic data modeling concepts (links, references, scalars) useful regardless of HDF5
• Already established in LINDI (_SCALAR, _COMPOUND_DTYPE, _REFERENCE, _EXTERNAL_ARRAY_LINK)
• The _UPPER_CASE pattern is distinctive enough to avoid collisions with user attributes in practice
• hdmf-zarr adopts LINDI's naming, minimizing LINDI's migration burden

Reserved Attributes (Unified)

Attribute	On	Replaces	Value
_LINKS	Groups	zarr_link / _SOFT_LINK	List of {"name", "source", "path"} dicts
_DTYPE	Datasets	zarr_dtype (when string)	"object_reference" for ref datasets
_SCALAR	Datasets	zarr_dtype = "scalar" / _SCALAR	true
_REFERENCE	Attributes	{"zarr_dtype": "object", ...} / {"_REFERENCE": ...}	Self-describing ref wrapper with just {"source", "path"}
_EXTERNAL_ARRAY_LINK	Datasets	_EXTERNAL_ARRAY_LINK	{"link_type", "url", "name"} (unchanged from LINDI)

| _REFERENCE_FIELDS | Datasets | (new) | List of field names containing object references in compound datasets |

Note: _COMPOUND_DTYPE is not needed. Zarr v3's native structured data_type carries full field information (names and types). See section 4 below.

Feature-by-Feature Specification

1. Links

Adopt: hdmf-zarr's parent-centric model (simplified).

Parent group attribute _LINKS:

"_LINKS": [
    {"name": "device", "source": ".", "path": "/general/devices/array"}
]

• source = "." for internal links, relative path for external links
• Only name, source, path — no object_id fields (those are just regular attributes on the target object, readable after resolution)

Why parent-centric: No phantom groups, cleaner enumeration, naturally supports external links. LINDI's child-centric approach creates empty groups that aren't real objects.

LINDI migration: Replace _SOFT_LINK child groups with _LINKS entries on parent. Change LindiH5pyGroup.__getitem__ to check parent's _LINKS before Zarr children.

2. Object References in Attributes

Adopt: LINDI's self-describing wrapper (simplified).

"table": {
    "_REFERENCE": {
        "path": "/general/extracellular_ephys/electrodes",
        "source": "."
    }
}

Only source and path are needed. object_id is just a regular attribute on the target — the reader can fetch it after resolving the reference.

Why: Attributes are heterogeneous — you can't put a type marker on one attribute within a dict of attributes. The _REFERENCE wrapper key makes references self-identifying without needing a separate type marker.

hdmf-zarr migration: Change {"zarr_dtype": "object", "value": {...}} to {"_REFERENCE": {...}}. Update __read_attrs detection from v["zarr_dtype"] == "object" to "_REFERENCE" in v. Drop object_id/source_object_id from ref dicts.

3. Object References in Datasets

Adopt: StringDType array with _DTYPE = "object_reference" marker.

• Dataset attribute: _DTYPE = "object_reference"
• Data: StringDType array where each element is a plain target path string:

  "/acquisition/timeseries"
  

Since _DTYPE = "object_reference" already identifies every element as a reference, there is no need to wrap each value in a {"_REFERENCE": ...} dict. The source defaults to "." (same file). For future cross-file references, the format can be extended to store {"source": "other.nwb", "path": "/target"} dicts instead of plain strings.

Why "object_reference" over "object": More explicit and self-documenting. "object" is ambiguous — in numpy/Python it means "any Python object". "object_reference" precisely describes HDF5 object references. This is a new convention that both hdmf-zarr and LINDI will adopt.

Why plain path strings: Since the dataset-level _DTYPE marker already communicates that these are references, each element only needs to carry the target path. This is simpler and more compact than JSON-encoded dicts. Same-file references (source=".") are by far the most common case.

LINDI migration: Switch from object arrays to StringDType arrays for reference datasets.

4. Compound Data Types

Use zarr v3's native structured data_type directly — no _COMPOUND_DTYPE attribute needed.

Zarr v3 metadata data_type:

"data_type": {
    "name": "structured",
    "configuration": {
        "fields": [
            ["x", "uint32"],
            ["y", "float64"],
            ["label", {"name": "null_terminated_bytes", "configuration": {"length_bytes": 10}}]
        ]
    }
}

Data storage: Raw numpy structured array bytes (headless numpy arrays). The HDF5 compound data bytes are directly compatible with zarr v3's structured data_type — no data transformation needed. Byte-range references to HDF5 chunks work for large compound datasets, while small ones are inlined. Fixed-length byte string fields (S10, etc.) are represented as null_terminated_bytes in the zarr v3 structured fields.

Why no _COMPOUND_DTYPE: Zarr v3's structured data_type already carries full field information (names and types) natively. A separate attribute is redundant. Readers detect compound datasets via zarr_obj.dtype.names is not None.

Why structured data_type: zarr-python natively supports this (via UnstableSpecificationWarning). It stores the raw byte layout of numpy structured arrays, enabling direct byte-range reads from HDF5 compound chunks without data transformation. This is much more efficient than JSON-encoding each row.

Compound datasets with reference fields: Reference fields contain opaque HDF5 handles that must be resolved during generation — these compounds are always inlined. Reference fields are resolved to target path strings and stored as fixed-length Unicode (U{N}) fields in the structured array (mapped to fixed_length_utf32 in zarr v3). A _REFERENCE_FIELDS attribute lists which field names contain references:

"_REFERENCE_FIELDS": ["electrode", "group"]

This tells the reader which fields to interpret as object reference paths rather than regular strings.

LINDI migration: This is the biggest change for LINDI. Currently uses dtype='object' + numcodecs.JSON() (Zarr v2 only — object_codec doesn't exist in v3). Must switch to structured numpy arrays with zarr v3's structured data_type.

5. Scalar Datasets

Adopt: LINDI's boolean marker + native dtype preservation.

Dataset attribute: _SCALAR = true
Storage: shape=(1,), dtype matches original data (not always StringDType).

Why boolean marker: Cleaner than overloading zarr_dtype = "scalar". A scalar integer should remain an integer array, not a string.

hdmf-zarr migration: Change zarr_dtype = "scalar" to _SCALAR = true. In __scalar_fill__, preserve numeric dtypes instead of always using StringDType.

6. String Handling

Adopt: hdmf-zarr's Zarr v3 approach.

• Variable-length strings: StringDType() arrays
• Bytes: decoded to UTF-8
• Compound type string fields: fixed-length byte strings (S{N}) mapped to null_terminated_bytes in zarr v3

LINDI migration: Replace dtype='object' + numcodecs.JSON() with StringDType() throughout. This is required for Zarr v3 anyway.

7. NaN/Inf in Attributes

Adopt: LINDI's encoding.

Attributes are JSON-serialized, and JSON doesn't support NaN/Inf. Encode as "NaN", "Infinity", "-Infinity" strings.

hdmf-zarr migration: Add NaN/Inf encoding/decoding in write_attributes and __read_attrs.

8. External Array Links (Optional)

Adopt: LINDI's feature (unchanged).

Dataset attribute _EXTERNAL_ARRAY_LINK:

{
    "link_type": "hdf5_dataset",
    "url": "https://...",
    "name": "/path/in/hdf5"
}

hdmf-zarr should recognize this attribute on read (or raise a clear error), even if it doesn't write it.

9. Region References (Future)

Reserve _REGION_REFERENCE for future use. Neither library implements this yet. Proposed format:

{"_REGION_REFERENCE": {"path": "/target", "source": ".", "region": {"start": [0, 0], "count": [10, 20]}}}

---

Shared Package

Create a small Python package (working name TBD — could be zarr-data-conventions or similar) containing:

1. Specification document — formal definition of the convention
2. Constants — attribute name strings (LINKS, DTYPE, SCALAR, COMPOUND_DTYPE, REFERENCE, etc.)
3. Helper functions — create/parse refs, links, compound dtype descriptors, NaN/Inf encoding
4. Validation — check whether a Zarr store conforms to the convention
5. Convention version — root attribute for format detection

Dependencies: only numpy + zarr. Both hdmf-zarr and LINDI add this as a dependency.

---

Migration Plan

Phase 1: Create shared package

• Define constants and helpers
• Write specification document
• Publish on PyPI as v0.1.0

Phase 2: Migrate hdmf-zarr (on zarr-v3-migration branch)

Key file: src/hdmf_zarr/backend.py

• Replace zarr_link → _LINKS (lines 1022–1028, 1770–1783)
• Replace zarr_dtype → _DTYPE / _SCALAR (lines 1664–1665, 1790–1798, 1818–1833); drop _COMPOUND_DTYPE in favor of zarr v3 native structured data_type
• Replace {"zarr_dtype": "object", "value": ...} → {"_REFERENCE": ...} in attrs (lines 816, 1850–1858)
• Simplify ZarrReference / _create_ref to only include source + path (drop object_id, source_object_id)
• Update __reserve_attribute tuple (line 1516)
• Add NaN/Inf encoding in write_attributes / __read_attrs
• Preserve native dtypes for scalars in __scalar_fill__
• Update __is_ref to check for "object_reference" instead of "object"

Key file: src/hdmf_zarr/zarr_utils.py

• Update reference parsing in _get_ref, BuilderZarrReferenceDataset, BuilderZarrTableDataset

Key file: docs/source/storage.rst

• Update all attribute names and examples

Add backward compat: on read, check for BOTH old (zarr_dtype, zarr_link) and new names with deprecation warning.

Phase 3: Migrate LINDI

• Upgrade to Zarr v3
• _SCALAR, _REFERENCE, _EXTERNAL_ARRAY_LINK names stay the same
• Drop _COMPOUND_DTYPE — use zarr v3 native structured data_type instead
• Add _REFERENCE_FIELDS for compound datasets with reference fields
• Replace child-centric _SOFT_LINK with parent-centric _LINKS
• Add _DTYPE attribute for dataset type annotation
• Switch from numcodecs.JSON() object arrays to StringDType / structured arrays
• Change "<REFERENCE>" / "object" to "object_reference" for reference fields
• Drop object_id / source_object_id from _REFERENCE dicts
• Key files: LindiH5ZarrStore.py, LindiH5pyGroup.py, LindiH5pyDataset.py, LindiH5pyAttributes.py, create_zarr_dataset_from_h5_data.py, h5_ref_to_zarr_attr.py

Phase 4: Integration testing

• Write with hdmf-zarr, read with LINDI (and vice versa)
• Test all data types: links, references, compounds, scalars, strings
• Test edge cases: refs in compound types, NaN attributes, external links

---

Key Design Decisions Summary

Decision	Chose	Over	Rationale
Attribute naming	LINDI-style _ALL_CAPS	zarr_ / _h5z_ prefix	HDF5-agnostic, already established in LINDI
Link model	Parent-centric (_LINKS on parent)	Child-centric (_SOFT_LINK on child)	No phantom groups, supports external links
Ref format	Just {source, path}	Include object_id, source_object_id	object_id is a regular attr, readable after resolution
Ref in attrs	Self-describing _REFERENCE wrapper	zarr_dtype + value	Attrs are heterogeneous, need self-ID
Ref in datasets	Plain path strings + _DTYPE = "object_reference"	JSON dicts with _DTYPE = "object"	More explicit naming; simpler storage since _DTYPE already marks them
Compound storage	Zarr v3 native structured data_type (no _COMPOUND_DTYPE attr)	JSON-encoded object arrays / separate _COMPOUND_DTYPE attr	Zarr v3 carries field info natively; no redundant attribute needed
Compound ref fields	_REFERENCE_FIELDS attr + resolved path strings in U{N} fields	_COMPOUND_DTYPE with "object" dtype	Only adds metadata when reference fields exist; paths stored as plain strings
Scalar marker	_SCALAR = true	zarr_dtype = "scalar"	Clean separation, preserves native dtype
Strings	StringDType	Object + JSON codec	Zarr v3 native

[bendichter]

Cross-reference: NeurodataWithoutBorders/lindi#125