Skip to content

fix: read mixed regular/rectilinear chunk grids written by zarr 3.2.x - #4375

Draft
d-v-b wants to merge 27 commits into
zarr-developers:mainfrom
d-v-b:fix/mixed-regular-chunk-grid-4374
Draft

d-v-b wants to merge 27 commits into
zarr-developers:mainfrom
d-v-b:fix/mixed-regular-chunk-grid-4374

Conversation

@d-v-b

@d-v-b d-v-b commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

🤖 AI text below 🤖

Closes #4374.

Note

Depends on #4334, which is merged into this branch, so until #4334 lands this diff includes its changes. Both PRs read invalid regular chunk grids that older releases wrote, so the two readers now live in one function (see "One reader for stored regular grids" below). Review #4334 first; after it merges, this diff shrinks to this PR's own changes.

Problem

With array.rectilinear_chunks enabled, zarr 3.2.0 and 3.2.1 classified a mixed chunk spec such as chunks=(2, (5, 10, 5)) as regular, because _is_rectilinear_chunks only looked at the first element. They stored it as

{"name": "regular", "configuration": {"chunk_shape": [2, [5, 10, 5]]}}

while laying the chunks out as a rectilinear grid. The classifier was fixed in #4218, but the read side still has a hole: RegularChunkGridMetadata's validator delegates to the rectilinear validator, so it accepts the nested entry and then casts the result to tuple[int, ...]. The bad metadata gets through parsing and fails later in create_codec_pipeline with TypeError: Expected an iterable of integers.

Changes

  • RegularChunkGridMetadata now rejects non-integer chunk edge lengths with a TypeError that names the dimension.
  • A stored "regular" grid whose chunk_shape contains edge lists is read as the rectilinear grid it describes. It emits a ZarrUserWarning that says what is wrong with the metadata and how to re-save it (array.update_attributes({}), the same hint fix(chunk-grids): one invariant for zero-length axes across model, clamps, and metadata #4334 uses).
  • If rectilinear chunks are disabled, reading such an array raises a ValueError that explains what happened and how to enable them, instead of the unrelated TypeError. Neither message names the releases that wrote such metadata; that history is in the changelog fragment.

One reader for stored regular grids

#4334 reads a different invalid regular grid: a chunk size of 0, which releases before 3.4 wrote for arrays created with a zero-length axis. Both are compatibility readings of a stored regular grid, so they share one function, _read_stored_regular_chunk_grid, called from ArrayV3Metadata.__init__ (the zero-size reading needs the array shape, which chunk grid metadata does not carry). It sends a chunk_shape containing edge lists to the 3.2.x mixed-grid reader, and an all-integer one to #4334's parse_stored_regular_chunk_shape. parse_chunk_grid accepts only what the spec allows, and both warnings end with the same re-save hint, RESAVE_METADATA_HINT. No release wrote a mixed grid with a 0 in it: 3.2.0 and 3.2.1 reject -1, False and 0 inside a mixed spec, as measured with real installs. So the two readings never apply to the same document.

Integer vs sequence, not exact types

The chunk_shapes parsers decided what a dimension was by exact type, which rejected valid input: RectilinearChunkGridMetadata.from_dict accepted a dimension only as int or list, expand_rle accepted an RLE pair only as a list, and the reader above special-cased tuple to convert it before delegating. A metadata document built in Python holds tuples where parsed JSON holds lists, and a numpy array means the same thing as either.

One structural predicate, declares_chunk_edges, now answers "is this a sequence of edge lengths rather than a single integer size" — any non-integer iterable except str/bytes — and the four sites that make that decision use it: the stored-grid reader's detection of a mixed grid, the mixed-grid reader, from_dict, and expand_rle. It is a TypeGuard rather than a TypeIs because it is False for strings, which are iterable, so negative narrowing would be unsound; making it a guard removed three type: ignore comments. The reader's tuple special case is gone.

Consequences worth calling out:

  • Integers are int | np.integer throughout, matching _parse_chunk_shape, and _validate_chunk_shapes coerces edges, so building a rectilinear grid from numpy values works as it already did for a regular one. Without this, from_dict would accept np.int64(5) as a bare dimension while rejecting it inside an edge list.
  • The from_dict error message changed from "expected int or list" to "expected an integer or a sequence of chunk edge lengths".

Verification

  • Stores written by real zarr==3.2.0 and zarr==3.2.1 installs, with fix(chunk-grids): one invariant for zero-length axes across model, clamps, and metadata #4334 merged in: a mixed grid, a mixed grid resized to 0 along the regular axis, a mixed grid with an empty rectilinear axis, and a rectilinear grid on an empty axis. All open, take an append without losing data, and re-save to metadata that reopens without a warning. Full suite with fix(chunk-grids): one invariant for zero-length axes across model, clamps, and metadata #4334 merged in: 11807 passed.
  • New tests in tests/test_metadata/test_v3.py: the read path is parametrized over where the edge lists sit; there is one test per error case (flag disabled, direct construction of a regular grid with edges); and an end-to-end open, read, and re-save test.
  • For the parsing change I extended the existing tables in tests/test_unified_chunk_grid.py (test_rle_expand, test_rle_expand_rejects_invalid, test_rectilinear_from_dict) rather than adding parallel ones, plus one test for the new from_dict error branch and one for direct construction from numpy values. Each new case was confirmed to fail against the pre-change source.

🤖 Generated with Claude Code

d-v-b added 10 commits September 9, 2026 20:28
…, clamps and metadata

Invariant: a chunk edge length is always >= 1; a dimension's extent may be 0,
in which case the dimension has zero chunks (ceildiv(0, size) == 0).

Zero-length-axis bugs have recurred since 2017 (#150, #241, #303, zarr-developers#972,
zarr-developers#1977, zarr-developers#2434, zarr-developers#3711, zarr-developers#4305, zarr-developers#4307, zarr-developers#4328) because the layers disagreed on
this invariant and every span-derived chunk spelling clamped on its own:

- The metadata layer (common.py, metadata/v3.py) required chunk edges >= 1,
  but the in-memory FixedDimension allowed size == 0 with four special-case
  branches left over from zarr-developers#2434, so normalization could build a grid the
  metadata constructor then rejected. FixedDimension now rejects size < 1
  and the four `if self.size == 0` branches are gone. VaryingDimension
  already required edges > 0 and is unchanged.
- `chunks=-1`, `chunks=False`, `chunks="auto"` (_guess_regular_chunks, both
  the typesize == 0 early return and the np.maximum line) and `shards="auto"`
  each derived "one chunk covering the axis" independently. They now all go
  through one helper, `_full_span_chunk_size(span) = max(span, 1)`, which is
  the single definition of that phrase for a possibly zero-length axis.
- Zarr format 2 metadata had no chunk >= 1 check, so a legacy `chunks: [0]`
  document opened fine and read uninitialised memory after a resize. It now
  raises a clear ValueError at parse time, matching the format 3 grid.
- Rectilinear grids had no creation-time spelling for a zero-length axis:
  normalize_chunks_1d required sum(edges) == span, which no list of positive
  edges can satisfy for span 0, even though the same state is reachable via
  resize((0,)) and round-trips through reopen. For span == 0 any non-empty
  list of positive edges is now accepted verbatim, producing the same
  VaryingDimension(edges, extent=0) that resize produces; the strict sum
  check is kept for span > 0.

Tests: the per-spelling regression test from zarr-developers#4328 is replaced by one matrix
over {-1, False, "auto", 1, (1,...), [[2, 2]]} x {(0,), (0, 4), (4, 0),
(0, 0), ()} x {v2, v3} x {no shards, shards="auto" with and without a byte
budget, explicit shards}, with separate small tests for each error case.
Tests that constructed FixedDimension(size=0) now assert it raises, and a
zero-extent test covers the behaviour the old special cases were guarding.

Assisted-by: ClaudeCode:claude-fable-5-1
zarr-python 2.18.7 writes `chunks: [0]` for `zarr.zeros((0,), chunks=False)`
and for `chunks=(0,)`, so stores with that document exist. Rejecting them
at open would turn a previously-readable array into an error; leaving the
0 in place read uninitialised memory after a resize. Normalize the edge to
1 with a ZarrUserWarning instead — the same grid every other "one chunk
spans the axis" spelling produces — and keep rejecting a zero edge on an
axis that has data.

Assisted-by: ClaudeCode:claude-fable-5-1
Assisted-by: ClaudeCode:claude-fable-5-1
Measured against zarr 2.18.7: `zeros((0,), chunks=False)`, `chunks=-1` and
`chunks=(0,)` all write `chunks: [0]`, after which nchunks, read, write,
append, resize and reopen-then-read every raise ZeroDivisionError. There
was never a working behaviour to preserve; normalizing the edge to 1 makes
such arrays usable for the first time. Say so in the comment and fragment
instead of claiming the stores were previously readable.

Assisted-by: ClaudeCode:claude-fable-5-1
zarr 3.2.0 and 3.2.1 classified mixed chunk specs like (2, (5, 10, 5)) as
regular and stored them as a "regular" grid whose chunk_shape contains an
edge list, while laying the chunks out as a rectilinear grid. Newer versions
accepted that metadata and failed later with an unrelated TypeError.

RegularChunkGridMetadata now rejects edge lists. When reading stored
metadata, a "regular" grid with edge lists is read as the rectilinear grid
it describes, with a warning explaining how to re-save it; if rectilinear
chunks are disabled, the error says what happened and how to enable them.

Closes zarr-developers#4374

Assisted-by: ClaudeCode:claude-opus-5
Assisted-by: ClaudeCode:claude-opus-5
@read-the-docs-community

read-the-docs-community Bot commented Sep 18, 2026 •

Copy link
Copy Markdown

Documentation build overview

📚 zarr-metadata | 🛠️ Build #34756224 | 📁 Comparing e324baa against latest (1187a43)

  🔍 Preview build  

No files changed.

@read-the-docs-community

read-the-docs-community Bot commented Sep 18, 2026 •

Copy link
Copy Markdown

Documentation build overview

📚 zarr-indexing | 🛠️ Build #34756223 | 📁 Comparing e324baa against latest (1187a43)

  🔍 Preview build  

No files changed.

@codecov

codecov Bot commented Sep 18, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.74468% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.42%. Comparing base (f58644a) to head (09639f4).

Files with missing lines Patch % Lines
src/zarr/core/metadata/v3.py 93.33% 3 Missing ⚠️
src/zarr/core/chunk_grids.py 90.90% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4375      +/-   ##
==========================================
+ Coverage   94.37%   94.42%   +0.04%     
==========================================
  Files          93       93              
  Lines       13171    13217      +46     
==========================================
+ Hits        12430    12480      +50     
+ Misses        741      737       -4     
Files with missing lines Coverage Δ
src/zarr/core/common.py 91.42% <100.00%> (+0.95%) ⬆️
src/zarr/core/metadata/common.py 100.00% <100.00%> (ø)
src/zarr/core/metadata/v2.py 90.17% <100.00%> (+0.78%) ⬆️
src/zarr/core/chunk_grids.py 96.73% <90.90%> (-0.06%) ⬇️
src/zarr/core/metadata/v3.py 96.07% <93.33%> (+0.63%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…n the 3.2.x shim

Review follow-ups for zarr-developers#4375:

- RegularChunkGridMetadata now accepts numpy integer scalars and stores
  Python ints, mirroring parse_shapelike. The strict integer check had
  reported np.int64(2) as if it were a list of chunk edges.
- The mixed "regular" grid reader converts tuple entries to lists before
  delegating to RectilinearChunkGridMetadata.from_dict, so metadata dicts
  built in Python are handled the same as parsed JSON.
- The error and warning share one message prefix.
- Tests cover numpy ints, run-length encoded edges, and tuple input, and
  the test section header says the reader is broader than the 3.2.x bug.

Assisted-by: ClaudeCode:claude-fable-5-1
…k shapes

The chunk_shapes parsers checked exact types: `from_dict` accepted a
dimension only as `int` or `list`, `expand_rle` accepted an RLE pair only
as a `list`, and the reader for 3.2.x mixed grids special-cased `tuple` to
convert it to a list before handing it to `from_dict`. A metadata dict
built in Python holds tuples where parsed JSON holds lists, and a numpy
array is as good as either, so these checks rejected valid input.

One structural predicate, `declares_chunk_edges`, now answers "is this a
sequence of edges rather than a single integer size" for all of them:
any non-integer iterable except `str`/`bytes`. It is a `TypeGuard`, not a
`TypeIs`, because it is False for strings, which are iterable. The four
sites that make that decision use it: `parse_chunk_grid`'s detection of a
mixed grid, the mixed-grid reader, `RectilinearChunkGridMetadata.from_dict`,
and `expand_rle`. The shim no longer needs its tuple special case.

Integers are `int | np.integer` throughout, matching `_parse_chunk_shape`,
and every parser stores Python ints: `_validate_chunk_shapes` now
coerces, so constructing a rectilinear grid from numpy values works the
way it already did for a regular grid.

Assisted-by: ClaudeCode:claude-opus-5
Parsing a regular chunk shape repeated its work at two levels:

- `_parse_chunk_shape` type-checked and coerced every dimension, then
  handed the result to `_validate_chunk_shapes`, which re-ran the same
  isinstance test and coercion and added only the `>= 1` check. Its
  edge-list branch was unreachable from this caller, which is why a
  `cast` was needed on the way out.
- `RegularChunkGridMetadata.from_dict` parsed the chunk shape and passed
  it to the constructor, whose `__post_init__` parsed it again.

Together that was four passes over the dimensions for `from_dict`. It
is now one: `_parse_chunk_shape` checks the range itself and no longer
calls the rectilinear validator, and `from_dict` hands the dimensions to
the constructor unparsed. Beyond the redundancy, the shared validator
was the coupling that let a rectilinear chunk shape be stored as a
regular grid (zarr-developers#4374), so the two grid kinds now validate separately.

`RectilinearChunkGridMetadata.from_dict` also had its own `>= 1` check
for bare-int dimensions, duplicating `_validate_chunk_shapes`, which
`__post_init__` runs over the result anyway. It now only puts the JSON
into shape (integer vs sequence, RLE expansion), and a bad bare int is
reported by the validator, which names the dimension. `expand_rle` keeps
its own checks because it is called directly.

Assisted-by: ClaudeCode:claude-opus-5
… format 3

The compatibility policy for a stored chunk size of 0 on a zero-length
axis covered Zarr format 2 only, so arrays written by zarr-python 3.0 and
3.1 with `chunk_shape: [0]` — or `[false]`, which 3.0 wrote for
`chunks=False` — still could not be opened at all.

`ArrayV3Metadata` now applies the same policy to a regular chunk grid: a
stored chunk size of 0 on a zero-length axis is read as 1 with a
`ZarrUserWarning`, and a zero chunk size on a positive-length axis is
left for the chunk grid parser to reject. It runs in `__init__` rather
than in the grid parser because the policy needs the array shape, which
chunk grid metadata does not carry.

Both warnings now say how to store a corrected chunk size — open the
array writable and call `array.update_attributes({})`, which rewrites the
whole document from the parsed metadata — from one shared constant. The
Zarr format 2 warning also named only zarr-python 2.x; measured against
real installs, every 3.x release before 3.4 wrote a zero chunk size for
an empty array too (3.3.0 for `chunks=-1` and `chunks=False`).

Tested against stores written by zarr 3.0.10, 3.1.6 and 3.3.0: they open,
append without losing data, and re-save to a chunk size that reopens
without a warning.

Assisted-by: ClaudeCode:claude-opus-5
d-v-b and others added 8 commits September 19, 2026 18:05
… in one routine

The legacy zero-chunk policy was written out twice: inline in
`ArrayV2Metadata.__init__`, and again in a Zarr format 3 helper. The V2
copy also zipped with `strict=False` and re-appended any trailing chunk
entries only so that a separate length check, `parse_metadata`, could
report a dimensionality mismatch after construction.

`parse_stored_chunk_shape` in `zarr.core.metadata.common` is now the one
place a stored chunk shape is checked against its array's shape, for both
formats: one entry per axis, every integer chunk size at least 1, and a
size of 0 (or JSON `false`) on a zero-length axis read as 1 with a
warning that names the writer and how to re-save. Non-integer entries,
such as edge lists, pass through for the caller's own parser.

`ArrayV2Metadata.__init__` calls it directly and `parse_metadata` is
gone. The Zarr format 3 adapter only locates a regular grid's
`chunk_shape` in the stored document and hands it over; it still runs in
`ArrayV3Metadata.__init__` because chunk grid metadata has no array shape.
The Zarr format 3 import changes that existed only for the old helper are
reverted.

Tests for the policy now target the routine: one table of valid and
legacy inputs, and one test per rejection (dimension mismatch, zero on a
non-empty axis, negative). They replace metadata-level tests in
test_v2.py and test_v3.py that only re-tested the same rules; the
end-to-end tests still cover both formats' wiring against stored arrays.

Assisted-by: ClaudeCode:claude-opus-5
…nk grids

`parse_stored_chunk_shape` passed non-integer entries through "for the
caller's own parser", which made a regular-grid policy look like a
general chunk shape routine and let it decide what a 0-length chunk means
for grids it does not own. A rectilinear grid, or any other grid, is free
to define its own semantics for 0-length chunks.

It is now `parse_stored_regular_chunk_shape`, typed `Sequence[int]`, with
no pass-through, and its docstring says it applies to Zarr format 2
`chunks` and Zarr format 3 `regular` grids only. The Zarr format 3 caller
hands it a chunk shape only when the grid is named `regular` and every
entry is an integer (`_is_regular_chunk_shape`); anything else is not a
regular chunk shape and goes to the chunk grid parser untouched.

Assisted-by: ClaudeCode:claude-opus-5
… into fix/mixed-regular-chunk-grid-4374

# Conflicts:
#	src/zarr/core/metadata/v3.py
…place

zarr-developers#4334 read a zero chunk size on an empty axis in `ArrayV3Metadata.__init__`
and zarr-developers#4375 read a 3.2.x mixed grid inside `parse_chunk_grid`, each with its
own predicate, warning text and re-save advice. Both are compatibility
readings of a stored `regular` grid, so `_read_stored_regular_chunk_grid`
now dispatches to both, `parse_chunk_grid` accepts only what the spec
allows, and both warnings use `RESAVE_METADATA_HINT`.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…panning chunk

A stored chunk size of 0 was tolerated only on a zero-length axis and
rejected otherwise, because the metadata supposedly could not say how the
stored chunks were laid out. But a chunk size of 0 gives a grid of zero
chunks, so no release could store a chunk under it, and the writers of
that metadata let the axis grow: 3.4.0 appends to a Zarr format 2 array
created empty by 3.3.0 (shape grows, no chunk written), and 3.1.6 records
a Zarr format 3 resize and the resize half of a failed append. Measured
with real installs. Those arrays open in 3.4.0, attributes included; the
rejection would have made them unopenable.

`parse_stored_regular_chunk_shape` now reads a stored 0 (or JSON `false`)
on any axis as one chunk spanning it, `max(extent, 1)`, which is what the
`-1`/`False` spec that wrote it meant. On a grown axis the warning also
says that data written to it was not saved. Negative sizes are still
rejected.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The only state machine that touched arrays compared zarr on one store with
zarr on a MemoryStore, so a chunk grid bug showed up identically on both
sides; it had no append rule, covered Zarr format 3 only, and kept every
empty axis at 0 when resizing, which is where the zero-length bugs live.

`ArrayLifecycle` checks one array against a NumPy model across both
formats, every chunk spelling (-1, False, "auto", ints, sharded,
rectilinear) and the stored chunk size of 0 that releases before 3.4
wrote, including on an axis those releases grew. Rules append, resize
(growing and shrinking to and from 0), write and re-save the metadata;
the invariant reopens the array and compares shape, values and whether
the legacy warning is due. Deliberately breaking the grown-axis policy,
the legacy warning, or append on an empty axis each fails it.

`resize` keeps partly retained chunks whole, so cells cut off by a shrink
can come back with their old values when the axis grows (as in 2.x); the
model marks such cells unknown until written instead of encoding that.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… into fix/mixed-regular-chunk-grid-4374

# Conflicts:
#	src/zarr/core/metadata/v3.py
d-v-b and others added 3 commits September 25, 2026 15:26
The warning for a stored chunk size of 0 said which zarr-python releases
wrote it. The check runs on every metadata construction, so metadata built
in code, such as VirtualiZarr's kerchunk writer passing an empty array's
shape as `chunks`, was told it came from zarr-python 2.x. The warning now
says what the chunk size is read as and, on an axis of positive length,
that the axis holds only the fill value. `legacy_writers` is gone, and
the docstrings no longer narrate release history; the changelog keeps it.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… into fix/mixed-regular-chunk-grid-4374

# Conflicts:
#	src/zarr/core/metadata/v2.py
#	src/zarr/core/metadata/v3.py
The warning and error for a stored `regular` grid that lists chunk edges
said which zarr releases wrote it. They now say what is wrong with the
metadata, that it lists chunk edges only a rectilinear grid can declare,
and how it is read. The reader's docstrings no longer narrate release
history either; the changelog fragment keeps it.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Mixed regular/rectilinear chunk grids written by 3.2.x are unreadable by 3.3+

1 participant