feat(zarr-metadata)!: a shard is judged against the chunk it is handed, and its pipelines are read

[d-v-b]

🤖 AI text below 🤖

The last of four pieces of layer 4, depending on #4442 (4c, the codec pipeline): the sharding codec is judged against the shard it is handed, and its two pipelines are read as the array's is. The first fourteen commits are #4440's, #4441's and #4442's; this PR's are the last three.

document["shape"] = [16, 16]
document["chunk_grid"] = {"name": "regular", "configuration": {"chunk_shape": [8, 6]}}
document["codecs"] = [{"name": "sharding_indexed", "configuration": {
    "chunk_shape": [4, 4],
    "codecs": [{"name": "bytes", "configuration": {"endian": "little"}}],
    "index_codecs": ["bytes"],
}}]
validate_array_metadata_v3(document)
# codecs.0.configuration.chunk_shape.1: expected an inner chunk length that divides each length the shards take along this axis, [6], got 4
# codecs.0.configuration.index_codecs.0.configuration.endian: expected an endian, since each 'uint64' value takes several bytes

The shard. The sharding codec's chunk_shape has an inner chunk length for each axis of the shard it is handed, and each length divides every length the shards take along that axis. The shard is the chunk the codec is handed, after any codec before it. A transposed shard is divided along its transposed axes, as decided for layer 4. zarr-python reads divisibility from the untransposed grid instead.

The pipelines it holds. A codec definition gains one more hook, pipelines(configuration, nested, chunk). For each member of its configuration that holds a pipeline, it gives the chunk that pipeline's first codec is handed:

• the inner codecs are handed the inner chunks, of chunk_shape and of the shard's data type;
• the index codecs are handed the shard index: uint64, with the chunks per shard along each axis and a final axis of 2.

Each pipeline is read as the array's is: its order, then each codec against the chunk it is handed, with problems where they sit. A Stage keeps the inner stages as inner. An index bytes codec without an endian is now caught, and so is an inner transpose of the wrong rank. This also holds behind a codec nothing in scope claims, since the index is uint64 whatever the shard.

Like transition, pipelines is asked whatever the chunk rules found, and gives only what holds. A shard of another number of axes than its chunk_shape hands both pipelines lengths nothing is known of, since which of the two is wrong is not known. So it draws one problem, not three.

A problem inside is the inner field's own. This is the first commit. resolve used to read a field as invalid when anything inside it had a problem. So a shard holding a gzip with a level out of range, or a codec with must_understand: false, was left unread, and nothing else about it was judged. A document's fields are not read that way: a bad codec at one index leaves the others read. Now the fields a configuration holds are read the same way:

• a problem of one is its own, reported where it sits, with its own resolution in nested;
• the field holding it is read when its own configuration and rules are sound;
• its rules are still asked only when each field it holds is named, since a rule may read one by name.

Every hook already treats an unread nested field as unknown, so this adds no false problems. It changes what resolve gives for a container (layer 2, merged, unreleased); two tests encoded the old answer. Without it, the correctness review found problems missing from 553 of 3,000 generated documents with shards, although the verdicts were right. The same codec also drew different problems at the top level and inside a shard.

Also here. The codec functions no codec of a kind is asked are now one declared table, _UNASKED, instead of one if each. A bytes -> bytes codec is handed bytes, and only an array -> array codec hands on a chunk.

Breaking. A document whose shard does not divide, or whose shard pipelines do not fit what they are handed, now has a problem where it sits, where the package accepted it before. A field holding a field with a problem now resolves as read, where it resolved as invalid.

Choices worth a look

• The index's data type is a hand-built reading of the package's uint64 definition. Hooks get no scope, and the spec fixes the index type whatever the scope holds.
• Pipelines are keyed by the member of the configuration that holds them. No codec holds a pipeline deeper than that.
• Handed a chunk nothing is known of, a shard still hands on the inner chunks it declares, and an index of len(chunk_shape) + 1 axes. That is its own declaration, as cast_value names its data type.
• A shard's pipelines are judged only as a document's codecs are read. resolve or canonicalize of a shard field alone does not see that codecs: [] has no array -> bytes codec. That rule needs no chunk, so it could move into the shard's rules, with the walk skipping it for held pipelines. Left for a follow-up.
• A pipelines naming a member that holds data type fields nothing in scope claims cannot be told from one holding codecs nothing claims. It reads them as codecs nothing claims. Only a custom definition can do this.

Reviews. Two, with separate lenses.

The correctness review extended the 4c spec reference with the sharding rules:

• 120,000 generated documents, with every inner and index stage compared, gave 0 disagreements against a reference that mirrors the walk.
• All 17 planted bugs were caught, two only by the stage comparison, since no rule reads the index lengths.
• 33,000 fuzzed documents raised nothing.
• A strict reference found the problems hidden inside shards, which the first commit fixes. It also found a rank mismatch that drew three problems, now one.
• It found a divisibility message that listed every length: on a rectilinear axis of 20,000 edges it was 134,548 characters, and it now names the smallest length not divided.

The design review's changes are in:

• pipelines are keyed by member, not by path, so an untested path walk is gone;
• a member holding no codecs is a TypeError;
• the rank mismatch is fixed;
• the tests are split per case, and the nested shard is folded into the main table.

Where zarr-python differs. The spec is the authority. The first two are filed: the transpose case on #2050, and the shard inside a shard as #4437.

• zarr-python judges divisibility on the untransposed grid. It accepts [transpose(1,0), sharding_indexed chunk_shape [4,3]] on chunks of (4, 6), and a write and read returns different data. It refuses (6, 4), which its runtime handles.
• It never checks a shard held inside a shard. It accepts a shard [4,4] holding a shard [3,4], and a round trip returns different data.
• It accepts index_codecs of [crc32c], of [bytes, bytes], of bytes without an endian, or with a transpose of the wrong rank, and fails at the first write.

The stack. Layers 0 (#4420, #4421, #4422), 1 (#4432), 2 (#4434) and 3 (#4436) have merged. 4a is #4440, 4b is #4441, 4c is #4442, and this is 4d, which completes layer 4.

🤖 Generated with Claude Code

[read-the-docs-community]

Documentation build overview

📚 zarr-metadata | 🛠️ Build #34798189 | 📁 Comparing 074c7c9 against latest (0401a7f)

  🔍 Preview build  

7 files changed · ± 7 modified

± Modified

• index.html
• api/index.html
• api/model/index.html
• api/v3/chunk_grid/index.html
• api/v3/codec/index.html
• api/v3/data_type/index.html
• api/v3/definition/index.html

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.37%. Comparing base (0401a7f) to head (074c7c9).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4443   +/-   ##
=======================================
  Coverage   94.37%   94.37%           
=======================================
  Files          93       93           
  Lines       13174    13174           
=======================================
  Hits        12433    12433           
  Misses        741      741           

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.