Skip to content

fix(metadata): validate codec chains against the threaded chunk spec - #303

Open
d-v-b wants to merge 35 commits into
mainfrom
fix/codec-chain-validation
Open

d-v-b wants to merge 35 commits into
mainfrom
fix/codec-chain-validation

Conversation

@d-v-b

@d-v-b d-v-b commented Aug 20, 2026

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

ArrayV3Metadata incorrectly rejects codec chains when an earlier array-to-array codec changes chunk rank or shape. Evolution used the array-level shape, and later validation revisited the original shape instead of the resolved chunk spec.

The new evolve_and_validate_codecs helper threads a chunk ArraySpec through evolution and validation. Metadata no longer repeats the stale array-level validation. Sharding evolves and validates its inner chain with the real inner ArraySpec, including the actual fill value. Codec.validate's public signature stays unchanged; direct ShardingCodec.validate remains the geometry check.

Distinct rectilinear chunk shapes are checked, with Cartesian enumeration capped at 4,096. Above that cap validation warns and continues with a sample; it does not certify unexamined transformed shapes.

Tests cover reshape/transpose round trips, persisted metadata reopening, and invalid chains. A new Hypothesis property varies nonzero unsigned fill values across sharded and unsharded layouts; it caught validation using an invented zero fill value, which rejected a valid uint8 ScaleOffset(offset=1), fill_value=1 array.

Validation against current main: 1,191 codec/metadata/fused/parity tests passed, 17 optional/unsupported skips and 2 optional-backend xfails; all commit hooks including mypy passed. Misconfigured inner chains may now be rejected during construction rather than encoding.

d-v-b added a commit to d-v-b/zarr-metadata.js that referenced this pull request Aug 30, 2026
The same bug class d-v-b/zarr-python#303 fixes in zarr-python's
metadata validation existed here: transpose/sharding rules judged every
codec against the array-level dimensional facts, falsely rejecting the
spec-endorsed reshape->transpose chain and shards sized for the
transposed chunk, while missing shards sized for the un-transposed one.
The pipeline walk now threads context: a valid transpose permutes the
per-dimension chunk sizes, sharding recurses with its inner chunk rank
and spends the context at the array->bytes boundary, and any codec the
layer cannot reason about drops the context for the remaining codecs.

Assisted-by: ClaudeCode:claude-fable-5
d-v-b added a commit that referenced this pull request Sep 9, 2026
…, clamps and metadata

Invariant: a chunk edge length is always >= 1; a dimension's extent may be 0,
in which case the dimension has zero chunks (ceildiv(0, size) == 0).

Zero-length-axis bugs have recurred since 2017 (#150, #241, #303, zarr-developers#972,
zarr-developers#1977, zarr-developers#2434, zarr-developers#3711, zarr-developers#4305, zarr-developers#4307, zarr-developers#4328) because the layers disagreed on
this invariant and every span-derived chunk spelling clamped on its own:

- The metadata layer (common.py, metadata/v3.py) required chunk edges >= 1,
  but the in-memory FixedDimension allowed size == 0 with four special-case
  branches left over from zarr-developers#2434, so normalization could build a grid the
  metadata constructor then rejected. FixedDimension now rejects size < 1
  and the four `if self.size == 0` branches are gone. VaryingDimension
  already required edges > 0 and is unchanged.
- `chunks=-1`, `chunks=False`, `chunks="auto"` (_guess_regular_chunks, both
  the typesize == 0 early return and the np.maximum line) and `shards="auto"`
  each derived "one chunk covering the axis" independently. They now all go
  through one helper, `_full_span_chunk_size(span) = max(span, 1)`, which is
  the single definition of that phrase for a possibly zero-length axis.
- Zarr format 2 metadata had no chunk >= 1 check, so a legacy `chunks: [0]`
  document opened fine and read uninitialised memory after a resize. It now
  raises a clear ValueError at parse time, matching the format 3 grid.
- Rectilinear grids had no creation-time spelling for a zero-length axis:
  normalize_chunks_1d required sum(edges) == span, which no list of positive
  edges can satisfy for span 0, even though the same state is reachable via
  resize((0,)) and round-trips through reopen. For span == 0 any non-empty
  list of positive edges is now accepted verbatim, producing the same
  VaryingDimension(edges, extent=0) that resize produces; the strict sum
  check is kept for span > 0.

Tests: the per-spelling regression test from zarr-developers#4328 is replaced by one matrix
over {-1, False, "auto", 1, (1,...), [[2, 2]]} x {(0,), (0, 4), (4, 0),
(0, 0), ()} x {v2, v3} x {no shards, shards="auto" with and without a byte
budget, explicit shards}, with separate small tests for each error case.
Tests that constructed FixedDimension(size=0) now assert it raises, and a
zero-extent test covers the behaviour the old special cases were guarding.

Assisted-by: ClaudeCode:claude-fable-5-1
d-v-b added a commit that referenced this pull request Sep 9, 2026
…, clamps and metadata

Invariant: a chunk edge length is always >= 1; a dimension's extent may be 0,
in which case the dimension has zero chunks (ceildiv(0, size) == 0).

Zero-length-axis bugs have recurred since 2017 (#150, #241, #303, zarr-developers#972,
zarr-developers#1977, zarr-developers#2434, zarr-developers#3711, zarr-developers#4305, zarr-developers#4307, zarr-developers#4328) because the layers disagreed on
this invariant and every span-derived chunk spelling clamped on its own:

- The metadata layer (common.py, metadata/v3.py) required chunk edges >= 1,
  but the in-memory FixedDimension allowed size == 0 with four special-case
  branches left over from zarr-developers#2434, so normalization could build a grid the
  metadata constructor then rejected. FixedDimension now rejects size < 1
  and the four `if self.size == 0` branches are gone. VaryingDimension
  already required edges > 0 and is unchanged.
- `chunks=-1`, `chunks=False`, `chunks="auto"` (_guess_regular_chunks, both
  the typesize == 0 early return and the np.maximum line) and `shards="auto"`
  each derived "one chunk covering the axis" independently. They now all go
  through one helper, `_full_span_chunk_size(span) = max(span, 1)`, which is
  the single definition of that phrase for a possibly zero-length axis.
- Zarr format 2 metadata had no chunk >= 1 check, so a legacy `chunks: [0]`
  document opened fine and read uninitialised memory after a resize. It now
  raises a clear ValueError at parse time, matching the format 3 grid.
- Rectilinear grids had no creation-time spelling for a zero-length axis:
  normalize_chunks_1d required sum(edges) == span, which no list of positive
  edges can satisfy for span 0, even though the same state is reachable via
  resize((0,)) and round-trips through reopen. For span == 0 any non-empty
  list of positive edges is now accepted verbatim, producing the same
  VaryingDimension(edges, extent=0) that resize produces; the strict sum
  check is kept for span > 0.

Tests: the per-spelling regression test from zarr-developers#4328 is replaced by one matrix
over {-1, False, "auto", 1, (1,...), [[2, 2]]} x {(0,), (0, 4), (4, 0),
(0, 0), ()} x {v2, v3} x {no shards, shards="auto" with and without a byte
budget, explicit shards}, with separate small tests for each error case.
Tests that constructed FixedDimension(size=0) now assert it raises, and a
zero-extent test covers the behaviour the old special cases were guarding.

Assisted-by: ClaudeCode:claude-fable-5-1
d-v-b and others added 10 commits September 12, 2026 21:11
`ArrayV3Metadata` validated every codec against the array-level shape and
chunk grid, and threaded the *array* spec (not a chunk spec) through
`resolve_metadata` during evolution. Both wrongly reject chains in which an
earlier array->array codec changes a chunk's shape or rank, e.g. the
zarr-extensions `reshape` codec followed by `transpose` with an order of the
reshaped rank -- a combination the reshape spec explicitly endorses and that
the encode path already handles correctly.

Codecs are now evolved and validated in a single threaded pass
(`evolve_and_validate_codecs`): each codec sees the chunk spec produced by
the previous codec's `resolve_metadata`, exactly as at encode time. The
array-level shape/chunk grid are passed to `Codec.validate` unchanged until a
codec changes the chunk shape, after which the resolved chunk shape (and a
regular grid of it) stands in for them. `ShardingCodec.validate` now validates
its inner chain the same way against the inner chunk shape.

Assisted-by: ClaudeCode:claude-fable-5
…rectilinear chunk shape

Review feedback on the threaded-chunk-spec validation: after a codec changes
the chunk shape, validating the rest of the chain against a single
representative (max-edge) chunk shape is unsound for rectilinear grids --
an inner shard size that divides the largest chunk need not divide the
others. Concretely, transpose over a rectilinear grid followed by sharding
falsely accepted an inner chunk shape that only divided the largest
transposed chunk. The representative was also used to *detect* shape
changes, which could miss changes affecting only non-representative chunks.

`evolve_and_validate_codecs` now threads every distinct chunk shape of the
grid (the cross product of per-dimension distinct edges, capped at 4096 with
a ZarrUserWarning on truncation) through `resolve_metadata`, and validates
each one individually once any codec has changed a chunk shape. The
representative spec remains the single spec used for codec evolution and
dtype tracking.

Assisted-by: ClaudeCode:claude-fable-5
…idation

Two hypothesis oracles over the threaded-chunk-spec validation:

- acceptance implies round-trip: any reshape of a chunk into a valid
  factorization followed by a transpose of the reshaped rank (with and
  without sharding) is accepted, encodes/decodes losslessly, and its
  metadata survives JSON serialization; a transpose order of any other
  rank is rejected.

- transpose-then-shard over a rectilinear grid is accepted exactly when
  every chunk shape in the grid, transposed, is divisible by the inner
  shard shape (verified against a brute-force cross-product oracle; this
  test fails on the max-edge-representative implementation).

Assisted-by: ClaudeCode:claude-fable-5
Validate inner codecs during evolution, where the actual fill value is
available, and cover fill-changing chains with a round-trip property.

Assisted-by: Codex:GPT-6
Review fixes for the threaded codec-chain validation:

- Attach a note to any exception raised by a codec's evolve_from_array_spec,
  validate or resolve_metadata naming the codec's position in the chain, its
  class and the shape it was checked against. Codec messages talk about "the
  array", which is misleading once an earlier codec has changed the chunk
  shape; the exception type and message are unchanged.
- Drop the unused `evolve` parameter of evolve_and_validate_codecs left over
  from the removed ShardingCodec.validate inner-chain check.
- Document in ShardingCodec.evolve_from_array_spec why the inner chain is
  validated there (validate has no fill value) and against which grid.
- Docstrings use single backticks; the towncrier fragment is 303.bugfix.md.
- Tests: one parametrized happy-path test for rectilinear grids plus one
  test per error case; the sharding inner-chain test asserts the note.

Assisted-by: ClaudeCode:claude-fable-5-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s the transposed chunk

Add the `transposed_sharding_chains` strategy: a codec chain with a
TransposeCodec (random permutation) ahead of a ShardingCodec in one of three
layouts (transpose then shard; nested shard with the transpose between the
levels; transpose inside a shard as the always-valid control), together with
an oracle for its validity (every edge of the sharding codec's chunk shape
divides the transposed edge it applies to). Half the drawn chains are
invalid, breaking exactly one axis.

The property asserts that create_array accepts a chain exactly when the
oracle says it is valid, and that an accepted chain round-trips its data and
its persisted metadata. On main before the fix it fails two ways: a nested
sharding codec's inner chain is never validated, so an invalid inner chunk
shape is accepted and reads back wrong data, and a valid transpose-then-shard
chain is rejected because the sharding codec was validated against the
untransposed chunk grid.

Assisted-by: ClaudeCode:claude-fable-5-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…rom chain validation

Apply the second-lens review cuts:

- Enumerate every distinct rectilinear chunk shape without a cap. The
  warn-and-continue path validated only a subset of shapes and then accepted
  the metadata, which is worse than either failing or checking everything.
- Drop the exception notes and the try/except blocks around evolve, validate
  and resolve_metadata; they were not part of the fix.
- Move `transposed_sharding_chains` out of the public `zarr.testing.strategies`
  module into `tests/test_properties.py`, its only user.
- Drop the metadata-only rectilinear oracle property; the create_array plus
  round-trip property in `test_properties.py` covers the same acceptance
  oracle for regular and nested sharding.

Assisted-by: ClaudeCode:claude-fable-5-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix: complete imagecodecs codec package hints

Assisted-by: Codex:GPT-6

* docs: associate codec changelog with PR 338

Assisted-by: Codex:GPT-6

* docs: associate codec changelog with upstream PR 4351

Assisted-by: Codex:GPT-6
…r-developers#4345)

* docs(indexing): ground design and integration claims in current behavior

Assisted-by: Codex:GPT-6

* docs(indexing): correct reader lazy-array and cache contracts

Assisted-by: Codex:GPT-6

* fix(indexing): validate wire boundaries and clarify format contracts

Assisted-by: Codex:GPT-6

* fix(indexing): validate selector bounds and shared dependencies

Correct mathematical API documentation to match supported coordinate, grid, and chunk projection contracts.

Assisted-by: Codex:GPT-6

* docs(indexing): reconcile reader contracts and record audit fixes

Assisted-by: Codex:GPT-6

* docs(indexing): reconcile audit with current partition implementation

Retain the existing unsigned selector fix and update the unsupported mixed-dependency error assertion for general intersection routing.

Assisted-by: Codex:GPT-6

* docs(indexing): clarify planning coverage and benchmark measurement boundaries

Assisted-by: Codex:GPT-6

* fix(indexing): group signed chunk coordinates without collisions

Use lexicographic tuple grouping when chunk indices contain negative values. Cover shared one-axis and two-axis array dependencies, repeated points, and extreme signed coordinates.

Assisted-by: Codex:GPT-6

* docs(indexing): state remaining planner limits precisely

Assisted-by: Codex:GPT-6

* docs(indexing): number audit changelog entries for PR 4345

Assisted-by: Codex:GPT-6

* docs(indexing): describe current contracts in docstrings

Remove implementation history and unsupported historical claims from source and test docstrings. Distinguish immutable coordinate mappings from mutable source values.

Assisted-by: Codex:GPT-6
* docs(indexing): ground design and integration claims in current behavior

Assisted-by: Codex:GPT-6

* docs(indexing): correct reader lazy-array and cache contracts

Assisted-by: Codex:GPT-6

* fix(indexing): validate wire boundaries and clarify format contracts

Assisted-by: Codex:GPT-6

* fix(indexing): validate selector bounds and shared dependencies

Correct mathematical API documentation to match supported coordinate, grid, and chunk projection contracts.

Assisted-by: Codex:GPT-6

* docs(indexing): reconcile reader contracts and record audit fixes

Assisted-by: Codex:GPT-6

* docs(indexing): reconcile audit with current partition implementation

Retain the existing unsigned selector fix and update the unsupported mixed-dependency error assertion for general intersection routing.

Assisted-by: Codex:GPT-6

* docs(indexing): clarify planning coverage and benchmark measurement boundaries

Assisted-by: Codex:GPT-6

* fix(indexing): group signed chunk coordinates without collisions

Use lexicographic tuple grouping when chunk indices contain negative values. Cover shared one-axis and two-axis array dependencies, repeated points, and extreme signed coordinates.

Assisted-by: Codex:GPT-6

* docs(indexing): state remaining planner limits precisely

Assisted-by: Codex:GPT-6

* test(indexing): broaden planner property coverage

Assisted-by: Codex:GPT-6

* test(indexing): generate mixed affine planner dependencies

Assisted-by: Codex:GPT-6

* docs(indexing): number audit changelog entries for PR 4345

Assisted-by: Codex:GPT-6

* docs(indexing): describe current contracts in docstrings

Remove implementation history and unsupported historical claims from source and test docstrings. Distinguish immutable coordinate mappings from mutable source values.

Assisted-by: Codex:GPT-6
d-v-b and others added 14 commits September 14, 2026 12:30
Assisted-by: ClaudeCode:claude-fable-5-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Preserve grid geometry for identity, cast, scale, and transpose codecs. Stream arbitrary resolver chains without retaining the cross product, and test bounded work for common paths.

Assisted-by: Codex:GPT-6
…developers#4347)

* docs(indexing): ground design and integration claims in current behavior

Assisted-by: Codex:GPT-6

* docs(indexing): correct reader lazy-array and cache contracts

Assisted-by: Codex:GPT-6

* fix(indexing): validate wire boundaries and clarify format contracts

Assisted-by: Codex:GPT-6

* fix(indexing): validate selector bounds and shared dependencies

Correct mathematical API documentation to match supported coordinate, grid, and chunk projection contracts.

Assisted-by: Codex:GPT-6

* docs(indexing): reconcile reader contracts and record audit fixes

Assisted-by: Codex:GPT-6

* docs(indexing): reconcile audit with current partition implementation

Retain the existing unsigned selector fix and update the unsupported mixed-dependency error assertion for general intersection routing.

Assisted-by: Codex:GPT-6

* docs(indexing): clarify planning coverage and benchmark measurement boundaries

Assisted-by: Codex:GPT-6

* fix(indexing): group signed chunk coordinates without collisions

Use lexicographic tuple grouping when chunk indices contain negative values. Cover shared one-axis and two-axis array dependencies, repeated points, and extreme signed coordinates.

Assisted-by: Codex:GPT-6

* docs(indexing): state remaining planner limits precisely

Assisted-by: Codex:GPT-6

* fix(indexing): reject unsupported wire index array bounds

Assisted-by: Codex:GPT-6

* docs(indexing): clarify wire bounds rejection contract

Assisted-by: Codex:GPT-6

* docs(indexing): number audit changelog entries for PR 4345

Assisted-by: Codex:GPT-6

* docs(indexing): describe current contracts in docstrings

Remove implementation history and unsupported historical claims from source and test docstrings. Distinguish immutable coordinate mappings from mutable source values.

Assisted-by: Codex:GPT-6

* fix(indexing): validate raw index values against wire bounds

Accept valid finite and one-sided bounds with eager validation shared by both JSON loaders. Validate before affine adjustment and simplification; immutable validated maps need no retained constraint.

Assisted-by: Codex:GPT-6
…4348)

* docs(indexing): ground design and integration claims in current behavior

Assisted-by: Codex:GPT-6

* docs(indexing): correct reader lazy-array and cache contracts

Assisted-by: Codex:GPT-6

* fix(indexing): validate wire boundaries and clarify format contracts

Assisted-by: Codex:GPT-6

* fix(indexing): validate selector bounds and shared dependencies

Correct mathematical API documentation to match supported coordinate, grid, and chunk projection contracts.

Assisted-by: Codex:GPT-6

* docs(indexing): reconcile reader contracts and record audit fixes

Assisted-by: Codex:GPT-6

* docs(indexing): reconcile audit with current partition implementation

Retain the existing unsigned selector fix and update the unsupported mixed-dependency error assertion for general intersection routing.

Assisted-by: Codex:GPT-6

* docs(indexing): clarify planning coverage and benchmark measurement boundaries

Assisted-by: Codex:GPT-6

* fix(indexing): group signed chunk coordinates without collisions

Use lexicographic tuple grouping when chunk indices contain negative values. Cover shared one-axis and two-axis array dependencies, repeated points, and extreme signed coordinates.

Assisted-by: Codex:GPT-6

* docs(indexing): state remaining planner limits precisely

Assisted-by: Codex:GPT-6

* fix(indexing): define explicit source token contract

Assisted-by: Codex:GPT-6

* fix(indexing): reject mmap-backed token buffers

Assisted-by: Codex:GPT-6

* docs(indexing): number audit changelog entries for PR 4345

Assisted-by: Codex:GPT-6

* docs(indexing): describe current contracts in docstrings

Remove implementation history and unsupported historical claims from source and test docstrings. Distinguish immutable coordinate mappings from mutable source values.

Assisted-by: Codex:GPT-6

* fix(indexing): delegate source tokenization to Dask

Assisted-by: Codex:GPT-6
…umerating chunks

Validating codec chains on rectilinear grids streamed every combination of
per-axis chunk edges through each codec whose `resolve_metadata` was
overridden, e.g. numcodecs `delta`. A 3-d grid with 100 distinct edges per
axis took ~3.8 s per metadata construction, growing as n**ndim.

Follow zarrs: codecs map a whole chunk grid via the new
`BaseCodec.resolve_chunk_grid`. Dtype/fill-value codecs declare the
identity and transpose declares a permutation, so those chains stay exact.
An undeclared codec is exact on regular grids; on rectilinear grids it
makes the rest of the chain chunk-local, validated against one
representative chunk, with other chunk shapes checked at encode/decode
time. ShardingCodec now enforces divisibility at run time, which would
otherwise floor-divide and silently corrupt data.

Document the trade-off in the codec docstrings, the extending guide, the
rectilinear sharding docs and the changelog.

Assisted-by: ClaudeCode:claude-opus-5
…chunk shape

`create_codec_pipeline` evolved V3 pipelines against an all-ones placeholder
chunk spec whenever the grid was not regular, so any codec whose
`resolve_metadata` depends on the chunk shape (e.g. a reshape filter) failed
at array creation for rectilinear grids even though metadata validation had
accepted the chain, and shape-sensitive evolution output (sharding's inner
chain, BytesCodec endian) was computed against a meaningless shape.

Use `representative_chunk_shape`, the same shape `ArrayV3Metadata.__init__`
threads through evolution, so the pipeline carries exactly the codecs the
metadata produced. The ChunkTransform itself remains shape-agnostic: it only
holds the evolved codecs and resolves specs per call.

Assisted-by: ClaudeCode:claude-fable-5-1
…arr-developers#4358)

Zarr v2 retried DirectoryStore's rename because Windows intermittently
refuses to replace a destination (zarr-developers#597, fixed by zarr-developers#698). Atomic writes
arrived in v3's LocalStore in zarr-developers#3412 without that retry, so the failure is
back: _atomic_write's tmp_path.replace(path) raises

    PermissionError: [WinError 5] Access is denied:
      '...zarr.<hex>.partial' -> '...zarr.json'

and aborts the write. Reported in zarr-developers#3522.

_move_with_retry wraps the final move, retrying only the two Windows codes
that mean the destination could not be superseded right now. It needs no
platform test: off Windows an OSError carries no winerror, so the first
attempt either succeeds or raises.

The exclusive path is routed through it too but is unaffected by
construction -- the FileExistsError it relies on to report an existing node
is ERROR_ALREADY_EXISTS (183), which is not in the retried set, so it still
propagates on the first attempt.

Measured on Windows 11, 4,000 group-attr rewrites (each a replace onto an
existing zarr.json): 155-171 raised before, 0 after, for 3.68 s -> 4.08 s of
wall clock on a workload that is nothing but replace-onto-existing. In a
narrower stdlib-only loop of 20,000 replaces, 475 of 498 recoveries needed
only the second attempt and the worst needed the fourth.
…ers#4358 (zarr-developers#4359)

Follow-up cleanup to the retry landed in zarr-developers#4358, no behaviour change:

- drop the leading 0.0 sentinel, the bare `last_error` annotation, and
  the possibly-unbound re-raise from `_move_with_retry`; the final
  attempt now runs after the loop and propagates its own traceback
- correct the comment on ERROR_ACCESS_DENIED: Windows also reports it
  for destinations that will never clear (directory, read-only, ACL),
  so those surface the same error after the bounded delay
- align the changelog with the code: the exclusive path is routed
  through the retry, only FileExistsError is excluded; rename the
  fragment to the PR number and describe zarr-developers#3522 as mitigated, since a
  second process holding the destination open past the budget still
  fails
- tests record `time.sleep` instead of sleeping (~1.2 s per run before,
  now instant) and assert the actual delay schedule; the closure test
  double becomes a small callable class, removing four type-ignores
- add a regression test for `_atomic_write` onto an existing directory

Assisted-by: ClaudeCode:claude-fable-5-1

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* docs(blog): skeleton for the 3.4.0 release post

Headings and front matter in the 3.3.0 house style, with the proposed
highlights, evidence, PR references and snippet ideas as markdown comments
for the prose to be written against. Marked draft: true so it is excluded
from the built site until it is ready.

Assisted-by: ClaudeCode:claude-fable-5-1

* docs(blog): 3.4.0 post prose, copy edits, references, and contributor handles

Prose for the stack, roadmap, rectilinear, codec-error and msgspec
sections; typo and grammar fixes; PR/issue links throughout; a runnable
example of the new unknown-codec error; contributors listed by GitHub
handle with first-time contributors marked. The 3.3.0 post gains the
same contributor section for its own release range.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(blog): drop sections about fixes for 3.3.0 regressions

* docs(blog): link the previous post from the contributor note

* AI disclaimer

* Update 3.4.0-release.md

* Update docs/blog/posts/3.4.0-release.md

Co-authored-by: Ilan Gold <ilanbassgold@gmail.com>

* prose

* docs: add link to registry docs

* docs: fix the 3.3.0 release-notes link that pointed at a nonexistent PR

The load/open docstring entry cited zarr-developers#3984, which does not exist upstream
as a PR or an issue, so the weekly link check (zarr-developers#4357) reports a 404.
Point at the commit that landed the change instead.

Assisted-by: ClaudeCode:claude-fable-5-1

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Ilan Gold <ilanbassgold@gmail.com>
Built with `towncrier build --version 3.4.0`, consuming the 33 fragments
under changes/. Two touch-ups to the generated section: the codec
error-message example fence gets a `text` language (markdownlint MD040),
and the three fragments that had been named after issues rather than
the pull requests that merged them (3285 -> zarr-developers#4063; 4174 and 4272 ->
zarr-developers#4218) now link to those pull requests.

Assisted-by: ClaudeCode:claude-fable-5-1
…zarr-developers#4361)

GitHub's "Generate release notes" lists every pull request merged since
the previous tag, which in this repository includes the zarr-metadata,
zarr-indexing and zarr-http-server work that ships with those packages'
own releases. Label pull requests confined to one packages/ directory
with that package's name, and exclude those labels in .github/release.yml;
dependabot updates get their own section.

Assisted-by: ClaudeCode:claude-fable-5-1
d-v-b and others added 11 commits September 16, 2026 08:48
…developers#4349)

* docs(indexing): ground design and integration claims in current behavior

Assisted-by: Codex:GPT-6

* docs(indexing): correct reader lazy-array and cache contracts

Assisted-by: Codex:GPT-6

* fix(indexing): validate wire boundaries and clarify format contracts

Assisted-by: Codex:GPT-6

* fix(indexing): validate selector bounds and shared dependencies

Correct mathematical API documentation to match supported coordinate, grid, and chunk projection contracts.

Assisted-by: Codex:GPT-6

* docs(indexing): reconcile reader contracts and record audit fixes

Assisted-by: Codex:GPT-6

* docs(indexing): reconcile audit with current partition implementation

Retain the existing unsigned selector fix and update the unsupported mixed-dependency error assertion for general intersection routing.

Assisted-by: Codex:GPT-6

* docs(indexing): clarify planning coverage and benchmark measurement boundaries

Assisted-by: Codex:GPT-6

* fix(indexing): group signed chunk coordinates without collisions

Use lexicographic tuple grouping when chunk indices contain negative values. Cover shared one-axis and two-axis array dependencies, repeated points, and extreme signed coordinates.

Assisted-by: Codex:GPT-6

* docs(indexing): state remaining planner limits precisely

Assisted-by: Codex:GPT-6

* feat(indexing): execute partitions with their reader context

Assisted-by: Codex:GPT-6

* docs(indexing): number audit changelog entries for PR 4345

Assisted-by: Codex:GPT-6

* docs(indexing): describe current contracts in docstrings

Remove implementation history and unsupported historical claims from source and test docstrings. Distinguish immutable coordinate mappings from mutable source values.

Assisted-by: Codex:GPT-6

* fix(indexing): plan every view read with its source grid

Remove Partition.result and partition-local source windows. Preserve the source grid in derived views and supply projections for every LazyArray reader call, including unpartitioned reads.

Assisted-by: Codex:GPT-6
…ers#4350)

* feat(indexing): make LazyArray indexing lazy by default

Add synchronous writes through composed selections and an explicit eager adapter for array consumers. Cover deferred reads, source mutation, masks, aliases, and repeated destinations.

Assisted-by: Codex:GPT-6
Rebased-onto: upstream/main after zarr-developers#4345-zarr-developers#4349 merged; conflicts resolved with ClaudeCode:claude-fable-5-1

* fix(indexing): plan fancy writes against the source's write grid

The non-affine write fallback assigned one element at a time, which on a
chunked source is one chunk read-modify-write per element: a 50-row
orthogonal write into a 1000x1000 zarr array with 100x100 chunks cost
49,999 chunk writes and 27 s against 100 writes and 10 ms natively.

`write_into` now scatters in bulk. NumPy sources get one fancy assignment.
A readable source with a write grid, which `LazyArray.write` discovers
from `write_chunk_sizes` then `chunks`, is written one cell at a time:
the cell's touched hull is read once, updated in memory with
last-occurrence-wins semantics, and written back with one basic slice.
The same write into zarr now costs 100 chunk writes. Sources with no
grid, transforms the planner cannot factor, and sources that cannot be
read keep the per-element path, which never reads.

Also: order the rank check before the value copy while still validating
values for an empty selection; keep the payload of a masked zero-rank
affine write, which `np.flip` with no axes was turning into the `masked`
singleton; note that writes bypass the reader and pin that with a test;
note that iterated elements are views; add API pages for the writer and
eager adapter; and describe the shipped indexing surface in the
unreleased `LazyArray` changelog fragment instead of its removed `.lazy`
accessor.

Assisted-by: ClaudeCode:claude-fable-5-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(indexing): keep literal domains on views, positional keys unchanged

`LazyArray` re-zeroed the domain of every derived view, so a view forgot
which coordinates it was cut from and placement needed a side channel.
TensorStore keeps literal coordinates on views; NumPy users expect
positional keys. These are separable: the key dialect says how a key is
read, the domain says what the view remembers.

Views now keep their literal domain: `a[10:20]` has domain `[10, 20)`
and `a[10:20][2:5]` has `[12, 15)`, while `a[10:20][0]` is still the
first element because positional keys are normalized against the
domain's origin, which `normalize_positional_selection` already did.
Two literal keys join the NumPy ones, as in TensorStore's `__getitem__`:
an `IndexDomain` restricts the view to coordinates of its own domain
(empty intervals outside it are refused too) and an `IndexTransform`
composes onto it. Box partitions keep the request's coordinates, so a
part view's domain is a sub-domain of its parent's; a part placed by
index arrays keeps a fresh zero-origin domain, and `out_selection` is
the placement in both cases. A reversed view shows the negative origin
the algebra already produced.

Two frames stay zero-origin by construction so no consumer has to
subtract an origin: `ReadContext` re-bases its transform, and `parts()`
re-bases each projection's `cell_transform` from the request's literal
domain to positions in the view's result buffer, which is what
`Partition.projection` has always documented.

A table-driven test states the expected literal domain for every
indexing form and nested chain, checked against the transform algebra.

Assisted-by: ClaudeCode:claude-fable-5-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(indexing): state that a key's type picks relative or absolute indexing

Make the one rule about indexing a LazyArray prominent: NumPy keys are
positions relative to the view, an `IndexDomain` key names absolute
coordinates of the view's domain, an `IndexTransform` key composes onto
it, no key type has two readings, and every view keeps its absolute
domain whichever key produced it. A guide section with a runnable
snippet carries the table and the pandas `ix` / TensorStore comparison;
the module docstring, README, and API page point at it.

Assisted-by: ClaudeCode:claude-fable-5-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…elopers#4363)

* fix: replace runtime assert statements with explicit checks

Asserts are stripped under `python -O`, so checks that matter for
correctness or type narrowing must be explicit. Two were load-bearing:
`GroupMetadata.from_dict` asserted on node_type and the array-to-group
fallback in `zarr.api.asynchronous.open` caught the AssertionError, and
`make_store` asserted on mode ahead of the real validation.

Redundant asserts are deleted, narrowing asserts are restructured so
mypy narrows on its own, and the rest become explicit raises. Ruff S101
is enabled with `tests/` and `src/zarr/testing/` excluded.

Assisted-by: ClaudeCode:claude-fable-5-1

* docs: add changelog fragment for zarr-developers#4363

Assisted-by: ClaudeCode:claude-fable-5-1

* ci: scope the no-assert lint to runtime code across the monorepo

The S101 per-file ignores only covered the root tests/ and
src/zarr/testing/. The packages under packages/ inherit the root ruff
config, so the rule fired on 1524 asserts in their tests, examples, and
test-support modules and broke the ruff, Lint, pre-commit.ci, and
zarr-http-server jobs. Widen the ignores to **/tests/**, **/examples/**,
and **/testing/**, and exclude zarr-indexing's runtime source for now;
its own asserts are tracked as a separate change.

Assisted-by: ClaudeCode:claude-fable-5-1
zarr-developers#4365)

* fix(zarr-metadata): v2 array document is open and filters may be empty

Two v2 structural rules were stricter than the spec, and the shared
conformance corpus (zarr-metadata.js, conformance/v2_array.json cases 5
and 8) has been updated first; this brings the reference implementation
back into agreement.

- Members outside the .zarray definition were rejected. The spec: "Other
  keys SHOULD NOT be present within the metadata object and SHOULD be
  ignored by implementations" — a recommendation, unlike .zgroup's
  "Other keys MUST NOT be present". Extras are now tolerated by the
  validator, dropped by the model, and permitted by the Pydantic schema
  (the TypedDict is open, like the v3 one). The on-disk `.zarray` rule
  that `attributes` belongs in `.zattrs` is unchanged.
- filters: [] was rejected ("expected at least one filter"). The spec:
  "A list of JSON objects providing codec configurations, or null" — an
  empty list is a list. The Pydantic schema's min_length is dropped.

Assisted-by: ClaudeCode:claude-fable-5-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(zarr-metadata): changelog fragment for zarr-developers#4365

Assisted-by: ClaudeCode:claude-fable-5-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(zarr-metadata): link the spec text the v2 open-array change cites

Assisted-by: ClaudeCode:claude-fable-5-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(zarr-metadata): link the spec text behind every spec statement

Every docstring or comment that cites the Zarr spec or a zarr-extensions
README now carries a commit-pinned permalink with a line range
(zarr-specs fc7dd9c; zarr-extensions 4da7b37, the registry commit the
TypeScript port vendors). Unpinned zarr-extensions `tree/main` page links
are pinned the same way.

Three statements were wrong or stale and are corrected:

- zstd: `checksum` was typed required "per the proposed specification"
  (zarr-specs PR #256, never merged). The published zarr-extensions entry
  makes it optional ("Should be omitted if false"; schema requires only
  `level`), so it is now `NotRequired[bool]`.
- v3 consolidated metadata was described as "not a spec artifact"; since
  zarr-specs zarr-developers#373 the core spec names the field and fixes its envelope
  (core/index.rst L802-L816); the entry format remains a convention.
- The v2 array `*Partial` docstring spoke of a "closed shape"; the array
  document is open (other keys SHOULD be ignored), unlike `.zgroup`.

Assisted-by: ClaudeCode:claude-fable-5-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…developers#4371)

Bumps the python-dependencies group with 5 updates:

| Package | From | To |
| --- | --- | --- |
| [numpy](https://github.com/numpy/numpy) | `2.5.2` | `2.5.3` |
| [hypothesis](https://github.com/HypothesisWorks/hypothesis) | `6.167.1` | `6.168.0` |
| [uv](https://github.com/astral-sh/uv) | `0.12.9` | `0.12.12` |
| [towncrier](https://github.com/twisted/towncrier) | `25.8.0` | `26.9.0` |
| [ruff](https://github.com/astral-sh/ruff) | `0.16.5` | `0.16.6` |


Updates `numpy` from 2.5.2 to 2.5.3
- [Release notes](https://github.com/numpy/numpy/releases)
- [Changelog](https://github.com/numpy/numpy/blob/main/doc/RELEASE_WALKTHROUGH.rst)
- [Commits](numpy/numpy@v2.5.2...v2.5.3)

Updates `hypothesis` from 6.167.1 to 6.168.0
- [Release notes](https://github.com/HypothesisWorks/hypothesis/releases)
- [Commits](HypothesisWorks/hypothesis@v6.167.1...v6.168.0)

Updates `uv` from 0.12.9 to 0.12.12
- [Release notes](https://github.com/astral-sh/uv/releases)
- [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md)
- [Commits](astral-sh/uv@0.12.9...0.12.12)

Updates `towncrier` from 25.8.0 to 26.9.0
- [Release notes](https://github.com/twisted/towncrier/releases)
- [Changelog](https://github.com/twisted/towncrier/blob/trunk/NEWS.rst)
- [Commits](twisted/towncrier@25.8.0...26.9.0)

Updates `ruff` from 0.16.5 to 0.16.6
- [Release notes](https://github.com/astral-sh/ruff/releases)
- [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md)
- [Commits](astral-sh/ruff@0.16.5...0.16.6)

---
updated-dependencies:
- dependency-name: numpy
  dependency-version: 2.5.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: hypothesis
  dependency-version: 6.168.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: python-dependencies
- dependency-name: uv
  dependency-version: 0.12.12
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: towncrier
  dependency-version: 26.9.0
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: python-dependencies
- dependency-name: ruff
  dependency-version: 0.16.6
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…pers#4368)

* fix(ci): preserve branch zarr in xarray downstream tests

* chore: use pull request number for changelog fragment

---------

Co-authored-by: glaziermag <glaziermag@users.noreply.github.com>
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
* chore: update pre-commit hooks

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.16.0 → v0.16.6](astral-sh/ruff-pre-commit@v0.16.0...v0.16.6)
- [github.com/codespell-project/codespell: v2.4.2 → v2.4.3](codespell-project/codespell@v2.4.2...v2.4.3)
- [github.com/DavidAnson/markdownlint-cli2: v0.22.1 → v0.23.2](DavidAnson/markdownlint-cli2@v0.22.1...v0.23.2)
- [github.com/scientific-python/cookie: 2026.06.18 → 2026.08.14](scientific-python/cookie@2026.06.18...2026.08.14)
- [github.com/zizmorcore/zizmor-pre-commit: v1.26.1 → v1.30.0](zizmorcore/zizmor-pre-commit@v1.26.1...v1.30.0)
- [github.com/twisted/towncrier: 25.8.0 → 26.9.0](twisted/towncrier@25.8.0...26.9.0)

* style: reformat a README example for ruff 0.16.6

ruff-format v0.16.6 formats Python code blocks in Markdown, and normalizes
this inline comment to PEP 8's two spaces. Fallout from the hook bump in this
pull request, not a behavior change.

Assisted-by: ClaudeCode:claude-opus-5
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* ci: pin just-version in setup-just steps

zizmor v1.30.0 adds the `unpinned-tools` audit, which flags all twelve
`extractions/setup-just` steps across the three package workflows: pinning the
action by SHA still leaves the action free to install whatever just is newest
at run time, so a just release can change CI without a commit here.

Pinned to 1.58.0, which is what the action resolves to today, so this changes
nothing about the current builds while making that choice explicit.

Assisted-by: ClaudeCode:claude-opus-5
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…zarr-developers#4370)

Bumps the actions group with 4 updates in the / directory: [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv), [github-community-projects/issue-metrics](https://github.com/github-community-projects/issue-metrics), [scientific-python/upload-nightly-action](https://github.com/scientific-python/upload-nightly-action) and [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action).


Updates `astral-sh/setup-uv` from 10.0.1 to 10.1.0
- [Release notes](https://github.com/astral-sh/setup-uv/releases)
- [Commits](astral-sh/setup-uv@20cfd1b...bec219d)

Updates `github-community-projects/issue-metrics` from 5.0.1 to 5.0.2
- [Release notes](https://github.com/github-community-projects/issue-metrics/releases)
- [Commits](github-community-projects/issue-metrics@61084fa...a7dc2fb)

Updates `scientific-python/upload-nightly-action` from 0.6.4 to 0.6.5
- [Release notes](https://github.com/scientific-python/upload-nightly-action/releases)
- [Commits](scientific-python/upload-nightly-action@e76cfec...16fa02e)

Updates `zizmorcore/zizmor-action` from 0.6.3 to 0.6.4
- [Release notes](https://github.com/zizmorcore/zizmor-action/releases)
- [Commits](zizmorcore/zizmor-action@70fb788...cc914d7)

---
updated-dependencies:
- dependency-name: astral-sh/setup-uv
  dependency-version: 10.1.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: github-community-projects/issue-metrics
  dependency-version: 5.0.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: scientific-python/upload-nightly-action
  dependency-version: 0.6.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: zizmorcore/zizmor-action
  dependency-version: 0.6.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
… validators return tuples (zarr-developers#4378)

* fix(zarr-metadata)!: make entity types assignable to metadata fields; validators return tuples

`ZarrV3NamedConfigJSON.name` and `.configuration` are `ReadOnly` and the
envelope is `closed`, so the concrete codec / chunk-grid / chunk-key-encoding
/ data-type TypedDicts are assignable to the fields they describe. Every
concrete `*Object` / `*Configuration` is `closed` and object forms declare
`must_understand: NotRequired[bool]`.

Every `validate_*` in `zarr_metadata.model` returns
`tuple[ValidationProblem, ...]`, `MetadataValidationError.problems` is a
tuple, and `load_store_json` returns `object` rather than `Any`. `ANN401`
is enforced package-wide. `ZarrV2ConsolidatedMetadataJSON.zarr_consolidated_format`
is `Literal[1]`.

Split from #296 (part 1 of 3).

Assisted-by: ClaudeCode:claude-fable-5-1

* chore(zarr-metadata): number changelog fragments for #317

Assisted-by: ClaudeCode:claude-fable-5-1

* test(zarr-metadata): expect tuples from validate_array_metadata_v2 in zarr-developers#4365's tests

Assisted-by: ClaudeCode:claude-opus-5
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* chore(zarr-metadata): number changelog fragments for zarr-developers#4378

Assisted-by: ClaudeCode:claude-opus-5
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants