Conversation
d-v-b
added a commit
to d-v-b/zarr-metadata.js
that referenced
this pull request
Aug 30, 2026
The same bug class d-v-b/zarr-python#303 fixes in zarr-python's metadata validation existed here: transpose/sharding rules judged every codec against the array-level dimensional facts, falsely rejecting the spec-endorsed reshape->transpose chain and shards sized for the transposed chunk, while missing shards sized for the un-transposed one. The pipeline walk now threads context: a valid transpose permutes the per-dimension chunk sizes, sharding recurses with its inner chunk rank and spends the context at the array->bytes boundary, and any codec the layer cannot reason about drops the context for the remaining codecs. Assisted-by: ClaudeCode:claude-fable-5
d-v-b
added a commit
that referenced
this pull request
Sep 9, 2026
…, clamps and metadata Invariant: a chunk edge length is always >= 1; a dimension's extent may be 0, in which case the dimension has zero chunks (ceildiv(0, size) == 0). Zero-length-axis bugs have recurred since 2017 (#150, #241, #303, zarr-developers#972, zarr-developers#1977, zarr-developers#2434, zarr-developers#3711, zarr-developers#4305, zarr-developers#4307, zarr-developers#4328) because the layers disagreed on this invariant and every span-derived chunk spelling clamped on its own: - The metadata layer (common.py, metadata/v3.py) required chunk edges >= 1, but the in-memory FixedDimension allowed size == 0 with four special-case branches left over from zarr-developers#2434, so normalization could build a grid the metadata constructor then rejected. FixedDimension now rejects size < 1 and the four `if self.size == 0` branches are gone. VaryingDimension already required edges > 0 and is unchanged. - `chunks=-1`, `chunks=False`, `chunks="auto"` (_guess_regular_chunks, both the typesize == 0 early return and the np.maximum line) and `shards="auto"` each derived "one chunk covering the axis" independently. They now all go through one helper, `_full_span_chunk_size(span) = max(span, 1)`, which is the single definition of that phrase for a possibly zero-length axis. - Zarr format 2 metadata had no chunk >= 1 check, so a legacy `chunks: [0]` document opened fine and read uninitialised memory after a resize. It now raises a clear ValueError at parse time, matching the format 3 grid. - Rectilinear grids had no creation-time spelling for a zero-length axis: normalize_chunks_1d required sum(edges) == span, which no list of positive edges can satisfy for span 0, even though the same state is reachable via resize((0,)) and round-trips through reopen. For span == 0 any non-empty list of positive edges is now accepted verbatim, producing the same VaryingDimension(edges, extent=0) that resize produces; the strict sum check is kept for span > 0. Tests: the per-spelling regression test from zarr-developers#4328 is replaced by one matrix over {-1, False, "auto", 1, (1,...), [[2, 2]]} x {(0,), (0, 4), (4, 0), (0, 0), ()} x {v2, v3} x {no shards, shards="auto" with and without a byte budget, explicit shards}, with separate small tests for each error case. Tests that constructed FixedDimension(size=0) now assert it raises, and a zero-extent test covers the behaviour the old special cases were guarding. Assisted-by: ClaudeCode:claude-fable-5-1
d-v-b
added a commit
that referenced
this pull request
Sep 9, 2026
…, clamps and metadata Invariant: a chunk edge length is always >= 1; a dimension's extent may be 0, in which case the dimension has zero chunks (ceildiv(0, size) == 0). Zero-length-axis bugs have recurred since 2017 (#150, #241, #303, zarr-developers#972, zarr-developers#1977, zarr-developers#2434, zarr-developers#3711, zarr-developers#4305, zarr-developers#4307, zarr-developers#4328) because the layers disagreed on this invariant and every span-derived chunk spelling clamped on its own: - The metadata layer (common.py, metadata/v3.py) required chunk edges >= 1, but the in-memory FixedDimension allowed size == 0 with four special-case branches left over from zarr-developers#2434, so normalization could build a grid the metadata constructor then rejected. FixedDimension now rejects size < 1 and the four `if self.size == 0` branches are gone. VaryingDimension already required edges > 0 and is unchanged. - `chunks=-1`, `chunks=False`, `chunks="auto"` (_guess_regular_chunks, both the typesize == 0 early return and the np.maximum line) and `shards="auto"` each derived "one chunk covering the axis" independently. They now all go through one helper, `_full_span_chunk_size(span) = max(span, 1)`, which is the single definition of that phrase for a possibly zero-length axis. - Zarr format 2 metadata had no chunk >= 1 check, so a legacy `chunks: [0]` document opened fine and read uninitialised memory after a resize. It now raises a clear ValueError at parse time, matching the format 3 grid. - Rectilinear grids had no creation-time spelling for a zero-length axis: normalize_chunks_1d required sum(edges) == span, which no list of positive edges can satisfy for span 0, even though the same state is reachable via resize((0,)) and round-trips through reopen. For span == 0 any non-empty list of positive edges is now accepted verbatim, producing the same VaryingDimension(edges, extent=0) that resize produces; the strict sum check is kept for span > 0. Tests: the per-spelling regression test from zarr-developers#4328 is replaced by one matrix over {-1, False, "auto", 1, (1,...), [[2, 2]]} x {(0,), (0, 4), (4, 0), (0, 0), ()} x {v2, v3} x {no shards, shards="auto" with and without a byte budget, explicit shards}, with separate small tests for each error case. Tests that constructed FixedDimension(size=0) now assert it raises, and a zero-extent test covers the behaviour the old special cases were guarding. Assisted-by: ClaudeCode:claude-fable-5-1
`ArrayV3Metadata` validated every codec against the array-level shape and chunk grid, and threaded the *array* spec (not a chunk spec) through `resolve_metadata` during evolution. Both wrongly reject chains in which an earlier array->array codec changes a chunk's shape or rank, e.g. the zarr-extensions `reshape` codec followed by `transpose` with an order of the reshaped rank -- a combination the reshape spec explicitly endorses and that the encode path already handles correctly. Codecs are now evolved and validated in a single threaded pass (`evolve_and_validate_codecs`): each codec sees the chunk spec produced by the previous codec's `resolve_metadata`, exactly as at encode time. The array-level shape/chunk grid are passed to `Codec.validate` unchanged until a codec changes the chunk shape, after which the resolved chunk shape (and a regular grid of it) stands in for them. `ShardingCodec.validate` now validates its inner chain the same way against the inner chunk shape. Assisted-by: ClaudeCode:claude-fable-5
…rectilinear chunk shape Review feedback on the threaded-chunk-spec validation: after a codec changes the chunk shape, validating the rest of the chain against a single representative (max-edge) chunk shape is unsound for rectilinear grids -- an inner shard size that divides the largest chunk need not divide the others. Concretely, transpose over a rectilinear grid followed by sharding falsely accepted an inner chunk shape that only divided the largest transposed chunk. The representative was also used to *detect* shape changes, which could miss changes affecting only non-representative chunks. `evolve_and_validate_codecs` now threads every distinct chunk shape of the grid (the cross product of per-dimension distinct edges, capped at 4096 with a ZarrUserWarning on truncation) through `resolve_metadata`, and validates each one individually once any codec has changed a chunk shape. The representative spec remains the single spec used for codec evolution and dtype tracking. Assisted-by: ClaudeCode:claude-fable-5
…idation Two hypothesis oracles over the threaded-chunk-spec validation: - acceptance implies round-trip: any reshape of a chunk into a valid factorization followed by a transpose of the reshaped rank (with and without sharding) is accepted, encodes/decodes losslessly, and its metadata survives JSON serialization; a transpose order of any other rank is rejected. - transpose-then-shard over a rectilinear grid is accepted exactly when every chunk shape in the grid, transposed, is divisible by the inner shard shape (verified against a brute-force cross-product oracle; this test fails on the max-edge-representative implementation). Assisted-by: ClaudeCode:claude-fable-5
Validate inner codecs during evolution, where the actual fill value is available, and cover fill-changing chains with a round-trip property. Assisted-by: Codex:GPT-6
Review fixes for the threaded codec-chain validation: - Attach a note to any exception raised by a codec's evolve_from_array_spec, validate or resolve_metadata naming the codec's position in the chain, its class and the shape it was checked against. Codec messages talk about "the array", which is misleading once an earlier codec has changed the chunk shape; the exception type and message are unchanged. - Drop the unused `evolve` parameter of evolve_and_validate_codecs left over from the removed ShardingCodec.validate inner-chain check. - Document in ShardingCodec.evolve_from_array_spec why the inner chain is validated there (validate has no fill value) and against which grid. - Docstrings use single backticks; the towncrier fragment is 303.bugfix.md. - Tests: one parametrized happy-path test for rectilinear grids plus one test per error case; the sharding inner-chain test asserts the note. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s the transposed chunk Add the `transposed_sharding_chains` strategy: a codec chain with a TransposeCodec (random permutation) ahead of a ShardingCodec in one of three layouts (transpose then shard; nested shard with the transpose between the levels; transpose inside a shard as the always-valid control), together with an oracle for its validity (every edge of the sharding codec's chunk shape divides the transposed edge it applies to). Half the drawn chains are invalid, breaking exactly one axis. The property asserts that create_array accepts a chain exactly when the oracle says it is valid, and that an accepted chain round-trips its data and its persisted metadata. On main before the fix it fails two ways: a nested sharding codec's inner chain is never validated, so an invalid inner chunk shape is accepted and reads back wrong data, and a valid transpose-then-shard chain is rejected because the sharding codec was validated against the untransposed chunk grid. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…rom chain validation Apply the second-lens review cuts: - Enumerate every distinct rectilinear chunk shape without a cap. The warn-and-continue path validated only a subset of shapes and then accepted the metadata, which is worse than either failing or checking everything. - Drop the exception notes and the try/except blocks around evolve, validate and resolve_metadata; they were not part of the fix. - Move `transposed_sharding_chains` out of the public `zarr.testing.strategies` module into `tests/test_properties.py`, its only user. - Drop the metadata-only rectilinear oracle property; the create_array plus round-trip property in `test_properties.py` covers the same acceptance oracle for regular and nested sharding. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix: complete imagecodecs codec package hints Assisted-by: Codex:GPT-6 * docs: associate codec changelog with PR 338 Assisted-by: Codex:GPT-6 * docs: associate codec changelog with upstream PR 4351 Assisted-by: Codex:GPT-6
…r-developers#4345) * docs(indexing): ground design and integration claims in current behavior Assisted-by: Codex:GPT-6 * docs(indexing): correct reader lazy-array and cache contracts Assisted-by: Codex:GPT-6 * fix(indexing): validate wire boundaries and clarify format contracts Assisted-by: Codex:GPT-6 * fix(indexing): validate selector bounds and shared dependencies Correct mathematical API documentation to match supported coordinate, grid, and chunk projection contracts. Assisted-by: Codex:GPT-6 * docs(indexing): reconcile reader contracts and record audit fixes Assisted-by: Codex:GPT-6 * docs(indexing): reconcile audit with current partition implementation Retain the existing unsigned selector fix and update the unsupported mixed-dependency error assertion for general intersection routing. Assisted-by: Codex:GPT-6 * docs(indexing): clarify planning coverage and benchmark measurement boundaries Assisted-by: Codex:GPT-6 * fix(indexing): group signed chunk coordinates without collisions Use lexicographic tuple grouping when chunk indices contain negative values. Cover shared one-axis and two-axis array dependencies, repeated points, and extreme signed coordinates. Assisted-by: Codex:GPT-6 * docs(indexing): state remaining planner limits precisely Assisted-by: Codex:GPT-6 * docs(indexing): number audit changelog entries for PR 4345 Assisted-by: Codex:GPT-6 * docs(indexing): describe current contracts in docstrings Remove implementation history and unsupported historical claims from source and test docstrings. Distinguish immutable coordinate mappings from mutable source values. Assisted-by: Codex:GPT-6
* docs(indexing): ground design and integration claims in current behavior Assisted-by: Codex:GPT-6 * docs(indexing): correct reader lazy-array and cache contracts Assisted-by: Codex:GPT-6 * fix(indexing): validate wire boundaries and clarify format contracts Assisted-by: Codex:GPT-6 * fix(indexing): validate selector bounds and shared dependencies Correct mathematical API documentation to match supported coordinate, grid, and chunk projection contracts. Assisted-by: Codex:GPT-6 * docs(indexing): reconcile reader contracts and record audit fixes Assisted-by: Codex:GPT-6 * docs(indexing): reconcile audit with current partition implementation Retain the existing unsigned selector fix and update the unsupported mixed-dependency error assertion for general intersection routing. Assisted-by: Codex:GPT-6 * docs(indexing): clarify planning coverage and benchmark measurement boundaries Assisted-by: Codex:GPT-6 * fix(indexing): group signed chunk coordinates without collisions Use lexicographic tuple grouping when chunk indices contain negative values. Cover shared one-axis and two-axis array dependencies, repeated points, and extreme signed coordinates. Assisted-by: Codex:GPT-6 * docs(indexing): state remaining planner limits precisely Assisted-by: Codex:GPT-6 * test(indexing): broaden planner property coverage Assisted-by: Codex:GPT-6 * test(indexing): generate mixed affine planner dependencies Assisted-by: Codex:GPT-6 * docs(indexing): number audit changelog entries for PR 4345 Assisted-by: Codex:GPT-6 * docs(indexing): describe current contracts in docstrings Remove implementation history and unsupported historical claims from source and test docstrings. Distinguish immutable coordinate mappings from mutable source values. Assisted-by: Codex:GPT-6
d-v-b
force-pushed
the
fix/codec-chain-validation
branch
from
September 14, 2026 10:30
d9bf256 to
04dd144
Compare
Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Preserve grid geometry for identity, cast, scale, and transpose codecs. Stream arbitrary resolver chains without retaining the cross product, and test bounded work for common paths. Assisted-by: Codex:GPT-6
…developers#4347) * docs(indexing): ground design and integration claims in current behavior Assisted-by: Codex:GPT-6 * docs(indexing): correct reader lazy-array and cache contracts Assisted-by: Codex:GPT-6 * fix(indexing): validate wire boundaries and clarify format contracts Assisted-by: Codex:GPT-6 * fix(indexing): validate selector bounds and shared dependencies Correct mathematical API documentation to match supported coordinate, grid, and chunk projection contracts. Assisted-by: Codex:GPT-6 * docs(indexing): reconcile reader contracts and record audit fixes Assisted-by: Codex:GPT-6 * docs(indexing): reconcile audit with current partition implementation Retain the existing unsigned selector fix and update the unsupported mixed-dependency error assertion for general intersection routing. Assisted-by: Codex:GPT-6 * docs(indexing): clarify planning coverage and benchmark measurement boundaries Assisted-by: Codex:GPT-6 * fix(indexing): group signed chunk coordinates without collisions Use lexicographic tuple grouping when chunk indices contain negative values. Cover shared one-axis and two-axis array dependencies, repeated points, and extreme signed coordinates. Assisted-by: Codex:GPT-6 * docs(indexing): state remaining planner limits precisely Assisted-by: Codex:GPT-6 * fix(indexing): reject unsupported wire index array bounds Assisted-by: Codex:GPT-6 * docs(indexing): clarify wire bounds rejection contract Assisted-by: Codex:GPT-6 * docs(indexing): number audit changelog entries for PR 4345 Assisted-by: Codex:GPT-6 * docs(indexing): describe current contracts in docstrings Remove implementation history and unsupported historical claims from source and test docstrings. Distinguish immutable coordinate mappings from mutable source values. Assisted-by: Codex:GPT-6 * fix(indexing): validate raw index values against wire bounds Accept valid finite and one-sided bounds with eager validation shared by both JSON loaders. Validate before affine adjustment and simplification; immutable validated maps need no retained constraint. Assisted-by: Codex:GPT-6
…4348) * docs(indexing): ground design and integration claims in current behavior Assisted-by: Codex:GPT-6 * docs(indexing): correct reader lazy-array and cache contracts Assisted-by: Codex:GPT-6 * fix(indexing): validate wire boundaries and clarify format contracts Assisted-by: Codex:GPT-6 * fix(indexing): validate selector bounds and shared dependencies Correct mathematical API documentation to match supported coordinate, grid, and chunk projection contracts. Assisted-by: Codex:GPT-6 * docs(indexing): reconcile reader contracts and record audit fixes Assisted-by: Codex:GPT-6 * docs(indexing): reconcile audit with current partition implementation Retain the existing unsigned selector fix and update the unsupported mixed-dependency error assertion for general intersection routing. Assisted-by: Codex:GPT-6 * docs(indexing): clarify planning coverage and benchmark measurement boundaries Assisted-by: Codex:GPT-6 * fix(indexing): group signed chunk coordinates without collisions Use lexicographic tuple grouping when chunk indices contain negative values. Cover shared one-axis and two-axis array dependencies, repeated points, and extreme signed coordinates. Assisted-by: Codex:GPT-6 * docs(indexing): state remaining planner limits precisely Assisted-by: Codex:GPT-6 * fix(indexing): define explicit source token contract Assisted-by: Codex:GPT-6 * fix(indexing): reject mmap-backed token buffers Assisted-by: Codex:GPT-6 * docs(indexing): number audit changelog entries for PR 4345 Assisted-by: Codex:GPT-6 * docs(indexing): describe current contracts in docstrings Remove implementation history and unsupported historical claims from source and test docstrings. Distinguish immutable coordinate mappings from mutable source values. Assisted-by: Codex:GPT-6 * fix(indexing): delegate source tokenization to Dask Assisted-by: Codex:GPT-6
…umerating chunks Validating codec chains on rectilinear grids streamed every combination of per-axis chunk edges through each codec whose `resolve_metadata` was overridden, e.g. numcodecs `delta`. A 3-d grid with 100 distinct edges per axis took ~3.8 s per metadata construction, growing as n**ndim. Follow zarrs: codecs map a whole chunk grid via the new `BaseCodec.resolve_chunk_grid`. Dtype/fill-value codecs declare the identity and transpose declares a permutation, so those chains stay exact. An undeclared codec is exact on regular grids; on rectilinear grids it makes the rest of the chain chunk-local, validated against one representative chunk, with other chunk shapes checked at encode/decode time. ShardingCodec now enforces divisibility at run time, which would otherwise floor-divide and silently corrupt data. Document the trade-off in the codec docstrings, the extending guide, the rectilinear sharding docs and the changelog. Assisted-by: ClaudeCode:claude-opus-5
…chunk shape `create_codec_pipeline` evolved V3 pipelines against an all-ones placeholder chunk spec whenever the grid was not regular, so any codec whose `resolve_metadata` depends on the chunk shape (e.g. a reshape filter) failed at array creation for rectilinear grids even though metadata validation had accepted the chain, and shape-sensitive evolution output (sharding's inner chain, BytesCodec endian) was computed against a meaningless shape. Use `representative_chunk_shape`, the same shape `ArrayV3Metadata.__init__` threads through evolution, so the pipeline carries exactly the codecs the metadata produced. The ChunkTransform itself remains shape-agnostic: it only holds the evolved codecs and resolves specs per call. Assisted-by: ClaudeCode:claude-fable-5-1
…arr-developers#4358) Zarr v2 retried DirectoryStore's rename because Windows intermittently refuses to replace a destination (zarr-developers#597, fixed by zarr-developers#698). Atomic writes arrived in v3's LocalStore in zarr-developers#3412 without that retry, so the failure is back: _atomic_write's tmp_path.replace(path) raises PermissionError: [WinError 5] Access is denied: '...zarr.<hex>.partial' -> '...zarr.json' and aborts the write. Reported in zarr-developers#3522. _move_with_retry wraps the final move, retrying only the two Windows codes that mean the destination could not be superseded right now. It needs no platform test: off Windows an OSError carries no winerror, so the first attempt either succeeds or raises. The exclusive path is routed through it too but is unaffected by construction -- the FileExistsError it relies on to report an existing node is ERROR_ALREADY_EXISTS (183), which is not in the retried set, so it still propagates on the first attempt. Measured on Windows 11, 4,000 group-attr rewrites (each a replace onto an existing zarr.json): 155-171 raised before, 0 after, for 3.68 s -> 4.08 s of wall clock on a workload that is nothing but replace-onto-existing. In a narrower stdlib-only loop of 20,000 replaces, 475 of 498 recoveries needed only the second attempt and the worst needed the fourth.
…ers#4358 (zarr-developers#4359) Follow-up cleanup to the retry landed in zarr-developers#4358, no behaviour change: - drop the leading 0.0 sentinel, the bare `last_error` annotation, and the possibly-unbound re-raise from `_move_with_retry`; the final attempt now runs after the loop and propagates its own traceback - correct the comment on ERROR_ACCESS_DENIED: Windows also reports it for destinations that will never clear (directory, read-only, ACL), so those surface the same error after the bounded delay - align the changelog with the code: the exclusive path is routed through the retry, only FileExistsError is excluded; rename the fragment to the PR number and describe zarr-developers#3522 as mitigated, since a second process holding the destination open past the budget still fails - tests record `time.sleep` instead of sleeping (~1.2 s per run before, now instant) and assert the actual delay schedule; the closure test double becomes a small callable class, removing four type-ignores - add a regression test for `_atomic_write` onto an existing directory Assisted-by: ClaudeCode:claude-fable-5-1 Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* docs(blog): skeleton for the 3.4.0 release post Headings and front matter in the 3.3.0 house style, with the proposed highlights, evidence, PR references and snippet ideas as markdown comments for the prose to be written against. Marked draft: true so it is excluded from the built site until it is ready. Assisted-by: ClaudeCode:claude-fable-5-1 * docs(blog): 3.4.0 post prose, copy edits, references, and contributor handles Prose for the stack, roadmap, rectilinear, codec-error and msgspec sections; typo and grammar fixes; PR/issue links throughout; a runnable example of the new unknown-codec error; contributors listed by GitHub handle with first-time contributors marked. The 3.3.0 post gains the same contributor section for its own release range. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(blog): drop sections about fixes for 3.3.0 regressions * docs(blog): link the previous post from the contributor note * AI disclaimer * Update 3.4.0-release.md * Update docs/blog/posts/3.4.0-release.md Co-authored-by: Ilan Gold <ilanbassgold@gmail.com> * prose * docs: add link to registry docs * docs: fix the 3.3.0 release-notes link that pointed at a nonexistent PR The load/open docstring entry cited zarr-developers#3984, which does not exist upstream as a PR or an issue, so the weekly link check (zarr-developers#4357) reports a 404. Point at the commit that landed the change instead. Assisted-by: ClaudeCode:claude-fable-5-1 --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Ilan Gold <ilanbassgold@gmail.com>
Built with `towncrier build --version 3.4.0`, consuming the 33 fragments under changes/. Two touch-ups to the generated section: the codec error-message example fence gets a `text` language (markdownlint MD040), and the three fragments that had been named after issues rather than the pull requests that merged them (3285 -> zarr-developers#4063; 4174 and 4272 -> zarr-developers#4218) now link to those pull requests. Assisted-by: ClaudeCode:claude-fable-5-1
…zarr-developers#4361) GitHub's "Generate release notes" lists every pull request merged since the previous tag, which in this repository includes the zarr-metadata, zarr-indexing and zarr-http-server work that ships with those packages' own releases. Label pull requests confined to one packages/ directory with that package's name, and exclude those labels in .github/release.yml; dependabot updates get their own section. Assisted-by: ClaudeCode:claude-fable-5-1
…developers#4349) * docs(indexing): ground design and integration claims in current behavior Assisted-by: Codex:GPT-6 * docs(indexing): correct reader lazy-array and cache contracts Assisted-by: Codex:GPT-6 * fix(indexing): validate wire boundaries and clarify format contracts Assisted-by: Codex:GPT-6 * fix(indexing): validate selector bounds and shared dependencies Correct mathematical API documentation to match supported coordinate, grid, and chunk projection contracts. Assisted-by: Codex:GPT-6 * docs(indexing): reconcile reader contracts and record audit fixes Assisted-by: Codex:GPT-6 * docs(indexing): reconcile audit with current partition implementation Retain the existing unsigned selector fix and update the unsupported mixed-dependency error assertion for general intersection routing. Assisted-by: Codex:GPT-6 * docs(indexing): clarify planning coverage and benchmark measurement boundaries Assisted-by: Codex:GPT-6 * fix(indexing): group signed chunk coordinates without collisions Use lexicographic tuple grouping when chunk indices contain negative values. Cover shared one-axis and two-axis array dependencies, repeated points, and extreme signed coordinates. Assisted-by: Codex:GPT-6 * docs(indexing): state remaining planner limits precisely Assisted-by: Codex:GPT-6 * feat(indexing): execute partitions with their reader context Assisted-by: Codex:GPT-6 * docs(indexing): number audit changelog entries for PR 4345 Assisted-by: Codex:GPT-6 * docs(indexing): describe current contracts in docstrings Remove implementation history and unsupported historical claims from source and test docstrings. Distinguish immutable coordinate mappings from mutable source values. Assisted-by: Codex:GPT-6 * fix(indexing): plan every view read with its source grid Remove Partition.result and partition-local source windows. Preserve the source grid in derived views and supply projections for every LazyArray reader call, including unpartitioned reads. Assisted-by: Codex:GPT-6
…ers#4350) * feat(indexing): make LazyArray indexing lazy by default Add synchronous writes through composed selections and an explicit eager adapter for array consumers. Cover deferred reads, source mutation, masks, aliases, and repeated destinations. Assisted-by: Codex:GPT-6 Rebased-onto: upstream/main after zarr-developers#4345-zarr-developers#4349 merged; conflicts resolved with ClaudeCode:claude-fable-5-1 * fix(indexing): plan fancy writes against the source's write grid The non-affine write fallback assigned one element at a time, which on a chunked source is one chunk read-modify-write per element: a 50-row orthogonal write into a 1000x1000 zarr array with 100x100 chunks cost 49,999 chunk writes and 27 s against 100 writes and 10 ms natively. `write_into` now scatters in bulk. NumPy sources get one fancy assignment. A readable source with a write grid, which `LazyArray.write` discovers from `write_chunk_sizes` then `chunks`, is written one cell at a time: the cell's touched hull is read once, updated in memory with last-occurrence-wins semantics, and written back with one basic slice. The same write into zarr now costs 100 chunk writes. Sources with no grid, transforms the planner cannot factor, and sources that cannot be read keep the per-element path, which never reads. Also: order the rank check before the value copy while still validating values for an empty selection; keep the payload of a masked zero-rank affine write, which `np.flip` with no axes was turning into the `masked` singleton; note that writes bypass the reader and pin that with a test; note that iterated elements are views; add API pages for the writer and eager adapter; and describe the shipped indexing surface in the unreleased `LazyArray` changelog fragment instead of its removed `.lazy` accessor. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(indexing): keep literal domains on views, positional keys unchanged `LazyArray` re-zeroed the domain of every derived view, so a view forgot which coordinates it was cut from and placement needed a side channel. TensorStore keeps literal coordinates on views; NumPy users expect positional keys. These are separable: the key dialect says how a key is read, the domain says what the view remembers. Views now keep their literal domain: `a[10:20]` has domain `[10, 20)` and `a[10:20][2:5]` has `[12, 15)`, while `a[10:20][0]` is still the first element because positional keys are normalized against the domain's origin, which `normalize_positional_selection` already did. Two literal keys join the NumPy ones, as in TensorStore's `__getitem__`: an `IndexDomain` restricts the view to coordinates of its own domain (empty intervals outside it are refused too) and an `IndexTransform` composes onto it. Box partitions keep the request's coordinates, so a part view's domain is a sub-domain of its parent's; a part placed by index arrays keeps a fresh zero-origin domain, and `out_selection` is the placement in both cases. A reversed view shows the negative origin the algebra already produced. Two frames stay zero-origin by construction so no consumer has to subtract an origin: `ReadContext` re-bases its transform, and `parts()` re-bases each projection's `cell_transform` from the request's literal domain to positions in the view's result buffer, which is what `Partition.projection` has always documented. A table-driven test states the expected literal domain for every indexing form and nested chain, checked against the transform algebra. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(indexing): state that a key's type picks relative or absolute indexing Make the one rule about indexing a LazyArray prominent: NumPy keys are positions relative to the view, an `IndexDomain` key names absolute coordinates of the view's domain, an `IndexTransform` key composes onto it, no key type has two readings, and every view keeps its absolute domain whichever key produced it. A guide section with a runnable snippet carries the table and the pandas `ix` / TensorStore comparison; the module docstring, README, and API page point at it. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…elopers#4363) * fix: replace runtime assert statements with explicit checks Asserts are stripped under `python -O`, so checks that matter for correctness or type narrowing must be explicit. Two were load-bearing: `GroupMetadata.from_dict` asserted on node_type and the array-to-group fallback in `zarr.api.asynchronous.open` caught the AssertionError, and `make_store` asserted on mode ahead of the real validation. Redundant asserts are deleted, narrowing asserts are restructured so mypy narrows on its own, and the rest become explicit raises. Ruff S101 is enabled with `tests/` and `src/zarr/testing/` excluded. Assisted-by: ClaudeCode:claude-fable-5-1 * docs: add changelog fragment for zarr-developers#4363 Assisted-by: ClaudeCode:claude-fable-5-1 * ci: scope the no-assert lint to runtime code across the monorepo The S101 per-file ignores only covered the root tests/ and src/zarr/testing/. The packages under packages/ inherit the root ruff config, so the rule fired on 1524 asserts in their tests, examples, and test-support modules and broke the ruff, Lint, pre-commit.ci, and zarr-http-server jobs. Widen the ignores to **/tests/**, **/examples/**, and **/testing/**, and exclude zarr-indexing's runtime source for now; its own asserts are tracked as a separate change. Assisted-by: ClaudeCode:claude-fable-5-1
zarr-developers#4365) * fix(zarr-metadata): v2 array document is open and filters may be empty Two v2 structural rules were stricter than the spec, and the shared conformance corpus (zarr-metadata.js, conformance/v2_array.json cases 5 and 8) has been updated first; this brings the reference implementation back into agreement. - Members outside the .zarray definition were rejected. The spec: "Other keys SHOULD NOT be present within the metadata object and SHOULD be ignored by implementations" — a recommendation, unlike .zgroup's "Other keys MUST NOT be present". Extras are now tolerated by the validator, dropped by the model, and permitted by the Pydantic schema (the TypedDict is open, like the v3 one). The on-disk `.zarray` rule that `attributes` belongs in `.zattrs` is unchanged. - filters: [] was rejected ("expected at least one filter"). The spec: "A list of JSON objects providing codec configurations, or null" — an empty list is a list. The Pydantic schema's min_length is dropped. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(zarr-metadata): changelog fragment for zarr-developers#4365 Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(zarr-metadata): link the spec text the v2 open-array change cites Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(zarr-metadata): link the spec text behind every spec statement Every docstring or comment that cites the Zarr spec or a zarr-extensions README now carries a commit-pinned permalink with a line range (zarr-specs fc7dd9c; zarr-extensions 4da7b37, the registry commit the TypeScript port vendors). Unpinned zarr-extensions `tree/main` page links are pinned the same way. Three statements were wrong or stale and are corrected: - zstd: `checksum` was typed required "per the proposed specification" (zarr-specs PR #256, never merged). The published zarr-extensions entry makes it optional ("Should be omitted if false"; schema requires only `level`), so it is now `NotRequired[bool]`. - v3 consolidated metadata was described as "not a spec artifact"; since zarr-specs zarr-developers#373 the core spec names the field and fixes its envelope (core/index.rst L802-L816); the entry format remains a convention. - The v2 array `*Partial` docstring spoke of a "closed shape"; the array document is open (other keys SHOULD be ignored), unlike `.zgroup`. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…developers#4371) Bumps the python-dependencies group with 5 updates: | Package | From | To | | --- | --- | --- | | [numpy](https://github.com/numpy/numpy) | `2.5.2` | `2.5.3` | | [hypothesis](https://github.com/HypothesisWorks/hypothesis) | `6.167.1` | `6.168.0` | | [uv](https://github.com/astral-sh/uv) | `0.12.9` | `0.12.12` | | [towncrier](https://github.com/twisted/towncrier) | `25.8.0` | `26.9.0` | | [ruff](https://github.com/astral-sh/ruff) | `0.16.5` | `0.16.6` | Updates `numpy` from 2.5.2 to 2.5.3 - [Release notes](https://github.com/numpy/numpy/releases) - [Changelog](https://github.com/numpy/numpy/blob/main/doc/RELEASE_WALKTHROUGH.rst) - [Commits](numpy/numpy@v2.5.2...v2.5.3) Updates `hypothesis` from 6.167.1 to 6.168.0 - [Release notes](https://github.com/HypothesisWorks/hypothesis/releases) - [Commits](HypothesisWorks/hypothesis@v6.167.1...v6.168.0) Updates `uv` from 0.12.9 to 0.12.12 - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](astral-sh/uv@0.12.9...0.12.12) Updates `towncrier` from 25.8.0 to 26.9.0 - [Release notes](https://github.com/twisted/towncrier/releases) - [Changelog](https://github.com/twisted/towncrier/blob/trunk/NEWS.rst) - [Commits](twisted/towncrier@25.8.0...26.9.0) Updates `ruff` from 0.16.5 to 0.16.6 - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md) - [Commits](astral-sh/ruff@0.16.5...0.16.6) --- updated-dependencies: - dependency-name: numpy dependency-version: 2.5.3 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: hypothesis dependency-version: 6.168.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: python-dependencies - dependency-name: uv dependency-version: 0.12.12 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: towncrier dependency-version: 26.9.0 dependency-type: direct:development update-type: version-update:semver-major dependency-group: python-dependencies - dependency-name: ruff dependency-version: 0.16.6 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…pers#4368) * fix(ci): preserve branch zarr in xarray downstream tests * chore: use pull request number for changelog fragment --------- Co-authored-by: glaziermag <glaziermag@users.noreply.github.com> Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
* chore: update pre-commit hooks updates: - [github.com/astral-sh/ruff-pre-commit: v0.16.0 → v0.16.6](astral-sh/ruff-pre-commit@v0.16.0...v0.16.6) - [github.com/codespell-project/codespell: v2.4.2 → v2.4.3](codespell-project/codespell@v2.4.2...v2.4.3) - [github.com/DavidAnson/markdownlint-cli2: v0.22.1 → v0.23.2](DavidAnson/markdownlint-cli2@v0.22.1...v0.23.2) - [github.com/scientific-python/cookie: 2026.06.18 → 2026.08.14](scientific-python/cookie@2026.06.18...2026.08.14) - [github.com/zizmorcore/zizmor-pre-commit: v1.26.1 → v1.30.0](zizmorcore/zizmor-pre-commit@v1.26.1...v1.30.0) - [github.com/twisted/towncrier: 25.8.0 → 26.9.0](twisted/towncrier@25.8.0...26.9.0) * style: reformat a README example for ruff 0.16.6 ruff-format v0.16.6 formats Python code blocks in Markdown, and normalizes this inline comment to PEP 8's two spaces. Fallout from the hook bump in this pull request, not a behavior change. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * ci: pin just-version in setup-just steps zizmor v1.30.0 adds the `unpinned-tools` audit, which flags all twelve `extractions/setup-just` steps across the three package workflows: pinning the action by SHA still leaves the action free to install whatever just is newest at run time, so a just release can change CI without a commit here. Pinned to 1.58.0, which is what the action resolves to today, so this changes nothing about the current builds while making that choice explicit. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…zarr-developers#4370) Bumps the actions group with 4 updates in the / directory: [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv), [github-community-projects/issue-metrics](https://github.com/github-community-projects/issue-metrics), [scientific-python/upload-nightly-action](https://github.com/scientific-python/upload-nightly-action) and [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action). Updates `astral-sh/setup-uv` from 10.0.1 to 10.1.0 - [Release notes](https://github.com/astral-sh/setup-uv/releases) - [Commits](astral-sh/setup-uv@20cfd1b...bec219d) Updates `github-community-projects/issue-metrics` from 5.0.1 to 5.0.2 - [Release notes](https://github.com/github-community-projects/issue-metrics/releases) - [Commits](github-community-projects/issue-metrics@61084fa...a7dc2fb) Updates `scientific-python/upload-nightly-action` from 0.6.4 to 0.6.5 - [Release notes](https://github.com/scientific-python/upload-nightly-action/releases) - [Commits](scientific-python/upload-nightly-action@e76cfec...16fa02e) Updates `zizmorcore/zizmor-action` from 0.6.3 to 0.6.4 - [Release notes](https://github.com/zizmorcore/zizmor-action/releases) - [Commits](zizmorcore/zizmor-action@70fb788...cc914d7) --- updated-dependencies: - dependency-name: astral-sh/setup-uv dependency-version: 10.1.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions - dependency-name: github-community-projects/issue-metrics dependency-version: 5.0.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: scientific-python/upload-nightly-action dependency-version: 0.6.5 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: zizmorcore/zizmor-action dependency-version: 0.6.4 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
… validators return tuples (zarr-developers#4378) * fix(zarr-metadata)!: make entity types assignable to metadata fields; validators return tuples `ZarrV3NamedConfigJSON.name` and `.configuration` are `ReadOnly` and the envelope is `closed`, so the concrete codec / chunk-grid / chunk-key-encoding / data-type TypedDicts are assignable to the fields they describe. Every concrete `*Object` / `*Configuration` is `closed` and object forms declare `must_understand: NotRequired[bool]`. Every `validate_*` in `zarr_metadata.model` returns `tuple[ValidationProblem, ...]`, `MetadataValidationError.problems` is a tuple, and `load_store_json` returns `object` rather than `Any`. `ANN401` is enforced package-wide. `ZarrV2ConsolidatedMetadataJSON.zarr_consolidated_format` is `Literal[1]`. Split from #296 (part 1 of 3). Assisted-by: ClaudeCode:claude-fable-5-1 * chore(zarr-metadata): number changelog fragments for #317 Assisted-by: ClaudeCode:claude-fable-5-1 * test(zarr-metadata): expect tuples from validate_array_metadata_v2 in zarr-developers#4365's tests Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore(zarr-metadata): number changelog fragments for zarr-developers#4378 Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
ArrayV3Metadata incorrectly rejects codec chains when an earlier array-to-array codec changes chunk rank or shape. Evolution used the array-level shape, and later validation revisited the original shape instead of the resolved chunk spec.
The new evolve_and_validate_codecs helper threads a chunk ArraySpec through evolution and validation. Metadata no longer repeats the stale array-level validation. Sharding evolves and validates its inner chain with the real inner ArraySpec, including the actual fill value. Codec.validate's public signature stays unchanged; direct ShardingCodec.validate remains the geometry check.
Distinct rectilinear chunk shapes are checked, with Cartesian enumeration capped at 4,096. Above that cap validation warns and continues with a sample; it does not certify unexamined transformed shapes.
Tests cover reshape/transpose round trips, persisted metadata reopening, and invalid chains. A new Hypothesis property varies nonzero unsigned fill values across sharded and unsharded layouts; it caught validation using an invented zero fill value, which rejected a valid uint8 ScaleOffset(offset=1), fill_value=1 array.
Validation against current main: 1,191 codec/metadata/fused/parity tests passed, 17 optional/unsupported skips and 2 optional-backend xfails; all commit hooks including mypy passed. Misconfigured inner chains may now be rejected during construction rather than encoding.