feat(zarr-metadata)!: a shard is judged against the chunk it is handed, and its pipelines are read

packages/zarr-metadata/README.md

@@ -56,17 +56,20 @@ members that the strict model parser rejects.
 
 The model validators enforce the declared document structure and a small set
 of context-free consistency rules, including fixed format literals, finite
-JSON numbers, non-negative dimensions, non-empty v3 codec pipelines, and one
-`dimension_names` entry per array dimension. In a v3 document they also read
+JSON numbers, non-negative dimensions, and one `dimension_names` entry per
+array dimension. In a v3 document they also read
 each extension point -- the data type, chunk grid, chunk key encoding, each
 codec and each storage transformer -- through the definition that claims its
 name in a scope, `CORE_AND_EXTENSIONS` unless a `context` is passed: a
 configuration its definition refuses is refused, and a key it does not
 declare is reported as `unknown_key`. A name nothing in the scope claims is
-left unjudged, and whether to support it is the consumer's decision. The
-validators do not judge fields against each other: a fill value against its
-data type, a codec against the array it is handed, a chunk grid against the
-shape.
+left unjudged, and whether to support it is the consumer's decision. A v3
+fill value is judged against the data type it names, by that data type's
+definition, the chunk grid against the shape, by the grid's definition,
+and the codecs as a pipeline: in order, each judged by its definition
+against the chunk it is handed, a shard's inner and index codecs too.
+The validators do no arithmetic on values: whether a fill value survives
+a `cast_value` round trip is not judged.
 
 The Pydantic integration's generated JSON Schemas express independently
 checkable document structure and field constraints, but they are not a

packages/zarr-metadata/changes/4440.feature.md

@@ -0,0 +1,10 @@
+**Breaking:** a v3 array's `fill_value` is judged against its `data_type`.
+Each data type's definition declares the JSON shape of its fill value,
+`fill_value`, and the rules for one of that shape, `fill_value_rules`: an
+`int8` fill value of 300, a `float32` hex string of another width, and a
+struct fill value missing a field are each a problem at `fill_value`,
+where the package accepted them before. `fill_value_problems(data_type,
+value)` judges a fill value against a data type field a scope read, and
+a field that is read keeps the fields it read inside as
+`Resolved.nested`, a `Nested` mapping by location. A data type nothing
+in scope claims leaves its fill value unjudged.

packages/zarr-metadata/changes/4441.bugfix.md

@@ -0,0 +1,3 @@
+`ZarrV3ArrayMetadata.create_default(shape=...)` derives a chunk length of 1
+for a dimension of length 0, where it wrote 0: the regular grid asks for
+chunk lengths greater than zero, and `zarr` does not open a grid with one.

packages/zarr-metadata/changes/4441.feature.md

@@ -0,0 +1,7 @@
+**Breaking:** a v3 array's `chunk_grid` is judged against its `shape`, by
+the grid's definition. A regular grid whose `chunk_shape` does not have
+one length per dimension of the shape, or has a length of 0 for a
+dimension that is not empty, and a rectilinear grid whose `chunk_shapes`
+does not have one entry per dimension, or whose chunk lengths fall short
+of their dimension, each have a problem in `chunk_grid.configuration`,
+where the package accepted them before.

packages/zarr-metadata/changes/4442.feature.md

@@ -0,0 +1,20 @@
+**Breaking:** a v3 array's `codecs` are read as a pipeline: in order --
+array -> array codecs, then one array -> bytes codec, then bytes -> bytes
+codecs -- and each codec judged against the chunk it is handed, which is
+the grid's chunks of the array's data type, as the array -> array codecs
+before it hand them on. A codec out of order, a second array -> bytes
+codec or none at all; a `bytes` codec without an `endian` handed numbers
+of several bytes, or handed values that vary in size; a `transpose` whose
+`order` has another number of axes than its chunk; a `cast_value` to or
+from a data type that models no real numbers, wrapping to one that is
+not integral, or mapping a scalar that is not a fill value of the data
+type on its side; a `scale_offset` handed values that are no numbers, or
+with an `offset` or `scale` that is not a value of its data type; and a
+struct field whose values vary in size are each a problem where they
+sit, where the package accepted them before. `read_pipeline(codecs,
+chunk)` gives each codec with the chunk it is handed,
+`chunk_grid_lengths(grid, shape)` the lengths a grid's chunks take, and
+`storage_of(data_type)` how a data type's values are stored. A
+definition's `rules` are handed the fields its configuration holds as
+the scope read them, and the kinds gain `chunk_lengths` (grids),
+`storage` (data types), and `chunk_rules` and `transition` (codecs).

packages/zarr-metadata/changes/4443.feature.1.md

@@ -0,0 +1,5 @@
+**Breaking:** a problem of a field a configuration holds is that field's
+own, reported where it sits, and the field holding it is read when its
+own configuration and rules are sound -- as a document's fields are. A
+codec a shard holds with a `level` out of range, or a `must_understand`
+of false, no longer hides the shard's other problems.

packages/zarr-metadata/changes/4443.feature.md

@@ -0,0 +1,13 @@
+**Breaking:** a v3 array's sharding codec is judged against the shard it
+is handed, and its two pipelines are read as the array's is. Its
+`chunk_shape` has an inner chunk length for each axis of the shard,
+dividing each length the shards take along it -- the shard being the
+chunk the codec is handed, so a transposed shard is divided along its
+transposed axes. Its inner codecs are handed the inner chunks, of the
+shard's data type, and its index codecs the shard index, of `uint64`,
+with an axis more than the shard. A shard that does not divide, or a
+pipeline that does not fit what it is handed -- an index `bytes` codec
+without an `endian` -- has a problem where it sits, where the package
+accepted it before. A codec definition
+says what the pipelines it holds are handed, `pipelines`, and each
+codec's `Stage` keeps the stages of those pipelines as `inner`.

packages/zarr-metadata/docs/api/index.md

@@ -21,8 +21,9 @@ The package is organized to mirror the structure of the Zarr specifications:
   and [data types](v3/data_type.md)
 - [`zarr_metadata.v3.definition`](v3/definition.md) — each extension's
   metadata as a definition: the TypedDict its configuration is, and the
-  rules on it; check JSON against a TypedDict, judge a configuration, or
-  read a whole field in a scope. Its module docstring is the guide
+  rules on it; check JSON against a TypedDict, judge a configuration,
+  read a whole field in a scope, or read a codec pipeline. Its module
+  docstring is the guide
 
 The document types, models, and spec vocabulary — including the store keys —
 are re-exported at the top level, so

packages/zarr-metadata/docs/index.md

@@ -71,17 +71,20 @@ members that the strict model parser rejects.
 
 The model validators enforce the declared document structure and a small set
 of context-free consistency rules, including fixed format literals, finite
-JSON numbers, non-negative dimensions, non-empty v3 codec pipelines, and one
-`dimension_names` entry per array dimension. In a v3 document they also read
+JSON numbers, non-negative dimensions, and one `dimension_names` entry per
+array dimension. In a v3 document they also read
 each extension point -- the data type, chunk grid, chunk key encoding, each
 codec and each storage transformer -- through the definition that claims its
 name in a scope, `CORE_AND_EXTENSIONS` unless a `context` is passed: a
 configuration its definition refuses is refused, and a key it does not
 declare is reported as `unknown_key`. A name nothing in the scope claims is
-left unjudged, and whether to support it is the consumer's decision. The
-validators do not judge fields against each other: a fill value against its
-data type, a codec against the array it is handed, a chunk grid against the
-shape.
+left unjudged, and whether to support it is the consumer's decision. A v3
+fill value is judged against the data type it names, by that data type's
+definition, the chunk grid against the shape, by the grid's definition,
+and the codecs as a pipeline: in order, each judged by its definition
+against the chunk it is handed, a shard's inner and index codecs too.
+The validators do no arithmetic on values: whether a fill value survives
+a `cast_value` round trip is not judged.
 
 ## Scope
 

packages/zarr-metadata/src/zarr_metadata/_json.py

@@ -200,20 +200,23 @@ def is_canonical_json(value: object, *, finite: bool = True) -> TypeGuard[JSONVa
 
     A non-finite number counts only when `finite` is false, as a document's
     guard passes it: where one may be is the document's validator's to say.
+    One frame per level of nesting, as `refine_json` takes, so a value
+    `refine_json` reads is one this can walk.
     """
     if isinstance(value, float):
         return not finite or math.isfinite(value)
     if isinstance(value, (str, int, bool)) or value is None:
         return True
     if isinstance(value, (list, tuple)):
-        sequence = cast("list[object] | tuple[object, ...]", value)
-        return all(is_canonical_json(item, finite=finite) for item in sequence)
+        for item in cast("list[object] | tuple[object, ...]", value):
+            if not is_canonical_json(item, finite=finite):
+                return False
+        return True
     if isinstance(value, dict):
-        mapping = cast("dict[object, object]", value)
-        return all(
-            isinstance(key, str) and is_canonical_json(item, finite=finite)
-            for key, item in mapping.items()
-        )
+        for key, item in cast("dict[object, object]", value).items():
+            if not isinstance(key, str) or not is_canonical_json(item, finite=finite):
+                return False
+        return True
     return False
 
 

packages/zarr-metadata/src/zarr_metadata/model/__init__.py

@@ -4,14 +4,17 @@
 representation of the JSON documents. Validators check a document's JSON
 structure and, in a v3 document, read each extension point (codecs, chunk
 grids, data types, ...) through the definition that claims its name in a
-scope, `CORE_AND_EXTENSIONS` unless a `context` is passed; they do not judge
-fields against each other. Each document concept gets a `validate_*`
-function returning every problem found (a tuple of `ValidationProblem`, each
-with a machine-readable `kind`), an `is_*` type guard, and a `parse_*`
-function that narrows or raises `MetadataValidationError`. Model
-`from_json` / `from_key_value` constructors raise `MetadataValidationError`
-for every ingestion failure, including missing store keys and undecodable
-bytes, and the v3 ones take the same `context`.
+scope, `CORE_AND_EXTENSIONS` unless a `context` is passed, and judge the
+fill value against the data type it names, the chunk grid against
+the shape, and the codecs as a pipeline, each against the chunk it is
+handed. Each document concept gets a
+`validate_*` function returning every problem found (a tuple of
+`ValidationProblem`, each with a machine-readable `kind`), an `is_*` type
+guard, and a `parse_*` function that narrows or raises
+`MetadataValidationError`. Model `from_json` / `from_key_value` constructors
+raise `MetadataValidationError` for every ingestion failure, including
+missing store keys and undecodable bytes, and the v3 ones take the same
+`context`.
 """
 
 from zarr_metadata._json import (

packages/zarr-metadata/src/zarr_metadata/model/_array.py

@@ -191,19 +191,28 @@ def create_default(cls, **overrides: Unpack[ZarrV3ArrayMetadataPartial]) -> Zarr
         analog of `list()` returning `[]`. Any field can be overridden by keyword
         (the same fields accepted by `update`). Overriding `shape` without
         `chunk_grid` derives a consistent default grid: one regular chunk
-        covering the array (`chunk_shape` equal to `shape`).
+        covering the array (`chunk_shape` equal to `shape`, with a length of
+        1 for a dimension of length 0, since a chunk length is at least 1:
+        https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/chunk-grids/regular-grid/index.rst#L40).
 
         The derivation is deliberately one-way. A user-supplied `chunk_grid`
         is an extension point and is taken verbatim — deriving `shape` from
         it would require interpreting the grid's configuration, which this
         layer never does (and cannot do for unrecognized grid names). So
         overriding `chunk_grid` without `shape` keeps the scalar default
-        `shape=()`, and consistency between the two is the caller's
-        responsibility.
+        `shape=()`, which a grid of another rank does not fit: consistency
+        between the two is the caller's responsibility, so pass them
+        together. So is a fill value for an overridden `data_type`:
+        the default `fill_value` is `0`, which a data type whose fill value
+        is not an integer -- `bool`, `string`, a complex or struct type --
+        refuses, so pass the two together; and so are its codecs: the
+        default `bytes` codec has no `endian`, which a data type whose
+        values take several bytes needs.
         """
         if "shape" in overrides and "chunk_grid" not in overrides:
+            chunk_shape = tuple(max(length, 1) for length in overrides["shape"])
             overrides["chunk_grid"] = ZarrV3NamedConfig(
-                name="regular", configuration={"chunk_shape": tuple(overrides["shape"])}
+                name="regular", configuration={"chunk_shape": chunk_shape}
             )
         default = cls(
             shape=(),