feat(zarr-metadata)!: a fill value is judged by the data type it fills

[d-v-b]

🤖 AI text below 🤖

The first of four pieces of layer 4, the field among fields, on top of layer 3 (#4436): a v3 array's fill_value is judged against its data_type. The other three pieces follow as draft PRs, each depending on the one before: 4b the grid against the shape, 4c the codec pipeline, 4d sharding.

document["data_type"] = "int8"
document["fill_value"] = 300
validate_array_metadata_v3(document)
# (ValidationProblem(loc=('fill_value',), message='expected an integer in [-128, 127], got 300', kind='invalid_value'),)

What a data type knows. DataTypeDefinition gains fill_value, the JSON shape of a fill value as an annotation the checker reads (each module's existing <X>FillValue alias), and fill_value_rules(configuration, nested, value), what the spec disallows in a fill value of that shape. It mirrors configuration plus rules: the type check stays declarative, the rules are plain functions in the extension's module. Every package data type has both:

• integers within the type's range; bool a boolean; string a string;
• floats a number, "NaN", "Infinity", "-Infinity", or a hex string of the type's own width; complex a pair of those, judged by the component type's own rules;
• raw bits one byte value for each 8 bits (r16 takes two); bytes byte values or base64;
• the numpy time types a signed 64-bit integer or "NaT";
• struct a fill value for every field, each judged by that field's own type, and none for a name no field has (unknown_key).

A data type that says nothing of its fill value takes any JSON, and one nothing in scope claims leaves its fill value unjudged.

What a read field keeps. Resolved.nested holds the fields a configuration holds, as the scope read them, by where each sits in the configuration: a struct's field types at ("fields", i, "data_type"), a shard's codecs, a cast's target. A definition's functions receive it (Nested), which is how a struct reaches its field types. canonicalize now spells nested fields from it rather than reading each subtree again.

Where it is judged. fill_value_problems(data_type, value) is public: not JSON is its first verdict, then the shape, then the rules, which see past a key the shape does not declare. validate_array_metadata_v3 uses it, and so from_json, from_key_value, to_key_value, the pydantic types and each array in an inline consolidated_metadata.

Breaking. A document whose fill value its data type refuses, which the package accepted, now has a problem at fill_value: "NaN" on uint8, 0 on bool. Two tests did exactly that and now use a fitting type; create_default says an overridden data_type needs its own fill_value.

Choices worth a look

• A float fill value takes any number: the spec says a reader rounds it to the nearest representable value. feat(zarr-metadata): composition rules layer, shape-exact entity validators #4379 refused numbers beyond the type's largest finite value; this does not.
• r<N> takes N/8 byte values. The spec's text says N, but N counts bits (r16 is two bytes), which reads as a spec defect.
• A base64 string with nonzero padding bits ("AB==") is accepted, as base64_bytes on main accepts it.
• zarr-python is more lenient than this in several places (a bool fill judged by truthiness, integral floats and numeric strings for integers, float hex of any width, str() of anything for string, a struct fill missing fields) and refuses r* entirely.

Reviews. Two, with separate lenses. The correctness review compared 25,000 generated fill values against a reference written from the spec text (no mismatches) and found four bugs, each fixed with a test: the JSON check took two stack frames per level, so a struct fill value 350 deep raised RecursionError (it now walks one frame per level, as refine_json does); the rule factories returned closures, so definitions stopped pickling (they are partial applications of module-level functions); an undeclared key hid the rules' findings; a hand-built struct reading without nested raised KeyError. The design review's changes are in too: one annotation-derived check that a definition's function members are functions, fill_value_problems owning the JSON check, complex rules reusing the float rules, a plain-dict nested, and the canonicalize cut above.

The stack. Layers 0 (#4420, #4421, #4422), 1 (#4432), 2 (#4434) and 3 (#4436) have merged. This is 4a; 4b (#4441), 4c (#4442) and 4d (#4443) follow.

🤖 Generated with Claude Code

[read-the-docs-community]

Documentation build overview

📚 zarr-metadata | 🛠️ Build #34798145 | 📁 Comparing 4c24bb4 against latest (0401a7f)

  🔍 Preview build  

4 files changed

± index.html
± api/model/index.html
± api/v3/data_type/index.html
± api/v3/definition/index.html

[d-v-b]

🤖 AI text below 🤖

This landed with #4443, which carried its commits and merged all of layer 4. Nothing is left to merge here. #4444 renames this PR's changelog fragment to #4443.