fix(dtype): accept platform-equivalent numpy dtypes in from_native_dtype

[Yi-111-a]

Fixes #3282.

ZDType._check_native_dtype in src/zarr/core/dtype/wrapper.py tested the dtype class with

return type(dtype) is cls.dtype_cls

That is too strict for NumPy's C type-name spellings. np.dtype("q") is a np.dtypes.LongLongDType instance, not a np.dtypes.Int64DType instance, even though long long is 64-bit on the platform NumPy was built for. The result is that a plain 64-bit signed integer is rejected:

>>> import zarr
>>> zarr.create_array(store={}, shape=(3,), dtype="q")
ValueError: No Zarr data type found that matches dtype 'dtype('int64')'

while two other spellings of the identical layout work fine:

>>> zarr.create_array(store={}, shape=(3,), dtype="<i8").metadata.data_type
Int64(endianness='little')
>>> zarr.create_array(store={}, shape=(3,), dtype="l").metadata.data_type
Int64(endianness='little')

The width of long long depends on the build platform, so the dtype class on its own is not a reliable test. What a Zarr data type actually encodes is the layout.

Change

Keep the exact type check as the fast path, and fall back to comparing kind and itemsize:

if type(dtype) is cls.dtype_cls:
    return True
try:
    expected = cls.dtype_cls()
except TypeError:
    # Flexible/parametric dtypes (e.g. VoidDType) have no fixed layout
    # to compare against, so the exact dtype class check above stands.
    return False
return dtype.kind == expected.kind and dtype.itemsize == expected.itemsize

np.dtype("Q") (ULongLongDType vs UInt64DType) is fixed by the same change, as is np.dtype("l") on a platform where long is 32-bit.

The except TypeError branch matters: VoidDType, DateTime64DType and TimeDelta64DType cannot be instantiated (Preliminary-API: Flexible/Parametric legacy DType), and DateTime64 relies on the base _check_native_dtype. Without that guard those classes would start raising TypeError where they previously returned False. Structured overrides _check_native_dtype outright and is unaffected.

Why this is safe

The registry requires exactly one wrapper to match a dtype, so a looser test could in principle create ambiguity. Comparing on (kind, itemsize) cannot, because that pair is unique across every registered wrapper:

('b', 1) Bool        ('f', 2) Float16    ('i', 1) Int8     ('u', 1) UInt8
                     ('f', 4) Float32    ('i', 2) Int16    ('u', 2) UInt16
('c', 8) Complex64   ('f', 8) Float64    ('i', 4) Int32    ('u', 4) UInt32
('c', 16) Complex128                    ('i', 8) Int64    ('u', 8) UInt64
('O', 8) VariableLengthBytes           ('T', 16) VariableLengthUTF8

Verification

I diffed the complete resolution behaviour before and after the patch, over every C spelling, every explicit byte-order/width spelling, float128/complex256, string, bytes, void, object and datetime dtypes, structured dtypes, and every registered wrapper asked about every one of those dtypes (2065 recorded results). The only differences are the two intended ones:

- np.dtype("q") : no match, create_array -> ValueError
+ np.dtype("q") : Int64(endianness='little')
- np.dtype("Q") : no match, create_array -> ValueError
+ np.dtype("Q") : UInt64(endianness='little')

Everything else is byte-identical, including:

• float128 and complex256 still resolve to nothing (no Zarr data type has that layout), so they still raise rather than silently becoming Float64/Complex128.
• object, fixed-width bytes, null-terminated bytes, structured and datetime dtypes are unchanged.
• Each of q and Q still matches exactly one wrapper, so the "multiple wrappers match" error cannot trigger.

Test runs:

• pytest tests/test_dtype/ tests/test_dtype_registry.py — 1948 passed, 8 skipped
• ruff check and ruff format --check on the three changed files — clean

The new coverage fails on main and passes with the patch:

• tests/test_dtype/test_npy/test_int.py: np.dtype("q") and np.dtype("Q") added to the TestInt64 / TestUInt64 valid_dtype tuples, which the shared BaseTestZDType drives through _check_native_dtype and from_native_dtype.
• tests/test_dtype_registry.py: a new test_match_dtype_c_type_spelling asserting that both spellings resolve to exactly one data type through match_dtype and parse_dtype for Zarr formats 2 and 3.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.37%. Comparing base (f58644a) to head (f5f5a58).
⚠️ Report is 14 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4416   +/-   ##
=======================================
  Coverage   94.37%   94.37%           
=======================================
  Files          93       93           
  Lines       13171    13181   +10     
=======================================
+ Hits        12430    12440   +10     
  Misses        741      741           

Files with missing lines	Coverage Δ
src/zarr/core/dtype/wrapper.py	98.18% <100.00%> (+0.26%)	⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[d-v-b]

i'm not really convinced we need to do this. the whole idea of zarr is to save data in a platform-agnostic way. good for numpy that they have many ways of saying the same thing, but I think it's OK to not copy that.

[d-v-b]

in fact I wonder if we should actually reject numpy data types that are platform-dependent

[Yi-111-a]

Thanks for the review, @d-v-b. One clarification that might change the picture: this doesn't make Zarr copy NumPy's aliasing into what it stores. The data type that ends up in the metadata is still the platform-agnostic Int64 (or whichever fixed-layout type matches). The PR only changes which native input spellings resolve to it, by comparing kind and item size instead of the dtype class.

Today the behaviour is already inconsistent. On the same machine, dtype="l" and dtype="<i8" resolve to Int64, but dtype="q" (whose repr is also dtype('int64')) raises No Zarr data type found that matches dtype 'dtype('int64')'. That error message is what #3282 reported.

On rejecting platform-dependent dtypes: that would be the stricter and more explicit option, but it's a behaviour change. "l", "i" and np.dtype(int) all work today. If you'd prefer that direction, I'm happy to close this, or to rework it into a clear error that names the fixed-width spelling to use. Let me know which one you want.

(I also pushed a fix for the mypy failure in Lint.)

[read-the-docs-community]

Documentation build overview

📚 zarr-metadata | 🛠️ Build #34789784 | 📁 Comparing f5f5a58 against latest (8dc50b8)

  🔍 Preview build  

10 files changed · ± 8 modified · - 2 deleted

± Modified

• api/index.html
• api/model/index.html
• api/pydantic/index.html
• api/v2/index.html
• api/v3/chunk_grid/index.html
• api/v3/chunk_key_encoding/index.html
• api/v3/codec/index.html
• api/v3/data_type/index.html

- Deleted

• api/typed_json/index.html
• api/v3/definition/index.html

[d-v-b]

The width of long long depends on the build platform

this is the concerning part. if I understand correctly, with these data types, the same python script that writes data from numpy arrays will produce different zarr data, on different platforms. I think we should avoid this, and only accept numpy data types that are platform-independent. as you note this would require a behavioral change, because we do accept some of these platform-sensitive data types today. we would need to implement this change through a deprecation cycle, which I'm open to.

cc @zarr-developers/python-core-devs, thoughts on having this library adhere to the following principle: when coercing numpy data types to zarr data types, we reject ambiguous numpy data types that have different widths on different platforms, and encourage users to use a platform-independent numpy data type instead.

[dcherian]

when coercing numpy data types to zarr data types, we reject ambiguous numpy data types that have different widths on different platforms, and encourage users to use a platform-independent numpy data type instead.

I agree