fix(arrow-array): align FFI buffers before validation under force_validate - #10798
fix(arrow-array): align FFI buffers before validation under force_validate#10798bit2swaz wants to merge 2 commits into
Conversation
Jefffrey
left a comment
There was a problem hiding this comment.
how is this change tested if the test already passes on main?
|
it passes on main because arrow-rs/arrow-array/Cargo.toml Line 72 in d6b0bd7 so in the and being honest with you: i really couldnt find a way to write a test that exercises the fix under arrow-rs/arrow-data/src/data.rs Lines 2267 to 2268 in d6b0bd7 you cant source it from a valid array either: to be under aligned for the import type youd need a source type with the same byte width but smaller alignment and no arrow primitive fits that ( what is covered: normal builds run the realignment test. under so i can realistically think of two options: leave it as is with the gate and the comment explaining why, or i hand build the which would you prefer? |
Which issue does this PR close?
from_ffirealignment is bypassed underforce_validate#10034.Rationale for this change
from_ffirealigned under-aligned C Data Interface buffers (e.g. an 8-byte alignedDecimal128from a JVM producer) afterconsume(). underforce_validate,consume()'sbuild()validates first and rejects the buffer before the realign runs, so spec-legal input errors. reachable via thearrowcrate withfeatures = ["force_validate", "ffi"]callingarrow::ffi::from_ffi.What changes are included in this PR?
ImportedArrowArray::consumebuilds throughArrayDataBuilderwithalign_buffers(true)before validation, matchingarrow-ipc'screate_array_from_builderalign_buffers()calls infrom_ffi/from_ffi_and_data_type.Are these changes tested?
covered by
test_decimal128_under_aligned_round_trip. the issue suggested ungating it underforce_validate, but that isn't possible as its fixture is a misalignedArrayDatabuilt withbuild_unchecked, which validates underforce_validateand so rejects the input at construction, beforefrom_ffiruns. the gate stays with a comment explaining whyAre there any user-facing changes?
no public API change. behavior only changes under
force_validate, where spec-legal under-aligned input is realigned instead of erroring.