Skip to content

fix(arrow-array): align FFI buffers before validation under force_validate - #10798

Open
bit2swaz wants to merge 2 commits into
apache:mainfrom
bit2swaz:fix/ffi-align-before-validate
Open

fix(arrow-array): align FFI buffers before validation under force_validate#10798
bit2swaz wants to merge 2 commits into
apache:mainfrom
bit2swaz:fix/ffi-align-before-validate

Conversation

@bit2swaz

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

from_ffi realigned under-aligned C Data Interface buffers (e.g. an 8-byte aligned Decimal128 from a JVM producer) after consume(). under force_validate, consume()'s build() validates first and rejects the buffer before the realign runs, so spec-legal input errors. reachable via the arrow crate with features = ["force_validate", "ffi"] calling arrow::ffi::from_ffi.

What changes are included in this PR?

  • ImportedArrowArray::consume builds through ArrayDataBuilder with align_buffers(true) before validation, matching arrow-ipc's create_array_from_builder
  • dropped the now-redundant outer align_buffers() calls in from_ffi / from_ffi_and_data_type.

Are these changes tested?

covered by test_decimal128_under_aligned_round_trip. the issue suggested ungating it under force_validate, but that isn't possible as its fixture is a misaligned ArrayData built with build_unchecked, which validates under force_validate and so rejects the input at construction, before from_ffi runs. the gate stays with a comment explaining why

Are there any user-facing changes?

no public API change. behavior only changes under force_validate, where spec-legal under-aligned input is realigned instead of erroring.

@github-actions github-actions Bot added arrow Changes to the arrow crate arrow-array labels Aug 22, 2026

@Jefffrey Jefffrey left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how is this change tested if the test already passes on main?

@bit2swaz

Copy link
Copy Markdown
Contributor Author

it passes on main because arrow-array's force_validateis empty and doesnt turn on the validation that lives in arrow-data:

force_validate = []

so in the arrow-array test job the misaligned buffer is never rejected, with or without this change. the validation only actually runs when arrow-data/force_validate is on which only happens thru the arrow meta-crate's force_validate (it forwards to both crates) and that job doesnt run arrow-array's unit tests. so factually speaking today no CI config runs this test with the validation enabled

and being honest with you: i really couldnt find a way to write a test that exercises the fix under arrow-data/force_validate. the test has to hand from_ffi a buffer thats under aligned for its type. the only way to get one is an ArrayData from build_unchecked but under force_validate that validates anyway (the gate ignores skip_validation), so the fixture is rejected at construction, before from_ffi runs

if !skip_validation.get() || cfg!(feature = "force_validate") {
data.validate_data()?;

you cant source it from a valid array either: to be under aligned for the import type youd need a source type with the same byte width but smaller alignment and no arrow primitive fits that (align_of == size_of). so the misaligned input is genuinely unconstructable in a force_validate build

what is covered: normal builds run the realignment test. under arrow-data/force_validate the full ffi test module still passes since valid arrays go through the same align-before-validate path in consume so the reorder doesnt break anything. the one gap is the under aligned case under force_validate which is the case that cant be built based on my research

so i can realistically think of two options: leave it as is with the gate and the comment explaining why, or i hand build the FFI_ArrowArray from a raw under aligned pointer in the test to bypass ArrayData entirely and get real coverage, but at the cost of more unsafe test code.

which would you prefer?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

arrow Changes to the arrow crate arrow-array

Projects

None yet

Development

Successfully merging this pull request may close these issues.

from_ffi realignment is bypassed under force_validate

2 participants