debug_assertions changes written file bytes: the is_sorted assert on patch indices caches a statistic

[gerchowl]

Summary

Writing the same array with the same session and strategy produces different file bytes depending
only on whether debug_assertions is enabled. The compression is identical — same chosen encodings,
same compressed sizes — but the container metadata differs, and the file is 24 bytes longer in a debug
build.

Still present on develop (checked at the current tip).

Root cause

Patches::new validates its indices under #[cfg(debug_assertions)]
(vortex-array/src/patches.rs):

#[cfg(debug_assertions)]
{
    use crate::aggregate_fn::fns::is_sorted::is_sorted;
    let mut ctx = legacy_session().create_execution_ctx();
    assert!(
        is_sorted(&indices, &mut ctx).unwrap_or(false),
        "Patch indices must be sorted"
    );
}

is_sorted is not observationally pure: is_sorted_impl ends in cache_is_sorted, which does

array_stats.set(Stat::IsSorted, Precision::Exact(true.into()));

so the check mutates the array it inspects. Cached statistics are serialised into the written
file, so an assertion changes the bytes that are later written.

A test at 0.75.0 confirms the annotation directly — after Patches::new, the indices array's
statistics set has gained exactly one entry and nothing else:

left:  ["IsSorted=Exact(Bool(true))"]
right: []

Reproducer

let arr = Buffer::copy_from([i64::MIN, 0, i64::MAX].as_slice()).into_array();
let mut buf = ByteBufferMut::empty();
rt.block_on(
    s.write_options()
        .with_strategy(WriteStrategyBuilder::default().build())   // stock strategy
        .write(&mut buf, arr.to_array_stream()),
).unwrap();
println!("{}", buf.freeze().len());

build	bytes
cargo run	2796
cargo run --release	2772

Stable within a profile over repeated runs. [i64::MIN, 0, i64::MAX] zigzags to
[u64::MAX, 0, u64::MAX - 1], which bitpacks with exactly one exception — hence patches. Controls
that produce no patches are byte-identical in both profiles: [i64::MIN, i64::MAX] (2444) and
[-1, 0, 1] (2228).

The axis is debug_assertions alone. Holding the profile and target dir fixed and toggling only the
cargo knobs:

build	debug-assertions	overflow-checks	bytes
release (default)	off	off	2916
release	on	on	2940
release	on	off	2940
release	off	on	2916
debug (default)	on	on	2940
debug	off	off	2916

(Struct-wrapped numbers; the bare-array pair moves identically.) So this is not UB and not codegen.

Also ruled out by measurement

• Not scheme selection. vortex_compressor::encode at TRACE is identical in both profiles: same
  winner chain vortex.int.zigzag -> vortex.int.bitpacking, input_nbytes=24,
  compressed_nbytes=19, estimated_ratio = achieved_ratio, accepted=true.
• Not hash-iteration order. Output is stable across repeated runs of each binary, and the
  encoding-id string table appears in the same order.
• Not a consumer customisation. Reproduces with the stock WriteStrategyBuilder::default().

The debug trace also contains a rewrite the release trace does not —
rewrote vortex.cast(u8, len=1) -> vortex.primitive(u8, len=1) — i.e. the indices array is
materialised in debug, which is what evaluating that aggregate causes.

Why it matters

We content-address written files: a blake3 root over the bytes is the artifact's identity. A byte
difference that is a function of the build configuration rather than of the data breaks that identity,
and makes a build-configuration change indistinguishable from a data change. Concretely, our CI
disagreed with itself because one check runs the writer in the dev profile and another in release.

The narrow reading needs no opinion on content-addressing, though: an assertion should not be able
to change the output.

Fix

PR to follow: add an is_sorted_uncached that computes sortedness without recording it, and use that
for the assert. The assert keeps its protective value, and is_sorted/is_strict_sorted keep caching
for query callers.

Happy to take a different shape if you would rather the writer not serialise incidentally-computed
statistics at all — that would also fix it, and would cover this whole class rather than one site.

Downstream tracking: vig-os/tessera#468.