Format 1.4: compact entry metadata and key-less pages for exact digests

[hadashiA]

Summary

Storage format 1.4: shrinks the fixed per-entry cost of every B+Tree page with two new page-level layouts, both written by default and gated per page by NodeFlags.

CompactMeta

Entry metadata becomes an absolute ushort offsets[entry_count + 1] array; every length is derived from adjacent offsets, and the final slot closes the last entry's payload.

• Leaf pages that store keys append ushort key_lengths[entry_count] (the key/value boundary is the one thing offsets cannot derive): meta 8B → 4B per entry.
• Internal pages need no key lengths at all — the payload per entry is key + 8-byte child ordinal, so the key length falls out of the offsets: meta 6B → 2B per entry.
• Bit 15 of a leaf offset flags an overflow entry (inline payload = 8-byte blob page ordinal), replacing the 0xFFFF value-length sentinel. This is why the layout requires pageSize <= 32767; larger pages keep the classic meta records automatically.

OmittedKeys

When the key encoding declares the new IKeyEncoding.IsKeyDigestExact (Int64: the sign-flipped digest is a bijection), pages store no key bytes at all:

• The digest array doubles as the key column: every key comparison reduces to a digest comparison, and enumeration (RangeIterator.CurrentKey) reconstructs the original bytes via the new IKeyEncoding.TryDecodeKeyFromDigest.
• Leaf entries: digest 8 + offset 2 + value. Internal pages drop their meta area entirely — the payload is a dense array of 8-byte child ordinals (digest 8 + child 8 = 16B per entry).
• Never combined with Eytzinger digests (entry i's sorted-order digest must be addressable for reconstruction), and digests are now computed once when entries are appended instead of re-derived from the page buffer at flush time — which is what allows the key bytes to be absent.

An Int64 entry with an 8-byte value drops from 32B to 18B of fixed cost; a 4KB leaf holds ~227 entries instead of ~128, which widens fanout and roughly doubles cache/page density.

Digests are mandatory (option and capability flag both removed)

With 1.4 the DatabaseBuilder.KeyDigests switch no longer represented a real trade-off: for exact-digest encodings the digest column replaces the key bytes, so disabling digests made files bigger (12B vs 10B fixed cost per Int64 entry) and lookups slower at the same time. Going further, the capability flag IKeyEncoding.SupportsKeyDigest is removed too: GetKeyDigest is now a required, order-preserving member, and every tree page carries a digest array — "the key column is always separated" becomes a format invariant rather than a per-table mode.

• Uuidv7 gains a real digest (the first three Guid fields packed in Guid.CompareTo comparison order — leading with the UUIDv7 timestamp), and Ulid gains the big-endian first 8 bytes (byte-lexicographic order). Both are covered by a new pairwise order-preservation contract test.
• Orders that cannot spread keys into 64 bits (collation rules) should normalize at encode time; keys that collide in their leading bytes still work and fall back to full comparisons (covered by the shared-prefix test).
• The readers drop the digest-less search paths entirely: the HasKeyDigests page flag (bit 8, formats 1.1–1.3) is retired and the per-search hasKeyDigest plumbing is gone.
• PmcProbe drops its no-digest row (that layout no longer exists; the historical 1.3 numbers stay in its README).

Compatibility

• The builder writes format 1.4 and the reader accepts 1.4 only: because readers now assume the digest area on every page, pre-1.4 files (which may lack it) are rejected — strict single-version, as 1.3 was in its day.
• IsKeyDigestExact / TryDecodeKeyFromDigest are default interface members. Non-unique secondary indexes (rid-suffixed composite keys) have inexact digests and keep their keys.
• Source-breaking, folded into the same release as the format break: DatabaseBuilder.KeyDigests is removed, IKeyEncoding.SupportsKeyDigest is removed, and IKeyEncoding.GetKeyDigest loses its default implementation (custom encodings must now provide an order-preserving digest).

Measured (Apple M5, .NET 10, same-machine ShortRun; µs / 1000 lookups)

Benchmark	main	this branch
FindByKey (10k, fixed key)	14.8	17.3–18.2
FindByKey_RandomKeys (10k)	23.7	22.1
FindByKey_RandomKeys_NoRepeat (10k)	53.2	47.9 (−10%)
FindByKey_ParallelSpread (8 threads)	111.4	99.7 (−11%)
FindByKey_1M (fixed key)	22.4–26.1	21.4–22.9
FindByKey_1M_RandomKeys	61.3	48.6 (−21%)
GetRange_1M	40.3–43.5	41.9–46.0
OpenAndFirstRead	26.1 (19.1KB alloc)	17.3 (−34%, 13.6KB alloc)

The hot search loop only touches the digest array, so per-probe speed is unchanged; the wins come from density (more entries per page/cache line), and grow with working-set size and randomness. Fixed-key rows are known to swing ±20–40% between builds from code alignment; the 10k fixed-key delta is within that band while 1M fixed is at parity (the second commit fuses the adjacent offset loads into one 4-byte read, which recovered the initial fixed-key/range regression).

File size (pageSize 4096, Int64 keys):

Dataset	main	this branch
10k × 8B values	325,312 B	182,528 B (−44%)
10k × 100B values	1,296,792 B	1,166,778 B (−10%)
1M × 8B values	32,523,742 B	18,241,064 B (−44%)

Tests

95/95 passing. New CompactMetaTest covers: Int64 omitted-keys flags + full lookups over negative/positive keys, iterator key reconstruction from digests, overflow entries through the bit-15 path (point + range reads), ASCII compact pages (keys kept), the pageSize > 32767 fallback to classic records, Eytzinger pages keeping their keys, and the file-size shrink. KeyDigestContractTest verifies the order-preserving digest contract pairwise over randomized keys for i64/ascii/uuidv7/ulid, plus i64 digest round-tripping. Two existing assertions changed only because leaves now hold more entries (4 instead of 3 in the 128-byte-page fixture); the digest-less-page test was removed together with the code path it covered.

🤖 Generated with Claude Code