perf(diff): skip summary stats for the compare grid's hidden columns

[paddymul]

Stacked on #309. The base is feat/diff-parquet-view so the diff shows only this change; retarget to main when #309 merges.

Problem

Buckaroo runs a batch aggregate and one histogram query for every column of the grid it is handed. The compare grid is three to four times as wide as the entries it compares, and a third to a half of its columns are hidden: each column's before-value ({col}), its equality sentinel ({col}_eq), membership, and the file's __row_order. Nothing reads a stat of any of them, but Buckaroo computes all of it. After #309 those queries run over a file and are cheap each, but there are still 56 of them on a 19-column pair and 138 on a 45-column pair. Under #304's stats_delivery: "deferred" the first rows do not wait for them, and the grid's statistics do. They also occupy Buckaroo's single event loop while they run.

plans/diff-performance-proposals.md calls this B1.

Fix

The diff's /load_expr body gains skip_stat_columns, the list of columns the grid's overrides hide.

• tallyman_companion.diff.diff_skip_stat_columns(overrides) returns the keys of the overrides whose merge_rule is "hidden", sorted. It reads the same dict the grid is configured with (view.column_config_overrides), so the hidden set and the skip set cannot drift apart. A column that changed to an incomparable dtype (diff stat path raises XorqTypeError when a shared-name column changed type across revisions (string→date) #68) keeps its before-value visible, side by side with the new one, so it is not in the list. Keys are never hidden.
• _load_compare_view in app.py adds the field to the body. Nothing else in the body changes.

Why a hidden column's stats are never read: color_categorical reads the row value at val_column (the key color reads the membership cell, the changed-value color reads the {col}_eq cell), and the simple tooltip reads the {col} cell. Both are cells, not stats. What the grid does read are min, max and the histogram bins of the delta columns (diff_extras/display/*.py, and color_map reads histogram_stats[val_column].histogram_bins), and the delta columns are never hidden.

A skipped column keeps its name, dtype and length and gets no stat queries. Its min, max and distinct_count are null, and it has no null_count, mean, histogram or the rest. If a visible column were in the list it would lose its displayer (fare_amount_v2's 3-decimal float displayer drops to obj, checked by adding it to the list), which is why the function takes the hidden set and nothing else, and why a test checks it from the other side.

Not in this PR: promoted diff entries (they load through the entry path, and their overrides are persisted and user-editable), init_sd from the source entries' stats (B2), narrowing the compare to changed columns (B3), primary-key detection, splitting the route.

What it saves

A throwaway project (TALLYMAN_HOME in a temp dir, nothing of mine read or written), synthetic data, 2,000,000 rows. trips is 19 source columns (87 columns in the file, 36 skipped, 51 visible); parking is 45 source columns (173 in the file, 90 skipped, 83 visible). The real route through TestClient, a real Buckaroo 0.15.9 subprocess on a random port, a cold stat cache and a fresh Buckaroo process every open, the diff file already written. "Without" drops skip_stat_columns from the body before it is posted, which is #309's body. Six cold opens of each, in shuffled order, over three runs of the script.

	/load_expr POST without	with	stat queries (cache files) without / with
trips	3.72 to 3.84 s	2.00 to 2.17 s	56 / 22
parking	10.11 to 10.35 s	4.33 to 4.43 s	138 / 48

The whole route took 0.03 s longer than the POST in each case, since the file was already there. The ranges are the spread of the six opens. The machine was also running another job's tests during some of them.

Checks on the same runs, over the real WebSocket:

• The grid loads with the list: all four views (main, detailed_pct, detailed_absolute, summary) come back, and a page request for 300 rows returns without an error.
• Visible columns' stats are the same with and without the list. 552 stat cells on trips and 838 on parking were compared. All are identical except histogram cells (5 and 26). The same cells also differ between two runs that both have no skip, so that is run-to-run variation in those columns' histograms, not an effect of the list.
• Every skipped column has stat values without the list (36 of 36 and 90 of 90) and none with it.

The table above is buckaroo 0.15.9 with inline stats. After rebasing onto #309's rebased head, which is on main with #304 (buckaroo 0.15.10, stats_delivery: "deferred"), I ran the same route and a WebSocket client that advertises ?caps=stats_update as the frontend does, on a fresh throwaway project of 2,000,000 rows by 20 columns, one run per pair. Without this PR (#309's rebased head) and with it:

	first page after connect	stats_update after connect, without	with
worthy pair	0.05 s, both	3.19 s	1.58 s
cheap pair	0.04 s, both	3.23 s	1.48 s

So on 0.15.10 the list does not move the first page. It brings the statistics in about twice as soon. A separate run on 0.15.10 (not repeated for this PR) found the list honored by the deferred second stage, by /reload_expr, and by the cell-level stat cache (a full run after a skip run computes only the missing cells, a skip run after a full run computes none). It also measured the stretch where deferred stats occupy Buckaroo's single event loop: a second session's page request waited 7.8 s during a 45-column load without the list and 3.2 s with it.

Relation to #304

#304 is on main and #309 is rebased onto it, so the body in _load_compare_view now carries skip_stat_columns, data_id and stats_delivery: "deferred", and this branch is rebased onto #309 with no conflict. The skip list does not enter the cache identity: skipped columns send no queries, and the queries that remain are keyed by data_id and the stat hash, so data_id does not need to change.

Verification

After the rebase onto #309's rebased head (89095f7), local, macOS:

• uv run pytest: 1184 passed, 6 skipped, 1 failed. The failure is test_fouc.py::test_unknown_api_path_404s, which fails because this worktree has no built packages/app/dist, the same failure perf(diff): write the diff's join to a parquet file and hand Buckaroo a read of it #309 records; CI builds the app.
• uvx ruff check: clean.
• The rebased tests-only commit d7c6a09, checked out alone, fails the same 10 tests. One of them, the test that pins the other fields of the body, was amended in that commit for the rebase: the body now also carries data_id and stats_delivery.

CI on the rebased head 4b73509 (run 37647153416): ruff clean; fast suite 1189 passed, 2 skipped (vitest 15 passed); integration suite 7 passed, 1 skipped.

Before the rebase, on #309's 0.15.9 base (TDD):

• Red: 9b6f956 (tests/test_diff_skip_stats.py only, 10 tests), run 37638467476: 10 failed, 1168 passed, 2 skipped. Three fail with ImportError (no diff_skip_stat_columns) and seven with KeyError: 'skip_stat_columns' or the body-fields assertion. Ruff passed.
• Green: 5dae01e, run 37641097890: ruff clean; fast suite 1178 passed, 2 skipped (vitest 15 passed); integration suite 7 passed, 1 skipped (the macOS-only Buckaroo crash test). The perf report workflow does not run on a PR whose base is not main.

The tests: the list is exactly the hidden columns of a compare of two entries, sorted and without repeats, and excludes keys, _v2, deltas and the before-value of a column whose dtype changed to an incomparable one; a key that changed dtype is never in it; the route posts it as a list for a worthy pair and a cheap pair, equal to the hidden columns of the posted overrides and disjoint from every visible column of the file; no other field of the body changed. Over a real Buckaroo dataflow built from the posted body: no view draws a skipped column and every color_map rule reads the histogram of a column that keeps its stats, every view's display config is identical with and without the list, and the stat pipeline runs fewer queries with it on a cold cache while visible columns keep their histograms and skipped ones keep their dtype and length. I checked that this guard bites by adding a delta column to the list: the route tests, the "no view draws a skipped column" test and the display-config test all failed.

🤖 Generated with Claude Code