perf(gfql): build grouped-aggregate outputs in one pass to keep pandas blocks consolidated

[lmeyerov]

Recovers the one real regression the whole-stack board A/B found: pandas q1 +11.3% @20k / +6.9% @100k, q2 +10.0% / +6.7% (q2 runs q1's shape as its first leg). polars was flat and every board cell returned byte-identical values, so it was pure overhead.

Bisected to a single 4-line hunk in #1901's first commit. Making a.id AS id survive as an output required moving the key rename() before the per-output column assignments — but rename() returns a consolidated copy, so running it first leaves the later per-column writes creating unconsolidated blocks, and the ~200k-row inner merge then pays per-block take plus _merge_blocks/vstack. Instrumented on the real q1: parent 1 block / 5.58 ms, culprit 2 blocks / 7.63 ms, against a +2.6 ms board delta; cProfile agreed with identical call counts and a byte-identical compiled plan.

Fix: build the projected frame in ONE construction (a dict of Series handed to df_cons) instead of assigning column-by-column after the rename — which is also this repo's standing cheap-write guidance. No private pandas API. The .copy() became unnecessary and is gone. The correctness #1901 added is intact: the key is still renamed away from node_col, so a.id AS id survives and an output named like the node-id column cannot corrupt the join key.

Local microbenchmark (10 interleaved rounds × 9 reps, medians of round medians — not the board; a dgx A/B confirms separately):

shape	blocks base → fix	lookup merge	query total
grouped-agg, int key	2 unconsolidated → 1 consolidated	11.41 → 6.78 ms (−40.6%)	36.72 → 30.01 ms (−18.3%, fix faster 10/10 rounds)
grouped-agg, int + string	3 → 2 consolidated	14.63 → 10.16 ms (−30.6%)	52.58 → 45.49 ms (−13.5%, 10/10)
string key only (control)	2 consolidated → 2 consolidated	9.79 → 9.79 ms	−0.3% (ns)
two-star grouped count	3 → 2 consolidated	11.70 → 8.41 ms (−28.1%)	−2.0% (7/10)

The string-only row is the honest control: that shape is already consolidated at base, so there is nothing to win — the mechanism confirming itself rather than a uniform speedup. Result-frame sha256 identical base vs fix on all four shapes.

Sibling sweep: all 113 rename(columns=…) sites in graphistry/compute/ checked for a following column-write; exactly two had the pathology (the culprit and the two-star grouped-count twin), both fixed. The polars twins already build in one select/with_columns, consistent with polars measuring flat on the board.

Gates: full-tree failure set byte-identical (md5 matched), fast path still engages, typecheck 329 clean, hygiene guard no growth — the first draft needed a cast() that tripped the explicit-cast guard, fixed by annotating the dict properly rather than raising the cap.

Not claimed: the two-star sibling's query-level −2.0% is weak and shouldn't be quoted (its merge-level −28% and block-state change are unambiguous); no cuDF verification on a GPU-less box; board confirmation still outstanding.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi

[lmeyerov]

Board A/B: 18 slots per build per scale (108 slots, 1944 cell measurements), frozen builds sha-verified byte-exact against their git blobs, --network none, zero errors. Values identical in all three builds, every cell, every slot.

Verdict: recovers, but I will not claim provably 100%.

scale	cell	TIP vs BASE	FIX vs BASE (recovery)	FIX vs TIP (help)
100k	pandas q1	+8.1% SIG	+2.8% p=0.34 ns	−4.9% SIG
100k	pandas q2	+7.6% SIG	+2.7% p=0.38 ns	−4.6% SIG
20k	pandas q1	+15.4% SIG	+2.9% p=0.85 ns	−10.9% SIG
20k	pandas q2	+14.5% SIG	+4.1% p=0.49 ns	−9.1% SIG
both	polars q1/q2 (control)	ns	ns	ns — flat within ±2.2%

FIX vs BASE is statistically indistinguishable from zero in all four pandas cells, and FIX vs TIP is a large Bonferroni-significant improvement. Honest caveat: every FIX-vs-BASE point estimate sits positive (+2.7% to +4.1%) and the bootstrap recovery fraction is 65% [56,112] / 65% [54,109] at 100k, so a residual up to ~+3.5% at 100k cannot be excluded. What the data firmly excludes is anything near the original 8%/15%. The defensible statement is "recovered to within noise, with a possible ~3% residual at 100k" — not "fully recovered".

The regression reproduced somewhat larger than first measured (+8.1/+7.6 @100k vs +6.9/+6.7; +15.4/+14.5 @20k vs +11.3/+10.0), consistent with the known cross-session drift of intermediate heads.

q5–q9 @100k, FIX vs TIP: all ten cells ns. The two-star hunk did not perturb q5/q6/q7 — which are exactly its shape — and also produced no measurable win there, consistent with its weak local result. The one cell worth a re-check is pandas q6 @20k (FIX vs TIP +1.3%, uncorrected-sig only, ns under Bonferroni; +0.8% ns at 100k).

Separate finding, not this PR's: polars q8 shows TIP vs BASE +7.8% SIG @100k with FIX vs TIP ns — a stack-level residual this change neither causes nor fixes. That is the known trail-correctness cost (self-loop exclusion from count(*)), previously measured at +3.7% after the #1907 optimizations; the larger figure here needs a re-measure at the final tip before release rather than being quoted as-is. Favourable movers, unattributed: polars q9 −4.8% SIG @100k and −12.1% @20k, polars q6 −9.3% @20k.