fix(strategy): dedup OPF byte cap, scope it per checkpoint ref, run it off git push's blocking path by peyton-alt · Pull Request #2533 · entireio/cli

peyton-alt · 2026-09-18T23:25:35Z

https://entire.io/gh/entireio/cli/trails/1380

Problem

With OPF on, the git-refs backend had four delivery-correctness bugs:

Delivery checked the OPF trailer on one ref tip and then pushed the symbolic ref, so a concurrent write could ship a later tip that was never checked.
Queue cleanup removed refs by name, so it could erase a newer generation of the same ref.
The byte caps were counted across every queued ref, so one large session withheld all of them.
The prose-leaf cap counted a repeated leaf every time it appeared, although OPF processes it once.

Result

The push queue records (ref, hash) generations. Delivery pushes the exact hash whose trailer it verified and removes only that generation. Rotation after a bounded flush (from fix: bound and explain a wholesale checkpoint ref push failure #2522) also moves exact generations.
Divergence recovery replays the exact candidate and installs the result only if the local ref still names that candidate.
Each queued ref is collected, capped, scanned and rebuilt on its own. A ref that trips a cap or fails stays queued while its siblings ship in the same push.
Each blob's size is checked against the raw-byte budget before it is read.
The prose-leaf count deduplicates by leaf text, matching what OPF actually processes.
doctor migrate-checkpoints reports refs that landed before a failure.

The default caps are unchanged from main (2 MiB prose, 200 MiB raw). OPF still runs inline in git push on both backends, as it does on main.

Scope

This PR was cut down from an earlier version that also moved OPF into a background worker that rewrote refs, and raised the caps. That design is being replaced by a scan-only worker with an OPF result cache that covers both git-refs and git-branch, in #2631, stacked on this one. The follow-up also brings back the higher caps, the size-scaled timeout and chunked model calls. This PR is safe to merge on its own.

Where to start reviewing

checkpoint/pushqueue.go: generation entries, exact removal, rotation.
strategy/manual_commit_push.go and strategy/push_common.go: delivery pinned to exact hashes, and recovery.
strategy/manual_commit_opf_refs.go: per-ref collect, cap and rewrite.

Validation

mise run check (unit, integration, Vogon 56/56, Roger-Roger 4/4)
Real binary with the real opf model, local bare remote:
- git-refs, two checkpoints, ENTIRE_OPF_BATCH_LIMIT=4000: the small ref was scanned and shipped in the same push (6s), and the oversized ref was withheld with a per-ref error. The next push with the default cap shipped it (8s). Both refs on the remote have 0 cleartext names and [REDACTED_PERSON] in their place.
- No categories enabled: the ref is withheld and never trailered.
- git-branch: inline scan, and v1 is pushed with the OPF trailer and redacted content.

🤖 Generated with Claude Code

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit ac850d0. Configure here.}

cursor · 2026-09-18T23:28:23Z

+	// for it. Only when the user asked for OPF on this push: see the gate's doc.
+	if opfDecision == OPFRun {
+		maybeSpawnOPFFlush(ctx, repo)
+	}


Stuck status survives successful rewrite

Medium Severity

The consecutive-failure log is only updated by RunOPFFlush. A later inline rewrite through opfGateForCheckpointRefs can trailer a previously stuck ref and even push it, but nothing records that success, and StuckOPFRefs only checks that the ref still exists. entire status keeps warning that the checkpoint cannot be privacy-filtered — including after the user follows the printed ENTIRE_OPF_BATCH_LIMIT remediation and the rewrite actually succeeds.

Additional Locations (2)

cmd/entire/cli/strategy/manual_commit_opf_flush.go#L294-L320

cmd/entire/cli/status.go#L615-L640

^{Reviewed by Cursor Bugbot for commit ac850d0. Configure here.}

Copilot

Copilot review overview

🟡 Changes recommended

Unresolved OPF worker/push races and stale failure/status handling require fixes before approval.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 2 · 1 · 1

Open (4)

Invalidate in-flight workers after an OPF opt-out · New
Serialize background rewrites with queue delivery · New
Gate status reads on the active checkpoint backend · New
Correct contradictory git-refs byte cap documentation · New

What changed in this PR

Improves git-refs OPF handling by deduplicating cap accounting, isolating ref rewrites, adding detached flushing, and exposing backlog status.

Changes:

Deduplicates prose-leaf byte measurements and scopes rewriting per ref.
Adds detached OPF flushing, throttling, and failure tracking.
Adds text/JSON status visibility and supporting tests.

File	Description
`redact/batch.go`	Deduplicates leaf-byte measurements.
`redact/batch_test.go`	Tests deduplication behavior.
`docs/security-and-privacy.md`	Documents OPF backlog behavior.
`cmd/entire/cli/trail_context_cache.go`	Reuses shared spawn throttling.
`cmd/entire/cli/strategy/manual_commit_push.go`	Integrates OPF decisions and worker spawning.
`cmd/entire/cli/strategy/manual_commit_opf_rewrite.go`	Clarifies backend cap behavior.
`cmd/entire/cli/strategy/manual_commit_opf_rewrite_test.go`	Tests rewrite isolation.
`cmd/entire/cli/strategy/manual_commit_opf_refs.go`	Implements per-ref OPF rewriting.
`cmd/entire/cli/strategy/manual_commit_opf_flush.go`	Adds the detached OPF worker.
`cmd/entire/cli/strategy/manual_commit_opf_flush_test.go`	Tests worker behavior and failures.
`cmd/entire/cli/status.go`	Adds OPF backlog status output.
`cmd/entire/cli/status_test.go`	Tests status reporting.
`cmd/entire/cli/spawnmarker/spawnmarker.go`	Provides shared spawn throttling.
`cmd/entire/cli/settings/settings.go`	Adds OPF configuration access.
`cmd/entire/cli/session_sweep.go`	Reuses shared spawn throttling.
`cmd/entire/cli/root.go`	Registers the hidden worker command.
`cmd/entire/cli/opf_flush_cmd.go`	Defines the detached flush command.
`cmd/entire/cli/opf_flush_cmd_test.go`	Tests command wiring and logging.
`cmd/entire/cli/checkpoint/opf_failures.go`	Tracks per-ref failures.
`cmd/entire/cli/checkpoint/opf_failures_test.go`	Tests failure tracking.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+// follow-up; an explicit OPFSkip (or an unresolvable decision, mapped to
+// OPFAbort) must not, since running OPF in the background after the user
+// declined it for this push would redact content they chose to flush as-is
+// and diverge the local ref from what was actually pushed. Not permanent: a
+// later push whose decision resolves to OPFRun re-evaluates the same backlog


+	opfDecision, opfErr := opfGateForCheckpointRefs(ctx, repo)
+	// Hand any remaining OPF backlog to a detached worker whether the gate
+	// succeeded or not: a withheld flush leaves the whole backlog, and a gate
+	// that succeeded can still have skipped a ref over the per-ref cap. Placed
+	// before the withheld early-return so both outcomes get the follow-up, and
+	// it never blocks — the child makes the model call, this push does not wait
+	// for it. Only when the user asked for OPF on this push: see the gate's doc.
+	if opfDecision == OPFRun {
+		maybeSpawnOPFFlush(ctx, repo)
+	}


+func opfBacklogForStatus(ctx context.Context, s *EntireSettings) (pending int, stuck []string) {
+	if !s.OPFEnabled() {
+		return 0, nil
+	}


+> **Stale, needs a rewrite:** this section still describes one inference call
+> batching every commit/ref in the push. On `git-refs`, that changed: the cap
+> and the OPF call are now scoped per checkpoint ref, not per flush, and a
+> `git-refs` flush that can't finish inline hands the remainder to a detached
+> `entire __opf_flush` worker instead of blocking the push (see "Seeing
+> outstanding OPF work" below, and the caps subsection above, for the accurate
+> current shape). `git-branch`'s flow below is still accurate: its cap stays
+> cumulative across the whole unpushed chain. Rewrite this section for
+> `git-refs` rather than trusting the flow as written.


…on still-redacting siblings Delivery of queued checkpoint refs was all-or-nothing: prePushCheckpointRefs and PushQueuedCheckpointRefs early-returned the moment opfGateForCheckpointRefs reported any error, so flushCheckpointRefsQueue never ran and nothing shipped — including refs the gate had already rewritten and trailered. PR #2533 made the rewrite per-ref; this makes delivery per-ref too. flushCheckpointRefsQueue now takes a notReady set and excludes those refs from the push, leaving them queued exactly like a rejected ref. Both callers compute it from RefsAwaitingOPF, which re-reads local refs, so "ready to ship" stays "already carries Entire-OPF-Applied" and never "we gave up waiting". An unreadable backlog still withholds everything. OPFAbort now ships already-trailered refs rather than withholding them: the abort declines inference for this push, which is orthogonal to whether previously-redacted content may leave the machine. PushQueuedCheckpointRefs can therefore return a non-zero pushed alongside a non-nil error, so doctor migrate-checkpoints reports what landed before surfacing the error. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Entire-Checkpoint: 01M2VGJS5T5ARMYH2SAC7M24XW

SumProseLeafBytes counted every occurrence of a leaf, but the batch sends each unique leaf to OPF once. A ref whose ancestry re-carries the same full.jsonl was measured several times over and could trip the cap on content OPF would process only once. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Entire-Checkpoint: 01M3QEGVWCWS5ND3EGSF4TZDPE

… ref git-refs delivery checked the OPF trailer on one ref tip and then pushed the symbolic ref, so a concurrent write could ship a later, unchecked tip. Queue cleanup removed refs by name and could erase a newer generation. And the byte caps were counted across every queued ref, so one large session withheld all of them. - The push queue records (ref, hash) generations. Delivery pushes the exact hash whose trailer it verified and removes only that generation; rotation after a bounded flush moves exact generations too. - Divergence recovery replays the exact candidate and installs it only if the local ref still names that candidate. - Each queued ref is collected, capped, scanned, and rebuilt on its own, so a ref that trips a cap or fails stays queued while its siblings ship. - Each blob's size is checked against the raw-byte budget before it is read. - doctor migrate-checkpoints reports refs that landed before a failure. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Entire-Checkpoint: 01M3QEHF4J9VQYQ6BFXJZ45RNW

…ure first Ctrl-C at the OPF prompt on git-refs now withholds the whole flush, already-trailered refs included, matching the documented contract that cancelled checkpoint refs stay queued. A decision that could not be resolved still ships refs that already carry the trailer. When a broken OPF runtime stops the per-ref rewrite, it now outranks an earlier ref's cap error, so the user is pointed at the runtime rather than the cap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Entire-Checkpoint: 01M47DRA0SE3ZW70K0QWT0J7AY

# Conflicts: # cmd/entire/cli/strategy/manual_commit_push.go

- A ref whose OPF call fails at runtime moves to the back of the push queue, so the next push reaches the refs it was withholding. - Recovery that finds the remote already contains the local commits counts the ref as delivered instead of trailer-checking the remote's own commit. - EnqueueRef queues an unresolvable ref without a generation instead of dropping it. - Recovery wraps the fetch error instead of discarding it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Entire-Checkpoint: 01M4EN1M5Y35NJQ68TF0PN60WB

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Entire-Checkpoint: 01M4EP2XKTZ61K1MGVRYR9DKEX

Copilot AI lite review requested due to automatic review settings September 18, 2026 23:25

peyton-alt requested a review from a team as a code owner September 18, 2026 23:25

Copilot started reviewing on behalf of peyton-alt September 18, 2026 23:26 View session

cursor Bot reviewed Sep 18, 2026

View reviewed changes

Copilot AI reviewed Sep 18, 2026

View reviewed changes

peyton-alt and others added 2 commits September 29, 2026 13:43

peyton-alt force-pushed the peyton/opf-batch-cap-fix branch from 9c609c5 to faca839 Compare September 29, 2026 20:54

peyton-alt mentioned this pull request Sep 30, 2026

feat(opf): scan in the background so git push never waits on the privacy filter #2631

Open

peyton-alt mentioned this pull request Oct 6, 2026

fix(opf): push the v1 commit the OPF rewrite verified, not the branch tip #2669

Open

peyton-alt and others added 3 commits October 8, 2026 13:44

Merge remote-tracking branch 'origin/main' into peyton/opf-2533-rework

d47fec7

# Conflicts: # cmd/entire/cli/strategy/manual_commit_push.go

fix(checkpoint): keep a queued ref whose recorded generation is corrupt

cb127fe

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Entire-Checkpoint: 01M4EP2XKTZ61K1MGVRYR9DKEX

Navigation Menu

Repository navigation

fix(strategy): dedup OPF byte cap, scope it per checkpoint ref, run it off git push's blocking path - #2533

fix(strategy): dedup OPF byte cap, scope it per checkpoint ref, run it off git push's blocking path#2533
peyton-alt wants to merge 6 commits into
mainfrom
peyton/opf-batch-cap-fix

peyton-alt commented Sep 18, 2026 •

edited

Loading

Uh oh!

cursor Bot left a comment

Uh oh!

cursor Bot Sep 18, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

Conversation

peyton-alt commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Result

Scope

Where to start reviewing

Validation

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Sep 18, 2026

Choose a reason for hiding this comment

Stuck status survives successful rewrite

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

peyton-alt commented Sep 18, 2026 •

edited

Loading