Count each checkpoint's tokens once by peyton-alt · Pull Request #2674 · entireio/cli

peyton-alt · 2026-10-06T03:33:20Z

https://entire.io/gh/entireio/cli/trails/1495

Problem

Checkpoints store the tokens used since the previous checkpoint, and entire.io adds those up for session totals, repo charts, home activity and recap. Several paths count from the wrong place, so tokens get counted twice, or not at all:

Partial commits. When a commit leaves agent-changed files uncommitted, carry-forward resets CheckpointTranscriptStart to 0 so the next checkpoint shows the whole conversation. Token counting used the same offset, so the next checkpoint counted the whole session again.
session attach always counted from the start of the transcript and never recorded how far it got. It double counted turns already in a checkpoint, and the session's next checkpoint counted the attached turns again.
session adopt reset the offset to 0. The session keeps its ID, so if the source session's checkpoints are in the same repo, the server added them up together.
Mid-turn commits. The turn-end advance after a mid-turn commit skipped the rest of the turn, so those tokens were never counted.

Affects agents whose tokens come from the transcript: Claude Code, Codex, Copilot CLI, Factory Droid, OpenCode, Pi and external agents. Cursor and Antigravity get their tokens from hooks.

Fix

A second offset, SessionState.TokenTranscriptStart (read through TokenStart()), records how far tokens have been counted. It only moves forward, and only when a condensation stores tokens (AdvanceCheckpointWindow). CheckpointTranscriptStart keeps its job of choosing which part of the transcript a checkpoint shows.

Condensation, the live-transcript path and review sessions count tokens from TokenStart().
Carry-forward and the mid-turn turn-end advance leave the token offset alone.
Attach counts from the token offset, falling back to the pending hook-reported total the way condensation does. Afterwards it records the transcript end and resets the pending window. What the next checkpoint shows is unchanged.
Adopt carries over the source's token offset, pending total and subagent baseline. The retired source never condenses again, so tokens it used but never checkpointed are counted once in the target, and tokens it did checkpoint aren't counted again. This reverses the subagent rebaseline from d39ecfd and the pending-total reset from cf4e18d, which counted only what happened after the adopt.
State written by an older CLI has no token offset. On load it starts from CheckpointTranscriptStart, which is the current behaviour, so a session already in the middle of a partial commit double counts one more time.

Verification

Integration tests (integration_test/token_scope_test.go):
- Partial commits: 100 / 7 / 3, where today's code stores 107 for the second checkpoint.
- Attach after a partial commit, then the next checkpoint: 9 / 4.
- Mid-turn tail lands in the next checkpoint: 12. The test fails with 7 if the token offset moves at turn end.
Unit tests: loading older state, copies of a session state not sharing the offset, adopt.
End-to-end with real git hooks and the Claude Code hook commands, comparing this build with main:

Step	Correct	main	this PR
Commit after a partial commit	7	107	7
Commit leftover file, no new turn	0	110	0
Commit after a mid-turn commit (tail 5 + 7)	12	7	12
`session attach` (new turn: 9)	9	59	9
Commit after attach (new turn: 4)	4	63	4

mise run check

Not in this PR

Checkpoints already written stay inflated. Recalculating them would be a server-side job.
Snapshot checkpoints (entire checkpoint create) still don't advance the offset.
A session state that is deleted and then re-created starts counting from 0 again.

🤖 Generated with Claude Code

Note

Medium Risk
Changes token accounting across condensation, attach, and adopt—areas that feed server-side session totals and billing-style metrics; migration may allow one extra double-count for sessions mid partial-commit on upgrade.

Overview
Fixes double-counting and missed tokens in per-checkpoint metadata by splitting display transcript windows from token accounting.

TokenTranscriptStart (via TokenStart()) records how far tokens are already stored in checkpoints; it only advances when condensation stores tokens (AdvanceCheckpointWindow). CheckpointTranscriptStart can still reset to 0 on carry-forward so the UI shows the full conversation, without recounting earlier turns.

Condensation, live-transcript paths, and review token hydration now count from TokenStart() instead of the checkpoint display offset. session attach uses AttachTokenUsage / ConsumeAttachTokenWindow so checkpoints store only uncheckpointed tokens while session state keeps the full-transcript total. session adopt preserves the source's token offset, pending checkpoint usage, and subagent baseline instead of opening a fresh window.

Older state files migrate on load (token_transcript_start defaults from checkpoint_transcript_start). New integration tests cover partial commits, attach, and mid-turn commit tails.

^{Reviewed by Cursor Bugbot for commit 4ea92d8. Configure here.}

CheckpointTranscriptStart chose both which part of the transcript a checkpoint shows and where its token count starts. Carry-forward after a partial commit and session adopt reset it to 0 so the next checkpoint shows the whole conversation, which also made that checkpoint count every token from the start of the session again. Session attach always counted from 0 and never recorded how far it got, and the turn-end advance after a mid-turn commit skipped that turn's remaining tokens entirely. Add a separate offset, TokenTranscriptStart, that advances only when a condensation stores tokens. Token counting in condensation, the live transcript path, review sessions and attach reads it through TokenStart(). Attach records the transcript end afterwards and resets the pending window. Adopt keeps the source's token offset, pending total and subagent baseline, because the retired source never condenses again. State written by an older CLI starts from CheckpointTranscriptStart, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Entire-Checkpoint: 01M47MB89PBDYQTEFR7ACGRJCS

Copilot

Copilot review overview

🟡 Changes recommended

Attach can lose pending subagent usage and can report success without durably persisting its consumed token window.

Review effort: Balanced
Findings: 2 · 1

Open (3)

Prevent token double-counting when state persistence fails · New
Preserve pending subagent usage during attach · New
Update token validation documentation for new offset contract · New

What changed in this PR

Separates transcript display offsets from token-accounting offsets to prevent duplicate or missed checkpoint token totals.

Changes:

Adds and migrates TokenTranscriptStart.
Updates condensation, attach, adopt, and review token scoping.
Adds unit and integration coverage for partial commits, attach, and mid-turn commits.

File	Description
`docs/development/checkpoint-implementation.md`	Documents token-offset semantics.
`cmd/entire/cli/strategy/manual_commit_hooks.go`	Advances both offsets after condensation.
`cmd/entire/cli/strategy/manual_commit_condensation.go`	Implements token-scoped condensation and attach helpers.
`cmd/entire/cli/strategy/condensation_prompts_test.go`	Updates helper invocation.
`cmd/entire/cli/session/state.go`	Adds persistent token offset state.
`cmd/entire/cli/session/state_test.go`	Tests migration and copy isolation.
`cmd/entire/cli/session_adopt.go`	Preserves source token accounting during adoption.
`cmd/entire/cli/session_adopt_test.go`	Tests adopted pending and subagent usage.
`cmd/entire/cli/review/manifest.go`	Uses the token offset for review usage.
`cmd/entire/cli/integration_test/token_scope_test.go`	Covers key token-accounting scenarios.
`cmd/entire/cli/attach.go`	Scopes attach tokens and consumes the token window.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+	if err := saveAttachSessionState(logCtx, repo, existingState, sessionID, ag.Type(), transcriptPath, checkpointID, meta, sessionUsage, tokenPos, opts, reviewSkills); err != nil {
 		logging.Warn(logCtx, "failed to save session state", "error", err)
 	}


cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 4ea92d8. Configure here.}

The turn-end advances after a mid-turn commit move CheckpointTranscriptStart without storing tokens. An unset TokenTranscriptStart falls back to CheckpointTranscriptStart, so it would follow the display window forward and skip that turn's tail tokens. AdvanceDisplayWindow pins the token offset before moving the window; both turn-end sites use it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Entire-Checkpoint: 01M47MXV4A361ASRFC7NM860TX

…ding subagent tokens Attach writing into HEAD's checkpoint replaces this session's entry. It now reads that entry first and keeps its stored tokens, adding only the tokens since TokenStart; without state the whole-transcript count already covers them. Attach also fills the pending window's subagent tokens before consuming the window, as condensation does, so they land in the attach checkpoint instead of nowhere. The existing-entry read moves into attachReplacedEntryTokens; review attaches still refuse an existing entry and fail on a read error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Entire-Checkpoint: 01M4BSKPVWH6285NNJ70J7DD7M

…isplay window Carry-forward set CheckpointTranscriptStart to 0 directly. An unset TokenTranscriptStart would follow it through TokenStart's fallback and bring the recount back; AdvanceDisplayWindow(0) pins it first, like the turn-end advances. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Entire-Checkpoint: 01M4BT334KYX2QZYXY99PCF0XJ

AttachTokenUsage merged the replaced checkpoint entry with accumulateTokenUsage, which replaces SubagentTokens because it was written for cumulative snapshots. Both sides here are window deltas, so use types.AddTokenUsage to add the nested subagent totals. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Entire-Checkpoint: 01M4EKWN5WQQMEA5BW95K9GPC4

…en-scope # Conflicts: # cmd/entire/cli/session_adopt.go # cmd/entire/cli/session_adopt_test.go # cmd/entire/cli/strategy/condensation_prompts_test.go # cmd/entire/cli/strategy/manual_commit_condensation.go # docs/development/checkpoint-implementation.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Entire-Checkpoint: 01M4EMEBSKX0SARDNN8YHG8PS8

Attach added the replaced checkpoint entry's tokens on top of a count from TokenStart even when the state was recreated from scratch (TokenStart 0), so the entry was counted twice. Only keep it when tokens were already checkpointed; otherwise count the whole transcript, as without state. A condensation that could not read the transcript reports 0 lines, and AdvanceCheckpointWindow(0) moved the token offset back to 0, so the next checkpoint recounted everything. Leave the token offset alone then. Also point docs and comments that still named CheckpointTranscriptStart as the token scope at the token offset. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Entire-Checkpoint: 01M4EWKP5QA813YPJNBMTS6W7C

…en-scope # Conflicts: # cmd/entire/cli/attach.go

Resolves the attach.go conflict with #2672 without bringing back the double count: main's attach counted tokens from the displayed window (CheckpointTranscriptStart), which carry-forward leaves at the session start, so attach after a partial commit recounted checkpointed turns. - The checkpoint's displayed window stays main's window.start; tokens come from strategy.AttachTokenUsage, counted from TokenStart(). - ConsumeAttachTokenWindow runs whether or not the session is active; main's rule still applies to CheckpointTranscriptStart, which a running session's hooks own. - Main now returns early when the checkpoint already holds the session, so attach never replaces its own entry and the replaced-entry token carry-over is unreachable; it is removed and the own-checkpoint test now checks the entry keeps its tokens and the next checkpoint counts the turn attach skipped. - New test: attach during an active turn stores the turn's tokens so far, leaves CheckpointTranscriptStart alone, and the turn's checkpoint doesn't count them again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Entire-Checkpoint: 01M4GVA294QN563TJP9TB6NBBS

saveAttachSessionState reloads the session from disk, so a hook that checkpointed while attach waited at its prompt has already moved the token offset. Consuming attach's window from the stale position moved it back and had the next checkpoint recount those tokens. Consume only when the offset is still where attach counted from; otherwise leave it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Entire-Checkpoint: 01M4GVZXYFGM6SXT3S93J4TEVE

Attaching mid-turn left the turn's subagent tokens out of the attach checkpoint: hooks record them in the pending usage only at Stop, so the next checkpoint picked them up instead. Attach now reads them live the way a mid-turn condensation does, and keeps the cumulative subagent total so the re-baseline covers what the attach checkpoint stored. TestTokenScope_EachTurnCountedOnce runs a session through hook commits, a partial commit (carry-forward), two attaches to successive commits and a subagent turn, for an ended and a running session, and checks that each checkpoint holds exactly the turns it newly covers and that the checkpoints sum to the session's tokens. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Entire-Checkpoint: 01M4GXCE3389JHD45NHRKZNDY9

The race guard skipped only the offset move, so attach still overwrote the session's token total with the older figure it computed before its prompt. Skip both. Also read the on-disk token offset before the display offset moves: TokenStart falls back to CheckpointTranscriptStart for a state without its own token offset, so a brand-new session looked raced and its token window was never consumed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Entire-Checkpoint: 01M4GZ2C3JW2V9XPMR034Y6RFB

…activity Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Entire-Checkpoint: 01M4H5WYHBXPY1YJJJJHDEK133

Copilot AI balanced review requested due to automatic review settings October 6, 2026 03:33

peyton-alt requested a review from a team as a code owner October 6, 2026 03:33

Copilot started reviewing on behalf of peyton-alt October 6, 2026 03:34 View session

Copilot AI reviewed Oct 6, 2026

View reviewed changes

cursor Bot reviewed Oct 6, 2026

View reviewed changes

Comment thread cmd/entire/cli/strategy/manual_commit_condensation.go

peyton-alt and others added 8 commits October 5, 2026 20:43

Remove stray patch backup file

47b540f

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Entire-Checkpoint: 01M4EMEBSKX0SARDNN8YHG8PS8

Merge remote-tracking branch 'origin/main' into peyton/checkpoint-tok…

78ebe92

…en-scope # Conflicts: # cmd/entire/cli/attach.go

gtrrz-victor previously approved these changes Oct 9, 2026

View reviewed changes

peyton-alt dismissed gtrrz-victor’s stale review via 895c841 October 9, 2026 17:28

peyton-alt and others added 4 commits October 9, 2026 10:39

attach: also treat a changed pending total or new checkpoint as hook …

47e01ff

…activity Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Entire-Checkpoint: 01M4H5WYHBXPY1YJJJJHDEK133

Navigation Menu

Repository navigation

Count each checkpoint's tokens once - #2674

Count each checkpoint's tokens once#2674
peyton-alt wants to merge 14 commits into
mainfrom
peyton/checkpoint-token-scope

peyton-alt commented Oct 6, 2026 •

edited by entire Bot

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

Conversation

peyton-alt commented Oct 6, 2026 • edited by entire Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Fix

Verification

Not in this PR

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

peyton-alt commented Oct 6, 2026 •

edited by entire Bot

Loading