Repository navigation
#35 docs: curate the context surface — all four over-budget files cleared - #40
Conversation
…ared Measured exactly (Anthropic count_tokens), not estimated: AGENTS.md 6349 -> 5954 (budget 6000) docs/ARCHITECTURE.md 11075 -> 9510 (budget 10000) docs/contracts/content-replicate-... 10284 -> 9274 (budget 10000) docs/contracts/replicator-boundaries.md 9152 already under The issue's first bullet is refuted: replicator-boundaries.md was never over budget. The write guard's offline estimator runs +13.5% high on the contract docs because measure-context.sh calibrates bytes/token on the policy file alone (AGENTS.md tokenizes at 2.52 bytes/token; the contract docs at ~2.83). The issue's cl100k cross-check undercounts for the opposite reason. Two files the issue did not name were genuinely over: docs/ARCHITECTURE.md by 1075, and AGENTS.md itself by 349 after #29. Relocations (verbatim; nothing paraphrased in transit): - ARCHITECTURE.md's stream-agnostic bus rules -> new docs/CONVENTIONS.md - "Why the issuer renders" -> docs/plans/2026-08-14-...-settled.md, the same split #24 made for the fetch_failed open question; the contract keeps the shipped rule, the plan keeps the alternatives - AGENTS.md test-layout bullet -> docs/TESTING.md - AGENTS.md founding-design pointers -> docs/ARCHITECTURE.md Tightenings, each warranted by duplication already in the surface: the replicate-gcs bullet, the normative-contracts bullet, Project Layout, the co-core floor tail, the duplicated env-load snippet, the duplicated --no-cov rationale. .socraticodecontextartifacts.json re-registered in the same change per the manifest-is-a-source rule (#19 CR #17); codebase_context_index re-run on merge. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…k per skills#115 CR #1 — restore `src/core/` to AGENTS.md Project Layout. The tightening left three of four src/ packages listed, which reads as a complete enumeration and denies a package the same file imports from two sections later. CR #2 — restore the integration-specific `--no-cov` reason. The previous cut claimed a class-D verbatim-duplicate warrant that does not hold: neither the comment three lines above nor docs/COMMANDS.md carries "these do not exercise all of src/". Reclassified C, not D. CR #3 — Bus Conventions regained an inline route to its deep docs. Deleting the old ARCHITECTURE.md pointer left the section reachable only from the Detail Docs index, right after this branch split its reasoning across two files. CR #4 — drop the counts from the manifest's AGENTS.md description ("nine bus rules", "two normative contracts"); both had drifted, and a count in the highest-authority answer outranks the correct one in the file. CR #5 — drop the content-free `## Conventions` heading in docs/CONVENTIONS.md. CR #6 — index the relocated plan doc in the replicate contract's "Where the rest of the contract lives", matching what the fetch contract already does for 2026-07-31-fetch-failed-fact-settled.md. CR #7 — stet. Separately, per review direction: the Code Exploration Policy block is brought onto init-socraticode's post-#115 shape (gregoryfoster/skills#115, upstream 9125148, which our vendored pin 3fc7b71 predates). Variant A keeps the indexed-state line, the negative rule, a three-row table and a link; the full table, prefetch, per-tool gotchas and cross-repo search move to the new docs/SOCRATICODE.md. Repo-authored content is rescued into `## Code Exploration Notes (repo-specific)` OUTSIDE the marker pair, per the step-1a contract that same change added — otherwise the next init-socraticode run deletes it silently. Code Exploration Policy 935 -> 446 tokens AGENTS.md 5,954 -> 5,837 (budget 6,000) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CR round 1 applied —
|
| Item | Action | Result |
|---|---|---|
| 1 | Fixed | AGENTS.md:63 — src/core/ restored to Project Layout |
| 2 | Fixed | AGENTS.md:178 — integration --no-cov reason restored; warrant reclassified D → C |
| 3 | Fixed | AGENTS.md:158-159 — Bus Conventions routes to ARCHITECTURE.md + CONVENTIONS.md inline |
| 4 | Fixed | manifest agent-guidelines — counts removed ("nine bus rules", "two normative contracts") |
| 5 | Fixed | docs/CONVENTIONS.md — ## Conventions heading dropped |
| 6 | Fixed | replicate contract's "Where the rest of the contract lives" now indexes the plan doc, matching the fetch contract |
| 7 | Stet | — |
Plus: the policy block, per review direction
init-socraticode did ship the split — gregoryfoster/skills#115, upstream commit 9125148. Our vendored pin 3fc7b71 predates it, so it had not reached this repo. The block is now on the upstream Variant A shape:
AGENTS.mdkeeps the indexed-state line, the negative rule, a three-row table and a link — 935 → 446 tokens.- New
docs/SOCRATICODE.md(1,414 tok) carries the full ten-row table, the prefetch/hook note, per-tool gotchas, cross-repo search, graph-yield health, and.socraticodeignorescope. - Repo-authored content is rescued into
## Code Exploration Notes (repo-specific)outside the marker pair — the step-1a contract that same upstream change added. The manifest-is-a-source gotcha (Consume content.fetch-policy — replace the single env default with Watcher's per-host numbers (cannobserv#285) #19 CR A body-less 304 dead-letters — make "not modified" a first-class outcome before conditional GET is usable #17) lives there now; left inside the markers, the nextinit-socraticoderun would have deleted it silently.
This is why finding 3 mattered more than it looked: the file now routes to four detail docs from the sections that need them.
Numbers
| Before run | After CR | |
|---|---|---|
AGENTS.md |
6,349 | 5,837 (budget 6,000) |
Code Exploration Policy |
935 | 446 |
| headroom | −349 | +163 |
Every live doc under 10,000. 0 dead links, 0 orphans. ruff check clean, ruff format --check 92 files formatted, 683 tests pass (99.83%).
prove-no-loss.sh: 149 inline, 10 relocated verbatim (8 → SOCRATICODE.md, 1 → ARCHITECTURE.md, 1 → TESTING.md), 31 unaccounted — the template-owned policy block regenerated from upstream Variant A (not repo prose to preserve), the class-D duplications listed in the original PR body, and reflows. Ledger row rewritten in place rather than appended, since this is one run.
Follow-up beyond this PR
The vendored skills submodule sits at 3fc7b71; upstream main is c3f40f7 and carries #115 plus other work. Bumping the pin is separate from this docs curation — worth its own issue so the init-socraticode re-run and its report land deliberately.
CR #8 — docs/SOCRATICODE.md's header claimed a re-run overwrites it. False at our pin: 3fc7b71 predates the #115 split (0 mentions of SOCRATICODE.md, an 8-row block, no step-1a rescue), so a re-run today restores the pre-split block and orphans this file. Header now states the required version and points at #41, which tracks the bump. The rescued Notes section is safe either way — the old sweep targets only `## Code Exploration Policy`. CR #9 — the manifest's AGENTS.md description ended in a "…live in X, Y, Z respectively" list that had drifted to seven docs against six clauses, so every doc after the inserted one mis-mapped. Replaced with a pointer to the Detail Docs index rather than a corrected list: a positional enumeration in the highest-authority answer rots on every doc added, which is how it broke here. CR #10 — restore the graph-health command. mcp-driver.mjs exists in this checkout, so "check yield, not status" was unactionable advice about a trap (READY with almost no edges) that answers empty rather than erroring. CR #11 — Bus Conventions routing block rendered as a four-sentence run-on; now four bullets. CR #12 — stet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`measure-context.sh` flagged four contract docs `over_budget` even though #40 cleared them. Exact counts refute it — every live doc is under the 10,000 budget, the largest being ARCHITECTURE.md at 9,510: ARCHITECTURE.md est 10,610 exact 9,510 (+11.6%) content-replicate-issuer est 10,681 exact 9,372 (+14.0%) replicator-boundaries est 10,470 exact 9,152 (+14.4%) content-fetch-issuer-ref est 10,078 exact 9,234 (+ 9.1%) That is the bias skills#172 describes: one global bytes/token ratio fitted on the policy file alone, applied to prose that tokenizes differently. The remedy vendored in with #41 — per-file anchors (#145) plus a surface-fitted ratio (#175) — needed one `--exact` run to populate, which this is. `.skills/context-token-counts` now anchors each file to its own last exact measurement, so the offline estimate reports `tokens_source: "file"` and reproduces the exact figures; the global ratio (2.47 -> 2.67, now fitted across the whole surface) is the fallback for a file never counted or since drifted >25%. Nothing about the docs changed — only what the estimator believes about them. An over-flag wastes a reviewer's attention; the direction that would matter is an under-flag, which is what the anchors also close. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Closes #35.
Before / after — measured exactly, not estimated
AGENTS.mddocs/ARCHITECTURE.mddocs/contracts/content-replicate-issuer-contract.mddocs/contracts/replicator-boundaries.mddocs/CONVENTIONS.md(new)docs/TESTING.mdtotals.tokens_live73,998 → 73,081. Dead links: 0. Orphans: 0. Full suite: 682 passed.The issue's premise, checked
replicator-boundaries.mdwas never over budget. Exact count 9,152 — 848 under, not 385 over.The issue guessed the hook was "measuring something wider than the file". It isn't.
measure-context.shpersists a bytes-per-token ratio computed from the policy file alone (RATIO_X100=$(( P_BYTES * 100 / P_TOKENS ))), and the write guard then applies that one ratio to every reference doc.AGENTS.mdtokenizes at 2.52 bytes/token; the contract docs, which are much denser in tables and links, tokenize at ~2.83. So the guard over-reports them:replicator-boundaries.mdcontent-replicate-issuer-contract.mdARCHITECTURE.mdAGENTS.md10,386 is the "~385 over" in the issue. The cl100k cross-check in the issue (6,148 / 6,026) errs the other way — it is OpenAI's tokenizer and undercounts Claude badly.
Two files the issue did not name were genuinely over:
docs/ARCHITECTURE.mdby 1,075 (the largest breach), andAGENTS.mditself by 349 — it grew 477 tokens between the last ledger row and#29landing.Filed upstream against the skill: the ratio should be calibrated over the whole measured surface, not the policy file.
Per-section disposition
ARCHITECTURE.md→ stream-agnostic bus rules (15 bullets)docs/CONVENTIONS.mdcontent-replicate-issuer-contract.md→## Why the issuer rendersdocs/plans/2026-08-14-why-the-issuer-renders-settled.mdAGENTS.md→ Conventions/General test-layout bulletdocs/TESTING.md(## Test layout)AGENTS.md→ Project Overview founding-design pointersdocs/ARCHITECTURE.mdheaderAGENTS.md→ Bus Conventions "replicate loop writes forgcs"AGENTS.md→ Bus Conventions "Three normative contracts"AGENTS.md→ Project LayoutAGENTS.md→ Environment & Tooling co-core floor tailAGENTS.md→ env-load snippet under Environment VariablesAGENTS.md→--no-covrationale, 2nd occurrenceAGENTS.md→ "Read ARCHITECTURE.md before changing what any of them carries"AGENTS.md→ Bus Conventions (9 core rules), Code Exploration Policy, Detail Docs, Common Commands, Environment VariablesRelocations, with destinations
docs/CONVENTIONS.md(new, 1,870 tok) — the 15 co-core/Redis-Streams bullets that hold for every stream: outbox posture,extra="ignore"validation, batch-poison /count=1,dead_letter, decoding a fieldless frame, the globalfrom_wiredispatch table,claim_stale,times_delivered, theXINFO CONSUMERSliveness trap, message-vs-cycle failure, injection-only clients, store-then-publish, tz-awareoccurred_at, keyword-onlytopic/message_id,sha256's home.ARCHITECTURE.mdkeeps what each individual stream carries. Indexed in Detail Docs and in the context manifest.docs/plans/2026-08-14-why-the-issuer-renders-settled.md(new) — T3's rejected alternatives, what the chosen shape costs, and the consequences for cannobserv#303. Same split#24made for thefetch_failedopen question. The contract keeps a six-line stub pointing at it; the inbound#why-the-issuer-rendersanchor at T3 was repointed. The doc's own URL is unchanged, so sibling-repo links are unaffected — this is why the section was relocated rather than the document split.docs/TESTING.md## Test layout— the split-by-concern rule and the_integrationfilename exception.AGENTS.mdkeeps the mirror rule and routes.docs/ARCHITECTURE.mdheader — the founding-design and parent-strategy plan pointers (archival paths; the Detail Docs line now names them).Deletions, with warrants
Three, all class D, all verbatim duplication — quoted above in the disposition table. No deletion rests on an UNVERIFIABLE verdict.
Verification
verify-facts.sh --issues: 0 FALSE, 50 TRUE, 15 UNVERIFIABLE — all 15 benign (absolute paths outside the repo, illustrative templates likesrc/foo.py, bare filenames named in prose). Nothing deleted on an UNVERIFIABLE.prove-no-loss.sh --base main: 160 lines still inline, 2 relocated verbatim, 28 unaccounted. Every one is a class-C reflow or a class-D duplicate; each warrant is named above and each is verifiable againstdocs/ARCHITECTURE.md,docs/CONVENTIONS.md, or the Detail Docs index. Ledger recordsno_loss: "failed"honestly rather than claiming a clean pass.uv run pytest: 682 passed, coverage 99.83%.tests/test_boundaries.py(106 assertions incl. the AGENTS.md structural ones) andtests/test_deploy.pygreen.Note on the ledger delta
The row reads
+82because it diffs against the last recorded row (5,872 on 2026-08-11), not against today's baseline. The run's actual effect onAGENTS.mdis 6,349 → 5,954 (−395); the intervening +477 was#29landing without a curation pass.Follow-up
codebase_context_indexmust be re-run after merge — the manifest is a source, not the artifact (#19 CR #17). The manifest edits are in this PR; the embeddings are not.🤖 Generated with Claude Code