Skip to content

#22 docs: curate the context surface to the 6,000-token budget (wave A, pair 3) - #23

Merged
gregoryfoster merged 4 commits into
mainfrom
feat/22-curate-context
Aug 6, 2026
Merged

gregoryfoster merged 4 commits into
mainfrom
feat/22-curate-context

Conversation

@gregoryfoster

Copy link
Copy Markdown
Contributor

Closes #22. Wave A, pair 3 — adopt curating-context v1.2 and run it once.

Before / after

Exact counts via count_tokens (--exact succeeded; tokens_exact: true).

Before After
AGENTS.md 19,032 5,773
Lines 309 232
Budget 6,000 6,000 — under
Largest section share 52% (Bus Conventions) 18% (Code Exploration Policy)
Orphaned docs 4 0
Dead links 0 0

The issue quoted 17,521 against remote HEAD; the branch point measured 19,032. Same file, later commits.

tokens_live rose 19,032 → 50,759, which is the expected shape of a successful demotion: the always-paid cost fell 70% while the reachable surface grew because four previously-orphaned docs are now linked and five new ones exist.

Per-section disposition

Section Before Class Disposition After
(preamble) 66 A keep 66
Project Overview 354 A keep 354
Development Methodology 46 A keep 46
Environment & Tooling 515 A keep — wheelhouse mechanism + the load-bearing patch floor 543
Code Exploration Policy 1,042 — untouched; owned by init-socraticode, has its own idempotency contract 1,097
Project Layout 900 B tree → docs/ARCHITECTURE.md; 6-line signpost stays 138
Infrastructure 533 A+B single-VM rule + the Redis ≥7.0 floor stay; topology → docs/DEPLOYMENT.md 176
Server Lifecycle 745 A+B deploy/debug commands + the /etc copy trap stay; table → docs/DEPLOYMENT.md 175
Environment Variables 2,991 A+B two-env-file boundary stays; per-variable reference → docs/DEPLOYMENT.md 495
Bus Conventions 9,994 A+B nine condensed rules stay; all reasoning → docs/ARCHITECTURE.md, docs/STORAGE.md, docs/TESTING.md 994
Common Commands 352 A keep 371
Agent Skills 46 A keep 46
Conventions 1,494 A+B commit/date/general rules stay; three logging deep-dives → docs/STYLE.md 541
Detail Docs — A new — the index; class A by construction 493

Bus Conventions at 52% was the finding rather than a symptom, so it was demoted wholesale and rebuilt inline as nine rules that bind every change: two idempotency keys, store-then-publish, count=1, fail-loud from_wire with a global dispatch table, DLQ vs retry, no outbox on the consume path, the three stream kinds, seed_fetch.py as the only writer to content.fetch, and the two normative contracts.

Every relocated block, with its destination

Verbatim moves. prove-no-loss.sh --base <branch-point> accounts for all 226 non-blank lines: 106 still inline, 120 relocated, 0 unaccounted for.

Destination Tokens Lines moved Content
docs/ARCHITECTURE.md 8,091 55 ## Project Layout tree (62–92); ## Bus Conventions intro and bullets 189–211 and 214–220
docs/DEPLOYMENT.md 4,232 48 ## Infrastructure table + port prose (94–103); ### Redis is Archiver-operated (105–111); ## Server Lifecycle whole (113–136); the environment-variable reference (155–183)
docs/STORAGE.md 1,842 10 blob path and blob-mode bullets (212–213); ### Retention whole (222–231)
docs/STYLE.md 1,034 3 the three logging paragraphs (294, 296, 298)
docs/TESTING.md 905 4 Testing the bus, the fakeredis divergences, real_redis, the dedupe-key namespace (233–239)

Two normalisations the checker permits were the only edits in transit: ### Retention became ## Retention at the top of its section, and ### Redis is Archiver-operated… kept its level under ## Infrastructure. No relative markdown links needed re-levelling — the original file had none (links.refs was empty, which is why every doc was an orphan).

Every destination is under the 10,000 per-doc budget, so the demotion removed cost rather than relocating it.

Deletions

None. Phase 2 returned 41 TRUE, 50 UNVERIFIABLE, 0 FALSE — no warrant existed. The 50 UNVERIFIABLE are the expected classes: absolute VM paths (/etc/replicator/.env), URI schemes (file://, redis://…, gs://…), illustrative templates (src/foo.py), file.py::symbol references, and cross-repo issue numbers. None was treated as a licence to delete.

Also in this PR

Verification

  • uv run pytest — 444 passed, 18 deselected, coverage 99.77%
  • uv run ruff check . — clean
  • prove-no-loss.sh — exit 0, 0 unaccounted for
  • measure-context.sh --exact — links.dead: [], links.orphans: []

Two things this run did not fix

  1. docs/contracts/content-fetch-issuer-contract.md is still 14,772 tokens, over the 10,000 per-doc budget. Nothing was demoted into it, per the issue. It needs splitting on its top-level headings — worth its own issue.
  2. The daily refresh hook will break the v1.2 pin. .claude/hooks/skills-submodule-update.sh runs git submodule update --remote --merge -- skills-vendor/ once per UTC day on main and commits the result. Left alone it bumps gregoryfoster-skills past 3fc7b71 within a day, which puts two skill versions inside the wave-A arm and makes score-cohort.sh return INCONCLUSIVE. The hook has no opt-out flag and the submodule is one repo, so pinning curating-context alone is not possible — holding the pin means suspending the daily refresh for the duration of the experiment. Flagged rather than decided: that trade is a maintainer call, not a curation one.

🤖 Generated with Claude Code

gregoryfoster and others added 3 commits August 6, 2026 05:40
Bump the vendored skills pointer 549a88b -> 3fc7b71 (the merge carrying
curating-context v1.2 and the cohort validation gate) and link the skill
through the repo's two-hop vendoring convention.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
AGENTS.md was 19,032 tokens (exact, count_tokens) against a 6,000 budget,
with `Bus Conventions` alone at 52% of the file, and all four reference
docs orphaned — nothing in the policy file linked any of them.

Now 5,773 tokens, under budget, with zero orphans and zero dead links.
Every relocated block moved verbatim; prove-no-loss.sh accounts for all
226 non-blank lines of the original (106 inline, 120 relocated, 0 lost).

Five new reference docs, all under the 10,000 per-doc budget:

  docs/ARCHITECTURE.md  8,091  layout tree + the bus-contract reasoning
  docs/DEPLOYMENT.md    4,232  topology, unit lifecycle, env-var reference
  docs/STORAGE.md       1,842  blob tree + retention
  docs/STYLE.md         1,034  the logging stack
  docs/TESTING.md         905  fakeredis divergences, live-broker fixtures

A `## Detail Docs` index links those plus COMMANDS, SKILLS, and both
contracts. Nothing was deleted — Phase 2 found no FALSE verdicts, so no
warrant existed.

The context manifest is re-embedded in the same change, per this repo's
own rule that editing a description there is not what a search returns.

Also installs the context-budget write guard (PostToolUse), the other
half of the ratchet: the weekly run recovers ground, the guard stops
regrowth between runs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`skills-submodule-update.sh` runs `git submodule update --remote --merge --
skills-vendor/` once per UTC day on main and commits the result, which would
move `gregoryfoster-skills` past `3fc7b71` within a day and put two skill
versions inside the wave-A arm.

The hook takes no per-submodule exclusion and one submodule carries every
`gregoryfoster` skill, so pinning `curating-context` alone is not expressible.
Removing the SessionStart entry is the only remedy a consumer repo has today —
the same one `CannObserv/cli` took (gregoryfoster/skills#100). The hook script
stays in place, so restoring the refresh is a one-entry edit.

`docs/SKILLS.md` records the pin, why the suspension is blunter than the
problem, the manual update path while it holds, and when to restore.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gregoryfoster

Copy link
Copy Markdown
Contributor Author

Two follow-ups landed on this branch and one filed:

  • Daily skills refresh suspended (0f4fded). The hook's SessionStart entry is removed from .claude/settings.json; the script stays, so restoring it is a one-entry edit. docs/SKILLS.md records the pin at 3fc7b71, why the suspension is blunter than the problem, the manual update path while it holds, and when to restore. Upstream remedy tracked at gregoryfoster/skills#100, which is open with no fix yet — this is the same remedy CannObserv/cli took.
  • docs: split and compress content-fetch-issuer-contract.md — 14,772 tokens, over the 10,000 per-doc budget #24 filed for splitting and compressing docs/contracts/content-fetch-issuer-contract.md (14,772 tokens, the only live doc still over the 10,000 per-doc budget), with the heading seams, the sibling-link constraint, and the verification command.

1. tests/core/test_logging.py — docs/DEPLOYMENT.md is a fourth uvicorn
   command site since the Server Lifecycle demotion; it was outside
   UVICORN_COMMAND_SOURCES, so the --log-config guard passed by not looking.
2. README.md — pointed at AGENTS.md for "Not everything in the journal is
   JSON", which moved to docs/STYLE.md on this branch.
3. docs/DEPLOYMENT.md — two relocated bullets end "see **Testing the bus**"
   and "see **Retention**", intra-file references at the branch point and
   cross-file now. Destinations named on continuation lines so the original
   lines stay verbatim.
4. docs/ARCHITECTURE.md — the layout tree read "docs/ — Reference docs
   (COMMANDS, SKILLS)". Warrant: disproven. `ls docs/*.md` returns seven,
   and this branch is what made it false. The roster now points at the
   Detail Docs index rather than re-listing.
5. .socraticodecontextartifacts.json — `architecture` named ./README.md
   while ./docs/ARCHITECTURE.md was `architecture-detail`, implying a
   parent/child pair that does not exist. Now `readme` and `architecture`.
   Re-embedded via codebase_context_index.
6. docs/ARCHITECTURE.md — dropped the duplicated "Replicator is a consumer
   first" lead-in; the line stays inline in AGENTS.md.
7. docs/SKILLS.md — records skills#99: the write-guard hook symlink is
   outside doctor.sh's heal scope and dangles on a submodule-less checkout,
   failing on every edit. Also completes the manual-update snippet, which
   omitted the commit the suspended hook used to perform.
8. README.md — its 24-row environment table duplicated docs/DEPLOYMENT.md
   with neither marked authoritative. Now a five-row table of what this VM
   actually overrides, pointing at DEPLOYMENT.md for the rest.

prove-no-loss.sh reports exactly one line dropped — finding 4, with the
warrant above. Everything else is still inline or relocated verbatim.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gregoryfoster

Copy link
Copy Markdown
Contributor Author

Code review round 1 — 8 findings, all actioned (a60a9b1).

# Action Result
1 Fixed tests/core/test_logging.py — docs/DEPLOYMENT.md is a fourth uvicorn command site since the Server Lifecycle demotion and was outside UVICORN_COMMAND_SOURCES; the --log-config guard was passing by not looking
2 Fixed README.md pointed at AGENTS.md for "Not everything in the journal is JSON", which moved to docs/STYLE.md on this branch
3 Fixed Two relocated bullets in docs/DEPLOYMENT.md end see **Testing the bus** / see **Retention** — intra-file at the branch point, cross-file now. Destinations named on continuation lines so the original lines stay verbatim
4 Fixed docs/ARCHITECTURE.md layout tree read docs/ — Reference docs (COMMANDS, SKILLS)
5 Fixed Manifest architecture named ./README.md while ./docs/ARCHITECTURE.md was architecture-detail — now readme and architecture, re-embedded
6 Fixed Dropped the "Replicator is a consumer first" lead-in duplicated between AGENTS.md and docs/ARCHITECTURE.md
7 Documented docs/SKILLS.md records gregoryfoster/skills#99 — the guard-hook symlink is outside doctor.sh's heal scope and dangles on a submodule-less checkout, failing on every edit. Also completed the manual-update snippet, which omitted the commit the suspended hook used to perform
8 Fixed README.md's 24-row environment table duplicated docs/DEPLOYMENT.md with neither authoritative — now five rows of what this VM actually overrides, pointing at DEPLOYMENT.md for the rest

One deletion, with its warrant

prove-no-loss.sh --base 6a4b1c0 now reports exactly one line dropped: finding 4's docs/ — Reference docs (COMMANDS, SKILLS).

Warrant: disproven. Refuting command:

$ ls docs/*.md
docs/ARCHITECTURE.md docs/COMMANDS.md docs/DEPLOYMENT.md docs/SKILLS.md docs/STORAGE.md docs/STYLE.md docs/TESTING.md

Seven, not two — and this branch is what made the claim false. The replacement points at the Detail Docs index rather than re-listing, so it cannot go stale the same way again. The --no-loss ok recorded on the ledger row remains accurate for the state it measured; this is a subsequent, reviewed correction.

Gates, re-run after the fixes

@gregoryfoster
gregoryfoster merged commit d4a5e02 into main Aug 6, 2026
1 of 2 checks passed
@gregoryfoster
gregoryfoster deleted the feat/22-curate-context branch August 6, 2026 06:35
gregoryfoster added a commit that referenced this pull request Aug 14, 2026
CR #22: the stated rule and the enforcing code were different rules. The
docstring said percent-encoding was refused outright; the check was
`unquote(d) != d`, which only rewrites well-formed %XX — so `%zz` and a trailing
`%` passed through it and were refused by the character allow-list instead. Both
outcomes were right and the reasons disagreed, which is the drift the next reader
inherits. One check now owns every `%`, and the refusal names the rule.

CR #24: the invalid_destination message embedded the offending segment whole, and
that message is not only a log line — it becomes the failure fact's `detail` on
the wire and the `dlq_reason` on the DLQ entry. So a multi-kilobyte destination
reached two places the logging bound was introduced to protect. Bounded, along
with the character list.

CR #23: load_alias_table returns early four times before the populated case, each
building its own table, and the immutability test pinned one of them. A fifth
early return with a bare {} would have left one path mutable with every test
green — which is precisely how the original frozen=True claim came to be untrue.
One `_empty()` constructor, and the test now runs over all six construction
paths.

CR #25: recorded that provider *availability* is deliberately a code decision
while provider *provisioning* is the operator's. An env-driven provider list
would satisfy T5 literally — one variable and a public, undeletable archive.org
item is reachable. Keeping the tuple in code means adding a provider means adding
its writer, its containment rule and its tests in the same change, which is the
review an irreversible destination should get.

Verified: 645 unit tests (13 new, red first), 18 integration against the live
broker, ruff check and format clean, coverage 99.82% with both new modules at
100%.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
gregoryfoster added a commit that referenced this pull request Aug 18, 2026
CR #17 — deploy/replicator.service's own comment still said the guard
"warns only on a stale, unpushed, or dirty tree". That is the file an
operator reads while a start is failing, now describing the opposite
verdict for the condition that refused them. Needs the cp to
/etc/systemd/system/ like any unit edit, comment-only though it is.

CR #18 — docs/STYLE.md's journald inventory listed unpushed among the
warn-tier lines. It is a refusal line now, which is the half of that
inventory that matters: refusals appear exactly when something is
already wrong.

CR #19 — the deploy sequence front-loaded `git push`, but the usual path
is a PR merged on GitHub, after which this checkout is *behind* and the
push is a no-op. A deploy line whose first command routinely prints
"Everything up-to-date" is one an operator learns to skip. Both merge
paths now get their own row: pull when it merged on GitHub, push when it
merged here (AGENTS.md, README.md, docs/DEPLOYMENT.md, docs/COMMANDS.md).

CR #21 — the manifest's deployment-topology description still called the
guard branch-only.

CR #22 — the dirty-tree warning moved above the origin comparison, so
every warn prints before the one refusal that can still fire. An
operator who fixes the ahead refusal should not then discover a dirty
tree on the next start; same reasoning as printing `behind` before the
`ahead` refusal. Pinned by a test.

CR #20 (re-index the context manifest) is deliberately deferred to after
the merge — codebase_context_index has to run from the main checkout.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

curating-context: adopt the context budget (wave A, pair 3) — AGENTS.md is 17,521 tokens and all 4 docs are orphaned

1 participant