Skip to content

#138 PR 2/2: the content.capture consumer, off by default - #158

Merged
gregoryfoster merged 20 commits into
mainfrom
138-capture-consumer
Oct 26, 2026
Merged

gregoryfoster merged 20 commits into
mainfrom
138-capture-consumer

Conversation

@gregoryfoster

Copy link
Copy Markdown
Contributor

Closes #138. PR 2 of 2: the content.capture consumer, on PR 1's runtime (#157). Off by default (REPLICATOR_CAPTURE_ENABLED); merge after runbook phase A is deployed and read back.

  • Consumer: CAPTURE_SPEC (group replicator.capture, dedupe segment capture), capture.py (validate → pace → render → store → capture_stored) and capture_reporter.py, both facts on content.artifacts. No content.blobs fact, no change fingerprint.
  • Boot: the group is created at 0, never $; one test launch; a browser that can't start degrades capture alone (worker ready: capture: unavailable, a CRITICAL naming the runbook).
  • One HostPacer for fetch and capture, lifted into main.run; pace re-reads after each wait, so two loops on one host go out an interval apart (CR 14).
  • Settings: the flag, group/consumer, REPLICATOR_CAPTURE_MAX_WAIT_SECONDS (3 × wait + 30 ≤ REPLICATOR_MAX_FETCH_SECONDS, CR 5), …_MEMORY_MAX_BYTES, …_CGROUP, REPLICATOR_PLAYWRIGHT_BROWSERS_PATH.
  • Contract: docs/contracts/content-capture-issuer-contract.md, the normative reason registry (invalid_request_options, wait_timeout (terminal), destination_refused, not_fetchable, http_status, too_large, plus the loop's two). The charter moves archival capture out of future; capture_reporter.py joins DOMAIN_ECHO_MODULES.
  • Docs: AGENTS, STREAMS, COMMANDS (four streams, four DLQs), ENVIRONMENT, ARCHITECTURE, POLITENESS, CONVENTIONS, README; runbook phase B; CI's import smoke names the capture types.
  • Tests: scripts/seed_capture.py (scratch only); the loop against a scratch broker; the real shell through the handler to capture_stored.

Its final tree is the one reviewed (CR 1–36 on #138) and tested: full suite 1671 passed, -m browser 23/23, scratch-broker integration 15/15.

Post-merge host steps: runbook phase B (deploy still off → the operator writes the flag → worker ready reads capture: enabled → broker#116's acladmin read-back → archiver#380 may issue).

🤖 Generated with Claude Code

gregoryfoster and others added 20 commits October 26, 2026 14:59
…ng helpers shared with fetch

Co-authored-by: Shelley <shelley@exe.dev>
…ture alone, one pacer for fetch and capture; scratch-only seed_capture

Co-authored-by: Shelley <shelley@exe.dev>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ell through the handler to capture_stored

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…h stream across AGENTS, STREAMS, COMMANDS, ENVIRONMENT, ARCHITECTURE; runbook phase B

AGENTS.md pays for its lines with duplicate pointers (5991 -> 5989 tokens).
COMMANDS.md tightens its DLQ history and loops the per-stream rcli lines.
CI's import smoke names the capture types.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…first response is bounded too

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…full page is as wide as its overflow

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…history prose tightened, the journal-line note moved to the runbook

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… on one pacer go out an interval apart; the total wait stays under the park bound

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rest of a navigation's net::ERR either retries or reaches the delivery ceiling

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… a long screenshot; say so

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e contract's size reads width × height; the design names the real setting

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, as a 429 does

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e pacer serves two concurrent loops

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s certificate and version/cipher failures under handler_error, as the code does

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…overs, rather than "the publishers"

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… slept past (fails on the pre-CR-14 pace)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, so a leak into the default fails it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he runbook, as an unstartable one does

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…h the consumer's two end-to-end ones

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…laim lands a second into boot

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gregoryfoster
gregoryfoster merged commit 9b37afe into main Oct 26, 2026
4 checks passed
@gregoryfoster
gregoryfoster deleted the 138-capture-consumer branch October 26, 2026 16:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Viewport capture: consume the capture command, render headless, store, publish the stored-capture fact

1 participant