You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Viewport capture: consume the capture command, render headless, store, publish the stored-capture fact #138
Requested by CannObserv/archiver, which is splitting archiver#5's old fetch settings into one issue each. Operator direction (2026-10-14): viewport screenshots live on Replicator, as the byte manager, answering a dedicated command stream and publishing a fact after storage. The contracts are in the cannobserv issue linked from the comments; archiver's side is in the archiver issue linked there too.
Scope
Consume the capture command, using the competing-consumer group pattern.
Render the page in a headless browser at the requested viewport, honouring full_page and the wait condition, within the existing fetch-timeout ceiling.
Store the image in the blob store, content-addressed, with the same retention machinery as fetched blobs.
Publish the stored-capture fact only after storage succeeds. On failure, publish the failure variant.
Deduplicate on command_id, so a redelivered command does nothing.
Linked: the contracts are CannObserv/cannobserv#571. Archiver's capture policy, issuance and consumption are CannObserv/archiver#380. The browser runtime is shared with #63.
Released: cannobserv v0.37.1 (PATCH, tag v0.37.1, release commit 7b7e37f0). It carries the capture contracts from CannObserv/cannobserv#571. The bump is additive: the floors stay >=0.37,<0.38, so only the [tool.uv.sources] tags move.
Command:ContentCaptureCommand (content_capture) on content.capture. Archiver sends it to Replicator; your group is replicator.capture. It carries command_id, url, info_source_id, a viewport (CaptureViewport: width, height), full_page and an optional wait (CaptureWait: networkidle, selector with selector, or delay with seconds). It carries no headers and no credential: every value is already resolved.
Success fact:CaptureStoredEvent (capture_stored) on content.artifacts. Its fields are command_id, url, info_source_id, content_fingerprint (bare hex), blob_uri, media_type, size_bytes, image_width, image_height, final_url, captured_at and blob_expires_at. Publish it with CaptureStoredEmit.
Failure fact:CaptureFailedEvent (capture_failed) on content.artifacts, with FetchFailedEvent's shape: reason, terminal, status_code, attempts, detail. Publish it with CaptureFailedEmit. reason is your vocabulary, a str on both classes.
Keys: the command keys on command_id, and both facts on command_id:occurred_at.
What the contracts leave to Replicator (from CannObserv/cannobserv#571's post-merge notes):
Guard every request the page makes: subresources and scripts, not just the navigation. A request into a private range is destination_refused.
A capture is not a fetch occasion: no content.blobs fact, and no change fingerprint.
invalid_request_options covers an unknown wait kind, a non-positive dimension, or a delay over your ceiling. The consumer class decodes all three so you can refuse them. Whether wait_timeout is terminal is your call.
wait=None is documented as "after the load event". If the runtime behaves differently, say so here, and the docstring follows.
Write the capture issuer contract as the normative token registry: the three reason tokens fixed in the docstring (invalid_request_options, wait_timeout, destination_refused) plus any you add.
Broker:content.capture and replicator.capture still need registering, with Replicator read/ack on the stream and write on content.capture.dlq. Archiver files it (CannObserv/archiver#380). The facts need no new ACL because they ride content.artifacts.
It registers content.capture and replicator.capture. Replicator gets a group read on the stream (the same selector shape as content.fetch, content.replicate and content.persist), plus +xadd, +xlen, +xrange and +xdel on content.capture.dlq, matching its other DLQs. The capture_stored and capture_failed facts need no new grant: replicator already holds +xadd on content.artifacts. If replicator needs anything different on the DLQ, say so on CannObserv/broker#114.
Broker: content.capture is registered, and the grants went live 2026-10-20T15:06:05Z (CannObserv/broker#114; apply record: CannObserv/broker#114 (comment)). This is the broker step in this issue's order. Replicator's consumer is next.
What replicator now holds (deploy/redis-acl.conf at 7fe8f44):
content.capture is in your consume selector, beside content.fetch, content.replicate and content.persist: +xreadgroup +xack +xautoclaim +xgroup|create +xpending +xrange +xinfo|stream. These were derived from the spec-driven loop, not captured. If the capture loop issues anything else on this stream, say so on broker#114 before it ships.
content.capture.dlq: +xadd, +xlen +xrange and +xdel, but no +xtrim. That matches your other three queues.
Dedupe keys replicator:cmd:capture:<command_id> are already inside ~replicator:cmd:*.
The outcomes, capture_stored and capture_failed, ride content.artifacts, which you already hold +xadd on. No new grant was needed.
The group: create replicator.capture when the consumer starts, before archiver's first command. Never create it at $ against a stream that already holds entries. Broker's probe row for it is not dormant: if archiver writes the stream before the group exists, every tick reports group-missing. Don't verify the group as replicator. You hold no XINFO GROUPS, so the read would be denied and would land in broker's ACL LOG. Broker reads it back as acladmin at your deploy: CannObserv/broker#116.
The undelivered threshold is the shared 300 s. It's sized from your own stop-budget sum for one handler: a 5 s poll, 5 s pacing, the 120 s REPLICATOR_MAX_FETCH_SECONDS a render runs within, and the 30 s REPLICATOR_BLOB_TIMEOUT_SECONDS, for 160 s total (tests/test_deploy.py at 9fcd33b). Broker mirrors that as CAPTURE_HANDLER_CEILING_SECONDS. Tell broker if this issue changes it: a capture ceiling of renders' own past about 260 s, or a read loop that isn't a blocking count=1 read, needs a per-group threshold.
Doc drift on your side:docs/COMMANDS.md still says "the three command streams" and "the three .dlq streams" (line 147) and quotes (+xdel ~content.fetch.dlq ~content.replicate.dlq ~content.persist.dlq) (lines 167-171). The selector now names four queues. That's yours to update when the capture loop lands. Broker hasn't changed it.
Correction to broker's notice above: replicator.capture's undelivered threshold is 320 s, not 300 s (CannObserv/broker#114, CR 2, 8f09282).
The notice sized the threshold against one capture handler at your ceiling, 160 s by your stop-budget sum. Your PollCadence docstring (src/worker/loop.py, #98) gives the right bound: the handler in flight plus at most one reclaim, "two handler durations, which is the figure an undelivered-age threshold has to clear". So the threshold is now 2 * CAPTURE_HANDLER_CEILING_SECONDS, which is 320 s, derived.
What replaces "past about 260 s": the threshold follows the ceiling, so there's no fixed limit to stay under. Tell broker (on CannObserv/broker#114 or a new broker issue) if this issue:
gives renders a capture ceiling of their own, outside REPLICATOR_MAX_FETCH_SECONDS;
changes the handler sum's terms (poll, pacing bound, fetch deadline, store timeout); or
moves the capture loop off the blocking count=1 read with PollCadence.
Any of those means broker re-mirrors the sum. replicator.fetch shares the 160 s handler and still takes 300 s: that gap is CannObserv/broker#117, and broker owns it.
Where these disagree with each other or with the code, stop and ask.
Size L. Design doc first:docs/plans/<date>-viewport-capture-design.md, committed as #138 docs: add design doc for viewport capture. This issue is its approval issue. It's a contract change (a new command stream and a new issuer contract), so AGENTS.md requires one. The doc also chooses the browser runtime that #63 will reuse. Let the doc propose the PR split. Expect at least two: runtime plus guard, then the consumer.
The types are in co_core.pure.models.changes: ContentCaptureCommand, CaptureStoredEvent / CaptureFailedEvent, CaptureStoredEmit / CaptureFailedEmit, and CONTENT_CAPTURE.
CONTENT_CAPTURE is a command kind in _STREAM_KINDS: one group, competing consumers.
The keys are command_id for the command and command_id:occurred_at for both facts, the same as persist.
Consume with the canonical ContentCaptureCommand (extra="ignore"), never the *Emit classes. ContentCaptureCommandEmit re-validates viewport / wait through model_dump().
CI's import smoke doesn't name the capture types yet. Add them with the consumer.
tests/test_co_core_pin.py guards the four pin sites if you raise the floor. Nothing here should need it.
Operator decisions (starting the thread approves the recommendation)
[Operator decision: the browser runtime — recommend a pinned, service-owned headless Chromium (e.g. a pinned Playwright release and its chromium-headless-shell, installed to a path the unit owns), not /headless-shell. That binary (Chromium 156.0.8078.48, 243M, dated 2026-10-21) [measured] is documented only as Shelley's browser tool, not as a platform runtime. Shelley is disabled here (Disable the Shelley agent on co-replicator — the documented command does not, because the socket re-activates it #97), and the image can change it under us. The design doc may propose a lighter CDP-direct option, with evidence.]
[Operator decision: concurrency — recommend one render at a time, in its own loop. That's a blocking count=1 read with PollCadence, like the other three, so broker's threshold arithmetic holds.]
All browser traffic through a local proxy that applies the guard's policy, with direct connections disabled.
Request interception, which carries the TOCTOU residual egress.py already names (DNS rebinding).
Either way, also turn off WebRTC and QUIC, which escape a proxy.
The OOM posture [measured]. replicator.service runs at OOMScoreAdjust=-900 with MemoryLow=128M and no MemoryMax. A child process inherits -900, so a runaway render would make the kernel kill sessions and tailscaled first. The browser needs its own positive oom_score_adj and a memory cap of its own (a scope, or a cgroup the unit delegates). Read docs/MEMORY-PROTECTION.md and host memory first. The host has 7.7Gi in total, 5.9Gi available.
The group. Create it at boot, never at $ against a stream that holds entries. Never read it back as replicator: XINFO GROUPS is denied and lands in broker's ACL LOG.
The charter.docs/contracts/replicator-boundaries.md:90 names "alternate fetch drivers (browser, archival capture)" as future. Run its three tests against capture, and change the charter and tests/test_boundaries.py together. That doc is about 9726 of 10000 tokens [stated, 2026-10-18], so it may need splitting before it grows.
The AGENTS.md budget. It's at 5991 of 6000 [estimate, 2026-10-24]. Bus Conventions' "Three stream kinds" bullet and the Detail Docs list both grow. Pay for it with curating-context tightenings, not overflow.
Doc drift that broker flagged:
docs/COMMANDS.md:147 says "the three command streams" and "the three .dlq streams";
its +xdel selector quote names three queues;
docs/STREAMS.md and docs/ARCHITECTURE.md carry the stream roster too.
The contract's open items are yours:
write docs/contracts/content-capture-issuer-contract.md as the normative reason registry (invalid_request_options, wait_timeout, destination_refused, plus any you add);
decide whether wait_timeout is terminal;
report on cannobserv if wait=None isn't "after the load event" in the chosen runtime;
captures go in the temporary BlobStore under the same TTL, with blob_expires_at populated;
per-host pacing applies to the navigation;
a capture produces no content.blobs fact and no fingerprint.
Template
#114's persist stream, the last new command stream here. It had:
an enable flag, off by default;
the group at boot;
an issuer contract (docs/contracts/content-persist-issuer-contract.md);
a design doc and a runbook (docs/plans/2026-09-25-*).
Brief review against the code and host (2026-10-25)
Read against origin/main2790a53, plus a throwaway measurement spike in /tmp (Playwright 1.64.1, chromium_headless_shell-1248; nothing installed into the repo or the unit). Where the brief holds I say nothing. These are corrections and additions.
Corrections
Disk: ~416M, not 200–300M [measured]. The playwright package is 138M, because it bundles its own Node (v24.22.1) driver. chromium_headless_shell-1248 is 273M, and Playwright's installer also pulls ffmpeg-1013, 5.1M. uv's cache holds a second copy of the wheel. The root has 4.5G free.
The pacer isn't shareable as wired [traced]. build_handler builds its own HostPacer when none is injected (src/worker/handler.py:268), and main.run injects none. If a capture loop builds a second pacer, the two loops are each polite on their own, which is the "fiction" charter test 2 names. The pacer has to move up to main.run and be passed to both handlers.
The info_source_id carve-out is a module list [traced]. DOMAIN_ECHO_MODULES (tests/test_boundaries.py:171) names four modules today, and replicate_reporter.py is one of them. A capture handler or reporter that echoes info_source_id joins that list. Per AGENTS.md, that's a charter edit, separate from moving "archival capture" out of future at replicator-boundaries.md:90.
Additions (hazards the brief doesn't name)
4. Playwright launches Chromium with --no-sandbox by default [measured: the launch argv]. A renderer compromise would then run as exedev, which can read /etc/replicator/.env (the GCS key path and the broker URL). chromium_sandbox=True launches cleanly on this host, because unprivileged user namespaces are available. The design turns the sandbox on and makes it a test.
5. Playwright's own DNS-leak rule is inert in this build [measured]. With a SOCKS proxy it passes --host-resolver-rules="MAP * ~NOTFOUND , EXCLUDE 127.0.0.1", and chrome-headless-shell logs Failed parsing rule for it on every launch. What actually holds is SOCKS5 remote resolution: the proxy received hostnames (ATYP 3), and strace of the browser showed no port-53 traffic. The only non-loopback connect() was the IPv6 reachability probe to 2001:4860:4860::8888, a UDP connect that sends no packet.
6. WebRTC escapes the proxy unless flagged [measured]. A loopback canary saw four STUN datagrams without --force-webrtc-ip-handling-policy=disable_non_proxied_udp, and none with it. TCP to loopback, [::1], and RFC 1918, including through a 302 hop, fetch(), WebSocket, sendBeacon and prefetch, all went through the proxy either way: Playwright adds <-loopback> itself. That confirms the brief's hazard with a number.
7. The host has 2 vCPUs [measured]. A full-page render of a long article (en.wikipedia Cannabis, 1366×28108, 7.9 MB PNG) spent 31 s in the screenshot step alone. That's inside the 120 s REPLICATOR_MAX_FETCH_SECONDS, but it's CPU the fetch loop shares.
8. Memory, by cgroup memory.peak [measured]: the whole tree peaked at 199 MiB for example.com. For the Wikipedia full page, the browser peaked at 768 MiB and Python plus the Node driver at 230 MiB.
A cap has to sit on the browser's own cgroup. A cap on a scope holding Python OOM-killed Python. A browser-only child cgroup with memory.oom.group=1 at 400M group-killed the six browser processes, left the worker untouched, and Playwright raised TargetClosedError.
systemd here is 255, so DelegateSubgroup= is available.
wait=None = after load holds [measured]. Playwright's goto waits for load by default. A spike page whose text changed 1.5 s after load was captured with the old text. There's nothing to report on cannobserv.
Recommends: pinned Playwright plus chromium-headless-shell; a per-render SOCKS5 egress guard that connects to the address it checked; a delegated browser/ cgroup (memory.max, oom.group, adj +500); the renderer sandbox on.
PR split: 1. runtime and guard; 2. the consumer.
Stop budget: unchanged at 160 s, so there's nothing to tell broker#116.
Open calls: six, listed under For the operator. No code until approved.
PR 1 of 2 merged: the browser runtime, egress guard and delegated cgroup (#157, 6580979)
Shelley's thread was picked up mid-PR 2; three review rounds (CR 1–36) ran over both halves before the split. This half carries its own fixes.
What's in it
DestinationGuard shared by fetch and the browser; browser_egress.py, a per-render SOCKS5 guard that vets every connection a page opens and dials the address it checked (CR 3, 4: IDNA2008 names relayed; a name that can't be encoded is unreachable, not a refusal).
browser.py: pinned Playwright 1.64.1 / chromium-headless-shell 1248, sandbox on, scrubbed env; size limits checked before and after the shot (CR 2, 12, 18); net::ERR classified as a network failure, an impossible navigation (ERR_UNSAFE_PORT, ERR_UNSAFE_REDIRECT, both measured) or neither (CR 15, 23); teardown always reaches cgroup.kill (CR 6).
The unit: Delegate=memory pids, DelegateSubgroup=worker, KillMode=mixed (CR 7: a stop no longer kills the render in flight; uv 0.13.5 forwards SIGTERM, measured). The worker enables the memory controller and re-claims memory.low on its leaf at every boot (CR 1: without it the worker would have lost its Protect tailscaled on co-replicator: second on the OOM list, and the worker's only path to the bus #113 reservation on the first boot of the new unit).
CI's browser job: tolerant apt, packages from the dry run, ldd decides (CR 24, 31–33).
Verified
Merged main: pre-ship 1560 passed, ruff clean.
-m browser21/21 on GitHub's ubuntu-24.04 runner (the job's first run; the delegated user scope works there) and 21/21 here.
Accepted limits / residuals: an unauthenticated per-render listener a local process can trip (CR 10); Playwright's Node driver unconfined in worker/, unreachable from a page (CR 11). IPv6 forms embedding a private IPv4 address: #156.
Not deployed yet. Phase A of docs/plans/2026-10-25-viewport-capture-runbook.md is the operator's (the unit cp is a production write). PR 2 (the consumer) follows; it carries Closes #138.
For #63: reuse Renderer, the SOCKS guard and the cgroup; the cap is per render, so concurrent renders need serialising or per-render child cgroups.
Shipped: PR 2 of 2, the content.capture consumer, off by default (#158, 9b37afe)
What's in it
Consumer:CAPTURE_SPEC (group replicator.capture) with capture.py and capture_reporter.py. The flow is validate, pace, render, store, then capture_stored on content.artifacts. There's no content.blobs fact and no change fingerprint.
Boot: the group is created at 0, never $. A browser that can't start disables capture alone; worker ready then reads capture: unavailable.
Pacing: fetch and capture share one HostPacer, and pace re-reads the window after each wait (CR 14).
Contract:docs/contracts/content-capture-issuer-contract.md is the normative reason registry. wait_timeout is terminal. not_fetchable also covers URLs the browser won't open.
Verified: CI green on #158's head (the browser job ran 23/23); pre-ship passed on merged main. Phase A is done on co-replicator:
the delegated unit is live, with worker/memory.low = 134217728 and 13/13 host deploy tests passing;
the pinned shell is installed (revision 1248, 273M; root 81%, 3.6G free), ldd finds nothing missing, and the real-browser suite passed 21/21 there.
B1: deploy 9b37afe with the flag still off (git pull is a no-op here; uv sync --frozen, then sudo systemctl restart replicator). worker ready should read "capture": "disabled" and "build": "9b37afe".
B2, the operator's write: back up /etc/replicator/.env, append REPLICATOR_CAPTURE_ENABLED=true, restart. Expect "capture": "enabled", "capture_group": "replicator.capture", no CRITICAL line, and 0 pending on content.capture.
Verification debt: the first real stored a capture and published capture_stored waits on archiver#380. Broker's read-back waits on B3.
Follow-ups:#156 (IPv6 forms that embed a private IPv4 address). A pre-existing race on main between the boot sweep and the store's shard mkdir makes test_run_dispatches_to_the_byte_path fail about 1 run in 50; not filed yet.
For #63: reuse Renderer, the SOCKS guard and the browser/ cgroup. The memory cap is per render, so concurrent renders need serialising or per-render child cgroups.
No CRITICAL or ERROR lines; XPENDING content.capture replicator.capture shows 0.
The browser/ cgroup is at 1536 MiB, swap 0, oom.group 1, 512 pids, and empty after the boot launch.
Hand-offs posted: broker's acladmin read-back on CannObserv/broker#116, and issuing go-ahead on CannObserv/archiver#380. The remaining debt is the first real capture_stored (waits on archiver#380) and broker's read-back. Finding 37 is filed as #159.
Activity
gregoryfoster commentedon Oct 14, 2026
Linked: the contracts are CannObserv/cannobserv#571. Archiver's capture policy, issuance and consumption are CannObserv/archiver#380. The browser runtime is shared with #63.
gregoryfoster commentedon Oct 19, 2026
Released: cannobserv v0.37.1 (PATCH, tag
v0.37.1, release commit7b7e37f0). It carries the capture contracts from CannObserv/cannobserv#571. The bump is additive: the floors stay>=0.37,<0.38, so only the[tool.uv.sources]tags move.The contracts (pair doc: docs/change_bus/capture.md):
ContentCaptureCommand(content_capture) oncontent.capture. Archiver sends it to Replicator; your group isreplicator.capture. It carriescommand_id,url,info_source_id, aviewport(CaptureViewport:width,height),full_pageand an optionalwait(CaptureWait:networkidle,selectorwithselector, ordelaywithseconds). It carries no headers and no credential: every value is already resolved.CaptureStoredEvent(capture_stored) oncontent.artifacts. Its fields arecommand_id,url,info_source_id,content_fingerprint(bare hex),blob_uri,media_type,size_bytes,image_width,image_height,final_url,captured_atandblob_expires_at. Publish it withCaptureStoredEmit.CaptureFailedEvent(capture_failed) oncontent.artifacts, withFetchFailedEvent's shape:reason,terminal,status_code,attempts,detail. Publish it withCaptureFailedEmit.reasonis your vocabulary, astron both classes.command_id, and both facts oncommand_id:occurred_at.What the contracts leave to Replicator (from CannObserv/cannobserv#571's post-merge notes):
destination_refused.BlobStoreas fetched blobs, under the same TTL, and populateblob_expires_at. The Boundary decision: may the blob store hold a derived sidecar Replicator cannot rebuild? (blocks watcher#222 and three dependents) #69 ruling holds only while a capture is TTL-bounded. To keep a capture, Archiver issues acontent.persist.content.blobsfact, and no change fingerprint.invalid_request_optionscovers an unknown waitkind, a non-positive dimension, or adelayover your ceiling. The consumer class decodes all three so you can refuse them. Whetherwait_timeoutis terminal is your call.wait=Noneis documented as "after theloadevent". If the runtime behaves differently, say so here, and the docstring follows.reasontokens fixed in the docstring (invalid_request_options,wait_timeout,destination_refused) plus any you add.Broker:
content.captureandreplicator.capturestill need registering, with Replicator read/ack on the stream and write oncontent.capture.dlq. Archiver files it (CannObserv/archiver#380). The facts need no new ACL because they ridecontent.artifacts.Changelog: https://github.com/CannObserv/cannobserv/blob/v0.37.1/CHANGELOG.md#0371---2026-10-18
Refs CannObserv/cannobserv#581.
gregoryfoster commentedon Oct 19, 2026
Broker registration filed: CannObserv/broker#114 (from broker's coordinator, CannObserv/broker#80, 2026-10-19, on the operator's direction).
It registers
content.captureandreplicator.capture. Replicator gets a group read on the stream (the same selector shape ascontent.fetch,content.replicateandcontent.persist), plus+xadd,+xlen,+xrangeand+xdeloncontent.capture.dlq, matching its other DLQs. Thecapture_storedandcapture_failedfacts need no new grant: replicator already holds+xaddoncontent.artifacts. If replicator needs anything different on the DLQ, say so on CannObserv/broker#114.Order on broker's side: after CannObserv/broker#93. CannObserv/broker#114 will record the time its ACL goes live, which is this issue's step after the contracts.
gregoryfoster commentedon Oct 20, 2026
Broker:
content.captureis registered, and the grants went live 2026-10-20T15:06:05Z (CannObserv/broker#114; apply record: CannObserv/broker#114 (comment)). This is the broker step in this issue's order. Replicator's consumer is next.What
replicatornow holds (deploy/redis-acl.confat7fe8f44):content.captureis in your consume selector, besidecontent.fetch,content.replicateandcontent.persist:+xreadgroup +xack +xautoclaim +xgroup|create +xpending +xrange +xinfo|stream. These were derived from the spec-driven loop, not captured. If the capture loop issues anything else on this stream, say so on broker#114 before it ships.content.capture.dlq:+xadd,+xlen +xrangeand+xdel, but no+xtrim. That matches your other three queues.replicator:cmd:capture:<command_id>are already inside~replicator:cmd:*.capture_storedandcapture_failed, ridecontent.artifacts, which you already hold+xaddon. No new grant was needed.The group: create
replicator.capturewhen the consumer starts, before archiver's first command. Never create it at$against a stream that already holds entries. Broker's probe row for it is not dormant: if archiver writes the stream before the group exists, every tick reportsgroup-missing. Don't verify the group asreplicator. You hold noXINFO GROUPS, so the read would be denied and would land in broker'sACL LOG. Broker reads it back asacladminat your deploy: CannObserv/broker#116.The undelivered threshold is the shared 300 s. It's sized from your own stop-budget sum for one handler: a 5 s poll, 5 s pacing, the 120 s
REPLICATOR_MAX_FETCH_SECONDSa render runs within, and the 30 sREPLICATOR_BLOB_TIMEOUT_SECONDS, for 160 s total (tests/test_deploy.pyat9fcd33b). Broker mirrors that asCAPTURE_HANDLER_CEILING_SECONDS. Tell broker if this issue changes it: a capture ceiling of renders' own past about 260 s, or a read loop that isn't a blockingcount=1read, needs a per-group threshold.Doc drift on your side:
docs/COMMANDS.mdstill says "the three command streams" and "the three.dlqstreams" (line 147) and quotes(+xdel ~content.fetch.dlq ~content.replicate.dlq ~content.persist.dlq)(lines 167-171). The selector now names four queues. That's yours to update when the capture loop lands. Broker hasn't changed it.gregoryfoster commentedon Oct 20, 2026
Correction to broker's notice above:
replicator.capture's undelivered threshold is 320 s, not 300 s (CannObserv/broker#114, CR 2,8f09282).The notice sized the threshold against one capture handler at your ceiling, 160 s by your stop-budget sum. Your
PollCadencedocstring (src/worker/loop.py, #98) gives the right bound: the handler in flight plus at most one reclaim, "two handler durations, which is the figure an undelivered-age threshold has to clear". So the threshold is now2 * CAPTURE_HANDLER_CEILING_SECONDS, which is 320 s, derived.What replaces "past about 260 s": the threshold follows the ceiling, so there's no fixed limit to stay under. Tell broker (on CannObserv/broker#114 or a new broker issue) if this issue:
REPLICATOR_MAX_FETCH_SECONDS;count=1read withPollCadence.Any of those means broker re-mirrors the sum.
replicator.fetchshares the 160 s handler and still takes 300 s: that gap is CannObserv/broker#117, and broker owns it.gregoryfoster commentedon Oct 25, 2026
Brief (coordinator, epic #154, 2026-10-25)
Authorities, in order:
Where these disagree with each other or with the code, stop and ask.
Size L. Design doc first:
docs/plans/<date>-viewport-capture-design.md, committed as#138 docs: add design doc for viewport capture. This issue is its approval issue. It's a contract change (a new command stream and a new issuer contract), so AGENTS.md requires one. The doc also chooses the browser runtime that #63 will reuse. Let the doc propose the PR split. Expect at least two: runtime plus guard, then the consumer.Changed since filed
2790a53, deployed 2026-10-24T22:36Z) [measured].co_core.pure.models.changes:ContentCaptureCommand,CaptureStoredEvent/CaptureFailedEvent,CaptureStoredEmit/CaptureFailedEmit, andCONTENT_CAPTURE.CONTENT_CAPTUREis acommandkind in_STREAM_KINDS: one group, competing consumers.command_idfor the command andcommand_id:occurred_atfor both facts, the same as persist.ContentCaptureCommand(extra="ignore"), never the*Emitclasses.ContentCaptureCommandEmitre-validatesviewport/waitthroughmodel_dump().tests/test_co_core_pin.pyguards the four pin sites if you raise the floor. Nothing here should need it.replicator.capturesharesFETCH_HANDLER_WARN_UNDELIVERED_AGE_SECONDS: 320 s, which is 2 × the 160 s fetch-handler sum (replicator.fetch's undelivered threshold (300 s) is under its two-handler bound (320 s) broker#117).Operator decisions (starting the thread approves the recommendation)
chromium-headless-shell, installed to a path the unit owns), not/headless-shell. That binary (Chromium 156.0.8078.48, 243M, dated 2026-10-21) [measured] is documented only as Shelley's browser tool, not as a platform runtime. Shelley is disabled here (Disable the Shelley agent on co-replicator — the documented command does not, because the socket re-activates it #97), and the image can change it under us. The design doc may propose a lighter CDP-direct option, with evidence.]REPLICATOR_CAPTURE_ENABLED, off by default, with the persist precedent. Turning it on createsreplicator.captureat boot, so the operator turns it on at go-live, before Archiver's first command, and broker reads the group back asacladmin(Go-live check for content.capture: confirm replicator.capture exists before archiver's first command broker#116).]count=1read withPollCadence, like the other three, so broker's threshold arithmetic holds.]Hazards, to weigh
The egress guard doesn't cover a browser [traced,
src/worker/egress.py].GuardedTransportis an httpx transport, called once per hop. Chromium has its own network stack, so every subresource, redirect, script fetch, websocket, service worker, prefetch and DNS-over-HTTPS lookup bypasses it. The contract requiresdestination_refusedfor any request the page makes into a private range (content.fetch trust: the contract's escalation trigger has fired - decide signing, a URL allowlist, or a destination guard #89, Destination guard: refuse loopback, private and tailnet addresses before a fetch leaves the host (#89 decision) #95: loopback, RFC 1918, the tailnet). Weigh two options:egress.pyalready names (DNS rebinding).Either way, also turn off WebRTC and QUIC, which escape a proxy.
The OOM posture [measured].
replicator.serviceruns atOOMScoreAdjust=-900withMemoryLow=128Mand noMemoryMax. A child process inherits -900, so a runaway render would make the kernel kill sessions andtailscaledfirst. The browser needs its own positiveoom_score_adjand a memory cap of its own (a scope, or a cgroup the unit delegates). Readdocs/MEMORY-PROTECTION.mdand host memory first. The host has 7.7Gi in total, 5.9Gi available.Disk. The root is at 77%, and Host disk GC: adopt provisioner#61's canonical GC + headroom sensor on co-replicator (root hit 97%) #148's GC isn't installed yet. A browser install is roughly 200–300M. Measure it, and put the number in the doc.
The handler ceiling. A render inside
REPLICATOR_MAX_FETCH_SECONDSkeeps broker's 320 s valid. Tell broker on Go-live check for content.capture: confirm replicator.capture exists before archiver's first command broker#116 if the design gives renders a ceiling of their own, or a read loop that isn't a blockingcount=1read.The group. Create it at boot, never at
$against a stream that holds entries. Never read it back asreplicator:XINFO GROUPSis denied and lands in broker'sACL LOG.The charter.
docs/contracts/replicator-boundaries.md:90names "alternate fetch drivers (browser, archival capture)" as future. Run its three tests against capture, and change the charter andtests/test_boundaries.pytogether. That doc is about 9726 of 10000 tokens [stated, 2026-10-18], so it may need splitting before it grows.The AGENTS.md budget. It's at 5991 of 6000 [estimate, 2026-10-24]. Bus Conventions' "Three stream kinds" bullet and the Detail Docs list both grow. Pay for it with
curating-contexttightenings, not overflow.Doc drift that broker flagged:
docs/COMMANDS.md:147says "the three command streams" and "the three.dlqstreams";+xdelselector quote names three queues;docs/STREAMS.mdanddocs/ARCHITECTURE.mdcarry the stream roster too.The contract's open items are yours:
docs/contracts/content-capture-issuer-contract.mdas the normativereasonregistry (invalid_request_options,wait_timeout,destination_refused, plus any you add);wait_timeoutis terminal;wait=Noneisn't "after theloadevent" in the chosen runtime;BlobStoreunder the same TTL, withblob_expires_atpopulated;content.blobsfact and no fingerprint.Template
#114's persist stream, the last new command stream here. It had:
docs/contracts/content-persist-issuer-contract.md);docs/plans/2026-09-25-*).Expected side effects
XADDtocontent.capture, so the first real capture waits on Viewport screenshots: per-source capture policy, issued to Replicator on a dedicated command stream archiver#380's issuance. Say what you verified instead (a scratch broker, an integration test against a known page), and list the end-to-end check as debt.Out of scope
Post-merge host steps (for the issue, when shipping)
REPLICATOR_CAPTURE_ENABLEDto/etc/replicator/.env. Production writes are the operator's.worker readyreports capture.acladminread-back and itsACL LOGcheck.codebase_update, if the manifest changed.gregoryfoster commentedon Oct 25, 2026
Brief review against the code and host (2026-10-25)
Read against
origin/main2790a53, plus a throwaway measurement spike in/tmp(Playwright 1.64.1,chromium_headless_shell-1248; nothing installed into the repo or the unit). Where the brief holds I say nothing. These are corrections and additions.Corrections
playwrightpackage is 138M, because it bundles its own Node (v24.22.1) driver.chromium_headless_shell-1248is 273M, and Playwright's installer also pullsffmpeg-1013, 5.1M. uv's cache holds a second copy of the wheel. The root has 4.5G free.build_handlerbuilds its ownHostPacerwhen none is injected (src/worker/handler.py:268), andmain.runinjects none. If a capture loop builds a second pacer, the two loops are each polite on their own, which is the "fiction" charter test 2 names. The pacer has to move up tomain.runand be passed to both handlers.info_source_idcarve-out is a module list [traced].DOMAIN_ECHO_MODULES(tests/test_boundaries.py:171) names four modules today, andreplicate_reporter.pyis one of them. A capture handler or reporter that echoesinfo_source_idjoins that list. Per AGENTS.md, that's a charter edit, separate from moving "archival capture" out of future atreplicator-boundaries.md:90.Additions (hazards the brief doesn't name)
4. Playwright launches Chromium with
--no-sandboxby default [measured: the launch argv]. A renderer compromise would then run asexedev, which can read/etc/replicator/.env(the GCS key path and the broker URL).chromium_sandbox=Truelaunches cleanly on this host, because unprivileged user namespaces are available. The design turns the sandbox on and makes it a test.5. Playwright's own DNS-leak rule is inert in this build [measured]. With a SOCKS proxy it passes
--host-resolver-rules="MAP * ~NOTFOUND , EXCLUDE 127.0.0.1", andchrome-headless-shelllogsFailed parsing rulefor it on every launch. What actually holds is SOCKS5 remote resolution: the proxy received hostnames (ATYP 3), andstraceof the browser showed no port-53 traffic. The only non-loopbackconnect()was the IPv6 reachability probe to2001:4860:4860::8888, a UDPconnectthat sends no packet.6. WebRTC escapes the proxy unless flagged [measured]. A loopback canary saw four STUN datagrams without
--force-webrtc-ip-handling-policy=disable_non_proxied_udp, and none with it. TCP to loopback,[::1], and RFC 1918, including through a 302 hop,fetch(), WebSocket,sendBeaconand prefetch, all went through the proxy either way: Playwright adds<-loopback>itself. That confirms the brief's hazard with a number.7. The host has 2 vCPUs [measured]. A full-page render of a long article (en.wikipedia
Cannabis, 1366×28108, 7.9 MB PNG) spent 31 s in the screenshot step alone. That's inside the 120 sREPLICATOR_MAX_FETCH_SECONDS, but it's CPU the fetch loop shares.8. Memory, by cgroup
memory.peak[measured]: the whole tree peaked at 199 MiB forexample.com. For the Wikipedia full page, the browser peaked at 768 MiB and Python plus the Node driver at 230 MiB.memory.oom.group=1at 400M group-killed the six browser processes, left the worker untouched, and Playwright raisedTargetClosedError.DelegateSubgroup=is available.memory_recursiveprot, the unit'sMemoryLow=128Misn't handed down to the worker's leaf cgroup, so the worker would have to claim it there. That touches Protect tailscaled on co-replicator: second on the OOM list, and the worker's only path to the bus #113's slice-sum test.wait=None= afterloadholds [measured]. Playwright'sgotowaits forloadby default. A spike page whose text changed 1.5 s afterloadwas captured with the old text. There's nothing to report on cannobserv.The design doc follows on this branch.
gregoryfoster commentedon Oct 25, 2026
Design doc for approval:
docs/plans/2026-10-25-viewport-capture-design.mdon branch138-viewport-capture(56dacdb).chromium-headless-shell; a per-render SOCKS5 egress guard that connects to the address it checked; a delegatedbrowser/cgroup (memory.max,oom.group, adj +500); the renderer sandbox on.#138 refactor: lift the destination check into DestinationGuard for a…
#138 feat: the browser's SOCKS5 egress guard, vetting every connectio…
#138 feat: the confined renderer — pinned Playwright, delegated brows…
#138 docs: runbook for the browser runtime and the delegated unit; me…
#138 fix: CR 1 — the memory.low claim enables the memory controller f…
#138 fix: CR 2 — refuse a PNG taller than the limit, not only a page …
#138 fix: CR 3 — a SOCKS hostname is ASCII passed through, and a non-…
12 remaining items
#138 fix: CR 32 — a dependency dry run that failed is warned about, n…
#138 fix: CR 33 — the CI dependency install is tolerant, so ldd decid…
#138 docs: CR 25 — runbook A2 checks the shell with ldd; the dry run'…
#138 docs: CR 34 — the dry run's 21 are installs and upgrades, not 21…
#138 docs: CR 20 — runbook A's pass condition is PR 1's 21 browser te…
#138 test: the sandbox test polls for the renderer instead of one loo…
Merge #138: PR 1 — the browser runtime, its SOCKS egress guard and th…
gregoryfoster commentedon Oct 26, 2026
PR 1 of 2 merged: the browser runtime, egress guard and delegated cgroup (#157,
6580979)Shelley's thread was picked up mid-PR 2; three review rounds (CR 1–36) ran over both halves before the split. This half carries its own fixes.
What's in it
DestinationGuardshared by fetch and the browser;browser_egress.py, a per-render SOCKS5 guard that vets every connection a page opens and dials the address it checked (CR 3, 4: IDNA2008 names relayed; a name that can't be encoded is unreachable, not a refusal).browser.py: pinned Playwright 1.64.1 /chromium-headless-shell1248, sandbox on, scrubbed env; size limits checked before and after the shot (CR 2, 12, 18);net::ERRclassified as a network failure, an impossible navigation (ERR_UNSAFE_PORT,ERR_UNSAFE_REDIRECT, both measured) or neither (CR 15, 23); teardown always reachescgroup.kill(CR 6).Delegate=memory pids,DelegateSubgroup=worker,KillMode=mixed(CR 7: a stop no longer kills the render in flight; uv 0.13.5 forwards SIGTERM, measured). The worker enables the memory controller and re-claimsmemory.lowon its leaf at every boot (CR 1: without it the worker would have lost its Protect tailscaled on co-replicator: second on the OOM list, and the worker's only path to the bus #113 reservation on the first boot of the new unit).browserjob: tolerant apt, packages from the dry run,ldddecides (CR 24, 31–33).Verified
main: pre-ship 1560 passed, ruff clean.-m browser21/21 on GitHub's ubuntu-24.04 runner (the job's first run; the delegated user scope works there) and 21/21 here.Accepted limits / residuals: an unauthenticated per-render listener a local process can trip (CR 10); Playwright's Node driver unconfined in
worker/, unreachable from a page (CR 11). IPv6 forms embedding a private IPv4 address: #156.Not deployed yet. Phase A of
docs/plans/2026-10-25-viewport-capture-runbook.mdis the operator's (the unitcpis a production write). PR 2 (the consumer) follows; it carriesCloses #138.For #63: reuse
Renderer, the SOCKS guard and the cgroup; the cap is per render, so concurrent renders need serialising or per-render child cgroups.#138 docs: runbook A1 reads the worker leaf after worker ready; the c…
Merge #138: PR 2 — the content.capture consumer, off by default (#158)
gregoryfoster commentedon Oct 26, 2026
Shipped: PR 2 of 2, the
content.captureconsumer, off by default (#158,9b37afe)What's in it
CAPTURE_SPEC(groupreplicator.capture) withcapture.pyandcapture_reporter.py. The flow is validate, pace, render, store, thencapture_storedoncontent.artifacts. There's nocontent.blobsfact and no change fingerprint.0, never$. A browser that can't start disables capture alone;worker readythen readscapture: unavailable.HostPacer, andpacere-reads the window after each wait (CR 14).docs/contracts/content-capture-issuer-contract.mdis the normativereasonregistry.wait_timeoutis terminal.not_fetchablealso covers URLs the browser won't open.Verified: CI green on #158's head (the browser job ran 23/23); pre-ship passed on merged
main. Phase A is done onco-replicator:worker/memory.low= 134217728 and 13/13 host deploy tests passing;lddfinds nothing missing, and the real-browser suite passed 21/21 there.Post-merge host steps (runbook phase B,
docs/plans/2026-10-25-viewport-capture-runbook.md)9b37afewith the flag still off (git pullis a no-op here;uv sync --frozen, thensudo systemctl restart replicator).worker readyshould read"capture": "disabled"and"build": "9b37afe"./etc/replicator/.env, appendREPLICATOR_CAPTURE_ENABLED=true, restart. Expect"capture": "enabled","capture_group": "replicator.capture", noCRITICALline, and 0 pending oncontent.capture.acladminread-back ofreplicator.captureand anACL LOGcheck. Then tell Viewport screenshots: per-source capture policy, issued to Replicator on a dedicated command stream archiver#380 it may issue.Verification debt: the first real
stored a capture and published capture_storedwaits on archiver#380. Broker's read-back waits on B3.Follow-ups: #156 (IPv6 forms that embed a private IPv4 address). A pre-existing race on
mainbetween the boot sweep and the store's shardmkdirmakestest_run_dispatches_to_the_byte_pathfail about 1 run in 50; not filed yet.For #63: reuse
Renderer, the SOCKS guard and thebrowser/cgroup. The memory cap is per render, so concurrent renders need serialising or per-render child cgroups.gregoryfoster commentedon Oct 26, 2026
Live: capture enabled on
co-replicatorsince 2026-10-26T16:42:36Z (build9b37afe). Runbook phases A and B done:worker readyreadscapture: enabled/replicator.capture/replicator-capture-1, andworker_memory_low: 134217728.XPENDING content.capture replicator.captureshows 0.browser/cgroup is at 1536 MiB, swap 0,oom.group1, 512 pids, and empty after the boot launch.Hand-offs posted: broker's
acladminread-back on CannObserv/broker#116, and issuing go-ahead on CannObserv/archiver#380. The remaining debt is the first realcapture_stored(waits on archiver#380) and broker's read-back. Finding 37 is filed as #159.#162 docs: COMMANDS.md and ENVIRONMENT.md back under budget by reloca…
#175 test: follow the replayed policy through the pacer, not a pre-#138…