Repository layout and the content-acquisition data flow. AGENTS.md keeps only
the non-obvious boundaries and the rules that constrain them; the enumeration
lives here.
The authoritative list of live reference documents is the Detail Docs index in AGENTS.md; the
docs/entry in the tree below mirrors it.
Central registry + authoring service for the Cannabis Observer information layer. FastAPI + PostgreSQL. Owns five registry tables (info_items, info_sources, source_revisions, rep_specs, info_item_rep_specs) plus the join table info_item_sources; the dashboard adds app_users and api_keys. Consumed by Replicator and external callers via the archiver-client Python SDK - not by Watcher (watcher#254). Produces info.changes, info.registry, content.replicate, content.persist and, since flip F1 (#438), content.fetch-policy (#366) via an internal outbox publisher; consumes content.revisions, info.watch-status and content.artifacts (docs/BUS.md and docs/BUS_CONSUMERS.md carry each stream's provenance). Never content.blobs: that role boundary is unqualified, with no read-only exception.
Phase 4 (the current model — Archiver v2) shipped 2026-05-09 on branch phase-4-archiver-v2. Design + implementation plan:
docs/plans/2026-05-08-archiver-v2-architecture-design.mddocs/plans/2026-05-08-phase-4-archiver-v2-implementation.md
tests/mirrorssrc/;tests/deploy/pins host contracts -deploy.shend to end, installed units against the live release, the needrestart drop-in and the memory reservation, the production read role (#348), and the SocratiCode client config and server pin.tests/ci/pins.github/: the Dependabot config, the action SHA pins, the runner image and the uv cache prune (#192).
src/api/ FastAPI routes, deps, schemas, serializers
src/dashboard/ HTML/HTMX admin dashboard (routes/, templates/, static/, deps.py,
pagination.py — shared clamped limit/offset dependency; see
docs/SCREENS.md for why the dashboard clamps where the API 422s;
watch_panel.py — pure watched-item panel context from local
state: states, health rule, next-due, drift, archiver#151;
temporal_profile_form.py — the profile row, #365;
replication_actions.py — the manual-replication outcome→flash
translation shared by both screens that offer the action, and
the one place that records why a refusal is a 200 rather than
a 4xx, archiver#171;
bus_triage.py — the Bus triage screen's outcome→flash per
core outcome, the discard confirm (names a foreign parking
group) and parked-twice detection, archiver#271;
rep_fields_block.py — pure Replication Fields block view model:
rows per required raw key, source badges, the form parse,
archiver#307;
rep_spec_picker.py — pure Add-a-spec picker view model:
readiness and path preview per unassigned spec, archiver#308;
org_row.py — the Overview's Organization row: the linked
Power Map org's notices (pure) and the type-ahead's local
suggestions; routes/power_map.py serves its search, archiver#306;
providers.py — UNWRITABLE_PROVIDERS, shared by the RepSpec form
and the picker, archiver#202/#308)
src/core/ Domain logic
models/ ORM (info_item, info_source, source_revision,
info_item_source, rep_spec, info_item_rep_spec,
changes_outbox, replication_command,
persist_command, pm_organization)
source_spec_schema/ SourceSpec JSON Schema v1 + validator
rep_spec_schema/ RepSpec envelope + per-provider sub-schemas
(providers/{gcs,gdrive,ia}/v1.json)
rep_fields_schema/ rep_fields meta-schema + validator
replication/ The RepSpec path contract and the destination
renderer (archiver#168). template.py holds the
ONE placeholder parser — used by the create/update
gate (via rep_spec_schema/validator.py) and by the
renderer, so a document that validates is one that
renders; it also owns the occasion namespace
(source_revision.*, reserved out of
required_fields) and the R2 discriminator rule.
destination.py renders, refuses (segment charset,
T3 path guards, naive datetimes), pre-flights a
fan-out set for colliding paths, and probes
renderability at assignment time. It also supplies
the occasion values themselves — the framework's
date forms, and the extension implied by the
origin's media type via co-core's
extension_for_media_type, the cluster's one table
— and resolves the bag's _slug companions through
rep_fields.effective_rep_fields on the way in
(archiver#205, #206, #210, #303).
permanent_store.py derives a persisted
digest's gs://co-gcs-replicator URI (archiver#276).
errors.py is the
single base both raise under, so archiver#169 can
record a skip by catching one class. Archiver
renders because the issuer contract's T3 says so
— Replicator receives strings and never
interpolates. Do not write a second renderer.
watch_spec_schema/ WatchSpec (cadence policy) v1 + validator. Cadence
only — pause state is info_items.watch_active
changes/ The background asyncio tasks and their shared
pacing. group_consumer.py owns the consumer-group
loop — read, claim, quarantine, ack, back off,
re-arm — and is SHARED by every group consumer
rather than copied: nearly every line encodes an
incident or a review finding, and a copy inherits
those once and then drifts. artifacts_consumer.py
applies replication outcomes from content.artifacts
(archiver#170) — public_url's automated writer —
and persist outcomes (archiver#276, via
services/persist_writeback.py);
replication_reaper.py closes commands that produced
no fact at all, on a timer because it detects an
absence, and re-issues silent persist commands
(services/persist_issuance.py, which also issues
content.persist on receipt, archiver#276). publisher.py drains changes_outbox to
info.changes; consumer.py ingests
source_revision_observed from content.revisions
(archiver#139) and is gated on
ARCHIVER_BUS_CONSUMER; backoff.py holds the
retry/log-throttle constants both use — shared,
not copied, because they encode incident history
(#107, #128). diagnostics.py renders an exception
plus its __cause__/__context__ chain for both
paths: co-core's wrappers name only the
event_type, so repr() alone drops the remedy —
and that string is the whole diagnostic a
dead-lettered row or a quarantined message leaves
(#141). dlq_triage.py lists, discards and
reprocesses the three DLQs archiver drains
(archiver#238) - XDEL by id, never XTRIM;
reprocess runs the consumer's own
handle_message on owned entries only.
outbox_triage.py is the exit from a
dead-lettered outbox row (archiver#191):
discard or rearm, by row id only.
outbox_prune.py is the published-row
retention pass (archiver#189) - it rides the
drain loop rather than a systemd timer, because a
timer would need a third holder of
ARCHIVER_ALLOW_PRODUCTION_DB; live and
dead-lettered rows are never pruned.
registry_snapshot.py is the info.registry
full-set republish timer — direct publish,
bypassing the outbox (no transactional obligation
to carry), no retry (the next period is the
repair), reading generations without bumping
(#141). fetch_policy_snapshot.py is the same
for content.fetch-policy (#366): every Domain
and tombstone each 300 s, the cap floored at
ten complete sets. The event
payload models live in co-core since #106, not
here. watch_status_consumer.py tails
info.watch-status groupless (AsyncBusTailReader,
archiver#151) into the watch_status cache —
resumes from bus_tail_cursors, no DLQ, and
deliberately no ARCHIVER_BUS_CONSUMER gate: a
tail removes nothing from a group PEL.
bus_client.py builds the one Redis client all six
loops share, with an explicit connection policy
(archiver#193) - never a bare from_url, which was
safe only while the broker was on loopback. Its
socket_timeout has a FLOOR, not a ceiling:
redis-py does not extend it for a blocking
command, so a value at or below a loop's BLOCK
raises on every idle read. It also carries the
startup reachability probe, because from_url is
lazy and nothing else would notice a broker that
is configured but down. read_windows.py holds the
blocking-read windows those consumers use, split
out as a LEAF so bus_client can derive its timeout
without importing the consumer modules - that
inversion put client construction downstream of
the loops consuming its client, and would become
an import cycle the moment a consumer built its
own client.
services/ Registry write paths shared by the HTTP surface
and the bus consumers. A service owns one
mutation end to end — domain validation, the
write, and its changes_outbox row — raises domain
errors rather than HTTPException, and never
commits: the caller owns the transaction, because
"row and event in one transaction" is the outbox
guarantee. source_revision.py is why the bus and
HTTP paths cannot emit divergent payloads.
replication_status.py is the one read-only member:
a projection of replication_commands for the
dashboard's assignment tables (archiver#171), so
a public_url with an automated writer can say
which occasion wrote it.
domain_policy.py (update_domain) is the one write
that can suspend or resume a host (#360) or change
its default cadence (#364): API
PATCH/archive/restore and the dashboard's
archive/restore/fetch-policy, announcing the
host's items whose resolved state moved.
info_item_lifecycle.py: archive/restore (#361).
cadence_resolution.py (resolve_interval) is the
announced interval: the profile's stored result
(#365), item, domain default (#364).
temporal_evaluation.py, temporal_profiles.py:
profile semantics, writes, boundary (#365).
delete_domain is the delete: it locks the row
before drawing the tombstone's generation (#366).
fetch_policy.py is content.fetch-policy's delta
half (#366): draws the host's generation from
one sequence when its frame changes, writes the
outbox row only while
ARCHIVER_FETCH_POLICY_PRODUCER is set (unit-only,
live since F1), and records a deleted host's
tombstone.
spec_match.py Compares an observed spec_fingerprint against the
InfoSource's own source_specs via co-core's shared
derivation (cannobserv#309). Every uncertain branch
resolves to "incomparable", never "superseded" — a
false mismatch reads exactly like a real one.
fingerprints.py The sha256:<64 hex> content-fingerprint spelling.
Its own module so the API schema and the bus
consumer can share the rule without either
dragging in the ORM.
rep_fields.py The derived half of an InfoItem's rep_fields bag:
<key>_slug companions via co-core's
normalize_string, the one slugger the cluster
shares (archiver#206). A leaf beside
fingerprints.py rather than a tools/ helper —
replication and the validator both consume it,
and from under tools/ that inverted the layering.
An empty derivation is omitted, never written as
"": present-and-empty satisfies a presence check
for a value that can never be a path segment.
effective_rep_fields(bag, org) is the one
resolution point (archiver#303): the stored bag
over the linked org, then derived. A guard test
refuses any other resolve_rep_fields reference.
power_map/ Power Map integration (archiver#304). client.py is
the ONLY importer of the power-map-client SDK:
get_org -> Snapshot | NotModified | Merged | Gone,
search_orgs, PowerMapUnavailableError for every
non-answer. snapshots.py owns pm_organizations
(apply_org_snapshot, load_org_values). Imports
no package that imports it - tools, services,
api, dashboard routes (archiver#327, guard test).
Edge rule: Power Map is called on the authoring
path and by the follower, NEVER during
replication - render reads the snapshot. Beside
the no-HTTP-to-Watcher rule (#142); Power Map is
an identity source, not a sibling in the content
pipeline.
notifications/ Content options (#363); composer, text reader (#367): docs/NOTIFICATION_COMPOSER.md.
Cutover, drain, error reminders (#369): docs/NOTIFICATIONS.md.
notifier/ Notifier gate + adapter, one dispatch (#368, #367): docs/NOTIFICATIONS.md.
tools/ Authoring helpers (assign_rep_spec + lock_rep_specs,
set_rep_fields + the rep_fields_gate they share,
link_org, update_rep_spec, preview_extraction, etc.)
swap_primary_source.py is the one primary swap
(#45): PUT …/primary-source and both dashboard
swaps; a swap-back starts a new tenure row.
bind_info_source.py is the only binding writer
and emits info_item_primary_changed (#390;
tests/core/test_binding_guard.py).
audit_stored_documents.py re-judges every stored
authored document against its current schema
(archiver#164); its KINDS registry is guarded
against every *_schema/ function and JSON column.
refresh_orgs.py is the hourly Power Map org
follower (archiver#305, moved here by #327):
refresh_linked_orgs + the timer's main, beside
the other writers of the effective bag.
import_watcher_domains.py imports Watcher's
domains export through the API (#360, design
D4): operator-run, --dry-run first, idempotent.
import_watcher_item_cadence.py does the same for
the items' own cadence, fill-if-absent (#364).
temporal_boundary.py: the boundary timer (#365).
logging.py Structured logging config (configure_logging at
entry points). Service-local — Watcher keeps its
own copy and there is NO parity requirement; see
"No cross-repo mirror discipline".
build_json_formatter() is the single source of
truth for the JSON field contract. Fetch +
extract + fingerprint come from co-core — see
"Content-acquisition via co-core"
database.py Lazy async engine + async_sessionmaker singletons
(get_database_url / get_engine /
get_session_factory / reset_engine)
log_config.json uvicorn --log-config dictConfig — routes
uvicorn/.access/.error through build_json_formatter
so uvicorn lines match app logs (archiver#122);
each uvicorn logger also runs the
ColorMessageFilter to strip uvicorn's ANSI
color_message extra at the record source
(archiver#123 — filter on the loggers, not a sink);
wired into archiver.service + dev_server.sh
url_canonicalization.py Write-time URL normalization for info_sources
db_safety.py Production-DB startup guard — refuses to serve a
database whose name lacks a _test/_dev suffix
unless ARCHIVER_ALLOW_PRODUCTION_DB=1 (set only
by the deploy/ units and scripts/deploy.sh's
migration). Called from the FastAPI lifespan,
alembic/env.py and schema_state.py; mirrored in
scripts/dev_server.sh and kept in step by
tests/scripts/test_db_guard_parity.py
schema_state.py The database's Alembic revision against the code's
head: current / behind / ahead / unmigrated
(archiver#330 D3). CLI exits 0/3/2 for the deploy;
archiver-bus-health WARNs on a state that cannot serve
reader_grants.py The production read role's grants (archiver#348): SELECT
on information, column allowlists for app_users and
api_keys, default privileges. deploy.sh re-applies them
after every migration; --sql prints them for the runbook
build.py /health's build_id: the serving release's REVISION,
else null (archiver#330 D8)
drift.py The drift check (archiver#338): does live's REVISION
lag origin/main in code that runs, past an 8 h grace?
Hourly from archiver-drift.timer; asks GitHub,
reports ok/alert to co-status's co-archiver-drift
status_checkin.py One check-in to a co-status monitor; the key is a
systemd credential, never in the environment
liveness.py The service's heartbeat (archiver#297): checks in ok
to co-archiver-live every 5 min while every expected
bus loop stamps progress; silent otherwise
backup.py The nightly dump (archiver#233): pg_dump as its own
pg_read_all_data role, verified, created create-only
in gs://co-gcs-archiver-backup; ok/alert to
co-status's co-archiver-backup. BACKUP.md
restore.py Its operator CLI: list, fetch + sha256-verify, restore
into an empty database (refuses production unless
--allow-production)
clients/python/ archiver_client SDK v5.x (generated + hand-written wrappers).
Version lives in clients/python/pyproject.toml and
bumps only when the SDK surface changes
alembic/ Migration root (information schema scoped within the archiver database)
tests/ Mirrors src/ structure; tests/integration/ for cross-component flows
(HTTP + DB + bus); tests/api/ for single-route HTTP behavior;
tests/core/ for domain logic; tests/dashboard/ for the HTMX UI;
tests/js/ for the dashboard JS modules; tests/scripts/ for the
shell/Python helpers under scripts/ (incl.
test_db_guard_parity.py, which keeps dev_server.sh in step
with db_safety.py);
tests/deploy/ asserts the installed systemd artifacts match
deploy/ — archiver.service, the bus-health pair,
the pm-org-refresh pair, the drift pair, the
temporal-boundary pair and the backup pair (each skips when absent, so CI passes) — and the
needrestart drop-in against the live config chain
(archiver#278; asserts only on the archiver host),
the memory reservation's effective protection,
the swapfile and swappiness, every capped command
in docs/SOCRATICODE.md bounding swap (archiver#286)
and the SocratiCode server pin (archiver#237; live
halves skip off-host). The redis-server
drop-in and its parity test left with the broker
(archiver#193 D6 → CannObserv/broker); the LIVE broker
config is still checked here by check_redis_floor.sh
scripts/ deploy.sh (build a release from a pushed main commit,
rehearse + migrate, switch /srv/archiver/live, verify,
switch back on failure; archiver#330, DEPLOYMENT.md) +
sync_wheelhouse.py (mirror co-core wheels from the private
GCS index into ./.wheelhouse; run before uv sync, and by
deploy.sh into each release) +
dump_openapi.py +
setup-backup-role.sql (the backup's database role,
once per cluster; archiver#233, BACKUP.md) +
check_client_drift.py (regen vendored clients from
committed OpenAPI snapshots; diff vs generated/;
CI gate, see client-drift job) +
check_changelog_on_push.sh (pre-push guard;
wired via .pre-commit-config.yaml) +
check_changelog_lib.sh (the shared trigger-path
predicate; sourced by the guard) +
test_check_changelog.sh (its unit tests; runs as a
pre-commit hook) +
check_version_lockstep.py (CI lint job: pyproject
[project].version must equal the newest CHANGELOG
`## vX.Y.Z` heading — #85 silent drift) +
check_redis_floor.sh (archiver.service ExecStartPre:
≥7.0 broker floor + warn-only live-maxmemory check,
each probe bounded by ARCHIVER_REDIS_FLOOR_TIMEOUT) +
dev_server.sh (ONLY sanctioned way to start the
8001 dev server; refuses to resolve onto the
production DB — see "Server Lifecycle") +
audit_lock.sh (report-only uv audit of both locks,
runtime and dev apart; archiver#420, DEPLOYMENT.md) +
pre-ship.sh (ship gate; SKILLS.md) +
lib/env.sh (their env allowlist; DEPLOYMENT.md)
deploy/ README.md (install instructions) + systemd units, all
running /srv/archiver/live and installed by deploy.sh:
archiver.service +
needrestart.conf.d/archiver.conf (list restarts, never
perform them; archiver#278) +
the memory reservation (archiver#237):
99-archiver-memory.conf (sysctl; vm.swappiness for
the 4 G /swapfile, archiver#286),
system.slice.d/, system-postgresql.slice.d/,
postgresql@16-main.service.d/ +
the OOM order of the tail (archiver#285):
tailscaled.service.d/, postgresql/16/main/environment +
archiver-bus-health.service/.timer (the OUTBOX probe;
the broker-side half is CannObserv/broker's since
archiver#193 D6) +
archiver-pm-org-refresh.service/.timer (the
Power Map org follower, archiver#305) +
archiver-drift.service/.timer (archiver#338) +
archiver-temporal-boundary.service/.timer (#365) +
archiver-backup.service/.timer (the nightly dump,
as user archiver_backup; archiver#233). The #128 OOM lockstep now spans two
repos: the cap is in CannObserv/broker's drop-in,
the transient classification in publisher.py
docs/ Live reference docs, the dashboard living docs
among them (see "Dashboard living docs").
reference/ holds host-shaped facts that are not
about the code: tailscale.md (this node, the
ACL, why there is no tailnet-only bind).
Indexed by the Detail Docs section of AGENTS.md.
Archival subtrees: plans/ + research/ + archive/
(retired operational history)
skills/ Agent skills (committed overrides + symlinks → skills-vendor/)
skills-vendor/ Git submodules for external skill repos
.skills/doctor.sh Committed skill-symlink doctor (real file, not a
symlink) — see SKILLS.md "SessionStart Hooks"
.claude/skills/ Claude Code skill discovery (symlinks → ../../skills/<name>)
.github/workflows/ CI (ci.yml) — lint job (ruff check + ruff format --check
+ check_version_lockstep.py),
test job (Postgres service container, alembic upgrade,
pytest), client-drift job (regen vendored clients
from committed OpenAPI snapshots, fail on diff;
then the SDK's own pytest suite under its lockfile),
and changelog job (changes under
alembic/versions/, src/api/routes/,
src/api/schemas/, or clients/python/ must touch
CHANGELOG.md — path-triggered, not commit-type;
opt out via `no-changelog` PR label).
Triggers on push/PR to main.
audit.yml — scripts/audit_lock.sh on push, PR, weekly
and dispatch; report-only, outside deploy.sh's gate
(#420; DEPLOYMENT.md "Lock audit")
.github/dependabot.yml Weekly github-actions bumps of the workflows' SHA pins
(#192); see DEPLOYMENT.md "Dependabot"
.pre-commit-config.yaml ruff check + ruff format + standard pre-commit-hooks
+ test_check_changelog.sh (pre-commit stage), plus a pre-push guard
(scripts/check_changelog_on_push.sh) that mirrors
the CI changelog check before the push leaves the
clone. Run `uv run pre-commit install` once per
clone — installs both pre-commit and pre-push
hook types via default_install_hook_types.
Quoted section names in the tree above ("Server Lifecycle", "Dashboard living docs", "Content-acquisition via co-core", "No cross-repo mirror discipline") refer to sections of AGENTS.md, unless a doc is named.
Fetch, extract, and the content fingerprint are consumed from co-core,
the canonical implementation upstreamed in #255. The former
src/core/{fetchers,extractors,simhash,extraction_defaults} mirror was deleted when
Phase 1b landed — no more mirror discipline for the acquisition pipeline.
- Fetch —
co_core_aio.fetch.AsyncFetchDriverexecutes theco_core.effects.fetch.FetchContenteffect (raw bytes, captures non-2xx asFetchResult). Wired intoapp.state.fetch_driverat the FastAPI composition root (main.lifespan), injected viadeps.get_fetch_driver. - Extract + fingerprint —
co_core.pure.extract.*(html/csv_excel/pdf/xml_feedextractors, synchronous;sha256+simhashperChunk). Requires theco-core[extract]extra; import parsers from their submodules (co_core.pure.extract.html, …) — they are not re-exported from__init__.preview_extractiondispatches as Processor does (#317):resolve_dispatch_essence→extractor_for_essence(co_core.pure.extract.dispatch) →canonical_text_fingerprint, in a worker thread.
The broker is CannObserv/broker's, not this repo's (#193 D6); the one seam that split left untested: docs/BUS_CLIENT.md.
The earlier InfoItem ↔ InfoSpec Phase 1–3a model is gone — info_specs table dropped, info_spec_schema package deleted, all SDK methods retired. The 2026-05-06 trajectory research doc remains as historical context.