Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,11 @@ repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v6.0.0
hooks:
# tests/fixtures/diff/ holds captured page text, byte-exact (archiver#367).
- id: trailing-whitespace
exclude: ^(skills-vendor/|clients/python/src/archiver_client/generated/)
exclude: ^(skills-vendor/|clients/python/src/archiver_client/generated/|tests/fixtures/diff/)
- id: end-of-file-fixer
exclude: ^(skills-vendor/|clients/python/src/archiver_client/generated/)
exclude: ^(skills-vendor/|clients/python/src/archiver_client/generated/|tests/fixtures/diff/)
- id: check-yaml
- id: check-toml
- id: check-merge-conflict
Expand Down
6 changes: 6 additions & 0 deletions .skills/context-claims-ok
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,9 @@
# still names the module as `src/core/db_safety.py`.
# (pruned 2026-09-09 round 2: the atom is present again at this base,
# so the round-1 warrant is spent - prove-no-loss.sh reported it matched nothing)

# archiver#367 D7 (2026-10-30): ARCHIVER_PUBLIC_BASE_URL is now an origin; the item page is
# /dashboard/info-items/{id} (src/core/public_base_url.py item_page_url), set in deploy/archiver.service.
docs/ENVIRONMENT.md :: disproven :: https://archiver.example.com
docs/ENVIRONMENT.md :: disproven :: https://archiver.example.com`
docs/ENVIRONMENT.md :: disproven :: {ARCHIVER_PUBLIC_BASE_URL}/info-items/{id}
11 changes: 11 additions & 0 deletions .skills/context-loss-ok
Original file line number Diff line number Diff line change
Expand Up @@ -100,3 +100,14 @@ AGENTS.md :: tighten :: the temporal-profile boundary, #365) and
AGENTS.md :: tighten :: `root:root 0600`: read by systemd for the units and by `deploy.sh` via sudo, never by an agent shell (#339).
AGENTS.md :: tighten :: never 8000. Why the script exists, its knobs, and the 2026-07-18
AGENTS.md :: tighten :: production-write incident: [docs/ENVIRONMENT.md](docs/ENVIRONMENT.md). The one

# archiver#367 D7 (2026-10-30): scoped ENVIRONMENT.md pass to fit the #368/#367 entries.
# Broker rationale moved verbatim to docs/BUS_CLIENT.md § The broker boundary; these are
# the lines rewritten in place.
docs/ENVIRONMENT.md :: disproven :: `AGENTS.md` keeps the four variables that carry safety rules; the reference
docs/ENVIRONMENT.md :: tighten :: > D5), so the fallback is no longer the normal case on this host. The knob had
docs/ENVIRONMENT.md :: tighten :: > always been honoured; there was simply never a database for it to name, so
docs/ENVIRONMENT.md :: tighten :: > every dev server and every test run shared one and raced. Provisioning a
docs/ENVIRONMENT.md :: tighten :: > dedicated VM was the moment to close that rather than port it.
docs/ENVIRONMENT.md :: tighten :: Note the fourth mover: the repo-root `.env` loads **after** `/etc/archiver/.env` and overrides it
docs/ENVIRONMENT.md :: disproven :: - `ARCHIVER_PUBLIC_BASE_URL` — *optional*. Public-facing base URL of this Archiver instance
15 changes: 14 additions & 1 deletion .skills/context-seams-ok
Original file line number Diff line number Diff line change
Expand Up @@ -213,7 +213,7 @@ tests/test_conftest_guard.py :: docs/DEPLOYMENT.md and the failure message both
docs/CONVENTIONS.md :: holds no transaction open (CLAUDE.md § Reading production).
docs/CONVENTIONS.md :: the rule sits in the file a tightening edits rather than in AGENTS.md.
docs/DEPLOYMENT.md :: (CLAUDE.md § Environment & Tooling, the `using-git-worktrees` override). Until #341 it was in
docs/ENVIRONMENT.md :: `AGENTS.md` keeps the four variables that carry safety rules; the reference
docs/ENVIRONMENT.md :: `AGENTS.md` keeps the five variables that carry safety rules; the reference
scripts/sync_wheelhouse.py :: A checkout runs it by hand (AGENTS.md, Environment & Tooling).
src/dashboard/org_row.py :: ``pm_organizations``, never Power Map (CLAUDE.md's edge rule).
tests/alembic/test_env_db_safety.py :: ``alembic upgrade`` or ``downgrade`` against production, and CLAUDE.md told
Expand Down Expand Up @@ -407,3 +407,16 @@ docs/SCHEMA_BUS.md :: `info.changes` row to the drain. See [BUS.md](BUS.md).
src/api/main.py :: # trigger fires (docs/BUS.md, content.replicate).
src/dashboard/bus_triage.py :: ``src/core/changes/outbox_triage.py`` (``docs/BUS.md``) and
src/dashboard/templates/bus_triage/index.html :: asked. The rules are the runbooks' - <code>docs/BUS.md</code> for the

# archiver#367 D7 (2026-10-30), --file docs/ENVIRONMENT.md: broker rationale moved to
# BUS_CLIENT.md § The broker boundary. Each pointer below names a variable entry, the
# outbound-env-var steps or the dev-server material ENVIRONMENT.md still holds.
docs/BUS.md :: - **Off unless `ARCHIVER_PERSIST_ISSUANCE` is set** ([ENVIRONMENT.md](
docs/BUS_CLIENT.md :: `ARCHIVER_REDIS_STREAM_MAXLEN` ([ENVIRONMENT.md](ENVIRONMENT.md)) move
docs/BUS_CLIENT.md :: `ARCHIVER_REGISTRY_STREAM_MAXLEN` ([ENVIRONMENT.md](ENVIRONMENT.md)) m
docs/BUS_CLIENT.md :: `ARCHIVER_DEV_REDIS_URL` ([ENVIRONMENT.md](ENVIRONMENT.md)) is dormant
docs/DEPLOYMENT.md :: variable reference and the dev server's internals are [ENVIRONMENT.md]
docs/DEPLOYMENT.md :: `scripts/dev_server.sh` exists: [ENVIRONMENT.md](ENVIRONMENT.md).
scripts/dev_server.sh :: # throwaway, see docs/ENVIRONMENT.md) or runs bus-dormant. And a dev o
src/core/changes/registry_snapshot.py :: env var can move it with no commit in either repo. ``docs/ENVIRONMENT.
tests/test_conftest_guard.py :: docs/ENVIRONMENT.md and the failure message both advertise.
19 changes: 19 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,25 @@ with any notable release. SDK version in `clients/python/pyproject.toml` bumps
only when the SDK surface changes (new methods, changed types, removals); a
service-only patch does not require an SDK bump.

## v4.40.0 (2026-10-30)

[service] **Notification composer, preview and test-send (archiver#367, epic #359).** SDK unchanged (5.16.0): no route or schema is added or reshaped. The only API change is the **value** of `dashboard_url`.

- **`dashboard_url` now points at the dashboard page:** `{ARCHIVER_PUBLIC_BASE_URL}/dashboard/info-items/{id}`. Before, it was `…/info-items/{id}`, a 404 under the configured origin.
- `ARCHIVER_PUBLIC_BASE_URL` is now read as an **origin**, `http(s)://host[:port]`, with any trailing `/` dropped.
- A value with a path, query, fragment or userinfo yields `null`, the same as unset.
- `deploy/archiver.service` sets `https://co-registrar.exe.xyz`.
- **The composer** (`src/core/notifications/`) is ported from Watcher `3ba20be`, with the default text word for word.
- Every render is sandboxed, with bounded `*`/`**`. The context is built only from documented variables.
- A failing custom template falls back to the event's default, never to raw template source.
- **The change diff** reads both texts from `gs://co-gcs-processor` only, through the `processor-text-reader` unit credential. Each read is capped and hash-checked. A missing key leaves diffs unavailable and refuses nothing.
- **Dashboard:**
- the subscription form gains per-block Jinja title/body fields with a live strict preview;
- the edit page gains a test-send that makes one Notifier dispatch, with the composed text passed as variables;
- neither has an API route.

Nothing dispatches on events yet: that's #369. Details: [docs/NOTIFICATION_COMPOSER.md](docs/NOTIFICATION_COMPOSER.md).

## v4.39.0 (2026-10-28)

[both] **Notification subscriptions: who hears which events, and how (archiver#363, design D3/D5, item A7).** Migration `c4d81f6a2e07` adds `information.notification_subscriptions`. SDK 5.16.0. Nothing sends or publishes yet: composition is #367, dispatch #369.
Expand Down
2 changes: 1 addition & 1 deletion clients/python/archiver-openapi.json
Original file line number Diff line number Diff line change
Expand Up @@ -3232,7 +3232,7 @@
},
"info": {
"title": "archiver",
"version": "4.39.0"
"version": "4.40.0"
},
"openapi": "3.1.0",
"paths": {
Expand Down
2 changes: 1 addition & 1 deletion deploy/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ user](../docs/DEPLOYMENT.md#the-service-user-archiver339)). `/etc/archiver/.env`

| Unit / file | Type | Purpose |
|---|---|---|
| `archiver.service` | service | The live API on port 8000 (see CLAUDE.md -> Server Lifecycle). Its `ExecStartPre` asserts the Redis >=7.0 floor when the bus is active. It checks in to co-status's `co-archiver-live` while its bus loops progress, and bounds restarts at 5 in 300 s (`docs/MONITORING.md` § Liveness). The one Notifier client (#368): its URL and opt-in are the unit's own `Environment=` lines, the key a `LoadCredential=` with no fallback, so `scripts/deploy.sh` refuses a host without `/etc/archiver/notifier.key` (`docs/NOTIFICATIONS.md` § The credential). |
| `archiver.service` | service | The live API on port 8000 (see CLAUDE.md -> Server Lifecycle). Its `ExecStartPre` asserts the Redis >=7.0 floor when the bus is active. It checks in to co-status's `co-archiver-live` while its bus loops progress, and bounds restarts at 5 in 300 s (`docs/MONITORING.md` § Liveness). The one Notifier client (#368): its URL and opt-in are the unit's own `Environment=` lines, the key a `LoadCredential=` with no fallback, so `scripts/deploy.sh` refuses a host without `/etc/archiver/notifier.key` (`docs/NOTIFICATIONS.md` § The credential). The processor-text reader (#367): the bucket is its own `Environment=` line, the key `/etc/archiver/processor-text-reader.json` a `LoadCredential=` **with** an empty fallback, so a missing key only leaves diffs unavailable (`docs/NOTIFICATION_COMPOSER.md`). `ARCHIVER_PUBLIC_BASE_URL` is its line too. |
| `archiver-bus-health.service` | service (oneshot) | One WARN-only tick of the **outbox** probe: depth, oldest-unpublished age, dead-lettered count (#130, reduced by #193). Never blocks anything; see *Outbox health timer* below. |
| `archiver-bus-health.timer` | timer | Runs the probe every 10 min. Enable with `systemctl enable --now archiver-bus-health.timer`. |
| `archiver-pm-org-refresh.service` | service (oneshot) | One sweep of the Power Map org follower: a conditional GET per linked org, applying renames, merges and misses to `pm_organizations` (#305). Exits 0 with no database access when `ARCHIVER_POWER_MAP_API_KEY` is unset. See *Power Map org follower* below. |
Expand Down
17 changes: 17 additions & 0 deletions deploy/archiver.service
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,23 @@ Environment=ARCHIVER_NOTIFIER_BASE_URL=http://notifier:9000
Environment=ARCHIVER_NOTIFIER_ENABLED=1
LoadCredential=notifier-key:/etc/archiver/notifier.key

# The processor-text reader (archiver#367 D4): change diffs read the two stored
# texts from the processor bucket. Its own service account, objectViewer on
# gs://co-gcs-processor alone, so this identity cannot read raw blobs
# (content.blobs). The key is a credential, in no environment and never logged,
# WITH the empty fallback, like Status's: a missing key starts the service with
# every diff unavailable and one WARNING, and deploy.sh's preflight passes it.
# Never GOOGLE_APPLICATION_CREDENTIALS here: that would make the identity ambient
# for every GCS call this process makes. The client is built from this file.
Environment=ARCHIVER_PROCESSOR_TEXT_BUCKET=co-gcs-processor
LoadCredential=processor-text-reader:/etc/archiver/processor-text-reader.json
SetCredential=processor-text-reader:\n

# Where notification and API links point (archiver#367): the origin people open,
# never an internal address. A line in /etc/archiver/.env would override this one
# (EnvironmentFile= wins), so that file must not carry it.
Environment=ARCHIVER_PUBLIC_BASE_URL=https://co-registrar.exe.xyz

# No wheelhouse sync here any more: the private wheels are fetched into the
# release at deploy time (archiver#330 D10), so a start never depends on GCS.

Expand Down
4 changes: 2 additions & 2 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -235,8 +235,8 @@ src/core/ Domain logic
the no-HTTP-to-Watcher rule (#142); Power Map is
an identity source, not a sibling in the content
pipeline.
notifications/ Content options (archiver#363): docs/NOTIFICATIONS.md.
notifier/ Notifier gate + read-only adapter (#368): docs/NOTIFICATIONS.md.
notifications/ Content options (#363); composer, text reader (#367): docs/NOTIFICATION_COMPOSER.md.
notifier/ Notifier gate + adapter, one dispatch (#368, #367): docs/NOTIFICATIONS.md.
tools/ Authoring helpers (assign_rep_spec + lock_rep_specs,
set_rep_fields + the rep_fields_gate they share,
link_org, update_rep_spec, preview_extraction, etc.)
Expand Down
16 changes: 16 additions & 0 deletions docs/BUS_CLIENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,10 +32,26 @@ loses - the outbox is the durable buffer. The URL, the measured path, and the
Consumer loops need no transient/poison classification of their own; what they do
on a dropped connection is pinned by `tests/core/changes/test_bus_reconnect.py`.

## The broker boundary

**The broker is not operated from this repo (#193 D6).** Its tuning, health
probe, and stream inventory live in
[CannObserv/broker](https://github.com/CannObserv/broker); archiver is a client.
One seam survives that split with no test spanning it: the broker config's
`maxmemory` cap and `OutOfMemoryError` being transient in
`_TRANSIENT_PUBLISH_ERRORS` are **one decision**, and each repo names the other
in a comment (R5).

**Lockstep invariant (archiver#128) — now spans two REPOSITORIES (archiver#193 R5).** The broker config's `maxmemory` cap and `OutOfMemoryError` being listed in `_TRANSIENT_PUBLISH_ERRORS` (`src/core/changes/publisher.py`) are **one decision; never change either alone.** The cap now lives in `CannObserv/broker:deploy/redis.conf.broker`, so no test spans the pair — each side names the other in a comment and that is the whole mechanism. It rode a systemd drop-in overriding `ExecStart` while this repo tuned the broker; broker#1 Phase 5 appended it to `redis.conf` instead, because `requirepass` cannot ride an `ExecStart` argument without landing in `argv` and journald, and splitting the rest of the tuning across two mechanisms then buys nothing (archiver#196). `redis.conf.broker` is **not** a drop-in. `maxmemory-policy noeviction` with the default `maxmemory 0` is inert — nothing is ever refused, so an untrimmed stream is OOM-killed rather than erroring. The cap restores bounded, instance-wide `OOM command not allowed` errors; the transient classification is what stops those from dead-lettering valid `info.changes` events (`OutOfMemoryError` is a `ResponseError` subclass, so the default "possibly-permanent" branch would otherwise catch it). Removing the cap makes the classification pointless; removing the classification makes the cap lossy. The blast radius is instance-wide — **every** producer on the shared broker is refused, and every durable path retries through it: Archiver's outbox via this entry, Watcher's and Replicator's as verified in CannObserv/watcher#288 and CannObserv/replicator#79. At the cap the bus stalls rather than losing those writes. The periodic full-set publishes do not retry - `info.registry` snapshots, `content.fetch-policy` (Watcher's until F1, then Archiver's, archiver#366) and Watcher's `info.watch-status` - because the next period supersedes a lost one.

`ARCHIVER_REDIS_STREAM_MAXLEN` ([ENVIRONMENT.md](ENVIRONMENT.md)) moves a broker threshold:

**Setting this obliges a note to [CannObserv/broker](https://github.com/CannObserv/broker).** Their bus-health probe mirrors the *default* (`DEFAULT_STREAM_MAXLEN`, `src/core/changes/publisher.py`) as `CHANGES_PRODUCER_MAXLEN` (`FACT_PRODUCER_MAXLEN` until CannObserv/broker#60), which reaches `info.changes` only, and derives a stream-too-long warning threshold from it. Naming the source is not enough when an env var can move it underneath — an override here leaves their threshold stale with no commit in either repo (the CannObserv/broker#44 failure mode, agreed with broker-agent 2026-09-23). Unset in production as of 2026-10-22 (#326), so the mirror is correct today.

`ARCHIVER_REGISTRY_STREAM_MAXLEN` ([ENVIRONMENT.md](ENVIRONMENT.md)) moves two broker checks:

**Setting this obliges a note to [CannObserv/broker](https://github.com/CannObserv/broker)**, for the same reason as the fact-stream cap above: their `REGISTRY_PRODUCER_MAXLEN` mirrors this knob's *default* and feeds `REGISTRY_WARN_LENGTH`, the stream-too-long **ceiling**. It also bounds their shrink check (`entries-removed`, CannObserv/broker#42). That check has no threshold on what it counts, but it reports only **under** the mirrored cap (why: `DEFAULT_REGISTRY_STREAM_MAXLEN`'s docstring), so an override here moves **both** checks. **Lowered**, the shrink check fires at each trim at the real cap. **Raised past the 10% margin**, the ceiling fires early. **Within the margin**, neither fires. Unset in production as of 2026-10-22 (#326).

`ARCHIVER_DEV_REDIS_URL` ([ENVIRONMENT.md](ENVIRONMENT.md)) is dormant or a local throwaway broker, never an index on the shared one:

**Do not re-add a `.../1` recommendation: a logical DB index is not a boundary.** Redis ACLs cannot partition by index — any client that can reach db1 can `SELECT 0` — so isolation by index is isolation by good behaviour. The shared broker closes that axis with `databases 1` and no `+select` in archiver's ACL (after its 2026-09-10 incident, `docs/INCIDENT-2026-09-10.md` in [CannObserv/broker](https://github.com/CannObserv/broker)), so `.../1` there is `ERR DB index is out of range`. Nor is a scratch bus on the shared instance wanted: it shares `maxmemory` under `noeviction` instance-wide, so a dev run that fills it stalls every production publisher (the R5 lockstep, firing for a reason nobody would look for). A prefix-confined dev credential (`~archiver.dev.*`) is available from broker on request — not requested until a real need appears.
Loading
Loading