Echo info_source_id on blob_available and fetch_failed; revise MUST-2 #28

Activity

1. gregoryfoster commented on Aug 7, 2026

   

   gregoryfoster

   on Aug 7, 2026

   ContributorAuthor

   More actions

   Unblocked: cannobserv v0.8.0 is published. Pin co-core>=0.8,<0.9 at tag v0.8.0.

   Three things landed differently from what CannObserv/cannobserv#300 proposed, and all three change this issue's scope.

   1. The field is REQUIRED, not optional

   The issue specified str | None = None with a tightening follow-up; that was reversed. info_source_id is required on ContentFetchCommand, BlobAvailableEvent, and FetchFailedEvent.

   So the echo is not best-effort — a BlobAvailableEvent or FetchFailedEvent constructed without it raises at construction, in Replicator's own emit path. There is no None case to handle on either fact.

   2. BlobAvailableEvent.command_id is required too

   This one reverses cannobserv#266 and is worth checking against your emit path. It was optional so a "non-command-triggered emit (e.g. a seed harness)" would stay legal. The review could not find that producer — scripts/seed_fetch.py issues commands and tails outcomes rather than emitting blobs.

   If Replicator has any path that emits blob_available without a command_id, this breaks it and the finding was wrong — say so and we will restore the optionality in a patch. Otherwise nothing to do.

   3. blob_available is re-keyed to content_fingerprint:command_id

   Previously the bare content_fingerprint. This is a delivery-behaviour change on your stream: emissions that previously collapsed now all deliver. Consumers are already required to be idempotent, so it moves in the safe direction, but the volume characteristics change.

   The reason it had to change: a bare fingerprint collapsed every fetch producing identical bytes into one fact — fine while the payload described only the bytes, not fine once it carried a domain key. Two InfoSources on one URL (exactly the case MUST-3 exists for) would collapse to a single fact naming whichever issuer won the race.

   It also closes a pre-existing hole worth knowing about: the second command for identical bytes was deduped away and therefore never closed, leaving its issuer waiting on a fact that would never arrive. That looked like a slow origin, not a bug.

   MUST-2 revision — scope it carefully

   The issue asked for a doc revision here. With the field required, MUST-2 can relax from a correctness requirement to bookkeeping — but only once CannObserv/watcher#252 also ships. Until then an issuer cannot rely on the wire carrying the mapping, so the contract doc should describe the guarantee as new, not retroactive. Wording that survives both states is better than wording that has to be revised twice.

   Also new, no action required

   • BlobAvailableEvent.blob_expires_at (optional) — when the blob stops being retrievable. Populating it lets Watcher/Archiver record a cache horizon honestly instead of hard-coding MUST-7's TTL, whose clock runs from last fetch reference and which no consumer can observe. Worth populating if you have the value cheaply.
   • The old "validator freshness caveat" is dissolved by the re-key — every fetch now emits its own fact with its own etag / last_modified / final_url.
2. gregoryfoster commented on Aug 7, 2026

   

   gregoryfoster

   on Aug 7, 2026

   ContributorAuthor

   More actions

   Deploy constraints this release introduces

   1. Flush the streams. schema_version stays 1. cannobserv#266's justification (zero producers/consumers) no longer held — Replicator emits these today — so the decision rests instead on the wire being pre-production and discardable. Bumping to 2 would imply a v1 consumers must branch on, when the correct operation is to drop the v1 messages. Flushing content.fetch and content.blobs is a prerequisite step of this deploy, not optional cleanup.

   2. Deploy the step-2 pair together. Required fields mean a half-deployed cluster does not degrade — it dead-letters. A Replicator on 0.8.0 will fail from_wire on any command from a Watcher still on 0.7.x, and a dead-lettered command destroys the command_id correlator. #28 and CannObserv/watcher#252 may be worked in parallel but must ship together, streams flushed between. Same for step 3's pair (CannObserv/archiver#139 before CannObserv/watcher#253).
3. gregoryfoster commented on Aug 7, 2026

   

   gregoryfoster

   on Aug 7, 2026

   ContributorAuthor

   More actions

   Replicator's read, and the two answers the comments ask for

   1. No blob_available emit without a command_id — the finding was right

   Nothing to restore; keep it required.

   • src/worker/handler.py is the only BlobAvailableEvent construction site in src/, and it sits inside handle(command) with command_id=command.command_id.
   • src/worker/loop.py dead-letters a blank command_id before the handler runs, so the field is non-empty by the time the handler sees it.
   • scripts/seed_fetch.py issues commands and decodes facts read-only. It never emits one. The hypothetical seed-harness producer cannobserv#266 protected does not exist here either.

   2. blob_expires_at — yes, populating it

   The value is cheap and, more importantly, honest. LocalBlobStore.store os.utimes on the content-addressed short-circuit, so the TTL clock genuinely means "since last referenced" and fetched_at + REPLICATOR_BLOB_TTL_SECONDS is correct at emit time. A re-fetch only ever pushes it later; the sweep runs on an interval so the real reap is also later. The announced horizon errs conservative in both directions and never optimistic, which is the property a consumer's cache horizon needs.

   ---

   One blocker the issue does not mention: the boundaries charter refuses the field

   tests/test_boundaries.py carries DOMAIN_TOKENS = frozenset({"info_source", "info_item", "watched_item", "tenant", "aspect"}), and its vocabulary scan reads identifiers and string literals across all of src/. The first info_source_id=command.info_source_id trips test_no_module_names_a_domain_concept — the charter's own load-bearing assertion, written precisely against "one optional field, one small settings table, and Replicator has a domain model."

   That test is not wrong; it is doing its job. The charter is what changed. Amending both in the same commit:

   • Keep info_source in DOMAIN_TOKENS. Add a narrow (module, token) allowlist — the two emit sites and the report dataclass, nothing else.
   • Add the compensating assertion the allowlist now requires: info_source_id appears nowhere in src/core/config.py (never a setting), nowhere in src/storage/ (never a path component), and never in a comparison, if, or dict-lookup subscript anywhere in src/. "Verbatim echo" is mechanically checkable; "pass-through" as prose is not, and prose is what erodes.
   • docs/contracts/replicator-boundaries.md gains the reasoning: one echo, required by the wire contract, interpreted nowhere.

   Also not adding a blank-info_source_id guard to mirror the command_id one. command_id is refused because a fact is unkeyable without it; a blank domain id is the issuer's problem and refusing it would be interpretation. If co-core sets min_length, the refusal lives there, which is the right place.

   Doc surface is wider than the three spots in the body

   • content-fetch-issuer-contract.md opens with "There is no info_source_id, and no other domain identity" — that reverses outright, ahead of the MUST-2 edit.
   • The envelope-key table row for blob_available → content_fingerprint:command_id.
   • The success-fact table's command_id | str \| None / "None only for non-command emits" → required.
   • MUST-5 ("do not dedupe facts on content_fingerprint") is strengthened by the re-key — the hole it warned consumers about is now closed at the producer too. Saying so is worth a sentence.
   • Two Replicator-internal docs assert the fact dedupes on the bare content_fingerprint (AGENTS.md Bus Conventions, docs/ARCHITECTURE.md). Both now wrong; ARCHITECTURE's carries the reasoning and gains from the correction.

   MUST-2 wording — taking the "survives both states" advice literally and phrasing by deploy epoch rather than by "now": the durable record stays a MUST for request options, audit, and the MUST-6 reaper; facts emitted from the step-2 deploy onward carry info_source_id, so the crash-between-mint-and-record window stops permanently orphaning the mapping for those facts. True before and after CannObserv/watcher#252, and needs no second revision.

   Failure-mode note for the half-deployed window

   With the field required, a pre-0.8 command does not reach the fetch_failed path at all — it fails from_wire and lands in content.fetch.dlq via the anomaly route, silently. That is already the reference doc's "frame that fails from_wire entirely" row, so no new taxonomy, but it is the concrete shape of the dead-lettering this issue's second comment warns about: the correlator is destroyed at decode, one step earlier than a rejected command.

   Flush list

   content.fetch, content.blobs, and both .dlq streams — v1 dead-letters are unreplayable under 0.8.0, so leaving them is leaving a trap for whoever triages next. Plus replicator:cmd:* if any command_id will be reused across the flush.

   Not content.fetch-policy. It is read groupless as a state stream; flushing it leaves the worker with no policy until the next publish. Will confirm FetchPolicyState is unchanged in 0.8.0 during the pin bump.

   Proceeding TDD-first, starting with the charter amendment.
4. gregoryfoster commented on Aug 8, 2026

   

   gregoryfoster

   on Aug 8, 2026

   ContributorAuthor

   More actions

   Implemented — PR #33, not yet merged

   Branch 28-echo-info-source-id, five commits (f455deb..1a0d0a8). Held short of main deliberately: on this VM main is the deployed code, and this cannot deploy without CannObserv/watcher#252.

   Done when — all three met.

   • Both facts carry info_source_id, echoed verbatim. Required on the wire, so there is no None case on either.
   • Round-trip tested in both directions: the value crosses from command to blob_available (test_the_domain_key_is_echoed_verbatim_onto_the_success_fact) and to fetch_failed (test_the_domain_key_is_echoed_verbatim_onto_the_failure_fact), asserted against a value the emit site could not have derived from anything else on the command. A command without the field no longer exists — it fails from_wire and dead-letters, which is the half-deploy failure mode rather than a case to test.
   • Issuer contract and reference both revised: MUST-2 narrowed by deploy epoch, MUST-3 kept with its last excuse removed, MUST-5 noted as closed on the producer side too, the envelope-key table re-keyed, and blob_expires_at documented in MUST-7.

   Also landed, beyond the issue's scope.

   • blob_expires_at is populated (cannobserv#301). Cheap and honest here: store() touches mtime on its content-addressed short-circuit, so the TTL clock genuinely means "since last referenced", and the stamp is read before the store so the published horizon can only fall earlier than the real reap.
   • blob_ttl_seconds is bounded. Making the horizon arithmetic turned an unbounded float into an OverflowError raised after the bytes were stored — dead-lettering a command whose blob was fine and leaving it orphaned. A config typo should not be able to manufacture those.
   • The seed harness takes --info-source-id and the live target refuses both the placeholder and a blank.

   The boundaries charter needed amending, and that was the substantive work.

   tests/test_boundaries.py forbids domain vocabulary anywhere in src/, so the first info_source_id=command.info_source_id failed the charter's own load-bearing assertion. Correctly — the rule it protects is that Replicator never learns what a domain key means. Rather than delete the token, the exemption is three modules wide and one identifier wide, and a second scan took over the real invariant.

   That scan is the part worth knowing about. It began as a deny-list of reading positions, and three review rounds each found positions it had not enumerated — six in the first, one in the second, two in the third — while the charter cited it as what made the carve-out safe. It is now inverted: an echo has four legal shapes and everything else fails, including shapes nobody has thought of. Probed against fifteen violations it was never told about — assert, comprehension filters, set literals, del, sort keys, async for, setdefault among them — all fifteen caught, the three legal shapes clean.

   Answers to the two questions the release comments asked.

   1. No path emits blob_available without a command_id. handler.py is the only construction site and always sits inside handle(command); loop.py dead-letters a blank id before the handler runs; seed_fetch.py issues commands and decodes facts read-only. The finding was right — keep it required, nothing to restore.
   2. blob_expires_at populated, per above.

   Verification. 498 unit + 18 live-broker integration tests, coverage 99.77%, ruff clean, ty at its pre-existing baseline. AGENTS.md and all three contract docs back inside their context budgets.

   Not closing this issue yet — it closes when the step-2 pair is deployed together, streams flushed between. Order of operations is in docs/DEPLOYMENT.md.
5. gregoryfoster

   closed this as completedin #33on Aug 8, 2026
6. gregoryfoster commented on Aug 8, 2026

   

   gregoryfoster

   on Aug 8, 2026

   ContributorAuthor

   More actions

   Deployed — step-2 pair shipped together, 2026-08-08 22:38 UTC

   Merged as #33 (f4ea53b), deployed alongside CannObserv/watcher#252 in one window. Both services on co-core 0.8.0.

   Order executed. Both consumers stopped → XPENDING confirmed 0 → streams snapshotted, then flushed (content.fetch, content.blobs, both .dlqs: 69/69/0/0 entries, plus 27 replicator:cmd:* keys) → Replicator uv sync --frozen, Watcher alembic upgrade head (d5a71c93e0f2 → e7c4b2a91f60) + uv sync --frozen → Replicator started first, Watcher second.

   content.fetch-policy was deliberately not flushed, and the boot proved why: the replay rebuilt 3 tracked hosts from all 1992 messages in 417 ms. Flushing it would have left every host on the fallback interval until Watcher republished.

   Verified on the live wire, one real command end to end:

   info_source_id    "isrc-deploy-smoke-28"          echoed verbatim
   blob_expires_at   2026-08-15T22:38:43.416719Z     stored_at + 7d
   envelope key      <content_fingerprint>:<command_id>   the #300 re-key
   

   Exit 0, blob on disk, both DLQs empty, no ERROR or WARNING on Replicator's side.

   The cross-service half is the part worth recording. Watcher consumed that fact, decoded it under 0.8.0, read info_source_id off it, and discarded it:

   blob fact matched no fetch command — discarding · command_id=01KZH… · info_source_id=isrc-deploy-smoke-28

   That is MUST-3 behaving exactly as written — it correlated on command_id, found no match for a synthetic seed id, and refused to fall back to url or to the domain key now sitting in front of it. The one warning in the window is the contract working.

   Two notes.

   • The smoke test used a fake --info-source-id against --production, so one command and one fact naming a nonexistent InfoSource remain on the live streams. Watcher discarded the fact; the blob expires 2026-08-15. Left in place rather than XDEL'd, since removing entries a consumer has already processed is more disruptive than leaving two aged frames to fall off.
   • watcher.service reports failed after a clean systemctl stop because uvicorn exits 143 on SIGTERM and the unit lacks SuccessExitStatus=143. Cosmetic, pre-existing, and worth a one-line fix in that repo so a real failure is distinguishable from an ordinary stop.

   Closing: both "Done when" criteria met, contract and reference revised, and the pair is live.
7. gregoryfoster

   added 2 commits that reference this issue on Aug 8, 2026

   #252 merge: populate info_source_id on content_fetch commands

   ...

   d5ab227

   #29 fix: apply round-3 review findings (CR #26-#38)

   ...

   119267a
8. gregoryfoster

   added a commit that references this issue on Sep 24, 2026

   #118 docs: curate DEPLOYMENT.md — two misplaced blocks home, two hist…

   ...

   ef0f981