Committed-session index survives a throwing scan instead of wedging it - #690
Conversation
…he daemon (#685) `createCommittedSessionIndex`'s self-clearing guard attaches a fulfillment-only handler to the scan promise. Nothing awaits the derived promise, so a scan that ever rejected became an unhandled rejection and Node terminated the daemon. `source.js` catches whatever `projectExchange` rejects with, so that orphan was the one path out of the projector that reached Node's default handler. `rebuild()` now normalizes a rejecting scan into the same `undefined` "could not scan" outcome the scan already reports for a failed partition discovery, which makes the guard total and lets `mightHaveCommittedRows` err toward scanning instead of propagating. `scanCommittedSessionIds` documents the non-rejecting contract the guard depends on. Co-Authored-By: Claude <noreply@anthropic.com>
Neutral review round - PR #690 @
|
Review round 2 finding. Now that a rejecting scan degrades the index instead of killing the daemon, the log line is the only thing that says capture is running on a degraded index, and both ways to get there emitted the same `aigw.session_index_scan_failed` with the same fields. An operator could not tell `discover_failed` (an I/O condition the next rebuild may clear on its own) from `scan_rejected` (the scan broke its documented "resolve, never reject" contract, so it will keep failing until someone fixes the code). Add an `error_kind` discriminator to both sites, per the repo's log-driven-development convention, and assert the rejection path's warn in the regression test, which previously passed no log and so proved nothing about it. Co-Authored-By: Claude <noreply@anthropic.com>
Neutral review ROUND 2 - PR #690 @
|
| # | scenario | outcome |
|---|---|---|
| S1 | 10 consecutive index-scan rejections | 10/10 rows still emitted, 0 throws, exactly 20 discover calls (1 build + 1 fallback per exchange), exactly 10 warns, 0 unhandled |
| S2 | scan rejects after a previous successful build | stale miss past the window rebuilds, that rebuild rejects, row still emitted, built self-clears, next exchange rebuilds successfully and recovers. Calls: [ok, throw, ok, ok] |
| S3 | rejection while N=6 callers await the shared promise | all 6 fulfilled, 1 row each, exactly one index scan shared, 6 fallback scans, 0 unhandled |
| S4 | N=6 concurrent stale misses past the rebuild window where the rebuild rejects | all 6 fulfilled, exactly one shared rebuild, 0 unhandled |
| S5 | control: persistent discover_failed (the undefined path #683 already shipped) |
10 rows, 10 warns, identical steady state |
Three things fall out of this that I consider the strongest evidence in the PR's favour, and that round 1 did not produce:
- No unbounded retry and no leak. Failure costs exactly one rebuild per exchange. Not one per caller (S3/S4 share it), not exponential, not a storm. Each rebuild allocates two collectable promises.
- No wedge and no poisoned cache. S2 shows the index recovering on the very next exchange after a failure that followed a success.
- S5 is the decisive one. The new
.catchproduces a steady state identical to theundefinedoutcome The gateway daemon leaks until GC thrash: recorder retention, unbounded dedupe and seed scans (LLP 0204) #683 already shipped. That is precisely what "normalize a rejection into the same could-not-scan outcome" should mean, and it is measurable rather than asserted.
I also checked the tempting alternative fix, guarding the loop with Array.isArray(partitions). It would be wrong here: a malformed answer would yield an empty Set, i.e. an authoritative "no session has committed rows", which suppresses seeding and loses real rows. .catch to undefined ("could not scan", err toward scanning) is the right normalization. The PR picked correctly.
Is undefined correct per LLP 0204? CONFIRMED.
llp/0204-gateway-daemon-memory-leak.issue.md "Fix" describes the seed index as tolerant by construction: "a stale miss only risks the duplicate seeding guards against, which settlement/compaction still collapse". Erring toward true costs an extra scan and loses nothing. false would silently drop rows.
Does swallowing the rejection mask a permanent systemic failure?
Partly, and it is worth stating plainly, but it is not a defect this PR introduces. On permanent failure the daemon returns to pre-0204 behaviour: a whole-table per-session scan for every new session, which is root cause #3 of the very incident 0204 documents. So a permanently broken index silently walks the daemon back toward the 4.8 GB shape, with only warn lines to notice. S5 proves this property is identical to the undefined path already shipped in #683, so it belongs to 0204's design as shipped, not to this change. Surviving is still unambiguously better than dying: a crash-looping daemon takes down the proxy and every other dataset with it. I did act on the observability half of this (see F1).
Conventions
Re-checked independently: no semicolons, no U+2014 in the diff, no @typedef, no inline import() types. @ref LLP 0204#fix [constrained-by] resolves (0204 has a ## Fix heading), sits with no blank line above const ids, and the gloss carries information the code does not. No LLP text edited, which is right: this is a defect in 0204's implementation, not a change to what 0204 decided.
2. FIXED AND PUSHED
F1 - a survivable failure that does not say which failure it was. (Round 1 logged the shape of this as nit N2 and did not act; it has more teeth than a nit, because this PR is exactly what makes it matter.)
Before this PR, a rejecting scan killed the process, which is at least loud. After it, the daemon keeps serving on a degraded index and the log line is the only thing that says so. Both routes to that line emitted the same message with the same fields:
message_projector.js:426-discoverCachePartitionsthrew. An I/O condition; the next rebuild may well clear it on its own.message_projector.js:353(new in this PR) - the scan rejected, breaking its own documented "resolve, never reject" contract. A code defect; it will keep degrading the index on every exchange until someone fixes it.
Identical output, opposite responses. CLAUDE.md's Log-Driven Development section calls for exactly this discriminator ("prefer structured attributes such as component, operation, status, error_kind").
Pushed as 3bb9a52:
- added
error_kind: 'scan_rejected'at the new site anderror_kind: 'discover_failed'at the pre-existing one, with a comment saying why the two must be distinguishable - extended the regression test to actually pass a log and assert the warn. Round 1 noted (N3) that the test builds its projector via
freshSessionProjector(storage, () => 0)with nolog, so nothing asserted the warn at all; round 1 verified it by hand and left it untested. It is now tested: exactly one warn,error_kind === 'scan_rejected', error message matching/not iterable/.
No behaviour change beyond the added log field.
Verification the fix actually landed (not merely that the suite is green), against the pushed tree rather than my working copy:
$ git show origin/fix/issue-685:.../ai-gateway/src/message_projector.js | grep -n error_kind
352: // `error_kind` separates the two ways this message is reached, because
360: error_kind: 'scan_rejected',
434: error_kind: 'discover_failed',
$ git show origin/fix/issue-685:test/plugins/ai-gateway-message-projector.test.js | grep -n "scan_rejected\|collectingLogger(logged)"
934: const projector = freshSessionProjector(storage, () => 0, collectingLogger(logged))
952: assert.equal(scanWarns[0].fields.error_kind, 'scan_rejected')
And the new assertion bites. Deleting only the error_kind: 'scan_rejected' line:
not ok 1 - committed-session index: a scan that throws degrades the index instead of killing the process
expected: 'scan_rejected'
# pass 0 # fail 1
3. STILL ACTIONABLE (not fixed here)
A1 - hard conflict with PR #689. CONFIRMED. Severity: medium (merge coordination). Round 1's finding is real; I reproduced it and re-confirmed it at the new head:
$ git merge-tree --write-tree 3bb9a52 da2cd4f
CONFLICT (content): Merge conflict in .../ai-gateway/src/message_projector.js
CONFLICT (content): Merge conflict in test/plugins/ai-gateway-message-projector.test.js
I also read #689's diff rather than trusting the summary, and can sharpen round 1's warning. #689 replaces the floating .then with attempt.ids = scan.then(...) and that chain has no rejection handler. So a merge that takes #689's rebuild() wholesale does not just lose one row: attempt.ids stays permanently rejected, and because the guard is a fulfillment handler it never runs, so built is never cleared. Every later mightHaveCommittedRows awaits the same rejected promise. That is a permanent, total, silent loss of ai_gateway_messages capture, not a transient miss. Keeping both changes is mandatory, not stylistic. Round 1's suggested merged form is correct; it now also needs the error_kind field from 3bb9a52.
Round 1 checked #689 only. I also checked #688 (6127364), which touches the same test file: git merge-tree --write-tree 3bb9a52 6127364 is clean. No action needed there.
This remains owned by the merge rung, not by this PR.
A2 - NEW, not raised in round 1. Severity: medium. Pre-existing on master; this PR un-masks it.
scanCommittedMessageIds (message_projector.js:480) has the identical unguarded for (const part of partitions ?? []) outside its try/catch that this PR just hardened one function away, and its JSDoc at :454 states it "NEVER throws". That claim is false for the same malformed input.
Round 1 did not reach this because the PR's own fixture is shaped around it: the test makes only the first discoverCachePartitions malformed, so the per-session fallback scan is well-formed. A real storage that violates the contract violates it on every call. Driven that way, against the post-fix code:
results: [ THREW: (partitions ?? []) is not iterable, <- sess-same, exchange 1
THREW, THREW, THREW, <- sess-same, exchanges 2-4
other THREW ] <- a different session
warns: [ aigw.session_index_scan_failed,
aigw.session_index_scan_failed ] <- only TWO, for five exchanges
unhandled: []
Read that carefully:
- The daemon survives. Zero unhandled rejections. This PR's stated goal is met, and
source.js:228absorbs the awaited rejection. - But every
projectExchangerejects, so every row is dropped, for every session. - And it goes silent.
seedPromises(:285-291) memoizes the rejected seed promise per session forever, so exchanges 2-4 forsess-sameshort-circuit on the poisoned memo, never reach the index, and emit no warn at all. Five exchanges, two warn lines, 100% row loss.
So this PR converts "daemon dies loudly in a restart loop" into "daemon lives and silently drops all gateway capture". Surviving is still the right trade, and the poisoned-memo behaviour is genuinely pre-existing (scanCommittedMessageIds and the memo are untouched here). But it is only reachable now, and it deserves its own issue LLP + PR rather than a drive-by widening of this one: the honest fix is to make scanCommittedMessageIds honour its own "never throws" contract (there, unlike in the session scan, degrading to "seeded nothing" is the documented behaviour), plus a decision about not memoizing a rejected seed. That is a separate change set with its own regression test.
Flagged, not fixed here. Note that F1 partially mitigates it: the first warn now says error_kind: 'scan_rejected', which is the operator's one signal that this is a code defect rather than a storage blip.
Nits carried forward, no change requested
- N1
message_projector.js:359/373- the inner.then((ids) => ...)shadows the outerconst ids. - N4 if
log.warnitself threw inside the.catch, the derived promise would be unhandled again. Theoretical (fieldsis plain strings); the identical exposure predates this PR at:426.
Checks run
At 8ae05e2 (unmodified):
npm test -> 1..3862 # pass 3858 # fail 0 # skipped 6 (exit 0)
npm run typecheck -> clean, exit 0
At 3bb9a52 (after F1):
npm test -> 1..3862 # pass 3858 # fail 0 # skipped 6 (exit 0)
npm run typecheck -> clean, exit 0
node --test test/plugins/ai-gateway-message-projector.test.js -> # pass 36 # fail 0
Neutral triage - PR #690 @
|
Both sides changed `createCommittedSessionIndex().rebuild()`: - master (#689) chained the self-clearing guard onto the scan so `atMs` is stamped when the scan COMPLETES; - this branch (#685) normalizes a rejecting scan to the `undefined` "could not scan" outcome so the failure degrades the index instead of escaping. Resolved by keeping both: the `.catch` normalization now wraps the scan promise that master's chain is built on, so the stamp-on-completion and the shared in-flight attempt are untouched and the chained handler still only ever sees a fulfilled value. master's chaining incidentally gave the rejection an awaiter, so the unhandled-rejection-kills-the-daemon shape is no longer reachable on its own; the wedge it caused still is. Without the `.catch`, the rejected attempt stays published in `built` and never self-clears, so every later exchange on that listener re-awaits the same rejection (verified: 3 sequential exchanges, all rejected, index never rebuilt). The regression test now pins that directly with a second exchange, and keeps the unhandled-rejection assertion so the chaining cannot silently regress. Co-Authored-By: Claude <noreply@anthropic.com>
Neutral triage - PR #690 @
|
What was wrong
hypaware-core/plugins-workspace/ai-gateway/src/message_projector.js,createCommittedSessionIndex().rebuild():The self-clearing guard attached a fulfillment-only handler to the scan promise. Nothing awaits the derived promise, so if
scanCommittedSessionIdsever rejected, that derived promise was an unhandled rejection and Node terminated the daemon.This is the one escape hatch out of the projector that reaches Node's default handler:
source.jsonExchangeFinishedwrapsprojectExchangeintry/catch, so a rejection travelling the normal awaited path is absorbed and logged. Only the orphan derived promise is fatal. That makes it strictly worse than the pre-#683 behaviour, where the scan promise always had an awaiter, and it sits in the shutdown-safety path LLP 0204 exists to harden.Secondary consequence of the same defect: because the fulfillment handler never ran on a rejection,
builtwas left pointing at the rejected attempt, so every subsequentmightHaveCommittedRowsre-awaited the same rejected promise for the listener's lifetime instead of retrying: a permanently wedged index, not a degraded one.Root cause
scanCommittedSessionIdssignals "the scan could not run" by resolving toundefined, never by rejecting (both awaits sit insidetry/catch), and the guard was written to assume that contract rather than enforce it. Nothing structural held it.The fix
rebuild()normalizes a rejecting scan into the sameundefined"could not scan" outcome the scan already reports for a failed partition discovery (logging the sameaigw.session_index_scan_failedwarn):One
.catchat the single point where the promise is created makes every consumer total, not just the guard:ids === undefinedbranch fires, sobuiltself-clears and the next caller retries;mightHaveCommittedRows'sawait current.idsyieldsundefinedand returnstrue, so the index degrades to "err toward scanning" and the row is still emitted.Adding only
attempt.ids.then(handler, () => {})(the two-character fix suggested in the issue) would silence the crash but leave the index wedged on the rejected attempt and still lose the exchange. This is the same size of change and satisfies the issue's own gate.scanCommittedSessionIds's JSDoc now states the non-rejecting contract explicitly, as the issue suggested, and the normalization carries// @ref LLP 0204#fix [constrained-by]: the seed index is documented best-effort, so a scan it cannot complete must degrade the index, never end the process. No LLP text was edited (nothing it decided has changed).Regression test
test/plugins/ai-gateway-message-projector.test.jscommitted-session index: a scan that throws degrades the index instead of killing the processDriven by a storage whose first
discoverCachePartitions(the index build) answers with a non-iterable, so thefor (const part of partitions ?? [])loop throws from outsidescanCommittedSessionIds'stry/catchand the scan promise genuinely rejects. Later calls are well-formed, so the per-session fallback scan is unaffected and the "err toward scanning" degradation is observable on its own. The test installs aprocess.on('unhandledRejection')collector, drains the loop, and asserts it is empty, plus asserts the row is still emitted and the fallback scan ran.FAIL before the fix
git stashonmessage_projector.jsonly, test unchanged:The runner names the defect directly:
triggered an unhandledRejection event.Standalone confirmation outside the test runner, with
projectExchangecaught exactly assource.jscatches it (so the only escaping rejection is the orphan), on unfixed code:The same script on fixed code:
(the process no longer dies; and inside the test, where the fallback scan is left working, the exchange's row survives too).
PASS after the fix
Full suite
🤖 Generated with Claude Code
Fixes #685
Update at head
045abc33(merge oforigin/master): what this PR fixes has shiftedMaster moved under this PR (5 PRs,
b28aae3->3e8f05a), andorigin/masterwas merged into the branch to resolve the conflict with #689. That changes the rationale above in two ways:.thenwith a chain (attempt.ids = scan.then(...)), which incidentally gives a rejecting scan an awaiter: the rejection propagates throughmightHaveCommittedRowsintoprojectExchange, whichsource.jscatches. Verified on the merged-but-unfixed tree: a rejecting index scan produces zero unhandled rejections, exit 0..catch, the rejected attempt stays published inbuilt, the stamp-and-self-clear handler runs on fulfilment only, andatMsstaysInfinity(never stale), so the failed attempt never clears and never ages out. Verified on the merged-but-unfixed tree: 3 sequential exchanges all rejected,discoverCallsstuck at 1, zero rows emitted, zero warns. With the fix: every row emitted, the index self-clears and rebuilds, oneerror_kind: 'scan_rejected'warn.The root cause and the fix are unchanged:
scanCommittedSessionIdscan reject while nothing inrebuild()tolerates a rejection, and one.catchwhere the promise is created makes every consumer total. Only the surviving symptom changed, from "kills the daemon" to "silently loses every subsequent exchange on the listener".The regression test is renamed accordingly (
committed-session index: a scan that throws degrades the index instead of wedging it) and now pins both shapes: a second exchange asserts the failed attempt clears itself (fails unfixed: the awaited rejection throws before it), and theunhandledRejectioncollector assertion keeps the crash shape closed, since nothing structural on master holds #689's chaining in place.The merge resolution is the composed form both review rounds prescribed: the
.catchnormalization wraps the scan promise that master's completion-stamping chain is built on, so #689's stamp-at-completion and shared in-flight attempt are untouched and the chained handler only ever sees a fulfilled value. At this head: 3895 tests, 3889 pass, 0 fail, 6 skipped; typecheck clean.