Concurrent entity occurrence integration test can time out after 60 seconds #878

Current result: repair published; consumer CI green; merge awaiting review verdict

A controlled shutdown schedule identifies a hanging promise below the runtime, in Plumbing's Node session input closure. The old implementation awaited stdin finish or error; stdin and the child process can close without that final-flush event. This strands typed Git protocol close, git-cas reader retirement, and git-warp history shutdown.

• Plumbing #19 is merged at 405e34badaa4133ef2a7bca4ef2cbc671624cd8a and released as 3.3.1. Publication and its Node/Bun/Deno gates passed.
• git-warp #880 contains the deterministic consumer regression and adoption of Plumbing ^3.3.1, locked to the public npm artifact. The existing git-cas 6.5.10 dependency accepts this patch; it needs no new release.
• The upstream fix settles input closure on stdin close or process completion and cleans up temporary listeners. Protocol completion still checks process exit status.

The consumer test advances the 1000 ms idle timer with fake timers. It uses real Git in owned scratch space, controls the Writable final callback while delivering real EOF, witnesses one persistent cat-file --batch-command reader and closed stdin without finish, then asserts that history shutdown settles. One event-loop turn drains continuations; elapsed time is not the oracle.

The hosted red at 99dc6fb passed all 7,521 unit tests and 118 other integration tests, then failed history close must settle after idle reader exit. Its first failure is preserved. Final commit 14fa42ae1 includes the aligned Deno import map and an array guard for the scheduling fixture's spawn overloads. All normal push gates and hosted CI pass: 7,521 unit tests with two pre-existing skips, 138 integration tests, eight BATS CLI tests, Bun, Deno, coverage, lint, types, documentation, and path hygiene. Performance comparison and release preflight also pass. CodeRabbit confirmed that its only finding is fixed and there are no remaining change requirements; all review threads are resolved. Its earlier CHANGES_REQUESTED verdict remains active while its review quota is exhausted. The ordinary head-matched merge was rejected by branch policy, and auto-merge is disabled. #880 awaits a fresh formal verdict or explicitly authorized dismissal of that stale review. No admin merge or review-rule change has been made.

The original timeout log had no operation-level diagnostic, so its exact historical event ordering remains unknown. The original failure and merged #879 stage diagnostics are preserved. The controlled hanging mechanism and red/green evidence establish this repair independently of any later successful retry. The original concurrent-occurrence assertions and runner watchdog remain unchanged; there is no release waiver.

Acceptance

• Identify and bound a closure stall through a controlled Node stream/process schedule.
• Observe the consumer assertion red on the unfixed dependency, locally and in hosted CI.
• Fix the upstream lifecycle and publish the reviewed package.
• Validate concurrent occurrence and closure contracts plus the Node 22 integration lane against a fresh registry install.
• Land fix(storage): complete idle Git session shutdown #880 after final CI and review; retain diagnostics for any recurrence of the original timeout.

Earlier investigation record

The following sections preserve the evidence and rejected proposal from before the controlled reproduction above.

Observed failure

The Node 22 CI job for PR #877 at b9f63a26488afa378e5581ed709b23e7ccb9d47d failed because keeps concurrent occurrences incomparable but deterministically ordered exceeded 60,000 ms. The other four tests in that file passed. The integration lane finished with 117 passes and one failure.

• Failed CI job and original log
• Test at the failing revision

The test opens two Runtime instances against one disposable repository, writes to the same lane with different writers using Promise.all, checks the occurrence relations and ordering, then closes both runtimes. The timeout does not identify which operation stalled.

One local diagnostic run of the unchanged test file passed all five tests. That is not a reproduction or resolution of the CI failure. The original failing signal remains evidence; no retry has replaced it.

Release consequence

Resolve or explicitly disposition this failure before v19.2.0 release tracking #876 closes. Do not raise the timeout, remove the assertion, or rerun CI until green as a substitute for diagnosis. Current ownership: release campaign operator; no quarantine or coverage removal is proposed.

Required evidence

• Identify whether the stall occurs in runtime opening, lane setup, admission, or closure, with bounded diagnostic artifacts.
• Reproduce the mechanism with controlled ordering or fault injection where feasible; preserve the original CI failure and any diagnostic runs.
• Fix the responsible runtime or test-harness behavior with red-on-unfixed evidence, or document a specific reviewed risk disposition if the mechanism remains irreproducible.
• Validate the relevant concurrent write and closure contracts, then the Node 22 integration lane.

Investigation update

PR #879 at 699c2b0e3dba20afa4e7b1f1fb49d782f9d1a5f7 adds failure-only reporting for public runtime/lane opening, both writes, and both closes. A deliberately stalled close produced the expected test failure and named close writer-b: pending; that calibrates the diagnostic, not the original bug. The exact-head Node 22 CI passed: integration job.

Bounded diagnostic exploration in a disposable Linux arm64 container used Node 22.23.2 and source-built Git 2.55.0. The baseline, five idle-retirement schedules, and 24 payload cases (1 byte through 128 KiB) completed. One-shot cat-file, mktree, and fast-import failures recovered; a one-shot update-ref failure rejected with PUBLICATION_REF_UPDATE_FAILED and both closes completed. The original runner was Ubuntu 24.04 x64; these are not identical environments or exhaustive interleaving evidence.

The original failure remains unreproduced. No runtime fix, waiver, quarantine, timeout increase, or assertion removal has been made. The next occurrence must retain the pending lifecycle stage; publication still requires a fix or a specific reviewed risk disposition.

Deferral rejected — investigation remains a release blocker

The operator rejected deferral and directed the campaign to solve the timeout, including deterministic clock/schedule control where time affects behavior. Keep this issue priority:asap, status:active, and in v19.2.0. No release waiver is authorized. The earlier proposed deferral and its conditions are retained below as historical context only; they are not active release permission.

Basis: the original failed CI log is preserved; merged PR #879 supplies calibrated pending-stage diagnostics; bounded Linux Node 22/Git 2.55 idle-retirement, payload, and process-failure probes did not reproduce the stall; subsequent exact-head CI and the prepared v19.2.0 full local preflight passed.

Residual risk: the stalled operation and mechanism remain unknown. A runtime admission or closure could stall again; the passing probes do not establish absence of that failure in production. No data-corruption mechanism has been demonstrated, and no claim of safety from corruption follows from that absence of evidence.

Conditions if approved:

• Keep the original concurrent-occurrence test gating with the same assertions and timeout. Preserve first-failure diagnostics; do not retry it into an unqualified green result.
• A recurrence, reproducible mechanism, or production report immediately restores priority:asap and blocks the next publication pending diagnosis.
• Retain the original failure, replay details, diagnostic PR, and explicit limitations in the release evidence.
• Re-evaluate the accepted risk by 2026-09-21; the deadline is an investigation review, not automatic test deletion or a claim of resolution.
• Think continues through its isolated proof gates after public package verification. This disposition does not authorize a production capture-store cutover or retained-mind migration.