Skip to content

fix(sentinel): bound triage calls per run and keep Sentinel off the GPU around chat turns (#866) - #875

Merged
goruck merged 2 commits into
mainfrom
fix/sentinel-triage-burst-866
Oct 27, 2026
Merged

goruck merged 2 commits into
mainfrom
fix/sentinel-triage-burst-866

Conversation

@goruck

@goruck goruck commented Oct 27, 2026 •

Copy link
Copy Markdown
Owner

Step 1 of the plan on #866 (part of #866; the model selector follows as its own PR).

What happens

On a single-GPU host where one Ollama server serves both chat and Sentinel, a run with many findings sends one triage call per finding, back to back. A voice turn arriving mid-burst cancels the call in flight; on Ollama a cancelled request can wedge the runner, and the voice turn that cancelled it then stalls for minutes. The reporter saw 17 triage calls in a couple of seconds, triage_suppress_rate: 0 across ~500 findings, and a multi-minute voice stall.

What changes

  • Triage calls per run — new option in Sentinel's Advanced setup (sentinel_triage_max_per_run, default 5, 1–50). On a local model shared with chat, a run sends at most that many findings to the triage model; the rest pass through untriaged with reason_code: triage_cap. The same limit caps the run's explanation calls, and a finding that passed through gets no explanation. A cloud provider, or a server marked model_provider_uncontended, is never limited.
  • A gap between calls (SENTINEL_TRIAGE_CALL_GAP_S, 1 s) measured from a shared clock of the last Sentinel model call of any kind (triage, explanation, discovery), across runs, so a chat turn arriving between two calls starts without cancelling anything.
  • Once chat takes the model during a run, the run's remaining triage and explanation calls are skipped (triage_deferred) instead of resuming between voice turns. The deferral is signalled by the admission gate's exception, never by the model's reason code.
  • A grace period after a chat turn (SENTINEL_CHAT_GRACE_S, 30 s): on a shared local model no Sentinel LLM call starts within 30 s of a chat turn ending, so a conversation's follow-up turns never land on a call started between them. Admission defers at once while chat is active or in its grace, and waits only for video. The grace yields, with a warning, after a streak of deferrals (no more than a grace apart) that lasts 300 s, so an automation calling conversation.process every few seconds cannot hold Sentinel off for good, while a quiet house keeps its grace.
  • model_provider_uncontended now bypasses every local gate as documented (chat session, video session, Sentinel admission and grace, probe gate). Before, chat still marked itself active and Sentinel still deferred with it on.
  • The stalled-provider probe (Show a stalled model backend and fail fast instead of waiting out long timeouts #828) is not sent while chat or video holds the GPU (the grace does not apply to it), is retried 10 s after a skipped round, is never cancelled once sent (a cancel can re-wedge a runner that just recovered), and a recovery recorded by another call lets it finish; a re-stall reuses the live loop. stalled_model_providers shows probe_rounds_skipped. Closes the TODOS entry "Stall probes skip the chat-priority gate", the other way round from what it asked: the review showed cancelling the probe is the hazard.
  • Health sensor: triage_calls_last_run and triage_passed_through_last_run; triage_suppress_rate now counts only findings the model decided on.
  • Reload safety: a chat or video turn that entered before a config-entry reload no longer touches the counters the reload reset.
  • Docs (docs/sentinel.md, docs/constants.md, docs/configuration.md): the new option and attributes, the grace period, and the two points from the issue: keep model_provider_uncontended off on a shared-GPU host, and a cancel cannot un-wedge an Ollama runner. TODOS.md gains an entry for a value-ordered / rate-based triage budget.
  • Strings in en, cs and zh-Hans. CHANGELOG entry under Unreleased.

Review

gstack review, three fix cycles (testing, maintainability, security, performance, simplification specialists; Red Team; Claude adversarial; Codex gpt-6-astra), about 40 findings fixed, including a resolver allowlist omission that made the option a no-op, an explanation burst the cap did not cover, two wrong clocks for the grace yield, a model-spoofable deferral code, and a duplicate probe loop. Record ends converged: false by protocol (edits in cycle 3), with no known open defect. QA evidence in .gstack/qa-reports/review-866/.

Tests

New: cap and pass-through, begin_run renewal, the gap within and across runs and before explanations, the per-run short-circuit after a deferral, a model reply that cannot fake a deferral, an interrupted call staying counted, cloud deployments never capped, the grace (deferred at once, streak-based yield with its warning, stale deferrals ignored, chat arriving during the video wait), uncontended bypassing every gate, reload-generation safety for chat and video turns, the probe runner's branches (cloud, uncontended, gate closed, grace ignored, a sent probe finishing), a skipped probe round retried and counted, a re-stall reusing the live loop, a recovery leaving a sent probe alone, explanation cap/skip/gap and the option-to-limit path, the explainer's deferred flag, KPI exclusion, the resolver propagating the option, the sensor allow-list, and _run_once renewing both budgets each run.

make lint, make test, make typecheck clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01QytwFSMNhTRnAfuCqngGL7

…PU around chat turns (#866)

On a single-GPU host where one Ollama server serves chat and Sentinel, a
run with many findings sent one triage call per finding back to back; a
voice turn arriving mid-burst cancelled the call in flight, which can
wedge the Ollama runner and stall the voice turn for minutes.

- New Sentinel Advanced option "Triage calls per run"
  (sentinel_triage_max_per_run, default 5); the rest of a run's findings
  pass through untriaged (triage_cap). Calls are 1 s apart, and once chat
  takes the model during a run the run's remaining findings pass through.
- No Sentinel LLM call starts within 30 s of a chat turn ending on an edge
  deployment (SENTINEL_CHAT_GRACE_S), so a conversation's follow-up turns
  never land on a call started between them.
- The stalled-provider probe (#828) follows the same gate and is cancelled
  by chat like any other background call; closes the TODOS entry "Stall
  probes skip the chat-priority gate".
- sentinel_health: triage_calls_last_run, triage_passed_through_last_run;
  triage_suppress_rate counts only findings the model saw.
- Docs, strings (en/cs/zh-Hans), CHANGELOG.

Part of #866.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QytwFSMNhTRnAfuCqngGL7
@claude

claude Bot commented Oct 27, 2026

Copy link
Copy Markdown

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

🤖 Generated with Claude Code

…race and stall probe (#866)

Three gstack review cycles (specialists, Claude adversarial, Codex, Red
Team) on PR #875. The fixes:

- The new option was missing from the resolver allowlist, so a saved cap
  never reached runtime; added, with a resolver test.
- The same limit now caps explanation calls per run, on every dispatch
  path, and a finding that passed through triage gets no explanation;
  the 1 s gap is measured from a shared last-call clock that triage,
  explanations and discovery all wait on. Cap and gap apply only to a
  local model shared with chat: a cloud provider, or a server marked
  model_provider_uncontended, is never limited.
- sentinel_admission defers at once while chat is active and waits only
  for video; the 30 s chat grace yields, with a warning, after a streak
  of deferrals no more than a grace apart that lasts 300 s, so chat turns
  every few seconds cannot hold Sentinel off for good while a quiet house
  keeps its grace. model_provider_uncontended is now a module flag that
  bypasses every local gate, as documented. A chat or video turn from
  before a reload no longer touches the counters the reload reset.
- A triage deferral is signalled by the gate's exception, never by the
  model's reason code; reserved triage_* codes in a reply are normalised.
  An interrupted call stays counted; a refused one does not.
- The stall probe is gated at start only (active chat or video; the grace
  does not apply), retried 10 s after a skipped round, never cancelled
  once sent, and a recovery recorded by another call lets it finish; a
  re-stall reuses the live probe loop; skipped rounds are counted on the
  health sensor and refresh it once per streak.
- triage_suppress_rate counts only findings the model decided on.
- Test-only APIs removed (run_stats, max_per_run, sentinel_may_start,
  run_tracked_sentinel_call, retry_in_s); docs, strings and CHANGELOG
  follow the final behaviour; TODOS.md gains the value-ordered budget.

Part of #866.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QytwFSMNhTRnAfuCqngGL7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant