Skip to content

feat: herdr-app backend that shows CAO sessions in the Herdr desktop app - #1356

Open
anilkmr-a2z wants to merge 27 commits into
awslabs:mainfrom
anilkmr-a2z:feat/herdr-app-backend
Open

anilkmr-a2z wants to merge 27 commits into
awslabs:mainfrom
anilkmr-a2z:feat/herdr-app-backend

Conversation

@anilkmr-a2z

@anilkmr-a2z anilkmr-a2z commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Adds a herdr-app terminal backend. It runs CAO terminals in a dedicated herdr-core session, so each CAO session shows up in the Herdr desktop app as one space, with one tab per terminal. It also adds a one-time script that moves herdr TUI sessions into the app.

Fixes #1355

Why. The Herdr app runs on herdr-core (hc), not on the TUI server that the herdr backend drives, so CAO terminals don't appear in the app. A common setup is the app on a laptop and CAO on a remote box the laptop reaches over SSH.

Approach

  • Connection. CAO talks to herdr-core through hc connect --session <s>, the core's own entry point and the one the app uses over SSH. The socket path differs between Linux and macOS, so CAO doesn't open it directly. Every hc child, including hc attach, is started without HC_TERMINAL_ID, HC_TERMINAL_SOCKET, HC_SCOPE and HC_SOCKET. A cao-server started inside an app terminal therefore never reaches the user's own core.
  • Placement. Terminals are placed with the app's documented herdr.space and herdr.tab labels. Space and tab names exist only in the app's herdr-app layout document, so CAO sets them with one version-checked write to it. Writing the whole document for placement was rejected: the app resolves concurrent edits as "newest accepted document wins".
  • Tab order. The app inserts an adopted tab after the selected one, so workers pile up next to the conductor. CAO's naming pass puts its own tabs back in order (each after the terminal that created it, siblings in creation order). It moves only its own tabs, within the positions they already hold, so a person's tab or another cao-server's tab keeps its place and two servers sharing a space settle. Dragging a CAO tab is undone on the next layout change.
  • Ownership. A cao.home label keeps each cao-server to its own terminals.
  • Status and inbox. HerdrAppInboxService follows the core's event stream for status, exits, lost terminals and layout changes. Reconcile only drops a CAO row whose core record has finished, so rows from tmux or the herdr TUI are left alone. A native error status is ignored, because herdr-core keeps it after a failed turn while the agent is back at its prompt.
  • Input. Each terminal gets its own input connection, so a send waiting on a person's draft (up to 60 s) blocks neither status lookups nor input to other terminals. Inbox deliveries confirm the submit and resend Enter when the agent did not pick the message up, because herdr-core presses Enter 200 ms after the paste whatever the provider needs.

Behavior change. This only applies with terminal.backend: herdr-app. tmux and the herdr TUI backend are unchanged.

Rollback. Set terminal.backend back and restart cao-server.

Verified

  • Tests. The backend, client, inbox, factory and migration-script suites pass: 283 tests, with CAO_EAGER_INBOX_DELIVERY unset. The e2e suite against a throwaway herdr-core: 3 passed. mypy: no new errors.
  • Live with mock_cli. A cao-server against a throwaway core: two terminals in one space, inbox delivery while idle and while busy, server restart, and session delete.
  • Live with kiro-cli 3.3.0. Placement, kiro's own status reports, and delayed inbox delivery. Model turns were not exercised, because the account was over quota.
  • Manual test with the app. A trial cao-server on herdr-app, viewed in the Herdr app on another machine over SSH (hc machine add <host> --session <s>). A conductor assigned a worker, and the worker's tab appeared in the conductor's space. The worker's callback reached the conductor. Typing into tabs from the app and deleting the session both worked.

@codecov-commenter

codecov-commenter commented Nov 16, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.79717% with 95 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@b2b3666). Learn more about missing BASE report.

Files with missing lines Patch % Lines
...i_agent_orchestrator/backends/herdr_app_backend.py 89.94% 29 Missing and 8 partials ⚠️
...t_orchestrator/services/herdr_app_inbox_service.py 89.01% 26 Missing and 4 partials ⚠️
src/cli_agent_orchestrator/clients/hc_rpc.py 82.50% 19 Missing and 9 partials ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1356   +/-   ##
=======================================
  Coverage        ?   91.92%           
=======================================
  Files           ?      299           
  Lines           ?    57923           
  Branches        ?     9520           
=======================================
  Hits            ?    53245           
  Misses          ?     3291           
  Partials        ?     1387           
Flag Coverage Δ
unittests 91.92% <88.79%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

malacration added a commit to malacration/cli-agent-orchestrator that referenced this pull request Nov 16, 2026
Integrate awslabs#1356 at 9224f3e.
Preserve the profile-env support from awslabs#665 by forwarding trusted_env through
HerdrAppBackend and applying the shared environment and runtime identity policy.

Validation: 711 focused tests pass, including profile environment regression
tests; migration self-test, Ruff, Black, and isort checks pass. Type checking
has the same 117 diagnostics as the prior branch in the local environment,
with no new per-file/error-code counts. Live core and graphical checks require
desktop/socket access and were not performed.

@haofeif haofeif left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at 9224f3e: Request changes (0 P1, 2 P2, 1 P3)

Thank you for this backend. Its design is careful:

  • Every hc child is started without the HC_* routing variables.
  • Layout writes are version-checked and retried, and they change only the spaces and tabs that CAO owns.
  • The cao.home label keeps each cao-server to its own terminals.
  • Reconcile drops only rows whose core record has finished.
  • The migration script only reads the TUI.

I found two P2 problems on the input path:

  • All terminals share one input connection. So a send that waits for a person's draft in one terminal stops input to all other terminals.
  • herdr-core waits a fixed 200 ms between the paste and Enter. CAO's per-provider submit delay is not used, and on this backend inbox deliveries have no resubmit.

There is also one P3 for the "Sends wait" docs.

Other reviewers

  • Codecov (comment): "patch coverage 87.94%, 89 lines missing" (backend 87.79%, inbox service 88.97%, hc_rpc.py 84.56%): Confirmed. The report has no BASE upload, so it shows no project delta. The coverage gap is not the reason for my findings. The tests that exist do not check either P2 path. The input-connection test checks status lookups only, and no test checks the paste-to-Enter gap.
  • PR description claims:
    • "Input goes over a separate connection, so a send waiting on a person's draft (up to 60 s) doesn't block status lookups": Partly. This is true for status lookups (test_herdr_app_backend.py:430-478). But all terminals share the input connection, so a held send blocks input to every other terminal (P2-1).
    • "12033 passed, 1 failed … test_skips_when_processing … fails the same way on main": Not reproduced. The test passes locally at the PR head and on main, and the Unit Tests job passes. The PR does not change inbox_service.py, so the failure comes from the local environment, not from this PR.
    • "tmux and the herdr TUI backend are unchanged": Confirmed. The shared changes are a new factory.py branch, a new lifespan branch (api/main.py), a structural EventInboxService type in herdr_inbox_registry.py, and one constant. One small visible change: /health now always lists a herdr-core component, in the same way that it already lists herdr. The web card shows unavailable as "not installed".
    • "Live with kiro-cli 3.3.0 … delayed inbox delivery": Confirmed as stated. Claude Code was not part of the live tests, so those tests did not cover P2-2.

Findings

P2-1: All terminals share one input connection, so a send that waits for a draft in one terminal stops input to all other terminals (herdr_app_backend.py:113-118, :148-156; clients/hc_rpc.py:178-205)

  • _input() opens one HcRpcClient for the whole backend. _input_call sends all terminal.send and terminal.keys calls through it with INPUT_TIMEOUT_S = 90 s (:47). HcRpcClient.call holds self._lock while it writes the request and waits for the reply (hc_rpc.py:181-205). So the backend sends one input request at a time for all terminals.
  • herdr-core v0.1.21 holds terminal.send while a person has a draft in that terminal. The hold is up to 60 s after their last key (HC_DRAFT_MS or delivery.draft_wait), and each new key restarts it (crates/herdr-core/src/input.rs:81-89, :267-277, :314-370). A plain hc connect connection runs one request at a time (server.rs:170-214). Inside the core, each terminal has its own queue, so only CAO's shared connection joins them.
  • Effect: a person types in one CAO terminal from the app, and CAO sends to that terminal (for example, an inbox delivery to the idle conductor). Then all CAO input to every other terminal stops: inbox deliveries, send_input, startup-dialog keys (codex, kimi, antigravity), and typed launch commands (kiro, cursor, opencode, hermes, omp). The stop lasts up to 60 s after the person's last key, and up to 90 s if they keep typing. After 90 s, the first send fails with "herdr-core lost terminal.send", and the next caller continues. tmux has no such link between terminals.
  • I reproduced it with the real HerdrAppBackend and HcRpcClient and a fake hc connect that answers one request at a time, as hc plain mode does. The fake holds terminal.send to terminal A for 4 s. send_special_key to terminal B waited 3.76 s. The result was the same on the PR head and on the merge with main.
  • Options: (a) Open one input HcRpcClient for each terminal address, on first use, and close it in kill_address and cleanup. (b) Use a client that matches replies by id, with framed mode (server.rs:374 starts one task per request). Or keep terminal.keys on a shared connection and give terminal.send one connection for each terminal. In both cases, add a test: a send held on terminal A must not delay keys or a send to terminal B.

P2-2: herdr-app replaces each provider's paste-to-Enter delay with a fixed 200 ms, and inbox deliveries on herdr-app have no resubmit (herdr_app_backend.py:437, :452-453)

  • send_keys accepts submit_delay but does not use it. It calls terminal.send with force=True (:453). herdr-core writes the paste, waits for the terminal's template paste_gap_ms, else PASTE_GAP = 200 ms, and then writes Enter (input.rs:11, :459-467). A terminal gets paste_gap_ms only from a template (terminals.rs:127-134, :306-308). CAO starts its terminals without a template (herdr_app_backend.py:193-211), so every CAO terminal gets 200 ms.

  • CAO sets this delay for each provider because a short gap loses the submit:

    • Claude Code: 2.0 s. "A too-early Enter is swallowed and the message sits unsubmitted in the prompt box" (claude_code.py:1922-1928).
    • Kiro: 1.0 s (kiro_cli.py:605-608).
    • OpenCode: 1.0 s (opencode_cli.py:169-173).
    • Grok: 0.4 s.
    • All other providers: 0.3 s (backends/base.py:301).

    send_input passes the provider's value with force_bracketed_paste=True (terminal_service.py:6345-6351). tmux uses that value. The herdr TUI backend waits a fixed 2 s after each bracketed paste (herdr_backend.py:724-725). The base docstring lets a backend ignore the value only if it has no paste step (base.py:318-320). herdr-app has a paste step.

  • Inbox deliveries cannot recover:

    • HerdrAppBackend.supports_event_inbox() returns True (:500-501). So deliver_pending does not run _confirm_submitted, which is on by default (inbox_service.py:321-337, constants.py:380-382).
    • The reason for that skip is that the herdr TUI service delivers on the event loop. herdr-app already delivers in a worker thread (herdr_app_inbox_service.py:324).
    • Only the first task to a new worker still gets _confirm_worker_started_or_resubmit (terminal_service.py:5405-5417).
  • Effect: a callback from a worker to a Claude Code conductor can stay unsubmitted in its prompt box. The conductor does not start a turn, and the flow stops until a person presses Enter. I did not run Claude Code on herdr-app. This finding comes from CAO's own provider contract and from the herdr-core source.

  • Options: (a) Give herdr-core the provider's delay. For example, start each terminal from a CAO template that sets paste_gap_ms, or ask herdr-core for a gap on each send. (b) Let herdr-app run the inbox submit confirmation, because its delivery is already off the event loop. Use the same None boundary that the first-task path uses for event-inbox backends (terminal_service.py:5405-5416). In both cases, add a test: a provider with paste_submit_delay 2.0 must get at least that gap, or a swallowed Enter must be sent again.

P3-1: "Sends wait" does not describe what herdr-core waits for (docs/herdr-app.md:49; herdr_app_backend.py:45-46)

  • The docs say: "Typing into a terminal waits for about 3 seconds of output quiet". The backend comment says that the calls wait "for output to settle before typing".
  • herdr-core waits 3 s after a person's last key (HC_QUIET_MS), and only if a person typed (input.rs:81-89, :267-268). The output wait (800 ms of quiet, at most 5 s) applies only during the first 10 s of a terminal. After 5 s it fails with unavailable and types nothing (input.rs:17-19, :486-513).
  • Options: (a) Change the docs to: "waits until 3 s after a person's last key, and for their unfinished draft (up to 60 s). During its first 10 s, a new terminal also waits for its output to settle." Change the comment in the same way. (b) Remove the "output quiet" part from both.

What I checked

  • The full diff against main (24 files, +4556/-17):
    • The backend, the hc RPC client and the inbox service.
    • The lifespan and factory wiring, the registry type, and the docs.
    • The migration script.
  • The herdr-core v0.1.21 source for each method that CAO calls: terminal.run, send, keys, read, list, doc.set, events and the connection modes.
  • The input path, end to end, with a fake hc probe (P2-1).
  • The submit path from send_input to herdr-core, compared with tmux and the herdr TUI backend (P2-2).
  • The migration script:
    • It only reads the TUI (session list, api snapshot, pane process-info).
    • Its doc writes are version-checked.
    • Its login wrapper quotes "$@".
    • --no-daemon is a real codex flag.
    • --self-test passes.
  • send_keys(enter_count=0) callers send one character each (codex, antigravity, kimi), so the 64-character limit is safe.
  • Merge drift: the branch is behind main, which moved only by test-only #1357. git merge-tree is clean, and the P2-1 probe gives the same result on the merged tree.
  • Overfitting: the constants are generic (timeouts, HC_READ_MAX_LINES from a live measurement), and the status map follows the hc status words. The argument grammars in the script are part of rewriting CLI arguments, not special cases for one prompt.
  • CI: 27 checks pass at 9224f3e. Codecov is covered above.

Notes

  • Neither CI nor pytest runs scripts/herdr_tui_to_app.py --self-test. A small pytest wrapper would keep it from going stale.
  • "It never uses your default session" (docs/herdr-app.md:24): nothing refuses terminal.herdr_app_session default. Reconcile and kill touch only labelled terminals, so the risk is low. A check in factory.py, or a docs note, would make the statement true.
  • force_bracketed_paste is ignored on herdr-app. herdr-core chooses bracketed paste from the program's DEC mode 2004 (input.rs:465), so this is fine.

🤖 Reviewed with CAO review workflows

@anilkmr-a2z

Copy link
Copy Markdown
Contributor Author

Thanks for the careful review. All three findings and both notes are addressed in 3d98c65..e0d7536.

P2-1, shared input connection. Agreed, and fixed in 3d98c65. Input now uses one HcRpcClient per terminal address, opened on first use (herdr_app_backend.py:162).

  • I chose your option (a) over framed mode. herdr-core already queues input per terminal, so one connection per terminal gives the same concurrency without adding the binary frame protocol to CAO.
  • A terminal's input connection closes in kill_address (which covers kill_window, kill_session and cleanup_terminal_exact), on a removed event, when the naming pass no longer lists the terminal, and at shutdown.
  • HcRpcClient.close() no longer waits for a held call.
  • A new test holds a send on terminal A and checks that keys and a send to terminal B still complete.
  • docs/herdr-app.md now states the process count: N + 2 hc connect children.

P2-2, fixed paste-to-Enter gap. Agreed. hc 0.1.21 has no per-send gap: terminal.send takes caller, force, if, terminal, text, timeout and wait. paste_gap_ms comes only from template files in ~/.config/herdr-core/templates, which the user's own sessions share, so I did not do (a).

  • I did (b) in 5ada19a. A new event_inbox_confirms_submit() capability (backends/base.py:520, true at herdr_app_backend.py:627) lets herdr-app inbox deliveries run _confirm_submitted.
  • Pickup uses the first-task rule for event-inbox backends: PROCESSING, or a change from the status read just before the send. Native activity after the send also counts.
  • The herdr TUI backend still skips the confirmation, and tmux is unchanged.
  • A new test delivers to Claude Code (2.0 s paste delay) with the status stuck at COMPLETED and checks that Enter is sent again.

P3-1, docs. Agreed; I confirmed it against input.rs. Fixed in 1d58272. docs/herdr-app.md now uses your option (a) wording. The comment above INPUT_TIMEOUT_S says the same, plus the up-to-5 s wait for the agent to start after typing, which the 90 s timeout also has to cover.

Notes.

  • a2db0ff: BackendFactory refuses terminal.herdr_app_session = default with a ConfigurationError that says why (factory.py:91).
  • e0d7536: adds test/scripts/test_herdr_tui_to_app_self_test.py, which runs --self-test and asserts exit 0.

You were right about test_skips_when_processing. That failure came from my environment: CAO_EAGER_INBOX_DELIVERY was set in my shell. With it unset, the test passes, along with the backend, client, inbox, factory and script suites (283 tests). I'll correct the PR description.

@haofeif haofeif left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at 8cd84d9: Request changes (0 P1, 1 P2, 1 P3)

Thank you for the quick and careful fixes. All three earlier findings are fixed, and both notes are done. Each fix has a test that fails when I revert the fix.

The rebase onto 3ef0667 also added two commits that the reply does not list: 5068057 (sort tabs in creation order) and 1f13b97 (put each tab after the terminal that created it). These two commits cause one new P2 and one new P3:

  • P2-3: Two cao-servers that use the same session name sort the same tabs against each other, and the writes do not stop. docs/herdr-app.md:40 describes this setup as supported.
  • P3-2: The docs and the PR description do not tell users that CAO now sorts tabs.

Earlier findings

Finding Status Evidence
P2-1: all terminals share one input connection Fixed in cf9a46f _input(address) opens one HcRpcClient for each address (herdr_app_backend.py:162-172). All input goes through _input_call (:569-585). The connection closes in kill_address (:433), in the naming pass (_prune_inputs, :190-202), on a removed event (herdr_app_inbox_service.py:277-280), and at shutdown (api/main.py:1985). In herdr-core v0.1.21, a closed connection cancels a send that is still in its draft hold (server.rs:197-211, input.rs:330, :341), so nothing is typed later. When I put all input on one connection again, 6 tests fail, including test_held_send_does_not_delay_another_terminals_input.
P2-2: fixed 200 ms gap, and no resubmit for inbox deliveries Fixed in 2082dbe herdr-app returns True from event_inbox_confirms_submit() (herdr_app_backend.py:627). The base default is False (base.py:529). So herdr-app inbox deliveries run the submit confirmation (inbox_service.py:277-293, :337). Pickup counts PROCESSING, a change from the status read before the send, or native activity after the send (inbox_service.py:560-581, status_monitor.py:1169-1210). When I force native_confirm off, 2 tests fail. When I make the pickup rule always true, test_swallowed_enter_is_resent_for_claude_code fails. The herdr TUI backend still skips the confirmation (test_herdr_tui_keeps_skipping_confirmation).
P3-1: "Sends wait" docs Fixed in c7d36dd docs/herdr-app.md:49 and the comment above INPUT_TIMEOUT_S agree with herdr-core: HC_QUIET_MS 3000 (input.rs:267-268), HC_DRAFT_MS 60000 (:271-272), STARTUP_MS 10 s (:17), SETTLE_MAX 5 s (:19), OUTPUT_SETTLE 800 ms (:15).
Note: the default session Done in ea0723c factory.py:91-97 raises ConfigurationError before it connects. Two tests cover it.
Note: pytest does not run --self-test Done in 8cd84d9 test/scripts/test_herdr_tui_to_app_self_test.py runs it in a subprocess. Unit Tests runs test/ (ci.yml:76), so CI runs it.

Other reviewers

Claim Source Verdict
"All three findings and both notes are addressed in 3d98c65..e0d7536" Author reply Confirmed. These SHAs are from before the rebase. At the head, the same changes are cf9a46f, 2082dbe, c7d36dd, ea0723c and 8cd84d9. The reply does not list 5068057 and 1f13b97 (see P2-3 and P3-2).
"HcRpcClient.close() no longer waits for a held call" Author reply Confirmed. close() does not take the call lock. It ends the hc connect child, so a held call gets EOF (hc_rpc.py:240-256).
"A new test holds a send on terminal A and checks that keys and a send to terminal B still complete" Author reply Confirmed. The test fails when all input shares one connection.
"N + 2 hc connect children" Author reply, docs/herdr-app.md:41 Confirmed from the code. I did not measure the 2.4 MB for each child.
"hc 0.1.21 has no per-send gap … paste_gap_ms comes only from template files" Author reply Confirmed. terminal.send takes caller, force, if, terminal, text, timeout and wait (api/types.rs:480-504). Only a template sets paste_gap_ms (input.rs:464).
"Pickup uses the first-task rule … Native activity after the send also counts" Author reply Confirmed (inbox_service.py:560-581, herdr_app_inbox_service.py:288-292).
"The herdr TUI backend still skips the confirmation, and tmux is unchanged" Author reply, PR description Confirmed. The new path needs both supports_event_inbox() and event_inbox_confirms_submit(). Only herdr-app returns True for the second one.
"A new test delivers to Claude Code (2.0 s paste delay) … checks that Enter is sent again" Author reply Confirmed (see the P2-2 mutation above).
"BackendFactory refuses … default" and "e0d75367 adds … test_herdr_tui_to_app_self_test.py" Author reply Confirmed.
"I'll correct the PR description" Author reply Confirmed. The description now says 283 tests, with CAO_EAGER_INBOX_DELIVERY unset.
"Each terminal gets its own input connection … Inbox deliveries confirm the submit and resend Enter" PR description Confirmed.
"Space and tab names … CAO sets them with one version-checked write" PR description, "Placement" Partly. At 8cd84d9, the same write also sorts the space's tabs, and it runs again on every layout doc event. The description does not say this (P3-2). It causes P2-3.
"283 tests … The e2e suite … 3 passed. mypy: no new errors" PR description Consistent. The targeted suites pass here (339 tests). The blocking mypy ratchet in CI passes. I did not run the e2e suite, because it needs hc, and CI does not run it.
Live runs with mock_cli and kiro-cli, and the manual test with the app PR description Not reproduced. These are the author's results. The description does not give their head, so they can be from before cf9a46f and 2082dbe. See Notes.
"Patch coverage is 88.62559% with 96 lines in your changes missing coverage" Codecov Noted. The report has no BASE upload, so it shows no project delta. The tab-order tests use one server only, so they do not find P2-3.

Findings

P2-3: Two cao-servers that use the same session name sort the same tabs against each other, and the writes do not stop (herdr_app_backend.py:327-413; herdr_app_inbox_service.py:246-249)

  • The docs support this setup: "Several cao-servers with different homes can share one core … Two servers that use the same session name share that session's space in the app" (docs/herdr-app.md:40). The space id comes from the session name only (space_id_for, herdr_app_backend.py:80-82). Two homes get the same session name with cao launch --session-name <name>, or with the same idempotency key (utils/orchestration.py:1304).
  • sync_layout_names ranks tabs only by this server's terminals. A tab with no terminal of this server gets the rank len(rank), so it goes after this server's tabs (:366-413). Each server sees only the terminals with its own cao.home, and each server has its own database. So server A puts its tabs first, and server B puts its tabs first.
  • Each sort that changes the order is a doc.set. herdr-core v0.1.21 saves docs.json and sends a doc event to every unfiltered subscriber (api/doc.rs:24-37, events.rs:127-134). CAO subscribes unfiltered (hc_rpc.py:265). Every layout doc event runs sync_layout_names again (herdr_app_inbox_service.py:246-249). No check stops a server when the other server changed the order.
  • Result: a write by A makes B write, and a write by B makes A write. This continues while both servers have terminals in the space. Each write saves the document to disk and sends an event to every client of the core. herdr-app replaces its model with each new version (herdr-app 317e673, src/window.rs:2141-2181). So the tabs change order on the screen again and again, and a local layout edit that is not saved yet gets "Layout changed elsewhere" (window.rs:2181).
  • I reproduced it with one FakeHc and two real HerdrAppBackend objects. They have different cao.home values, one session name, two terminals each, and a registry for each server:
    • I ran sync_layout_names 6 times, alternating A and B. Each pass wrote once, and the order changed each time between [t1, t2, t3, t4] and [t3, t4, t1, t2]. With the sort off (_creation_order returns {}), the same probe wrote twice (the names), and then it stopped.
    • I connected each doc.set to a doc event that goes to both real HerdrAppInboxService.handle() methods. After 50 events, there were 100 writes and 51 events in the queue.
  • Options:
    • (a) Sort this server's tabs only among the positions that they already have. Tabs that this server does not know stay where they are. Then the servers do not move each other's tabs, and each pass gets to a stable order. One server alone keeps the full feature.
    • (b) Do not sort when the space has a tab that this server does not know. This is simple. But a shared space, or a space with a person's own tab, gets no order.
    • (c) Sort only when a new tab of this server first appears in the layout (when the app adopts a new terminal), not on every doc event. This writes less, and a person's drag stays. Two servers can still move each other's tabs, but only one time for each new tab.
    • For each option, add a test: two backends with different cao.home values on one FakeHc and one session name. Alternate sync_layout_names and make sure that the writes stop.

P3-2: The docs and the PR description do not say that CAO sorts tabs (docs/herdr-app.md:37-50; PR description, "Placement")

  • 5068057 and 1f13b97 sort the session's tabs on every naming pass. The order is creation order, and each tab goes after the tab of the terminal that created it (caller_id, herdr_app_backend.py:327-364). The naming pass runs on every layout doc event.
  • A user sees these effects, and nothing in the docs tells them:
    • A person drags a CAO tab to a new position. The drag is a layout write, so the next naming pass moves the tab back.
    • A person's own tab in a CAO space moves after all CAO tabs.
  • The PR description says that the layout write sets names, and that placement uses labels. The reply does not list these commits.
  • Options: Add a "Tab order" item to "How CAO uses the core" in docs/herdr-app.md, and one line to the PR description. If a person's drag must stay, option (c) of P2-3 also does that.

What I checked

  • The first 19 commits are patch-identical to the commits that I reviewed at 9224f3e. I read the 7 new commits and their tests.
  • The targeted suites for the changed files at 8cd84d9: 339 passed.
  • Mutations. Each one made the new tests fail:
    • One shared input connection: 6 failures.
    • native_confirm forced off: 2 failures.
    • No tab sort: 3 failures.
    • caller_id ignored: test_grouped_by_parent fails.
    • Pickup rule always true: test_swallowed_enter_is_resent_for_claude_code fails.
  • The upstream source for each claim: herdr-core v0.1.21 (server.rs, input.rs, api/types.rs, api/doc.rs, events.rs, terminals.rs) and herdr-app 317e673 (src/model.rs, src/window.rs).
  • The two probes for P2-3 (above).
  • git merge-tree with main (b2b3666): no conflicts. Since 3ef0667, main has one new commit, b2b3666 (#1399, codex.py). It does not touch these files.
  • CI at 8cd84d9: 27 of 27 checks pass.

Notes

  • Multi-agent flow on herdr-app. The e2e suite covers the session lifecycle, history depth, and inbox delivery when the agent is idle. The live runs in the description use one worker. cf9a46f and 2082dbe change the input and inbox paths that a multi-agent flow uses. Before merge, please run examples/assign with terminal.backend: herdr-app at the final head, and confirm these results:

    1. The supervisor assigns 3 tasks to data analysts in parallel, and then hands off to the report generator in sequence.
    2. Each data analyst completes its task, and its message gets back to the supervisor.
    3. The supervisor completes the report. It does not do the analysts' tasks itself.

    This is the same check that new providers must pass (step 9 of the cao-provider skill). It is a request for evidence, not a finding.

  • Branch state. The branch is one commit behind main. The merge has no conflicts.

  • Delivery time. One inbox delivery can now keep its worker thread for up to 3 × 8 s (INBOX_SUBMIT_CONFIRM_TIMEOUT, INBOX_SUBMIT_MAX_RESUBMITS; constants.py:386, :391). The deliveries and the event reader use the same default asyncio executor. So many slow deliveries at the same time can delay events. Pickup also reads status on the request connection, so it does not need those events. This is not a finding.

🤖 Reviewed with CAO review workflows

terminal.send and terminal.keys can wait up to about 60 s for a person's
unfinished draft. HcRpcClient serializes exchanges on one connection, so
every status lookup and terminal.list queued behind such a send. Input now
goes through a second HcRpcClient for the same session, opened on first
use. An injected client (tests) doubles as the input client unless
input_client= is given.

Assisted-by: AI
Every terminal the backend creates now carries cao.home, the first 16
hex of the SHA-256 of the resolved CAO_HOME_DIR, and every lookup
(sessions, windows, exact cleanup, list_managed, naming) filters on it.
Two cao-servers with different homes sharing one core no longer see,
reconcile, or kill each other's terminals.

Space ids depend only on the session name, so two servers using the
same session name share a space; kill_session now leaves the space in
the layout document while any live terminal still carries it.

Assisted-by: AI
HerdrAppInboxService follows the CAO core's event stream on its own
hc connect: delivers pending messages on idle/done, notes native
activity on working/blocked, records each mapped status change once,
tears down CAO terminals on finished/lost, kills shells that replaced
a torn-down agent (restarted), names spaces and tabs on herdr-app doc
events, rebuilds its map on missed, and reconnects with 1-30 s backoff
on EOF or 45 s of silence, reconciling rows and terminals under the
session lifecycle lock after each connect.

Changes from the plan:
- Only terminals with this server's cao.home label are mapped,
  restarted-killed or reconciled; another server's are ignored.
- After teardown on finished/lost the terminal is killed and removed
  by address: it is no longer live, so kill_window's label lookup
  cannot find it and its record would otherwise linger in the core.
- A failed teardown, removal, row drop or orphan kill is logged and
  does not end the stream (which would reconnect and re-reconcile).

Assisted-by: AI
Delivery types through terminal.send, which can wait up to about 60 s
for a person's unfinished draft. Awaiting it inline held every other
terminal's events (status, finished, doc) for that long. Each delivery
now runs as its own task, at most one in flight per terminal; an idle or
done that arrives meanwhile triggers one rerun when it ends. Shutdown
cancels deliveries still in flight.

Assisted-by: AI
…--terminal

- herdr_inbox_registry types its singleton as a runtime-checkable
  EventInboxService protocol (register_terminal, unregister_terminal),
  so terminal_service's call sites type-check for both services and the
  module still imports neither.
- The lifespan starts HerdrAppInboxService for HerdrAppBackend, with the
  same inbox_service.deliver_pending callback as herdr; the existing
  shutdown block cancels and unregisters it.
- /health reports terminal_backend "herdr-app" and a "herdr-core"
  component (hc or herdr-core on PATH).
- cao-server --terminal accepts herdr-app.

Audit of herdr-specific branches outside the backends:
- terminal_service.py:1746 isinstance(HerdrBackend) Kiro home path
  refusal: herdr CLI argument quoting only; herdr-app passes env as
  JSON. No change.
- terminal_service.py:3277 FIFO setup skipped for event-inbox
  backends: herdr-app pipe_pane is a no-op. Correct as is.
- terminal_service.py:3477 and :7019 register/unregister through
  get_herdr_inbox_service(): works for both via the protocol; pane_id
  is the hc address. No change.
- terminal_service.py:5350, :5415 and api/main.py:5647 event-inbox
  dispatch recency and boundary: herdr-app has no output generation
  either. Correct as is.
- inbox_service.py:322 skips submit confirmation for event-inbox
  backends because herdr delivers on the event loop. herdr-app delivers
  in a worker thread, so confirmation could run there; left skipped
  (same behavior as herdr), follow-up.
- status_monitor.py:1135, :1182, :1431, providers/base.py:698,
  utils/terminal.py:153: generic supports_event_inbox() paths using
  get_native_status/get_history. Correct as is.
- session_service.py:631 and terminal_service.py:858: docstrings about
  HerdrBackend's unconfirmed kill_session and lenient strict check;
  HerdrAppBackend confirms kill_session with a fresh lookup and its
  session_exists_strict raises. No change.

Assisted-by: AI
The reconciler deleted every CAO row in a session that had no live
herdr-core terminal. An operator switching from the tmux or herdr TUI
backend to herdr-app would lose all their existing CAO rows on the
first connect, because this core has never heard of those terminals.

A row is now dropped only when the core has at least one record for
its cao.terminal_id under this server's cao.home (terminal.list with
all: true, so finished records count) and every such record is in
state "finished". A row with no record is another backend's and stays;
a record in a state other than "finished" keeps the row too. The
orphan side (live terminal with our labels and no row) is unchanged,
and a failed lookup still skips the session.

HerdrAppBackend.list_managed takes include_finished to ask for the
finished records.

Assisted-by: AI
…erdr-core

Each test starts its own core: a fresh HC_SESSION and XDG runtime,
state and config dirs under a short /tmp scratch dir (unix socket paths
are limited to 108 bytes, too short for pytest's tmp_path).

- Lifecycle: two terminals share one herdr.space and carry this
  server's cao.home; "echo hi-$((40+2))" proves the shell ran the
  command (hi-42), not just echoed it; kill_session leaves nothing live.
- Depth: terminal.read with lines=20000 returned 2994 rows on hc 0.1.21
  (history plus the 120-row screen; the cap counts rows, so wrapped
  lines count more than once). HC_READ_MAX_LINES goes from 1000 to
  2000, N rounded down to a multiple of 1000, and the test asserts the
  core can return at least that many.
- Inbox: HerdrAppInboxService runs against the core with its database
  seams patched; a terminal printing OSC 7501 state=working then
  state=idle drives PROCESSING then IDLE and the delivery callback.

Assisted-by: AI
… script

docs/herdr-app.md covers prerequisites (herdr-core 0.1.21+), enabling
the backend (config, env, --terminal) and its session setting, showing
the core in the app (hc machine add), /health's herdr-core component,
the labels CAO sets, cao.home (several cao-servers with different homes
share one core; same session names share a space), the three hc
connect children per server, reconcile leaving other backends' rows
alone, keys typed without Enter (what terminal.keys accepts, checked on
hc 0.1.21: space maps to "space", shifted punctuation is rejected),
sends waiting for ~3 s quiet and a person's draft, the 2000-line read
cap, removing the app's machine entry before stopping the core, and a
"Migrating from the herdr TUI" section.

configuration.md lists herdr-app and terminal.herdr_app_session;
herdr.md points app users at the new page.

scripts/herdr_tui_to_app.py, the one-time herdr TUI to Herdr app
migration the new section describes, is added (black and isort
applied; --self-test passes).

Assisted-by: AI
After a native idle or done, providers/base.py reports PROCESSING for a
10 s buffer flush wait. The delivery the status event triggers runs
inside that window, finds the terminal busy, and leaves the message to
the 30 s inbox reconciliation sweep. Measured against a live cao-server
with a mock_cli terminal: 36 s from native idle to delivery.

On idle or done the service now also schedules one more delivery
SETTLE_RETRY_S (10.5 s) later. working, blocked, finished, lost and
removed cancel it; a later idle or done restarts it, so there is never
more than one per terminal. deliver_pending re-reads pending rows under
its per-terminal lock, so an extra call with nothing pending is free.
Same live check with the retry: 10.6 s.

Assisted-by: AI
The include_finished flag on _list made every **labels call site an
arg-type error (8 new mypy errors). list_managed now issues the
terminal.list with all: true itself and _list is unchanged.

Assisted-by: AI
…te lint

test_timeout_literals_are_derived_or_justified flagged wait(timeout=5)
and run(timeout=30) in clients/hc_rpc.py. They become
CONNECT_REAP_TIMEOUT_S and SERVER_START_TIMEOUT_S, each with the bound
it is chosen against.

Assisted-by: AI
A cao-server started inside a Herdr app terminal inherits HC_TERMINAL_ID,
HC_TERMINAL_SOCKET and HC_SCOPE, and an operator may set HC_SOCKET, which
herdr-core's docs/security.md says overrides the client endpoint. The browser
attach argv kept all four (the PTY inherits os.environ), so it attached to the
user's own core, and HC_SOCKET was stripped nowhere.

- hc_rpc strips HC_SOCKET too, for hc connect and hc server start (which now
  also gets stdin=/dev/null).
- prepare_web_attach returns `env -u ... HC_SESSION=<s> <hc> attach <addr>`;
  attach_session execs the resolved hc with the same scrubbed environment.
  Both use hc_binary() instead of a bare "hc".
- HcSubscription closes its hc connect child when the subscribe write, read
  or reply fails; the caller never gets the object, so nothing else could.
- The TUI migration script strips HC_SOCKET as well.

Tests cover all three spawn paths with the four variables set, run the web
attach argv against a stand-in hc that prints its environment, and check the
subscription's child is reaped and that events.subscribe carries no filter.

Assisted-by: AI
Live with kiro-cli 3.3.0, herdr-core kept a terminal at `error` after a failed
turn while the agent was back at its prompt. HerdrAppBackend mapped that to
TerminalStatus.ERROR, and terminal_service._refuse_if_provider_gone then
refused every later input to the terminal.

`error` is no longer mapped, as on the herdr TUI backend: get_native_status
returns None and the provider's buffer analysis decides. The inbox shares the
map, so an `error` event is neither recorded as a transition nor delivered on;
the next `idle` or `done` records and delivers as before.

docs/herdr-app.md now says a native `error` is ignored and that inbox delivery
lands about 10 s after a turn ends (CAO's completion wait).

Assisted-by: AI
…s own deletes

- register_terminal / unregister_terminal run on worker threads (the API runs
  delete_terminal in asyncio.to_thread) but cancelled loop tasks and mutated
  the maps the loop iterates. The address and last-state maps now sit behind a
  threading.Lock and change before the call returns; a pending retry is
  cancelled on the loop (call_soon_threadsafe off it). start() records the
  loop.
- Cancelling the service in the reconnect backoff skipped cancelling pending
  deliveries and retries. start() now cancels them on any CancelledError.
- On `missed` the service also reconciles: a terminal that finished in the
  gap is in no snapshot and its `finished` never arrived, so its row stayed.
- delete_session tore each terminal down twice: its kill_session produces a
  `finished` event and the inbox ran delete_terminal again ("Failed to
  snapshot terminal ... no live herdr-core terminal"). _teardown now takes the
  session's lifecycle lock, which session teardown holds from kill_session
  until its rows are gone, and skips a row that is no longer there. A
  single-terminal delete unregisters before it kills, and the map change is
  now synchronous, so its event is never mapped.

Assisted-by: AI
- FakeHc can keep a terminal alive through terminal.kill and terminal.rm:
  kill_session then returns False and keeps the space, and
  cleanup_terminal_exact reports STILL_PRESENT, never DELETED.
- Reconcile keeps a row whose records are one finished and one in a state
  CAO does not know.
- Naming leaves the user's own space, and the user's tab inside a CAO space,
  untouched while naming CAO's; a space the app placed before the CAO tab is
  named on its own. The spec now states that rule (the inbox naming pass's)
  instead of "write nothing when the space or tab is missing", which the code
  never did for a missing tab.
- The event stream is read with the 45 s silence limit.

Each new test fails against a mutation that drops the check it covers.

Assisted-by: AI
The app adopts a labelled terminal with Model::add_tab, which inserts
the tab after the space's selected tab. The conductor is usually
selected, so every new worker landed right after it and the newest
worker ended up leftmost.

The naming pass now also sorts each CAO space's tabs by creation
order, taken from the registry's oldest-first terminal order
(list_terminals_by_session, rowid order). Tabs CAO does not own go
last, and a registry read failure skips ordering for that pass. The
cost is a brief jump after each adoption, and a manual drag of a CAO
tab is undone, as renames already are. Asking herdr-app to adopt with
add_tab_unselected (append, keep selection) would remove the jump.

Verified: new tab-order tests fail without the change and pass with
it; herdr-app backend and inbox suites pass; mypy clean on the file.

Assisted-by: AI
With creation order alone, a worker a supervisor spawns lands at the
far right, away from its supervisor. Tabs now follow the delegation
tree: each terminal comes right after its parent's group (the
terminal recorded as its caller_id), siblings in creation order. A
terminal whose parent is not in the session, such as one created from
the CLI or the operator API, is a top-level entry in creation order,
so sessions without caller data keep the flat order.

caller_id is read per terminal with get_terminal_metadata rather than
added to list_terminals_by_session, whose rows also back API
responses.

Verified: the grouped test fails on the creation-order-only code and
passes with this change; herdr-app backend and inbox suites pass;
mypy clean on the file.

Assisted-by: AI
All terminals shared one input HcRpcClient, and herdr-core answers a
plain `hc connect` one request at a time (server.rs connection). It
holds terminal.send while a person has a draft in that terminal, up
to 60 s after their last key (input.rs type_in), so one draft stopped
CAO input to every other terminal: inbox deliveries, send_input,
startup-dialog keys and typed launch commands.

Input now opens one HcRpcClient per terminal address on first use.
The core already queues input per terminal, so this is the same
concurrency framed mode would give, without a second wire protocol in
CAO. A connection closes when CAO kills its terminal (kill_address,
so kill_window, kill_session and cleanup_terminal_exact), when the
terminal is removed outside CAO, when the naming pass's fresh listing
no longer shows it, and at server shutdown. HcRpcClient.close() no
longer waits for a call in flight: it ends the child, the held call
fails as lost, and the core cancels a send still in its draft hold.
A closed client never reconnects.

Trade-off: one `hc connect` child per terminal CAO typed into (N + 2
in total, about 2.4 MB each), documented in docs/herdr-app.md.

Verified: the new test holds terminal.send on terminal A and shows
keys and a send to B complete meanwhile; it fails when every address
shares one connection. Live on a scratch core (cao-dev-*): 3
terminals typed into gave 3 input children, killing one left 2,
close_inputs left none. Backend, client and inbox suites pass; mypy
at baseline (122).

Assisted-by: AI
…fter 200 ms

herdr-core's terminal.send pastes, waits PASTE_GAP (200 ms) and
presses Enter (input.rs send_message). The only override is a
template's paste_gap_ms, read from ~/.config/herdr-core/templates
files that are shared with the user's own sessions; terminal.send has
no gap parameter (hc 0.1.21 schema: caller, force, if, terminal,
text, timeout, wait). So send_keys cannot honor submit_delay, and
Claude Code asks for 2.0 s because an early Enter is swallowed. Inbox
deliveries on herdr-app also skipped _confirm_submitted, because
supports_event_inbox() is true, so a swallowed Enter stayed in the
prompt box until a person pressed Enter.

That skip exists because the herdr TUI inbox service delivers on its
event loop. HerdrAppInboxService already delivers via
asyncio.to_thread, so a new backend capability,
event_inbox_confirms_submit() (default False, True on herdr-app),
lets herdr-app run the confirmation. Event-inbox status has no output
generation, so pickup uses the first-task path's rule for these
backends (_wait_for_post_dispatch_start): a started status counts if
it is PROCESSING or differs from the status read just before the
dispatch, kept across resubmits. Native activity after the send
(started_on_last_input) also counts, because terminal.send waits up
to 5 s for the agent to start, so a short turn can be over before the
wait begins. Recovery is the existing helper (bare Enter when the
text is still in the composer). tmux and the herdr TUI are unchanged.

Trade-off: a herdr-app delivery can hold its worker thread and the
receiver's delivery lock for up to 8 s x 3 while unconfirmed, as on
tmux. Option (a) was not done: no supported per-send or per-terminal
gap exists without writing template files into the user's herdr-core
config.

Verified: new tests deliver to a ClaudeCodeProvider
(paste_submit_delay 2.0) on a HerdrAppBackend whose status stays
COMPLETED and see Enter resent; pickup by PROCESSING, by a
transition, or by native activity sends nothing; the herdr TUI keeps
the base False. Two of them fail without the inbox change.
test/services and test/backends pass except
test_skips_when_processing, which fails the same way without this
change (unpatched provider lookup on the eager path). mypy at
baseline (122).

Assisted-by: AI
docs/herdr-app.md said a send waits for about 3 s of output quiet, and
the INPUT_TIMEOUT_S comment said calls wait for output to settle. In
herdr-core 0.1.21 the 3 s is measured from a person's last key in that
terminal (HC_QUIET_MS) and applies only if someone typed
(input.rs Gate::wait). terminal.send also waits for their draft, up
to 60 s after that key. The output wait (800 ms quiet, at most 5 s,
then `unavailable` with nothing typed) applies only during a
terminal's first 10 s (input.rs settle).

The docs now use the reviewer's wording. The comment states the same
waits, plus the up-to-5 s wait for the agent to start after typing,
which is part of what the 90 s timeout must cover.

Verified: wording checked against input.rs at v0.1.21 (quiet,
default_draft_timeout, Gate::wait, settle, STARTUP_MS, SETTLE_MAX,
started_window) and the terminal.send and terminal.keys descriptions
in `hc schema`. Docs and comment only.

Assisted-by: AI
docs/herdr-app.md says CAO never uses your `default` session, the one
the Herdr app opens for your own terminals, but nothing enforced it:
terminal.herdr_app_session (or CAO_HERDR_APP_SESSION) set to
"default" made cao-server start that core, write its app layout and
run reconcile and kills there. Those only touch CAO-labelled
terminals, so the risk was low, but the docs promised more.

BackendFactory now raises ConfigurationError for "default", before
any core is started or contacted, and the message says why and
suggests "cao". No other name is refused.

Verified: new factory tests (env and settings file) fail without the
check and pass with it; test/backends/test_factory.py passes.

Assisted-by: AI
scripts/herdr_tui_to_app.py has assert-based offline checks behind
--self-test, but neither CI nor pytest ran them, so they could go
stale unnoticed.

A pytest wrapper now runs `python3 scripts/herdr_tui_to_app.py
--self-test` in a subprocess with the test interpreter and asserts
exit 0 and the "all checks passed" line. It needs no herdr or hc.

Verified: the wrapper passes; the whole run takes under a second.

Assisted-by: AI
Two cao-servers with different homes can share a space when they use
the same session name. The tab sort ranked a server's own tabs first
and put every tab it did not own after them. Each server therefore
reordered the other's tabs, and every resulting layout write woke the
other server's naming pass: the order flipped and the document was
rewritten without end.

The sort now permutes a server's own tabs only among the positions
they already hold. Tabs it does not own, a person's own tab or the
other server's, keep their place, so each server reaches a fixed point
and stops writing. A single server still gets the full ordering.

The docs now describe the tab order, including that dragging a CAO tab
is undone on the next layout change.

Verified: the two-server test fails on the previous sort and passes
with this change; the herdr-app backend, inbox and docs-link suites
pass; mypy clean on the file.

Assisted-by: AI
@anilkmr-a2z
anilkmr-a2z force-pushed the feat/herdr-app-backend branch from 8cd84d9 to dfff087 Compare November 19, 2026 05:18
@anilkmr-a2z

Copy link
Copy Markdown
Contributor Author

Thanks for the second review. The P2-3 loop is real, and it is fixed in dfff087 (the branch is also rebased onto main at b2b3666).

P2-3, two servers undoing each other. I took your option (a), in a slightly stricter form. sync_layout_names now reorders this server's tabs only among the positions they already hold (herdr_app_backend.py, the slots block in the naming pass). A tab it does not own, whether a person's own tab or another cao-server's, keeps its position. Each server therefore reaches a fixed point and stops writing, and a single server still gets the full ordering. I did not take (b) or (c) because they give up the ordering in shared spaces, or leave one-time moves. TestTwoServersOneSpace puts two backends with different cao.home values on one FakeHc and one session name, with their tabs interleaved and reversed. It alternates the naming pass and asserts that writes stop. It fails on the previous sort and passes now. test_unknown_tab_goes_last... changed to match: a tab CAO does not own now keeps its slot instead of going last.

P3-2, docs. docs/herdr-app.md has a new "Tab order" item under "How CAO uses the core". It covers the ordering rule, that only CAO's own tabs move, and that dragging a CAO tab is undone on the next layout change. The PR description has the same note.

For reference, the two commits you flagged are 5068057 (creation order) and 1f13b97 (group under the creator). My earlier reply only covered the review fixes. On the branch behind main: it was rebased just now.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feat] herdr-app backend: show CAO sessions in the Herdr desktop app (herdr-core)

3 participants