Repository navigation
feat: herdr-app backend that shows CAO sessions in the Herdr desktop app - #1356
anilkmr-a2z wants to merge 27 commits into
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #1356 +/- ##
=======================================
Coverage ? 91.92%
=======================================
Files ? 299
Lines ? 57923
Branches ? 9520
=======================================
Hits ? 53245
Misses ? 3291
Partials ? 1387
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Integrate awslabs#1356 at 9224f3e. Preserve the profile-env support from awslabs#665 by forwarding trusted_env through HerdrAppBackend and applying the shared environment and runtime identity policy. Validation: 711 focused tests pass, including profile environment regression tests; migration self-test, Ruff, Black, and isort checks pass. Type checking has the same 117 diagnostics as the prior branch in the local environment, with no new per-file/error-code counts. Live core and graphical checks require desktop/socket access and were not performed.
haofeif
left a comment
There was a problem hiding this comment.
Review at 9224f3e: Request changes (0 P1, 2 P2, 1 P3)
Thank you for this backend. Its design is careful:
- Every
hcchild is started without theHC_*routing variables. - Layout writes are version-checked and retried, and they change only the spaces and tabs that CAO owns.
- The
cao.homelabel keeps each cao-server to its own terminals. - Reconcile drops only rows whose core record has finished.
- The migration script only reads the TUI.
I found two P2 problems on the input path:
- All terminals share one input connection. So a send that waits for a person's draft in one terminal stops input to all other terminals.
- herdr-core waits a fixed 200 ms between the paste and Enter. CAO's per-provider submit delay is not used, and on this backend inbox deliveries have no resubmit.
There is also one P3 for the "Sends wait" docs.
Other reviewers
- Codecov (comment): "patch coverage 87.94%, 89 lines missing" (backend 87.79%, inbox service 88.97%,
hc_rpc.py84.56%): Confirmed. The report has no BASE upload, so it shows no project delta. The coverage gap is not the reason for my findings. The tests that exist do not check either P2 path. The input-connection test checks status lookups only, and no test checks the paste-to-Enter gap. - PR description claims:
- "Input goes over a separate connection, so a send waiting on a person's draft (up to 60 s) doesn't block status lookups": Partly. This is true for status lookups (
test_herdr_app_backend.py:430-478). But all terminals share the input connection, so a held send blocks input to every other terminal (P2-1). - "12033 passed, 1 failed …
test_skips_when_processing… fails the same way onmain": Not reproduced. The test passes locally at the PR head and onmain, and the Unit Tests job passes. The PR does not changeinbox_service.py, so the failure comes from the local environment, not from this PR. - "tmux and the herdr TUI backend are unchanged": Confirmed. The shared changes are a new
factory.pybranch, a new lifespan branch (api/main.py), a structuralEventInboxServicetype inherdr_inbox_registry.py, and one constant. One small visible change:/healthnow always lists aherdr-corecomponent, in the same way that it already listsherdr. The web card showsunavailableas "not installed". - "Live with kiro-cli 3.3.0 … delayed inbox delivery": Confirmed as stated. Claude Code was not part of the live tests, so those tests did not cover P2-2.
- "Input goes over a separate connection, so a send waiting on a person's draft (up to 60 s) doesn't block status lookups": Partly. This is true for status lookups (
Findings
P2-1: All terminals share one input connection, so a send that waits for a draft in one terminal stops input to all other terminals (herdr_app_backend.py:113-118, :148-156; clients/hc_rpc.py:178-205)
_input()opens oneHcRpcClientfor the whole backend._input_callsends allterminal.sendandterminal.keyscalls through it withINPUT_TIMEOUT_S= 90 s (:47).HcRpcClient.callholdsself._lockwhile it writes the request and waits for the reply (hc_rpc.py:181-205). So the backend sends one input request at a time for all terminals.- herdr-core v0.1.21 holds
terminal.sendwhile a person has a draft in that terminal. The hold is up to 60 s after their last key (HC_DRAFT_MSordelivery.draft_wait), and each new key restarts it (crates/herdr-core/src/input.rs:81-89,:267-277,:314-370). A plainhc connectconnection runs one request at a time (server.rs:170-214). Inside the core, each terminal has its own queue, so only CAO's shared connection joins them. - Effect: a person types in one CAO terminal from the app, and CAO sends to that terminal (for example, an inbox delivery to the idle conductor). Then all CAO input to every other terminal stops: inbox deliveries,
send_input, startup-dialog keys (codex, kimi, antigravity), and typed launch commands (kiro, cursor, opencode, hermes, omp). The stop lasts up to 60 s after the person's last key, and up to 90 s if they keep typing. After 90 s, the first send fails with "herdr-core lost terminal.send", and the next caller continues. tmux has no such link between terminals. - I reproduced it with the real
HerdrAppBackendandHcRpcClientand a fakehc connectthat answers one request at a time, as hc plain mode does. The fake holdsterminal.sendto terminal A for 4 s.send_special_keyto terminal B waited 3.76 s. The result was the same on the PR head and on the merge withmain. - Options: (a) Open one input
HcRpcClientfor each terminal address, on first use, and close it inkill_addressand cleanup. (b) Use a client that matches replies by id, with framed mode (server.rs:374starts one task per request). Or keepterminal.keyson a shared connection and giveterminal.sendone connection for each terminal. In both cases, add a test: a send held on terminal A must not delay keys or a send to terminal B.
P2-2: herdr-app replaces each provider's paste-to-Enter delay with a fixed 200 ms, and inbox deliveries on herdr-app have no resubmit (herdr_app_backend.py:437, :452-453)
-
send_keysacceptssubmit_delaybut does not use it. It callsterminal.sendwithforce=True(:453). herdr-core writes the paste, waits for the terminal's templatepaste_gap_ms, elsePASTE_GAP= 200 ms, and then writes Enter (input.rs:11,:459-467). A terminal getspaste_gap_msonly from a template (terminals.rs:127-134,:306-308). CAO starts its terminals without a template (herdr_app_backend.py:193-211), so every CAO terminal gets 200 ms. -
CAO sets this delay for each provider because a short gap loses the submit:
- Claude Code: 2.0 s. "A too-early Enter is swallowed and the message sits unsubmitted in the prompt box" (
claude_code.py:1922-1928). - Kiro: 1.0 s (
kiro_cli.py:605-608). - OpenCode: 1.0 s (
opencode_cli.py:169-173). - Grok: 0.4 s.
- All other providers: 0.3 s (
backends/base.py:301).
send_inputpasses the provider's value withforce_bracketed_paste=True(terminal_service.py:6345-6351). tmux uses that value. The herdr TUI backend waits a fixed 2 s after each bracketed paste (herdr_backend.py:724-725). The base docstring lets a backend ignore the value only if it has no paste step (base.py:318-320). herdr-app has a paste step. - Claude Code: 2.0 s. "A too-early Enter is swallowed and the message sits unsubmitted in the prompt box" (
-
Inbox deliveries cannot recover:
HerdrAppBackend.supports_event_inbox()returnsTrue(:500-501). Sodeliver_pendingdoes not run_confirm_submitted, which is on by default (inbox_service.py:321-337,constants.py:380-382).- The reason for that skip is that the herdr TUI service delivers on the event loop. herdr-app already delivers in a worker thread (
herdr_app_inbox_service.py:324). - Only the first task to a new worker still gets
_confirm_worker_started_or_resubmit(terminal_service.py:5405-5417).
-
Effect: a callback from a worker to a Claude Code conductor can stay unsubmitted in its prompt box. The conductor does not start a turn, and the flow stops until a person presses Enter. I did not run Claude Code on herdr-app. This finding comes from CAO's own provider contract and from the herdr-core source.
-
Options: (a) Give herdr-core the provider's delay. For example, start each terminal from a CAO template that sets
paste_gap_ms, or ask herdr-core for a gap on each send. (b) Let herdr-app run the inbox submit confirmation, because its delivery is already off the event loop. Use the sameNoneboundary that the first-task path uses for event-inbox backends (terminal_service.py:5405-5416). In both cases, add a test: a provider withpaste_submit_delay2.0 must get at least that gap, or a swallowed Enter must be sent again.
P3-1: "Sends wait" does not describe what herdr-core waits for (docs/herdr-app.md:49; herdr_app_backend.py:45-46)
- The docs say: "Typing into a terminal waits for about 3 seconds of output quiet". The backend comment says that the calls wait "for output to settle before typing".
- herdr-core waits 3 s after a person's last key (
HC_QUIET_MS), and only if a person typed (input.rs:81-89,:267-268). The output wait (800 ms of quiet, at most 5 s) applies only during the first 10 s of a terminal. After 5 s it fails withunavailableand types nothing (input.rs:17-19,:486-513). - Options: (a) Change the docs to: "waits until 3 s after a person's last key, and for their unfinished draft (up to 60 s). During its first 10 s, a new terminal also waits for its output to settle." Change the comment in the same way. (b) Remove the "output quiet" part from both.
What I checked
- The full diff against
main(24 files, +4556/-17):- The backend, the
hcRPC client and the inbox service. - The lifespan and factory wiring, the registry type, and the docs.
- The migration script.
- The backend, the
- The herdr-core v0.1.21 source for each method that CAO calls:
terminal.run,send,keys,read,list,doc.set, events and the connection modes. - The input path, end to end, with a fake
hcprobe (P2-1). - The submit path from
send_inputto herdr-core, compared with tmux and the herdr TUI backend (P2-2). - The migration script:
- It only reads the TUI (
session list,api snapshot,pane process-info). - Its doc writes are version-checked.
- Its login wrapper quotes
"$@". --no-daemonis a real codex flag.--self-testpasses.
- It only reads the TUI (
send_keys(enter_count=0)callers send one character each (codex, antigravity, kimi), so the 64-character limit is safe.- Merge drift: the branch is behind
main, which moved only by test-only #1357.git merge-treeis clean, and the P2-1 probe gives the same result on the merged tree. - Overfitting: the constants are generic (timeouts,
HC_READ_MAX_LINESfrom a live measurement), and the status map follows the hc status words. The argument grammars in the script are part of rewriting CLI arguments, not special cases for one prompt. - CI: 27 checks pass at
9224f3e. Codecov is covered above.
Notes
- Neither CI nor pytest runs
scripts/herdr_tui_to_app.py --self-test. A small pytest wrapper would keep it from going stale. - "It never uses your
defaultsession" (docs/herdr-app.md:24): nothing refusesterminal.herdr_app_session default. Reconcile and kill touch only labelled terminals, so the risk is low. A check infactory.py, or a docs note, would make the statement true. force_bracketed_pasteis ignored on herdr-app. herdr-core chooses bracketed paste from the program's DEC mode 2004 (input.rs:465), so this is fine.
🤖 Reviewed with CAO review workflows
|
Thanks for the careful review. All three findings and both notes are addressed in 3d98c65..e0d7536. P2-1, shared input connection. Agreed, and fixed in 3d98c65. Input now uses one
P2-2, fixed paste-to-Enter gap. Agreed. hc 0.1.21 has no per-send gap:
P3-1, docs. Agreed; I confirmed it against Notes.
You were right about |
39ae84a to
8cd84d9
Compare
haofeif
left a comment
There was a problem hiding this comment.
Re-review at 8cd84d9: Request changes (0 P1, 1 P2, 1 P3)
Thank you for the quick and careful fixes. All three earlier findings are fixed, and both notes are done. Each fix has a test that fails when I revert the fix.
The rebase onto 3ef0667 also added two commits that the reply does not list: 5068057 (sort tabs in creation order) and 1f13b97 (put each tab after the terminal that created it). These two commits cause one new P2 and one new P3:
- P2-3: Two cao-servers that use the same session name sort the same tabs against each other, and the writes do not stop.
docs/herdr-app.md:40describes this setup as supported. - P3-2: The docs and the PR description do not tell users that CAO now sorts tabs.
Earlier findings
| Finding | Status | Evidence |
|---|---|---|
| P2-1: all terminals share one input connection | Fixed in cf9a46f |
_input(address) opens one HcRpcClient for each address (herdr_app_backend.py:162-172). All input goes through _input_call (:569-585). The connection closes in kill_address (:433), in the naming pass (_prune_inputs, :190-202), on a removed event (herdr_app_inbox_service.py:277-280), and at shutdown (api/main.py:1985). In herdr-core v0.1.21, a closed connection cancels a send that is still in its draft hold (server.rs:197-211, input.rs:330, :341), so nothing is typed later. When I put all input on one connection again, 6 tests fail, including test_held_send_does_not_delay_another_terminals_input. |
| P2-2: fixed 200 ms gap, and no resubmit for inbox deliveries | Fixed in 2082dbe |
herdr-app returns True from event_inbox_confirms_submit() (herdr_app_backend.py:627). The base default is False (base.py:529). So herdr-app inbox deliveries run the submit confirmation (inbox_service.py:277-293, :337). Pickup counts PROCESSING, a change from the status read before the send, or native activity after the send (inbox_service.py:560-581, status_monitor.py:1169-1210). When I force native_confirm off, 2 tests fail. When I make the pickup rule always true, test_swallowed_enter_is_resent_for_claude_code fails. The herdr TUI backend still skips the confirmation (test_herdr_tui_keeps_skipping_confirmation). |
| P3-1: "Sends wait" docs | Fixed in c7d36dd |
docs/herdr-app.md:49 and the comment above INPUT_TIMEOUT_S agree with herdr-core: HC_QUIET_MS 3000 (input.rs:267-268), HC_DRAFT_MS 60000 (:271-272), STARTUP_MS 10 s (:17), SETTLE_MAX 5 s (:19), OUTPUT_SETTLE 800 ms (:15). |
Note: the default session |
Done in ea0723c |
factory.py:91-97 raises ConfigurationError before it connects. Two tests cover it. |
Note: pytest does not run --self-test |
Done in 8cd84d9 |
test/scripts/test_herdr_tui_to_app_self_test.py runs it in a subprocess. Unit Tests runs test/ (ci.yml:76), so CI runs it. |
Other reviewers
| Claim | Source | Verdict |
|---|---|---|
| "All three findings and both notes are addressed in 3d98c65..e0d7536" | Author reply | Confirmed. These SHAs are from before the rebase. At the head, the same changes are cf9a46f, 2082dbe, c7d36dd, ea0723c and 8cd84d9. The reply does not list 5068057 and 1f13b97 (see P2-3 and P3-2). |
"HcRpcClient.close() no longer waits for a held call" |
Author reply | Confirmed. close() does not take the call lock. It ends the hc connect child, so a held call gets EOF (hc_rpc.py:240-256). |
| "A new test holds a send on terminal A and checks that keys and a send to terminal B still complete" | Author reply | Confirmed. The test fails when all input shares one connection. |
"N + 2 hc connect children" |
Author reply, docs/herdr-app.md:41 |
Confirmed from the code. I did not measure the 2.4 MB for each child. |
"hc 0.1.21 has no per-send gap … paste_gap_ms comes only from template files" |
Author reply | Confirmed. terminal.send takes caller, force, if, terminal, text, timeout and wait (api/types.rs:480-504). Only a template sets paste_gap_ms (input.rs:464). |
| "Pickup uses the first-task rule … Native activity after the send also counts" | Author reply | Confirmed (inbox_service.py:560-581, herdr_app_inbox_service.py:288-292). |
| "The herdr TUI backend still skips the confirmation, and tmux is unchanged" | Author reply, PR description | Confirmed. The new path needs both supports_event_inbox() and event_inbox_confirms_submit(). Only herdr-app returns True for the second one. |
| "A new test delivers to Claude Code (2.0 s paste delay) … checks that Enter is sent again" | Author reply | Confirmed (see the P2-2 mutation above). |
"BackendFactory refuses … default" and "e0d75367 adds … test_herdr_tui_to_app_self_test.py" |
Author reply | Confirmed. |
| "I'll correct the PR description" | Author reply | Confirmed. The description now says 283 tests, with CAO_EAGER_INBOX_DELIVERY unset. |
| "Each terminal gets its own input connection … Inbox deliveries confirm the submit and resend Enter" | PR description | Confirmed. |
| "Space and tab names … CAO sets them with one version-checked write" | PR description, "Placement" | Partly. At 8cd84d9, the same write also sorts the space's tabs, and it runs again on every layout doc event. The description does not say this (P3-2). It causes P2-3. |
| "283 tests … The e2e suite … 3 passed. mypy: no new errors" | PR description | Consistent. The targeted suites pass here (339 tests). The blocking mypy ratchet in CI passes. I did not run the e2e suite, because it needs hc, and CI does not run it. |
Live runs with mock_cli and kiro-cli, and the manual test with the app |
PR description | Not reproduced. These are the author's results. The description does not give their head, so they can be from before cf9a46f and 2082dbe. See Notes. |
"Patch coverage is 88.62559% with 96 lines in your changes missing coverage" |
Codecov | Noted. The report has no BASE upload, so it shows no project delta. The tab-order tests use one server only, so they do not find P2-3. |
Findings
P2-3: Two cao-servers that use the same session name sort the same tabs against each other, and the writes do not stop (herdr_app_backend.py:327-413; herdr_app_inbox_service.py:246-249)
- The docs support this setup: "Several cao-servers with different homes can share one core … Two servers that use the same session name share that session's space in the app" (
docs/herdr-app.md:40). The space id comes from the session name only (space_id_for,herdr_app_backend.py:80-82). Two homes get the same session name withcao launch --session-name <name>, or with the same idempotency key (utils/orchestration.py:1304). sync_layout_namesranks tabs only by this server's terminals. A tab with no terminal of this server gets the ranklen(rank), so it goes after this server's tabs (:366-413). Each server sees only the terminals with its owncao.home, and each server has its own database. So server A puts its tabs first, and server B puts its tabs first.- Each sort that changes the order is a
doc.set. herdr-core v0.1.21 savesdocs.jsonand sends adocevent to every unfiltered subscriber (api/doc.rs:24-37,events.rs:127-134). CAO subscribes unfiltered (hc_rpc.py:265). Every layoutdocevent runssync_layout_namesagain (herdr_app_inbox_service.py:246-249). No check stops a server when the other server changed the order. - Result: a write by A makes B write, and a write by B makes A write. This continues while both servers have terminals in the space. Each write saves the document to disk and sends an event to every client of the core. herdr-app replaces its model with each new version (herdr-app
317e673,src/window.rs:2141-2181). So the tabs change order on the screen again and again, and a local layout edit that is not saved yet gets "Layout changed elsewhere" (window.rs:2181). - I reproduced it with one
FakeHcand two realHerdrAppBackendobjects. They have differentcao.homevalues, one session name, two terminals each, and a registry for each server:- I ran
sync_layout_names6 times, alternating A and B. Each pass wrote once, and the order changed each time between[t1, t2, t3, t4]and[t3, t4, t1, t2]. With the sort off (_creation_orderreturns{}), the same probe wrote twice (the names), and then it stopped. - I connected each
doc.setto adocevent that goes to both realHerdrAppInboxService.handle()methods. After 50 events, there were 100 writes and 51 events in the queue.
- I ran
- Options:
- (a) Sort this server's tabs only among the positions that they already have. Tabs that this server does not know stay where they are. Then the servers do not move each other's tabs, and each pass gets to a stable order. One server alone keeps the full feature.
- (b) Do not sort when the space has a tab that this server does not know. This is simple. But a shared space, or a space with a person's own tab, gets no order.
- (c) Sort only when a new tab of this server first appears in the layout (when the app adopts a new terminal), not on every
docevent. This writes less, and a person's drag stays. Two servers can still move each other's tabs, but only one time for each new tab. - For each option, add a test: two backends with different
cao.homevalues on oneFakeHcand one session name. Alternatesync_layout_namesand make sure that the writes stop.
P3-2: The docs and the PR description do not say that CAO sorts tabs (docs/herdr-app.md:37-50; PR description, "Placement")
5068057and1f13b97sort the session's tabs on every naming pass. The order is creation order, and each tab goes after the tab of the terminal that created it (caller_id,herdr_app_backend.py:327-364). The naming pass runs on every layoutdocevent.- A user sees these effects, and nothing in the docs tells them:
- A person drags a CAO tab to a new position. The drag is a layout write, so the next naming pass moves the tab back.
- A person's own tab in a CAO space moves after all CAO tabs.
- The PR description says that the layout write sets names, and that placement uses labels. The reply does not list these commits.
- Options: Add a "Tab order" item to "How CAO uses the core" in
docs/herdr-app.md, and one line to the PR description. If a person's drag must stay, option (c) of P2-3 also does that.
What I checked
- The first 19 commits are patch-identical to the commits that I reviewed at
9224f3e. I read the 7 new commits and their tests. - The targeted suites for the changed files at
8cd84d9: 339 passed. - Mutations. Each one made the new tests fail:
- One shared input connection: 6 failures.
native_confirmforced off: 2 failures.- No tab sort: 3 failures.
caller_idignored:test_grouped_by_parentfails.- Pickup rule always true:
test_swallowed_enter_is_resent_for_claude_codefails.
- The upstream source for each claim: herdr-core
v0.1.21(server.rs,input.rs,api/types.rs,api/doc.rs,events.rs,terminals.rs) and herdr-app317e673(src/model.rs,src/window.rs). - The two probes for P2-3 (above).
git merge-treewithmain(b2b3666): no conflicts. Since3ef0667,mainhas one new commit,b2b3666(#1399,codex.py). It does not touch these files.- CI at
8cd84d9: 27 of 27 checks pass.
Notes
-
Multi-agent flow on herdr-app. The e2e suite covers the session lifecycle, history depth, and inbox delivery when the agent is idle. The live runs in the description use one worker.
cf9a46fand2082dbechange the input and inbox paths that a multi-agent flow uses. Before merge, please runexamples/assignwithterminal.backend: herdr-appat the final head, and confirm these results:- The supervisor assigns 3 tasks to data analysts in parallel, and then hands off to the report generator in sequence.
- Each data analyst completes its task, and its message gets back to the supervisor.
- The supervisor completes the report. It does not do the analysts' tasks itself.
This is the same check that new providers must pass (step 9 of the
cao-providerskill). It is a request for evidence, not a finding. -
Branch state. The branch is one commit behind
main. The merge has no conflicts. -
Delivery time. One inbox delivery can now keep its worker thread for up to 3 × 8 s (
INBOX_SUBMIT_CONFIRM_TIMEOUT,INBOX_SUBMIT_MAX_RESUBMITS;constants.py:386,:391). The deliveries and the event reader use the same default asyncio executor. So many slow deliveries at the same time can delay events. Pickup also reads status on the request connection, so it does not need those events. This is not a finding.
🤖 Reviewed with CAO review workflows
…r CAO Assisted-by: AI
terminal.send and terminal.keys can wait up to about 60 s for a person's unfinished draft. HcRpcClient serializes exchanges on one connection, so every status lookup and terminal.list queued behind such a send. Input now goes through a second HcRpcClient for the same session, opened on first use. An injected client (tests) doubles as the input client unless input_client= is given. Assisted-by: AI
Every terminal the backend creates now carries cao.home, the first 16 hex of the SHA-256 of the resolved CAO_HOME_DIR, and every lookup (sessions, windows, exact cleanup, list_managed, naming) filters on it. Two cao-servers with different homes sharing one core no longer see, reconcile, or kill each other's terminals. Space ids depend only on the session name, so two servers using the same session name share a space; kill_session now leaves the space in the layout document while any live terminal still carries it. Assisted-by: AI
HerdrAppInboxService follows the CAO core's event stream on its own hc connect: delivers pending messages on idle/done, notes native activity on working/blocked, records each mapped status change once, tears down CAO terminals on finished/lost, kills shells that replaced a torn-down agent (restarted), names spaces and tabs on herdr-app doc events, rebuilds its map on missed, and reconnects with 1-30 s backoff on EOF or 45 s of silence, reconciling rows and terminals under the session lifecycle lock after each connect. Changes from the plan: - Only terminals with this server's cao.home label are mapped, restarted-killed or reconciled; another server's are ignored. - After teardown on finished/lost the terminal is killed and removed by address: it is no longer live, so kill_window's label lookup cannot find it and its record would otherwise linger in the core. - A failed teardown, removal, row drop or orphan kill is logged and does not end the stream (which would reconnect and re-reconcile). Assisted-by: AI
Delivery types through terminal.send, which can wait up to about 60 s for a person's unfinished draft. Awaiting it inline held every other terminal's events (status, finished, doc) for that long. Each delivery now runs as its own task, at most one in flight per terminal; an idle or done that arrives meanwhile triggers one rerun when it ends. Shutdown cancels deliveries still in flight. Assisted-by: AI
…--terminal - herdr_inbox_registry types its singleton as a runtime-checkable EventInboxService protocol (register_terminal, unregister_terminal), so terminal_service's call sites type-check for both services and the module still imports neither. - The lifespan starts HerdrAppInboxService for HerdrAppBackend, with the same inbox_service.deliver_pending callback as herdr; the existing shutdown block cancels and unregisters it. - /health reports terminal_backend "herdr-app" and a "herdr-core" component (hc or herdr-core on PATH). - cao-server --terminal accepts herdr-app. Audit of herdr-specific branches outside the backends: - terminal_service.py:1746 isinstance(HerdrBackend) Kiro home path refusal: herdr CLI argument quoting only; herdr-app passes env as JSON. No change. - terminal_service.py:3277 FIFO setup skipped for event-inbox backends: herdr-app pipe_pane is a no-op. Correct as is. - terminal_service.py:3477 and :7019 register/unregister through get_herdr_inbox_service(): works for both via the protocol; pane_id is the hc address. No change. - terminal_service.py:5350, :5415 and api/main.py:5647 event-inbox dispatch recency and boundary: herdr-app has no output generation either. Correct as is. - inbox_service.py:322 skips submit confirmation for event-inbox backends because herdr delivers on the event loop. herdr-app delivers in a worker thread, so confirmation could run there; left skipped (same behavior as herdr), follow-up. - status_monitor.py:1135, :1182, :1431, providers/base.py:698, utils/terminal.py:153: generic supports_event_inbox() paths using get_native_status/get_history. Correct as is. - session_service.py:631 and terminal_service.py:858: docstrings about HerdrBackend's unconfirmed kill_session and lenient strict check; HerdrAppBackend confirms kill_session with a fresh lookup and its session_exists_strict raises. No change. Assisted-by: AI
The reconciler deleted every CAO row in a session that had no live herdr-core terminal. An operator switching from the tmux or herdr TUI backend to herdr-app would lose all their existing CAO rows on the first connect, because this core has never heard of those terminals. A row is now dropped only when the core has at least one record for its cao.terminal_id under this server's cao.home (terminal.list with all: true, so finished records count) and every such record is in state "finished". A row with no record is another backend's and stays; a record in a state other than "finished" keeps the row too. The orphan side (live terminal with our labels and no row) is unchanged, and a failed lookup still skips the session. HerdrAppBackend.list_managed takes include_finished to ask for the finished records. Assisted-by: AI
…erdr-core Each test starts its own core: a fresh HC_SESSION and XDG runtime, state and config dirs under a short /tmp scratch dir (unix socket paths are limited to 108 bytes, too short for pytest's tmp_path). - Lifecycle: two terminals share one herdr.space and carry this server's cao.home; "echo hi-$((40+2))" proves the shell ran the command (hi-42), not just echoed it; kill_session leaves nothing live. - Depth: terminal.read with lines=20000 returned 2994 rows on hc 0.1.21 (history plus the 120-row screen; the cap counts rows, so wrapped lines count more than once). HC_READ_MAX_LINES goes from 1000 to 2000, N rounded down to a multiple of 1000, and the test asserts the core can return at least that many. - Inbox: HerdrAppInboxService runs against the core with its database seams patched; a terminal printing OSC 7501 state=working then state=idle drives PROCESSING then IDLE and the delivery callback. Assisted-by: AI
… script docs/herdr-app.md covers prerequisites (herdr-core 0.1.21+), enabling the backend (config, env, --terminal) and its session setting, showing the core in the app (hc machine add), /health's herdr-core component, the labels CAO sets, cao.home (several cao-servers with different homes share one core; same session names share a space), the three hc connect children per server, reconcile leaving other backends' rows alone, keys typed without Enter (what terminal.keys accepts, checked on hc 0.1.21: space maps to "space", shifted punctuation is rejected), sends waiting for ~3 s quiet and a person's draft, the 2000-line read cap, removing the app's machine entry before stopping the core, and a "Migrating from the herdr TUI" section. configuration.md lists herdr-app and terminal.herdr_app_session; herdr.md points app users at the new page. scripts/herdr_tui_to_app.py, the one-time herdr TUI to Herdr app migration the new section describes, is added (black and isort applied; --self-test passes). Assisted-by: AI
After a native idle or done, providers/base.py reports PROCESSING for a 10 s buffer flush wait. The delivery the status event triggers runs inside that window, finds the terminal busy, and leaves the message to the 30 s inbox reconciliation sweep. Measured against a live cao-server with a mock_cli terminal: 36 s from native idle to delivery. On idle or done the service now also schedules one more delivery SETTLE_RETRY_S (10.5 s) later. working, blocked, finished, lost and removed cancel it; a later idle or done restarts it, so there is never more than one per terminal. deliver_pending re-reads pending rows under its per-terminal lock, so an extra call with nothing pending is free. Same live check with the retry: 10.6 s. Assisted-by: AI
The include_finished flag on _list made every **labels call site an arg-type error (8 new mypy errors). list_managed now issues the terminal.list with all: true itself and _list is unchanged. Assisted-by: AI
…te lint test_timeout_literals_are_derived_or_justified flagged wait(timeout=5) and run(timeout=30) in clients/hc_rpc.py. They become CONNECT_REAP_TIMEOUT_S and SERVER_START_TIMEOUT_S, each with the bound it is chosen against. Assisted-by: AI
A cao-server started inside a Herdr app terminal inherits HC_TERMINAL_ID, HC_TERMINAL_SOCKET and HC_SCOPE, and an operator may set HC_SOCKET, which herdr-core's docs/security.md says overrides the client endpoint. The browser attach argv kept all four (the PTY inherits os.environ), so it attached to the user's own core, and HC_SOCKET was stripped nowhere. - hc_rpc strips HC_SOCKET too, for hc connect and hc server start (which now also gets stdin=/dev/null). - prepare_web_attach returns `env -u ... HC_SESSION=<s> <hc> attach <addr>`; attach_session execs the resolved hc with the same scrubbed environment. Both use hc_binary() instead of a bare "hc". - HcSubscription closes its hc connect child when the subscribe write, read or reply fails; the caller never gets the object, so nothing else could. - The TUI migration script strips HC_SOCKET as well. Tests cover all three spawn paths with the four variables set, run the web attach argv against a stand-in hc that prints its environment, and check the subscription's child is reaped and that events.subscribe carries no filter. Assisted-by: AI
Live with kiro-cli 3.3.0, herdr-core kept a terminal at `error` after a failed turn while the agent was back at its prompt. HerdrAppBackend mapped that to TerminalStatus.ERROR, and terminal_service._refuse_if_provider_gone then refused every later input to the terminal. `error` is no longer mapped, as on the herdr TUI backend: get_native_status returns None and the provider's buffer analysis decides. The inbox shares the map, so an `error` event is neither recorded as a transition nor delivered on; the next `idle` or `done` records and delivers as before. docs/herdr-app.md now says a native `error` is ignored and that inbox delivery lands about 10 s after a turn ends (CAO's completion wait). Assisted-by: AI
…s own deletes
- register_terminal / unregister_terminal run on worker threads (the API runs
delete_terminal in asyncio.to_thread) but cancelled loop tasks and mutated
the maps the loop iterates. The address and last-state maps now sit behind a
threading.Lock and change before the call returns; a pending retry is
cancelled on the loop (call_soon_threadsafe off it). start() records the
loop.
- Cancelling the service in the reconnect backoff skipped cancelling pending
deliveries and retries. start() now cancels them on any CancelledError.
- On `missed` the service also reconciles: a terminal that finished in the
gap is in no snapshot and its `finished` never arrived, so its row stayed.
- delete_session tore each terminal down twice: its kill_session produces a
`finished` event and the inbox ran delete_terminal again ("Failed to
snapshot terminal ... no live herdr-core terminal"). _teardown now takes the
session's lifecycle lock, which session teardown holds from kill_session
until its rows are gone, and skips a row that is no longer there. A
single-terminal delete unregisters before it kills, and the map change is
now synchronous, so its event is never mapped.
Assisted-by: AI
- FakeHc can keep a terminal alive through terminal.kill and terminal.rm: kill_session then returns False and keeps the space, and cleanup_terminal_exact reports STILL_PRESENT, never DELETED. - Reconcile keeps a row whose records are one finished and one in a state CAO does not know. - Naming leaves the user's own space, and the user's tab inside a CAO space, untouched while naming CAO's; a space the app placed before the CAO tab is named on its own. The spec now states that rule (the inbox naming pass's) instead of "write nothing when the space or tab is missing", which the code never did for a missing tab. - The event stream is read with the 45 s silence limit. Each new test fails against a mutation that drops the check it covers. Assisted-by: AI
The app adopts a labelled terminal with Model::add_tab, which inserts the tab after the space's selected tab. The conductor is usually selected, so every new worker landed right after it and the newest worker ended up leftmost. The naming pass now also sorts each CAO space's tabs by creation order, taken from the registry's oldest-first terminal order (list_terminals_by_session, rowid order). Tabs CAO does not own go last, and a registry read failure skips ordering for that pass. The cost is a brief jump after each adoption, and a manual drag of a CAO tab is undone, as renames already are. Asking herdr-app to adopt with add_tab_unselected (append, keep selection) would remove the jump. Verified: new tab-order tests fail without the change and pass with it; herdr-app backend and inbox suites pass; mypy clean on the file. Assisted-by: AI
With creation order alone, a worker a supervisor spawns lands at the far right, away from its supervisor. Tabs now follow the delegation tree: each terminal comes right after its parent's group (the terminal recorded as its caller_id), siblings in creation order. A terminal whose parent is not in the session, such as one created from the CLI or the operator API, is a top-level entry in creation order, so sessions without caller data keep the flat order. caller_id is read per terminal with get_terminal_metadata rather than added to list_terminals_by_session, whose rows also back API responses. Verified: the grouped test fails on the creation-order-only code and passes with this change; herdr-app backend and inbox suites pass; mypy clean on the file. Assisted-by: AI
All terminals shared one input HcRpcClient, and herdr-core answers a plain `hc connect` one request at a time (server.rs connection). It holds terminal.send while a person has a draft in that terminal, up to 60 s after their last key (input.rs type_in), so one draft stopped CAO input to every other terminal: inbox deliveries, send_input, startup-dialog keys and typed launch commands. Input now opens one HcRpcClient per terminal address on first use. The core already queues input per terminal, so this is the same concurrency framed mode would give, without a second wire protocol in CAO. A connection closes when CAO kills its terminal (kill_address, so kill_window, kill_session and cleanup_terminal_exact), when the terminal is removed outside CAO, when the naming pass's fresh listing no longer shows it, and at server shutdown. HcRpcClient.close() no longer waits for a call in flight: it ends the child, the held call fails as lost, and the core cancels a send still in its draft hold. A closed client never reconnects. Trade-off: one `hc connect` child per terminal CAO typed into (N + 2 in total, about 2.4 MB each), documented in docs/herdr-app.md. Verified: the new test holds terminal.send on terminal A and shows keys and a send to B complete meanwhile; it fails when every address shares one connection. Live on a scratch core (cao-dev-*): 3 terminals typed into gave 3 input children, killing one left 2, close_inputs left none. Backend, client and inbox suites pass; mypy at baseline (122). Assisted-by: AI
…fter 200 ms herdr-core's terminal.send pastes, waits PASTE_GAP (200 ms) and presses Enter (input.rs send_message). The only override is a template's paste_gap_ms, read from ~/.config/herdr-core/templates files that are shared with the user's own sessions; terminal.send has no gap parameter (hc 0.1.21 schema: caller, force, if, terminal, text, timeout, wait). So send_keys cannot honor submit_delay, and Claude Code asks for 2.0 s because an early Enter is swallowed. Inbox deliveries on herdr-app also skipped _confirm_submitted, because supports_event_inbox() is true, so a swallowed Enter stayed in the prompt box until a person pressed Enter. That skip exists because the herdr TUI inbox service delivers on its event loop. HerdrAppInboxService already delivers via asyncio.to_thread, so a new backend capability, event_inbox_confirms_submit() (default False, True on herdr-app), lets herdr-app run the confirmation. Event-inbox status has no output generation, so pickup uses the first-task path's rule for these backends (_wait_for_post_dispatch_start): a started status counts if it is PROCESSING or differs from the status read just before the dispatch, kept across resubmits. Native activity after the send (started_on_last_input) also counts, because terminal.send waits up to 5 s for the agent to start, so a short turn can be over before the wait begins. Recovery is the existing helper (bare Enter when the text is still in the composer). tmux and the herdr TUI are unchanged. Trade-off: a herdr-app delivery can hold its worker thread and the receiver's delivery lock for up to 8 s x 3 while unconfirmed, as on tmux. Option (a) was not done: no supported per-send or per-terminal gap exists without writing template files into the user's herdr-core config. Verified: new tests deliver to a ClaudeCodeProvider (paste_submit_delay 2.0) on a HerdrAppBackend whose status stays COMPLETED and see Enter resent; pickup by PROCESSING, by a transition, or by native activity sends nothing; the herdr TUI keeps the base False. Two of them fail without the inbox change. test/services and test/backends pass except test_skips_when_processing, which fails the same way without this change (unpatched provider lookup on the eager path). mypy at baseline (122). Assisted-by: AI
docs/herdr-app.md said a send waits for about 3 s of output quiet, and the INPUT_TIMEOUT_S comment said calls wait for output to settle. In herdr-core 0.1.21 the 3 s is measured from a person's last key in that terminal (HC_QUIET_MS) and applies only if someone typed (input.rs Gate::wait). terminal.send also waits for their draft, up to 60 s after that key. The output wait (800 ms quiet, at most 5 s, then `unavailable` with nothing typed) applies only during a terminal's first 10 s (input.rs settle). The docs now use the reviewer's wording. The comment states the same waits, plus the up-to-5 s wait for the agent to start after typing, which is part of what the 90 s timeout must cover. Verified: wording checked against input.rs at v0.1.21 (quiet, default_draft_timeout, Gate::wait, settle, STARTUP_MS, SETTLE_MAX, started_window) and the terminal.send and terminal.keys descriptions in `hc schema`. Docs and comment only. Assisted-by: AI
docs/herdr-app.md says CAO never uses your `default` session, the one the Herdr app opens for your own terminals, but nothing enforced it: terminal.herdr_app_session (or CAO_HERDR_APP_SESSION) set to "default" made cao-server start that core, write its app layout and run reconcile and kills there. Those only touch CAO-labelled terminals, so the risk was low, but the docs promised more. BackendFactory now raises ConfigurationError for "default", before any core is started or contacted, and the message says why and suggests "cao". No other name is refused. Verified: new factory tests (env and settings file) fail without the check and pass with it; test/backends/test_factory.py passes. Assisted-by: AI
scripts/herdr_tui_to_app.py has assert-based offline checks behind --self-test, but neither CI nor pytest ran them, so they could go stale unnoticed. A pytest wrapper now runs `python3 scripts/herdr_tui_to_app.py --self-test` in a subprocess with the test interpreter and asserts exit 0 and the "all checks passed" line. It needs no herdr or hc. Verified: the wrapper passes; the whole run takes under a second. Assisted-by: AI
Two cao-servers with different homes can share a space when they use the same session name. The tab sort ranked a server's own tabs first and put every tab it did not own after them. Each server therefore reordered the other's tabs, and every resulting layout write woke the other server's naming pass: the order flipped and the document was rewritten without end. The sort now permutes a server's own tabs only among the positions they already hold. Tabs it does not own, a person's own tab or the other server's, keep their place, so each server reaches a fixed point and stops writing. A single server still gets the full ordering. The docs now describe the tab order, including that dragging a CAO tab is undone on the next layout change. Verified: the two-server test fails on the previous sort and passes with this change; the herdr-app backend, inbox and docs-link suites pass; mypy clean on the file. Assisted-by: AI
8cd84d9 to
dfff087
Compare
|
Thanks for the second review. The P2-3 loop is real, and it is fixed in dfff087 (the branch is also rebased onto main at b2b3666). P2-3, two servers undoing each other. I took your option (a), in a slightly stricter form. P3-2, docs. For reference, the two commits you flagged are |
Adds a
herdr-appterminal backend. It runs CAO terminals in a dedicated herdr-core session, so each CAO session shows up in the Herdr desktop app as one space, with one tab per terminal. It also adds a one-time script that moves herdr TUI sessions into the app.Fixes #1355
Why. The Herdr app runs on herdr-core (
hc), not on the TUI server that theherdrbackend drives, so CAO terminals don't appear in the app. A common setup is the app on a laptop and CAO on a remote box the laptop reaches over SSH.Approach
hc connect --session <s>, the core's own entry point and the one the app uses over SSH. The socket path differs between Linux and macOS, so CAO doesn't open it directly. Everyhcchild, includinghc attach, is started withoutHC_TERMINAL_ID,HC_TERMINAL_SOCKET,HC_SCOPEandHC_SOCKET. A cao-server started inside an app terminal therefore never reaches the user's own core.herdr.spaceandherdr.tablabels. Space and tab names exist only in the app'sherdr-applayout document, so CAO sets them with one version-checked write to it. Writing the whole document for placement was rejected: the app resolves concurrent edits as "newest accepted document wins".cao.homelabel keeps each cao-server to its own terminals.HerdrAppInboxServicefollows the core's event stream for status, exits, lost terminals and layout changes. Reconcile only drops a CAO row whose core record has finished, so rows from tmux or the herdr TUI are left alone. A nativeerrorstatus is ignored, because herdr-core keeps it after a failed turn while the agent is back at its prompt.Behavior change. This only applies with
terminal.backend: herdr-app. tmux and the herdr TUI backend are unchanged.Rollback. Set
terminal.backendback and restart cao-server.Verified
CAO_EAGER_INBOX_DELIVERYunset. The e2e suite against a throwaway herdr-core: 3 passed. mypy: no new errors.mock_cli. A cao-server against a throwaway core: two terminals in one space, inbox delivery while idle and while busy, server restart, and session delete.herdr-app, viewed in the Herdr app on another machine over SSH (hc machine add <host> --session <s>). A conductor assigned a worker, and the worker's tab appeared in the conductor's space. The worker's callback reached the conductor. Typing into tabs from the app and deleting the session both worked.