Repository navigation
Description
Problem
When a worker answers its caller with send_message, the server closes the caller's delegation edge as replied as soon as the message is stored, before delivery is tried. The inbox POST then answers success: true after a 5 s wait whatever happens next. If delivery fails, nobody is told.
Seen on the herdr-app backend (cao-server 2.5.4) in a conductor -> supervisor -> forge chain. The forge worker's final reply to the supervisor (message 17944) was stored and its edge closed as replied. Delivery then failed before anything was typed: the provider reads the screen first (mark_input_received), and herdr-core's terminal.read answered nothing for 10 s.
ERROR Failed to deliver message 17944 to <supervisor>: herdr-core lost terminal.read: herdr-core sent nothing for 10.0s
The message was marked FAILED for good. The failure handler only tells callers of edges the failed message opened, and a reply opens none. The supervisor looked finished (idle, waiting on nobody), so after the 300 s pickup timeout the server auto-forwarded its stale last line to the conductor, and the worker's result was never seen. Resending the same 2,716-byte text later was delivered, so the failure was transient. In 31 server logs (Oct 17 to Nov 17, 9,576 delivery batches) it is the only delivery failure.
Four gaps:
- A reply closes its edge when it is stored, not when it is delivered.
- A reply that fails to deliver notifies no one.
- Any delivery exception other than "pane not found" and the dialog refusals is final, even when nothing was typed and a retry is safe.
- The inbox POST and
send_messagedo not report whether delivery worked, so a sender cannot know to resend.
Proposal
- A reply marks its edge (
reply_message_id) and leaves it open. The edge closes as replied when that message is delivered. If the message fails for good, the mark is cleared and the caller is waiting on the worker again. While a reply is in flight neither side is auto-forwarded. The 30 s sweep settles marks left by a restart. - Classify a failed attempt as
failed(nothing was typed: any step before the backend send) oruncertain(it failed while sending, so the text may have arrived). send_messagewaits up to 45 s for the outcome and returnsdelivery:delivered,queued,failed,uncertain, orin_progress.failedanduncertainreturnsuccess: falsewith the error, and the server does not retry them. Anuncertainerror starts with "May have been delivered:". The 45 s keeps the client timeout (55 s) under the lowest agent-CLI MCP tool limit measured (Codex default, 60 s).- A queued message that fails before typing is retried up to 3 times, then its sender gets a
[CAO delivery failed]notice. Anuncertainqueued message is not retried and the notice says it may have arrived.
Not in scope: undoing the "input sent" stamp of a failed attempt, making the pre-typing screen read non-fatal, and why herdr-core stalled.
The fix depends on the herdr-app input path, so the PR is stacked on #1356.
Activity