Skip to content

A reply that fails to deliver still counts as sent, and send_message reports success #1417

Open

Description

@anilkmr-a2z

Problem

When a worker answers its caller with send_message, the server closes the caller's delegation edge as replied as soon as the message is stored, before delivery is tried. The inbox POST then answers success: true after a 5 s wait whatever happens next. If delivery fails, nobody is told.

Seen on the herdr-app backend (cao-server 2.5.4) in a conductor -> supervisor -> forge chain. The forge worker's final reply to the supervisor (message 17944) was stored and its edge closed as replied. Delivery then failed before anything was typed: the provider reads the screen first (mark_input_received), and herdr-core's terminal.read answered nothing for 10 s.

ERROR Failed to deliver message 17944 to <supervisor>: herdr-core lost terminal.read: herdr-core sent nothing for 10.0s

The message was marked FAILED for good. The failure handler only tells callers of edges the failed message opened, and a reply opens none. The supervisor looked finished (idle, waiting on nobody), so after the 300 s pickup timeout the server auto-forwarded its stale last line to the conductor, and the worker's result was never seen. Resending the same 2,716-byte text later was delivered, so the failure was transient. In 31 server logs (Oct 17 to Nov 17, 9,576 delivery batches) it is the only delivery failure.

Four gaps:

  1. A reply closes its edge when it is stored, not when it is delivered.
  2. A reply that fails to deliver notifies no one.
  3. Any delivery exception other than "pane not found" and the dialog refusals is final, even when nothing was typed and a retry is safe.
  4. The inbox POST and send_message do not report whether delivery worked, so a sender cannot know to resend.

Proposal

  • A reply marks its edge (reply_message_id) and leaves it open. The edge closes as replied when that message is delivered. If the message fails for good, the mark is cleared and the caller is waiting on the worker again. While a reply is in flight neither side is auto-forwarded. The 30 s sweep settles marks left by a restart.
  • Classify a failed attempt as failed (nothing was typed: any step before the backend send) or uncertain (it failed while sending, so the text may have arrived).
  • send_message waits up to 45 s for the outcome and returns delivery: delivered, queued, failed, uncertain, or in_progress. failed and uncertain return success: false with the error, and the server does not retry them. An uncertain error starts with "May have been delivered:". The 45 s keeps the client timeout (55 s) under the lowest agent-CLI MCP tool limit measured (Codex default, 60 s).
  • A queued message that fails before typing is retried up to 3 times, then its sender gets a [CAO delivery failed] notice. An uncertain queued message is not retried and the notice says it may have arrived.

Not in scope: undoing the "input sent" stamp of a failed attempt, making the pre-typing screen read non-fatal, and why herdr-core stalled.

The fix depends on the herdr-app input path, so the PR is stacked on #1356.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions