Repository navigation
fix(tea): a durable agent's log is write-ahead, a repeated start is a no-op, one request runs at a time, and a late save is fenced [invariant 1, 2, 3, 6, 7, 8] - #991
Merged
Conversation
A run lost while its model call or its compaction was in flight restores as `running`, and `agentIsResumable` only said yes to `suspended`. The host booted it, dispatched nothing, and the run stayed `running` with nothing in flight. One table keyed by every `AgentStatus` kind now says which kinds are mid-run (`running` and `suspended`). `agentIsResumable` and `defineAgent`'s choice between `agent_start` and `agent_boot` both read it, so the second copy of that rule in `defineAgent` is deleted. The table reads `status`, so a failure annotation on a running phase is `failed` and a `stale` run is `running`. The recovery test restores a four-document run (57,344 characters per result, checkpoint over 100 KB) lost during the model call and during the compaction, three times each with three concurrent `host.runtime()` callers, and expects the final answer, no repeated read and one reissued operation. The agent matrix test covers every status kind, a failure annotation on a raw running phase and a stale run. ADR 0040 amends ADR 0004 only where it ties resumability to awaiting tools. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
`stepHost` could not refuse work for a run it had already been told to stop:
the engine's `resume` and `nextStep` were the only places to look, and a first
pull or a deferred enqueue reached neither in time.
`StepEngine` gains an optional `beforeResume(runId)`. It runs for every
authenticated pull, after the token is accepted and the give-up alarm is
re-armed and before the intake, the resume, the deferred readiness probe and
enqueue, `nextStep` and `terminalOutput`. `null` continues as before; the
existing terminal `{ done: true, output }` arm ends the pull there. A throw or a
rejection propagates before the intake is touched. Without the hook the old
path and both overloads are unchanged; no new wire arm is added.
The admission tests cover both hosts: the order, the short-circuit, a first
pull, duplicates, an async hook, `null`, a throw and a rejection, and the
unauthorized paths. A type test pins that an engine written before the hook
compiles unchanged and that the inline response keeps its two arms. The
recovery test persists a stop before the fourth of four 57,344-character reads is
posted and expects cancellation with no further model call, three times each on
the inline and the deferred host.
`step-host.test.ts` leaves the typecheck debt list in the same change: its eight
`noUncheckedIndexedAccess` errors are fixed.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…s [invariant 4, 6, 2, 3] The deploy guide stopped at `agent.run`. A host that holds the run itself had no page that says how the pieces fit, so each wrote its own wake check, its own stop check and its own alarm. Section 4 shows three distinct compositions from `@demlik/tea/do`: - Wake: `createAgentHost.runtime()` dispatches `agent_boot` once for a run that was running or suspended, so the host does not dispatch it again. - Stop: a `beforeResume` that reads the recorded stop, cancels the run through a runtime that boots nothing, and answers the pull with the terminal response. - Deadline: persist an absolute deadline at run creation, `await timer.rearm()` before answering, and rebuild the timer from the stored deadline after a wake. One function, `earliestActive`, owns the single alarm: the timer arms what it returns and the `AlarmStorage` handed to `stepHost` arms it too, so a pull moves the idle deadline and never the absolute one. The alarm only schedules a wake; the handler persists the outcome. The coexistence page lists the optional hook on `StepEngine` and the guide links to it for a host that does not own the alarm. Two composition tests join `step-host.test.ts`: a run no step request reached still expires at its startup deadline after a cold re-created timer fires, three times, and a pull cannot postpone an absolute deadline while an earlier idle deadline still wins. Both fail when the rearm is not awaited or when the first pull does not end the startup window. No timer is registered in the reducer, `init` or `interpret`. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…e [invariant 2, 3, 6] A minor bump, since `StepEngine`, a stable `./do` declaration, gains an optional member and carries the additive marker. The same changeset states the wake correction (a `running` run now boots like a `suspended` one) and names what it replaces: the private copy of the boot rule in `defineAgent`, which now reads the same table as `agentIsResumable`. The reference pages were regenerated in the commits that changed their TSDoc and a second run leaves no diff. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…nvariant 1, 6] The recovery tests ran three times, woke through `host.runtime()` alone, and read the stop from an in-memory flag, so none of them crossed `stepHost` on storage that outlives an activation. The fixture now drives `stepHost`'s `handle` over one retained storage that holds the agent's Model, the run capability and the recorded stop. The hands pull a step, run it and post the result. The activation is evicted while the run is `running`, once in its model call and once in its compaction, and a fresh host, runtime and step host over the same bytes answer the hands' retried post. The run reaches `done` with turns 1 to 5 returned once each, model invocations equal to returned turns plus the one lost call, the compaction reissued once, and each document step handed out once. A further pull and a re-posted result on a third activation return the same answer with no new call. A stop is written to the retained storage before the third result is posted. A fresh activation's `beforeResume` reads it, cancels through a runtime that boots nothing, and answers every duplicate post, a re-pull and a cold third activation `cancelled`, inline and deferred, with three model calls in all and no resume or enqueue. With no stop recorded the same post resumes as before. Every case now runs ten times. The step engine waits on the saved state, since a reissued operation parked on a tool keeps `host.runtime()` pending until the driver answers. The in-memory stop tests are deleted with their helpers. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The changeset said neither the removed copy nor `agentIsResumable` was a public declaration, but `agentIsResumable` is exported on the stable `./do` subpath. It now says the removed helper was the private copy inside `defineAgent`, and that `agentIsResumable` keeps its signature and answers true for a `running` run. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The integrated fixture showed that `host.runtime()` stays pending while a reissued operation is parked on a tool the driver has not answered, and that `host.reset()` waits for a build in that state. The guide's wake and stop sections did not say so. They now say that a `/step` engine starts the runtime without awaiting it and reads the saved state, and that the stop is read before anything in the activation has started the runtime. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…invariant 6, 2, 3] After a cold wake handed out a tool step, recording a stop and posting the tool result to the warm host hung in `host.reset()`: `build()` awaited the boot dispatch, the dispatch waits for its whole chain, and the chain waited on the tool the hands had not answered, so `cached` never settled and `reset()` awaited it. A host could not stop a run it could not reach. The same held for a `suspended` run, and now for a `running` one. `createAgentHost` keeps the `RunHandle` the moment `run` returns it, before any await, and `reset()` stops that handle directly. A second `runtime()` still shares the one build. Invariant 6 says "Boot is a named, awaitable moment" and names `await runtime.ready` as that moment, and `bootResume` dispatches after it. So the host's gate now opens when `agent_boot` is queued, not when the work it re-fired is done: the driver reaches the run through `runtime()`, so a `/step` engine cannot be asked to wait for the driver. `beginResume` (not on the barrel) hands back the dispatch chain, `bootResume` and `autoBoot` still wait for it, and the agent's port is written once for `autoBoot` and the host. The fixture's start-and-poll workaround and the guide paragraph that documented the limitation are deleted: the step engine calls `runtime()`, `reset()` and `status()` and waits on `subscribe`. New tests run on retained storage ten times for a run left `running` and one left `suspended`: cold recovery hands out a tool step, the stop is persisted, and the post to the warm host is answered `cancelled` in one request with no further model call and `reset()` resolved. ADR 0040 records the gate decision. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
MAINTAINING.md gives a call-breaking change the `**Breaking` marker and says a pull request holding both kinds states both, in separate changesets or one. `createAgentHost().runtime()` now resolves once `agent_boot` is queued, so a caller that read a finished run right after `await host.runtime()` must wait with `subscribe` or `status()`. A host that dispatched `agent_boot` by hand for a `running` run now boots it twice and must delete that dispatch. A second minor changeset carries both under `**Breaking`, and the sentences are moved out of the Additive one so each marker covers only its own kind. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… 3, 6, 8] The engine already holds the Msg when it saves: commit(next, msg, cmds) awaits store.save(next) and only then runs the Cmds. The store never saw the Msg, so a store that wanted to journal inputs had to hear them from runtime.observe, which fires after the Cmds. Store.save takes an optional second argument, the Msg the transition applied, and FencedStore.saveFenced takes the same optional context after the expected version. The loop passes its existing Msg; boot and the final flush at stop() have none, and a Msg that folds to null is not saved at all. The fencing wrapper and the Effect engine's failure wrapper forward it, so both engines deliver it and a rejected save still stops the transition before its Subs, listeners and Cmds with the original cause. One-argument stores and two-argument fenced stores still compile and behave as before. The skip-saving recipe forwards the Msg on the saves it allows, and says it is for snapshot stores: skipping a save also skips the input a log would need. ADR 0041 records the optional argument and amends ADR 0017 point 1, which kept Store untouched; it adds no second protocol beside the LoggedStore of ADR 0038. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
`createAgentHost` queues `agent_boot` and opens its gate without waiting for the chain, then dropped the chain's rejection. A save that failed on `agent_boot` left `runtime()` resolved with a `running` run, no model call and no report. The chain has no caller to reject at, so its rejection now goes to the host's existing `onError` under `"boot"`, through the engine's own `sinkOrDefault(onError ?? defaultOnError)`, so a throwing sink or a missing one is still loud. A rejection that is the host's own `reset()` teardown, or a `RuntimeDiscardNotice` the engine already reported under `"discard"`, is not reported twice. The `"boot"` phase doc names the new witness. The regression test restores a run lost during its model call, fails the save of `agent_boot` once, and runs ten times: `runtime()` still resolves early with the run `running` and no model call, `onError` hears exactly one `"boot"` report with the store's error, a second `runtime()` returns the same handle with no second boot or report, and `reset()` then `runtime()` rebuilds from storage, boots again and finishes with each document read once. It fails 10 of 10 on the swallowing host. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… 1, 3, 6, 8]
doEventSourcedStore fed its log from runtime.observe, which fires after a
transition's Cmds. An activation lost inside a model call left no row saying the
call had started, so the wake rebuilt the State from the log and found the run
idle.
The store's save(state, msg) now writes the log. It appends the applied Msg,
advances the meta cell and, when snapshotEvery events have accumulated, writes
the snapshot, in one storage transaction that settles before the engine starts
the transition's Cmds. Values are serialized before the transaction opens, and
the in-memory counters move only after it is acknowledged, so a refused write
rolls back, runs no Cmd and leaves the next event its seq. A save without a Msg
(boot, the flush at stop) writes nothing. snapshotNow pairs the State of the
highest event that was written with its seq, so a transition whose write failed
is never checkpointed.
The public append and the observer wiring that fed it are deleted, not kept as a
deprecated alias: both beside the new save would record every Msg twice. The
unexported workflow Durable Object wrapper that used them, and its only test, go
with it. The raw JSON events, the {lastSeq} meta cell and the {seq,state}
snapshot keep their bytes, so logs written before this read unchanged. Tests
seed those old bytes by hand and append after them.
Tests that fed the log through append or an observer now run the machine and let
save write it. The fake Durable Object storage moves to a shared fixture with
transaction rollback, structured-clone persistence and a refused-write switch.
Docs and the recovery pattern say that this store replays an applied-message log
and that a LoggedStore recovers differently.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…3, 7] agent.start and the agent_start cell reset the run on every call. A retried request, or two that arrived together, fired the first model call a second time, zeroed the run's usage and cleared a finished run's output. The runId is the run's identity (invariant 7), and the State already holds it. The decision now sits in one place, the start entry that both the public verb and the machine cell go through: a tag (begin, repeat, restart) read off the stored run, and one table, checked with satisfies Record<StartCase, ...>, that says what each does. A repeat returns the State it was given and no Cmd, ahead of transition(), so a State with lifecycle notes, a settled run and a failure annotation are all returned as they were, and agentEvents projects no RunDone twice. A begin or a restart runs the one existing reset. A run cancelled before it started holds no runId, so any new id starts a run. monitored-run's own restart is unchanged. isMidRun is the resume predicate and is false for terminal states, so it could not carry this rule. The start docs, the usage-reset wording and the durability explanation say that only a start for a new run resets. Tests cover every standing of a run (running, suspended, stale, compacting, done, failed by deadline, failure on a running phase, cancelled) through both the verb and the cell, compared by identity, plus a different id restarting from each, and two concurrent starts and a retry through the wired machine calling the model once per stage. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…riant 6] The detached agent_boot chain suppressed a rejection whenever the host had moved on (current !== booting), so a genuine boot failure that raced reset() reached no one. Classify by the error alone: only DispatchDiscardedError, what stop() rejects a pending dispatch with, is left out, because the engine already reported that teardown. Every other rejection goes to onError under "boot", reset in flight or not. The host-state check is deleted. One table over the four cases, ten repeats each: an ordinary failure, the same failure rejecting while reset() is in flight (the reviewer's race), the teardown notice, and no rejection. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… 6, 7, 8] stepHost let two requests of one run overlap. A result posted twice, or a pull that arrived while the first post was still resuming, was answered from the State in the middle of that resume: the intake marked the callId pending before the resume was awaited, so the duplicate skipped the resume and read the State immediately. A host author worked around it by making nextStep wait for the run to go idle. After the token check (invariant 8: nothing is admitted before it), each request of a run now waits for the run's earlier requests, then runs its whole turn: the alarm re-arm, the admission hook, the settle or the deferred probe and enqueue, and the read of the next step or the output. The queue is a private tail per runId inside the one stepHost, so a slow run does not hold another, a turn that throws is released and hands its own error to its caller, and an idle run's entry is removed. A request with a bad or unknown token returns at once and takes no turn, alarm or hook. The queue is not a lock across hosts or processes, and it cannot interrupt a model call that has started; both are written into the docs, along with the rule that a hook must not wait on a request for its own run. The intake ledger is now per run (invariant 7: a callId is only unique inside its run) and takes the settled result only after resume returns. A resume that threw used to leave its callId pending forever, so the retried post was dropped as a duplicate and never resumed. Now the callId stays open and the retry resumes. The first acceptance case is added to agent-recovery.test.ts on the real agent host, step host and retained event store: four documents of 73,728 characters, five model turns, a start sent twice at once and again, a duplicate held during turn two's resume, an eviction during turn three, a recovered State of about 149 KB over a snapshot and its tail, ten repeats on one backing store. The attempts are [1, 2, 3, 3, 4, 5]: the second 3 is the one reissue of the interrupted call, and effects remain at least once. It fails without the write-ahead save, without the start rule, and without this queue. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…rted plus a stop value [invariant 6] A reset that arrived while the cold load was pending still resumed into the stopped runtime, and the engine's refusal was reported as a boot failure. Rounds 3, 4 and 5 each fixed one cell of the same table, so this builds all of it. Each build owns one AbortController. reset() aborts it before it stops the handle, with the engine's DispatchDiscardedError as the reason. build() reads the signal after run() returns, and beginResume reads it after the load and before it dispatches, so an aborted build dispatches nothing and settles with that error; a runtime() still waiting rejects with it. The detached chain's rejection is teardown only when the signal is aborted AND the error is one of the engine's stop values (DispatchDiscardedError, RuntimeStoppedError). Any other error, or any error while not aborted, is reported once under "boot". The error-only check from the previous commit is deleted. One table of seven rows, ten repeats each, holds a build at each point with a controllable store: reset before run() returns, during the load, behind a pending save, a pending save that then rejects, no reset with a failing save, no reset with the runtime stopped under the boot, and no reset with a chain that resolves. Each row asserts the reports, the dispatches and stops on the old runtime, and that the next runtime() ends done with every tool run once. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ibes [invariant 6] The root guard no-per-test-timeout fails a test that carries a timeout of its own: a suite that needs a long one declares it once, on its describe, so a test added beside it inherits it. The turn-taking cases and the retained-event-log case passed a number to each it. Both suites now set the bound in their describe options. The bound stays: a gate that never opens fails the suite at 5 s (30 s for the ten-repeat recovery case) instead of hanging it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…and gates the next build [invariant 2, 6, 7] createAgentHost's lifecycle was three promise cells (cached, live and the build's own awaits), so each review found one more race between them. It is now one ExhaustiveTransitions table, empty | building | live | tearingDown across runtime | reset | built | failed | tornDown, with pure cells and one interpreter. Every value a cell may keep arrives on the message, and every effect leaves as a Cmd. One teardown is in flight at a time, and every overlapping reset() shares its promise. A runtime() during a teardown waits for it and then builds, so a replacement run never starts beside the old run's last save. Before, a second reset() returned at once, and the old stop save overwrote the replacement's persisted done (10/10). Deleted: the cached and live cells and the build's catch that cleared live. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…write-ahead-log-idempotent-start
…nvariant 1, 6] stop() gives up on a stuck save at disposeTimeoutMs but cannot call it back. A save that waits inside doEventSourcedStore (behind an earlier write, at the transaction) could land after a later run had loaded the same storage and saved its own State, writing an event over the later run's and forking the log. The store now checks once, at its write point. One generation per log (storage object plus meta cell) in the isolate: a load() advances it, a save or snapshotNow() carries the generation it was called under, and the first thing inside the transaction compares the two. A stale one throws StoreConflictError and touches no cell. Whether the engine gave up on the save is not an input: a save with no later load behind it still lands. The decision is one state x event table (src/do/save-fence.ts, issued/superseded/admitted/refused by takeOver/reach) with one interpreter; the tests walk it cell by cell, then drive real runs with disposeTimeoutMs: 0, ten times each. Nothing is written to storage for the fence, so logs written before this change read unchanged. The host lifecycle table is unchanged: a replacement still builds after the teardown it waited for. ADR 0042 records the check and amends ADR 0025's late-save cost. The changeset is Breaking: a save that used to write over a later run now rejects. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… on every attempt [invariant 1, 6] A storage transaction callback can run more than once: a conflict rolls the commit back and the platform calls it again. The save table judged a save once, so a save admitted at its first attempt stayed admitted, and a retry after a later run had loaded and saved wrote over that run (10 of 10 probes overwrote the snapshot). A refused save attempted again was not refused again either. "reach" is now "attempt", an event on every run of the callback. A takeover moves an admitted save to superseded, so its next attempt is refused, and every attempt ends in one Cmd: write or refuse. The tests add four paths, 10 repeats each, red against the old table: rolled back and called again with no later run (lands), after a later run saved (refused), a snapshot rolled back and called again (refused), and a refused write called again (refused again). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ruling [invariant 1] The amendment had rewritten ADR 0025's Cost on a late save. A decision record keeps what it ruled: the text is restored from before the fence change, and only the status line points at ADR 0042, as for ADR 0017 and ADR 0041. The amending ruling, and what it leaves uncovered, now lives in ADR 0042's Consequences. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Base automatically changed from
can/tea-host-resumes-running-and-stops-first
to
main
November 21, 2026 00:18
…og-idempotent-start # Conflicts: # .changeset/agent-host-boot-gate-breaks.md # .changeset/agent-host-resume-admission.md # .decisions/0040-running-agents-resume-on-wake.md # .decisions/index.md # packages/tea/docs/how-to/deploy-an-agent-to-a-durable-object.md # packages/tea/docs/reference/do.md # packages/tea/src/do/agent-host.ts # packages/tea/src/do/agent-recovery.test.ts # packages/tea/src/do/host-lifecycle.test.ts # packages/tea/src/do/host-lifecycle.ts # packages/tea/src/do/step-host.test.ts # packages/tea/src/do/step-host.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A durable agent's log is now written ahead of the work it describes, a repeated start leaves a running run alone, one request at a time reaches a run, and a save the engine abandoned can no longer overwrite a later run.
Closes #989
Built on #990, now merged; both ship in one release.
What changed
feat+fix, invariant 1, 2, 3, 6, 8): the engine hands the store the applied Msg with the State it saves (Store.save(state, msg)), anddoEventSourcedStoreappends the event there, before the Cmds run. After an eviction mid model call, the log says the call was in flight, and the wake reissues exactly that call.EventSourcedStore.appendand theruntime.observe(... append)wiring are removed.fix, invariant 1, 2, 3, 7): a start Msg for the storedrunIdon a run that is not idle changes nothing. A differentrunIdrestarts, as documented.fix, invariant 1, 6, 7, 8):stepHost'shandleserializes authenticated requests per run, so a duplicate post reads settled State.fix, invariant 1, 6):src/do/save-fence.tsholds the save lifecycle as one state × event table (issued, superseded, admitted, refused × takeOver, reach) with one interpreter. Each log has one generation per isolate:load()advances it, every save carries the generation it was issued under, and the first statement in the storage transaction refuses a stale one withStoreConflictError. The storage format is unchanged.Proof
agent-recovery.test.ts, 10 repeats on retained storage): a retried first request, two concurrent first requests, a duplicate post during a resume, and an eviction while a model call is in flight. Model attempts are [1,2,3,3,4,5]: the second 3 is the one reissue the eviction owes, since effects stay at-least-once. It fails without each of the log, start and serialization changes on its own.save-fence.test.ts): 8 per-cell tests and a count test, then 5 real-run paths ondisposeTimeoutMs: 0, 10 repeats each. 40 of 70 fail with the store unfenced, and 70 of 70 pass after.docs:reference:check,api:ratchet, and the root typecheck, lint and test (485 files, 9564 tests) all exit 0.Deleted
EventSourcedStore.append, the observer-based append wiring and its docs.src/internal/flow/workflow/do.tsand its test.Risk
FencedStore.doStoreis unchanged: itsputis issued inside the call, so nothing lands late. The platform's write ordering was not verified on Cloudflare.Changesets:
store-applied-messageminor, Additive (declaration only).do-store-write-aheadminor, Breaking (appendremoved).event-store-late-save-refusedminor, Breaking (a late save that used to overwrite a later run now rejects).agent-repeated-startandstep-host-one-request-per-runpatch. ADR 0042 is new and ADR 0025 is amended in part.🤖 Generated with Claude Code