Repository navigation
fix(runtime): guard Write/Edit against content changes before execution - #6089
jackeyfaker77 wants to merge 13 commits into
Conversation
Refs apache#4319 Generated-by: OpenAI Codex
Add Edit preparation and guarded execution, versioned mutation evidence, conversation-copy rebinding, and regression coverage for both tools. Refs apache#4319 Generated-by: OpenAI Codex
Keep package and recovery triggers aligned with their source closures. Generated-by: OpenAI Codex
Astro-Han
left a comment
There was a problem hiding this comment.
Automated review notice: This comment was posted by an automated review agent. It is not an independent human review and does not replace one.
I reviewed the complete 20-file change at 04d2b7d6c3f324e43f25dca3466ab6fa87f07375: versioned file evidence and decoding, Write/Edit preparation and execution, worker permissions/protocol, the existing T1 transaction, cancellation/locking and conversation-copy rebinding. One P2 remains: the new unconditional cleanup at packages/runtime/src/tool-runtime.ts:1922 breaks the existing successful Client Capability settlement contract and causes this PR's CI failure. The inline comment identifies the failing assertion and comparison.
The pre-dispatch evidence premise is observable, not just an issue proposal. In 18 independent child-process controls, terminating the actual Runtime process after committed T1 or before T2 left the operation prepared after SQLite reopen/rebuild. The file respectively remained unchanged or contained the expected new content. This head retained the bound before/expected-after fingerprints; removing preparation from the same builtin tool retained those file effects and operation states but no trusted file evidence. Seventy-two rehashed cross-identity substitutions were rejected, and historical events without the optional field remained readable. These observations demonstrate evidence persistence, not successful automatic recovery.
Scope and cost need a maintainer decision: the recovery registry, observer/reconciler, scan and Host recovery lifecycle are still absent. Worker-backed writes add a preparation process before execution, plus bounded hashing and result construction; production performance was not measured. Write/Edit retain never_auto_retry. The first contract covers regular UTF-8 files with before/after sizes up to 1 MiB and result JSON up to 256 KiB; unsupported observations retain ordinary execution without recovery eligibility. The in-process lock and descriptor checks are not cross-process CAS. The immediate change is therefore evidence production and guarded normal execution, not complete production file recovery.
The producer, execution guard, T1 binding and copy rebinding form one coherent contract: separating them would either persist a prediction that execution does not check or make copied events fail their identity validation. Keeping future recovery registration, reconciliation and lifecycle work separate is appropriate. Write and Edit could be stacked separately for review size, but I did not find an unrelated feature that must be split out of this patch. Whether to land this foundation before its recovery consumer remains a product/design choice, not approval implied by passing tests.
Validation: fresh dependency installation and root clean build; 307 passing tests across 19 complete Runtime/Core files covering file operations, durable boundaries, crash windows, copy and permissions. The complete settlement test file adds 13 passes and one failure, reproducing CI's extra cancel after successful admission; substituting the exact main ToolRuntime module in memory makes the same unchanged file pass all 14 tests. The other dependencies remain at this PR head, so that comparison isolates the changed module rather than claiming a full main-suite run. Core/Runtime/Host typechecks, 16-file Biome check, diff check and protocol guard passed. The independent crash probe used actual SIGKILL and the real local bypass executor/SQLite authority, not a simulated exception. Existing tests also exercised real one-shot workers and copy rebinding. Native Linux sandbox execution could not run locally because bubblewrap is absent; native macOS/Windows, the complete monorepo suite and a recovery consumer were not locally validated.
The internal filesystem worker protocol moves from 11 to 12; Runtime Host epoch remains 220 with no Host protocol change. The merge with current main 6c63fa05 is clean. Exact-head Windows packaging, recovery and W0 evidence checks and dependency audit succeeded, but CI 38395346636 failed on the settlement assertion above. This is a PR-caused failure, not one of the documented Windows baseline cleanup failures. Neither passing checks nor this review imply feature approval.
| }); | ||
| return sandboxError ? { error: msg, sandbox: sandboxError } : this.errorReturn(msg); | ||
| } finally { | ||
| await preparedExecution?.cancel(); |
There was a problem hiding this comment.
[P2] Close the settlement-contract regression introduced by cleanup
This unconditional cancel() now runs after a prepared Client Capability has executed successfully, not only when admission/T1 fails. Exact-head CI run 38395346636 fails the unchanged prepares Bypass Client Capability work before T1 and admits only after T1 test: actual order is ['prepare', 'T1', 'admit', 'cancel'], while its established success contract expects ['prepare', 'T1', 'admit'].
The complete test file reproduces this locally (13 pass, 1 fail); loading the exact 6c63fa05 ToolRuntime module in memory with the same test and other dependencies makes all 14 pass. This isolates the PR-caused CI failure. The production broker treats cancellation of an already-retired invocation as a no-op, so this evidence does not establish that completed production invocations are actually cancelled. Filesystem preparation needs reliable resource release, but its new cleanup must either be scoped appropriately or explicitly reconciled with the existing prepared-tool success contract and tests before this head is ready.
Cancel only preparations that never started execution. Once execute takes ownership, leave success and failure cleanup to the prepared executor. Preserve the existing Client Capability success contract and cover failures before execution and inside execution. Generated-by: OpenAI Codex
Retry validation after the isolated CodeMode timing test failure; no source changes. Generated-by: OpenAI Codex
Astro-Han
left a comment
There was a problem hiding this comment.
Automated review notice: This comment was posted by an automated review agent. It is not an independent human review and does not replace one.
At a0f156ff9b3a65dfeec33f30df955c1c3515d0bb, the previous P2 settlement-order finding is fixed. No additional P0–P3 issue was established in this increment.
The change transfers cleanup ownership to the prepared executor immediately before calling its execute method (packages/runtime/src/tool-runtime.ts:1648–1658). Runtime therefore cancels only preparations that never entered execution. The unchanged Client Capability success test again observes prepare → T1 → admit, without an extra cancel. Loading the exact 04d2b7d6 ToolRuntime into the same test still reproduces the old extra-cancel failure; the rest of the dependencies remain at the new head in this comparison.
The failure and cancellation paths also retain cleanup:
- A preparation cancelled after T1 publication fails, or when T1 itself fails, does not execute.
- Once execution starts, its executor releases its own resources on success, execution failure or abort; Runtime does not cancel that completed/executing object again.
- Twelve independent real Write/Edit checks cover success, T1 failure, publication failure, execution failure, abort before execution and abort during execution. Each check uses the actual filesystem preparation and SQLite commit paths, verifies the expected file contents and durable success/error/aborted outcome, then performs another prepared Write on the same target to prove that the real lock was released. The failure and abort timing are controlled test seams, not live provider incidents.
The original 18-process crash/reopen checks also pass: owned child processes are killed after committed T1 or before T2, and SQLite rebuild retains the expected evidence and operation state. The preparation-removal controls retain the same file effects but lack trusted evidence; 72 rehashed cross-identity substitutions are rejected and legacy events remain readable. These checks still do not exercise a Host recovery reconciler, native mount sandbox or cross-process compare-and-swap.
The increment contains two commits: the cleanup change and a validation-rerun commit with no source changes. The net diff affects Runtime and its settlement tests (+75/−4), with no rebase, new evidence format, or additional feature scope. A fresh dependency installation and normal clean build succeeded. Eight complete Runtime/Host test files passed 145 tests with no skips, including all 16 settlement tests and the existing Host broker/coordinator tests. Runtime and Host type checks, scoped Biome, whitespace checks and the protocol guard passed. This is not a full-monorepo, native-platform or live-provider validation.
Epoch 220 and the existing worker protocol 12 are unchanged from the preceding head; no storage-schema change was added. The head merges cleanly with fetched main 6c63fa057ed97916b77db3ff002a13636d6a564a; that synthetic merge was not separately built. At the final check, CI and audit were queued, Windows packaging was running, and the recovery and sandbox evidence checks had succeeded. This does not mean all checks are complete. The earlier feature-scope caveats remain: this PR produces evidence and guards execution, but does not implement recovery reconciliation; feature adoption and merging remain human decisions.
Preserve upstream chat image reads while advancing the prepared Write/Edit worker contract to v13. Add rejection coverage for upstream v12 workers. Generated-by: OpenAI Codex
The upstream chat-image parser requires marked@18.0.11 in production. Record the single package instance reported by Linux CI without changing the dependency-closure guard. Generated-by: OpenAI Codex
Upstream simplified the navigation-state initializer but left its nonTriviaTokens count stale. Record the measured 7280 tokens so strict base checks reflect the unchanged source tree. Generated-by: OpenAI Codex
Astro-Han
left a comment
There was a problem hiding this comment.
Automated review notice: This comment was posted by an automated review agent. It is not an independent human review and does not replace one.
At 2b3eff5a3931289f128c3e7555fdc45cea3f48d8, I found no new P0–P2 issue in the changes since a0f156ff. The earlier cleanup-ownership fix remains intact.
The merge with main 331340ab preserves all five original author commits. Its only textual conflict was the filesystem-worker version: both parents independently used v12 for different contracts. packages/runtime/src/filesystem-worker/protocol.ts:30 now uses v13 for the combined prepared Write/Edit and chat-image contract. Request and response schemas require that version, and the real client rejects a v12 response before accepting its result (client.ts:752). The separate Host compatibility epoch is 222, inherited from main; its guard reports no Host protocol change against the merge parent.
I verified the built worker through 16 actual subprocesses. Four v12 mutation requests were rejected without changing the file. For both Write and Edit, v13 preparation left the file unchanged, matching guarded execution succeeded, and an externally changed file caused execution to fail without overwriting those external bytes. Chat image reads preserved the exact bytes of a 2560×1 PNG; ordinary reads still refused that image, and oversized/invalid images retained their typed errors.
The cleanup transfer in tool-runtime.ts:1655 is unchanged. Twelve real Write/Edit scenarios again passed through filesystem preparation and SQLite persistence, covering success, T1 failure, publication failure, executor failure and cancellation before/during execution. Each case then acquired the same target's lock for another prepared Write. These use controlled failure timing; they do not test native OS sandbox enforcement or a recovery reconciler.
The two subsequent commits only add marked@18.0.11 to the packaged dependency ledger and correct the AppShell token count from 7285 to 7280. Both ledger files are byte-identical to their counterparts in #6129. These two lines duplicate/overlap that PR; rebase after #6129 lands to reconcile them.
Validation: the normal clean root build and Host type check passed before those ledger-only commits; production sources, tests, dependencies and the lockfile are unchanged by them. On the final head, all 151 focused tests and both subprocess/filesystem probes passed again. Full formatting, whitespace, strict renderer architecture (168 checker tests), artifact budget (7 tests; 242 package instances), Windows skip inventory and the epoch guard passed. This is not a full-monorepo, native Windows/macOS/Linux-sandbox or live-provider validation, and I did not rerun the earlier crash/reopen campaign.
At the final snapshot, CI, Windows packaging and sandbox evidence were still running; audit and Windows recovery had succeeded. The current head is therefore not yet all green. This remains an evidence-producing feature, not recovery reconciliation; feature adoption and merging remain human decisions.
Merge main after apache#6129 and retain its latest renderer ledger. Both CI ledgers now have no net diff against upstream; preserve the Write/Edit evidence changes. Validation: root build/typecheck/lint/format, Desktop/UI Knip, artifact budget, strict renderer architecture, 69 core regressions, ASF and protocol checks passed. Independent review found no issues. Renderer checker fixtures: 166 passed, 1 POSIX skip, 1 unchanged Windows absolute-path fixture failure. Generated-by: OpenAI Codex
Replace the fixed immediate-turn budget with bounded wall-clock polling while keeping the injected observation clock unchanged. Add a delayed-work regression that fails with the old helper. Validation: all 251 event-wait tests passed; the two CI failures passed 10 independent runs; Host build/typecheck, Biome, ASF and independent review passed. Generated-by: OpenAI Codex
Astro-Han
left a comment
There was a problem hiding this comment.
Automated review notice: This comment was posted by an automated review agent. It is not an independent human review and does not replace one.Incremental review:
2b3eff5a..50565abd, a merge of main atd14cec57plus one test-only commit. I found no P0–P2 issue, and CItestpasses on this head.
- Merge: the only hand resolution is the AppShell
nonTriviaTokensvalue inrenderer-architecture.json, which takes main's 7276. That correctly drops the CI-ledger lines this PR had duplicated from #6129. The Write/Edit evidence code is unchanged. 50565abd:eventually()inevent-wait-manager.test.tsnow polls on a 5 s wall-clock deadline with 5 ms sleeps, instead of 100 rounds ofsetImmediate. The injected observation clock stays fixed, so the timeout and Retry-After assertions stay exact. The new self-test checks that delayed host work completes without advancing that clock or firing its timers. This is a reasonable fix for a flaky test.
P3 (scope): event-wait-manager.test.ts belongs to the event-wait feature from #6113, not to Write/Edit evidence. If this is a flake on main, a small separate PR would let it land independently of this feature.
|
Thanks @Astro-Han for the review. I’ll move the event-wait test fix into a separate PR against main, run the tests and an independent agent review, then remove it from #6089 so this PR stays focused on Write/Edit evidence. I’ll link the separate PR here once it is ready. |
Remove the event-wait test-only change from the Write/Edit evidence PR now that it is proposed independently in apache#6140. Generated-by: OpenAI Codex
Astro-Han
left a comment
There was a problem hiding this comment.
Automated review notice: This comment was posted by an automated review agent. It is not an independent human review and does not replace one.Incremental review:
50565abd..c28bda56. The event-wait test change is reverted here and moved to its own PR, #6140. This resolves my scope P3. No other change. I found no P0–P2 issue, and CItestpasses.
The event-wait tests can exhaust their fixed `setImmediate` polling budget before asynchronous Host work reaches the expected checkpoint. In CI this caused the late-authorization and slow-cooldown-persistence tests to fail before their third authorization observation. Give the test helper a 5-second monotonic wall-clock deadline with 5 ms sleeps. The injected observation clock remains fixed, so existing timeout, Retry-After and cancellation assertions retain their exact semantics. A regression test verifies that delayed Host work completes without advancing that clock or firing its timers. Production code is unchanged. Extracted from #6089 following its scope review, so this test fix can land independently of Write/Edit evidence. Refs #6113. Failure evidence: https://github.com/apache/maka/actions/runs/39031806684/job/117393589335 Generated-by: OpenAI Codex
|
With #6130 now merged, I plan to narrow this PR to execution-time file-content consistency checks, aligning it with the current recovery contract. |
Merge upstream main and replace persisted recovery evidence with execution-time preconditions. Preserve permission and lifecycle boundaries, compare file state before mutation, and keep guarded Edit matching failures actionable. Remove recovery contracts, dispatch binding and conversation-copy rebinding; retain upstream T1/T2 schemas and never-auto-replay behavior. Generated-by: OpenAI Codex
Astro-Han
left a comment
There was a problem hiding this comment.
Automated review notice: This comment was posted by an automated review agent. It is not an independent human review and does not replace one.
Review at c41ca330f129f9e1836e16bd81216a76fc715d24. I found no new P0–P2 issue. I read the complete 17-file diff against main 2b0e4a25, the merge's additional resolutions and the removal of the earlier persisted-evidence path. The current implementation captures transient preconditions before T1 and checks the file descriptor's identity, length and content digest before mutation (file-mutation-guard.ts:215). Core contracts, storage, conversation copying, dependency manifests and the lockfile are byte-identical to current main. The previous concern about landing persisted recovery evidence before its consumer is addressed by removing that evidence, dispatch binding and copy rebinding. T1/T2 retain main's schemas and never_auto_retry behavior.
The narrower correctness premise is independently observable. Through the actual ToolRuntime, builtin Write/Edit, local filesystem executor and SQLite T1/T2, I changed an existing file on the same inode immediately after T1 committed. Both tools rejected execution and preserved before external\n, recording an ordinary error outcome. Loading only the actual main builtin-tools module instead makes Write replace that content and Edit change it to after external\n. This comparison isolates the missing preparation step while retaining the current dependencies and Runtime; it is not a full main-suite comparison. No model or native OS sandbox was used in these checks.
All 219 tests across 12 complete Runtime files passed, including real worker guard/error cases, read-only permission rejection, directory/target replacement, private argument isolation, unsupported observations, settlement and durable-boundary behavior, copy, locking and mutation outcomes. A separate 18-subprocess check of the built worker bundle rejected v12/v13 requests before mutation, preserved external bytes on stale guards, accepted matching guards and retained the chat-image read contract. Guarded missing/ambiguous Edit matches retain edit_conflict and the ordinary error message. Twelve actual prepared Write/Edit lifecycle scenarios also passed with SQLite outcomes and same-target lock reacquisition after success, T1/publication/executor failure and cancellation. The earlier successful Client Capability cleanup contract remains intact.
The normal root clean build passed after a fresh dependency installation and dependency patches. Runtime/Core/Host and five Desktop TypeScript configurations, Biome on 14 changed code files, ASF headers, six Windows workflow/inventory tests, Windows skip inventory, strict renderer architecture with 168 checker tests, artifact budget (242 package instances) and whitespace checks passed. The internal filesystem worker protocol is 14; the separate Host epoch remains 227, and its guard reports no Host protocol change. This head already includes current main and is mergeable.
The documented limits remain material: protection starts at preparation, regular UTF-8 pre-state/input strings are bounded to 1 MiB, unsupported observations use ordinary execution, and the final check/write still has a cross-process race. Worker-backed calls retain the extra preparation subprocess. I did not independently reproduce the author's Windows latency measurements, rerun the complete monorepo suite or qualify native Linux/macOS/Windows sandbox enforcement. Earlier crash/reopen evidence tests no longer describe this implementation and were not reused as acceptance evidence.
At this snapshot, exact-head CI test is still running; Windows packaging, Windows recovery and W0 protocol checks succeeded. Final CI still needs to complete before treating this head as all green.
|
I will make a small follow-up to reduce overhead within the current transient-guard design:
This will retain the two-process worker design, pre-T1 permission checks, directory/target/content validation, private execution arguments, locking/cancellation ownership and ordinary fallback for unsupported observations. It will not add a worker pool or change recovery behavior. I will run the affected tests and an independent automated review before pushing the follow-up. One clarification about the existing latency figures: the microbenchmark used unbundled worker-entry modules and an injected worker adapter, not the production bundled worker/client and native sandbox path. Those measurements demonstrate overhead in that harness; they are not released-application latency estimates. These small changes do not claim to eliminate the extra process-start cost. |
Compute input eligibility from private arguments before file observation and send only preparation metadata through the v15 worker protocol. Keep permission checks for ineligible inputs and preserve the two-process execution lifecycle. Hide the prepared guard interface, remove redundant Write wrappers and avoid copying freshly created results. Cover byte limits, fallback, private snapshot isolation, permissions and wire compatibility. Generated-by: OpenAI Codex
Astro-Han
left a comment
There was a problem hiding this comment.
Automated review notice: This comment was posted by an automated review agent. It is not an independent human review and does not replace one.
Incremental review at 7b801b4680e58688f944ea7f16629837b20b4192, following c41ca330. I read all nine changed files and found no new P0–P2 issue. Input eligibility is now computed from the private argument snapshot before content observation; preparation transports only kind, target and eligibility. The guard stays inside the prepared executor, and removing the Write-only wrappers/result clone preserves the shared execution path. Builtin tools, ToolRuntime, the permission client, dependencies and storage are unchanged by this commit.
All 224 tests across the same 12 complete Runtime files passed. The new cases cover the exact UTF-8 byte boundary and invalid surrogate inputs, ordinary execution for oversized inputs, private caller/backend-object isolation, metadata-only request shapes, write permission checks even when eligibility is false, rejected legacy wire versions and actionable Edit errors. The strict v15 schema rejects preparation payloads containing replacement text. The internal worker protocol advances from 14 to 15; the separate Host epoch is unchanged.
I also reran 20 actual built-worker subprocesses, including v12/v13/v14 rejection before mutation, matching/stale guards and chat-image behavior; all passed. Twelve real prepared Write/Edit lifecycle scenarios passed with SQLite outcomes and same-target lock reacquisition. The actual ToolRuntime/SQLite check again preserves externally changed bytes for both tools after T1; loading only main's builtin module instead applies the writes to those bytes. These checks confirm the retained execution protection and cleanup. They do not measure production latency or native sandbox enforcement.
Fresh dependency installation/patching, the normal clean root build:test, Runtime/Core/Host type checking, Biome on eight changed code files, ASF headers, six Windows workflow/inventory tests and whitespace checks passed. A static merge with main ca4ac75c9d157bb49bd7d0feae975f2574a7ca55 is clean and passes the Host epoch guard at 227; that merged tree was not separately built. The documented extra preparation process, 1 MiB eligibility limits, ordinary fallback and cross-process check/write race remain.
Exact-head CI test and Windows packaging are still running; Windows recovery and W0 checks succeeded. The preceding commit's green CI does not cover this new head. I did not repeat the full monorepo suite, native OS sandbox qualification or performance measurements for this increment.
Summary
Write/Edit can currently proceed after another writer changes a file during the gap before execution. This PR captures transient preconditions before T1 and checks the file descriptor's identity, byte count and content hash before mutation. A detected change rejects the operation without overwriting the changed bytes.
The scope has been narrowed following #6130 and the retirement of the old reconciliation roadmap. This revision removes this PR's persisted recovery evidence, Core contracts/exports, dispatch binding, conversation-copy rebinding and recovery-roadmap additions. It does not reintroduce a bundle writer, recovery registry or reconciler. T1/T2 schemas and the current unknown-outcome / no-auto-replay policy stay unchanged.
edit_conflicterrors.Boundaries and cost
This protects changes after preparation, not changes since the model's earlier Read. Eligible pre-state files are regular UTF-8 files up to 1 MiB; mutation input strings are bounded to 1 MiB each. The process-local lock does not exclude external writers, and checking then writing is not an atomic cross-process compare-and-swap. Both tools remain
never_auto_retry.Worker-backed preparation still adds a second subprocess. A Windows / Node 24.20.0 filesystem-executor microbenchmark (Write/Edit, 4 KiB and 256 KiB, two warm-ups; 7 worker or 21 local samples per case) measured these ranges of case medians:
2b0e4a25) ordinary executionc28bda560)c41ca330f)Baseline comparisons used isolated snapshots of the relevant source modules with shared dependencies, not separate clean builds. The harness launched unbundled worker-entry modules through an injected adapter; it did not exercise the production bundled worker/client path. The table predates the payload optimization; no new latency improvement is claimed for it. These are local microbenchmarks without SQLite/T1 or native OS sandbox enforcement, not production latency estimates. Narrowing removes unused persistence and duplicate computation but does not eliminate preparation overhead or improve every measured case.
Validation
Current revision:
7b801b468, following the reviewed narrowing atc41ca330f(which includes main at2b0e4a25).edit_conflictand its original message. Re-review independently passed the worker and lifecycle checks and found no remaining P0–P2 issue. This is automated review, not human approval./fixture/...paths; its test and plugin are identical to main. The actual architecture check passed with--base upstream/main --strict-base.AI use
OpenAI Codex assisted with implementation, tests, documentation, measurements, review and this description. Affected commits retain
Generated-by: OpenAI Codextrailers.Checklist
The second checkbox remains unchecked because of the baseline Windows failures described above.
Does this PR entail a change in behavior?