feat: stage and commit object writes with content ETags

.agents/environment.md

@@ -454,11 +454,18 @@ until something asks:
   in practice; the test stamps its fixtures on whole milliseconds and asserts
   the drift case separately.
 
-- **`rclone check` cannot compare hashes here.** The gateway's ETag is the
-  first 32 hex of sha256 over `dev:ino:size:mtimeMs` with a `-1` suffix, which
-  rclone reads as "not a plain MD5": it reports `N hashes could not be checked`
-  and falls back to **size plus modification time**. `--size-only` is the
-  comparison with no hash in it at all.
+- **`rclone check` compares hashes here.** The gateway's ETag for an object it
+  wrote is the content MD5, so plain `rclone check` announces `Using md5 for
+hash comparisons` and actually does them: `0 differences found` on a matching
+  tree, and nothing reported as unchecked. It catches a change that size and
+  modification time cannot see — same length, same stamp, different bytes —
+  which is what the oracle case now pins. This used to read the other way: while
+  the ETag was the first 32 hex of sha256 over `dev:ino:size:mtimeMs` with a
+  `-1` suffix, rclone read it as "not a plain MD5", reported `N hashes could not
+be checked` and fell back to size plus modification time. The fallback is still
+  reachable — an object this process never wrote, one after a restart, one the
+  bounded table evicted — and `--size-only` is still the comparison with no hash
+  in it at all.
 
 ## davfs2, the WebDAV mount client (installed 2026-07-31, this Linux host)
 

.agents/invariants.md

@@ -229,3 +229,57 @@ encode/decode bug.
 that file's host line with them. That covers `README.md` and `docs/` alike; the
 README links to the docs rather than repeating the numbers, so
 `docs/1.guide/6.tuning.md` is where they live.
+
+## The HTTP write path
+
+**Object writes over HTTP commit through `src/commit.ts`, and the ETag of anything
+written there is its content MD5.** `mountx/s3` and `mountx/webdav` serve the same
+driver and had the same three holes in the same place, so there is one `write()` and
+both call it for every body they store — `PutObject`, `UploadPart`, a multipart
+`Complete`, a `CopyObject`'s bytes, a `PUT`, a `COPY`'s bytes.
+
+**Why a bypass of `write()` reintroduces the stale-`If-Match` hole.** The validator
+before this existed was `sha256("dev:ino:size:mtimeMs")`, first 32 hex — still
+`derivedETag()`, still the fallback. It repeats for two same-size writes inside one
+filesystem timestamp tick, so a client's stale `If-Match` compared equal and the
+write was accepted. Measured through the session: roughly 180 of 200 stale writes
+accepted on ext4 under Linux 5.15 and 6.1, 200 of 200 on vfat under any kernel, 0 of
+200 on ext4 under 6.18 — whose multigrain timestamps (Linux 6.13) hide it, which is
+exactly what makes it easy to miss on a dev box while LTS distributions still ship
+pre-6.13 kernels. The memory and unstorage drivers had the same repeat until
+`src/drivers/clock.ts` gave them the same rule. A finer clock is not the fix: the fix
+is recording what the bytes hashed to. A write that goes straight to `driver.open()`
+records nothing, so the next read of that key falls back to the derived tag and
+compares two different contents equal again.
+
+**Why staging is capability-gated.** There is no atomic _replace_ in `FsDriver`.
+`link()` is an atomic create from a file that is already whole and a declared
+`atomicRename` `rename()` is an atomic replace, so the two compose into one — but
+only for a driver that has one of them _and_ a `mkdir` to make the staging root with.
+`unstorage` has neither primitive (no `link`; its `rename` is copy-then-delete), so
+it is written in place and the weaker guarantee is written down: the first byte, and
+past it a reader can see a partial object. Claiming otherwise would be invariant 5's
+faked capability, and the caller can tell which shape it got from the driver's own
+capabilities (`canStage()`). `EXDEV` from either primitive — a `node-fs` root that
+spans a mount point — falls back to an in-place copy of the already-verified staged
+body. A replace carries the destination's mode across and cannot carry its ownership;
+`uid`/`gid` need root and nothing here runs as root.
+
+**Why the table must be updated under the key's lock.** A recorded tag is only
+trustworthy because no other write to that key was in flight while it was recorded.
+`ObjectTable.serialize()` is a promise chain per key — not `src/lock.ts`'s
+`PathLock`, which is one writer against every reader of a whole path map and would
+make a server with ten clients behave like a server with one — and the _whole_ write
+runs inside it: the compare, the body and the swap, conditional or not. That is what
+stops an unconditional `PUT` landing between an `If-Match` compare and its swap, and
+it is why `settle` (the `x-amz-meta-mtime` `utimes`) runs before the `stat` that is
+recorded rather than after `write()` returns: `mtimeMs` is one of the four fields a
+record is believed by, so a later `utimes` would make the very next read disbelieve a
+record describing bytes nobody touched. Deletes, directory markers, a copy's
+destination and a metadata-only copy onto itself all run on the same chain for the
+same reason.
+
+The bound is `etagCacheEntries` (default 65536). Eviction costs one spurious `412`
+and a re-read, never a false `200`: a record is dropped rather than believed the
+moment `dev:ino:size:mtimeMs` stops matching, which is also how an external writer is
+noticed.

.agents/roadmap.md

@@ -243,20 +243,30 @@ area whose code it changes, not the area that motivated it.
 
 ## S3
 
-- **The multipart pair takes no conditionals.** `PutObject` evaluates
-  `If-None-Match`, `If-Match` and `If-Unmodified-Since` and answers `412`/`404`
-  (issue #19); `CreateMultipartUpload` and `CompleteMultipartUpload` still
-  ignore them, so a client that assembles an object in parts cannot express the
-  same compare-and-swap. The condition belongs on `Complete` — the moment the
-  key changes — and the key lock `#putObject` takes is already the right
-  granularity for it. No client observed needs it, which is why it is here
-  rather than done.
-- **A conditional `PUT` is a compare-and-swap within one process.**
-  `If-None-Match: *` is the driver's own (`O_CREAT|O_EXCL`, atomic against any
-  writer anywhere), but `If-Match` compares and then writes under a per-key
-  promise chain, so an _unconditional_ `PUT` — which takes no lock — can still
-  land between the two. Closing that needs every writer of a key on the same
-  chain, which is a cost on the ordinary path for a race no client has hit.
+- **The `x-amz-checksum-*` family is not verified.** crc32, crc32c, sha1 and
+  sha256, as request headers or as `aws-chunked` trailers, are carried no
+  further than any other header. `Content-MD5` **is** verified, on `PutObject`
+  and `UploadPart`, and the plumbing the rest would need is already there:
+  `write()` takes an `expectMd5` and answers `CommitError("digest")` from
+  inside the staged body, so a second algorithm is a second hash beside the MD5
+  in `drain()` and a second option beside `expectMd5` — not a new shape. What
+  it wants first is a client that sends one. The trailer form is the fiddlier
+  half: a trailing checksum arrives after the last chunk, so it can only be
+  compared where the digest already is, at the end of the body and before the
+  commit.
+- **A `mountx.*` compare-and-commit extension, for a driver whose store has its
+  own CAS.** `write()` composes an atomic commit out of `link`/`rename` because
+  `FsDriver` is a subset of `node:fs/promises` and that is all POSIX gives it.
+  A driver over a store with a native conditional put — an S3-compatible bucket,
+  an object store with an `If-Match` of its own, a KV with a compare-and-swap —
+  can do the whole thing in one round trip, and `write()` would prefer that call
+  where a driver declares it, exactly as it prefers `rename` over a copy today.
+  It is also the only path to compare-and-swap **across processes**: today the
+  guarantee is one gateway process per bucket, because `If-Match` is decided
+  under a lock this process holds. (`If-None-Match: *` is already
+  cross-process — it is the driver's own exclusive create.) That limit is
+  deliberate and documented rather than latent; what would lift it is a store
+  that can be asked, not a lock that spans machines.
 
 ## WebDAV
 
@@ -313,10 +323,11 @@ area whose code it changes, not the area that motivated it.
   define.
 - **The two HTTP servers duplicate their transport mechanics.**
   `src/webdav/server.ts` and `src/s3/server.ts` track connections, drain on
-  `close()` and write a streaming reply the same way, deliberately not shared
-  yet: the bind refusal's wording, the fallback error reply and the
-  authentication are each transport's own. If a third HTTP-shaped transport
-  appears, this is the duplication to remove first.
+  `close()`, call their session's `close()` after the drain and write a
+  streaming reply the same way, deliberately not shared yet: the bind refusal's
+  wording, the fallback error reply and the authentication are each transport's
+  own. If a third HTTP-shaped transport appears, this is the duplication to
+  remove first.
 
 ## Platforms
 

.agents/testing.md

@@ -115,6 +115,27 @@ test:9p:mount` / `pnpm test:root`) — 9P has no unprivileged route on any host,
   and the Tier-1 JS client (`client.ts`, which does its own path-walking and
   POSIX-vs-NFSv4 op-collapsing — `unlink` vs `rmdir`, OPEN not being for directories)
   plus `driver.ts` (the `FsDriver` over it) and `conformance.test.ts`.
+- `test/commit.test.ts` — Tier 0 for `src/commit.ts`, and the file that pins **both
+  driver shapes rather than one**. Three columns — `memory` and `node-fs` (staged:
+  hardlinks and/or an atomic `rename`, plus a `mkdir`) and `unstorage` (in place) —
+  run the same `write()` cases, and the two shape-specific blocks are named for
+  what they claim: `(staged shape)` asserts that a reader mid-write sees the whole
+  old object and that a failed body leaves it untouched with no debris,
+  `(in-place shape, the documented limitation)` asserts the opposite where the
+  driver cannot commit, so the weaker guarantee is written down rather than hidden.
+  Four synthetic drivers cover the routes no real one reaches: `link`/`rename`
+  crossing a mount point (`EXDEV`), an atomic `rename` with no hardlinks, a driver
+  with no `mkdir` to stage into, and a destination that cannot be `stat`'ed. Plus
+  `ObjectTable` alone — the identity check, the LRU bound, the per-key chain, the
+  injected staging names — and `sweepStaged`.
+- `test/clock.test.ts` — Tier 0 for `src/drivers/clock.ts`'s `nextStamp()`, away
+  from either driver that applies it: the wall clock is an argument, so the edges
+  the drivers' own suites cannot show are reachable — a clock that has not moved,
+  one that has stepped backwards, and a stamp `utimes` put in the future. The
+  drivers' halves are in `test/memory.test.ts` and `test/unstorage.test.ts`
+  (`timestamps`): every write of one file gets a stamp of its own, a directory gets
+  one per entry it gains, an explicit `utimes` is stored to the float and never
+  stepped past.
 - `test/xml.test.ts` — Tier 0 for the shared codec's own contract: namespaces, in
   both directions, against Namespaces in XML 1.0. Prefix resolution and scoping, the
   two rules that are deliberately lenient rather than conformant (an unbound prefix
@@ -131,6 +152,28 @@ test:9p:mount` / `pnpm test:root`) — 9P has no unprivileged route on any host,
   `oracle.test.ts` — a real `rclone`/`curl` against the gateway, gated on `command -v
 rclone`/`curl` and needing no root, so it runs as part of `pnpm test` and skips
   clean when either binary is absent.
+  Three blocks in `session.test.ts` cover the write path. **`a stale conditional
+write cannot win`** is the one that matters most, and it has two halves: fifty
+  rounds of a client's own compare-and-swap loop on an ordinary memory driver, and
+  the same fifty with the `stat` identity **frozen** — same `dev:ino:size:mtimeMs`
+  on every version, same body length — so the derived tag cannot separate two
+  versions and the only thing left that can is what the bytes hashed to. That
+  second case is the clock-independent pin: it fails on any host and any kernel
+  when a write bypasses the table, where the first would pass on a filesystem with
+  a fine enough stamp. Beside them: an unconditional `PUT` that must not land
+  between a conditional one's compare and its swap, an `If-None-Match: *` losing to
+  a creator that arrived after the check, a reader seeing the whole old object
+  mid-write, and eviction falling back to the derived tag. **`Content-MD5 on a body
+this gateway stores`** covers `PutObject` and `UploadPart`, the `InvalidDigest`
+  that reads nothing, the `BadDigest` that leaves the object alone, part ETags and
+  S3's md5-of-md5s, and the conditionals on `Complete`. **`over a driver that
+cannot commit atomically`** runs the same claims against `unstorage` and pins the
+  partial object a reader can catch there, named as that driver's limitation.
+  `oracle.test.ts`'s `rclone check` case was rewritten from its opposite: it used
+  to assert `N hashes could not be checked`, and now asserts `Using md5 for hash
+comparisons`, `0 differences found` and nothing unchecked — then changes a file
+  to the same length on the same stamp, which `--size-only` correctly calls
+  identical and the default check correctly calls `md5 differ`.
 - `test/webdav/` — Tier 0/1, all of it socket-optional: `protocol.test.ts` (the
   three request grammars, the `If` header's disjunction-of-conjunctions, the
   `multistatus`/`error`/lock documents, and the target↔`href` mapping round-tripped
@@ -153,6 +196,20 @@ rclone`/`curl` and needing no root, so it runs as part of `pnpm test` and skips
   misreading of RFC 4918 — including the whole class-2 round trip, where curl takes
   a lock on an unmapped URL, is refused `423` for an untokened `PUT`, and gets
   through with an `If` header. **No conformance column yet** — see Known gaps.
+  `session.test.ts`'s write-path blocks mirror the S3 ones: **`the ETag a resource
+answers`** (the content MD5, the derived fallback, a tag travelling with a `MOVE`
+  and forgotten by a `DELETE` so a same-size reseed is not the deleted resource's
+  tag), **`PUT stages the body, and commits it`** — whose stale-`If-Match` case is
+  the same clock-independent pin, fifty rounds against a frozen `stat` identity
+  with `etagCacheEntries: 0` as the shape of the old bug — plus the `405` for a
+  collection that appeared between the check and the commit and the create-only
+  `PUT` that loses, **`the reserved staging root`** (absent from a `PROPFIND` of
+  the share, `404` for every method that names it, `403` as a `COPY`/`MOVE`
+  destination), **`close()`** (sweeps `tmp-*` and nothing else; reports a staging
+  root it cannot read rather than throwing on the way out), and **`a driver that
+writes in place (unstorage)`**, which pins the partial resource as that driver's
+  limitation. `server.test.ts` adds the socket-level half: the server sweeps the
+  bodies a dead process staged, after its drain.
 - `test/webdav/mount.test.ts` — Tier 2, and the only test in the package that puts a
   **kernel** in front of this server: `mount.davfs` (davfs2, over FUSE) mounts the
   share and the workload is ordinary syscalls, not WebDAV. Gated on the in-file
@@ -192,7 +249,10 @@ rclone`/`curl` and needing no root, so it runs as part of `pnpm test` and skips
   `.agents/environment.md` § VM guests. Turning that into a Tier-2 column would mean
   carrying a VM in the suite.
 - **No NFSv4.1 or S3 benchmark column**, and the 9P one is from a later sitting than
-  the rest — see `.agents/benchmarks.md`.
+  the rest — see `.agents/benchmarks.md`. What the commit path cost was measured
+  through the session by hand and recorded in `.agents/architecture.md`, which is
+  not the same thing: it is not reproducible with `pnpm bench` and invariant 24
+  therefore keeps those numbers out of `docs/`. A column is what would change that.
 
 ## Runner scripts