Skip to content

fix(cli): upload server files in parallel, with timeouts and retries - #141

Merged
linyiru merged 2 commits into
mainfrom
fix/cli-parallel-serverfile-upload
Nov 3, 2026
Merged

linyiru merged 2 commits into
mainfrom
fix/cli-parallel-serverfile-upload

Conversation

@linyiru

@linyiru linyiru commented Nov 3, 2026

Copy link
Copy Markdown
Member

Problem

A customer's vinext production deploys (321 modules, 14 MB worker) hung or failed at Uploading bundle... with fetch failed / read ETIMEDOUT / read ECONNRESET, each leaving a deployment stuck in queued.

Production data (D1 + audit log + build logs) for that project:

  • Up to 2026-10-12 it deployed with a non-vinext build: upload 20-30 s.
  • From 2026-10-13 (first framework=vinext deploy) every upload took 350-1015 s; 50 of 92 CLI deploys since then were left queued.

Cause: stageDeploymentBundle sent one PUT /serverfile per module, sequentially, with no timeout and no retry. Each request waits out auth + several D1 round trips to the WNAM primary (~1-3 s from Taiwan), so the upload is latency-bound (it looked like "a few KB/s"), and any stall or reset in a 10+ minute run of 300+ requests killed the deploy.

Change

  • uploadServerFiles (packages/cli/src/utils/upload-server-files.ts): 8 requests at a time, at most 32 MB in flight (a larger single file still goes alone).
  • Each attempt is aborted after 60 s + 16 KB/s of its size. A network failure, a timeout, a 429 or a 5xx is retried up to 3 times with jittered backoff (1, 2, 4 s). PUT /serverfile only writes staging under the file's name, so a retry is safe even when the first attempt was applied. Other 4xx stop the upload; the missing-route 404 fallback to the inline bundle is kept.
  • After the first failure, no new file starts; the error surfaces once in-flight requests settle. A final timeout surfaces as fetch failed / ETIMEDOUT so the existing unreachable reporting applies.
  • Progress: one line per tenth of the files, a warning per retry, a summary. Nothing in --json mode.
  • SDK: uploadServerFile(..., { signal }). Release needs an sdk bump with the cli.

The final PUT /bundle is unchanged and not retried: it starts the deploy job, so it is not idempotent.

Verification (live, production api.creek.dev, 2026-11-02, from s-imac in the US)

Real vinext app (create-vinext-app, 147 routes), built once, deployed with --prod --skip-build to a test project vx-upload-e2e under the admin account. 214 server modules, 6.5 MB.

run server-file upload upload (server side: create -> bundle received) result
creek@0.4.55 (sequential) - 122.7 s active
this branch 16.2 s 17.5 s active, verified 200
this branch through a fault-injecting proxy 62.0 s - active, verified 200
this branch, --json - - stdout parses as JSON, no progress lines

The fault proxy sat in front of api.creek.dev and, for /serverfile PUTs, (#3) reset the connection, (#7) never answered, (#11) answered 502, and (#15) forwarded to the real API and then dropped the 200. The CLI retried all four (218 PUTs for 214 files) and the deploy went active. The 62 s is the stalled request's 60 s timeout. After that deploy, all 147 routes served their own <h1>page N</h1>, and /api/hello answered.

pnpm test (3720 passed), typecheck and format:check pass.

Not in this PR

  • The stale-deploy sweeper ignores queued, so a deployment whose upload never finished stays queued forever.
  • The per-request auth/D1 round trips that make each /serverfile slow from Asia.

Lets a caller bound one server-file upload with a timeout.
A vinext worker is hundreds of modules, and each went up as its own PUT
/serverfile, one after another, with no timeout or retry. Every request
waits out the API's auth and D1 round trips, so a 321-module worker took
6-15 minutes from Taiwan, and one stalled or reset connection hung or
failed the whole deploy, leaving the deployment queued.

Upload them 8 at a time (at most 32 MB in flight), give each attempt a
timeout of 60 s plus 16 KB/s of its size, and retry a network failure,
a timeout, a 429 or a 5xx up to 3 times with backoff. PUT /serverfile
only writes the file to staging under its name, so a retry is safe even
when the server applied the first attempt. Other 4xx still stop the
upload, and a missing-route 404 still falls back to the inline bundle.

Progress prints one line per tenth of the files, a warning per retry,
and a summary; nothing in --json mode.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The bounded parallel upload changes have no active review issues.

No open issues 🎉

What changed in this PR

Improves CLI deployment reliability and speed for workers with many server modules by replacing sequential uploads with bounded parallel uploads.

Changes:

  • Uploads up to eight files concurrently, with a 32 MB in-flight limit, size-based timeouts, and retries for transient failures.
  • Adds upload progress reporting and SDK abort-signal support while preserving the legacy inline-bundle fallback and leaving the final bundle request unchanged.
  • Adds tests for concurrency, retries, timeouts, failure handling, and progress output.
File Description
packages/​sdk/​src/​client/​index.ts Accepts an abort signal for server-file uploads.
packages/​cli/​src/​utils/​upload-server-files.ts Implements bounded parallel uploads, timeouts, and retries.
packages/​cli/​src/​utils/​upload-server-files.test.ts Tests upload limits, recovery, error handling, and progress reporting.
packages/​cli/​src/​commands/​deploy.ts Integrates the uploader and progress reporting into deployment staging.
packages/​cli/​src/​commands/​deploy.test.ts Verifies exhausted retries propagate errors without uploading the bundle.

Review effort: Balanced

@linyiru
linyiru merged commit 7a62cb2 into main Nov 3, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants