Skip to content

test(e2e): stop transient dev server crashes from failing E2E runs - #4269

Draft
kdaviduik wants to merge 4 commits into
kd-e2e-dev-server-exit-outputfrom
kd-e2e-retry-dev-server-startup
Draft

kdaviduik wants to merge 4 commits into
kd-e2e-dev-server-exit-outputfrom
kd-e2e-retry-dev-server-startup

Conversation

@kdaviduik

@kdaviduik kdaviduik commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

WHY are these changes introduced?

Stacked on #4268, which surfaced why E2E dev servers crash during startup.

About 5% of dev server starts crash when several Playwright workers start servers in the same project directory. React Router's dev plugin deletes and regenerates .react-router/types on every start, without retrying when the directory changes underneath it. An overlapping start fails with ENOTEMPTY or ENOENT, and the CLI exits with code 1. Playwright retries absorb most of these, but recipes gets only one retry, which is how the merge queue run for #4233 failed.

Separately, passing E2E runs take 9 to 14 minutes against a 20-minute job limit. A run with a few failing tests overruns it, GitHub reports the job as cancelled rather than failed, and the Playwright report upload (if: always() && !cancelled()) is skipped, which hides the failing tests.

WHAT is this pull request doing?

  • DevServer.start() retries a crash that happens before the server is ready, up to 3 attempts. Each retry logs the CLI's error line, so the frequency and cause stay visible in CI logs.
  • Startup timeouts and Shopify CLI login or link errors are not retried, because another attempt cannot fix them.
  • Raises the E2E job limit from 20 to 30 minutes.

Starting three dev servers at a time in templates/skeleton locally:

Starts Failed
CLI directly (current behavior) 165 8 crashed, all on .react-router/types
Through DevServer with this retry 120 0 (4 crashes retried successfully)

Out of scope

  • Preventing the race itself. The collision is in React Router's type generation, which deletes the directory with no retries. Avoiding it here would mean giving each worker its own project copy or serializing startups.

Risk

kdaviduik and others added 4 commits October 23, 2026 23:34
About 5% of dev server starts crash when several Playwright workers start
servers in the same project directory. React Router's dev plugin deletes
and regenerates `.react-router/types` on every start, without awaiting the
work, so an overlapping start fails with ENOTEMPTY (or ENOENT) and the
unhandled rejection exits the CLI with code 1. In CI this failed whole
spec files, and recipes, which get only one Playwright retry, could fail
the job outright.

`DevServer.start()` now retries a crash that happens before the server
reports ready, up to three attempts, logging the CLI's final output line
on each retry so the frequency and cause stay visible in CI logs.

Only early exits are retried. Startup timeouts already cost two minutes
and Shopify CLI login/link errors cannot be fixed by trying again, so
those still fail immediately.

Starting three servers at a time in one project locally: 8 of 165 starts
crashed through the CLI directly, all in `.react-router/types`.

Co-authored-by: AI (Pi/Claude Opus 5.5) <noreply@pi.dev>
Passing E2E runs take 9-14 minutes, but each retry of a failing test can
add a minute or more (consent specs get four attempts at 60s each). A run
with a handful of failures overran the 20-minute limit, GitHub reported
the job as cancelled rather than failed, and the Playwright report upload
(`if: always() && !cancelled()`) was skipped. The real test failures were
only visible by digging through the raw log.

Raising the limit to 30 minutes lets a failing run finish, report as a
failure, and upload its report.

Co-authored-by: AI (Pi/Claude Opus 5.5) <noreply@pi.dev>
The retry log used the CLI's last line of output, but the CLI prints its
error followed by a stack trace, so on the real skeleton the log read
"at watch (.../vite.js:1392:3)" instead of the ENOENT/ENOTEMPTY error that
explains the crash. The summary now uses the last line that names an
error, falling back to the last line when none does.

Co-authored-by: AI (Pi/Claude Opus 5.5) <noreply@pi.dev>
The dev server fixture tests log retries to the same CI output that is
searched for real startup crashes. Their simulated error was a copy of
the real React Router ENOTEMPTY error, so every run would have appeared
to contain several real crashes. The simulated error now says it is
simulated.

Co-authored-by: AI (Pi/Claude Opus 5.5) <noreply@pi.dev>
@kdaviduik
kdaviduik added this pull request to stack #4270 October 24, 2026 03:40
@kdaviduik kdaviduik changed the title kd e2e retry dev server startup test(e2e): stop transient dev server crashes from failing E2E runs Oct 24, 2026
@shopify

shopify Bot commented Oct 24, 2026

Copy link
Copy Markdown
Contributor

Oxygen deployed a preview of your kd-e2e-retry-dev-server-startup branch. Details:

Storefront Status Preview link Deployment details Last update (UTC)
Skeleton (skeleton.hydrogen.shop) ✅ Successful (Logs) Preview deployment Inspect deployment October 24, 2026 3:42 AM

Learn more about Hydrogen's GitHub integration.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant