Skip to content

Add an external backend option to the AWS emulator - #411

Merged
Railly merged 10 commits into
mainfrom
railly/aws-backend
Nov 16, 2026
Merged

Railly merged 10 commits into
mainfrom
railly/aws-backend

Conversation

@Railly

@Railly Railly commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

Problem

The AWS emulator covers S3, SQS, IAM, STS, KMS, and Cognito by hand. Projects that need broader AWS coverage have no way to use it without leaving emulate's config, ports, and Cognito flows behind.

Changes

  • aws.backend.url forwards S3, SQS, IAM, STS, and KMS to an AWS-compatible endpoint, such as AWS Emulator started with --auth off. Clients keep the emulator's port and /sqs, /iam, /sts, /kms, /s3 prefixes. Cognito, the inspector, and /_emulate stay local.
  • Requests without an Authorization header get one naming the target service, because the backend picks the service from the credential scope. Presigned requests are left alone.
  • SQS QueueUrl fields and backend redirects point back at the emulator. Bodies, including encoded ones, pass through byte for byte.
  • Seeded buckets, queues, IAM users and roles, and KMS keys are created through the backend's API once per emulator start, before the first forwarded request. Reset does not seed again, and resources from an earlier start are kept. A seed failure or an unreachable backend returns 502 on forwarded requests.
  • account_id and KMS key_id fail at startup with backend, since the backend assigns them.
  • The inspector's S3, SQS, and IAM tabs name the backend.
  • The AWS SDK e2e suites can target an external endpoint with AWS_EMULATOR_E2E_URL (plus AWS_EMULATOR_E2E_LAYOUT=root for a single-endpoint backend). STS and Cognito suites skip there because they cover emulate-specific behavior.
  • Docs: AWS page, skill, package README, root README, and CLI help.

Visual proof

Same config and script against AWS Emulator on 127.0.0.1:9297: an object uploaded through localhost:4007 lands on the backend.

Before (main ignores aws.backend):

Before

After:

After

Set aws.backend.url to proxy those services to an AWS-compatible endpoint such as AWS Emulator, keeping the emulator's port and path prefixes. Cognito and the inspector stay local. Seeded resources are created through the backend's API once per emulator start.

The AWS SDK e2e suites can also target an external endpoint through AWS_EMULATOR_E2E_URL.
@vercel

vercel Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
emulate Ready Ready Preview, v0 Nov 16, 2026 4:44am UTC

Comment thread packages/@emulators/aws/src/backend.ts Resolved
Comment thread packages/@emulators/aws/src/backend.ts Resolved
Comment thread packages/@emulators/aws/src/backend.ts Resolved
… responses

Seeding and unsigned forwarded requests now sign for aws.region, and buckets outside us-east-1 are created with a location constraint. Queue URLs on the backend's host, ListDeadLetterSourceQueues results, S3 POST <Location> bodies, and redirects keep pointing at the emulator. Compressed SQS responses pass through unchanged.

@lahfir lahfir left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me. a few small things:

Comment thread packages/@emulators/aws/src/backend.ts Resolved
Comment thread packages/@emulators/aws/src/backend.ts Resolved
Comment thread packages/@emulators/aws/src/backend.ts Resolved
Comment thread packages/@emulators/aws/src/backend.ts Resolved
Comment thread packages/@emulators/aws/src/backend.ts Resolved
Comment thread skills/aws/SKILL.md Resolved
A rejected seed stayed in the seeded WeakMap, so a reset reused the rejected promise and every later request failed until restart. Only successful seeds are kept now.
`new URL("localhost:4566")` parses with `localhost:` as the protocol, so a scheme-less backend URL passed validation and every request then failed with 502.
A bad s3.buckets[].region was reported as aws.region, which points at the wrong config key.
The service came only from emulate's path prefixes, so a bucket named kms, sqs, iam, or sts was forwarded to that service. The signed service now takes precedence and the path prefix is stripped only when it matches.
S3, IAM, and STS clients only parse XML errors, so a JSON 502 surfaced as an XML parse error and lost the message. Those services get awsErrorXml-style XML now, while SQS and KMS keep JSON.
Bare `emulate` in zsh runs the shell builtin, not the CLI.
Every describe in the cognito SDK suite is skipIf(externalTarget), so vitest never runs these hooks against an external endpoint.
Comment thread packages/@emulators/aws/src/backend.ts Resolved
A rejected seed promise stayed cached on the instance, so forwarded requests kept answering 502 after the backend came up and only a reset recovered. The seed attempt now lives in the per-store state, drops itself when it fails, and the proxy starts a new attempt; a successful seed still survives resets.
@Railly
Railly merged commit ad7b0de into main Nov 16, 2026
14 checks passed

This branch was successfully deployed

1 active deployment
Preview — a58fd31f Deployed Nov 16, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants