aws-event-driven-image-pipeline

An event-driven image processing pipeline on AWS, fully provisioned with Terraform. Uploads to S3 trigger asynchronous processing on EC2 workers that scale with the queue backlog and scale back down when the queue drains.

Problem

The fictional Bildwerk GmbH receives customer images in strongly varying volume: almost nothing at night, peaks of thousands of images during the day. The pipeline has to meet these requirements:

• processing runs automatically, without manual intervention
• capacity scales with the queue backlog, up and down
• no image is lost, even when processing fails
• workers run in private subnets, no public IPs
• IAM follows least privilege, no wildcards
• failures become visible within minutes

Architecture

An object lands in the upload bucket. S3 emits an event to EventBridge, a rule matching Object Created for that bucket forwards it to an SQS queue. EC2 workers in private subnets long-poll the queue, process the image, write a status record to DynamoDB and delete the message.

Capacity is controlled by an Auto Scaling group with a target tracking policy on a calculated CloudWatch metric: queue backlog divided by in-service instances, targeting 5 messages per instance. Messages that fail three times are moved to a dead letter queue. Two CloudWatch alarms watch the DLQ and the age of the oldest message.

Layer	Services
Network	VPC 10.0.0.0/16, 2 public + 2 private subnets across 2 AZs, IGW, NAT Gateway
Ingest	S3, EventBridge, SQS (+ DLQ)
Compute	EC2 Launch Template, Auto Scaling Group, target tracking policy
State	DynamoDB (on-demand)
Security	IAM roles, instance profile, SSM Session Manager, S3 public access block
Observability	CloudWatch dashboard, 2 metric alarms

Prerequisites

• Terraform >= 1.10 (uses native S3 state locking via use_lockfile)
• AWS CLI v2, configured credentials
• An S3 bucket for remote state, created once outside Terraform:

aws s3api create-bucket --bucket **your-name**-bildwerk-tfstate --region eu-central-1 --create-bucket-configuration LocationConstraint=eu-central-1
aws s3api put-bucket-versioning --bucket **your-name**-bildwerk-tfstate --versioning-configuration Status=Enabled
aws s3api put-public-access-block --bucket **your-name**-bildwerk-tfstate --public-access-block-configuration "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"

Deployment

cd terraform
terraform init
terraform apply

Roughly 32 resources. The NAT Gateway takes the longest to come up.

Verification

Upload an image and watch it flow through:

aws s3 cp **image.jpg** s3://**your-name**-bildwerk-uploads/test.jpg

# message arrives in the queue
aws sqs get-queue-attributes --queue-url **queue-url** --attribute-names ApproximateNumberOfMessages

# status record is written
aws dynamodb scan --table-name bildwerk-images --region eu-central-1

Load test the scaling behaviour (PowerShell):

1..60 | ForEach-Object { aws s3 cp **image.jpg** s3://**your-name**-bildwerk-uploads/load-$_.jpg }

Then poll the group every minute:

aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names bildwerk-worker --query "AutoScalingGroups[0].[DesiredCapacity,length(Instances)]"

60 messages against a target of 5 per instance drives the group to its ceiling of 4. Scale-in follows once the queue drains, deliberately slower than scale-out.

Workers are reachable through SSM Session Manager only. No SSH, no key pair, no public IP.

Cost

The NAT Gateway is the dominant cost driver — it is billed per hour plus per GB processed, and it runs whether or not anything is happening. EC2 instances add to that. Check the current rates in the AWS Pricing Calculator before leaving a stack running.

VPC, subnets, route tables, IGW, S3, SQS and an idle DynamoDB table are effectively free at this scale.

Tear down when done:

terraform destroy

For AWS-only traffic, VPC endpoints for SQS and DynamoDB would remove the need for a NAT Gateway entirely. That is the obvious next optimisation.

Lessons learned

• Target tracking on a calculated metric cannot scale from zero. The expression backlog / capacity is undefined at capacity = 0, so the metric reports no data and nothing triggers. Fixed with min_size = 1. The cleaner fix is a separate step scaling policy that starts the first instance.
• ASG group metrics are not published by default. GroupInServiceInstances only exists in CloudWatch once enabled_metrics is set on the Auto Scaling group. Without it the scaling policy is created successfully but never fires — a silent failure worth remembering.
• CloudWatch Metric Math rejects MAX([timeseries, scalar]). The guard against division by zero had to be dropped, which is what surfaced the scale-from-zero problem in the first place.
• The SQS API and CloudWatch use different names for the same thing. ApproximateNumberOfMessages for get-queue-attributes, ApproximateNumberOfMessagesVisible for the metric.
• S3 buckets only delete when empty, so terraform destroy fails on a bucket full of test images. force_destroy = true is acceptable for a lab, never for production.
• EventBridge needs to be switched on per bucket via aws_s3_bucket_notification with eventbridge = true, and the target queue needs a resource policy. Both fail silently — no error, just no messages.

Repository layout

.
├── docs/
│   ├── architecture.svg
│   └── runbook.md
└── terraform/
    ├── backend.tf        remote state
    ├── providers.tf      provider, region, default tags
    ├── network.tf        VPC, subnets, routing
    ├── ingest.tf         S3, EventBridge, SQS, DLQ
    ├── worker.tf         NAT, IAM, launch template, ASG, scaling policy
    ├── state.tf          DynamoDB
    ├── monitoring.tf     alarms, dashboard
    ├── variables.tf
    └── outputs.tf

License

MIT