WebhookRelay

A reliable webhook delivery service: ingest events over an API, fan them out to subscriber endpoints with retries, HMAC signing, idempotency, and replay.

Why this exists

This is the third in a small series of self-contained portfolio repos. BriefGenerator (Flask, OAuth2 API) and OpsDesk (Next.js full-stack app) are the other two. Unlike those, this one isn't modeled on a pattern from production work. I built it specifically to round out the backend and distributed-systems signal that a UI-heavy project can't show: queues, retries, idempotency, and observability, and I wrote it in Go instead of a third Node or Python repo.

The problem itself is a real, common one. It's the same shape as Stripe's webhook system, Svix, or Hookdeck, and I picked it because it's concrete enough to implement properly instead of as a toy. Retries need real backoff, "at least once" delivery needs real idempotency, and a queue needs to survive a crash in the middle of a fan-out.

Architecture

• API and worker are separate binaries and processes, coordinating only through Postgres, not a monolith with a background goroutine. The API holds an insert-only River client (it enqueues, but never dequeues); only the worker processes jobs. This is what lets docker compose show genuinely distributed shape, and it means ingestion stays responsive even when delivery is backed up.
• Queue: Postgres-native, using River instead of Redis or a BullMQ-equivalent. It's one less moving part to run, and InsertTx means the event write, the delivery row, and the queued job all commit, or roll back, as a single transaction. No dual-write problem between business data and the queue.
• Idempotency: events.idempotency_key is a unique DB constraint. Fan-out (creating delivery rows and enqueueing jobs) only happens on the branch where the event was newly inserted, so a retried request with the same key just returns the original event instead of double-delivering.
• Retries and dead-lettering: River's default policy is exponential backoff with jitter. Open-source River doesn't have a dead-letter-queue table (that's a paid feature), so this app keeps its own durable deliveries table. The worker explicitly marks a delivery dead-lettered when a failed attempt was also the last one allowed (job.Attempt >= job.MaxAttempts), independent of how long River itself keeps the row around.
• Replay: POST /deliveries/{id}/replay on a dead-lettered delivery tries River's JobRetry first, and falls back to a fresh Insert if that job's row has already aged out of River's retention window.
• Auth: a single static API key (Authorization: Bearer). This is a machine-to-machine ingestion API, not a human-facing app, so a full session system would be more scope than it needs.
• Rate limiting: one in-memory token bucket per subscriber endpoint. This is a known limitation, documented rather than hidden: it doesn't coordinate across horizontally-scaled worker replicas. See the Roadmap.
• Observability: structured log/slog logging, Prometheus metrics, and /health, /ready, and /metrics on both binaries, each on its own port.

Quickstart

git clone <this-repo>
cd webhookrelay
cp .env.example .env
docker compose up --build

This starts Postgres, runs migrations once, and brings up the API, the worker, and an example subscriber (examples/receiver) that verifies every webhook's signature and logs the result.

Try the full loop:

# Point a subscriber endpoint at the example receiver
curl -X POST localhost:8080/endpoints \
  -H "Authorization: Bearer dev-api-key-change-me" \
  -H "Content-Type: application/json" \
  -d '{"url":"http://receiver:9000/","secret":"demo-secret-change-me","topics":["order.created"]}'

# Ingest an event
curl -X POST localhost:8080/events \
  -H "Authorization: Bearer dev-api-key-change-me" \
  -H "Content-Type: application/json" \
  -d '{"topic":"order.created","payload":{"order_id":42},"idempotency_key":"demo-1"}'

# Watch the receiver's logs verify it, then check delivery status
docker compose logs -f receiver
curl localhost:8080/deliveries -H "Authorization: Bearer dev-api-key-change-me"

# Prometheus-format metrics from either binary
curl localhost:9090/metrics   # api
curl localhost:9091/metrics   # worker

(Running without Docker works too. Set DATABASE_URL in .env to your own Postgres, run go run ./cmd/migrate, then run go run ./cmd/api and go run ./cmd/worker in separate terminals.)

Testing

go test ./...

You don't need Docker or a system Postgres install. DB-backed tests boot a real, temporary Postgres instance using embedded-postgres, a pure-Go library that downloads and runs the actual postgres binary, migrated fresh for each test package. Coverage includes the store layer, the full HTTP API (idempotency, transactional fan-out, RBAC-style auth, replay), and the delivery worker's trickiest edge case: forcing a failure on a job's last allowed attempt and confirming it gets dead-lettered in exactly one Work() call.

Roadmap / stretch goals

• Multi-instance-aware rate limiting (Postgres- or Redis-backed instead of in-process) for horizontally-scaled worker replicas
• A minimal read-only dashboard for browsing deliveries. Deliberately API-only for now, since this repo is meant to showcase backend signal, not repeat OpsDesk's UI
• Configurable retry/backoff policy per endpoint instead of just one global setting
• Signature-verification helper libraries for a couple of common languages, mirroring what examples/receiver does by hand

License

MIT. See LICENSE.