SafeRetries

A small .NET demo that shows what goes wrong with retries, and how to fix it.

There are two projects:

• FlakyService - a fake downstream API you can make slow or broken on demand.
• SafeRetries.Web - the demo itself. It serves a small web page with a button per scenario, and prints a live log of every retry to the console at the same time.

What the demo shows

1. Manual retry loop (no library). 5 requests to a slow service, each retried 3 times. The service gets 15 calls. Retries multiply the load on a service that is already struggling.
2. Standard resilience handler against a failing service. 120 requests. The circuit breaker opens and the later calls fail fast instead of piling on.
3. POST with retries left on. The service saves the payment, then the reply is slow. The client times out and retries. Three payments are created from one request.
4. POST with DisableForUnsafeHttpMethods(). Same setup, but POST is not retried. Only one payment is created. The call still fails with a timeout, and that is fine.
5. POST with retries on plus an Idempotency-Key. The request is still retried, but the service recognises the key and returns the first result. Only one payment is created.
6. A retry doing its job. The service fails one call, then recovers. It is a GET, so retrying is safe, and the caller never sees the glitch. This is what retries are for.

How to run it

You need the .NET 10 SDK.

Start the service first, in one terminal:

cd FlakyService
dotnet run

It listens on http://localhost:5100.

Then run the demo in a second terminal:

cd SafeRetries.Web
dotnet run

Now open http://localhost:5000 and press a Run button, or "Run all 6".

The page shows live counters and a log of every retry as it happens. The same log is printed in the demo's console, so you can watch either one. The demo resets the service and sets the right mode before each scenario, so you can run them in any order, as many times as you like.

The flaky service

It runs on http://localhost:5100 and keeps everything in memory.

Endpoint	What it does
GET /data	Returns some JSON. Counts every call.
POST /payments	Creates a payment and returns 201. Counts every call. Honours an optional Idempotency-Key header.
POST /mode	Changes how the service behaves.
GET /stats	GET calls, POST calls, payments created.
POST /reset	Clears the counters and payments, sets mode back to normal.

Modes:

• normal - replies straight away
• slow - waits 5 seconds before replying
• failing - returns 500 every time
• blip - fails the first call only, then works normally
• slow-after-save - saves the payment first, then waits 5 seconds before replying. This is the one that causes duplicates: the work succeeded, only the reply was late.

You can poke it yourself while it is running:

curl http://localhost:5100/stats

Sample output

=== Scenario 1: manual retry loop (no library), slow service ===
Requests sent by client: 5
Calls received by service: 15   <-- retries multiply the load
Every call timed out after 3s
Time taken: 9.0s

=== Scenario 2: standard resilience handler, failing service ===
Requests sent by client: 120
Calls received by service: 100
Failed normally (500 after retries): 25
Failed fast (BrokenCircuitException): 95   <-- breaker is open
Succeeded: 0
Time taken: 5.9s

=== Scenario 3: POST with default retries ===
Requests sent by client: 1
Calls received by service: 3
Payments created: 3   <-- duplicate!
Result: TimeoutRejectedException
Time taken: 6.5s

=== Scenario 4: POST with DisableForUnsafeHttpMethods ===
Requests sent by client: 1
Calls received by service: 1
Payments created: 1   <-- no duplicate
Result: TimeoutRejectedException (a failure here is expected)
Time taken: 2.0s

=== Scenario 5: POST with retries plus an Idempotency-Key ===
Requests sent by client: 1
Calls received by service: 2   <-- it was still retried
Payments created: 1   <-- but only one payment
Result: HTTP 201
Time taken: 2.3s

=== Scenario 6: a retry doing its job ===
Requests sent by client: 1
Calls received by service: 2   <-- first failed, second worked
Result: HTTP 200   <-- the retry saved it
Time taken: 0.1s

Notes on the settings

If you change the resilience options, keep these rules or the app will fail to start:

• The total request timeout must be longer than the attempt timeout.
• The circuit breaker sampling duration must be at least double the attempt timeout.