test: catch the NextAuth signout loop and guard the shared request budget - #24
Merged
Conversation
On 2026-09-03 one client issued 228,211 requests in a day, peaking at 6,230 per minute, to /api/auth/session, /api/auth/csrf and /api/auth/signout. It exhausted the backend's per-IP rate limit and returned 429s to every other user behind the same NAT address. No test caught it. Every page rendered, every element was present, no console error appeared - the damage was entirely in the volume of requests, and nothing asserted on that. Two regression tests: - Page loads must not issue a runaway number of /api/auth/ requests. Measured on a healthy dev over 12s: 4 on the homepage, 1 on /datasets. The ceiling is 15 - well above healthy, orders of magnitude below a loop. A second assertion covers signout specifically, because an anonymous page load has no session to end, so any signout at all is the bug and no raised threshold can mask it. - A 20-request burst of GraphQL POSTs must not be throttled. That is a few page views' worth, and the budget is shared per IP - so it is shared by everyone behind a gateway. Deliberately does not try to reach the limit: doing so would spend the budget of every other client behind this address, which is the failure being guarded against. Reading the performance log rather than the DOM, and counting what the browser sent rather than what succeeded - a loop being rate-limited is still sending, so counting successful responses would hide it. The rate-limit test also documents where limiting actually lives (api/middleware/rate_limit.py). DRF's DEFAULT_THROTTLE_RATES was dead config and was misread as the cause during the incident. Verified against dev: 6 passed. Both proven able to fail - the volume test against a lowered ceiling, the burst test against a 429 host.
This was referenced Sep 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Coverage for the incident fixed in DataSpaceFrontend#453.
What happened, and why nothing caught it
One client issued 228,211 requests in a day, peaking at 6,230/minute, exhausting the backend's per-IP rate limit and returning 429 to every other user behind the same NAT address.
Every page rendered. Every element was present. No console error appeared. The damage was entirely in the volume of requests, and no test asserted on that. The existing suite would have stayed green throughout.
Tests
1. Page loads must not loop on auth requests (
smoke,regression)Reads Chrome's performance log and counts
/api/auth/requests over a 12s window.//datasetsWell above healthy, orders of magnitude below a loop.
A second assertion covers
signoutspecifically and requires zero — an anonymous page load has no session to end, so any signout at all is the bug. Split out deliberately: a future raised threshold could mask the count assertion, but cannot mask this one.Counts what the browser sent, not what succeeded — a loop that is being rate-limited is still sending, so counting successful responses would hide exactly the case that matters.
2. Ordinary request volume is not throttled (
api,smoke,regression)A 20-request GraphQL burst — a few page views' worth — must not be throttled. The POST budget is shared per IP, so it is shared by everyone behind a gateway; if this fails, the limit has been set low enough to break normal browsing from a shared office address.
Deliberately does not try to reach the limit: doing so would spend the budget of every other client behind this address, which is the failure being guarded against. A companion assertion confirms the endpoint answers normally, so a 500 on every request cannot produce zero 429s and pass vacuously.
It also documents where limiting actually lives (
api/middleware/rate_limit.py) — DRF'sDEFAULT_THROTTLE_RATESwas dead config and was misread as the cause during the incident (removed in DataSpaceBackend#143).Verification
6 passed against dev. Both proven able to fail: the volume test against a lowered ceiling (reports the real counts and names the loop), the burst test against a 429-returning host. Green again on revert.
Targets
CIbecause that is the ref the workflow checks out and the one callers use.