ci: make api-smoke actually run, and survive a slow dev backend - #19
Merged
Conversation
The dev backend is intermittently slow on authenticated routes. Measured on /api/activities/user/: 8.3s, 8.3s, 50.3s on consecutive calls, and similar on the token exchange - while Keycloak answers in 0.13s and the backend's own /health/ in 0.2s. Requests therefore cross nginx's 60s proxy timeout at random and come back as 504. That made api-smoke fail on whichever test drew the slow call: test_user_activities_response_is_list_or_dict went red while test_user_activities_returns_200_when_authenticated, hitting the same endpoint, passed. Retries only gateway statuses and transport errors, with a 90s timeout so slowness arrives as a status code rather than a ReadTimeout. A 4xx or an application 5xx is returned on the first attempt, so genuine failures still surface immediately. This makes an infrastructure problem stop presenting as an API contract failure. It does not fix the underlying slowness, which is inside the backend and worth its own issue.
/api/auth/keycloak/login/ intermittently exceeds nginx's 60s timeout on dev - roughly half of calls, with successes taking up to 42s. Both auth fixtures asserted 200 on the first attempt, so the whole authenticated suite errored out whenever the exchange was slow. Both now share one helper that retries gateway errors and client timeouts, then skips the session with a message naming the defect when no answer arrives at all. Skipping rather than failing is deliberate: the tests cannot say anything about the API when they cannot authenticate, and reporting that as a failure misattributes an endpoint outage to the code under test. A genuine 401 still fails immediately.
`inputs` is empty on push and pull_request events, so API_BASE_URL was unset, every api-smoke test skipped, and the job reported success having asserted nothing - the same hollow green the Keycloak preflight exists to prevent. The merge of #18 showed it plainly: "17 skipped in 1.09s", job green. Defaults to the dev backend, overridable by a repo variable, with a caller's api_base_url still winning. Because API_BASE_URL is now always set, the existing min_passed guard runs on every event too, so a fully-skipped api-smoke can no longer exit 0. The unauthenticated health tests satisfy min_passed on their own, which keeps this green when the token exchange is timing out and the authenticated tests skip. Verified against dev: 16 passed, 1 skipped, twice in a row (the skip is the deployed-SHA check, which has no SHA to compare on a push).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #18. That PR stopped the Keycloak config from silently skipping; this stops the whole
api-smokejob from doing the same.The hollow green
Merging #18 made it visible on the
CIpush run:Job green, nothing asserted.
API_BASE_URLcomes frominputs.api_base_url, andinputsis empty onpushandpull_requestevents — those only exist forworkflow_callandworkflow_dispatch. So the API tests have only ever run when a deploy workflow called this one.Now defaults to the dev backend, overridable by a repo variable, with a caller's value still winning. Because
API_BASE_URLis then always set, the existingmin_passedguard runs on every event too, so a fully-skippedapi-smokecan no longer exit 0.The dev backend is slow, and it was breaking the suite
Turning the tests on surfaced a real problem. Measured against dev:
/api/auth/keycloak/login//api/activities/user//health/So the latency is inside the backend's authenticated request path, not the Keycloak migration and not the network. Requests cross nginx's 60s proxy timeout at random and return 504.
The effect was arbitrary:
test_user_activities_response_is_list_or_dictfailed whiletest_user_activities_returns_200_when_authenticated— same endpoint — passed, purely on timing.How this handles it
ReadTimeout. A 4xx or an application 5xx returns on the first attempt, so genuine failures still surface immediately.min_passedon their own.Verification
16 passed, 1 skipped, twice in a row against dev. The skip is the deployed-SHA check, which has no SHA to compare on a push.
Not fixed here
The backend slowness itself. This stops it presenting as a test failure; it does not make it go away. Worth its own issue against DataSpaceBackend — the authenticated request path taking 40–60s while
/health/takes 0.2s points at something in the auth middleware or a connection pool, not at the tests.