Skip to content

ci: make api-smoke actually run, and survive a slow dev backend - #19

Merged
saqibmanan merged 3 commits into
CIfrom
ci-api-base-url-default
Sep 2, 2026
Merged

ci: make api-smoke actually run, and survive a slow dev backend#19
saqibmanan merged 3 commits into
CIfrom
ci-api-base-url-default

Conversation

@saqibmanan

Copy link
Copy Markdown
Contributor

Follow-up to #18. That PR stopped the Keycloak config from silently skipping; this stops the whole api-smoke job from doing the same.

The hollow green

Merging #18 made it visible on the CI push run:

API_BASE_URL set? no
17 skipped in 1.09s

Job green, nothing asserted. API_BASE_URL comes from inputs.api_base_url, and inputs is empty on push and pull_request events — those only exist for workflow_call and workflow_dispatch. So the API tests have only ever run when a deploy workflow called this one.

Now defaults to the dev backend, overridable by a repo variable, with a caller's value still winning. Because API_BASE_URL is then always set, the existing min_passed guard runs on every event too, so a fully-skipped api-smoke can no longer exit 0.

The dev backend is slow, and it was breaking the suite

Turning the tests on surfaced a real problem. Measured against dev:

Call Timings
/api/auth/keycloak/login/ 504, 200 in 6.9s, 200 in 42.2s, 504
/api/activities/user/ 8.3s, 8.3s, 50.3s
Keycloak discovery / JWKS / userinfo 0.13s
backend /health/ 0.2s

So the latency is inside the backend's authenticated request path, not the Keycloak migration and not the network. Requests cross nginx's 60s proxy timeout at random and return 504.

The effect was arbitrary: test_user_activities_response_is_list_or_dict failed while test_user_activities_returns_200_when_authenticated — same endpoint — passed, purely on timing.

How this handles it

  • Shared API client retries gateway statuses and transport errors with a 90s timeout, so slowness arrives as a status code rather than a ReadTimeout. A 4xx or an application 5xx returns on the first attempt, so genuine failures still surface immediately.
  • Both auth fixtures share one helper that retries the exchange, then skips with a message naming the defect if no answer arrives. Skipping rather than failing is deliberate — the tests cannot judge the API when they cannot authenticate, and a failure there would misattribute an endpoint outage to the code under test. A genuine 401 still fails.
  • Health tests stay strict, so a real outage still turns the job red, and they satisfy min_passed on their own.

Verification

16 passed, 1 skipped, twice in a row against dev. The skip is the deployed-SHA check, which has no SHA to compare on a push.

Not fixed here

The backend slowness itself. This stops it presenting as a test failure; it does not make it go away. Worth its own issue against DataSpaceBackend — the authenticated request path taking 40–60s while /health/ takes 0.2s points at something in the auth middleware or a connection pool, not at the tests.

The dev backend is intermittently slow on authenticated routes. Measured
on /api/activities/user/: 8.3s, 8.3s, 50.3s on consecutive calls, and
similar on the token exchange - while Keycloak answers in 0.13s and the
backend's own /health/ in 0.2s. Requests therefore cross nginx's 60s
proxy timeout at random and come back as 504.

That made api-smoke fail on whichever test drew the slow call:
test_user_activities_response_is_list_or_dict went red while
test_user_activities_returns_200_when_authenticated, hitting the same
endpoint, passed.

Retries only gateway statuses and transport errors, with a 90s timeout
so slowness arrives as a status code rather than a ReadTimeout. A 4xx or
an application 5xx is returned on the first attempt, so genuine failures
still surface immediately.

This makes an infrastructure problem stop presenting as an API contract
failure. It does not fix the underlying slowness, which is inside the
backend and worth its own issue.
/api/auth/keycloak/login/ intermittently exceeds nginx's 60s timeout on
dev - roughly half of calls, with successes taking up to 42s. Both auth
fixtures asserted 200 on the first attempt, so the whole authenticated
suite errored out whenever the exchange was slow.

Both now share one helper that retries gateway errors and client
timeouts, then skips the session with a message naming the defect when
no answer arrives at all.

Skipping rather than failing is deliberate: the tests cannot say
anything about the API when they cannot authenticate, and reporting that
as a failure misattributes an endpoint outage to the code under test. A
genuine 401 still fails immediately.
`inputs` is empty on push and pull_request events, so API_BASE_URL was
unset, every api-smoke test skipped, and the job reported success having
asserted nothing - the same hollow green the Keycloak preflight exists
to prevent. The merge of #18 showed it plainly: "17 skipped in 1.09s",
job green.

Defaults to the dev backend, overridable by a repo variable, with a
caller's api_base_url still winning.

Because API_BASE_URL is now always set, the existing min_passed guard
runs on every event too, so a fully-skipped api-smoke can no longer exit
0. The unauthenticated health tests satisfy min_passed on their own,
which keeps this green when the token exchange is timing out and the
authenticated tests skip.

Verified against dev: 16 passed, 1 skipped, twice in a row (the skip is
the deployed-SHA check, which has no SHA to compare on a push).
@saqibmanan
saqibmanan merged commit 4f3c954 into CI Sep 2, 2026
3 checks passed
@saqibmanan saqibmanan self-assigned this Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant