Two separate gaps found while adding the Keycloak migration tests (#11).
1. Shards report success despite failing tests
On the PR run for #11:
E2E Tests (shard 2/3): SUCCESS -> 26 failed, 85 passed, 13 errors
E2E Tests (shard 3/3): SUCCESS -> 24 failed, 44 passed, 10 errors
The GitHub check is green while 50 tests failed. Whatever decides the job's exit status is not reflecting the pytest result, so nobody sees these.
This is the same class of problem as CivicDataSpace-test#22 — a green CI signal that asserts nothing — and it means this suite's green cannot currently be trusted as evidence.
2. test_add_model_flow.py is collected by no shard
Grepping all three shard logs for add_model returns 0 matches, so the file never runs in CI. The session-expiry guard repaired in #11 therefore has no CI coverage; it was verified directly against the live URL shapes instead.
Worth checking whether other files are silently uncollected by the sharding.
Also observed
dev.api.parakh.civicdataspace.in returned ReadTimeout on several API tests during these runs. Checked for the same unconditional user.save() pattern that caused DataSpaceBackend#134 and found none, so this is a different cause — noted here rather than assumed.
Two separate gaps found while adding the Keycloak migration tests (#11).
1. Shards report success despite failing tests
On the PR run for #11:
The GitHub check is green while 50 tests failed. Whatever decides the job's exit status is not reflecting the pytest result, so nobody sees these.
This is the same class of problem as CivicDataSpace-test#22 — a green CI signal that asserts nothing — and it means this suite's green cannot currently be trusted as evidence.
2.
test_add_model_flow.pyis collected by no shardGrepping all three shard logs for
add_modelreturns 0 matches, so the file never runs in CI. The session-expiry guard repaired in #11 therefore has no CI coverage; it was verified directly against the live URL shapes instead.Worth checking whether other files are silently uncollected by the sharding.
Also observed
dev.api.parakh.civicdataspace.inreturnedReadTimeouton several API tests during these runs. Checked for the same unconditionaluser.save()pattern that caused DataSpaceBackend#134 and found none, so this is a different cause — noted here rather than assumed.