fix(postgres-patroni): pin base distro per major; REINDEX before refreshing collation versions - #142
Draft
paulocsanz wants to merge 7 commits into
Draft
paulocsanz wants to merge 7 commits into
paulocsanz wants to merge 7 commits into
Conversation
paulocsanz
force-pushed
the
paulo/collation-reindex-and-distro-pin
branch
3 times, most recently
from
September 28, 2026 18:10
564f765 to
097a28a
Compare
…re refreshing collation versions
A same-tag digest bump of the sibling postgres-ssl:16 image moved its base
from Debian 12 (glibc 2.36) to Debian 13 (glibc 2.41) on 2026-08-29 because
its Dockerfile floated `FROM postgres:16`; the wrapper then ran
`ALTER DATABASE ... REFRESH COLLATION VERSION` without reindexing, so every
btree index over a collatable column stayed sorted for the old libc while
Postgres's own warning was silenced. A customer read that as ~170k lost
records. This image had both hazards in the same shape.
Dockerfile / workflow: `FROM postgres:${POSTGRES_IMAGE_TAG}-${POSTGRES_BASE_DISTRO}`
with the release frozen per major (trixie for 14-18 -- what every published
tag already runs, verified against ghcr on 2026-09-20; bookworm would have
flipped the fleet 2.41 -> 2.36). The content gate keys on the suffixed base
and the publish step refuses an image whose codename is not the pinned one.
bootstrap/collation.rs: detect the mismatch from pg_database.datcollversion
and pg_collation.collversion against the running libc, REINDEX every index
that depends on a changed libc collation (CONCURRENTLY where possible) and
only then refresh the recorded versions -- leader only, re-checking
pg_is_in_recovery()/Patroni /leader/scheduled switchover/pause before every
index, catalog-driven so it is idempotent and failover-safe. ICU is reported,
not silenced. COLLATION_REINDEX_DISABLED=1 is the kill switch.
Tests: unit tests for the module; e2e t_image_base_distro_pinned and
t_ha_collation_mismatch_reindex_then_refresh.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
paulocsanz
force-pushed
the
paulo/collation-reindex-and-distro-pin
branch
from
September 28, 2026 21:29
6721f29 to
6f35fbe
Compare
…nescape COPY names Audit findings on the reindex-then-refresh procedure: - The `_ccnew` leftover scan matched an in-flight REINDEX CONCURRENTLY (its transient index is invalid until the swap); DROP INDEX then blocked on the builder's session lock and failed after the rename, aborting that database's repair. Exclude anything listed in pg_stat_progress_create_index. - patroni-runner's boot pass and the on_role_change callback can both fire for the same promotion and ran the whole procedure twice. One flock per node in the socket dir; the loser logs and returns. - COPY TO STDOUT escapes backslashes and control characters in identifiers; a name containing one was fed back to REINDEX misspelled and that database failed every pass. Undo the escaping on every name read back. - README: drop the stale "standalone does not run this yet" bullet and correct the paragraph on clusters an earlier image refreshed blindly — the stale predefined-collation stamps DO trigger the repair on the next leader boot, so it is a one-off full text-index REINDEX to budget for (converted bookworm volumes), not a manual step.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A library update can change collation ordering while indexes still contain the old order. Refreshing only the version stamp hides the problem. Pin the PostgreSQL base distro and rebuild dependent indexes before refreshing any stamps; failed rebuilds leave the database pending for the next boot or promotion.
Coverage includes libc and ICU, PG13/14 named-stamp inference, previously refreshed default stamps, explicit dependencies recorded in pg_depend, and partial/expression indexes with implicit default collation. Patroni leadership and switchover gates are checked before writes; replicas receive repaired indexes through WAL. Standalone runs the repair in its supervised maintenance sidecar, even without PITR.
Validation: full Rust workspace suite passed (470 tests, 9 doc tests ignored), cargo fmt and shell syntax checks. Production SQL selected all six explicit/implicit dependency fixtures on PostgreSQL 15. E2E covers leader repair, replica propagation, implicit dependencies, and a standalone custom-user volume. Image CI also builds pinned official MinIO/mc sources because their public registry images were removed.
Repairs run one at a time per node (file lock shared by the boot pass, the promotion callback and the standalone sidecar), an index another backend is still building concurrently is never mistaken for an abandoned leftover, and identifiers read back through COPY are unescaped before they reach REINDEX. Clusters an earlier image refreshed without reindexing (converted bookworm volumes) are repaired automatically on the next leader boot via their stale predefined-collation stamps; that is a one-off full text-index REINDEX to budget for, documented in the README. No automatic merge or rollout.