Repository navigation
Bug fixes, documentation and maintainability focus - #18
Merged
Merged
Conversation
The member's local copy of the name table and its position in the leader's stream — the ETS rows, the inverted index, applied_version and epoch — move into a new module that owns the rules for advancing it: offer_batch/5 RecvBcastApply / RecvBcastDrop / RecvBcastGap offer_snapshot/4 RecvSnap (with the version-monotonic guard) The module sends nothing and answers no one; it reports what happened (applied / stale_epoch / duplicate / gap, accepted / stale) and the member reacts. Both admission rules now sit side by side, so the epoch asymmetry (a newer-epoch batch applies without raising the epoch; a snapshot sets it) is documented as deliberate and matches the TLA+ model action for action. The member publishes `replica` to eta_observe in place of `epoch` and `applied_version`; the sim's Cluster.status/1 flattens it back so the invariants read the same keys as dgen_registry:status/1. No behaviour change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JCdT3buxZ1upFQovTcqDvC
adopt/4 (formerly the member's records_replace/2) cleared the ETS table and then reinserted the new content. The table is read lock-free by callers, so for the length of every resync and handoff re-baseline a bound name read as `undefined` from whereis_name/1 and get_metadata/1. It now writes the new rows first and deletes only the names the new content lacks, so a concurrent reader sees each name's old row or its new one. This was behind two intermittent failures under load: the Guarantee 11 gap test (a member's self-resync briefly emptied its table) and the sim "subject that dies during the handoff window is still reaped" test (the follower looked reaped while the leader still held the dead binding, so the successor was correctly refused). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JCdT3buxZ1upFQovTcqDvC
A non-leader member's unfinished requests lived in four stashes, each with
its own resolve function and its own rule on a leadership change:
pending_registers, pending_forwards (three kinds), deferred_yes and
pending_unregs. They become one dgen_registry_intents value: an intent is
a kind (register / unregister / set_meta) in a phase (awaiting_leader /
{forwarded, L} / {awaiting_version, V, L}), moved only by three events:
on_reply/4, on_version/2 and redrive/5. The module is pure; it returns
effects (reply, forward, retract, enqueue, resync) for the member to run.
The re-drive rule now follows idempotence instead of the field an intent
happened to sit in:
- register: re-sent to the new leader in every phase where its outcome is
unknown here. Previously a forwarded register, or a committed-but-gap-
deferred `yes`, was answered `no` when leadership moved — a
non-adjudicated `no` that OTP's via machinery reads as already_started.
- unregister: re-driven as a pid-guarded retract (unchanged).
- set_meta: a replace, not safe to re-send; `{error, no_leader}` if its
leader no longer leads (unchanged).
To support that, a member in its assume window postpones a forwarded
register until it leads, and a member that does not lead stays silent
rather than answer `no`; the forwarder re-drives on its next leadership
event. A snapshot re-sends only forwards whose leader changed; a restored
link (nodeup) re-sends forwards to the current leader too, since the cast
may have died with the link. Effects are emitted in arrival order, never
map order, so schedules stay reproducible under simulation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JCdT3buxZ1upFQovTcqDvC
The leader's write pipeline — the queue, the single in-flight commit, the
cached read version and the direct registrations awaiting holders — moves
into dgen_registry_pipeline. Its point is surrender/2: every op the leader
cannot finish leaves through it, for one of two reasons.
failed the commit was fenced, errored, its worker died, or it landed
under a leadership view that has since moved. Retried after
?REQUEUE_DELAY: re-enqueued if we still lead, else surrendered
as deposed.
deposed we no longer lead. By kind, the follower intents' rule: a local
registration becomes an intent re-driven to the next leader; a
forwarded one is left to its follower; set_meta answers
{error, no_leader}; an unregister or retract becomes an answered,
pid-guarded retract intent; a down is dropped.
This replaces five exits that each had their own rules
(maybe_start_commit's deposed flush, reject_plan + salvage_failed_plan,
the requeue_ops handlers, fail_pending_acks), and fixes two defects:
- A buffered unregister was silently dropped when the deposed flush ran
with no leader known — the member's own assume window after a
re-election it had not heard it lost. forward_retract(undefined, …) did
nothing. Reproduced by the new "ops caught in the assume window" test,
which fails on the previous commit.
- A registration was answered `no` when its commit failed or its leader
was deposed before answering. It is now retried or re-sent; `no` is left
to adjudicated refusals and the opt-in policies of §8.
A fenced commit also makes the member re-announce itself to the elector,
once per epoch, so a stale leader learns it was deposed now rather than at
the connector's next durable-epoch check (up to 10 s) while its retried ops
wait. A local registration whose caller has died is dropped on retry.
The partition-heal cluster test retried a register expecting a fast `no`
from a stale leader; it now treats a timed-out call as "not yet", per §3.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JCdT3buxZ1upFQovTcqDvC
detect_conflicts/3 decides a §5.6 conflict only between *live* pids, and probed liveness with is_pid_alive/1 — an erpc:call of up to ?ALIVE_PROBE_TIMEOUT (1 s) per remote pid — from the joiner-gather and assume-gather continuations, which run on the leader's own loop. Every other network step of a handoff was already off the loop; a partition heal with several divergent names could stall the leader for seconds. The probes now run in the gather helpers that already do the network work: - conflict_pids/3 names the pids an adjudication could turn on (each candidate name's authority and divergent claimants); - probe_alive/1 probes them in parallel, bounded by about one ?ALIVE_PROBE_TIMEOUT, inside the helper; - detect_conflicts/4 adjudicates on the loop with that evidence. A pid the helper did not probe (the leader's table changed meanwhile) is checked only if local; a remote one counts as not alive, so nothing is killed on evidence that was not gathered — the next gather re-checks it. The joiner helper reads the leader's own (protected) table itself, and probes nothing rather than crashing if the member has already died. detect_conflicts/3 is kept for the unit tests. No change to what counts as a conflict. Also fixes a race in the observability test "across a leadership change": the elector names the new leader before the member finishes its off-loop assume, so the published view and status/1 were compared mid-handoff. It now waits for the handoff to settle before requiring agreement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JCdT3buxZ1upFQovTcqDvC
`degraded` gates `reject_when_degraded` (§5.6, §8): may this leader's handoff reconstruction be missing a committed binding? It was computed from reachability — a member not marked fresh did not answer the gather. That is both too strong and too weak: - a member that did not answer but was behind held nothing extra, so a scale-down tripped it; - a binding whose only holders were gone — e.g. a degrade-open `yes` held by a leader that then crashed, so it is no longer in the member list — did not. Every replica is a prefix of the committed stream, so a reconstruction that reaches the durable version key IS the committed map, and one that stops short of it lacks exactly what was committed past it. `degraded` is now "gathered max version < durable version key" (degraded/5). gather_caught_up already read the key; it now returns it. If the key is unreadable it falls back to the reachability rule. The TLA+ model gains the property: - DegradedFlagSound: an assume that did not flag itself degraded adopted exactly histMap[dbVersion]. Added to the main config (10.66M states). - DegradedAssume (new constant, FALSE everywhere else) models the gather's deadline: an assume may proceed behind the key. The new DgenRegistryReplicationDegradedAssume config checks DegradedFlagSound there (10.85M states, pass); DgenRegistryReplicationDegradedReachable is its must-fail vacuity canary (violated at depth 9). Forcing the flag to never be set makes DegradedFlagSound fail, so it has teeth. All twelve configs re-run against the changed spec with their expected outcomes; both new ones are added to the CI matrix. Two tests drive a real handoff on one node and fail on the old rule in opposite directions: a non-answering, previously-synced member no longer degrades the leader, and a reconstruction behind the key now does. There was no prior coverage of `reject_when_degraded`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JCdT3buxZ1upFQovTcqDvC
Conflict repair (§5.6) was four fields of the member's state spread over seven sites: the release trail (written at commit, at batch apply, merged at two gathers, shipped in get_names_snapshot), the kill budget, the time of the last departure, and the `degraded` flag. They move into one pure module, organised as evidence, verdict, remedy and prevention: - trail/1, released/3, merge/2, departed/2, prune/4 — the release trail and its expiry rules (suspended while a member is disconnected or within a TTL of a departure), plus the kill-budget window; - conflict_pids/3 and detect/4 — which pids to probe off the loop, and the verdict given their liveness (detect_conflicts/3,4 and conflict_pids/3 leave the member); - resolve/5 — the remedy: kill the authority and every divergent pid and drop the name, within terminate_on_conflict and the per-name budget. It returns the kills and alarms as effects; the member carries them out (exit/2, logger and telemetry) in one place, repair_conflicts/5; - degraded/1, set_degraded/2, degraded/5 — prevention for reject_when_degraded. One behaviour change, completing 7569898: the remedy no longer re-probes the authority with is_pid_alive/1 — a remote RPC on the leader's loop — before killing it. A conflict only exists if the authority was alive, and a pid that has died since ignores the kill. The detector's unit tests move to test/dgen_registry_conflicts_test.exs, which adds coverage for the remedy (kill order, budget exhaustion and recovery, detect-only), the trail's expiry rules, and degraded/5. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JCdT3buxZ1upFQovTcqDvC
Comments only: history, restatements of the code, and module headers that duplicated the design doc. What stays is the non-obvious why — deadlock avoidance, message ordering, fencing, the reasons behind timing choices — plus § references and one-line notes where rolling-upgrade code remains. Verified token-identical to the previous revision with comments ignored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JCdT3buxZ1upFQovTcqDvC
Comments only, 1550 -> 492 lines. The module header becomes a short orientation (roles, where each concern lives, the statem states). Gone: statem-port phase notes, pre-fix and found-by stories, past timings and allocation counts, descriptions of old wire formats, and rationale the extracted modules already document (now a pointer). Kept, shortened: deadlock-avoidance reasons, FIFO assumptions, fences, § references and spec action names, one-line notes on mutation blocks and remaining rolling-upgrade code. Verified token-identical with comments ignored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JCdT3buxZ1upFQovTcqDvC
Comments only, in dgen_registry and the replica, intents, pipeline and conflicts modules (744 -> 331 lines). Each extracted module's header is now a compact statement of its concept and rules; gone are discovery stories, benchmarks (one reason kept per tuned constant), old wire formats and restatements. -doc/-moduledoc content is untouched. Verified token-identical with comments ignored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JCdT3buxZ1upFQovTcqDvC
739 -> ~410 lines. Numbered sections, guarantees and configuration are kept
(tightened); motivation, repeated explanations, glossary entries for
self-explanatory terms and code-level detail are cut. Three statements are
corrected to match src/dgen_server.erl: a dead lock holder blocks the queue
only until `lock_timeout` (manual intervention only with `infinity`); a
fenced holder's message is requeued and retried, so `handle_locked/4` must be
safe to run twice; and the exit is `{lock_fenced, requeued}`, or
`not_requeued` for a priority or inline lock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JCdT3buxZ1upFQovTcqDvC
formal/README.md 289 -> 124 lines: keeps what is modeled and out of scope, the invariants, one table of the twelve configs with what each proves, the spec<->code map (two stale entries corrected: the heartbeat handler is now a gen_statem clause; the snapshot guard names) and how to run. Cuts the discovery narratives, the "closed abstraction gap" story, timings and the jar's history. test/support/sim/README.md 459 -> 180: keeps running it, the invariants, how it works, the fault model and what is not simulated; the seven findings shrink to one or two lines each (tests and the spec cite them by number). README.md: drops duplicated links and the long eta paragraph. Two comments that cited removed README sections now point at the sim README's finding 1 and at design §5.7. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JCdT3buxZ1upFQovTcqDvC
1121 -> ~630 lines. Every numbered section is kept verbatim (code cites them as §N.N), as is every §6 guarantee and §7 non-goal (tightened, same meaning) and every §8 setting with its default and effect. Cut: how the design evolved, the race story behind the handoff guards, implementation detail that belongs in code docs (process-dictionary lookups, internal message shapes, index internals), hedging, and repeated explanations — each idea is now stated once and referenced by §: why writes block instead of answering `no` (§3), "a returned pid is a candidate" (§3), fencing (§5.1), the two-holder rule (§5.5), the kill contract (§5.6). §9 and §10 are short. Brought in line with current behaviour: on the minority side of a partition registrations block and time out rather than answer `no`, and an unregister answers `ok` and is held; `degraded` is defined in §5.7 as a handoff that stopped short of the durable version key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JCdT3buxZ1upFQovTcqDvC
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.