Skip to content

Bug fixes, documentation and maintainability focus - #18

Merged
jessestimpson merged 14 commits into
mainfrom
ccr-c10e23b1-d95ic1
Oct 29, 2026
Merged

jessestimpson merged 14 commits into
mainfrom
ccr-c10e23b1-d95ic1

Conversation

@jessestimpson

Copy link
Copy Markdown
Contributor

No description provided.

claude and others added 14 commits October 29, 2026 13:35
The member's local copy of the name table and its position in the leader's
stream — the ETS rows, the inverted index, applied_version and epoch — move
into a new module that owns the rules for advancing it:

  offer_batch/5     RecvBcastApply / RecvBcastDrop / RecvBcastGap
  offer_snapshot/4  RecvSnap (with the version-monotonic guard)

The module sends nothing and answers no one; it reports what happened
(applied / stale_epoch / duplicate / gap, accepted / stale) and the member
reacts. Both admission rules now sit side by side, so the epoch asymmetry
(a newer-epoch batch applies without raising the epoch; a snapshot sets it)
is documented as deliberate and matches the TLA+ model action for action.

The member publishes `replica` to eta_observe in place of `epoch` and
`applied_version`; the sim's Cluster.status/1 flattens it back so the
invariants read the same keys as dgen_registry:status/1.

No behaviour change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JCdT3buxZ1upFQovTcqDvC
adopt/4 (formerly the member's records_replace/2) cleared the ETS table and
then reinserted the new content. The table is read lock-free by callers, so
for the length of every resync and handoff re-baseline a bound name read as
`undefined` from whereis_name/1 and get_metadata/1.

It now writes the new rows first and deletes only the names the new content
lacks, so a concurrent reader sees each name's old row or its new one.

This was behind two intermittent failures under load: the Guarantee 11
gap test (a member's self-resync briefly emptied its table) and the sim
"subject that dies during the handoff window is still reaped" test (the
follower looked reaped while the leader still held the dead binding, so the
successor was correctly refused).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JCdT3buxZ1upFQovTcqDvC
A non-leader member's unfinished requests lived in four stashes, each with
its own resolve function and its own rule on a leadership change:
pending_registers, pending_forwards (three kinds), deferred_yes and
pending_unregs. They become one dgen_registry_intents value: an intent is
a kind (register / unregister / set_meta) in a phase (awaiting_leader /
{forwarded, L} / {awaiting_version, V, L}), moved only by three events:
on_reply/4, on_version/2 and redrive/5. The module is pure; it returns
effects (reply, forward, retract, enqueue, resync) for the member to run.

The re-drive rule now follows idempotence instead of the field an intent
happened to sit in:

- register: re-sent to the new leader in every phase where its outcome is
  unknown here. Previously a forwarded register, or a committed-but-gap-
  deferred `yes`, was answered `no` when leadership moved — a
  non-adjudicated `no` that OTP's via machinery reads as already_started.
- unregister: re-driven as a pid-guarded retract (unchanged).
- set_meta: a replace, not safe to re-send; `{error, no_leader}` if its
  leader no longer leads (unchanged).

To support that, a member in its assume window postpones a forwarded
register until it leads, and a member that does not lead stays silent
rather than answer `no`; the forwarder re-drives on its next leadership
event. A snapshot re-sends only forwards whose leader changed; a restored
link (nodeup) re-sends forwards to the current leader too, since the cast
may have died with the link. Effects are emitted in arrival order, never
map order, so schedules stay reproducible under simulation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JCdT3buxZ1upFQovTcqDvC
The leader's write pipeline — the queue, the single in-flight commit, the
cached read version and the direct registrations awaiting holders — moves
into dgen_registry_pipeline. Its point is surrender/2: every op the leader
cannot finish leaves through it, for one of two reasons.

  failed   the commit was fenced, errored, its worker died, or it landed
           under a leadership view that has since moved. Retried after
           ?REQUEUE_DELAY: re-enqueued if we still lead, else surrendered
           as deposed.
  deposed  we no longer lead. By kind, the follower intents' rule: a local
           registration becomes an intent re-driven to the next leader; a
           forwarded one is left to its follower; set_meta answers
           {error, no_leader}; an unregister or retract becomes an answered,
           pid-guarded retract intent; a down is dropped.

This replaces five exits that each had their own rules
(maybe_start_commit's deposed flush, reject_plan + salvage_failed_plan,
the requeue_ops handlers, fail_pending_acks), and fixes two defects:

- A buffered unregister was silently dropped when the deposed flush ran
  with no leader known — the member's own assume window after a
  re-election it had not heard it lost. forward_retract(undefined, …) did
  nothing. Reproduced by the new "ops caught in the assume window" test,
  which fails on the previous commit.
- A registration was answered `no` when its commit failed or its leader
  was deposed before answering. It is now retried or re-sent; `no` is left
  to adjudicated refusals and the opt-in policies of §8.

A fenced commit also makes the member re-announce itself to the elector,
once per epoch, so a stale leader learns it was deposed now rather than at
the connector's next durable-epoch check (up to 10 s) while its retried ops
wait. A local registration whose caller has died is dropped on retry.

The partition-heal cluster test retried a register expecting a fast `no`
from a stale leader; it now treats a timed-out call as "not yet", per §3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JCdT3buxZ1upFQovTcqDvC
detect_conflicts/3 decides a §5.6 conflict only between *live* pids, and
probed liveness with is_pid_alive/1 — an erpc:call of up to
?ALIVE_PROBE_TIMEOUT (1 s) per remote pid — from the joiner-gather and
assume-gather continuations, which run on the leader's own loop. Every
other network step of a handoff was already off the loop; a partition heal
with several divergent names could stall the leader for seconds.

The probes now run in the gather helpers that already do the network work:

- conflict_pids/3 names the pids an adjudication could turn on (each
  candidate name's authority and divergent claimants);
- probe_alive/1 probes them in parallel, bounded by about one
  ?ALIVE_PROBE_TIMEOUT, inside the helper;
- detect_conflicts/4 adjudicates on the loop with that evidence. A pid the
  helper did not probe (the leader's table changed meanwhile) is checked
  only if local; a remote one counts as not alive, so nothing is killed on
  evidence that was not gathered — the next gather re-checks it.

The joiner helper reads the leader's own (protected) table itself, and
probes nothing rather than crashing if the member has already died.
detect_conflicts/3 is kept for the unit tests. No change to what counts as
a conflict.

Also fixes a race in the observability test "across a leadership change":
the elector names the new leader before the member finishes its off-loop
assume, so the published view and status/1 were compared mid-handoff. It
now waits for the handoff to settle before requiring agreement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JCdT3buxZ1upFQovTcqDvC
`degraded` gates `reject_when_degraded` (§5.6, §8): may this leader's
handoff reconstruction be missing a committed binding? It was computed
from reachability — a member not marked fresh did not answer the gather.
That is both too strong and too weak:

- a member that did not answer but was behind held nothing extra, so a
  scale-down tripped it;
- a binding whose only holders were gone — e.g. a degrade-open `yes` held
  by a leader that then crashed, so it is no longer in the member list —
  did not.

Every replica is a prefix of the committed stream, so a reconstruction
that reaches the durable version key IS the committed map, and one that
stops short of it lacks exactly what was committed past it. `degraded` is
now "gathered max version < durable version key" (degraded/5).
gather_caught_up already read the key; it now returns it. If the key is
unreadable it falls back to the reachability rule.

The TLA+ model gains the property:

- DegradedFlagSound: an assume that did not flag itself degraded adopted
  exactly histMap[dbVersion]. Added to the main config (10.66M states).
- DegradedAssume (new constant, FALSE everywhere else) models the gather's
  deadline: an assume may proceed behind the key. The new
  DgenRegistryReplicationDegradedAssume config checks DegradedFlagSound
  there (10.85M states, pass); DgenRegistryReplicationDegradedReachable is
  its must-fail vacuity canary (violated at depth 9). Forcing the flag to
  never be set makes DegradedFlagSound fail, so it has teeth. All twelve
  configs re-run against the changed spec with their expected outcomes;
  both new ones are added to the CI matrix.

Two tests drive a real handoff on one node and fail on the old rule in
opposite directions: a non-answering, previously-synced member no longer
degrades the leader, and a reconstruction behind the key now does. There
was no prior coverage of `reject_when_degraded`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JCdT3buxZ1upFQovTcqDvC
Conflict repair (§5.6) was four fields of the member's state spread over
seven sites: the release trail (written at commit, at batch apply, merged
at two gathers, shipped in get_names_snapshot), the kill budget, the time
of the last departure, and the `degraded` flag. They move into one pure
module, organised as evidence, verdict, remedy and prevention:

- trail/1, released/3, merge/2, departed/2, prune/4 — the release trail
  and its expiry rules (suspended while a member is disconnected or within
  a TTL of a departure), plus the kill-budget window;
- conflict_pids/3 and detect/4 — which pids to probe off the loop, and the
  verdict given their liveness (detect_conflicts/3,4 and conflict_pids/3
  leave the member);
- resolve/5 — the remedy: kill the authority and every divergent pid and
  drop the name, within terminate_on_conflict and the per-name budget. It
  returns the kills and alarms as effects; the member carries them out
  (exit/2, logger and telemetry) in one place, repair_conflicts/5;
- degraded/1, set_degraded/2, degraded/5 — prevention for
  reject_when_degraded.

One behaviour change, completing 7569898: the remedy no longer re-probes
the authority with is_pid_alive/1 — a remote RPC on the leader's loop —
before killing it. A conflict only exists if the authority was alive, and
a pid that has died since ignores the kill.

The detector's unit tests move to test/dgen_registry_conflicts_test.exs,
which adds coverage for the remedy (kill order, budget exhaustion and
recovery, detect-only), the trail's expiry rules, and degraded/5.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JCdT3buxZ1upFQovTcqDvC
Comments only: history, restatements of the code, and module headers that
duplicated the design doc. What stays is the non-obvious why — deadlock
avoidance, message ordering, fencing, the reasons behind timing choices —
plus § references and one-line notes where rolling-upgrade code remains.
Verified token-identical to the previous revision with comments ignored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JCdT3buxZ1upFQovTcqDvC
Comments only, 1550 -> 492 lines. The module header becomes a short
orientation (roles, where each concern lives, the statem states). Gone:
statem-port phase notes, pre-fix and found-by stories, past timings and
allocation counts, descriptions of old wire formats, and rationale the
extracted modules already document (now a pointer). Kept, shortened:
deadlock-avoidance reasons, FIFO assumptions, fences, § references and
spec action names, one-line notes on mutation blocks and remaining
rolling-upgrade code. Verified token-identical with comments ignored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JCdT3buxZ1upFQovTcqDvC
Comments only, in dgen_registry and the replica, intents, pipeline and
conflicts modules (744 -> 331 lines). Each extracted module's header is now
a compact statement of its concept and rules; gone are discovery stories,
benchmarks (one reason kept per tuned constant), old wire formats and
restatements. -doc/-moduledoc content is untouched. Verified
token-identical with comments ignored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JCdT3buxZ1upFQovTcqDvC
739 -> ~410 lines. Numbered sections, guarantees and configuration are kept
(tightened); motivation, repeated explanations, glossary entries for
self-explanatory terms and code-level detail are cut. Three statements are
corrected to match src/dgen_server.erl: a dead lock holder blocks the queue
only until `lock_timeout` (manual intervention only with `infinity`); a
fenced holder's message is requeued and retried, so `handle_locked/4` must be
safe to run twice; and the exit is `{lock_fenced, requeued}`, or
`not_requeued` for a priority or inline lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JCdT3buxZ1upFQovTcqDvC
formal/README.md 289 -> 124 lines: keeps what is modeled and out of scope,
the invariants, one table of the twelve configs with what each proves, the
spec<->code map (two stale entries corrected: the heartbeat handler is now a
gen_statem clause; the snapshot guard names) and how to run. Cuts the
discovery narratives, the "closed abstraction gap" story, timings and the
jar's history.

test/support/sim/README.md 459 -> 180: keeps running it, the invariants, how
it works, the fault model and what is not simulated; the seven findings
shrink to one or two lines each (tests and the spec cite them by number).

README.md: drops duplicated links and the long eta paragraph.

Two comments that cited removed README sections now point at the sim
README's finding 1 and at design §5.7.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JCdT3buxZ1upFQovTcqDvC
1121 -> ~630 lines. Every numbered section is kept verbatim (code cites
them as §N.N), as is every §6 guarantee and §7 non-goal (tightened, same
meaning) and every §8 setting with its default and effect.

Cut: how the design evolved, the race story behind the handoff guards,
implementation detail that belongs in code docs (process-dictionary
lookups, internal message shapes, index internals), hedging, and repeated
explanations — each idea is now stated once and referenced by §: why writes
block instead of answering `no` (§3), "a returned pid is a candidate"
(§3), fencing (§5.1), the two-holder rule (§5.5), the kill contract (§5.6).
§9 and §10 are short.

Brought in line with current behaviour: on the minority side of a
partition registrations block and time out rather than answer `no`, and an
unregister answers `ok` and is held; `degraded` is defined in §5.7 as a
handoff that stopped short of the durable version key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JCdT3buxZ1upFQovTcqDvC
@jessestimpson
jessestimpson merged commit 5df83cc into main Oct 29, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants