diff --git a/.github/workflows/formal.yml b/.github/workflows/formal.yml index f08b669..d01da19 100644 --- a/.github/workflows/formal.yml +++ b/.github/workflows/formal.yml @@ -53,6 +53,17 @@ jobs: expect: pass - cfg: DgenRegistryReplicationDegradedReachable expect: fail + # The gather stops waiting once the old leader's DOWN has reached every + # live member. Every invariant holds there, and the canary proves the + # exit is taken. + - cfg: DgenRegistryReplicationLeaderGoneAssume + expect: pass + - cfg: DgenRegistryReplicationLeaderGoneReachable + expect: fail + # A held forwarded ack released by a later epoch's versions: the false + # `yes` the epoch fence closes. + - cfg: DgenRegistryReplicationNoEpochFence + expect: fail steps: - uses: actions/checkout@v3 - uses: actions/setup-java@v4 diff --git a/CHANGELOG.md b/CHANGELOG.md index f0c82a1..e8369de 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,6 @@ # Changelog -## v0.4.3 (TBD) +## v0.5.0 (2026-11-01) ### Bug fixes @@ -12,6 +12,13 @@ probes; they run off its loop. - `whereis_name/1` and `get_metadata/1` no longer briefly read a bound name as `undefined` while a member re-baselines its replica (a resync or handoff). +- A leader that dies after a commit but before broadcasting it no longer stalls + the next leader's handoff for 2 seconds. The new leader proceeds as soon as + every member has seen the old leader go down. +- A follower could answer `yes` to a forwarded registration that a degraded handoff + had lost. It held the answer until its replica reached the commit version, but the + new leader's versions could get it there without that commit. A held `yes` now + counts only in the epoch that committed it, and is otherwise sent again. ### Enhancements @@ -19,6 +26,13 @@ `dgen_registry_replica`, `dgen_registry_intents`, `dgen_registry_pipeline`, and `dgen_registry_conflicts`. +### Breaking changes + +- Some internal legacy behavior has been removed. It may affect systems during + a rolling upgrade from 0.4.x to 0.5.x. +- The handoff gather's peer reply changed shape. A 0.5.x leader skips a 0.4.x + member's replica, and the reverse, so don't run the two in one cluster. + ## v0.4.2 (2026-10-04) ### Enahancements diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..273a5f7 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,39 @@ +# Writing comments and docs + +These rules apply to code comments, `-moduledoc`/`-doc` strings, Elixir +`@moduledoc`/`@doc`, and the Markdown design docs. + +- Write the way you'd explain the code to a colleague out loud. +- Use full sentences with a subject that does something: "We drop the reply", + not "Reply dropped". +- Don't use field or type names as English words ("an intent is a kind in a + phase"). When you name a field, variable or atom, put it in backticks and use + it as a name. +- No stacked possessives ("its forward's reference"). No parentheticals that + carry the main point. Don't use semicolons or colons to glue sentences + together; write two sentences. +- One idea per sentence. Say why when the why isn't obvious. +- Function comments start with a third-person verb: "Sends...", "Returns...". +- Avoid shorthand jargon ("CP:", "adjudicated", "owed", "surrendered") unless + it is a term this codebase defines. Use defined terms (for example "fenced", + §5.1 of the registry design doc) consistently. +- Don't be wordy. Don't comment what the code already says, and don't turn a + short, accurate comment into a paragraph. +- Keep `§N.N` references to `docs/design/dgen_registry_design.md` accurate, and + don't renumber its sections. +- Keep the "LLM-generated" disclaimer on files that carry it. + +Example. Before: + +```erlang +%% Drop timed-out register/set_meta intents so a long leaderless window cannot grow +%% the map. Unregisters are kept: their effect is owed whether or not anyone waits. +``` + +After: + +```erlang +%% Drops timed-out registers and set_metas so the map can't grow during a long +%% stretch without a leader. Unregisters are kept, because the name must still be +%% removed even when nobody is waiting. +``` diff --git a/README.md b/README.md index 1902773..7e556c2 100644 --- a/README.md +++ b/README.md @@ -73,8 +73,8 @@ end configuration. - [formal/README.md](formal/README.md) — the TLA+ model of `dgen_registry`'s replication protocol. -- [test/support/sim/README.md](test/support/sim/README.md) — deterministic - simulation testing of `dgen_registry` against the same invariants, driven by +- [test/support/sim/dgen_registry_simulation.md](test/support/sim/dgen_registry_simulation.md) + — deterministic simulation testing of `dgen_registry` against the same invariants, driven by [`eta`](https://hexdocs.pm/eta). - API reference and getting-started guides: . - [CHANGELOG.md](CHANGELOG.md) — release notes. diff --git a/VERSION b/VERSION index 2b7c5ae..8f0916f 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.4.2 +0.5.0 diff --git a/docs/design/assets/registry_flows/direct.svg b/docs/design/assets/registry_flows/direct.svg new file mode 100644 index 0000000..738fa98 --- /dev/null +++ b/docs/design/assets/registry_flows/direct.svg @@ -0,0 +1,93 @@ + + +Callerany process + +Leadermember + +Commit workerdgen_transaction + +Followermember + + + +{register, Name, Pid, Meta} +1 + +Enqueues {add, …, {local, From}} + + +commit batch, fenced on {L, E} +2 + + +{dgen_transaction, Ref, {committed, V}} +3 + + +{names_batch, Ops, E, PrevV, V, L} +4 + + +{replicate_sync, BatchRef, L, V} +5 + +Starts the replicate_timeout timer + +Acks only if its applied version is at least V + + +{replicate_ack, BatchRef, F} +6 + +Enough distinct followers acked. Cancels the timer. + + +yes +7 + +IF TOO FEW ACKS ARRIVE IN TIME + +{replicate_timeout, BatchRef} + + +yes (degrade-open), or no plus a retract under strict_replication +8 + diff --git a/docs/design/assets/registry_flows/failover.svg b/docs/design/assets/registry_flows/failover.svg new file mode 100644 index 0000000..8bad791 --- /dev/null +++ b/docs/design/assets/registry_flows/failover.svg @@ -0,0 +1,125 @@ + + +Old leadermember + +New leadermember + +Helperspawned process + +Followermember + +Electordgen_server + +BackendFDB keys + +Process or node dies + + + +'DOWN' on the peer monitor +1 + +Sets leader_down. A helper reads L0's current +token. + + +{member_down, L0, Token} +2 + +Removes L0 and elects N. Epoch becomes E+1. + + +commit leader key {N, E+1} +3 + + + +{elector_assume_and_distribute, undefined, …, E+1} +4 + +Moves to assuming. Captures its own records +and version. + + +ok +5 + + +spawn the gather +6 + + + +get_names_snapshot +7 + + +{Records, V, Released, LeaderGone} +8 + + +read committed frontier +9 + +Gathers again every 25 ms until MaxVersion +reaches the frontier, every peer reports +LeaderGone, or 2 s pass + + + +get_subscriptions +10 + + +{assume_gathered, …, Freshest, {MaxVersion, Frontier}, …} +11 + +Repairs conflicts. Adopts the replica at +(E+1, MaxVersion). Becomes leader and starts +the monitor sweep. + + +{apply_names_snapshot, Records, N, …, E+1, MaxVersion} +12 + +Follows N. Releases or re-drives its held +replies. + diff --git a/docs/design/assets/registry_flows/fenced.svg b/docs/design/assets/registry_flows/fenced.svg new file mode 100644 index 0000000..7742e97 --- /dev/null +++ b/docs/design/assets/registry_flows/fenced.svg @@ -0,0 +1,84 @@ + + +Callerany process + +Old leadermember + +Commit workerdgen_transaction + +BackendFDB keys + +Electordgen_server + +L0 is partitioned. The elector has already moved leadership to N at epoch E+1. + + + +{register, Name, Pid, Meta} +1 + + +commit batch, fenced on {L0, E} +2 + + +check leader key +3 + + +{N, E+1}: conflict +4 + + +{dgen_transaction, Ref, {aborted, fenced}} +5 + +Retries the ops after 1 s. Once deposed, a local +register becomes an intent. + + +{join, L0, Token, false} +6 + +Rejoins at most once per epoch + +N's snapshot later moves L0 to follower, and its intents go to N. + diff --git a/docs/design/assets/registry_flows/forwarded.svg b/docs/design/assets/registry_flows/forwarded.svg new file mode 100644 index 0000000..e90e492 --- /dev/null +++ b/docs/design/assets/registry_flows/forwarded.svg @@ -0,0 +1,110 @@ + + +Callerany process + +Followermember + +Leadermember + +Commit workerdgen_transaction + +BackendFDB keys + +Other followersmembers + + + +{register, Name, Pid, Meta} +1 + +Stores an intent under Ref. It can't decide +alone. + + +{register_req, Ref, F, Name, Pid, Meta} +2 + +Enqueues {add, …, {forward, F, Ref}}. Ops that +arrive during a commit join the next batch. + + +start_commit(epoch E) +3 + + +fence on leader key {L, E}, bump version +4 + + +commit version V +5 + + +{dgen_transaction, Ref, {committed, V}} +6 + +Applies the batch to its replica. Monitors Pid +with tag {down, Name}. + + + + +{names_batch, Ops, E, PrevV, V, L} +7 + +PrevV matches its version, so the whole batch +applies. Replica is at (E, V). + + +{register_reply, Ref, yes, V, E} +8 + +Replica holds V in epoch E, so the yes is +released. + + +yes +9 + +If the batch was refused as a gap, F holds the yes +as {awaiting_version, V, E, L} and asks L for a +resync. A yes from another epoch never counts. + diff --git a/docs/design/assets/registry_flows/held-yes.svg b/docs/design/assets/registry_flows/held-yes.svg new file mode 100644 index 0000000..44a2574 --- /dev/null +++ b/docs/design/assets/registry_flows/held-yes.svg @@ -0,0 +1,102 @@ + + +Callerany process + +Followermember + +Old leadermember + +New leadermember + + + +{register, Name, Pid, Meta} +1 + + +{register_req, Ref, F, Name, …} +2 + +Commits at V in epoch E + + +{names_batch, …, E, PrevV, V, L0} +3 + + +{register_reply, Ref, yes, V, E} +4 + +Replica isn't at V in E. Holds {awaiting_version, V, E, L0} +and asks for a resync. + +Dies before answering + +N takes over degraded. Its rebuild lacks the commit at V. + + +{apply_names_snapshot, …, E+1, MaxVersion} +5 + +Its epoch moved past E, so the held yes no longer counts. +The intent goes back to the leader. + + +{register_req, Ref2, F, Name, …} +6 + +Commits at V2 in epoch E+1, or refuses + + +{names_batch, …, E+1, PrevV, V2, N} +7 + + +{register_reply, Ref2, Result, V2, E+1} +8 + + +Result +9 + +Before this was fixed, F applied N's later-epoch batches and released the old yes once their versions passed V. The caller heard yes for +a binding no live member held. + diff --git a/docs/design/assets/registry_flows/join.svg b/docs/design/assets/registry_flows/join.svg new file mode 100644 index 0000000..99ab920 --- /dev/null +++ b/docs/design/assets/registry_flows/join.svg @@ -0,0 +1,95 @@ + + +Joining membernew node + +Electordgen_server + +BackendFDB keys + +Leadermember + +Followermember + + +{join, J, Token, Fresh = true} +1 + +In one transaction, adds J and keeps leader L. The +epoch stays E. + + +commit state and leader key {L, E} +2 + +Runs the post-commit action + + + +{elector_assume_and_distribute, J, …, E} +3 + +Same epoch and synced, so it takes the fast path. +Monitors J. + + +ok +4 + + +{apply_names_snapshot, Records, L, AllIds, Tokens, E, V} +5 + +Adopts the replica at (E, V) and becomes a follower + + +{peer_joined, J, Tokens} +6 + +Monitors J. Its replica is untouched. + +J SYNCED BEFORE, AS AFTER A PARTITION + +A helper fetches J's replica and probes conflicting +pids, then sends {joiner_gathered, …} + +Repairs conflicts (§5.6), then sends the snapshot +as above + diff --git a/docs/design/assets/registry_flows/legend.svg b/docs/design/assets/registry_flows/legend.svg new file mode 100644 index 0000000..ed38020 --- /dev/null +++ b/docs/design/assets/registry_flows/legend.svg @@ -0,0 +1,39 @@ +cast (async)call (waits for a reply)replydurable: backend or elector queuemonitor signalthe hop the flow is aboutmessage lost or delayed diff --git a/docs/design/assets/registry_flows/reads.svg b/docs/design/assets/registry_flows/reads.svg new file mode 100644 index 0000000..fccf750 --- /dev/null +++ b/docs/design/assets/registry_flows/reads.svg @@ -0,0 +1,95 @@ + + +Callerany process + +Names tablelocal ETS + +Followermember + +Leadermember + +Helperspawned process + +BackendFDB keys + +WHEREIS_NAME/1 + + + +ets:lookup(Table, Name) +1 + + +Pid or undefined +2 + +WHEREIS_NAME_CONSISTENT/1 ON A FOLLOWER'S NODE + + + +{whereis, Name} +3 + + +{whereis_req, Name, From} +4 + +Reads the value from its replica now + + +spawn verify_leader(L, E) +5 + + +read leader key +6 + + +{L, E} +7 + + +gen_server:reply(From, Pid) +8 + +If the leader is unreachable, F answers undefined itself. If the key +shows another leader, the helper answers undefined. + diff --git a/docs/design/assets/registry_flows/resync.svg b/docs/design/assets/registry_flows/resync.svg new file mode 100644 index 0000000..9051193 --- /dev/null +++ b/docs/design/assets/registry_flows/resync.svg @@ -0,0 +1,89 @@ + + +Leadermember + +Followermember + +New leadermember + + +{names_batch, Ops1, E, 10, 20, L} +1 + + +{names_batch, Ops2, E, 20, 30, L} +2 + + +{names_batch, Ops3, E, 30, 40, L} +3 + +PrevV 30 doesn't match applied 20. A gap, so it applies nothing. + + +{resync_req, F} +4 + +Won't ask again for 2 s + + +{apply_names_snapshot, Records, L, [], #{}, E, 40} +5 + +Re-baselined at (E, 40) + +A LOST TAIL WITH NO LATER WRITES + + +heartbeat {names_batch, [], E, 50, 50, L} +6 + +The heartbeat is ahead of its version, so it resyncs as above + +A BATCH FROM A LATER EPOCH + + +{names_batch, Ops, E+1, PrevV, V, N} +7 + +A gap even if PrevV matches. Only a snapshot moves a replica into a +new epoch. + diff --git a/docs/design/assets/registry_flows/unregister.svg b/docs/design/assets/registry_flows/unregister.svg new file mode 100644 index 0000000..08dad7f --- /dev/null +++ b/docs/design/assets/registry_flows/unregister.svg @@ -0,0 +1,117 @@ + + +Callerany process + +Followermember + +Leadermember + +Commit workerdgen_transaction + +Other followersmembers + +Registered pidmonitored + +EXPLICIT UNREGISTER THROUGH A FOLLOWER + + + +{unregister, Name} +1 + +Deletes its own row now and notes the +released pid + + +{unregister_req, Ref, F, Name} +2 + +Deletes its row. Enqueues {remove, Name, +Pid}. + + +commit batch +3 + + +{committed, V} +4 + + + + +{names_batch, [{name_unregistered, Name, Pid}], …} +5 + + +{unregister_reply, Ref} +6 + + +ok +7 + +With no reachable leader, F answers ok right away and keeps the unregister +as an intent. It sends {retract_req, Name, Pid} once a leader is reachable. + +THE REGISTERED PROCESS EXITS + + +{{down, Name}, MRef, process, Pid, Reason} +8 + +Enqueues {down, Name, MRef}. Skipped if Name +was rebound under a new monitor. + + +commit batch +9 + + +{committed, V} +10 + + + + +{names_batch, [{name_unregistered, Name, Pid}], …} +11 + diff --git a/docs/design/dgen_registry_design.md b/docs/design/dgen_registry_design.md index f491c63..f76c17e 100644 --- a/docs/design/dgen_registry_design.md +++ b/docs/design/dgen_registry_design.md @@ -7,14 +7,15 @@ so `gen_server`, `gen_statem`, `gen_event` and friends can be registered and add by name with no special API. It coordinates through a **strongly-consistent database** (FoundationDB is the default -and reference backend). That choice defines it: +and reference backend). That choice shapes the rest of the design: -- **It is CP.** The side of a partition that cannot reach the database stops accepting - registrations rather than risk giving one name to two processes (§5.3). +- **It is CP.** In CAP terms it picks consistency over availability. The side of a + partition that cannot reach the database stops accepting registrations rather than + risk giving one name to two processes (§5.3). - **It is a *singleton* registry.** A name maps to at most one live process - cluster-wide, and the registry will **terminate** processes to keep that true, so + cluster-wide. The registry will **terminate** processes to keep that true, so register only processes that can withstand being killed (§5.6). -- **It stores almost nothing durably.** Pids are never written to the database; the +- **It stores almost nothing durably.** Pids are never written to the database, so the durable footprint does not grow with the number of registered names (§4.4). This document is the source of truth for what the registry does. The @@ -23,17 +24,21 @@ This document is the source of truth for what the registry does. The behaviour that contradicts a guarantee is a bug, and behaviour listed as a non-goal is intentional. +[dgen_registry Message Flows](dgen_registry_message_flows.md) shows the messages +behind each operation as swimlane diagrams. + **This document is LLM generated.** ## 1. The one-minute model - Each node runs a **member** process holding a local copy of the `name → pid` table. -- One member is the **leader**. Every change — register, unregister, cleanup after a - process dies — goes through it; it applies changes in order and tells the others. +- One member is the **leader**. Every change goes through it: register, unregister, + and cleanup after a process dies. The leader applies changes in order and tells the + others. - The leader is chosen through the database, so every node agrees who leads, and a stale leader cannot make changes. -- Lookups are usually a local, lock-free ETS read; an authoritative lookup is available - when you need it. +- A lookup is usually a local, lock-free ETS read. When you need an authoritative + answer, you can ask the leader instead. ## 2. Quick start @@ -90,26 +95,27 @@ its own leader, membership, name table and database keys. They share nothing. OTP's via routing uses `whereis_name/1`, so a `gen_server:call` through a via-tuple takes the fast path and never queues behind the member's mailbox. -**Registration blocks until decided; `no` is always adjudicated.** `register_name/2` -answers `yes` if the name is now yours, and `no` **only** when the name is held by a -different live process or under an opt-in §8 policy (`reject_when_degraded`, or -`strict_replication` failing closed). With no reachable leader (none elected yet, a -handoff window, a partition) the call **blocks** and is re-driven once one is -established; a registration whose leader was deposed, or whose commit failed, is -retried or re-sent to the next leader. A `no` there would tell OTP's via machinery -"already started" for a name that is not taken. - -If nothing decides in time the call **exits** on `register_timeout` (§8, default 5 s), +**Registration blocks until a leader decides it, and `no` is always a real answer.** +`register_name/2` answers `yes` if the name is now yours. It answers `no` **only** when +a different live process holds the name, or when an opt-in §8 policy refuses it +(`reject_when_degraded`, or `strict_replication` failing closed). When no leader is +reachable, the call **blocks**. That happens before the first election, during a +handoff, or across a partition. We re-drive the call once a leader is established. If +the leader was deposed or the commit failed, we retry or re-send the registration to +the next leader. We never answer `no` in these cases, because OTP's via machinery would +read it as "already started" for a name nobody holds. + +If nothing decides in time, the call **exits** on `register_timeout` (§8, default 5 s), as `global`, `syn` and distributed `gproc` do. The name *may* then be bound to the pid -you tried to register: a retry answers `yes` (§5.2), and if the pid is abandoned its +you tried to register. A retry answers `yes` (§5.2), and if you abandon the pid, its exit clears the binding. -**Automatic cleanup.** The leader monitors every registered process; when one exits, -its name is unregistered automatically. +**Automatic cleanup.** The leader monitors every registered process. When one exits, +the leader unregisters its name. **A returned pid is a candidate, not a liveness guarantee.** Any pid the registry hands -back — from a lookup, a query, or a presence notification — may have died by the time -you use it. +back may have died by the time you use it. That applies to lookups, queries and +presence notifications alike. ## 4. How it works @@ -122,87 +128,99 @@ supervisor: |---|---| | **member** | Holds the local `name → pid` replica in a `protected` ETS table that it alone writes and anyone reads. Serves reads. When leader, it is the sole writer of the name table. | | **elector** | Tracks membership and decides the leader, coordinating through the database. | -| **connector** | Keeps Erlang distribution in step with membership (§4.6): connects to member nodes, reaps stranded members, probes leader liveness. | +| **connector** | Keeps Erlang distribution in step with membership (§4.6). It connects to member nodes, reaps stranded members, and probes whether the leader is alive. | Application code only talks to the member (directly or via the via-tuple). ### 4.2 Leadership comes from the database -There is no heartbeat or bespoke election protocol. Membership changes (join, leave) -are entries on a durable queue that the database serialises: each is applied exactly -once, in one global order, whichever node consumes it. The resulting member set and -leader are committed to the database, so every node reads the same answer. +There is no heartbeat or bespoke election protocol. Each join or leave is an entry on a +durable queue that the database serialises. Every entry is applied exactly once, in one +global order, no matter which node consumes it. The elector commits the resulting member +set and leader to the database, so every node reads the same answer. -Leadership is **sticky**: the leader keeps the role while it remains a member. If it -departs, the next membership change deterministically elects a replacement. +Leadership is **sticky**. The leader keeps the role for as long as it remains a member. +If it departs, the next membership change deterministically elects a replacement. -Each leadership term carries an increasing **epoch**, also written into the durable -leadership record. Messages from an older epoch are ignored, and the epoch makes the -fence of §5.1 a monotonic fencing token. +Each leadership term carries an increasing **epoch**, which is also written into the +durable leadership record. Members ignore messages from an older epoch. Because the +epoch only grows, it also serves as the fencing token of §5.1. ### 4.3 The leader is the sole writer Every change to the name table — register, unregister, and auto-unregister on process death — is applied by the leader through its own mailbox, one at a time, so changes are totally ordered without locks. A follower **forwards** registrations to the leader and -relays the verdict. No member ever blocks its message loop waiting on another member, -which rules out circular waits. +passes the answer back to the caller. No member ever blocks its message loop waiting on +another member, which rules out circular waits. + +Writes reach the database through a group-commit pipeline. A worker outside the +leader's loop runs the commits, with at most one in flight. There is no batching +window, so an idle leader commits the first op immediately. Ops that arrive while a +commit is in flight queue up. When the worker reports its result, the leader starts the +next commit with everything queued, up to `commit_batch_size` (§8). Batches therefore +grow only under load, and an unloaded leader adds no batching delay. ### 4.4 What the database actually stores The database is for **coordination, not storage**. A pid read back later could name an -unrelated, reused process, so pids are never persisted; the name table lives only in -the members' ETS replicas, recreated empty whenever a member (re)starts. +unrelated process that reused it, so we never persist pids. The name table lives only +in the members' ETS replicas, and a member starts with an empty one every time it +starts or restarts. The name table costs **two keys**, regardless of how many names are registered: -- a **leader key** — the current leader and its epoch (the fencing token, §5.1); -- a **version key** — rewritten once per batch of committed changes with a - monotonically increasing commit version. It makes each leader write a real - transaction (so the fence engages) and gives every batch a global order. +- a **leader key**, which holds the current leader and its epoch. This is the fencing + token of §5.1. +- a **version key**, which the leader rewrites once per batch of committed changes + with a monotonically increasing commit version. Writing it turns every leader commit + into a real transaction, so the fence takes effect. It also gives every batch a + global order. -The elector's state (membership queue, §4.2; subscriptions, §4.9) is bounded by cluster -size, churn and subscription count. Nothing scales with the number of registered names. +The elector also keeps durable state: the membership queue (§4.2) and the subscriptions +(§4.9). Its size depends on cluster size, churn and subscription count. Nothing scales +with the number of registered names. ### 4.5 Replication After committing a batch, the leader broadcasts it to the other members as one message -carrying its version and the previous batch's version. A member applies a batch only -if it is **contiguous** with what it holds; on a gap (e.g. a dropped message) it stops -and asks the leader for a fresh snapshot. Every replica is therefore a *prefix* of the -leader's ordered stream — the property the handoff reconstruction (§5.7) relies on. +carrying its version and the version of the batch before it. A member applies a batch +only if it is **contiguous** with what it holds. If there is a gap, for example from a +dropped message, the member stops and asks the leader for a fresh snapshot. Every +replica is therefore a *prefix* of the leader's ordered stream. The handoff +reconstruction (§5.7) relies on this. ### 4.6 Joining, leaving, and connectivity -A node joins by starting the registry and leaves by stopping it (or failing). There is -no node list and no cluster-formation step: membership lives in the database and is +A node joins by starting the registry, and leaves by stopping it or by failing. There +is no node list and no cluster-formation step. Membership lives in the database and is read transactionally, so every node sees the same membership. -The registry also builds the Erlang distribution mesh. At startup, periodically, and -whenever a node becomes reachable, each member reads the member set and connects to any -member node it is not connected to. You do not need a discovery library or -`net_adm:ping`; a new node connects outward to the members it finds in the database, -which triggers its first state sync. Nodes must still be *able* to connect: shared -cookie, resolvable and reachable on the network. +The registry also builds the Erlang distribution mesh. Each member reads the member set +at startup, periodically, and whenever a node becomes reachable. It then connects to +any member node it is not yet connected to. You do not need a discovery library or +`net_adm:ping`. A new node connects outward to the members it finds in the database, +and that triggers its first state sync. Nodes must still be *able* to connect. They +need a shared cookie and must be resolvable and reachable on the network. #### Delegating connectivity (`connectivity => provided_externally`) -Distribution links are node-global, but each registry runs its own mesh (a periodic -member-set read and connect sweep), so on a node hosting many registries that work -scales with the number of registries. A registry started with -`connectivity => provided_externally` turns its mesh off and free-rides on links -another registry maintains. The intended shape is one **system registry** in the -default `self_managed` mode on **every** node, plus many `provided_externally` tenant -registries. - -**Contract.** The provider must run on a **node-superset** of every consumer's nodes. -Otherwise a tenant member on an uncovered node is **silently isolated**: it cannot -reach a leader, so its writes block and time out (§3). This is the deployer's -responsibility; as a backstop, a `provided_externally` registry with no elected leader -for ~60 s logs a warning naming the contract. - -Only the mesh is delegated: the stranded-member reap, the leader-liveness probe (§5.3) -and the durable-epoch nudge (§5.7) stay active. Any value other than +Distribution links belong to the whole node, but each registry runs its own mesh. The +mesh periodically reads the member set and sweeps through it connecting nodes. On a +node that hosts many registries, that work grows with the number of registries. A +registry started with `connectivity => provided_externally` turns its mesh off and +relies on the links another registry maintains. The intended setup is one **system +registry** in the default `self_managed` mode on **every** node, plus many +`provided_externally` tenant registries. + +**Contract.** The provider must run on every node that any consumer runs on. Otherwise +a tenant member on an uncovered node is **silently isolated**. It cannot reach a +leader, so its writes block and time out (§3). Meeting this contract is the deployer's +job. As a backstop, a `provided_externally` registry that has had no elected leader for +about 60 s logs a warning that names the contract. + +Only the mesh is delegated. The stranded-member reap, the leader-liveness probe (§5.3) +and the periodic epoch check (§5.7) stay active. Any value other than `provided_externally`, including a typo, resolves to `self_managed`. ### 4.7 Metadata and queries @@ -229,45 +247,47 @@ dgen_registry:query(my_registry, #{role => worker}), %% local snapsh dgen_registry:query_consistent(my_registry, #{role => worker}). %% leader-authoritative ``` -**Reads.** Metadata lives in the same replica row `whereis_name/1` reads. -`get_metadata/1` is a lock-free caller-side read like `whereis_name/1`; -`get_metadata_consistent/1` is its authoritative counterpart. Queries use a per-member -inverted index and run **through the member's mailbox** (on the leader for -`query_consistent/2`): the member applies a batch row by row, so a lock-free multi-key -read could see it half-applied, whereas a mailbox-serialised query sees whole batches. +**Reads.** Metadata lives in the same replica row that `whereis_name/1` reads. +`get_metadata/1` is a lock-free read in the calling process, like `whereis_name/1`. +`get_metadata_consistent/1` is its authoritative counterpart. Queries use an inverted +index on each member and run **through the member's mailbox**. For +`query_consistent/2` that is the leader's mailbox. We route queries this way because +the member applies a batch row by row. A lock-free read across several keys could see a +batch half-applied, but a query that waits its turn in the mailbox only sees whole +batches. **Query semantics.** `Constraints :: #{attr() => value()}` matches every registration -whose `index` satisfies *every* clause (AND of exact equalities; no ranges, `OR`, -negation or ordering). An empty map is rejected with `{error, empty_query}`. An -attribute no registration carries simply matches nothing — there is no schema. `data` -is never queryable. - -**Writes.** `register_name/3` and `set_metadata/2` ride the same fenced, sole-writer -commit pipeline as `register_name/2` (§4.3); metadata adds no durable state. -`set_metadata/2` on an unregistered name returns `{error, not_registered}`; if the leader -loses leadership before answering it, the call returns `{error, no_leader}` so the -caller can retry. A follower that forwarded a `set_metadata/2` answers only after its own -row is updated, so a following local `get_metadata/1` on that node sees the write. - -**Lifetime and handoff.** The unregister/`DOWN` path that clears a name clears its -metadata and index entries. On handoff, metadata travels with the binding through the -reconstruction of §5.7; conflict detection (§5.6) adjudicates pids only. +whose `index` satisfies *every* clause. Each clause is an exact equality, and there are +no ranges, `OR`, negation or ordering. An empty map is rejected with +`{error, empty_query}`. There is no schema, so an attribute that no registration +carries simply matches nothing. `data` is never queryable. + +**Writes.** `register_name/3` and `set_metadata/2` go through the same fenced, +single-writer commit pipeline as `register_name/2` (§4.3). Metadata adds no durable +state. `set_metadata/2` on an unregistered name returns `{error, not_registered}`. If +the leader loses leadership before it answers, the call returns `{error, no_leader}` so +the caller can retry. A follower that forwarded a `set_metadata/2` answers only after +its own row is updated, so a later local `get_metadata/1` on that node sees the write. + +**Lifetime and handoff.** When an unregister or a `DOWN` clears a name, it also clears +the name's metadata and index entries. On handoff, metadata travels with the binding +through the reconstruction of §5.7. Conflict detection (§5.6) compares pids only. ### 4.8 Process identity Starting a registry creates **zero atoms**, so an application can start many registries -dynamically (one per tenant, say) without leaking atoms. Only the caller-supplied `Name` -is used: the member is registered as `Name` and its ETS table is named `Name`; the -supervisor and elector are unregistered (the elector is found through the member's -supervisor). The supervisor is `one_for_all`, so the member's cached elector pid is -never stale. +dynamically, such as one per tenant, without leaking atoms. We use only the +caller-supplied `Name`. The member is registered as `Name`, and its ETS table is named +`Name`. The supervisor and the elector have no registered name. Callers find the +elector through the member's supervisor. The supervisor is `one_for_all`, so the +elector pid that the member caches can never be stale. ### 4.9 Presence: watch/notify subscriptions -A **presence subscription** turns a query (§4.7) into a live feed: under an -application-chosen id, a caller registers a **watch query** and a **notify query**, and -the registry pushes a message to the notify targets whenever the set of processes -matching the watch query changes. +A **presence subscription** turns a query (§4.7) into a live feed. A caller picks an id +and registers a **watch query** and a **notify query** under it. Whenever the set of +processes matching the watch query changes, the registry sends a message to every +process matching the notify query. ```erlang %% Keep every process matching `notify` up to date with the set matching `watch`. @@ -280,55 +300,59 @@ ok = dgen_registry:subscribe(my_registry, dgen_registry:unsubscribe(my_registry, {presence, RoomId}). ``` -A `query/0` is `{all, #{attr => value}}` (the `query/2` semantics) or a bare map read as -`{all, Map}`. The type is tagged so it can grow without breaking existing subscriptions. +A `query/0` is either `{all, #{attr => value}}`, with the same meaning as in `query/2`, +or a bare map, which is read as `{all, Map}`. The type is tagged so it can grow without +breaking existing subscriptions. **Durable, keyed by an application id.** Subscriptions live in the elector's durable -state (§4.2), are tied to no process, and survive a full cluster restart. `subscribe` -with an existing `SubId` is an idempotent **upsert**, so applications can model -subscriptions as database entities and scale to zero and back with them intact. +state (§4.2). They are not tied to any process, and they survive a full cluster +restart. Calling `subscribe` with an existing `SubId` is an idempotent **upsert**. That +lets applications model subscriptions as database entities, and scale to zero and back +with the subscriptions intact. **API.** `subscribe/4`, `unsubscribe/2`, `unsubscribe_all/1` and `subscriptions/1` go -to the elector. Writes are asynchronous: `ok` means accepted for durable processing; the -change commits (and shows in `subscriptions/1`) shortly after. Re-subscribing recovers a -write lost before it was enqueued. +to the elector. Writes are asynchronous. An `ok` means the elector accepted the change +for durable processing. The change commits shortly after, and then shows up in +`subscriptions/1`. If a write is lost before the elector enqueues it, subscribing again +recovers it. -**Teardown.** A registry's durable keys share one keyspace prefix; `delete/2(Name, -Tenant)`, called after stopping the supervisor, removes them all in one range clear. +**Teardown.** A registry's durable keys share one keyspace prefix. After you stop the +supervisor, `delete/2(Name, Tenant)` removes all of them in one range clear. `unsubscribe_all/1` clears only the subscriptions of a running registry. -**Change detection.** As the leader applies each committed batch it tests every changed -row against each watch query: a name that starts matching is `joined`, one that stops is -`left`, and a rebind to a new pid is both. This covers register, unregister, process -death and `set_metadata`. +**Change detection.** As the leader applies each committed batch, it tests every +changed row against each watch query. A name that starts matching is `joined`, and one +that stops matching is `left`. A name rebound to a new pid produces both. This covers +register, unregister, process death and `set_metadata`. -**Initial snapshot.** Whenever a process *enters* the notify set — at `subscribe` time -or later — it first receives the current watch set as `joined` events, then deltas. A -subscription can therefore predate its viewers. +**Initial snapshot.** Whenever a process *enters* the notify set, it first receives the +current watch set as `joined` events, and then the changes after that. This holds both +at `subscribe` time and later. A subscription can therefore exist before any of its +viewers do. -**Leader state.** Committed subscription changes are pushed to the current leader; a -newly-assuming leader pulls the full durable set and recomputes every watch set, which -also reconciles any change it missed. Followers hold no presence state. +**Leader state.** The elector pushes committed subscription changes to the current +leader. A newly elected leader pulls the full durable set and recomputes every watch +set, which also catches up on any change it missed. Followers hold no presence state. -**Consistency.** Delivery is asynchronous and in commit order: each viewer sees an -initial snapshot, then a prefix of the true sequence of watch-set transitions. Delivered +**Consistency.** Delivery is asynchronous and in commit order. Each viewer sees an +initial snapshot, then a prefix of the true sequence of watch-set changes. Delivered pids are candidates (§3). ## 5. Consistency and failure model ### 5.1 Fencing: a stale leader cannot write — or serve consistent reads -When leadership moves, the new leader's identity and epoch are committed to the -database. From then on the old leader is **fenced**: its commits conflict with the -leadership record and the database rejects them. Because the record carries the epoch, -a member deposed and later re-elected cannot land a write from its earlier term either. -At most one leader can write, enforced by transaction conflict detection rather than -timing — so there is no split brain. - -Consistent **reads** are fenced too. A deposed leader may not yet know it (e.g. on the -minority side of a partition), so every `*_consistent` answer is released only after -the member re-verifies its leadership (id and epoch) against the durable record. If -verification fails or cannot reach the database, it answers not-registered/empty. The +When leadership moves, the elector commits the new leader's identity and epoch to the +database. From then on the old leader is **fenced**. Its commits conflict with the +leadership record, and the database rejects them. Because the record carries the epoch, +a member that was deposed and later re-elected cannot land a write from its earlier +term either. Transaction conflict detection, not timing, ensures that at most one +leader can write, so there is no split brain. + +Consistent **reads** are fenced too. A deposed leader may not know it yet, for example +on the minority side of a partition. So before the member releases any `*_consistent` +answer, it re-checks its leader id and epoch against the durable record. If the check +fails or cannot reach the database, the member answers not-registered or empty. The cost is one database read per consistent read. ### 5.2 Linearizable registration @@ -338,33 +362,34 @@ time, so registration is **linearizable**: there is one order of decisions, and `register_name/2` returns `yes`, every later consistent read observes it until it changes. -Registration is **idempotent for the same holder**: registering a name to the pid that -already holds it returns `yes`. This makes redriving safe — a caller that timed out -(§3) but whose registration committed gets `yes` on retry. +Registration is **idempotent for the same holder**. Registering a name to the pid that +already holds it returns `yes`. That makes re-driving safe. A caller that timed out +(§3) after its registration committed gets `yes` when it retries. ### 5.3 Partition behaviour: CP Only the side that reaches the database and holds leadership can commit. On the other -side, registrations block and time out (§3), an unregister answers `ok` and is held -until a leader can take it (Non-goal 5), and consistent reads answer not-registered -(§5.1); local snapshot reads keep working but may be stale. Availability of writes is -traded for consistency. - -In an **Erlang-distribution-only** partition (mesh severed, database reachable from -both sides) the sides briefly contend over the single leadership record; then one side -leads and the other blocks as above until the mesh heals. The leader-liveness probe -only reaps a leader on a node the prober has *never* connected to (cold-restart -recovery), so an isolated member cannot depose a healthy leader. On heal, §5.6 repairs -any name reissued across the split. +side, registrations block and then time out (§3). An unregister answers `ok` and is +held until a leader can take it (Non-goal 5). Consistent reads answer not-registered +(§5.1). Local snapshot reads keep working, but they may be stale. The registry gives up +write availability to stay consistent. + +Sometimes only Erlang distribution is partitioned. The mesh is cut, but both sides +can still reach the database. The two sides then briefly compete for the single leadership +record. One side wins and leads, and the other blocks as above until the mesh heals. +The leader-liveness probe only reaps a leader on a node the prober has *never* +connected to. That case exists to recover from a cold restart, and it means an isolated +member cannot depose a healthy leader. When the partition heals, §5.6 repairs any name +that was reissued across the split. ### 5.4 Uniqueness is single-fault tolerant -**Uniqueness** — at most one live process per name — holds across the failure of **any -one member at a time**. Pids are not durable, so the only record that a name is taken -is the members' in-memory replicas. Every acknowledged registration is held by at least -two members (§5.5), so losing one member never loses that knowledge; losing *both* -holders at once can, after which the name could be reissued. The registry detects and -repairs that case (§5.6). +**Uniqueness** means at most one live process per name. It holds across the failure of +**any one member at a time**. Pids are not durable, so the only record that a name is +taken is in the members' in-memory replicas. Every acknowledged registration is held by +at least two members (§5.5), so losing one member never loses that record. Losing +*both* holders at once can lose it, and then the name could be reissued. The registry +detects and repairs that case (§5.6). ### 5.5 Addressability and the two-holder rule @@ -372,42 +397,52 @@ repairs that case (§5.6). When `register_name/2` returns `yes`, the binding is normally held by **two members**: - A registration **forwarded** from a follower is held by the leader and that follower. - The follower answers `yes` only once its replica has applied the batch's commit - version (deferring if it is mid-resync), so the second copy is always visible to the - handoff reconstruction (§5.7). + The follower answers `yes` only once its replica has applied the commit version of + that batch. If the follower is in the middle of a resync, it holds the answer until + the resync finishes. This way the second copy is always visible to the handoff reconstruction + (§5.7). + The commit version only counts in the epoch that committed it. A later leader may + have taken over without that commit, so the follower never releases the answer on a + later leader's versions. If its replica moves to a later epoch first, it sends the + registration again. For the same reason a follower treats a batch from a later epoch + as a gap, so only a snapshot moves it into a new epoch. - A registration made **on the leader's node** waits for `register_replicas` follower - confirmations (§8) before `yes`. If none arrive within `replicate_timeout`, the - default policy **degrades open** and answers `yes` with a single holder (emitting a - warning and telemetry event); `strict_replication` fails it closed instead. + confirmations (§8) before answering `yes`. If none arrive within + `replicate_timeout`, the default policy **degrades open**. It answers `yes` with a + single holder and emits a warning and a telemetry event. With `strict_replication` + the registration fails closed instead. The rule protects the *routing entry* against losing a node that does not host the -process (typically the leader). If the process's own node dies, the process is gone and -its name is released. +process, which is typically the leader's node. If the process's own node dies, the +process is gone and its name is released. ### 5.6 Conflict resolution by termination -If a process held a name on a node unreachable long enough to be dropped, and the name -was reissued elsewhere, two live processes can claim the name when the node returns. -On observing two *different, live* pids for one name, the registry **terminates both**, -clears the name, and raises an alarm (a log line and, if `telemetry` is present, a -`[dgen_registry, conflict]` event). Supervised processes restart and re-register under -the single leader. Killing both, rather than picking a winner, is deliberate. +Suppose a process holds a name on a node that stays unreachable long enough to be +dropped, and the name is reissued elsewhere. When the node returns, two live processes +claim the name. When the registry sees two *different, live* pids for one name, it +**terminates both**, clears the name, and raises an alarm. The alarm is a log line, +plus a `[dgen_registry, conflict]` event if `telemetry` is present. Supervised +processes then restart and re-register through the single leader. Killing both, rather +than picking a winner, is deliberate. -Detection runs at every **leadership handoff**, against the gathered replicas (§5.7), -and when a previously-synced member **rejoins a continuing leader** (the usual partition -heal), against the rejoiner's replica; a never-synced member is skipped. Repair needs a -*witness*: a binding no surviving replica reports cannot be detected (Non-goal 2). +Detection runs in two places. At every **leadership handoff**, the new leader checks +the replicas it gathered (§5.7). When a previously synced member **rejoins a continuing +leader**, which is how a partition usually heals, the leader checks the rejoining +member's replica. We skip a member that never synced. Repair needs a *witness*. If no +surviving replica reports a binding, we cannot detect the conflict (Non-goal 2). **Staleness is not conflict.** A lagging member may still report a binding that was -legitimately unregistered while its process lives on. Every member therefore keeps a -**release trail** of explicit unregisters of live processes, keyed by name-and-pid, for -`conflict_release_ttl` (§8); it is merged across handoffs. Entries expire only while -the cluster is fully connected and no member has departed within the last TTL. A member -gone longer than the TTL is adjudicated with whatever trail remains. +legitimately unregistered while its process kept running. So every member keeps a +**release trail**. It records each explicit unregister of a live process, keyed by name +and pid, for `conflict_release_ttl` (§8). The trail is merged across handoffs. Entries +expire only while the cluster is fully connected and no member has departed within the +last TTL. When a member has been gone longer than the TTL, we judge its bindings +against whatever trail remains. -**Kill budget.** `conflict_kill_budget` (§8) bounds terminations per name before -escalating to an operator. It is tracked by the current leader and reset on handoff, so -the bound is per name *per leader term*. +**Kill budget.** `conflict_kill_budget` (§8) limits how many times we terminate +processes for one name before escalating to an operator. The current leader tracks the +budget, and it resets on handoff, so the limit is per name *per leader term*. This is the registry's explicit contract: @@ -415,41 +450,51 @@ This is the registry's explicit contract: > Register only processes that can withstand being forcibly killed — for example, > supervised and restart-safe, or transient by design. -Termination can be turned off (`terminate_on_conflict`, leaving detect-and-alarm), and -`reject_when_degraded` prevents the reissue in the first place at some cost in -availability — see [Configuration](#8-configuration). +You can turn termination off with `terminate_on_conflict`, which leaves detection and +the alarm in place. You can also prevent the reissue in the first place with +`reject_when_degraded`, at some cost in availability. See +[Configuration](#8-configuration). ### 5.7 Leadership handoff -A new leader rebuilds its name table by gathering every reachable member's replica in -parallel and taking the freshest. The old leader is already fenced, so no global pause -is needed. Until the gather completes, writes block (§3) and are re-driven once the new -leader assumes. - -Because replicas are gap-checked prefixes of one ordered stream (§4.5), the freshest -replica holds the current binding for every name any reachable member knows, so a -two-holder binding survives any single unreachable member. A binding held *only* by an -unreachable member (a degrade-open single holder, or a multi-fault loss) is absent from -the rebuilt table and is not restored when that member returns; if the name was reissued -meanwhile, §5.6 resolves the conflict. - -Two guards keep this sound when a committed broadcast from the deposed leader is still -in flight to a member during the gather: - -- The gather is **fenced against the durable version key** (§4.4): no live member can be - ahead of it, so the new leader re-gathers (bounded retries) until its freshest replica - reaches the key. Past the bound it proceeds, and the handoff is **degraded** — it - stopped short of the version key and may be missing committed bindings - (`reject_when_degraded`, §8, acts on this). -- Every wholesale re-baseline (handoff snapshot or resync) is **version-monotonic**: a - member rejects one older than what it holds, so a handoff never rolls a replica back. +A new leader rebuilds its name table. It gathers the replica of every reachable member +in parallel and takes the freshest one. The old leader is already fenced, so no global +pause is needed. Until the gather completes, writes block (§3). We re-drive them once +the new leader takes over. + +Replicas are gap-checked prefixes of one ordered stream (§4.5). So the freshest replica +holds the current binding for every name that any reachable member knows, and a +two-holder binding survives any single unreachable member. Sometimes a binding is held +*only* by an unreachable member. That happens with a degrade-open single holder, or +after a multi-fault loss. Such a binding is missing from the rebuilt table, and we do +not restore it when that member returns. If the name was reissued in the meantime, §5.6 +resolves the conflict. + +The deposed leader may have committed a batch whose broadcast is still on its way to a +member during the gather. Two guards keep the handoff sound in that case: + +- The gather is **fenced against the durable version key** (§4.4). No live member can + be ahead of that key, so the new leader gathers again, a bounded number of times, + until its freshest replica reaches the key. Past that bound it proceeds anyway, and + the handoff is **degraded**. A degraded handoff stopped short of the version key and + may be missing committed bindings. `reject_when_degraded` (§8) acts on this. + + Waiting only helps while a broadcast can still arrive. If the deposed leader died + after a commit but before broadcasting it, no live member will ever reach the key. + Erlang delivers a process's messages before its DOWN, so each member reports whether + it has handled the DOWN for the leader it followed. Once the new leader and every + peer have, nothing more can arrive, and the new leader proceeds degraded without + waiting out the bound. +- Every wholesale replacement of a replica, whether a handoff snapshot or a resync, is + **version-monotonic**. A member rejects one older than what it holds, so a handoff + never rolls a replica back. The TLA+ model in [`formal/README.md`](../../formal/README.md) checks this argument. -A handoff always reaches its new leader eventually: every member periodically compares -the epoch it has heard with the committed one and re-announces itself on a mismatch. -Direct registrations awaiting replica confirmations when leadership is lost are resolved -under the replication timeout policy (§8). +A handoff always reaches its new leader eventually. Every member periodically compares +the epoch it has heard with the committed one, and announces itself again if they +differ. If leadership is lost while direct registrations are still waiting for +replica confirmations, the replication timeout policy (§8) decides their outcome. ## 6. Guarantees @@ -462,8 +507,8 @@ These are normative. Behaviour that violates one of these is a defect. 2. **No split-brain writes.** At most one leader can commit changes. A leader that has lost leadership cannot register, unregister, or otherwise mutate the name table. There is no merge step and no dual-accept window. -3. **Linearizable registration.** Successful registrations have a single global order; - after `register_name/2` returns `yes`, every later `whereis_name_consistent/1` +3. **Linearizable registration.** Successful registrations have a single global order. + After `register_name/2` returns `yes`, every later `whereis_name_consistent/1` reflects it until it changes. 4. **Two-holder durability of acknowledged registrations.** A registration acknowledged `yes` is held by at least two members and survives any single node failure — @@ -471,31 +516,31 @@ These are normative. Behaviour that violates one of these is a defect. may have one holder. `strict_replication` makes the property unconditional ([Configuration](#8-configuration)). 5. **Authoritative consistent reads.** `whereis_name_consistent/1` reflects the current - leader's committed view of the name, even across an unnoticed deposition: every - answer is fence-verified against the durable leadership record (§5.1), so a deposed - leader answers not-registered rather than a stale value. + leader's committed view of the name, even when a leader was deposed without knowing + it. The leader checks every answer against the durable leadership record (§5.1), so + a deposed leader answers not-registered rather than a stale value. 6. **Automatic unregistration.** When a registered process exits, its name is unregistered without an explicit call. -7. **CP under partition.** The side of a partition that cannot reach the database (or - does not hold leadership) does not accept registrations; it never invents a second - owner for a name. +7. **CP under partition.** The side of a partition that cannot reach the database, or + that does not hold leadership, does not accept registrations. It never invents a + second owner for a name. 8. **Registry isolation.** Distinct registries do not share leadership, membership, name tables, or database keys. 9. **Consistent membership and convergent connectivity.** Every node observes the same - committed member set. Given nodes able to connect (shared cookie, network - reachability), after a node's join commits the cluster converges so that `nodes()` on - each member returns every other reachable member, with no external discovery. In - `connectivity => provided_externally` mode (§4.6, §8) the registry does not connect - nodes itself, and convergence rests on the provider registry. + committed member set. Assume the nodes are able to connect, with a shared cookie and + network reachability. Then after a node's join commits, the cluster converges until + `nodes()` on each member returns every other reachable member, with no external + discovery. In `connectivity => provided_externally` mode (§4.6, §8) the registry does + not connect nodes itself, and convergence depends on the provider registry. 10. **Metadata lifetime is tied to the registration.** A registration's `index`, `data` and index entries exist exactly while the registration does and are removed on unregister or process exit, by the same path (§4.7). -11. **Metadata writes are fenced and linearizable.** `set_metadata/2` rides the same - fenced, single-leader commit pipeline as `register_name/2,3`; a leader that has +11. **Metadata writes are fenced and linearizable.** `set_metadata/2` goes through the + same fenced, single-leader commit pipeline as `register_name/2,3`. A leader that has lost leadership cannot apply or broadcast a metadata change. 12. **Metadata reads mirror name reads.** `get_metadata/1` is a lock-free snapshot read - with the freshness of `whereis_name/1`; `get_metadata_consistent/1` is - leader-authoritative, mirroring `whereis_name_consistent/1`. + with the freshness of `whereis_name/1`. `get_metadata_consistent/1` is answered by + the leader, like `whereis_name_consistent/1`. 13. **Queries are batch-consistent.** `query/2` and `query_consistent/2` never observe a half-applied commit batch (§4.7). Both are still snapshots, and a returned pid may already have died. @@ -509,43 +554,43 @@ does not promise. member of a registry is down at once, its names are gone and applications must re-register on startup. 2. **Multi-fault uniqueness.** The simultaneous loss of both holders of a binding may - lose the registration and can transiently allow the name to be reissued. This is - detected and repaired (§5.6) when a surviving replica still reports the old binding; - with no surviving witness the displaced process simply loses its registration and is - not terminated. It is not prevented unless `reject_when_degraded` is enabled, which - trades availability for it. -3. **Availability during a partition.** The disconnected/minority side cannot register - names (§5.3). + lose the registration and can briefly allow the name to be reissued. If a surviving + replica still reports the old binding, the registry detects and repairs this (§5.6). + With no surviving witness, the displaced process simply loses its registration and + is not terminated. Nothing prevents it unless `reject_when_degraded` is enabled, + which gives up some availability to do so. +3. **Availability during a partition.** The disconnected or minority side cannot + register names (§5.3). 4. **Strong freshness from `whereis_name/1`.** The snapshot read may briefly lag a change made elsewhere. Use `whereis_name_consistent/1` for the authoritative answer. 5. **Synchronous cluster-wide unregistration.** `unregister_name/1` normally returns `ok` - after the removal has committed on the leader, but does not wait for every node to - observe it; replication is asynchronous. Propagation is still guaranteed: an - unregister whose commit fails, whose leader is deposed, whose forward is dropped, or - that arrives with no reachable leader is re-driven to the current leader rather than - dropped. + after the removal has committed on the leader. It does not wait for every node to + see it, because replication is asynchronous. The removal still always propagates. + We re-drive an unregister to the current leader instead of dropping it when its + commit fails, its leader is deposed, its forward is lost, or no leader is reachable. 6. **Preservation of a process the registry must terminate.** The registry may terminate a registered process to enforce uniqueness (§5.6). A process that is neither - restart-safe nor transient by design is not a good fit, and its termination is by - design. + restart-safe nor transient by design is not a good fit. Terminating it is intended + behaviour, not a bug. 7. **Automatic re-registration after failure.** After a multi-fault loss, detecting the loss and re-registering (typically via a supervisor) is the application's job. 8. **Rich queries.** Queries support only a conjunction of exact-equality clauses over - indexed attributes — no ranges, `OR`, negation, prefix/substring matching, ordering, - pagination, or joins (§4.7). -9. **Querying non-indexed metadata.** `data` is returned only by `get_metadata/1` / - `get_metadata_consistent/1`; it is never matched by or used to filter a query. -10. **Durability of metadata across a full restart.** Metadata is in-memory only, like - names (item 1); applications re-set it when they re-register. + indexed attributes. There are no ranges, `OR`, negation, prefix or substring + matching, ordering, pagination, or joins (§4.7). +9. **Querying non-indexed metadata.** Only `get_metadata/1` and + `get_metadata_consistent/1` return `data`. A query never matches or filters on it. +10. **Durability of metadata across a full restart.** Like names (item 1), metadata + lives only in memory. Applications set it again when they re-register. 11. **Uniqueness or constraints on indexed values.** Many registrations may share an - indexed attribute/value pair; there is no unique-attribute constraint. -12. **Per-pid (cross-name) metadata.** Metadata is per registration: a process - registered under several names has independent metadata for each. + indexed attribute and value. There is no unique-attribute constraint. +12. **Per-pid (cross-name) metadata.** Metadata belongs to a registration, not a pid. + A process registered under several names has separate metadata for each. ## 8. Configuration -Defaults make the registry single-fault tolerant; the knobs move it along the -availability/strictness spectrum. Configuration is **per registry**, passed at start: +The defaults make the registry single-fault tolerant. The settings below trade +availability against strictness. Configuration is **per registry**, and you pass it at +start: ```erlang dgen_registry:start_link(orders, Tenant, #{ @@ -554,51 +599,52 @@ dgen_registry:start_link(orders, Tenant, #{ }). ``` -An unset key falls back to the `dgen` application environment (a global default), then -to the built-in default below. +An unset key falls back to the `dgen` application environment, which acts as a global +default, and then to the built-in default below. | Setting | Default | Effect | |---|---|---| | `register_timeout` | `5000` ms | How long a leader-routed write (`register_name/2,3`, `unregister_name/1`, `set_metadata/2`) waits before the caller exits (§3). | -| `register_replicas` | `1` | Follower copies a *direct* registration waits for before `yes` (§5.5), bounded by the number of followers. Higher widens durability at the cost of latency. | -| `commit_batch_size` | `5000` | Max write ops coalesced into one group commit; the rest ride the next. Bounds per-commit work so a burst (e.g. a departing node's flood of `DOWN`s) keeps the leader responsive. Affects latency/throughput only, never correctness. | +| `register_replicas` | `1` | How many follower copies a *direct* registration waits for before answering `yes` (§5.5), capped at the number of followers. A higher value makes registrations more durable but slower. | +| `commit_batch_size` | `5000` | The most write ops coalesced into one group commit. The rest go in the next commit. This bounds the work per commit, so the leader stays responsive during a burst such as the flood of `DOWN`s from a departing node. It affects only latency and throughput, never correctness. | | `replicate_timeout` | `1000` ms | How long the leader waits for those confirmations before applying the timeout policy. | -| `strict_replication` | `false` | Timeout policy. `false`: **degrade open** — answer `yes` with whatever holders exist (possibly one), with a warning and telemetry event. `true`: **fail closed** — answer `no` and retract the binding; the retraction is handed to the next leader if this one is deposed first, so the binding cannot survive. | -| `terminate_on_conflict` | `true` | Whether a detected conflict (§5.6) terminates the claimants, or only detects and alarms. | +| `strict_replication` | `false` | The timeout policy. With `false`, the leader **degrades open**. It answers `yes` with whatever holders exist, possibly just one, and emits a warning and a telemetry event. With `true`, it **fails closed**. It answers `no` and retracts the binding. If this leader is deposed first, the next leader takes over the retraction, so the binding cannot survive. | +| `terminate_on_conflict` | `true` | Whether a detected conflict (§5.6) terminates both claimants, or only raises the alarm. | | `conflict_kill_budget` | `{3, 60000}` | At most *N* terminations per name per window (ms) before escalating to an operator. Per name *per leader term* (§5.6). | | `conflict_release_ttl` | `600000` ms | How long an explicitly released live name/pid pair stays in the release trail (§5.6). Expiry pauses while any member is disconnected and for one TTL after any member departs. Entries are tiny. | -| `reject_when_degraded` | `false` | Optional prevention (§5.6). When `true`, a leader whose last handoff was **degraded** (reconstruction stopped short of the durable version key, §5.7) answers `no` to registering a name it does not already hold. Merely-behind or never-synced members do not cause degradation, so normal scale-up/down does not trip it; losing a binding's only holders does. Trades availability for prevention. | -| `connectivity` | `self_managed` | Who maintains the distribution mesh (§4.6). `self_managed`: this registry runs its own. `provided_externally`: mesh off; relies on a `self_managed` registry spanning a node-superset of this one's nodes, else the node is silently isolated (a sustained no-leader window logs a warning). Registry-scoped backstops stay active. Any other value resolves to `self_managed`. | +| `reject_when_degraded` | `false` | Prevents reissued names instead of repairing them (§5.6). When `true`, a leader whose last handoff was **degraded** answers `no` to registering a name it does not already hold. A handoff is degraded when its reconstruction stopped short of the durable version key (§5.7). Members that are merely behind or never synced do not cause this, so normal scaling up and down does not trip it. Losing every holder of a binding does. This gives up availability in exchange for prevention. | +| `connectivity` | `self_managed` | Who maintains the distribution mesh (§4.6). With `self_managed`, this registry runs its own mesh. With `provided_externally`, the mesh is off, and the registry relies on a `self_managed` registry that runs on every node this one runs on. Otherwise the node is silently isolated, and a long stretch with no leader logs a warning. The other per-registry backstops stay active. Any other value resolves to `self_managed`. | ## 9. Comparison with `global`, `gproc`, and Elixir's `Registry` -Elixir's `Registry` solves the single-node problem; `global`, `gproc` (global mode) and -`dgen_registry` solve the cluster-wide one in different ways. Each is a reasonable +Elixir's `Registry` solves the single-node problem. `global`, `gproc` in global mode, +and `dgen_registry` each solve the cluster-wide one in a different way. Each is a reasonable trade-off. ### Elixir's `Registry` (standard library, single node) A fast, partitioned, ETS-backed **single-node** registry with unique and duplicate keys, -via-tuple support, values and dispatch. If your naming need is local to a node it is the -natural, dependency-free choice. It does not provide one owner of a name across a +via-tuple support, values and dispatch. If you only need names on one node, it is the +natural choice and needs no dependency. It does not provide one owner of a name across a cluster, which is what `dgen_registry` is for. ### `global` (OTP, built in) -Built into OTP with no dependency: one cluster-wide namespace replicated on every node, -serialised by distributed locks. It is **AP-leaning** — both sides of a partition keep -accepting registrations, and clashes are reconciled on heal by a resolver function. -Choose it for a zero-dependency, always-available registry when you can define the -merge; choose `dgen_registry` to guarantee a single owner through a partition, if you -can run a consistent database. +Built into OTP with no dependency. It keeps one cluster-wide namespace, replicated on +every node and serialised by distributed locks. It leans toward availability. Both +sides of a partition keep accepting registrations, and when the partition heals, a +resolver function settles any clashes. Choose it for a zero-dependency, always-available +registry when you can define how to merge. Choose `dgen_registry` to guarantee a single +owner through a partition, if you can run a consistent database. ### `gproc` (third-party) -A feature-rich, fully in-memory library — names, properties, counters, aggregated -counters, `ets:select`-style queries — with a fast local mode and a global mode that -elects a leader among the Erlang nodes. Its breadth is outside `dgen_registry`'s scope, -and it needs no external database. `dgen_registry` pays for that dependency with strict -consistency and dynamic membership: no node list, stale leaders fenced automatically. +A feature-rich, fully in-memory library. It offers names, properties, counters, +aggregated counters and `ets:select`-style queries. It has a fast local mode and a +global mode that elects a leader among the Erlang nodes. Much of what it offers is +outside the scope of `dgen_registry`, and it needs no external database. In return for +depending on a database, `dgen_registry` gives strict consistency and dynamic +membership. There is no node list, and stale leaders are fenced automatically. ### At a glance @@ -618,10 +664,13 @@ consistency and dynamic membership: no node list, stale leaders fenced automatic ## 10. When `dgen_registry` is (and isn't) a good fit -**Good fit** when you need a strict cluster-wide singleton and prefer a name briefly -unavailable to briefly owned twice; your registered processes can withstand being -killed; you can operate a consistent database such as FoundationDB; and lookups beyond -name → pid are simple AND-equal searches (§4.7). +**Good fit** when all of these hold: + +- You need a strict cluster-wide singleton, and you would rather a name be briefly + unavailable than briefly owned twice. +- Your registered processes can withstand being killed. +- You can operate a consistent database such as FoundationDB. +- Any lookup beyond name → pid is a simple AND of equalities (§4.7). **Reconsider** when writes must stay available on every side of a partition, you cannot run an external database, a registered process must never be killed by the registry, or diff --git a/docs/design/dgen_registry_message_flows.md b/docs/design/dgen_registry_message_flows.md new file mode 100644 index 0000000..7ba5192 --- /dev/null +++ b/docs/design/dgen_registry_message_flows.md @@ -0,0 +1,121 @@ + + +# dgen_registry Message Flows + +Each diagram follows one operation through the processes that take part. Time runs +downward. Message names are the tuples in the code, and `E` and `V` are the epoch +and the commit version. A member's loop never blocks on another member. Calls +between members run in spawned helpers or come from the elector. + +Section numbers like §5.6 refer to the [dgen_registry Design](dgen_registry_design.md). + +![Arrow styles: cast, call, reply, durable, monitor signal, the highlighted hop, and a lost message](assets/registry_flows/legend.svg) + +**This document is LLM generated.** + +## Register through a follower + +A follower forwards the registration to the leader, which commits it, broadcasts the batch, then replies. FIFO puts the batch ahead of the reply, so the follower already holds the binding when it answers yes. + +![Swimlane diagram: Register through a follower](assets/registry_flows/forwarded.svg) + +- The follower's loop never blocks on the leader. Every hop between members is a cast tagged with `Ref`. +- The leader answers forwarded registrations as soon as they commit. The forwarding follower is the second holder, so no replica ack is needed. +- The worker only bumps the version key. Name state lives in members' memory. The leader key read is what fences a deposed leader. + +In the code: `dgen_registry_member`: register call, register_req, start_commit/1, apply_committed_plan/3, names_batch handler, register_reply handler. + +## Register on the leader's node + +A registration made on the leader's own node commits, then waits for register_replicas followers to confirm they hold the batch before the caller gets yes. + +![Swimlane diagram: Register on the leader’s node](assets/registry_flows/direct.svg) + +- `replicate_sync` is sent after the batch on the same channel, so FIFO delivers it only once the follower has seen the batch. +- A follower that refused the batch as a gap stays silent. It never claims to hold bindings it doesn't have. +- With no followers at all, the leader answers `yes` right away. + +In the code: `dgen_registry_member`: await_replica_acks/3, replicate_sync handler, replicate_ack handler, replicate_timeout handler. + +## Unregister and process exit + +An unregister deletes the local row at once and is committed by the leader. When a registered process exits, the leader's tagged monitor turns the exit into the same unregister batch. + +![Swimlane diagram: Unregister and process exit](assets/registry_flows/unregister.svg) + +- Only the leader monitors registered pids. The monitor's custom tag carries the name, so the DOWN says which name to clear. +- A retract only clears the name if the same pid still holds it, so it never removes a later registration. + +In the code: `dgen_registry_member`: route_unregister/3, unregister_req, {down, Name} handler, monitor_name/2. + +## Snapshot and consistent reads + +A snapshot read is a lock-free ETS lookup with no messages. A consistent read goes to the leader, and a helper checks the durable leader key before it replies to the caller directly. + +![Swimlane diagram: Snapshot and consistent reads](assets/registry_flows/reads.svg) + +- The snapshot table is `protected`. The member is its only writer, and callers read it without locking. +- `get_metadata_consistent/1` and `query_consistent/2` take the same path with their own request tags. +- Neither the follower nor the leader waits for the backend. The reply goes straight to the caller's `From`. + +In the code: `dgen_registry`: whereis_name/1; `dgen_registry_member`: forward_read/5, whereis_req, reply_fenced/4. + +## A member joins + +A joining member announces itself through the elector's durable queue. The elector keeps the sticky leader, and the leader onboards the joiner with one snapshot while existing followers only start monitoring it. + +![Swimlane diagram: A member joins](assets/registry_flows/join.svg) + +- Every node runs an elector process over the same durable state. Whichever one consumes the `join` runs the action. +- Because the snapshot comes from the leader process, FIFO puts it ahead of every later `names_batch` the joiner sees. +- The first member to join finds no leader, so the elector elects it with a new epoch and it takes the full handoff path. + +In the code: `dgen_registry_elector`: join transaction; `dgen_registry_elector_actions`: push_leader_to_members/1; `dgen_registry_member`: leader assume call, assume_leadership_fast_path/6, cast_data_to_new_member/4. + +## Leader failure and handoff + +When the leader dies, survivors report it to the elector, which elects a new leader in a new epoch. The new leader gathers every replica, waits until it has caught up to the durable version, and sends each member a snapshot in the new epoch. + +![Swimlane diagram: Leader failure and handoff](assets/registry_flows/failover.svg) + +- F sends its own `member_down` too. The elector drops the second one because L0 is already gone. +- The new leader key fences L0. If L0 is alive but cut off, its next commit aborts. +- Ending at the deadline without reaching the frontier marks the leader `degraded`. A committed binding may be missing, and §5.6 repairs it. +- A forwarded registration that reaches N while it is assuming is postponed until the gather finishes. + +In the code: `dgen_registry_member`: peer DOWN handler, prepare_to_assume_leadership/6, gather_until_caught_up/7, assume_gathered; `dgen_registry_elector`: member_down transaction, elect_leader/5. + +## Gap detection and resync + +Each batch names the version it follows. A follower that sees a gap applies nothing and asks the leader for a snapshot. The heartbeat exposes a lost tail, and a batch from a later epoch always counts as a gap. + +![Swimlane diagram: Gap detection and resync](assets/registry_flows/resync.svg) + +- Applying batches whole and in order keeps every replica a prefix of the leader's stream. Two replicas at the same version hold the same content. +- A batch from an older epoch is dropped as stale, and one at or behind the applied version is a duplicate. +- The leader sends the heartbeat every 5 s. + +In the code: `dgen_registry_replica`: offer_batch/5; `dgen_registry_member`: request_resync/2, resync_req, broadcast_heartbeat/1. + +## A held yes across a leader change + +A follower holding a yes from epoch E drops it when a snapshot moves it into a later epoch, and sends the registration to the new leader again instead of answering from versions the new leader produced. + +![Swimlane diagram: A held yes across a leader change](assets/registry_flows/held-yes.svg) + +- A yes from an epoch F has already left is re-dispatched along the current route, not held. +- A yes from an epoch F hasn't reached yet is held, and F asks for a resync to get there. +- The TLA+ config `NoEpochFence` turns this off and finds the false ack. + +In the code: `dgen_registry_intents`: on_reply/5, on_version/2; `dgen_registry_replica`: offer_batch/5; `dgen_registry_member`: apply_names_snapshot. + +## A deposed leader is fenced + +A leader cut off from the cluster keeps trying to commit. The backend rejects its commit because the leader key names a newer leader, and the old leader hands its work back and rejoins. + +![Swimlane diagram: A deposed leader is fenced](assets/registry_flows/fenced.svg) + +- The commit reads the leader key, or adds a read conflict on it. A newer leader key aborts the commit. +- A forwarded registration in the failed batch is dropped. Its follower re-drives it to the new leader. + +In the code: `dgen_registry_member`: handle_commit_result/3, maybe_rejoin/1; `dgen_registry_names`: verify_leader/4. diff --git a/docs/design/dgen_server_design.md b/docs/design/dgen_server_design.md index 56edaee..617a63d 100644 --- a/docs/design/dgen_server_design.md +++ b/docs/design/dgen_server_design.md @@ -2,11 +2,11 @@ `dgen_server` brings the `gen_server` programming model (`init/1`, `handle_call/3`, `handle_cast/2`, ...) to a distributed system. A `dgen_server` -is not an Erlang process: it is durable state plus a durable message queue in a -strongly-consistent backend (default: FoundationDB), and it exists as long as -that state does, independent of any process, node, or cluster. +is not an Erlang process. It is durable state plus a durable message queue in a +strongly-consistent backend (default: FoundationDB). It exists as long as that +state does, independent of any process, node, or cluster. -The [Guarantees](#6-guarantees) section is normative: behaviour that contradicts +The [Guarantees](#6-guarantees) section is normative. Behaviour that contradicts it is a bug. **This document is LLM generated.** @@ -15,17 +15,17 @@ it is a bug. ## 1. The one-minute model -- Zero or more **consumer** processes, on any node, pull messages off the queue, - run your callback, and commit the new state (and reply, if any) in one - database transaction. +- Zero or more **consumer** processes, on any node, pull messages off the queue + and run your callback. Each one commits the new state, and the reply if + there is one, in a single database transaction. - **Standard** messages (`call`, `cast`) go through the queue and are processed with strict serializability. **Priority** messages (`priority_call`, `priority_cast`, `handle_info`) skip the queue and run immediately, at the cost of ordering. -- Callbacks run inside that transaction and must be pure; side effects go in - post-commit **Actions** (§4.3) or `handle_locked/4` (§4.4). -- If a consumer crashes mid-transaction nothing commits; a queued message is - retried by the next consumer (priority messages are lost, §5). +- Callbacks run inside that transaction and must be pure. Side effects go in + **Actions**, which run after the commit (§4.3), or in `handle_locked/4` (§4.4). +- If a consumer crashes mid-transaction, nothing commits. The next consumer + retries a queued message. Priority messages are lost (§5). --- @@ -101,18 +101,18 @@ GenServer.stop(pid) `erlfdb`. - **tenant**: a database handle plus a directory handle defining a key subspace. Every `dgen_server` is started against one. -- **key-tuple**: a tuple encoded as a key in a tenant; extending it makes the - original a prefix containing other keys. +- **key-tuple**: a tuple encoded as a key in a tenant. Extending it makes the + original a prefix that contains other keys. - **tuid**: the key-tuple identifying a `dgen_server` in its tenant. `init/1` may return `{ok, Tuid, State}` to choose it. - **message-queue**: the durable queue a `dgen_server` processes requests from. - **call-request / cast-request**: a queued operation that does / does not expect a response. - **priority-request**: a call or cast handled immediately, ignoring the queue - and its ordering; for urgent, usually read-only work. -- **dgen_server process / consumer**: any process that can push requests onto - the queue / one that also processes them. A `dgen_server` may have zero, one, - or many consumers. + and its ordering. It is meant for urgent, usually read-only work. +- **dgen_server process**: any process that can push requests onto the queue. +- **consumer**: a dgen_server process that also processes requests. A + `dgen_server` may have zero, one, or many consumers. --- @@ -131,15 +131,17 @@ GenServer.stop(pid) ### 4.2 `consume_k` and inlining `consume_k` (default `1`) is the maximum number of queued messages a consumer -processes per transaction. A consume cycle peeks up to `consume_k` messages, -processes them in order with the state carried in memory, commits the batch in -**one** transaction (state read once, written once), and immediately re-arms. - -A larger `consume_k` amortises per-transaction cost and raises throughput when -callbacks are cheap, at the price of larger transactions and all-or-nothing -batches (a conflict retries the whole batch). A busy consumer keeps draining, -so it also tends to keep a **single** node as the active consumer, which helps -when consumer identity matters (e.g. election-style callbacks). +processes per transaction. A consume cycle peeks up to `consume_k` messages and +processes them in order, carrying the state in memory. It commits the batch in +**one** transaction, so the state is read once and written once. Then it starts +the next cycle right away. + +A larger `consume_k` spreads the cost of each transaction over more messages and +raises throughput when callbacks are cheap. The price is larger transactions and +all-or-nothing batches, because a conflict retries the whole batch. A busy +consumer keeps draining, so it also tends to keep a **single** node as the +active consumer. That helps when it matters which node consumes, as in +election-style callbacks. **Inlining.** When `consume_k =:= 1`, a `call/2` whose queue is empty and unlocked is processed immediately in the caller's own transaction, with the @@ -155,51 +157,57 @@ Callbacks may return `{reply, Reply, State, Actions}` or `{noreply, State, Actions}`, where `Actions` is a list of 1-arity functions that: -- run after the transaction commits; -- receive the committed `State` but cannot modify it; -- are the place for side effects (logging, telemetry, external publishing); -- may return `halt` to stop running the remaining actions. +- run after the transaction commits. +- receive the committed `State` but cannot modify it. +- are the place for side effects such as logging, telemetry, and external + publishing. +- may return `halt` to skip the remaining actions. With `consume_k > 1`, a batch's actions all run after its single commit and each receives the state as of the **end of the batch**. ### 4.4 Locking -A callback may return `{lock, State}` to enter locked mode, for work that must -finish before the next queued message is processed but cannot run inside a -transaction (external API calls, long computations). +A callback may return `{lock, State}` to enter locked mode. Locked mode is for +work that must finish before the next queued message is processed but cannot +run inside a transaction, such as external API calls or long computations. While locked: - Standard `call` and `cast` messages are queued but not processed. - Priority messages and `handle_info` still execute (§4.1). -- `handle_locked/4` runs **outside** a transaction: no transaction limits, side - effects permitted, and the state it returns is written back. +- `handle_locked/4` runs **outside** a transaction. It has no transaction + limits and may perform side effects, and the state it returns is written back. `lock_timeout` (default `infinity`) bounds how long a lock may be held. The holder normally clears the lock when `handle_locked/4` returns. If the holder is killed (SIGKILL, VM abort), another consumer clears the lock once -`lock_timeout` ms have passed since it was set. With `infinity`, a dead holder -blocks all consumers permanently. Set it safely above the worst-case -`handle_locked/4` duration. +`lock_timeout` ms have passed since it was set. We call that busting the lock. +With `infinity`, a dead holder blocks all consumers permanently. + +Busting is a guess that the holder is dead, and a slow holder may still be +running. So the lock is **fenced** (§4.7). If a holder was busted, we don't +commit its result. Its message goes back on the queue, and the holder exits +with `{lock_fenced, requeued}`. A priority or inline lock has no queued +message, so the holder exits with `{lock_fenced, not_requeued}` and the caller +receives the exit. Either way, the side effects of `handle_locked/4` have +already happened, so **`handle_locked/4` must be safe to run twice**. -Clearing a stale lock is a guess that the holder is dead, so the lock is -fenced (§4.7): a busted holder's result is not committed, its message is -requeued, and it exits with `{lock_fenced, requeued}`. (A priority or inline -lock has no queued message: it exits with `{lock_fenced, not_requeued}`, and the -caller receives the exit.) Its side effects have already happened, so -**`handle_locked/4` must be safe to run twice**. +Because of fencing, a wrong guess costs a retry rather than lost data. So set +`lock_timeout` to how quickly you want to recover from a dead holder, rather +than above the worst-case `handle_locked/4` duration. ### 4.5 Persisted state State is encoded to optimise partial updates: -1. **Assigns map**: a map with all atom keys; one key per entry, unordered. +1. **Assigns map**: a map whose keys are all atoms. Each entry gets its own + key, unordered. #{mykey => <<"my value">>, otherkey => 42} -2. **Components list**: a list of maps each with an atom `id` key holding a - binary; one key per item, ordered by fractional indexing. +2. **Components list**: a list of maps, each with an atom `id` key holding a + binary. Each item gets its own key, ordered by fractional indexing. [#{id => "item1", value => 1}, #{id => "item2", value => 2}] @@ -208,10 +216,10 @@ State is encoded to optimise partial updates: 4. **Nesting**: encodings nest recursively, e.g. a components list inside an assigns map. -On update, assigns maps and components lists write only changed entries -(clearing removed ones; reordering updates fractional indices); terms are -rewritten in full. If the encoding type changes, the old keys are cleared and -the new encoding written in full. +On update, assigns maps and components lists write only the entries that +changed. Removed entries are cleared, and reordering updates the fractional +indices. Terms are rewritten in full. If the encoding type changes, we clear the +old keys and write the new encoding in full. ### 4.6 Caching @@ -222,40 +230,42 @@ consumer modifies the state. ### 4.7 Request flows **Call:** the caller enqueues the request with its from-key and watches the -reply-sentinel-key. A consumer runs `handle_call/3`, writes the new state and, -if the sentinel still exists, the reply; then commits. The consumer then runs -the actions while the caller reads and clears the reply. A timed-out caller -clears its reply keys so none leak; as with `gen_server`, the callback still -runs and the state still changes. +reply-sentinel-key. A consumer runs `handle_call/3` and writes the new state. +If the sentinel still exists, it also writes the reply. Then it commits. After +that, the consumer runs the actions while the caller reads and clears the reply. +A caller that times out clears its reply keys so none leak. As with +`gen_server`, the callback still runs and the state still changes. -**Cast:** enqueue; a consumer runs `handle_cast/2`, commits, then runs the -actions. +**Cast:** the caller enqueues the request. A consumer runs `handle_cast/2`, +commits, then runs the actions. **Lock:** 1. The callback returns `{lock, NewState}`. In one transaction the consumer commits `NewState`, dequeues the message, and writes a lock key holding - `{Timestamp, Token}` (`Token` unique to this acquisition). + `{Timestamp, Token}`. `Token` is unique to this acquisition. 2. Other consumers see the lock and wait on a DB watch. 3. The holder calls `handle_locked/4` with the triggering event, message, and committed `NewState`. -4. The holder commits the result in a second transaction, only if the lock key - still carries its `Token`. Otherwise nothing is written, the message is +4. The holder commits the result in a second transaction, but only if the lock + key still carries its `Token`. Otherwise nothing is written, the message is requeued in that transaction, and the holder exits with - `{lock_fenced, requeued}` (`not_requeued` for a priority or inline lock). -5. Whatever `handle_locked/4` returns, even on exception, the lock key is - cleared (only if it still carries this holder's `Token`) and the queue watch - notified, so waiting consumers resume. + `{lock_fenced, requeued}`. A priority or inline lock exits with + `{lock_fenced, not_requeued}` instead. +5. However `handle_locked/4` ends, even with an exception, the holder clears the + lock key if it still carries its `Token`. Clearing notifies the queue watch, + so waiting consumers resume. ### 4.8 Internal terms - **from-key**: `{WaitingKey..., Timestamp, term_to_binary(Ref)}` identifies - one waiting caller under the **waiting-key** prefix; the timestamp lets + one waiting caller under the **waiting-key** prefix. The timestamp lets abandoned keys be garbage-collected. -- **reply-sentinel-key**: chunk 0 of the reply, stored as `{From, <<"t">>, N}` - so replies may exceed the backend value-size limit; the caller watches it. -- **quid**: prefix of all queue keys: **item-keys** (queued requests) and the - **push-key / pop-key** counters. +- **reply-sentinel-key**: chunk 0 of the reply. Replies are stored in chunks as + `{From, <<"t">>, N}` so they may exceed the backend value-size limit. The + caller watches this key. +- **quid**: the prefix of all queue keys. Under it are the **item-keys**, which + hold queued requests, and the **push-key** and **pop-key** counters. --- @@ -269,17 +279,18 @@ operation and **at-least-once** if a consumer crashes before commit (bounded by from scratch. **During `handle_call`, `handle_cast`, `handle_info`:** the transaction aborts -and state is unchanged. A `call`/`cast` stays in the queue and is retried by -the next consumer (subject to `dead_letter_threshold`). A `priority_call`, +and state is unchanged. A `call` or `cast` stays in the queue, and the next +consumer retries it, subject to `dead_letter_threshold`. A `priority_call`, `priority_cast`, or `handle_info` message is lost. **Dead-letter queue.** A *poison message* consistently crashes consumers. With `dead_letter_threshold` set to a positive integer, each failed attempt -increments a counter in the message envelope; at the threshold the message is -moved to the dead-letter queue (DLQ) instead of retried, and: +increments a counter in the message envelope. At the threshold, the message is +moved to the dead-letter queue (DLQ) instead of being retried, and: -- a blocked `call` caller raises `{dead_letter, N}` (`N` = attempt count); -- the optional `handle_dead_letter/2` callback receives the message and count; +- a blocked `call` caller raises `{dead_letter, N}`, where `N` is the attempt + count. +- the optional `handle_dead_letter/2` callback receives the message and count. - a warning is logged. Dead-lettering is **disabled by default** (`infinity`): @@ -342,35 +353,35 @@ entries = :dgen_queue.peek_dlq(tenant, quid) # [{key, envelope, attempts, ts_ms its attempt count reset to zero, for retry after the cause is fixed. **During `handle_locked`:** the state committed with the lock is already -persisted and the triggering message already dequeued. On an exception the lock -is cleared before the process exits and the message is not retried. If the -process dies without clearing it (SIGKILL, OOM, power loss), the lock blocks -the queue until `lock_timeout` expires (forever with `infinity`, requiring -manual intervention). If the holder is fenced, its message is requeued and -retried (§4.4). +persisted, and the triggering message is already dequeued. On an exception, the +lock is cleared before the process exits, and the message is not retried. If +the process dies without clearing the lock (SIGKILL, OOM, power loss), the lock +blocks the queue until `lock_timeout` expires. With `infinity` it blocks +forever and needs manual intervention. If the holder is fenced, its message is +requeued and retried (§4.4). **During actions:** state is already committed and the message consumed. A crashing action stops the remaining actions; nothing is retried. -**Supervisor restart:** if durable state exists, `init/1` is called but its -state is ignored; the process resumes from the persisted state and consumes -(if configured to) alongside any other consumers. +**Supervisor restart:** if durable state exists, `init/1` is called but the +state it returns is ignored. The process resumes from the persisted state and, +if configured to, consumes alongside any other consumers. --- ## 6. Guarantees -These are normative. Behaviour that violates one of these is a defect. +These are normative. Behaviour that violates one of them is a defect. 1. **Durability.** State and queued messages persist in the backend, independent of any Erlang process, node, or cluster. 2. **Delivery.** Standard `call`/`cast` messages are processed exactly-once under normal operation. If a consumer crashes before its transaction - commits, the message is retried by another consumer: at-least-once in that - case, bounded by `dead_letter_threshold` when configured (§5). + commits, another consumer retries the message. In that case delivery is + at-least-once, bounded by `dead_letter_threshold` when configured (§5). 3. **Serializability.** Standard messages are committed with strict - serializability via the durable queue: a batch's state read and write happen - in one transaction. + serializability through the durable queue. A batch reads and writes the + state in one transaction. 4. **Priority messages trade ordering for immediacy.** `priority_call/2`, `priority_cast/2`, and `handle_info/2` always bypass the queue and the lock, so they are not ordered with respect to the queued stream. @@ -380,9 +391,9 @@ These are normative. Behaviour that violates one of these is a defect. holder's `handle_locked/4`, priority messages, and `handle_info/2` execute; other standard `call`/`cast` messages wait until the lock clears. 7. **Callback purity.** `handle_call/3`, `handle_cast/2`, and `handle_info/2` - run inside a database transaction and must not perform side effects; those - belong in `Actions` (§4.3) or `handle_locked/4` (§4.4), which run outside a - transaction. + run inside a database transaction and must not perform side effects. Side + effects belong in `Actions` (§4.3) or `handle_locked/4` (§4.4), which run + outside a transaction. --- @@ -404,11 +415,11 @@ Options in the `Opts` proplist of `start/3,4` and `start_link/3,4`: ## 8. When `dgen_server` is (and isn't) a good fit **Good fit when** you want the `gen_server` model but need state and in-flight -messages to survive process, node, or cluster death; a database round-trip per -message is acceptable; and side effects can be deferred to `Actions` or done in -`handle_locked/4`. - -**Reconsider when** you need purely in-memory latency (a plain `gen_server` is -faster for state that need not survive a restart); your callbacks need -non-deferrable side effects inline; or you cannot or do not want to operate a -strongly-consistent external database. +messages to survive the death of a process, node, or cluster. A database +round-trip per message must be acceptable, and side effects must be able to +wait for `Actions` or run in `handle_locked/4`. + +**Reconsider when** you need in-memory latency. A plain `gen_server` is faster +for state that need not survive a restart. Also reconsider if your callbacks +need side effects inline that can't be deferred, or if you can't or don't want +to operate a strongly-consistent external database. diff --git a/docs/tools/registry_flows.py b/docs/tools/registry_flows.py new file mode 100644 index 0000000..50c5f67 --- /dev/null +++ b/docs/tools/registry_flows.py @@ -0,0 +1,641 @@ +"""Generates the swimlane diagrams in docs/design/dgen_registry_message_flows.md. + +Run it from anywhere with `python3 docs/tools/registry_flows.py`. It writes one SVG +per flow to docs/design/assets/registry_flows/ and regenerates the Markdown page. +The flows below are written by hand from the code, so update them when a message +changes. + +*This script is LLM-generated. See the AI disclosure in README.md.* +""" +import html +import math +import os +import textwrap + +HERE = os.path.dirname(os.path.abspath(__file__)) +MEMBER = "src/dgen_registry/dgen_registry_member.erl" +ELECTOR = "src/dgen_registry/dgen_registry_elector.erl" +ACTIONS = "src/dgen_registry/dgen_registry_elector_actions.erl" +REPLICA = "src/dgen_registry/dgen_registry_replica.erl" +INTENTS = "src/dgen_registry/dgen_registry_intents.erl" +NAMES = "src/dgen_registry/dgen_registry_names.erl" +API = "src/dgen_registry.erl" + +GUTTER = 40 +HEAD_H = 46 +TOP = 72 +MONO_W = 6.55 # px per char at 11px mono +SANS_W = 6.0 # px per char at 11.5px sans, conservative +LINE = 15 + +LANES = { + "C": ("Caller", "any process"), + "F": ("Follower", "member"), + "F2": ("Other followers", "members"), + "L": ("Leader", "member"), + "W": ("Commit worker", "dgen_transaction"), + "DB": ("Backend", "FDB keys"), + "E": ("Elector", "dgen_server"), + "H": ("Helper", "spawned process"), + "P": ("Registered pid", "monitored"), + "J": ("Joining member", "new node"), + "L0": ("Old leader", "member"), + "N": ("New leader", "member"), + "T": ("Names table", "local ETS"), +} + + +def esc(s): + return html.escape(s, quote=True) + + +class Diagram: + def __init__(self, lanes, spacing): + self.lanes = lanes + self.S = spacing + self.W = GUTTER + len(lanes) * spacing + 8 + self.parts = [] + self.y = TOP + self.step = 0 + + def x(self, key): + return GUTTER + self.S / 2 + self.lanes.index(key) * self.S + + def clamp(self, cx, w): + return max(w / 2 + 4, min(self.W - w / 2 - 4, cx)) + + def number(self, y): + self.step += 1 + self.parts.append( + f'' + f'{self.step}' + ) + + def head(self, x2, y, cls, left): + d = -1 if left else 1 + pts = f"{x2:.1f},{y:.1f} {x2 - d * 8:.1f},{y - 4.5:.1f} {x2 - d * 8:.1f},{y + 4.5:.1f}" + self.parts.append(f'') + + def msg(self, frm, to, label, kind="cast", opts=()): + targets = to if isinstance(to, list) else [to] + y = self.y + 22 + x1 = self.x(frm) + cls = f"m-{kind}" + (" accent" if "accent" in opts else "") + xs = [self.x(t) for t in targets] + far = max(xs, key=lambda v: abs(v - x1)) + near = min(xs, key=lambda v: abs(v - x1)) + lost = "lost" in opts + if lost: + x_end = x1 + (far - x1) * 0.62 + segs = [(x1, x_end)] + else: + # One segment per side, so a broadcast can fan out both ways. + segs = [(x1, min(xs) + 8)] if min(xs) < x1 else [] + if max(xs) > x1: + segs.append((x1, max(xs) - 8)) + for a, b in segs: + self.parts.append( + f'' + ) + if kind == "call": + self.parts.append(f'') + if lost: + cx = x_end + self.parts.append( + f'' + ) + else: + for xt in xs: + self.head(xt, y, cls, xt < x1) + w = len(label) * MONO_W + span_mid = (x1 + (near if not lost else x_end)) / 2 + cx = self.clamp(span_mid, w) + self.parts.append( + f'{esc(label)}' + ) + self.number(y) + self.y += 34 + + def act(self, lane, text, cls=""): + maxw = self.S * 1.75 + chars = max(14, int((maxw - 18) / SANS_W)) + lines = textwrap.wrap(text, chars) + w = min(maxw, max(len(l) for l in lines) * SANS_W + 20) + h = len(lines) * LINE + 10 + cx = self.clamp(self.x(lane), w) + y0 = self.y + 6 + self.parts.append( + f'' + ) + for i, l in enumerate(lines): + self.parts.append( + f'{esc(l)}' + ) + self.y = y0 + h + 4 + + def note(self, a, b, text): + xa, xb = sorted([self.x(a), self.x(b)]) + x0 = max(GUTTER - 4, xa - self.S / 2 + 10) + x1 = min(self.W - 4, xb + self.S / 2 - 10) + w = x1 - x0 + chars = max(20, int((w - 24) / SANS_W)) + lines = textwrap.wrap(text, chars) + h = len(lines) * LINE + 12 + y0 = self.y + 8 + self.parts.append( + f'' + ) + for i, l in enumerate(lines): + self.parts.append( + f'{esc(l)}' + ) + self.y = y0 + h + 6 + + def phase(self, text): + y = self.y + 20 + self.parts.append( + f'' + ) + w = len(text) * 6.9 + 16 + self.parts.append( + f'' + f'{esc(text.upper())}' + ) + self.y = y + 14 + + def render(self, aria): + H = self.y + 18 + out = [ + f'' + ] + for k in self.lanes: + x = self.x(k) + name, sub = LANES[k] + out.append( + f'' + ) + out.append( + f'' + f'{esc(name)}' + f'{esc(sub)}' + ) + out.extend(self.parts) + out.append("") + return "\n".join(out) + + +def build(spec): + d = Diagram(spec["lanes"], spec.get("S", 172)) + for s in spec["steps"]: + k = s[0] + if k == "m": + d.msg(s[1], s[2], s[3], s[4] if len(s) > 4 else "cast", s[5] if len(s) > 5 else ()) + elif k == "a": + d.act(s[1], s[2], s[3] if len(s) > 3 else "") + elif k == "n": + d.note(s[1], s[2], s[3]) + elif k == "p": + d.phase(s[1]) + return d.render(spec["claim"]) + + +FLOWS = [ + { + "id": "forwarded", + "title": "Register through a follower", + "claim": "A follower forwards the registration to the leader, which commits it, broadcasts the batch, then replies. FIFO puts the batch ahead of the reply, so the follower already holds the binding when it answers yes.", + "lanes": ["C", "F", "L", "W", "DB", "F2"], + "steps": [ + ("m", "C", "F", "{register, Name, Pid, Meta}", "call"), + ("a", "F", "Stores an intent under Ref. It can't decide alone."), + ("m", "F", "L", "{register_req, Ref, F, Name, Pid, Meta}"), + ("a", "L", "Enqueues {add, …, {forward, F, Ref}}. Ops that arrive during a commit join the next batch."), + ("m", "L", "W", "start_commit(epoch E)"), + ("m", "W", "DB", "fence on leader key {L, E}, bump version", "db"), + ("m", "DB", "W", "commit version V", "reply"), + ("m", "W", "L", "{dgen_transaction, Ref, {committed, V}}"), + ("a", "L", "Applies the batch to its replica. Monitors Pid with tag {down, Name}."), + ("m", "L", ["F", "F2"], "{names_batch, Ops, E, PrevV, V, L}", "cast", ("accent",)), + ("a", "F", "PrevV matches its version, so the whole batch applies. Replica is at (E, V)."), + ("m", "L", "F", "{register_reply, Ref, yes, V, E}", "cast", ("accent",)), + ("a", "F", "Replica holds V in epoch E, so the yes is released."), + ("m", "F", "C", "yes", "reply"), + ("n", "F", "L", "If the batch was refused as a gap, F holds the yes as {awaiting_version, V, E, L} and asks L for a resync. A yes from another epoch never counts."), + ], + "notes": [ + "The follower's loop never blocks on the leader. Every hop between members is a cast tagged with Ref.", + "The leader answers forwarded registrations as soon as they commit. The forwarding follower is the second holder, so no replica ack is needed.", + "The worker only bumps the version key. Name state lives in members' memory. The leader key read is what fences a deposed leader.", + ], + "code": [ + ("register call", MEMBER, 233), ("register_req", MEMBER, 571), ("start_commit/1", MEMBER, 1440), + ("apply_committed_plan/3", MEMBER, 1480), ("names_batch handler", MEMBER, 302), + ("register_reply handler", MEMBER, 270), + ], + }, + { + "id": "direct", + "title": "Register on the leader's node", + "claim": "A registration made on the leader's own node commits, then waits for register_replicas followers to confirm they hold the batch before the caller gets yes.", + "lanes": ["C", "L", "W", "F"], + "S": 196, + "steps": [ + ("m", "C", "L", "{register, Name, Pid, Meta}", "call"), + ("a", "L", "Enqueues {add, …, {local, From}}"), + ("m", "L", "W", "commit batch, fenced on {L, E}"), + ("m", "W", "L", "{dgen_transaction, Ref, {committed, V}}"), + ("m", "L", "F", "{names_batch, Ops, E, PrevV, V, L}"), + ("m", "L", "F", "{replicate_sync, BatchRef, L, V}", "cast", ("accent",)), + ("a", "L", "Starts the replicate_timeout timer"), + ("a", "F", "Acks only if its applied version is at least V"), + ("m", "F", "L", "{replicate_ack, BatchRef, F}", "cast", ("accent",)), + ("a", "L", "Enough distinct followers acked. Cancels the timer."), + ("m", "L", "C", "yes", "reply"), + ("p", "If too few acks arrive in time"), + ("a", "L", "{replicate_timeout, BatchRef}", "warn"), + ("m", "L", "C", "yes (degrade-open), or no plus a retract under strict_replication", "reply"), + ], + "notes": [ + "replicate_sync is sent after the batch on the same channel, so FIFO delivers it only once the follower has seen the batch.", + "A follower that refused the batch as a gap stays silent. It never claims to hold bindings it doesn't have.", + "With no followers at all, the leader answers yes right away.", + ], + "code": [ + ("await_replica_acks/3", MEMBER, 1637), ("replicate_sync handler", MEMBER, 280), + ("replicate_ack handler", MEMBER, 289), ("replicate_timeout handler", MEMBER, 373), + ], + }, + { + "id": "unregister", + "title": "Unregister and process exit", + "claim": "An unregister deletes the local row at once and is committed by the leader. When a registered process exits, the leader's tagged monitor turns the exit into the same unregister batch.", + "lanes": ["C", "F", "L", "W", "F2", "P"], + "S": 166, + "steps": [ + ("p", "Explicit unregister through a follower"), + ("m", "C", "F", "{unregister, Name}", "call"), + ("a", "F", "Deletes its own row now and notes the released pid"), + ("m", "F", "L", "{unregister_req, Ref, F, Name}"), + ("a", "L", "Deletes its row. Enqueues {remove, Name, Pid}."), + ("m", "L", "W", "commit batch"), + ("m", "W", "L", "{committed, V}"), + ("m", "L", ["F", "F2"], "{names_batch, [{name_unregistered, Name, Pid}], …}"), + ("m", "L", "F", "{unregister_reply, Ref}", "cast", ("accent",)), + ("m", "F", "C", "ok", "reply"), + ("n", "C", "L", "With no reachable leader, F answers ok right away and keeps the unregister as an intent. It sends {retract_req, Name, Pid} once a leader is reachable."), + ("p", "The registered process exits"), + ("m", "P", "L", "{{down, Name}, MRef, process, Pid, Reason}", "signal", ("accent",)), + ("a", "L", "Enqueues {down, Name, MRef}. Skipped if Name was rebound under a new monitor."), + ("m", "L", "W", "commit batch"), + ("m", "W", "L", "{committed, V}"), + ("m", "L", ["F", "F2"], "{names_batch, [{name_unregistered, Name, Pid}], …}"), + ], + "notes": [ + "Only the leader monitors registered pids. The monitor's custom tag carries the name, so the DOWN says which name to clear.", + "A retract only clears the name if the same pid still holds it, so it never removes a later registration.", + ], + "code": [ + ("route_unregister/3", MEMBER, 1315), ("unregister_req", MEMBER, 581), + ("{down, Name} handler", MEMBER, 403), ("monitor_name/2", MEMBER, 2217), + ], + }, + { + "id": "reads", + "title": "Snapshot and consistent reads", + "claim": "A snapshot read is a lock-free ETS lookup with no messages. A consistent read goes to the leader, and a helper checks the durable leader key before it replies to the caller directly.", + "lanes": ["C", "T", "F", "L", "H", "DB"], + "S": 162, + "steps": [ + ("p", "whereis_name/1"), + ("m", "C", "T", "ets:lookup(Table, Name)", "call"), + ("m", "T", "C", "Pid or undefined", "reply"), + ("p", "whereis_name_consistent/1 on a follower's node"), + ("m", "C", "F", "{whereis, Name}", "call"), + ("m", "F", "L", "{whereis_req, Name, From}"), + ("a", "L", "Reads the value from its replica now"), + ("m", "L", "H", "spawn verify_leader(L, E)"), + ("m", "H", "DB", "read leader key", "db"), + ("m", "DB", "H", "{L, E}", "reply"), + ("m", "H", "C", "gen_server:reply(From, Pid)", "reply", ("accent",)), + ("n", "C", "F", "If the leader is unreachable, F answers undefined itself. If the key shows another leader, the helper answers undefined."), + ], + "notes": [ + "The snapshot table is protected. The member is its only writer, and callers read it without locking.", + "get_metadata_consistent/1 and query_consistent/2 take the same path with their own request tags.", + "Neither the follower nor the leader waits for the backend. The reply goes straight to the caller's From.", + ], + "code": [ + ("whereis_name/1", API, 342), ("forward_read/5", MEMBER, 848), + ("whereis_req", MEMBER, 602), ("reply_fenced/4", MEMBER, 2037), + ], + }, + { + "id": "join", + "title": "A member joins", + "claim": "A joining member announces itself through the elector's durable queue. The elector keeps the sticky leader, and the leader onboards the joiner with one snapshot while existing followers only start monitoring it.", + "lanes": ["J", "E", "DB", "L", "F"], + "S": 186, + "steps": [ + ("m", "J", "E", "{join, J, Token, Fresh = true}", "db"), + ("a", "E", "In one transaction, adds J and keeps leader L. The epoch stays E."), + ("m", "E", "DB", "commit state and leader key {L, E}", "db"), + ("a", "E", "Runs the post-commit action"), + ("m", "E", "L", "{elector_assume_and_distribute, J, …, E}", "call"), + ("a", "L", "Same epoch and synced, so it takes the fast path. Monitors J."), + ("m", "L", "E", "ok", "reply"), + ("m", "L", "J", "{apply_names_snapshot, Records, L, AllIds, Tokens, E, V}", "cast", ("accent",)), + ("a", "J", "Adopts the replica at (E, V) and becomes a follower"), + ("m", "L", "F", "{peer_joined, J, Tokens}"), + ("a", "F", "Monitors J. Its replica is untouched."), + ("p", "J synced before, as after a partition"), + ("a", "L", "A helper fetches J's replica and probes conflicting pids, then sends {joiner_gathered, …}"), + ("a", "L", "Repairs conflicts (§5.6), then sends the snapshot as above"), + ], + "notes": [ + "Every node runs an elector process over the same durable state. Whichever one consumes the join runs the action.", + "Because the snapshot comes from the leader process, FIFO puts it ahead of every later names_batch the joiner sees.", + "The first member to join finds no leader, so the elector elects it with a new epoch and it takes the full handoff path.", + ], + "code": [ + ("join transaction", ELECTOR, 228), ("push_leader_to_members/1", ACTIONS, 26), + ("leader assume call", MEMBER, 540), ("assume_leadership_fast_path/6", MEMBER, 869), + ("cast_data_to_new_member/4", MEMBER, 1016), + ], + }, + { + "id": "failover", + "title": "Leader failure and handoff", + "claim": "When the leader dies, survivors report it to the elector, which elects a new leader in a new epoch. The new leader gathers every replica, waits until it has caught up to the durable version, and sends each member a snapshot in the new epoch.", + "lanes": ["L0", "N", "H", "F", "E", "DB"], + "S": 166, + "steps": [ + ("a", "L0", "Process or node dies", "warn"), + ("m", "L0", ["N", "F"], "'DOWN' on the peer monitor", "signal"), + ("a", "N", "Sets leader_down. A helper reads L0's current token."), + ("m", "N", "E", "{member_down, L0, Token}", "db"), + ("a", "E", "Removes L0 and elects N. Epoch becomes E+1."), + ("m", "E", "DB", "commit leader key {N, E+1}", "db", ("accent",)), + ("m", "E", "N", "{elector_assume_and_distribute, undefined, …, E+1}", "call"), + ("a", "N", "Moves to assuming. Captures its own records and version."), + ("m", "N", "E", "ok", "reply"), + ("m", "N", "H", "spawn the gather"), + ("m", "H", "F", "get_names_snapshot", "call"), + ("m", "F", "H", "{Records, V, Released, LeaderGone}", "reply"), + ("m", "H", "DB", "read committed frontier", "db"), + ("a", "H", "Gathers again every 25 ms until MaxVersion reaches the frontier, every peer reports LeaderGone, or 2 s pass"), + ("m", "H", "E", "get_subscriptions", "call"), + ("m", "H", "N", "{assume_gathered, …, Freshest, {MaxVersion, Frontier}, …}"), + ("a", "N", "Repairs conflicts. Adopts the replica at (E+1, MaxVersion). Becomes leader and starts the monitor sweep."), + ("m", "N", "F", "{apply_names_snapshot, Records, N, …, E+1, MaxVersion}", "cast", ("accent",)), + ("a", "F", "Follows N. Releases or re-drives its held replies."), + ], + "notes": [ + "F sends its own member_down too. The elector drops the second one because L0 is already gone.", + "The new leader key fences L0. If L0 is alive but cut off, its next commit aborts.", + "Ending at the deadline without reaching the frontier marks the leader degraded. A committed binding may be missing, and §5.6 repairs it.", + "A forwarded registration that reaches N while it is assuming is postponed until the gather finishes.", + ], + "code": [ + ("peer DOWN handler", MEMBER, 409), ("member_down transaction", ELECTOR, 237), + ("elect_leader/5", ELECTOR, 323), ("prepare_to_assume_leadership/6", MEMBER, 887), + ("gather_until_caught_up/7", MEMBER, 1103), ("assume_gathered", MEMBER, 720), + ], + }, + { + "id": "resync", + "title": "Gap detection and resync", + "claim": "Each batch names the version it follows. A follower that sees a gap applies nothing and asks the leader for a snapshot. The heartbeat exposes a lost tail, and a batch from a later epoch always counts as a gap.", + "lanes": ["L", "F", "N"], + "S": 250, + "steps": [ + ("m", "L", "F", "{names_batch, Ops1, E, 10, 20, L}"), + ("m", "L", "F", "{names_batch, Ops2, E, 20, 30, L}", "cast", ("lost",)), + ("m", "L", "F", "{names_batch, Ops3, E, 30, 40, L}"), + ("a", "F", "PrevV 30 doesn't match applied 20. A gap, so it applies nothing.", "warn"), + ("m", "F", "L", "{resync_req, F}", "cast", ("accent",)), + ("a", "F", "Won't ask again for 2 s"), + ("m", "L", "F", "{apply_names_snapshot, Records, L, [], #{}, E, 40}", "cast", ("accent",)), + ("a", "F", "Re-baselined at (E, 40)"), + ("p", "A lost tail with no later writes"), + ("m", "L", "F", "heartbeat {names_batch, [], E, 50, 50, L}"), + ("a", "F", "The heartbeat is ahead of its version, so it resyncs as above"), + ("p", "A batch from a later epoch"), + ("m", "N", "F", "{names_batch, Ops, E+1, PrevV, V, N}"), + ("a", "F", "A gap even if PrevV matches. Only a snapshot moves a replica into a new epoch.", "warn"), + ], + "notes": [ + "Applying batches whole and in order keeps every replica a prefix of the leader's stream. Two replicas at the same version hold the same content.", + "A batch from an older epoch is dropped as stale, and one at or behind the applied version is a duplicate.", + "The leader sends the heartbeat every 5 s.", + ], + "code": [ + ("offer_batch/5", REPLICA, 123), ("request_resync/2", MEMBER, 1996), + ("resync_req", MEMBER, 606), ("broadcast_heartbeat/1", MEMBER, 1974), + ], + }, + { + "id": "held-yes", + "title": "A held yes across a leader change", + "claim": "A follower holding a yes from epoch E drops it when a snapshot moves it into a later epoch, and sends the registration to the new leader again instead of answering from versions the new leader produced.", + "lanes": ["C", "F", "L0", "N"], + "S": 216, + "steps": [ + ("m", "C", "F", "{register, Name, Pid, Meta}", "call"), + ("m", "F", "L0", "{register_req, Ref, F, Name, …}"), + ("a", "L0", "Commits at V in epoch E"), + ("m", "L0", "F", "{names_batch, …, E, PrevV, V, L0}", "cast", ("lost",)), + ("m", "L0", "F", "{register_reply, Ref, yes, V, E}"), + ("a", "F", "Replica isn't at V in E. Holds {awaiting_version, V, E, L0} and asks for a resync."), + ("a", "L0", "Dies before answering", "warn"), + ("n", "L0", "N", "N takes over degraded. Its rebuild lacks the commit at V."), + ("m", "N", "F", "{apply_names_snapshot, …, E+1, MaxVersion}"), + ("a", "F", "Its epoch moved past E, so the held yes no longer counts. The intent goes back to the leader."), + ("m", "F", "N", "{register_req, Ref2, F, Name, …}", "cast", ("accent",)), + ("a", "N", "Commits at V2 in epoch E+1, or refuses"), + ("m", "N", "F", "{names_batch, …, E+1, PrevV, V2, N}"), + ("m", "N", "F", "{register_reply, Ref2, Result, V2, E+1}"), + ("m", "F", "C", "Result", "reply"), + ("n", "C", "N", "Before this was fixed, F applied N's later-epoch batches and released the old yes once their versions passed V. The caller heard yes for a binding no live member held."), + ], + "notes": [ + "A yes from an epoch F has already left is re-dispatched along the current route, not held.", + "A yes from an epoch F hasn't reached yet is held, and F asks for a resync to get there.", + "The TLA+ config NoEpochFence turns this off and finds the false ack.", + ], + "code": [ + ("on_reply/5", INTENTS, 121), ("on_version/2", INTENTS, 170), + ("offer_batch/5", REPLICA, 123), ("apply_names_snapshot", MEMBER, 318), + ], + }, + { + "id": "fenced", + "title": "A deposed leader is fenced", + "claim": "A leader cut off from the cluster keeps trying to commit. The backend rejects its commit because the leader key names a newer leader, and the old leader hands its work back and rejoins.", + "lanes": ["C", "L0", "W", "DB", "E"], + "S": 190, + "steps": [ + ("n", "L0", "E", "L0 is partitioned. The elector has already moved leadership to N at epoch E+1."), + ("m", "C", "L0", "{register, Name, Pid, Meta}", "call"), + ("m", "L0", "W", "commit batch, fenced on {L0, E}"), + ("m", "W", "DB", "check leader key", "db"), + ("m", "DB", "W", "{N, E+1}: conflict", "reply", ("accent",)), + ("m", "W", "L0", "{dgen_transaction, Ref, {aborted, fenced}}", "cast", ("accent",)), + ("a", "L0", "Retries the ops after 1 s. Once deposed, a local register becomes an intent."), + ("m", "L0", "E", "{join, L0, Token, false}", "db"), + ("a", "L0", "Rejoins at most once per epoch"), + ("n", "L0", "E", "N's snapshot later moves L0 to follower, and its intents go to N."), + ], + "notes": [ + "The commit reads the leader key, or adds a read conflict on it. A newer leader key aborts the commit.", + "A forwarded registration in the failed batch is dropped. Its follower re-drives it to the new leader.", + ], + "code": [ + ("handle_commit_result/3", MEMBER, 2053), ("maybe_rejoin/1", MEMBER, 2077), + ("verify_leader/4", NAMES, 172), + ], + }, +] + + +def legend_svg(): + items = [ + ("cast", "cast (async)", "m-cast", False), + ("call", "call (waits for a reply)", "m-call", True), + ("reply", "reply", "m-reply", False), + ("db", "durable: backend or elector queue", "m-db", False), + ("signal", "monitor signal", "m-signal", False), + ] + parts = [] + y = 14 + x = 4 + for _k, text, cls, dot in items: + parts.append(f'') + parts.append(f'') + if dot: + parts.append(f'') + parts.append(f'{esc(text)}') + y += 24 + parts.append(f'') + parts.append(f'') + parts.append(f'the hop the flow is about') + y += 24 + parts.append(f'') + parts.append(f'') + parts.append(f'message lost or delayed') + h = y + 14 + return (f'' + "".join(parts) + "") + + +# The SVGs are shown through , so each carries its own styles. System font +# stacks only, because an image can't load web fonts. The dark palette follows the +# reader's OS setting, and the opaque background keeps it legible either way. +SVG_CSS = """ +:root{--surface:#ffffff;--fg:#1a2226;--muted:#5a676d;--rule:#c9d2d5;--lane:#e6ecee; +--accent:#0a6a86;--durable:#8c5a06;--warn:#b0281f;--note:#f8f4e6;--note-rule:#c9b98a} +@media (prefers-color-scheme: dark){:root{--surface:#172024;--fg:#e1e8ea;--muted:#97a5ab; +--rule:#33424a;--lane:#1e2a30;--accent:#56c2df;--durable:#e2ab52;--warn:#ff8074; +--note:#23241d;--note-rule:#6d6440}} +.bg{fill:var(--surface)} +.leg-t{font-family:system-ui,-apple-system,"Segoe UI",Roboto,sans-serif;font-size:12px;fill:var(--fg)} +.life{stroke:var(--rule);stroke-width:1;stroke-dasharray:3 4} +.lane-head{fill:var(--lane);stroke:var(--rule)} +.lane-n{font-family:system-ui,-apple-system,"Segoe UI",Roboto,sans-serif;font-weight:600;font-size:13.5px;fill:var(--fg)} +.lane-s,.lbl,.phase-t,.num{font-family:ui-monospace,"SF Mono",Menlo,Consolas,"DejaVu Sans Mono",monospace} +.lane-s{font-size:10.5px;fill:var(--muted)} +.ln{stroke-width:1.4;fill:none} +.lbl{font-size:11px;paint-order:stroke;stroke-width:4px;stroke-linejoin:round} +.m-cast,.m-call{stroke:var(--fg);fill:var(--fg)} +.m-reply{stroke:var(--muted);fill:var(--muted)} +.ln.m-reply{stroke-dasharray:5 4} +.m-db{stroke:var(--durable);fill:var(--durable)} +.ln.m-db{stroke-dasharray:1.5 3;stroke-width:1.8;stroke-linecap:round} +.m-signal{stroke:var(--warn);fill:var(--warn)} +.accent{stroke:var(--accent);fill:var(--accent)} +.ln.accent{stroke-width:2.2} +.lbl{stroke:var(--surface)} +.lbl.accent{font-weight:600} +.head,.dot{stroke:none} +.lost{stroke:var(--warn);stroke-width:2;fill:none;stroke-linecap:round} +.act{fill:var(--surface);stroke:var(--rule)} +.act.warn{stroke:var(--warn)} +.act-t,.note-t{font-family:system-ui,-apple-system,"Segoe UI",Roboto,sans-serif;font-size:11.5px;fill:var(--fg)} +.act-t.warn{fill:var(--warn)} +.note{fill:var(--note);stroke:var(--note-rule);stroke-dasharray:4 3} +.note-t{font-style:italic} +.phase-ln{stroke:var(--rule);stroke-width:1} +.phase-bg{fill:var(--surface)} +.phase-t{font-size:10.5px;letter-spacing:.08em;fill:var(--muted)} +.num-dot{fill:none;stroke:var(--rule)} +.num{font-size:10px;fill:var(--muted)} +""" + +ROOT = os.path.abspath(os.path.join(HERE, "..", "..")) +ASSETS = os.path.join(ROOT, "docs", "design", "assets", "registry_flows") +DOC = os.path.join(ROOT, "docs", "design", "dgen_registry_message_flows.md") + + +def standalone(svg): + """Adds the styles and a background to an inline SVG so it works as a file.""" + head, rest = svg.split(">", 1) + return f'{head}>{rest}\n' + + +def code_refs(refs): + by_mod = {} + for lbl, path, _line in refs: + mod = os.path.splitext(os.path.basename(path))[0] + by_mod.setdefault(mod, []).append(lbl) + return "; ".join(f"`{m}`: " + ", ".join(ls) for m, ls in by_mod.items()) + + +def md_inline(text): + """Turns the notes' tags into Markdown backticks.""" + return text.replace("", "`").replace("", "`") + + +def markdown(): + out = [ + "", + "", + "# dgen_registry Message Flows", + "", + "Each diagram follows one operation through the processes that take part. Time runs", + "downward. Message names are the tuples in the code, and `E` and `V` are the epoch", + "and the commit version. A member's loop never blocks on another member. Calls", + "between members run in spawned helpers or come from the elector.", + "", + "Section numbers like §5.6 refer to the [dgen_registry Design](dgen_registry_design.md).", + "", + "![Arrow styles: cast, call, reply, durable, monitor signal, the highlighted hop, and a lost message](assets/registry_flows/legend.svg)", + "", + "**This document is LLM generated.**", + "", + ] + for f in FLOWS: + out += [ + f"## {f['title']}", + "", + f["claim"], + "", + f"![Swimlane diagram: {f['title'].replace(chr(39), chr(8217))}](assets/registry_flows/{f['id']}.svg)", + "", + ] + out += [f"- {md_inline(n)}" for n in f["notes"]] + out += ["", f"In the code: {code_refs(f['code'])}.", ""] + return "\n".join(out) + + +if __name__ == "__main__": + os.makedirs(ASSETS, exist_ok=True) + for f in FLOWS: + with open(os.path.join(ASSETS, f["id"] + ".svg"), "w") as fh: + fh.write(standalone(build(f))) + with open(os.path.join(ASSETS, "legend.svg"), "w") as fh: + fh.write(standalone(legend_svg())) + with open(DOC, "w") as fh: + fh.write(markdown()) + print("wrote", DOC) diff --git a/formal/DgenRegistryReplication.cfg b/formal/DgenRegistryReplication.cfg index edd80b1..9aae7d4 100644 --- a/formal/DgenRegistryReplication.cfg +++ b/formal/DgenRegistryReplication.cfg @@ -21,6 +21,8 @@ CONSTANTS SafeAssumeGather = TRUE SafeAssumeSnap = TRUE DegradedAssume = FALSE + LeaderGoneAssume = FALSE + EpochFencedAck = TRUE INVARIANTS TypeOK PrefixConsistency diff --git a/formal/DgenRegistryReplication.tla b/formal/DgenRegistryReplication.tla index c16bd96..aa9c548 100644 --- a/formal/DgenRegistryReplication.tla +++ b/formal/DgenRegistryReplication.tla @@ -35,6 +35,11 @@ CONSTANTS \* pre-fix epoch-only guard (a stale snapshot can rewind) DegradedAssume, \* TRUE = the gather may also give up waiting for the durable \* version key (the code's deadline) and assume degraded + LeaderGoneAssume, \* TRUE = the gather also stops waiting once every live + \* member has handled its old leader's DOWN + EpochFencedAck, \* TRUE = a held forwarded ack is released only in the + \* epoch that committed it, and a broadcast from a later + \* epoch is a gap; FALSE = the pre-fix false ack (mutation) NoPid, NoMember \* model values ASSUME NoPid \notin Pids @@ -58,7 +63,8 @@ VARIABLES epoch, \* [Members -> Nat]: last epoch learned (set by snapshots/assume) leaderView, \* [Members -> Members \cup {NoMember}]: believed leader; \* leaderView[m] = m <=> m believes it is the leader - deferred, \* [Members -> SUBSET (Names \X Pids \X Nat)]: intents awaiting_version + deferred, \* [Members -> SUBSET (Names \X Pids \X Nat \X Nat)]: intents + \* awaiting_version, with the epoch that committed them pendingAcks, \* [Members -> SUBSET (Names \X Pids \X Nat)]: leader-side \* direct registrations awaiting a replicate_ack resyncReq, \* [Members -> Members \cup {NoMember}]: outstanding resync @@ -68,12 +74,17 @@ VARIABLES \* "gathered max version < durable version key" (§5.6) complete, \* [Members -> BOOLEAN]: the reconstruction it adopted was the \* true committed map, histMap[dbVersion] + \* ---- per-member: the code's `leader_down` ---- + leaderDown, \* [Members -> BOOLEAN]: m has handled the DOWN for leaderView[m]. + \* Reset whenever leaderView[m] changes, as the code resets it on + \* every write of `leader`. \* ---- the network ---- chan \* [Members -> [Members -> Seq(Msg)]]: per-ordered-pair FIFO vars == <> + deferred, pendingAcks, resyncReq, chan, degraded, complete, + leaderDown>> (***************************************************************************) (* Basic definitions *) @@ -84,22 +95,31 @@ Live == {m \in Members : alive[m]} PidOpt == Pids \cup {NoPid} Versions == 0..MaxVersion Regs == Names \X Pids \X Versions +Held == Names \X Pids \X Versions \X (0..MaxEpoch) + +\* The registration a held ack answers, without its epoch. +Reg(d) == <> MaxOf(S) == CHOOSE x \in S : \A y \in S : x >= y \* Message shapes. bcast/reply mirror the stamped {name_registered} -\* broadcast and {register_reply, Ref, yes, Version} cast; sync/ack mirror +\* broadcast and {register_reply, Ref, yes, Version, Epoch} cast; sync/ack mirror \* {replicate_sync}/{replicate_ack}; snap mirrors {apply_names_snapshot}. Msg == [type : {"bcast"}, name : Names, pid : Pids, epoch : 0..MaxEpoch, prev : Versions, ver : Versions] \cup [type : {"hb"}, epoch : 0..MaxEpoch, ver : Versions] - \cup [type : {"reply"}, name : Names, pid : Pids, ver : Versions] + \cup [type : {"reply"}, name : Names, pid : Pids, ver : Versions, + epoch : 0..MaxEpoch] \cup [type : {"sync"}, ver : Versions] \cup [type : {"ack"}, ver : Versions] \cup [type : {"snap"}, rep : [Names -> PidOpt], epoch : 0..MaxEpoch, ver : Versions, ldr : Members] +\* With EpochFencedAck, a broadcast from a later epoch than f's is a gap: f missed +\* the new leader's snapshot, and only a snapshot moves f into a new epoch. +Later(m, f) == EpochFencedAck /\ m.epoch > epoch[f] + HeadIs(s, r, t) == /\ chan[s][r] # <<>> /\ Head(chan[s][r]).type = t @@ -124,6 +144,7 @@ Init == /\ chan = [s \in Members |-> [r \in Members |-> <<>>]] /\ degraded = [m \in Members |-> FALSE] /\ complete = [m \in Members |-> TRUE] + /\ leaderDown = [m \in Members |-> FALSE] (***************************************************************************) (* Elections and handoff *) @@ -142,7 +163,7 @@ Elect(m) == epoch, leaderView, deferred, pendingAcks, resyncReq, chan>> \* The genuine-leadership-change path: elector_assume_and_distribute -> -\* spawn_assume_gather -> gather_maps -> the {assume_gathered} continuation. +\* spawn_leadership_gather -> gather_maps -> the {assume_gathered} continuation. \* Modeled atomically; the gather reaches every live member (see module \* header). Freshest-wins: adopt the max-applied_version replica (version \* tie => identical content, which PrefixConsistency asserts). Deferred @@ -168,12 +189,20 @@ Elect(m) == \* then records the code's `degraded` flag (maxV < dbVersion) and, as ghost \* state, whether the adopted map really was the committed one — which \* DegradedFlagSound checks the flag against. +\* +\* With LeaderGoneAssume the gather also stops waiting, degraded, once every live +\* member (m included) has handled the DOWN for the leader it followed. Erlang +\* delivers a process's messages before its DOWN (ObserveDown below), so by then +\* nothing more from the old leader can arrive and waiting cannot help. This is +\* the code's `SelfLeaderGone andalso PeersSettled` exit in gather_until_caught_up. AssumeGather(m) == /\ alive[m] /\ dbLeader = m /\ epoch[m] < dbEpoch /\ LET maxV == MaxOf({appliedVer[x] : x \in Live}) - IN /\ (SafeAssumeGather => (maxV >= dbVersion \/ DegradedAssume)) + allDown == \A x \in Live : leaderDown[x] + IN /\ (SafeAssumeGather => + (maxV >= dbVersion \/ DegradedAssume \/ (LeaderGoneAssume /\ allDown))) /\ degraded' = [degraded EXCEPT ![m] = (maxV < dbVersion)] /\ \E src \in Live : /\ appliedVer[src] = maxV @@ -226,7 +255,8 @@ RegisterForward(f, l, n, p) == newRep == [rep[l] EXCEPT ![n] = p] b == [type |-> "bcast", name |-> n, pid |-> p, epoch |-> epoch[l], prev |-> appliedVer[l], ver |-> v] - r == [type |-> "reply", name |-> n, pid |-> p, ver |-> v] + r == [type |-> "reply", name |-> n, pid |-> p, ver |-> v, + epoch |-> epoch[l]] IN /\ dbVersion' = v /\ histMap' = [histMap EXCEPT ![v] = newRep] /\ rep' = [rep EXCEPT ![l] = newRep] @@ -280,13 +310,15 @@ RecvBcastApply(s, f) == /\ HeadIs(s, f, "bcast") /\ LET m == Head(chan[s][f]) IN /\ m.epoch >= epoch[f] + /\ ~Later(m, f) /\ m.prev = appliedVer[f] /\ LET newV == m.ver - flushed == {d \in deferred[f] : d[3] <= newV} + flushed == {d \in deferred[f] : + d[3] <= newV /\ (EpochFencedAck => d[4] = epoch[f])} IN /\ rep' = [rep EXCEPT ![f][m.name] = m.pid] /\ appliedVer' = [appliedVer EXCEPT ![f] = newV] /\ deferred' = [deferred EXCEPT ![f] = @ \ flushed] - /\ acked' = acked \cup flushed + /\ acked' = acked \cup {Reg(d) : d \in flushed} /\ resyncReq' = [resyncReq EXCEPT ![f] = NoMember] /\ chan' = Consume(s, f) /\ UNCHANGED <= epoch[f] + /\ ~Later(m, f) /\ m.prev # appliedVer[f] /\ m.ver <= appliedVer[f] /\ chan' = Consume(s, f) @@ -308,14 +341,16 @@ RecvBcastDrop(s, f) == pendingAcks, resyncReq>> \* A gap: we missed a batch. Refuse the broadcast (do NOT apply past the -\* hole) and flag a resync request to the stream's sender. +\* hole) and flag a resync request to the stream's sender. A broadcast from a +\* later epoch is a gap too (Later). RecvBcastGap(s, f) == /\ alive[f] /\ HeadIs(s, f, "bcast") /\ LET m == Head(chan[s][f]) IN - /\ m.epoch >= epoch[f] - /\ m.prev # appliedVer[f] - /\ m.ver > appliedVer[f] + \/ Later(m, f) + \/ /\ m.epoch >= epoch[f] + /\ m.prev # appliedVer[f] + /\ m.ver > appliedVer[f] /\ chan' = Consume(s, f) /\ resyncReq' = [resyncReq EXCEPT ![f] = IF @ = NoMember THEN s ELSE @] @@ -368,9 +403,9 @@ RecvHeartbeat(s, f) == /\ HeadIs(s, f, "hb") /\ LET m == Head(chan[s][f]) IN /\ chan' = Consume(s, f) - /\ IF m.epoch >= epoch[f] /\ m.ver = appliedVer[f] + /\ IF m.epoch >= epoch[f] /\ ~Later(m, f) /\ m.ver = appliedVer[f] THEN resyncReq' = [resyncReq EXCEPT ![f] = NoMember] - ELSE IF m.epoch >= epoch[f] /\ m.ver > appliedVer[f] + ELSE IF m.epoch >= epoch[f] /\ (Later(m, f) \/ m.ver > appliedVer[f]) THEN resyncReq' = [resyncReq EXCEPT ![f] = IF @ = NoMember THEN s ELSE @] ELSE UNCHANGED resyncReq @@ -387,6 +422,9 @@ RecvHeartbeat(s, f) == \* applied >= ver -> ack now (the broadcast/snapshot already delivered the \* row; the follower is a version-visible second holder); \* gapped + guard -> defer (awaiting_version), released by on_version; +\* With EpochFencedAck the version only counts in the epoch that committed it. +\* A reply from an epoch f has already left is dropped (the code re-sends it as +\* a fresh forward), and one from an epoch f hasn't reached yet is deferred. \* gapped, no guard-> the pre-guard bug: optimistic row write + immediate \* ack. The row exists but at a version the freshest-wins \* gather ignores — the silent-loss window. @@ -395,14 +433,15 @@ RecvReply(s, f) == /\ HeadIs(s, f, "reply") /\ LET m == Head(chan[s][f]) t == <> + h == <> IN /\ chan' = Consume(s, f) - /\ IF leaderView[f] # s + /\ IF leaderView[f] # s \/ (EpochFencedAck /\ m.epoch < epoch[f]) THEN UNCHANGED <> - ELSE IF appliedVer[f] >= m.ver + ELSE IF appliedVer[f] >= m.ver /\ (EpochFencedAck => m.epoch = epoch[f]) THEN /\ acked' = acked \cup {t} /\ UNCHANGED <> ELSE IF VersionGuardedAck - THEN /\ deferred' = [deferred EXCEPT ![f] = @ \cup {t}] + THEN /\ deferred' = [deferred EXCEPT ![f] = @ \cup {h}] /\ UNCHANGED <> ELSE /\ rep' = [rep EXCEPT ![f][m.name] = m.pid] /\ acked' = acked \cup {t} @@ -468,9 +507,11 @@ DegradeTimeout(l) == (***************************************************************************) \* Epoch-guarded (Epoch >= CurrentEpoch) wholesale re-baseline. On a -\* leadership change the deferred acks are rejected (do_leader_changed); +\* leadership change the deferred acks are rejected (the apply_names_snapshot handler); \* on a same-leader snapshot (a resync) the version advance releases them -\* (flush_deferred). A member that believed it led relinquishes. +\* (flush_deferred). A member that believed it led relinquishes. With +\* EpochFencedAck only acks held in the snapshot's own epoch are released, and +\* acks held from an earlier epoch are dropped (the code re-sends them). \* \* THE FIX, second half (SafeAssumeSnap): the re-baseline must be version-monotonic \* — a snapshot may not move a follower BACKWARD in applied_version. The pre-fix @@ -492,16 +533,23 @@ RecvSnap(s, f) == /\ chan' = Consume(s, f) /\ IF m.epoch >= epoch[f] /\ (SafeAssumeSnap => m.ver >= appliedVer[f]) THEN LET changed == leaderView[f] # m.ldr - flushed == IF changed THEN {} - ELSE {d \in deferred[f] : d[3] <= m.ver} + flushed == + IF EpochFencedAck + THEN {d \in deferred[f] : d[4] = m.epoch /\ d[3] <= m.ver} + ELSE IF changed THEN {} + ELSE {d \in deferred[f] : d[3] <= m.ver} + kept == + IF changed THEN {} + ELSE IF EpochFencedAck + THEN {d \in deferred[f] \ flushed : d[4] >= m.epoch} + ELSE deferred[f] \ flushed IN /\ rep' = [rep EXCEPT ![f] = m.rep] /\ appliedVer' = [appliedVer EXCEPT ![f] = m.ver] /\ epoch' = [epoch EXCEPT ![f] = m.epoch] /\ leaderView' = [leaderView EXCEPT ![f] = m.ldr] /\ resyncReq' = [resyncReq EXCEPT ![f] = NoMember] - /\ deferred' = [deferred EXCEPT ![f] = - IF changed THEN {} ELSE @ \ flushed] - /\ acked' = acked \cup flushed + /\ deferred' = [deferred EXCEPT ![f] = kept] + /\ acked' = acked \cup {Reg(d) : d \in flushed} /\ pendingAcks' = [pendingAcks EXCEPT ![f] = IF m.ldr # f THEN {} ELSE @] ELSE UNCHANGED <> +\* A member handles the DOWN for the leader it follows. Erlang delivers every +\* message a process sent before its DOWN, so the DOWN can only come once the +\* dead leader's channel to f is empty: each message was delivered or lost. Only +\* enabled under LeaderGoneAssume, so the other configs' state spaces are unchanged. +ObserveDown(f) == + /\ LeaderGoneAssume + /\ alive[f] + /\ ~leaderDown[f] + /\ LET l == leaderView[f] + IN /\ l \in Members /\ l # f + /\ ~alive[l] + /\ chan[l][f] = <<>> + /\ leaderDown' = [leaderDown EXCEPT ![f] = TRUE] + /\ UNCHANGED <> + (***************************************************************************) (* Next / Spec *) (***************************************************************************) @@ -581,9 +646,15 @@ Step == \/ RecvReply(s, r) \/ RecvSync(s, r) \/ RecvAck(s, r) \/ RecvSnap(s, r) \/ ServeResync(s, r) \/ DropMsg(s, r) +\* `leader_down` resets whenever a member's view of its leader changes. +ResetLeaderDown == + leaderDown' = [x \in Members |-> + IF leaderView'[x] # leaderView[x] THEN FALSE ELSE leaderDown[x]] + Next == - \/ \E m \in Members : AssumeGather(m) - \/ Step /\ UNCHANGED <> + \/ \E m \in Members : AssumeGather(m) /\ ResetLeaderDown + \/ \E m \in Members : ObserveDown(m) + \/ Step /\ UNCHANGED <> /\ ResetLeaderDown Spec == Init /\ [][Next]_vars @@ -601,13 +672,14 @@ TypeOK == /\ appliedVer \in [Members -> Versions] /\ epoch \in [Members -> 0..MaxEpoch] /\ leaderView \in [Members -> Members \cup {NoMember}] - /\ deferred \in [Members -> SUBSET Regs] + /\ deferred \in [Members -> SUBSET Held] /\ pendingAcks \in [Members -> SUBSET Regs] /\ resyncReq \in [Members -> Members \cup {NoMember}] /\ acked \subseteq Regs /\ crashes \in 0..MaxCrashes /\ degraded \in [Members -> BOOLEAN] /\ complete \in [Members -> BOOLEAN] + /\ leaderDown \in [Members -> BOOLEAN] /\ \A s \in Members : \A r \in Members : \A i \in 1..Len(chan[s][r]) : chan[s][r][i] \in Msg diff --git a/formal/DgenRegistryReplicationAckReachable.cfg b/formal/DgenRegistryReplicationAckReachable.cfg index a142298..d60969b 100644 --- a/formal/DgenRegistryReplicationAckReachable.cfg +++ b/formal/DgenRegistryReplicationAckReachable.cfg @@ -21,6 +21,8 @@ CONSTANTS SafeAssumeGather = TRUE SafeAssumeSnap = TRUE DegradedAssume = FALSE + LeaderGoneAssume = FALSE + EpochFencedAck = TRUE INVARIANTS NoAcks CONSTRAINT ChanBound diff --git a/formal/DgenRegistryReplicationDegradeOpen.cfg b/formal/DgenRegistryReplicationDegradeOpen.cfg index 99ae8c8..0793495 100644 --- a/formal/DgenRegistryReplicationDegradeOpen.cfg +++ b/formal/DgenRegistryReplicationDegradeOpen.cfg @@ -21,6 +21,8 @@ CONSTANTS SafeAssumeGather = TRUE SafeAssumeSnap = TRUE DegradedAssume = FALSE + LeaderGoneAssume = FALSE + EpochFencedAck = TRUE INVARIANTS DurableAcked CONSTRAINT ChanBound diff --git a/formal/DgenRegistryReplicationDegradeOpenSafety.cfg b/formal/DgenRegistryReplicationDegradeOpenSafety.cfg index 3fa1c47..9b78545 100644 --- a/formal/DgenRegistryReplicationDegradeOpenSafety.cfg +++ b/formal/DgenRegistryReplicationDegradeOpenSafety.cfg @@ -21,6 +21,8 @@ CONSTANTS SafeAssumeGather = TRUE SafeAssumeSnap = TRUE DegradedAssume = FALSE + LeaderGoneAssume = FALSE + EpochFencedAck = TRUE INVARIANTS TypeOK PrefixConsistency diff --git a/formal/DgenRegistryReplicationDegradedAssume.cfg b/formal/DgenRegistryReplicationDegradedAssume.cfg index 013bb0c..d081698 100644 --- a/formal/DgenRegistryReplicationDegradedAssume.cfg +++ b/formal/DgenRegistryReplicationDegradedAssume.cfg @@ -21,6 +21,8 @@ CONSTANTS SafeAssumeGather = TRUE SafeAssumeSnap = TRUE DegradedAssume = TRUE + LeaderGoneAssume = FALSE + EpochFencedAck = TRUE INVARIANTS TypeOK PrefixConsistency diff --git a/formal/DgenRegistryReplicationDegradedReachable.cfg b/formal/DgenRegistryReplicationDegradedReachable.cfg index 7faca3d..e3ab3c5 100644 --- a/formal/DgenRegistryReplicationDegradedReachable.cfg +++ b/formal/DgenRegistryReplicationDegradedReachable.cfg @@ -21,6 +21,8 @@ CONSTANTS SafeAssumeGather = TRUE SafeAssumeSnap = TRUE DegradedAssume = TRUE + LeaderGoneAssume = FALSE + EpochFencedAck = TRUE INVARIANTS NoDegradedAssume CONSTRAINT ChanBound diff --git a/formal/DgenRegistryReplicationHandoffGatherOnly.cfg b/formal/DgenRegistryReplicationHandoffGatherOnly.cfg index bc3e40a..7bf8217 100644 --- a/formal/DgenRegistryReplicationHandoffGatherOnly.cfg +++ b/formal/DgenRegistryReplicationHandoffGatherOnly.cfg @@ -21,6 +21,8 @@ CONSTANTS SafeAssumeGather = FALSE SafeAssumeSnap = TRUE DegradedAssume = FALSE + LeaderGoneAssume = FALSE + EpochFencedAck = TRUE INVARIANTS DurableAcked UniqueBinding diff --git a/formal/DgenRegistryReplicationHandoffRace.cfg b/formal/DgenRegistryReplicationHandoffRace.cfg index 4e5616e..26fa1c4 100644 --- a/formal/DgenRegistryReplicationHandoffRace.cfg +++ b/formal/DgenRegistryReplicationHandoffRace.cfg @@ -21,6 +21,8 @@ CONSTANTS SafeAssumeGather = FALSE SafeAssumeSnap = FALSE DegradedAssume = FALSE + LeaderGoneAssume = FALSE + EpochFencedAck = TRUE INVARIANTS DurableAcked UniqueBinding diff --git a/formal/DgenRegistryReplicationHandoffRaceUnique.cfg b/formal/DgenRegistryReplicationHandoffRaceUnique.cfg index 91dc9d1..d4f3a5a 100644 --- a/formal/DgenRegistryReplicationHandoffRaceUnique.cfg +++ b/formal/DgenRegistryReplicationHandoffRaceUnique.cfg @@ -21,6 +21,8 @@ CONSTANTS SafeAssumeGather = FALSE SafeAssumeSnap = FALSE DegradedAssume = FALSE + LeaderGoneAssume = FALSE + EpochFencedAck = TRUE INVARIANTS UniqueBinding CONSTRAINT ChanBound diff --git a/formal/DgenRegistryReplicationHeartbeat.cfg b/formal/DgenRegistryReplicationHeartbeat.cfg index 8b76326..40a4b1c 100644 --- a/formal/DgenRegistryReplicationHeartbeat.cfg +++ b/formal/DgenRegistryReplicationHeartbeat.cfg @@ -21,6 +21,8 @@ CONSTANTS SafeAssumeGather = TRUE SafeAssumeSnap = TRUE DegradedAssume = FALSE + LeaderGoneAssume = FALSE + EpochFencedAck = TRUE INVARIANTS TypeOK PrefixConsistency diff --git a/formal/DgenRegistryReplicationLeaderGoneAssume.cfg b/formal/DgenRegistryReplicationLeaderGoneAssume.cfg new file mode 100644 index 0000000..77193a7 --- /dev/null +++ b/formal/DgenRegistryReplicationLeaderGoneAssume.cfg @@ -0,0 +1,43 @@ +SPECIFICATION Spec +CONSTANTS + m1 = m1 + m2 = m2 + m3 = m3 + Members = {m1, m2, m3} + n1 = n1 + Names = {n1} + p1 = p1 + p2 = p2 + Pids = {p1, p2} + NoPid = NoPid + NoMember = NoMember + MaxVersion = 3 + MaxEpoch = 2 + MaxCrashes = 1 + MaxChanLen = 2 + VersionGuardedAck = TRUE + DegradeOpen = FALSE + Heartbeat = FALSE + SafeAssumeGather = TRUE + SafeAssumeSnap = TRUE + DegradedAssume = FALSE + LeaderGoneAssume = TRUE + EpochFencedAck = TRUE +INVARIANTS + TypeOK + PrefixConsistency + UniqueBinding + DurableAcked + LeaderEpochUnique + DegradedFlagSound +CONSTRAINT ChanBound +SYMMETRY Symm + +\* The gather stops waiting once every live member has handled its old leader's +\* DOWN (LeaderGoneAssume = TRUE), with the plain deadline off. The `degraded` flag +\* stays sound, as in DgenRegistryReplicationDegradedAssume.cfg. Unlike the +\* deadline, this exit also keeps UniqueBinding and DurableAcked. It only fires +\* once nothing more can arrive from the old leader, so a commit it misses was +\* never acked. DgenRegistryReplicationNoEpochFence.cfg shows that this relies on +\* EpochFencedAck. DgenRegistryReplicationLeaderGoneReachable.cfg is the canary +\* proving this exit is actually taken. diff --git a/formal/DgenRegistryReplicationLeaderGoneReachable.cfg b/formal/DgenRegistryReplicationLeaderGoneReachable.cfg new file mode 100644 index 0000000..124c5d5 --- /dev/null +++ b/formal/DgenRegistryReplicationLeaderGoneReachable.cfg @@ -0,0 +1,33 @@ +SPECIFICATION Spec +CONSTANTS + m1 = m1 + m2 = m2 + m3 = m3 + Members = {m1, m2, m3} + n1 = n1 + Names = {n1} + p1 = p1 + p2 = p2 + Pids = {p1, p2} + NoPid = NoPid + NoMember = NoMember + MaxVersion = 3 + MaxEpoch = 2 + MaxCrashes = 1 + MaxChanLen = 2 + VersionGuardedAck = TRUE + DegradeOpen = FALSE + Heartbeat = FALSE + SafeAssumeGather = TRUE + SafeAssumeSnap = TRUE + DegradedAssume = FALSE + LeaderGoneAssume = TRUE + EpochFencedAck = TRUE +INVARIANTS + NoDegradedAssume +CONSTRAINT ChanBound +SYMMETRY Symm + +\* Vacuity canary for DgenRegistryReplicationLeaderGoneAssume.cfg. The plain +\* deadline is off, so the only way to assume degraded is the leader-gone exit. +\* NoDegradedAssume MUST FAIL here, proving that exit is reached. diff --git a/formal/DgenRegistryReplicationNoEpochFence.cfg b/formal/DgenRegistryReplicationNoEpochFence.cfg new file mode 100644 index 0000000..4747638 --- /dev/null +++ b/formal/DgenRegistryReplicationNoEpochFence.cfg @@ -0,0 +1,37 @@ +SPECIFICATION Spec +CONSTANTS + m1 = m1 + m2 = m2 + m3 = m3 + Members = {m1, m2, m3} + n1 = n1 + Names = {n1} + p1 = p1 + p2 = p2 + Pids = {p1, p2} + NoPid = NoPid + NoMember = NoMember + MaxVersion = 3 + MaxEpoch = 2 + MaxCrashes = 1 + MaxChanLen = 2 + VersionGuardedAck = TRUE + DegradeOpen = FALSE + Heartbeat = FALSE + SafeAssumeGather = TRUE + SafeAssumeSnap = TRUE + DegradedAssume = FALSE + LeaderGoneAssume = TRUE + EpochFencedAck = FALSE +INVARIANTS + DurableAcked +CONSTRAINT ChanBound +SYMMETRY Symm + +\* The pre-fix false ack (EpochFencedAck = FALSE), under the same settings as +\* DgenRegistryReplicationLeaderGoneAssume.cfg, where DurableAcked holds. A +\* follower holds a forwarded `yes` until its replica reaches the commit version. +\* The leader that committed it dies before broadcasting, and the next leader takes +\* over degraded without it. The follower misses the new leader's snapshot, applies +\* its next batch, and releases the held `yes` for a binding nobody holds. MUST FAIL +\* on DurableAcked. diff --git a/formal/DgenRegistryReplicationNoGuard.cfg b/formal/DgenRegistryReplicationNoGuard.cfg index dbad23b..2094a3a 100644 --- a/formal/DgenRegistryReplicationNoGuard.cfg +++ b/formal/DgenRegistryReplicationNoGuard.cfg @@ -21,6 +21,8 @@ CONSTANTS SafeAssumeGather = TRUE SafeAssumeSnap = TRUE DegradedAssume = FALSE + LeaderGoneAssume = FALSE + EpochFencedAck = TRUE INVARIANTS UniqueBinding DurableAcked diff --git a/formal/DgenRegistryReplicationSnapMonotonicOnly.cfg b/formal/DgenRegistryReplicationSnapMonotonicOnly.cfg index aa386ab..59bc0d1 100644 --- a/formal/DgenRegistryReplicationSnapMonotonicOnly.cfg +++ b/formal/DgenRegistryReplicationSnapMonotonicOnly.cfg @@ -21,6 +21,8 @@ CONSTANTS SafeAssumeGather = TRUE SafeAssumeSnap = FALSE DegradedAssume = FALSE + LeaderGoneAssume = FALSE + EpochFencedAck = FALSE INVARIANTS DurableAcked UniqueBinding @@ -33,3 +35,9 @@ SYMMETRY Symm \* broadcast the follower has since applied — rewinds that follower without the \* monotonic guard. So each half of the §5.7 fix is independently necessary, \* which the paired HandoffGatherOnly config proves for the other half. +\* +\* EpochFencedAck is FALSE here on purpose. The rewind needs a follower that +\* applied a later epoch's broadcast before that epoch's snapshot, and the epoch +\* fence turns that broadcast into a gap. With the fence on, TLC found no +\* violation in 146M distinct states, so the fence alone also closes this race. +\* The code keeps both checks. diff --git a/formal/README.md b/formal/README.md index 8d4fe9a..f3a76e7 100644 --- a/formal/README.md +++ b/formal/README.md @@ -1,12 +1,12 @@ # dgen_registry Formal Model A TLA+ model of the replication protocol in -[`src/dgen_registry_member.erl`](../src/dgen_registry_member.erl) (design: -[`docs/design/dgen_registry_design.md`](../docs/design/dgen_registry_design.md) +[`src/dgen_registry/dgen_registry_member.erl`](../src/dgen_registry/dgen_registry_member.erl) +(design: [`docs/design/dgen_registry_design.md`](../docs/design/dgen_registry_design.md) §4.5, §5). TLC checks its safety properties against every interleaving of a small bounded cluster. Tests can only sample interleavings; the -[simulation harness](../test/support/sim/README.md) covers the other side: it checks -that the real code implements this protocol. +[simulation harness](../test/support/sim/dgen_registry_simulation.md) covers the +other side: it checks that the real code implements this protocol. ## What it models @@ -19,7 +19,8 @@ One module, `DgenRegistryReplication.tla`: - **Resync**: a follower with a gap requests a snapshot from its stream's sender. - **Forwarded registrations, version-guarded acks**: a follower answers its - caller only after its replica has applied the batch. + caller only after its replica has applied the batch, in the epoch that + committed it (`EpochFencedAck`). - **Direct registrations, replicate-before-ack**: the leader waits for a follower to confirm before answering `yes`. The degrade-open timeout is a constant. @@ -37,7 +38,7 @@ One module, `DgenRegistryReplication.tla`: **Deliberately out of scope**: unregister, retract and `DOWN` cleanup; the §5.6 conflict trail and kill budget; metadata, queries and presence; membership joins and partial gathers, including the continuing-leader join fast path -(`assume_fast_path/6` / `onboard_joiner` / `{peer_joined}`); `register_replicas +(`assume_leadership_fast_path/6` / `cast_data_to_new_member` / `{peer_joined}`); `register_replicas > 1` (`RecvAck` resolves on the first follower ack); Erlang-distribution-only partitions. The simulation harness covers register/unregister concurrency. The module header in the `.tla` has the full scope note. @@ -68,14 +69,17 @@ take seconds. | `DgenRegistryReplicationHeartbeat` | pass | the first five with the heartbeat on (`MaxVersion = 2`) | | `DgenRegistryReplicationDegradeOpenSafety` | pass | the default degrade-open mode keeps everything except `DurableAcked` | | `DgenRegistryReplicationDegradedAssume` | pass | with the gather deadline modeled, `DegradedFlagSound` holds: the flag never misses an incomplete reconstruction | +| `DgenRegistryReplicationLeaderGoneAssume` | pass | the gather stops waiting once every live member has handled its old leader's DOWN. All six invariants hold. The plain deadline can't keep `DurableAcked`, but this exit only fires once nothing more can arrive from the old leader | | `DgenRegistryReplicationHandoffRace` | fail | the pre-§5.7 handoff (both `SafeAssume*` off): one crash loses an acked registration (`DurableAcked`) | | `DgenRegistryReplicationHandoffRaceUnique` | fail | the same race checked against `UniqueBinding` alone: the freed name is re-issued to a second pid | | `DgenRegistryReplicationHandoffGatherOnly` | fail | the gather fence alone off: it is needed on its own | -| `DgenRegistryReplicationSnapMonotonicOnly` | fail | the snapshot guard alone off: it is needed on its own | -| `DgenRegistryReplicationNoGuard` | fail | optimistic follower acks (`VersionGuardedAck = FALSE`) lose `DurableAcked`. The code's versionless `{register_reply, Ref, Result}` rolling-upgrade clause has this shape | +| `DgenRegistryReplicationSnapMonotonicOnly` | fail | the snapshot guard alone off, with `EpochFencedAck` also off: it was needed on its own. The epoch fence alone also closes this race (see the `.cfg`) | +| `DgenRegistryReplicationNoGuard` | fail | optimistic follower acks (`VersionGuardedAck = FALSE`) lose `DurableAcked` | | `DgenRegistryReplicationDegradeOpen` | fail | degrade-open (`strict_replication = false`) loses `DurableAcked` | | `DgenRegistryReplicationAckReachable` | fail | vacuity canary: `NoAcks` fails, so acks are actually reachable | | `DgenRegistryReplicationDegradedReachable` | fail | vacuity canary: `NoDegradedAssume` fails, so degraded assumes are actually reachable | +| `DgenRegistryReplicationLeaderGoneReachable` | fail | vacuity canary: with the deadline off, `NoDegradedAssume` fails, so the leader-gone exit is actually taken | +| `DgenRegistryReplicationNoEpochFence` | fail | `EpochFencedAck = FALSE`: a follower releases a held ack when a later leader's batch passes its version, though that leader took over without the commit (`DurableAcked`) | If a vacuity canary ever passes, the model has stopped exercising its behaviour, and a green run on the other configs means much less. @@ -85,17 +89,18 @@ behaviour, and a green run on the other configs means much less. | Spec | Code | |---|---| | `Elect` | `dgen_registry_elector` committing a membership/leadership change (abstracted to one durable write) | -| `AssumeGather` (incl. the `SafeAssumeGather` version-key fence) | the `assuming` statem state: `assume_genuine/6` → `spawn_assume_gather` → `gather_caught_up/6` → the `{assume_gathered}` continuation, valid only in `assuming` for its arming ref. The continuing-leader join fast path (`assume_fast_path/6` / `onboard_joiner` / `{peer_joined}`) dispatches in the `leader` state and is out of scope | +| `AssumeGather` (incl. the `SafeAssumeGather` version-key fence) | the `assuming` statem state: `prepare_to_assume_leadership/6` → `spawn_leadership_gather` → `gather_until_caught_up/7` → the `{assume_gathered}` continuation, valid only in `assuming` for its arming ref. The continuing-leader join fast path (`assume_leadership_fast_path/6` / `cast_data_to_new_member` / `{peer_joined}`) dispatches in the `leader` state and is out of scope | | `HeartbeatBcast` / `RecvHeartbeat` | `handle_event(info, replica_heartbeat, leader, …)` → `broadcast_heartbeat/1` (`{names_batch, [], E, V, V, Self}`) / the empty batch through `dgen_registry_replica:offer_batch/5` | | `CanCommit`'s `dbLeader/dbEpoch` conjuncts | the fenced version-key bump in `dgen_registry_names:start_commit/4` (§5.1) | | `RegisterForward` | follower `route_register` forward → leader `{register_req}` → group commit → `broadcast_batch/5` + `{register_reply, Ref, yes, Version}` | -| `RegisterDirect` + `RecvSync`/`RecvAck` | leader-local `route_register` → `dgen_registry_pipeline:await_acks/5` / `{replicate_sync}` / `{replicate_ack}` → `dgen_registry_pipeline:ack/3` | +| `RegisterDirect` + `RecvSync`/`RecvAck` | leader-local `route_register` → `dgen_registry_pipeline:await_acks/5` / `{replicate_sync}` / `{replicate_ack}` → `dgen_registry_pipeline:record_ack/3` | | `DegradeTimeout` | `handle_info({replicate_timeout, _})` with `strict_replication = false` | -| `RecvBcastApply/Drop/Gap` | `dgen_registry_replica:offer_batch/5` → `applied` / `stale_epoch`+`duplicate` / `gap` (one `{names_batch, …}` per commit version). The member's `{names_batch}` handler reacts to the verdict | -| `RecvReply` | `dgen_registry_intents:on_reply/4` incl. the `awaiting_version` phase. Flushes are `dgen_registry_intents:on_version/2`, called from the `{names_batch}` handler's `applied` branch and on an accepted snapshot. On a leadership change the code re-drives a deferred ack as a fresh forward (`redrive/5`) where the spec drops it: a fresh `RegisterForward` in spec terms | +| `RecvBcastApply/Drop/Gap` | `dgen_registry_replica:offer_batch/5` → `applied` / `stale_epoch`+`duplicate` / `gap` (one `{names_batch, …}` per commit version). A batch from a later epoch is a `gap` (`Later`). The member's `{names_batch}` handler reacts to the verdict | +| `RecvReply` | `dgen_registry_intents:on_reply/5` incl. the `awaiting_version` phase. The reply carries the leader's epoch, and a held ack keeps it. Flushes are `dgen_registry_intents:on_version/2`, called from the `{names_batch}` handler's `applied` branch and on an accepted snapshot. Both release only in the held ack's epoch. On a leadership change, or once our replica passes that epoch, the code re-drives a deferred ack as a fresh forward (`redrive/5`) where the spec drops it: a fresh `RegisterForward` in spec terms | | `RecvSnap` (incl. the `SafeAssumeSnap` version-monotonic guard) | `dgen_registry_replica:offer_snapshot/4` (the `Epoch >= CurEpoch andalso V >= Applied` guard), called from `handle_cast({apply_names_snapshot, ...})` | | `ServeResync` / `resyncReq` | `{resync_req}` handler / `request_resync/2` | | `Crash` | member/node death; the ETS replica dies with the process | +| `ObserveDown` / `leaderDown` (with `LeaderGoneAssume`) | the peer-monitor `'DOWN'` handler setting `leader_down` when the dead member is our `leader`. `get_names_snapshot` reports it, and `gather_until_caught_up/7` stops waiting once our own flag and every peer's are set | | `DropMsg` | casts lost on an Erlang-distribution disconnect (signal-ordering semantics) | The member's `gen_statem` states (`searching`/`assuming`/`leader`/`follower`) diff --git a/include/dgen_eta.hrl b/include/dgen_eta.hrl index dd1917f..4c30a15 100644 --- a/include/dgen_eta.hrl +++ b/include/dgen_eta.hrl @@ -2,24 +2,22 @@ %% %% -include("../include/dgen_eta.hrl"). %% -%% Everything here hangs on the `DST` define, which the test build sets (see -%% `erlc_options/1` in mix.exs) and a release build does not, so the -%% instrumentation below can be left in the source of a module that ships. +%% Everything here depends on the `DST` define. The test build sets it, in +%% `erlc_options/1` in mix.exs, and a release build does not. That lets a module +%% that ships keep the instrumentation below in its source. %% %% ## Why not include `eta/include/eta.hrl` directly %% %% `-include_lib` resolves through the code path, so `eta` would have to be -%% findable in *every* compilation — which the library's own walkthrough handles by -%% telling adopters to depend on it with `runtime: false`. dgen is a published -%% package, and test tooling has no business becoming a package requirement for -%% every consumer, so `eta` stays behind `only: :test` (pinned from Hex, ~> 0.1 — -%% see mix.exs). +%% available in every compilation. The eta walkthrough handles this by telling +%% adopters to depend on it with `runtime: false`. But dgen is a published package, +%% and its consumers shouldn't have to pull in our test tooling. So `eta` stays +%% behind `only: :test`, pinned from Hex at ~> 0.1 in mix.exs. %% -%% So `eta` stays test-only, the `-include_lib` sits behind `-ifdef(DST)` where the -%% preprocessor never resolves it in a release build, and the `-else` branch below -%% supplies the macros that branch would otherwise leave undefined. Without that -%% branch, `?ETA_LOG` in shipped code is a compile error in the only build that -%% matters. +%% The `-include_lib` sits behind `-ifdef(DST)`, so the preprocessor never resolves +%% it in a release build. The `-else` branch below defines the macros that the +%% include would otherwise provide. Without it, `?ETA_LOG` in shipped code would be +%% a compile error in the release build, which is the one that matters. %% %% *This documentation is LLM-generated. See the AI disclosure in README.md.* @@ -28,16 +26,28 @@ -ifdef(DST). -%% Brings the parse transform and the real macros: timer and clock calls rewritten -%% to `eta_time`, sends to `eta_net`, spawns to `eta_sched`, and `-eta_observe` -%% fields published on every callback return. Inert unless a run is active. +%% Brings in the parse transform and the real macros. The transform rewrites timer +%% and clock calls to `eta_time`, sends to `eta_net` and spawns to `eta_sched`, and +%% publishes `-eta_observe` fields on every callback return. None of it does +%% anything unless a simulation run is active. -include_lib("eta/include/eta.hrl"). +%% Turns off the supervisor's `hibernate_after` in DST builds on OTP 28 and later. +%% It is a wall-clock timer, and elector_pid/1 calls into the supervisor, so it +%% would make runs with the same seed diverge. +-if(?OTP_RELEASE >= 28). +-define(SUP_SIM_FLAGS, #{hibernate_after => infinity}). -else. +-define(SUP_SIM_FLAGS, #{}). +-endif. + +-else. + +-define(SUP_SIM_FLAGS, #{}). -%% Release build. `Event` and `Label` are **not evaluated**, matching `eta.hrl`, so -%% anything logged must be side-effect free and a variable used only inside one of -%% these reads as unused. +%% Release build. Like `eta.hrl`, these macros do not evaluate `Event` or `Label`. +%% So anything logged must be free of side effects, and a variable used only inside +%% one of these macros counts as unused. -define(ETA_LABEL(Label), ok). -define(ETA_LOG(Event), 0). diff --git a/lib/dgen/registry.ex b/lib/dgen/registry.ex index d088559..a813eb6 100644 --- a/lib/dgen/registry.ex +++ b/lib/dgen/registry.ex @@ -9,9 +9,9 @@ defmodule DGen.Registry do {:ok, _sup} = DGen.Registry.start_link(:my_registry, tenant) - `start_link/2,3` returns the supervisor's pid — hold onto it (e.g. to - `Supervisor.stop/2` it later) rather than looking it up by name afterward; - see "Process identity" below. + `start_link/2,3` returns the pid of the supervisor. Keep that pid if you + need it later, for example to pass to `Supervisor.stop/2`. You can't look the + supervisor up by name, as "Process identity" below explains. ## Via-tuple usage @@ -21,32 +21,31 @@ defmodule DGen.Registry do ## Process identity - Starting a registry creates zero atoms. `name` is supplied by the caller and - is application-controlled, not generated by this module. The supervisor - itself is not registered under any name, and the elector has no registered - name either — only the member is, under `name` directly, with no further - atom derived from it. This matters for applications that start many - independent registries dynamically (e.g. one per tenant): atoms are never - garbage-collected, so this module never derives additional atoms per - registry beyond the one the caller already chose to create. + Starting a registry creates zero atoms. The caller supplies `name`, and this + module never generates one. Neither the supervisor nor the elector has a + registered name. Only the member is registered, under `name` itself, and no + other atom is derived from it. This matters for applications that start many + registries dynamically, such as one per tenant. Atoms are never + garbage-collected, so we avoid creating any beyond the one the caller chose. ## Consistency model `register_name/2`, `unregister_name/1`, and `whereis_name_consistent/1` - route through the elected leader. `whereis_name/1` (used by the OTP - via-tuple machinery) is served lock-free from the calling process — an ETS - read against the local member's table, no member round-trip, no network hop. - There is a short replication window after registration where a follower - node's snapshot read may still return `nil`. + route through the elected leader. The OTP via-tuple machinery uses + `whereis_name/1`, which the calling process serves itself without locks. It + reads the local member's ETS table directly, with no round-trip to the member + and no network hop. Right after a registration, replication takes a moment to + reach other nodes. During that window, a snapshot read on a follower node may + still return `nil`. ## Metadata and queries - A registration can carry metadata for its lifetime: `register_name/3` / - `set_metadata/2` attach an `index` map (queryable) and an opaque `data` - payload; `get_metadata/1` / `get_metadata_consistent/1` read it back - (lock-free / leader-routed, mirroring `whereis_name/1` / - `whereis_name_consistent/1`). `query/2` / `query_consistent/2` find every - registration whose `index` satisfies a conjunction of exact equalities. + A registration can carry metadata for as long as it lives. `register_name/3` + and `set_metadata/2` attach an `index` map, which queries can search, and an + opaque `data` payload. `get_metadata/1` reads the metadata back without locks, + like `whereis_name/1`. `get_metadata_consistent/1` asks the leader, like + `whereis_name_consistent/1`. `query/2` and `query_consistent/2` find every + registration whose `index` matches all of a set of exact equalities. *This documentation is LLM-generated. See the AI disclosure in `README.md`.* """ @@ -71,19 +70,19 @@ defmodule DGen.Registry do @doc """ Starts the registry `name` with default options. - Returns `{:ok, sup_pid}` — the supervisor is not registered under any name; - see "Process identity" above. + Returns `{:ok, sup_pid}`. The supervisor has no registered name, as + "Process identity" above explains. """ defdelegate start_link(name, tenant), to: :dgen_registry @doc """ Starts the registry `name` with per-registry options `opts`. - See `docs/dgen_registry_design.md` (§8 Configuration) for the available - tuning knobs (`register_replicas`, `strict_replication`, - `connectivity`, etc.). Set `connectivity: :provided_externally` to turn - off this registry's proactive distribution mesh so it free-rides on the - connections another registry maintains (§4.6). + See `docs/design/dgen_registry_design.md` (§8 Configuration) for the + available settings, such as `register_replicas`, `strict_replication` and + `connectivity`. Set `connectivity: :provided_externally` to turn off this + registry's own distribution mesh. It then relies on the connections that + another registry maintains (§4.6). """ defdelegate start_link(name, tenant, opts), to: :dgen_registry @@ -91,29 +90,31 @@ defmodule DGen.Registry do Registers `pid` under `{registry_name, logical_name}`. Routes through the local member, which forwards to the leader if needed. - Returns `:yes` on success, `:no` if the name is already taken or no leader - is currently elected. + Returns `:yes` on success. Returns `:no` if another live process holds the + name, or if an opt-in policy refuses it (design doc §8). + If no leader is reachable, the call waits for one. It exits if nothing decides + within `register_timeout` (design doc §3). """ defdelegate register_name(name, pid), to: :dgen_registry @doc """ Removes the registration for `{registry_name, logical_name}`. Returns `:ok`. - A tracked call mirroring the register path: the local member's snapshot is - updated immediately, and the removal is forwarded to the leader. `:ok` - normally means the leader committed it; with an unreachable leader the - removal is stashed and re-driven on the next leadership/rejoin event, so an + Works like the register path. The local member updates its snapshot right + away and forwards the removal to the leader. `:ok` normally means the leader + committed it. If the leader is unreachable, the member holds on to the + removal and sends it again when leadership changes or the member rejoins. An explicit unregister is never silently lost (design doc, Non-goal 5). """ defdelegate unregister_name(name), to: :dgen_registry @doc """ - Snapshot read — served lock-free from the calling process (an ETS lookup - against the local member's table, no member round-trip). + Looks up `name` in the local snapshot. The calling process reads the local + member's ETS table directly, without locks or a round-trip to the member. - Never blocks on the leader or the backend. May be slightly stale on - follower nodes in the brief window between a remote registration and the - replication cast arriving. Returns `nil` (`:undefined`) if not found. + Never blocks on the leader or the backend. On a follower node it may be + briefly stale, between a registration on another node and the replication + message arriving here. Returns `nil` (`:undefined`) if not found. """ defdelegate whereis_name(name), to: :dgen_registry @@ -126,29 +127,28 @@ defmodule DGen.Registry do defdelegate send(name, msg), to: :dgen_registry @doc """ - Registers `pid` under `{registry_name, logical_name}` with metadata, in one - fenced step. + Registers `pid` under `{registry_name, logical_name}` with metadata, in a + single commit by the leader. - `meta_spec` is `%{index: map(), data: term()}` (both optional; default `%{}` - and `nil`). Like `register_name/2`, returns `:yes` on success or `:no` if the - name is already taken or no leader is currently elected. + `meta_spec` is `%{index: map(), data: term()}`. Both keys are optional, and + default to `%{}` and `nil`. Answers the same way as `register_name/2`. """ defdelegate register_name(name, pid, meta_spec), to: :dgen_registry @doc """ Replaces the metadata of the existing registration `{registry_name, logical_name}`. - `meta_spec` is `%{index: map(), data: term()}` (both optional; an omitted - field becomes `%{}` / `nil` — this is a replace, not a merge). Returns `:ok`, - `{:error, :not_registered}` if the name is not currently bound, or - `{:error, :no_leader}` if no leader is elected (or leadership changed - mid-flight — retry). + `meta_spec` is `%{index: map(), data: term()}`. Both keys are optional. This + replaces the metadata rather than merging, so an omitted key becomes `%{}` or + `nil`. Returns `:ok`, or `{:error, :not_registered}` if the name is not + currently registered. Returns `{:error, :no_leader}` if no leader is elected, + or if leadership changed before the leader answered. In that case, retry. """ defdelegate set_metadata(name, meta_spec), to: :dgen_registry @doc """ - Snapshot read of a registration's metadata — served lock-free from the - calling process, mirroring `whereis_name/1`. + Reads a registration's metadata from the local snapshot. Like + `whereis_name/1`, the calling process reads it directly without locks. Returns `{:ok, %{pid: pid, index: map, data: term}}` for a registered name, or `nil` (`:undefined`) if not registered. @@ -156,83 +156,85 @@ defmodule DGen.Registry do defdelegate get_metadata(name), to: :dgen_registry @doc """ - Consistent metadata read routed through the leader (authoritative, never stale). + Reads a registration's metadata through the leader. The answer is + authoritative and never stale. Returns the same shape as `get_metadata/1`. """ defdelegate get_metadata_consistent(name), to: :dgen_registry @doc """ - Finds every registration in `registry_name` whose indexed metadata satisfies - *all* of `constraints` (a conjunction of exact equalities — AND-equal). + Finds every registration in `registry_name` whose indexed metadata matches + *all* of `constraints`. Each constraint is an exact equality. - Snapshot read, local to the calling member; batch-consistent (never observes - a half-applied group commit). Returns a list of `%{name:, pid:, index:, - data:}` matches, or `{:error, :empty_query}` if `constraints` is empty. + Reads the snapshot on the local member. It never sees a group commit that is + only partly applied. Returns a list of `%{name:, pid:, index:, data:}` + matches, or `{:error, :empty_query}` if `constraints` is empty. """ defdelegate query(registry_name, constraints), to: :dgen_registry @doc """ - Like `query/2`, but routed through the leader for an authoritative (never-stale) result. + Like `query/2`, but asks the leader, so the result is authoritative and never stale. """ defdelegate query_consistent(registry_name, constraints), to: :dgen_registry @doc """ - Creates (or updates) a durable presence subscription `sub_id`: keeps the processes - matching `notify` up to date with the set of processes matching `watch`. - - `watch` and `notify` are queries (`{:all, %{attr => value}}`, or a bare map read as - `{:all, map}`). `sub_id` is any application term (an idempotent upsert). Whenever a - committed batch changes which processes satisfy `watch`, every process currently - matching `notify` receives one message `{:dgen_presence, sub_id, events}` where - `events` is a non-empty list of `{:joined, name, pid}` / `{:left, name, pid}`. A - process is sent an initial snapshot of the current `watch` set (as `:joined` events) - whenever it enters the notify set — at subscribe time *or* when it registers into - `notify` later — and deltas thereafter, so the subscription can be created before any - viewer exists. - - The subscription is **durable** — stored in the elector's backend state, so it - outlives the cluster: tie its lifetime to a database entity and presence comes back - intact after a scale-to-zero. Sent as a durable cast (applied asynchronously), so - `:ok` means "accepted"; `subscriptions/1` reflects it shortly after. Returns `:ok`, or - `{:error, :empty_query}` if either query is empty. + Creates or updates the durable presence subscription `sub_id`. It keeps the + processes matching `notify` up to date with the set of processes matching `watch`. + + `watch` and `notify` are queries. Each is `{:all, %{attr => value}}`, or a bare map, + which is read as `{:all, map}`. `sub_id` is any application term. Subscribing again + with the same `sub_id` replaces the earlier subscription. Whenever a committed batch + changes which processes match `watch`, every process currently matching `notify` + receives one message `{:dgen_presence, sub_id, events}`. `events` is a non-empty list + of `{:joined, name, pid}` and `{:left, name, pid}` tuples. When a process enters the + notify set, it first gets the current `watch` set as `:joined` events, and then the + changes after that. This happens both at subscribe time and when a process registers + into `notify` later. So you can create the subscription before any viewer exists. + + The subscription is **durable**. The elector stores it in its backend state, so it + outlives the cluster. If you tie its lifetime to a database entity, presence comes + back intact after the cluster scales to zero and back. The request is sent as a + durable cast and applied asynchronously, so `:ok` only means it was accepted. + `subscriptions/1` shows it shortly after. Returns `:ok`, or `{:error, :empty_query}` + if either query is empty. """ defdelegate subscribe(registry_name, sub_id, watch, notify), to: :dgen_registry @doc """ - Removes the durable presence subscription `sub_id` (from `subscribe/4`). Returns `:ok` - (idempotent; applied asynchronously). + Removes the durable presence subscription `sub_id` created by `subscribe/4`. Returns + `:ok`. It is safe to call more than once, and it is applied asynchronously. """ defdelegate unsubscribe(registry_name, sub_id), to: :dgen_registry @doc """ - Removes all durable presence subscriptions from the registry. Returns `:ok` (applied - asynchronously). + Removes all durable presence subscriptions from the registry. Returns `:ok`. The + removal is applied asynchronously. """ defdelegate unsubscribe_all(registry_name), to: :dgen_registry @doc """ Returns the registry's durable presence subscriptions as - `%{sub_id => {watch, notify}}` (shared cluster-wide and durable). + `%{sub_id => {watch, notify}}`. Every node sees the same set. """ defdelegate subscriptions(registry_name), to: :dgen_registry @doc """ - Removes all of the registry's durable backend state (membership, leader/version fence, - and subscriptions). Returns `:ok`. + Removes all of the registry's durable backend state: membership, the leader and + version keys, and subscriptions. Returns `:ok`. - Stop the registry supervisor first, then call this with the tenant you started it with - — a running elector would re-create the keys from its in-memory state. Clears only this - registry's keys, not any other registry sharing the tenant. + Stop the registry supervisor first, then call this with the tenant you started it + with. A running elector would write the keys again from its in-memory state. This + clears only this registry's keys, not those of other registries in the same tenant. """ defdelegate delete(registry_name, tenant), to: :dgen_registry @doc """ - Consistent read routed through the leader. + Looks up `name` through the leader. Returns the authoritative pid for `{registry_name, logical_name}`, or - `nil` (`:undefined`) if not registered. More expensive than `whereis_name/1` - but never stale. + `nil` (`:undefined`) if not registered. It costs more than `whereis_name/1`, + but it is never stale. """ defdelegate whereis_name_consistent(name), to: :dgen_registry @@ -246,8 +248,9 @@ defmodule DGen.Registry do defdelegate get_members(registry_name), to: :dgen_registry @doc """ - Whether the registry on this node is ready to serve registrations right now - (a leader is known and this member has synced). Point-in-time; see `await_ready/2`. + Returns whether the registry on this node can serve registrations right now. + That means a leader is known and this member has synced. To wait for it, use + `await_ready/2`. """ defdelegate ready(registry_name), to: :dgen_registry @@ -262,40 +265,41 @@ defmodule DGen.Registry do defdelegate await_ready(registry_name, timeout), to: :dgen_registry @doc """ - This node's member's own view of the registry, as a map with `:member_id`, + Returns how the member on this node sees the registry, as a map with `:member_id`, `:leader`, `:is_leader`, `:epoch`, `:synced`, and `:applied_version`. - This is the member's *belief*, deliberately distinct from `get_leader/1`'s - committed answer from the elector — a deposed leader that has not yet heard about - the handoff still reports `is_leader: true` here. Side-effect free; returns - `:undefined` if the member is not running. + This is what the member *believes*, which can differ from the committed answer + that `get_leader/1` gets from the elector. For example, a deposed leader that has + not yet heard about the handoff still reports `is_leader: true` here. It has no + side effects. Returns `:undefined` if the member is not running. """ defdelegate status(registry_name), to: :dgen_registry @doc """ Returns the registered name of the member process for the given registry name. - Identity today (`member_name(name) == name`) — the member is registered - under `name` directly. + Today this returns its argument unchanged, because the member is registered + under `name` itself. """ defdelegate member_name(registry_name), to: :dgen_registry @doc """ Returns the ETS table name holding the registry's local `name => pid` replica. - Identity today (`names_table(name) == name`). Exported for introspection; - ordinary callers should use `whereis_name/1` / `get_metadata/1` rather than + Today this returns its argument unchanged. It is exported for introspection. + Ordinary callers should use `whereis_name/1` or `get_metadata/1` rather than reading the table directly. """ defdelegate names_table(registry_name), to: :dgen_registry @doc """ Returns the current elector pid for `registry_name`, or `nil` (`:undefined`) - if the member (and so the elector) cannot currently be found. + if the member, and so the elector, cannot be found right now. - The elector has no registered name; it is discovered via the shared - supervisor (all-local, no network hop). Exported for introspection — used - internally by `get_leader/1`, `get_epoch/1`, and `get_members/1`. + The elector has no registered name. We find it through the supervisor it + shares with the member, which is a local lookup with no network hop. + `get_leader/1`, `get_epoch/1`, and `get_members/1` use it internally, and it is + exported for introspection. """ defdelegate elector_pid(registry_name), to: :dgen_registry end diff --git a/mix.exs b/mix.exs index b086ba2..229b79c 100644 --- a/mix.exs +++ b/mix.exs @@ -47,22 +47,26 @@ defmodule Dgen.MixProject do [ main: "readme", source_url: "https://github.com/foundationdb-beam/dgen", + assets: %{"docs/design/assets" => "assets"}, extras: [ "README.md", "CHANGELOG.md", "LICENSE.md", "docs/design/dgen_server_design.md", "docs/design/dgen_registry_design.md", + "docs/design/dgen_registry_message_flows.md", "docs/getting_started/dgen-intro.livemd", "docs/getting_started/dgen-registry-intro.livemd", "docs/getting_started/presence-demo.livemd", {"formal/README.md", - [filename: "dgen_registry_formal", title: "dgen_registry Formal Model"]} + [filename: "dgen_registry_formal", title: "dgen_registry Formal Model"]}, + {"test/support/sim/dgen_registry_simulation.md", + [filename: "dgen_registry_simulation", title: "dgen_registry Simulation Harness"]} ], groups_for_extras: [ "Getting Started": ~r/getting_started/, Design: ~r/design/, - "Formal Methods": ~r/formal/ + "Formal Methods": ["formal/README.md", "test/support/sim/dgen_registry_simulation.md"] ] ] end diff --git a/src/dgen.erl b/src/dgen.erl index 793a3d0..c63625d 100644 --- a/src/dgen.erl +++ b/src/dgen.erl @@ -2,9 +2,9 @@ -define(DOCATTRS, ?OTP_RELEASE >= 27). -%% Runs *inside the caller's process*, which under simulation is a scheduled one, -%% so it needs the transform for exactly the reason the servers do -- see the note -%% above the include in `dgen_registry`. +%% This code runs inside the caller's process, which the simulator schedules like +%% any other, so it needs the transform for the same reason the servers do. See the +%% note above the include in `dgen_registry`. -include("../include/dgen_eta.hrl"). -if(?DOCATTRS). @@ -13,25 +13,25 @@ %% Internal helpers for the dgen_server call/reply protocol. %% -%% This module implements the client-side call flow: pushing a call request -%% onto the durable queue, waiting for the reply via a watch, and -%% cleaning up reply keys on timeout. +%% This module implements the client side of a call. It pushes the call request +%% onto the durable queue, waits on a watch for the reply, and cleans up the reply +%% keys on timeout. %% -%% Reply payloads are stored using the codec's chunked term encoding so that -%% replies can exceed the single-value size limit. The watch is placed on -%% the first chunk key; the client always reads via `get_range`. +%% We store replies with the codec's chunked term encoding so that a reply can be +%% larger than the backend allows for a single value. The watch sits on the first +%% chunk key, and the client always reads the reply with `get_range`. %% %% ### Call Request Flow %% %% 1. The caller invokes `dgen_server:call/3` which delegates to `call/4`. -%% 2. A call request is pushed onto the durable queue via `push_call/5`, -%% which writes `noreply` as a chunked term under the from-key and sets -%% a watch on the first chunk key. +%% 2. `push_call/6` pushes the call request onto the durable queue. It also +%% writes `noreply` as a chunked term under the `From` key and sets a watch +%% on the first chunk key. %% 3. The caller blocks in `await_call_reply/4` until the watch fires or %% the timeout expires. %% 4. A consumer processes the request and writes `{reply, Reply}` as a %% chunked term under the from-key. -%% 5. The watch fires, the caller reads and clears the chunked reply. +%% 5. The watch fires, and the caller reads and clears the chunked reply. %% 6. On timeout the caller clears the chunked reply keys to avoid leaks. %% %% *This documentation is LLM-generated. See the AI disclosure in README.md.* @@ -61,10 +61,10 @@ push_call(Td, Tuid, Quid, Request, WatchTo, Options) -> -if(?DOCATTRS). -doc """ -Like `push_call/6` but records `Attempts` as the initial attempt count in the +Like `push_call/6`, but records `Attempts` as the initial attempt count in the message envelope. Pass a non-zero value when the call has already failed at -least once (e.g. after an inline execution attempt) so the dead-letter -threshold accounts for all failures. +least once, for example in an inline execution attempt, so that the dead-letter +threshold counts every failure. """. -endif. -spec push_call(term(), tuid(), dgen_queue:quid(), term(), pid(), list(), non_neg_integer()) -> @@ -90,8 +90,8 @@ get_waiting_key(Tuple) -> -doc """ Builds a from-key tuple from a waiting key and a unique ref. -The key includes a system-time timestamp (seconds) so that abandoned -call keys can be garbage-collected using a time-based heuristic. +The key includes the system time in seconds, so that abandoned call keys can +be garbage-collected by age. Key structure: `{WaitingKey..., Timestamp, term_to_binary(Ref)}`. """. -endif. @@ -120,9 +120,9 @@ call(Module, Server, Request, Timeout) -> catch error:{timeout, _} -> % gen_server:call timed out before returning. If push_call - % committed, the reply keys are abandoned (we don't have From). - % They will be cleaned up when the dgen_server is killed, - % which clears the entire waiting-key range. + % committed, its reply keys are abandoned, because we never got + % `From` back. They are cleaned up when the dgen_server is + % killed, which clears the whole waiting-key range. {noreply, {undefined, undefined, undefined}} end, diff --git a/src/dgen_backend.erl b/src/dgen_backend.erl index 1d667ed..5da64f6 100644 --- a/src/dgen_backend.erl +++ b/src/dgen_backend.erl @@ -8,11 +8,11 @@ Behaviour defining the storage backend for dgen. A backend must implement all required callbacks to provide: -- **Transactions** — serializable read-write transactions over the keyspace. -- **Key-value operations** — get, set, clear, range reads, atomic adds. -- **Watches** — notifications when a key changes. -- **Versionstamps** — globally-ordered, unique write stamps for FIFO queuing. -- **Directory layer** — tuple-key packing, range calculation, namespace isolation. +- **Transactions.** Serializable read-write transactions over the keyspace. +- **Key-value operations.** Get, set, clear, range reads, and atomic adds. +- **Watches.** Notifications when a key changes. +- **Versionstamps.** Unique, globally ordered write stamps, used for FIFO queuing. +- **Directory layer.** Tuple-key packing, range calculation, and namespace isolation. The default backend is `dgen_erlfdb`, which delegates to FoundationDB via the `erlfdb` library. @@ -35,9 +35,9 @@ config :dgen, backend: MyBackend ## Implementing a backend -A backend module must implement this behaviour. All watch and future operations -must return `{dgen_future, Ref, BackendData}` tuples where `Ref` is a reference -that will appear in a `{Ref, ready}` message sent to the watching process. +A backend module must implement this behaviour. Every watch and future operation +must return a `{dgen_future, Ref, BackendData}` tuple. `Ref` is a reference, and +the backend later sends `{Ref, ready}` to the watching process. ## Types @@ -89,33 +89,36 @@ when %% Conflict ranges and read-version control. %% -%% `add_read_conflict_key/2` records a read-conflict on `Key` *without* reading -%% it: the commit fails if another transaction wrote `Key` between this -%% transaction's read version and its commit version. Used by the registry to -%% fence a registration commit against a concurrent leadership change without -%% paying a storage read for the leader key's value. +%% `add_read_conflict_key/2` records a read conflict on `Key` without reading it. +%% The commit then fails if another transaction wrote `Key` between this +%% transaction's read version and its commit version. The registry uses this to +%% fence a registration commit against a concurrent leadership change, without +%% paying for a storage read of the leader key. %% -%% `set_read_version/2` pins the read version (e.g. to a prior commit version) -%% so a fenced write-only batch need not pay a GRV; `get_committed_version/1` -%% returns that prior commit version after a successful commit. +%% `set_read_version/2` pins the read version, for example to an earlier commit +%% version, so a fenced write-only batch doesn't have to fetch a new read version. +%% `get_committed_version/1` returns a transaction's commit version after it +%% commits, which the caller can pin on the next batch. -callback add_read_conflict_key(Tx :: tx(), Key :: key()) -> ok. -callback get_read_version(Tx :: tx()) -> future(). -callback set_read_version(Tx :: tx(), Version :: integer()) -> ok. -callback get_committed_version(Tx :: tx()) -> integer(). -%% Transaction lifecycle (explicit, process-model form). +%% Explicit transaction lifecycle. %% -%% `transactional/2` runs and auto-retries a closure; these callbacks instead -%% expose the underlying steps so a caller (`dgen_transaction`) can own a single -%% transaction across messages, decide when to commit, and control retries. +%% `transactional/2` runs a closure and retries it automatically. These callbacks +%% expose the individual steps instead, so a caller such as `dgen_transaction` can +%% keep one transaction open across messages, decide when to commit, and control +%% retries itself. %% %% `create_transaction/1` opens a fresh transaction against a database handle. -%% `commit/1` returns a future; waiting it yields `ok` or **raises** the backend's -%% transaction error. `on_error/2` takes the FDB error code and returns a future -%% whose wait succeeds (after backoff, transaction reset for retry) iff the error -%% is retryable, and raises otherwise. `error_code/1` maps a caught exception -%% reason to its integer Backend error code, or `error` if it is not one — so -%% callers never match the backend-specific error tuple directly. +%% `commit/1` returns a future. Waiting on it returns `ok` or raises the backend's +%% transaction error. `on_error/2` takes the FDB error code and returns a future. +%% If the error is retryable, waiting on that future backs off, resets the +%% transaction for a retry, and succeeds. Otherwise it raises. `error_code/1` maps +%% a caught exception reason to the backend's integer error code, or to `error` if +%% it isn't a backend error. This way callers never match the backend-specific +%% error tuple directly. -callback create_transaction(Db :: db()) -> tx(). -callback commit(Tx :: tx()) -> future(). -callback on_error(Tx :: tx(), ErrorCode :: integer()) -> future(). @@ -131,7 +134,7 @@ when -callback wait(Future :: future()) -> term(). -callback wait_for_all(Futures :: [future()]) -> [term()]. -%% Watches — must return {dgen_future, Ref, BackendData} +%% Watches. These must return `{dgen_future, Ref, BackendData}`. -callback watch(Tx :: tx(), Key :: key()) -> future(). -callback watch(Tx :: tx(), Key :: key(), Opts :: list()) -> future(). diff --git a/src/dgen_config.erl b/src/dgen_config.erl index 616d449..629e302 100644 --- a/src/dgen_config.erl +++ b/src/dgen_config.erl @@ -25,16 +25,17 @@ backend() -> application:get_env(dgen, backend, dgen_erlfdb). -%% The registry tuning knobs below are **per-registry**: each is resolved from the -%% options map passed to `dgen_registry:start_link/3` for that registry, falling back -%% to the `dgen` application environment (a global default), then to a built-in -%% default. So one registry can be strict while another degrades open, with no global -%% coupling. They are documented for end users in `docs/dgen_registry_design.md` -%% (§8 Configuration); section references (e.g. `§5.6`) point there. +%% The registry tuning knobs below are set per registry. Each one is read from the +%% options map passed to `dgen_registry:start_link/3` for that registry. If it isn't +%% there, we fall back to the `dgen` application environment, and then to a built-in +%% default. This lets one registry be strict while another degrades open. The knobs +%% are documented for users in docs/design/dgen_registry_design.md (§8), and section +%% references such as `§5.6` point there. %% %% *This documentation is LLM-generated. See the AI disclosure in README.md.* -%% Resolve a knob: per-registry option > application env > built-in default. +%% Resolves a knob from the per-registry option, then the application env, then +%% the built-in default. -spec get(config(), atom(), term()) -> term(). get(Config, Key, Default) -> case maps:find(Key, Config) of @@ -42,128 +43,132 @@ get(Config, Key, Default) -> error -> application:get_env(dgen, Key, Default) end. -%% Replicate-before-ack target for a *direct* registration (§5.5, §8): the number -%% of distinct follower replicas the leader waits to confirm before acking `yes`. -%% Bounded by the number of followers (with none, the leader acks immediately — -%% nothing survives the sole node anyway). Default 1 (the two-holder invariant). +%% The number of distinct follower replicas that must confirm a direct registration +%% before the leader answers `yes` (§5.5, §8). It is capped at the number of +%% followers. With no followers the leader answers immediately, since nothing would +%% survive the loss of the only node anyway. The default of 1 keeps every name on +%% at least two nodes. -spec register_replicas(config()) -> non_neg_integer(). register_replicas(Config) -> get(Config, register_replicas, 1). -%% Maximum number of write ops (registrations, unregisters, auto-unregisters) the -%% leader coalesces into a single group commit. Ops beyond this ride the following -%% commit. This bounds the *inline* per-commit work — plan, replica apply, and the -%% broadcast fan-out are all O(batch) — so a burst (a node with many names leaving -%% floods the leader with DOWNs) is split across several bounded commits and the -%% leader's message loop stays responsive *between* them, instead of freezing under -%% one enormous batch. Lower it to cap the burst; raise it to coalesce more -%% aggressively (fewer commits, larger inline bursts). Milliseconds of latency, not -%% correctness: the batched outcome equals sequential processing at any size. +%% The most write operations (registrations, unregisters and auto-unregisters) the +%% leader groups into a single commit. Any extra operations go in the next commit. +%% Planning, applying to the replica, and broadcasting all cost O(batch), and they +%% run inside the leader's message loop. When a node with many names leaves, the +%% leader is flooded with DOWNs. The cap splits that burst into several bounded +%% commits, so the leader stays responsive between them instead of freezing on one +%% huge batch. Lower it to cap the burst. Raise it to make fewer, larger commits. +%% This only affects latency. The outcome of a batch is the same as processing its +%% operations one at a time, at any size. -spec commit_batch_size(config()) -> pos_integer(). commit_batch_size(Config) -> get(Config, commit_batch_size, 5000). -%% Caller-side bound (ms) on how long `dgen_registry:register_name/2,3` waits for -%% its verdict before **exiting** with a call timeout (§3 of the design doc): a -%% timeout is not converted to `no` — `no` means the leader adjudicated (taken / -%% no leader / unreachable), a timeout means nothing was adjudicated, and masking -%% it as `no` would hand OTP's via machinery a wrong `already_started` error. -%% Unlike the other knobs this one is resolved in the *calling* process, which has -%% no handle on the per-registry options map — so only the `dgen` application -%% environment and the built-in default apply (no per-registry override). +%% How long, in ms, `dgen_registry:register_name/2,3` waits for an answer before +%% it exits with a call timeout (§3). We don't turn a timeout into `no`. A `no` +%% means the leader decided against the registration. A timeout means nothing +%% was decided, and reporting it as `no` would make OTP's `via` machinery return +%% a wrong `already_started`. +%% Unlike the other knobs, the caller resolves this one in its own process, which +%% has no access to the per-registry options map. So on the caller side only the +%% `dgen` application environment and the built-in default apply. -spec register_timeout(config()) -> pos_integer(). register_timeout(Config) -> get(Config, register_timeout, 5000). -%% How long a direct registration waits for its replica acks before the timeout -%% policy (`strict_replication`) fires. Milliseconds. +%% How long, in ms, a direct registration waits for its replica acks before the +%% `strict_replication` policy decides the outcome. -spec replicate_timeout(config()) -> pos_integer(). replicate_timeout(Config) -> get(Config, replicate_timeout, 1000). -%% Timeout policy for a direct registration that did not gather `register_replicas` -%% acks in time (§8). `false` (default) → **degrade open**: ack `yes` leader-only -%% (uniqueness is already committed; the async broadcast keeps propagating) and emit -%% telemetry. `true` → **fail closed**: reject the registration (`no`) and retract -%% the just-applied binding, so the caller never sees a `yes` that was not replicated. +%% What to do when a direct registration doesn't get `register_replicas` acks in +%% time (§8). With `false`, the default, we degrade open. The leader answers `yes` +%% on its own and emits telemetry. Uniqueness is already committed, and the async +%% broadcast keeps spreading the name. With `true`, we fail closed. The leader +%% answers `no` and removes the binding it just applied, so the caller never sees +%% a `yes` that wasn't replicated. -spec strict_replication(config()) -> boolean(). strict_replication(Config) -> get(Config, strict_replication, false). -%% Partition-case *prevention* (§5.6). When `true`, a leader whose most recent -%% handoff gather was **incomplete** (a member in the committed member set did not -%% respond — it is unreachable) refuses to register a name it does not currently hold, -%% since that name might be held by the unreachable member and registering it would be -%% a re-issue. `false` (default) keeps the reactive posture: re-issues during a -%% partition are caught at heal by §5.6 termination instead of prevented. +%% Prevents duplicate names during a partition instead of cleaning them up later +%% (§5.6). Sometimes a member in the committed member set doesn't respond to the +%% leader's handoff gather, because it is unreachable. When this is `true` and the +%% last gather was incomplete like that, the leader refuses to register any name it +%% doesn't already hold. The unreachable member might hold that name, and +%% registering it again would create a duplicate. With `false`, the default, we +%% allow those registrations and rely on §5.6 termination to resolve duplicates +%% when the partition heals. %% -%% This is deliberately blunt: with no durable taken-set the leader cannot tell a -%% genuine re-issue from a fresh name, so it rejects *all* new names while degraded — -%% and it cannot tell a partition from a legitimate scale-down, so a member that -%% leaves and never returns keeps the leader degraded until the cluster is whole -%% again. Hence default `false`; enable it only where fail-closed-during-partition is -%% preferred over availability. +%% This is deliberately blunt. The leader has no durable record of taken names, so +%% it can't tell a real duplicate from a fresh name, and it rejects every new name +%% while degraded. It also can't tell a partition from a planned scale-down. A +%% member that leaves and never returns keeps the leader degraded until the cluster +%% is whole again. That is why the default is `false`. Enable it only where failing +%% closed during a partition matters more than availability. -spec reject_when_degraded(config()) -> boolean(). reject_when_degraded(Config) -> get(Config, reject_when_degraded, false). -%% Registry conflict termination (§5.6). When `true` (default), a confirmed -%% divergence — two different live pids holding one name, observed at a handoff -%% gather — forcibly terminates the conflicting processes (kill-both), broadcasts an -%% unregister, and alarms. Set `false` to detect + alarm only (no kill), leaving the -%% uniqueness hole open until an operator intervenes. Termination is the load-bearing -%% uniqueness backstop now that the per-name occupancy set is gone (§4.4), so it -%% defaults on. +%% Registry conflict termination (§5.6). A conflict is two different live pids +%% holding one name, found during a handoff gather. When this is `true`, the +%% default, the leader kills both processes, broadcasts an unregister, and raises +%% an alarm. With `false` it only raises the alarm, and the duplicate stays until +%% an operator steps in. The registry no longer keeps a per-name occupancy set +%% (§4.4), so termination is what ultimately guarantees uniqueness. That is why it +%% is on by default. -spec terminate_on_conflict(config()) -> boolean(). terminate_on_conflict(Config) -> get(Config, terminate_on_conflict, true). -%% Kill budget for conflict termination: at most `Count` kills per name per -%% `WindowMs`, after which the registry stops killing that name and escalates to an -%% operator (alarm only), so a bug that regenerates a conflict cannot loop. +%% Limits conflict termination to `Count` kills per name in each `WindowMs`. Past +%% that, the registry stops killing for that name and only raises an alarm for an +%% operator. This stops a bug that keeps recreating a conflict from looping forever. %% -%% Scope caveat: the budget is tracked in the *current leader's* process state and -%% is not merged across a handoff (unlike the release trail, §5.6) — a leadership -%% change starts the new leader with a fresh budget for every name. The bound is -%% therefore per-name *per leader term*, not global; a conflict that regenerates -%% across failovers can be killed up to `Count` times per term. +%% The budget lives in the current leader's process state. Unlike the release trail +%% (§5.6), it isn't carried over in a handoff, so a new leader starts with a fresh +%% budget for every name. The limit is therefore per name and per leader term, not +%% global. A conflict that keeps coming back across failovers can be killed up to +%% `Count` times in each term. -spec conflict_kill_budget(config()) -> {pos_integer(), pos_integer()}. conflict_kill_budget(Config) -> get(Config, conflict_kill_budget, {3, 60000}). -%% How long (ms) a pid explicitly released while alive (an unregister of a live -%% process) stays in the conflict-detector trail (§5.6). The trail is what stops -%% the detector from kill-both-ing a legitimately unregistered-then-re-registered -%% name when a lagging member reports the old binding at a handoff gather, so it -%% must comfortably outlast realistic disconnect windows — pruning is additionally -%% suspended while any member is disconnected. Entries are tiny (pid + timestamp) -%% and only explicit live releases create them, so a generous default is cheap. +%% How long, in ms, the conflict detector remembers a live pid that was explicitly +%% unregistered (§5.6). Suppose a name is unregistered and then registered to a new +%% pid, and a lagging member still reports the old binding at a handoff gather. The +%% trail is what tells the detector this isn't a conflict, so it doesn't kill both +%% pids. The TTL must comfortably outlast realistic disconnects, and we also pause +%% pruning while any member is disconnected. Entries are tiny (a pid and a +%% timestamp), and only explicit unregisters of live pids create them, so a +%% generous default is cheap. -spec conflict_release_ttl(config()) -> pos_integer(). conflict_release_ttl(Config) -> get(Config, conflict_release_ttl, 600000). -%% Who is responsible for keeping the Erlang-distribution mesh in step with this +%% Decides who keeps Erlang distribution connected to every node in this %% registry's membership (§4.6, §8). %% -%% `self_managed` (default) — the registry's connector runs the proactive mesh: it -%% periodically reads the authoritative member set and opens a distribution -%% connection to every member node, so `nodes()` converges without any external -%% discovery. Every registry is self-sufficient. +%% `self_managed` (default). The registry's connector does it. It periodically +%% reads the authoritative member set and connects to every member node, so +%% `nodes()` converges without any external discovery. Each registry looks +%% after itself. %% -%% `provided_externally` — the connector does **not** mesh: it opens no distribution -%% connections and does not read the member set on a timer. The registry assumes -%% the connections it needs are established by *something else on the node* — -%% typically another `self_managed` "system" registry whose member set spans a -%% superset of this registry's nodes (distribution links are node-global, so a -%% tenant registry free-rides on them). The per-registry liveness backstops (the -%% leader-liveness probe, the stranded-member reap, the durable-epoch nudge) stay -%% active — they are registry-scoped and cannot be delegated. +%% `provided_externally`. The connector opens no distribution connections and +%% doesn't poll the member set. The registry assumes something else on the +%% node sets up the connections it needs. Usually that is another +%% `self_managed` "system" registry whose members cover all of this registry's +%% nodes. Distribution links are shared by the whole node, so a tenant registry +%% can rely on them. The per-registry liveness checks stay active: the +%% leader-liveness probe, the stranded-member reap, and the durable-epoch nudge. +%% Each belongs to one registry and can't be handed off to another. %% -%% Only `provided_externally` disables meshing; any other value (including an -%% unrecognised one) resolves to `self_managed`, so a typo fails *safe* — toward doing -%% the connectivity work rather than silently isolating the node. See §4.6 for the -%% contract this mode places on the deployer and the isolation it risks if that -%% contract is broken. +%% Only `provided_externally` turns the connections off. Any other value, including +%% a typo, resolves to `self_managed`. That way a mistake errs toward connecting +%% rather than silently isolating the node. See §4.6 for what this mode requires of +%% the deployer and how the node can end up isolated if that isn't met. -spec connectivity(config()) -> self_managed | provided_externally. connectivity(Config) -> case get(Config, connectivity, self_managed) of diff --git a/src/dgen_frac_index.erl b/src/dgen_frac_index.erl index 6950e18..38c8d41 100644 --- a/src/dgen_frac_index.erl +++ b/src/dgen_frac_index.erl @@ -8,10 +8,10 @@ %% Fractional indexing for ordered sequences. %% -%% Generates lexicographically-sortable binary strings using base-62 digits -%% (0-9, A-Z, a-z in ASCII order). A new key can always be generated between -%% any two existing keys, making this suitable for maintaining ordered lists -%% in key-value stores without reindexing. +%% Generates binary strings that sort lexicographically, using base-62 digits +%% (0-9, A-Z, a-z in ASCII order). We can always generate a new key between any +%% two existing keys, so an ordered list in a key-value store never needs +%% reindexing. %% %% *This documentation is LLM-generated. See the AI disclosure in README.md.* @@ -72,7 +72,7 @@ n_first(N) when is_integer(N), N > 0 -> n_between(<<>>, <<>>, N). -if(?DOCATTRS). --doc "Returns `N` evenly-spaced keys between `A` and `B`. Use `<<>>` for unbounded start/end.". +-doc "Returns `N` evenly-spaced keys between `A` and `B`. Pass `<<>>` to leave either end unbounded.". -endif. -spec n_between(binary(), binary(), non_neg_integer()) -> [key()]. n_between(_A, _B, 0) -> diff --git a/src/dgen_mem.erl b/src/dgen_mem.erl index 4ce38c5..dfe1af9 100644 --- a/src/dgen_mem.erl +++ b/src/dgen_mem.erl @@ -1,9 +1,9 @@ -module(dgen_mem). -behaviour(dgen_backend). -%% Simulation build only: rewrites the `erlang:monotonic_time/1` calls below to -%% `eta_time`, so commit versions advance with the virtual clock when one is -%% running. Outside a simulation build this module has no `eta` dependency at all. +%% In a simulation build, this rewrites the `erlang:monotonic_time/1` calls below to +%% use `eta_time`, so commit versions advance with the virtual clock when one is +%% running. Outside a simulation build this module doesn't depend on `eta` at all. -include("../include/dgen_eta.hrl"). -define(DOCATTRS, ?OTP_RELEASE >= 27). @@ -12,72 +12,74 @@ -moduledoc """ A pure-Erlang, in-memory `dgen_backend`. -Implements the FoundationDB semantics the registry actually depends on — MVCC read -versions, read-conflict detection, versionstamps, watches — with no NIF, no server, -and no I/O. Two things follow: +It implements the FoundationDB semantics the registry actually depends on: MVCC +read versions, read-conflict detection, versionstamps, and watches. It needs no +NIF, no server, and no I/O. That gives us two things. -- **The suite runs without FoundationDB installed.** Worth having on its own. -- **Commit timing stops being a source of nondeterminism**, which is the last one +- **The suite runs without FoundationDB installed.** That is worth having on its own. +- **Commit timing is no longer a source of nondeterminism.** It was the last one outside `eta_sched` and `eta_time`. Commits happen synchronously, in the calling process, in a deterministic order. ## Fidelity, and where it stops -Key layout is *exactly* FDB's: `erlfdb_subspace` and `erlfdb_tuple` are pure Erlang, -so this reuses them rather than reimplementing order-preserving tuple encoding, -where a subtle error would surface as ordering bugs in `dgen_queue`. +Key layout is exactly the same as FDB's. `erlfdb_subspace` and `erlfdb_tuple` are +pure Erlang, so we reuse them instead of reimplementing order-preserving tuple +encoding. A subtle error there would show up as ordering bugs in `dgen_queue`. What is modelled faithfully: - **Read versions.** A read sees the newest value written at or before the - transaction's read version. Pinning a read version (`set_read_version/2`) really - does read the past, which is what makes the registry's fencing test meaningful. + transaction's read version. Pinning an older read version with + `set_read_version/2` really does read the past. That is what makes the registry's + fencing test meaningful. - **Read conflicts.** A commit aborts with `not_committed` (1020) if any key in a - read-conflict range was written after the transaction's read version. This is the - mechanism the leader fence rests on (§5.1), so it has to be real. -- **Versionstamps.** `set_versionstamped_value/3` substitutes a 10-byte stamp — - 8-byte commit version, 2-byte in-transaction batch order — so the registry's - version key decodes and orders exactly as it does against FDB. + read-conflict range was written after the transaction's read version. The leader + fence depends on this (§5.1), so it has to be real. +- **Versionstamps.** `set_versionstamped_value/3` fills in a 10-byte stamp, made of + an 8-byte commit version and a 2-byte batch order within the transaction. The + registry's version key then decodes and sorts exactly as it does against FDB. - **Watches.** A watch fires when its key is written by a later commit. What is not: - **No storage-level partitioning, no coordinators, no real distribution.** A - simulation runs in one VM, so the database is a single serialisation point rather - than a cluster that can partition beneath the system under test. -- **Write conflicts are not tracked.** FDB conflicts a commit whose *write* set - overlaps another transaction's read set. Here commits are serialised, so the - losing transaction is simply the one that commits second, and it sees the other's - writes. Nothing in `dgen` depends on the distinction. -- **No transaction *size* limit** (`transaction_too_large`). Nothing in `dgen` - approaches FoundationDB's 10MB transaction cap. -- **No retry backoff.** FoundationDB backs off exponentially with jitter; - `transactional/2` here retries at once, up to 100 attempts. A `timer:sleep/1` - would be wrong under a virtual clock — a sleeper is neither runnable nor blocked - on a receive, so `eta_sched` can never step it. + simulation runs in one VM, so the database is a single point where every commit + is serialised. It is not a cluster that can partition underneath the system + under test. +- **Write conflicts are not tracked.** FDB fails a commit whose write set overlaps + another transaction's read set. Here commits are serialised, so the losing + transaction is simply the one that commits second, and it sees the other's + writes. Nothing in `dgen` depends on the difference. +- **No transaction size limit** (`transaction_too_large`). Nothing in `dgen` comes + near FoundationDB's 10MB transaction cap. +- **No retry backoff.** FoundationDB backs off exponentially with jitter. Here + `transactional/2` retries at once, up to 100 attempts. A `timer:sleep/1` would be + wrong under a virtual clock. A sleeping process is neither runnable nor blocked + on a receive, so `eta_sched` could never step it. **The MVCC window is modelled.** Versions advance at FoundationDB's rate -(`VERSIONS_PER_SECOND` = 1e6) and the window is the same version distance -(`MAX_READ_TRANSACTION_LIFE_VERSIONS`, 5 seconds' worth), so a read older than the -window raises `transaction_too_old` as it would against the real thing — and under -a virtual clock those five seconds are reachable instantly. +(`VERSIONS_PER_SECOND` = 1e6), and the window is the same version distance +(`MAX_READ_TRANSACTION_LIFE_VERSIONS`, 5 seconds' worth). A read older than the +window raises `transaction_too_old`, as it would against the real thing. Under a +virtual clock, those five seconds pass instantly. -The version rate follows whatever clock `eta_time` exposes, so start the clock -before opening the database; doing it part-way through shifts the origin the rate -is measured from. +The version rate follows whatever clock `eta_time` exposes. Start the clock before +opening the database. Starting it later shifts the origin we measure the rate from. ## Fault injection -`set_faults/2` makes the failures the registry is built to survive reproducible: -`conflict_p` (spurious `not_committed`, which a correct caller retries) and -`commit_fail_p` (a non-retryable error). Both draw from a seeded RNG, so a run -replays. +`set_faults/2` injects the failures the registry is built to survive, in a way +that can be reproduced. `conflict_p` causes spurious `not_committed` errors, which a +correct caller retries. `commit_fail_p` causes a non-retryable error. Both draw from +a seeded RNG, so a run can be replayed. ## Determinism -State lives in ETS keyed by the database handle and nothing runs in a process of -its own, so no call into this module can read as quiescence to `eta_sched`. Commit -versions come from a counter, not a clock. +State lives in ETS, keyed by the database handle, and nothing runs in a process of +its own. So no call into this module can look like quiescence to `eta_sched`. +Commit versions come from the clock that `eta_time` exposes, so they are +deterministic under a virtual clock. *This documentation is LLM-generated. See the AI disclosure in `README.md`.* """. @@ -122,23 +124,24 @@ versions come from a counter, not a clock. -export_type([db/0, fault_opts/0]). -%% FDB error codes. The retryable set is what `on_error/2` resets for, and it has -%% to match FoundationDB's — a code missing from it turns a recoverable error into -%% a fatal one, which is exactly the kind of infidelity that makes a simulation -%% backend worse than useless. +%% FDB error codes. `on_error/2` resets the transaction for the retryable set, and +%% that set has to match FoundationDB's. A missing code turns a recoverable error +%% into a fatal one, and that kind of mismatch makes a simulation backend worse +%% than useless. -define(TRANSACTION_TOO_OLD, 1007). -%% Reading a key whose value this transaction has already made unreadable — a -%% pending versionstamped write, whose value is not known until commit. +%% Raised when a transaction reads a key it has a pending versionstamped write to. +%% That value isn't known until commit. -define(ACCESSED_UNREADABLE, 1036). -%% FoundationDB advances the read version at a fixed rate rather than per commit, -%% and expresses its MVCC window as a *version distance*: `VERSIONS_PER_SECOND` is -%% 1e6, and `MAX_READ_TRANSACTION_LIFE_VERSIONS` is 5 seconds' worth. Reading at a -%% version older than that raises `transaction_too_old`. +%% FoundationDB advances the read version at a fixed rate rather than once per +%% commit, and it defines its MVCC window as a distance in versions. +%% `VERSIONS_PER_SECOND` is 1e6, and `MAX_READ_TRANSACTION_LIFE_VERSIONS` is 5 +%% seconds' worth. Reading at a version older than that raises +%% `transaction_too_old`. %% -%% Modelling it the same way costs nothing and buys the behaviour outright — and -%% under a virtual clock it becomes *controllable*: five seconds of MVCC window is -%% five simulated seconds, reachable instantly, instead of a six-second sleep. +%% Modelling it the same way costs nothing and gets us the same behaviour. Under a +%% virtual clock it also becomes controllable. Five seconds of MVCC window are five +%% simulated seconds, which pass instantly instead of needing a six-second sleep. -define(VERSIONS_PER_MS, 1000). -define(MAX_READ_TRANSACTION_LIFE_VERSIONS, 5000000). -define(NOT_COMMITTED, 1020). @@ -180,8 +183,8 @@ open(Opts = #{name := Name}) -> _ = ets:new(watch_tab(Name), [named_table, public, bag]), Seed = maps:get(seed, Opts, 0), ets:insert(meta_tab(Name), [ - %% Starts above zero so a pinned read version of 0 is meaningfully "before - %% everything", matching how the registry treats an unwritten version key. + %% Starts above zero so that a pinned read version of 0 really is "before + %% everything". That matches how the registry treats an unwritten version key. {version, 1}, %% Versions advance with the clock (see ?VERSIONS_PER_MS), so the database %% needs an origin to measure from. Under the transform this reads the @@ -202,11 +205,11 @@ open(Opts = #{name := Name}) -> close({dgen_mem_db, Name}) -> lists:foreach( fun(Tab) -> - %% Tables are owned by whichever process opened the database, so the VM - %% destroys them when it exits. A checked delete would race that: the - %% table can vanish between the existence check and the delete, which - %% is exactly what an ExUnit `on_exit` — running after the test process - %% is gone — hits. Deleting something already deleted is a no-op here. + %% The process that opened the database owns the tables, so the VM + %% destroys them when that process exits. Checking first and then + %% deleting would race with that, because the table can vanish in + %% between. An ExUnit `on_exit` hits exactly this, since it runs after + %% the test process is gone. So we just delete, and ignore a missing table. try ets:delete(Tab) catch @@ -229,11 +232,11 @@ reset({dgen_mem_db, Name}) -> -doc """ Sets the commit fault policy. -- `conflict_p` — probability a commit is rejected with `not_committed` (1020) even - though nothing actually conflicted. Retryable, so a correct caller recovers; the - point is to exercise that it does. -- `commit_fail_p` — probability a commit fails with a non-retryable error, which - must surface to the caller rather than be retried away. +- `conflict_p` is the probability that a commit is rejected with `not_committed` + (1020) even though nothing conflicted. The error is retryable, so a correct caller + recovers. The point is to check that it does. +- `commit_fail_p` is the probability that a commit fails with a non-retryable + error. That error must reach the caller rather than being retried away. """. -endif. -spec set_faults(db(), fault_opts()) -> ok. @@ -242,7 +245,7 @@ set_faults({dgen_mem_db, Name}, Faults) -> ok. -if(?DOCATTRS). --doc "Commit count, conflict count, current version, and key count.". +-doc "Returns the commit count, conflict count, current version, and key count.". -endif. -spec stats(db()) -> #{atom() => non_neg_integer()}. stats({dgen_mem_db, Name}) -> @@ -260,9 +263,10 @@ watch_tab(Name) -> list_to_atom("dgen_mem_watch_" ++ atom_to_list(Name)). meta(Name, Key) -> ets:lookup_element(meta_tab(Name), Key, 2). -%% The current read version: whichever is further ahead, the last commit or the -%% clock. Commits therefore never go backwards, and an idle database still advances -%% — which is what makes the MVCC window mean elapsed time rather than commit count. +%% Returns the current read version, which is the last commit version or the clock +%% version, whichever is further ahead. So versions never go backwards, and an idle +%% database still advances. That is what makes the MVCC window measure elapsed time +%% rather than a number of commits. current_version(Name) -> max(meta(Name, version), clock_version(Name)). @@ -270,8 +274,8 @@ clock_version(Name) -> Elapsed = erlang:monotonic_time(millisecond) - meta(Name, start_ms), max(0, Elapsed) * ?VERSIONS_PER_MS. -%% FoundationDB's transaction_too_old: a read at a version the storage servers no -%% longer retain. +%% Raises FoundationDB's `transaction_too_old` for a read at a version the storage +%% servers would no longer keep. check_not_too_old(Name, RV) -> case current_version(Name) - RV > ?MAX_READ_TRANSACTION_LIFE_VERSIONS of true -> erlang:error({erlfdb_error, ?TRANSACTION_TOO_OLD}); @@ -295,11 +299,12 @@ create_transaction({dgen_mem_db, Name}) -> new_tx_state() -> #{ - %% `undefined` means "assign lazily on first read or at commit", which is - %% how FDB's GRV works — a transaction that only writes never pays for one. + %% `undefined` means we assign it on the first read or at commit. FDB works + %% the same way, so a transaction that only writes never fetches a read + %% version. read_version => undefined, - %% Mutations in issue order. clear_range and set interleave, so a map keyed - %% by key would lose the ordering that decides the outcome. + %% Mutations, newest first. Sets and clear_ranges interleave, and their order + %% decides the outcome, so a map keyed by key would lose information. muts => [], read_conflicts => [], committed_version => undefined, @@ -347,9 +352,9 @@ get_committed_version(Tx) -> -if(?DOCATTRS). -doc """ -Runs `Fun` in a transaction, retrying on retryable errors — the same contract as -`erlfdb:transactional/2`. A transaction handle passed in is used directly, so -nesting composes. +Runs `Fun` in a transaction and retries it on retryable errors. This is the same +contract as `erlfdb:transactional/2`. If you pass in a transaction handle, it is +used directly, so nested calls work. """. -endif. -spec transactional(db() | tx(), fun((tx()) -> Result)) -> Result when Result :: term(). @@ -381,8 +386,8 @@ get(Tx = {dgen_mem_tx, Name, _}, Key) -> RV = read_version(Tx), check_not_too_old(Name, RV), add_read_conflict_range(Tx, Key, key_strinc(Key)), - %% Read-your-writes: the transaction's own uncommitted mutations shadow the - %% store, as they do in FDB. + %% The transaction reads its own writes. Its uncommitted mutations take + %% precedence over the store, as they do in FDB. case pending_value(tx_state(Tx), Key) of {ok, Value} -> ready(Value); cleared -> ready(not_found); @@ -408,8 +413,8 @@ get_range(Tx = {dgen_mem_tx, Name, _}, StartKey, EndKey, Opts) -> Limit -> lists:sublist(Ordered, Limit) end. -%% Every live key in [StartKey, EndKey) as of `RV`, with the transaction's own -%% mutations overlaid. +%% Returns every live key in [StartKey, EndKey) as of `RV`, with the transaction's +%% own mutations applied on top. range_rows(Name, StartKey, EndKey, RV, S) -> Stored = [ {K, V} @@ -431,10 +436,10 @@ merge_rows(Stored, Pending) -> Cleared = [K || {K, cleared} <- Pending], lists:keysort(1, [{K, V} || {K, V} <- Merged, not lists:member(K, Cleared)]). -%% Every stored key in [StartKey, EndKey). `ets:next/2` gives the first key -%% *strictly greater* than its argument, so the start key has to be checked for -%% membership separately — omitting that silently drops it from every range, which -%% breaks single-key clear_range and, more quietly, read-conflict detection on a +%% Returns every stored key in [StartKey, EndKey). `ets:next/2` returns the first +%% key strictly greater than its argument, so we check the start key separately. +%% Without that check the start key would silently drop out of every range. That +%% breaks single-key clear_range and, less visibly, read-conflict detection on a %% key that exists. keys_in_range(Name, StartKey, EndKey) -> Tab = data_tab(Name), @@ -452,8 +457,8 @@ keys_from(Tab, Key, EndKey, Acc) when Key < EndKey -> keys_from(_Tab, _Key, _EndKey, Acc) -> lists:reverse(Acc). -%% The newest version at or before `RV`; `not_found` if the key did not exist then -%% (or was cleared). +%% Returns the newest value at or before `RV`. Returns `not_found` if the key didn't +%% exist then or had been cleared. stored_value(Name, Key, RV) -> case ets:lookup(data_tab(Name), Key) of [] -> @@ -468,18 +473,18 @@ stored_value(Name, Key, RV) -> value_or_not_found(cleared) -> not_found; value_or_not_found(Value) -> Value. -%% What this transaction's own mutations say about a key. +%% Returns what this transaction's own mutations say about a key. pending_value(#{muts := Muts}, Key) -> lists:foldl( fun ({set, K, V}, _Acc) when K =:= Key -> {ok, V}; - %% A pending versionstamped write makes the key *unreadable* for the - %% rest of the transaction: its value is not decided until commit - %% assigns the stamp. FoundationDB raises accessed_unreadable (1036) - %% here, and matching that matters more than it might seem — falling - %% through to the last committed value instead returns a stale answer - %% silently, which is how a caller comparing a cached version against - %% this key gets a spurious match and serves stale state. + %% A pending versionstamped write makes the key unreadable for the + %% rest of the transaction, because its value isn't decided until the + %% commit assigns the stamp. FoundationDB raises accessed_unreadable + %% (1036) here, and matching that matters. If we fell through to the + %% last committed value instead, we would silently return a stale + %% answer. A caller comparing a cached version against this key would + %% then get a false match and serve stale state. ({vs_value, K, _}, _Acc) when K =:= Key -> unreadable; ({clear_range, S, E}, _Acc) when Key >= S, Key < E -> cleared; (_, Acc) -> Acc @@ -536,17 +541,16 @@ stage(Tx, Mut) -> add_read_conflict_key(Tx, Key) -> add_read_conflict_range(Tx, Key, key_strinc(Key)). -%% Every non-snapshot read records a read conflict, because that is what -%% FoundationDB does — and it is load-bearing far more often than the explicit +%% Every non-snapshot read records a read conflict, because FoundationDB does the +%% same. Far more code depends on this than the few explicit %% `add_read_conflict_key/2` calls suggest. %% -%% Across all of `dgen` there is exactly *one* explicit conflict: the registry's -%% leader fence. Everything else relies on the implicit ones — most importantly -%% `dgen_queue:peek_k/3`, where two consumers read the same pending items and the -%% conflict is what stops both from consuming them. Without this, a backend offers -%% no serializable isolation for any read-then-write, and the failures it produces -%% (a double-consumed message, a lost increment) look nothing like a missing -%% conflict range. +%% Across all of `dgen` there is exactly one explicit conflict, the registry's +%% leader fence. Everything else relies on the implicit ones. The most important is +%% `dgen_queue:peek_k/3`. Two consumers can read the same pending items, and the +%% conflict is what stops both of them from consuming them. Without it, no +%% read-then-write would be serializable. The resulting failures, such as a message +%% consumed twice or a lost increment, look nothing like a missing conflict range. -spec add_read_conflict_range(tx(), dgen_backend:key(), dgen_backend:key()) -> ok. add_read_conflict_range(Tx, StartKey, EndKey) -> update_tx(Tx, fun(S = #{read_conflicts := Rs}) -> @@ -577,13 +581,13 @@ commit(Tx = {dgen_mem_tx, Name, _}) -> {error, Code} -> erlang:error({erlfdb_error, Code}) end. -%% Runs under the commit lock. Raising from here would escape before the lock's -%% `after` clause has run in some orderings, so failures come back as a value and -%% are raised by the caller once the lock is released. +%% Runs under the commit lock. In some orderings, raising here would escape before +%% the lock's `after` clause has run. So we return failures as a value, and the +%% caller raises them once the lock is released. try_commit(_Tx, _Name, #{muts := []}, _RV) -> - %% A read-only transaction commits trivially and can never conflict — FDB - %% treats it as a no-op, and now that every read records a conflict range, - %% checking them here would fail read-only transactions for no reason. + %% A read-only transaction always commits and can never conflict. FDB treats + %% it as a no-op. Every read records a conflict range, so checking them here + %% would fail read-only transactions for no reason. ok; try_commit(Tx, Name, S, RV) -> case commit_fault(Name) of @@ -602,14 +606,15 @@ try_commit(Tx, Name, S, RV) -> end. do_commit(Tx = {dgen_mem_tx, _, TxRef}, Name, S) -> - %% Strictly ahead of the last commit *and* of the clock — that is, of - %% `current_version/1`, which is what every read version already handed out was - %% drawn from. Equality there is not a rounding detail: both `has_conflict/3` - %% and `watch/3` ask whether a key was written *after* a read version, so a - %% commit landing exactly on one is a conflict that is not detected and a watch - %% that never fires. `max(Version + 1, Clock)` allowed exactly that whenever the - %% clock was ahead of the commit counter, which is most of the time. - %% Runs under the commit lock, so the read-modify-write is safe. + %% The commit version must be strictly ahead of both the last commit and the + %% clock. In other words, it must be ahead of `current_version/1`, which every + %% read version handed out so far came from. Landing exactly on a read version + %% is a real bug, not a rounding detail. Both `has_conflict/3` and `watch/3` ask + %% whether a key was written after a read version. A commit at that same + %% version would be a conflict we miss and a watch that never fires. An earlier + %% `max(Version + 1, Clock)` did exactly that whenever the clock was ahead of + %% the commit counter, which is most of the time. This runs under the commit + %% lock, so the read-modify-write is safe. Version = current_version(Name) + 1, ets:insert(meta_tab(Name), {version, Version}), Stamp = versionstamp(Version, 0), @@ -619,23 +624,23 @@ do_commit(Tx = {dgen_mem_tx, _, TxRef}, Name, S) -> fire_watches(Name, Written, TxRef), ok. -%% Serialise the commit critical section. +%% Runs `Fun` while holding the commit lock, so commits happen one at a time. %% %% Checking the conflict ranges, assigning a version, and applying the mutations -%% have to be one indivisible step. FoundationDB serialises commits; without the -%% same guarantee here a second transaction slips into the gap and three things -%% break, all of them quietly: +%% have to be one indivisible step. FoundationDB serialises commits. Without the +%% same guarantee here, a second transaction can slip into the gap, and three +%% things break quietly. %% -%% - a conflict is missed, so a transaction that should have been fenced commits -%% — which is precisely the safety property the registry's leader fence rests -%% on (§5.1); -%% - `write/4`'s read-modify-write of a key's version list loses an entry; -%% - versions land out of order in that list, and `stored_value/3` takes the head -%% of the matching entries, so a read returns an *older* value than it should. +%% - A conflict is missed, so a transaction that should have been fenced +%% commits. The registry's leader fence depends on exactly this (§5.1). +%% - The read-modify-write of a key's version list in `write/4` loses an entry. +%% - Versions land out of order in that list. `stored_value/3` takes the first +%% matching entry, so a read returns an older value than it should. %% -%% An ETS lock rather than a serialising process: a blocking call into a process -%% reads as quiescence to `eta_sched`, so every commit would end a scheduling step. -%% The critical section is pure in-memory work, so spinning is cheap. +%% We use an ETS lock rather than a process that serialises commits. A blocking +%% call into a process looks like quiescence to `eta_sched`, so every commit would +%% end a scheduling step. The critical section is pure in-memory work, so spinning +%% is cheap. with_commit_lock(Name, Fun) -> Tab = meta_tab(Name), case ets:insert_new(Tab, {commit_lock, self()}) of @@ -650,14 +655,14 @@ with_commit_lock(Name, Fun) -> with_commit_lock(Name, Fun) end. -%% A lock holder can die without releasing: `try ... after` does not run when a -%% process is killed untrappably, and killing a process mid-flight is routine here -%% — a simulated member crash is exactly that. Left alone, one such death wedges -%% every subsequent commit in the VM, which presents as the whole suite hanging in -%% an unrelated place. +%% A lock holder can die without releasing the lock. `try ... after` doesn't run +%% when a process is killed with an untrappable exit, and that happens routinely +%% here, since a simulated member crash is exactly that. If we did nothing, one such +%% death would block every later commit in the VM. It would look like the whole +%% suite hanging somewhere unrelated. %% -%% `delete_object/2` rather than `delete/2` so the reclaim can only remove the -%% *dead* holder's lock, never one a live process has just taken. +%% We use `delete_object/2` rather than `delete/2` so that we can only remove the +%% dead holder's lock, never one a live process has just taken. reclaim_or_wait(Tab) -> case ets:lookup(Tab, commit_lock) of [{commit_lock, Holder}] -> @@ -669,8 +674,8 @@ reclaim_or_wait(Tab) -> ok end. -%% A read conflict is a key in a read-conflict range written after our read version -%% — precisely the check the registry's leader fence relies on (§5.1). +%% Returns whether any key in a read-conflict range was written after our read +%% version. The registry's leader fence relies on this check (§5.1). has_conflict(_Name, [], _RV) -> false; has_conflict(Name, Ranges, RV) -> @@ -736,14 +741,14 @@ add_to(Current, N) -> <> = Padded, <<((Cur + N) band ((1 bsl (Size * 8)) - 1)):(Size * 8)/little-unsigned>>. -%% A 10-byte FDB versionstamp: 8-byte big-endian commit version, 2-byte batch -%% order. The registry decodes this as a big-endian integer and shifts the batch -%% bytes off to recover the commit version, so the layout has to match. +%% Builds a 10-byte FDB versionstamp from an 8-byte big-endian commit version and a +%% 2-byte batch order. The registry decodes it as a big-endian integer and shifts +%% off the batch bytes to get the commit version, so the layout has to match. versionstamp(Version, Batch) -> <>. -%% A versionstamped value carries a 10-byte placeholder plus a 4-byte little-endian -%% offset saying where it sits. +%% A versionstamped value carries a 10-byte placeholder, followed by a 4-byte +%% little-endian offset that says where the placeholder is. substitute_stamp(Value, Stamp) when byte_size(Value) >= 14 -> Body = binary:part(Value, 0, byte_size(Value) - 4), <> = binary:part(Value, byte_size(Value) - 4, 4), @@ -755,13 +760,15 @@ substitute_stamp(Value, _Stamp) -> -if(?DOCATTRS). -doc """ -The transaction's versionstamp, as a future that resolves **after** it commits. - -Laziness is not an optimisation here, it is the contract. Callers capture this -future *before* committing and wait it afterwards — `set_mod_state/4` in `dgen_server` -does exactly that to version its state cache. Resolving eagerly hands back the -`undefined` that exists before a commit version has been assigned, and the cache -then keys itself on a value that never matches, or worse, spuriously does. +Returns the transaction's versionstamp as a future that resolves **after** the +transaction commits. + +Resolving late isn't an optimisation here. It is the contract. Callers take this +future before committing and wait on it afterwards. `set_mod_state/4` in +`dgen_server` does exactly that to version its state cache. If we resolved it +early, we would return `undefined`, because no commit version exists yet. The cache +would then key itself on a value that never matches, or worse, one that matches by +accident. """. -endif. -spec get_versionstamp(tx()) -> dgen_backend:future(). @@ -781,10 +788,10 @@ error_code(_) -> error. -if(?DOCATTRS). -doc """ -Resets a transaction for retry when `Code` is retryable, mirroring -`erlfdb:on_error/2`: the returned future succeeds for a retryable error and raises -otherwise. A reset drops staged mutations and conflict ranges, cancels the -transaction's watches, and takes a fresh read version. +Resets a transaction for a retry when `Code` is retryable, like +`erlfdb:on_error/2`. For a retryable error it returns a future that succeeds. +Otherwise it raises. A reset drops staged mutations and conflict ranges, cancels +the transaction's watches, and takes a fresh read version. """. -endif. -spec on_error(tx(), integer()) -> dgen_backend:future(). @@ -820,8 +827,8 @@ roll(Name) -> %% Futures and watches %% --------------------------------------------------------------------------- -%% Every read here is already resolved, so a future is just a box. Watches are the -%% exception and carry a real ref. +%% Every read here is already resolved, so a future just wraps the value. Watches +%% are the exception and carry a real ref. ready(Value) -> {dgen_mem_future, Value}. -spec wait(dgen_backend:future()) -> term(). @@ -836,29 +843,28 @@ wait_for_all(Futures) -> [wait(F) || F <- Futures]. -if(?DOCATTRS). -doc """ -Watches `Key`, firing `{Ref, ready}` at the watcher when its value changes. +Watches `Key` and sends `{Ref, ready}` to the watcher when its value changes. **A watch is anchored to the transaction that creates it, not to the moment it is registered.** FoundationDB reports a change relative to the value the creating -transaction would read, and getting that wrong is not a subtle infidelity — it is -a silently lost wakeup: - -- A write that lands *after* the transaction's read version but *before* the watch - is registered still fires it. `consume_queued/5` in `dgen_server` reads an empty queue - and then watches the push key; without this, a push committed in that window - leaves the consumer asleep on a queue that already has work, and the caller's - `call/4` in `dgen` times out with nothing wrong anywhere else. The check and the - registration run under the commit lock so no write can slip between them either. - -- The creating transaction's *own* writes do not fire it — they are part of the - value it would read. `push_call/7` in `dgen` writes the reply sentinel and watches it - in one transaction; firing there costs a read-and-rewatch round trip on every - call, and that round trip is precisely the window the first case then loses a - reply in. - -Firing happens at registration rather than at commit, which is a deliberate -simplification: a watch that fires early is safe here because every consumer -re-reads the watched state before acting, whereas one that fires late is not. +transaction would read. Getting that wrong silently loses a wakeup. + +- A write that lands after the transaction's read version, but before the watch is + registered, still fires the watch. `consume_queued/5` in `dgen_server` reads an + empty queue and then watches the push key. Without this rule, a push committed in + between leaves the consumer asleep on a queue that already has work. The caller's + `call/4` in `dgen` then times out with nothing wrong anywhere else. The check and + the registration run under the commit lock, so no write can slip between them. + +- The creating transaction's own writes don't fire the watch, because they are part + of the value it would read. `push_call/7` in `dgen` writes the reply sentinel and + watches it in one transaction. Firing there would cost a read-and-rewatch round + trip on every call. That round trip is exactly the window in which the first case + loses a reply. + +We fire at registration rather than at commit. This is a deliberate +simplification. Firing early is safe here, because every consumer re-reads the +watched state before acting. Firing late is not. """. -endif. -spec watch(tx(), dgen_backend:key()) -> dgen_backend:future(). @@ -873,8 +879,8 @@ watch(Tx = {dgen_mem_tx, Name, TxRef}, Key, Opts) -> with_commit_lock(Name, fun() -> case newest_version(Name, Key) > RV of true -> - %% Already changed out from under the transaction. Nothing later - %% will fire this, so fire it now. + %% The key already changed after the transaction's read version. + %% Nothing later will fire this watch, so we fire it now. catch To ! {Ref, ready}; false -> ets:insert(watch_tab(Name), Entry), @@ -884,7 +890,7 @@ watch(Tx = {dgen_mem_tx, Name, TxRef}, Key, Opts) -> {dgen_future, Ref, {dgen_mem_future, ok}}. %% Runs under the commit lock, as part of `do_commit/3`. `TxRef` is the committing -%% transaction: its own writes leave its own watches armed and unfired. +%% transaction. Its writes don't fire its own watches, which stay armed. fire_watches(Name, Keys, TxRef) -> lists:foreach( fun(Key) -> @@ -903,8 +909,8 @@ fire_watches(Name, Keys, TxRef) -> ). %% Cancels a transaction's watches, as FoundationDB does when a transaction is -%% reset. Without this a retried transaction leaves an armed watch behind for every -%% attempt, each of which fires once and delivers to a caller that has moved on. +%% reset. Without this, a retried transaction leaves an armed watch behind for every +%% attempt. Each one later fires and delivers to a caller that has moved on. cancel_watches(Tx = {dgen_mem_tx, Name, _}) -> lists:foreach( fun(Entry) -> ets:delete_object(watch_tab(Name), Entry) end, @@ -912,9 +918,10 @@ cancel_watches(Tx = {dgen_mem_tx, Name, _}) -> ), update_tx(Tx, fun(S) -> S#{watches := []} end). -%% Drops a transaction that will never be used again. Only safe for one abandoned -%% by `do_transactional/3`, which replaces it wholesale — `on_error/2` resets a -%% transaction the caller still holds, so it resets state rather than removing it. +%% Drops a transaction that will never be used again. This is only safe for a +%% transaction that `do_transactional/3` abandons and replaces with a new one. +%% `on_error/2` works on a transaction the caller still holds, so it resets the +%% state instead of removing it. discard_tx(Tx = {dgen_mem_tx, Name, Ref}) -> cancel_watches(Tx), ets:delete(tx_tab(Name), Ref), @@ -924,14 +931,14 @@ discard_tx(Tx = {dgen_mem_tx, Name, Ref}) -> %% Directory layer %% --------------------------------------------------------------------------- %% -%% A directory is an `erlfdb_subspace` over a prefix this module allocates. Reusing -%% erlfdb's (NIF-free) subspace and tuple modules means the key layout is byte-for- -%% byte what the FoundationDB backend produces, so ordering — which `dgen_queue`'s -%% FIFO depends on — is identical rather than merely similar. +%% A directory is an `erlfdb_subspace` over a prefix this module chooses. We reuse +%% erlfdb's subspace and tuple modules, which need no NIF. That makes the key +%% layout byte-for-byte the same as the FoundationDB backend's, so key order is +%% identical, not just similar. The FIFO in `dgen_queue` depends on that order. %% -%% Prefixes are derived from the directory name rather than allocated from a -%% counter, so the same directory name always yields the same keyspace. That is -%% both simpler than FDB's allocator and better for replay. +%% Prefixes come from the directory name rather than from a counter, so the same +%% directory name always gives the same keyspace. That is simpler than FDB's +%% allocator, and it makes runs easier to replay. -spec dir_range(dgen_backend:dir(), dgen_backend:tuple_key()) -> {dgen_backend:key(), dgen_backend:key()}. @@ -966,8 +973,8 @@ to_bin(A) when is_atom(A) -> atom_to_binary(A, utf8). -if(?DOCATTRS). -doc """ -Opens a database and a root directory, mirroring `dgen_erlfdb:sandbox_open/2` so -`DGen.Case` can swap backends without knowing which one it has. +Opens a database and a root directory, like `dgen_erlfdb:sandbox_open/2`. This +lets `DGen.Case` swap backends without knowing which one it has. """. -endif. -spec sandbox_open(term(), term()) -> dgen_backend:tenant(). diff --git a/src/dgen_mod_state_codec.erl b/src/dgen_mod_state_codec.erl index 5a5c94a..db4e8c4 100644 --- a/src/dgen_mod_state_codec.erl +++ b/src/dgen_mod_state_codec.erl @@ -6,20 +6,20 @@ -moduledoc false. -endif. -%% Encoder/decoder for dgen_server mod_state. +%% Encodes and decodes the mod_state of a dgen_server. %% -%% Stores structured Erlang terms across FDB key-value pairs using a recursive -%% encoding scheme. Three encoding types are supported: +%% Spreads a structured Erlang term across many FDB key-value pairs, using one of +%% three encodings: %% -%% - **term** (fallback) - `term_to_binary`, chunked into 100KB values. -%% - **assigns map** - map with all atom keys; each entry stored at a -%% separate FDB key using `atom_to_binary(Key)` in the path. -%% - **component list** - list of maps where every item has an atom `id` -%% key with a binary value; each item stored separately, ordered by a -%% fractional index embedded in the FDB key. +%% - **term** (the fallback) - `term_to_binary`, split into 100KB chunks. +%% - **assigns map** - a map whose keys are all atoms. Each entry gets its own +%% FDB key, with `atom_to_binary(Key)` in the path. +%% - **component list** - a list of maps where every item has an `id` key with a +%% binary value. Each item is stored separately, and a fractional index in its +%% FDB key keeps the items in order. %% -%% The encoding is applied recursively. For example, an assigns map whose value -%% is a component list will nest both encodings in the key path. +%% The encoding applies recursively. For example, if a value in an assigns map +%% is a component list, both encodings nest in the key path. %% %% ### Key structure %% @@ -84,9 +84,9 @@ set(Tenant, BaseKey, OldModState, ModState) -> -if(?DOCATTRS). -doc """ -Diff-based partial write. Compares `OldModState` and `NewModState` and -only writes changed keys. Falls back to a full rewrite when the encoding -type changes or both values are plain terms. +Writes only the keys that differ between `OldModState` and `NewModState`. +Falls back to a full rewrite when the encoding type changes or when both +values use the term encoding. When `OldModState` is `undefined`, existing data is cleared first. """. @@ -202,7 +202,7 @@ term_first_key(Dir, BaseKey) -> -type encoding_type() :: term | map | list. -%% Classify a term into its encoding type. +%% Returns the encoding type for a term. %% - map : map with all atom keys (assigns map) %% - list : list of maps each with a binary id key (component list) %% - term : everything else (fallback) @@ -371,9 +371,9 @@ decode_list(B, Dir, BaseKey, KVs) -> [] -> {ok, []}; _ -> - %% Group by FracIndex (BaseSize+2). Each frac index maps to one item. - %% FDB returns keys sorted by (FracIndex, Id, ...), so groups are - %% already in the correct list order -- no sort needed. + %% Group by FracIndex at position BaseSize+2. Each fractional index + %% belongs to one item. FDB returns keys sorted by (FracIndex, Id, ...), + %% so the groups are already in list order and we don't need to sort. Groups = group_by_pos(BaseSize + 2, DataKVs), Items = [ begin @@ -426,7 +426,7 @@ diff_map(Td, BaseKey, OldMap, NewMap) -> ), ok. -%% Internal diff-set that operates on an already-transactional tenant +%% Does the same diff as set/5, inside a transaction the caller already holds. set_tx(_Td, _BaseKey, Same, Same) -> ok; set_tx(Td, BaseKey, OldModState, NewModState) -> @@ -511,8 +511,8 @@ diff_list({Tx, Dir} = Td, BaseKey, OldList, NewList) -> ok. -%% Walk the new list in order and assign fractional indices to newly added -%% items, placing each between its neighbours' existing indices. +%% Walks the new list in order and gives each added item a fractional index +%% between the indices of its neighbours. -spec assign_new_indices(list(), [binary()], map()) -> map(). assign_new_indices(NewList, AddedIds, OrderMap) -> AddedSet = sets:from_list(AddedIds), @@ -557,8 +557,9 @@ find_next_defined([{_Id, FI} | _Rest]) -> %% Helpers %% -%% Group a sorted list of pre-unpacked KVs by the element at Pos. -%% Relies on FDB key ordering to keep groups contiguous. +%% Groups a sorted list of already-unpacked KVs by the tuple element at Pos. +%% FDB key order keeps each group contiguous, so we only compare with the +%% previous group. -spec group_by_pos(pos_integer(), [{tuple(), binary(), binary()}]) -> [{term(), [{tuple(), binary(), binary()}]}]. group_by_pos(Pos, UnpackedKVs) -> diff --git a/src/dgen_queue.erl b/src/dgen_queue.erl index 8f2e486..f409946 100644 --- a/src/dgen_queue.erl +++ b/src/dgen_queue.erl @@ -8,9 +8,9 @@ -moduledoc """ Durable FIFO queue backed by the configured dgen backend. -Each dgen_server has its own queue keyed under `Quid`. Items are -ordered by versionstamps, guaranteeing strict FIFO across transactions. -Push and pop counts are tracked with atomic `add` operations for O(1) length. +Each dgen_server has its own queue, keyed under `Quid`. Items are ordered +by versionstamp, which keeps the queue strictly FIFO across transactions. +We track push and pop counts with atomic `add` operations, so `length/2` is O(1). *This documentation is LLM-generated. See the AI disclosure in `README.md`.* """. @@ -93,10 +93,11 @@ notify({Tx, Dir}, Quid) -> -doc """ Reads up to `K` items from the queue without removing them. -Returns `{ok, [{RawKey, RawBin}]}` where `RawBin` is the raw encoded value, -or `{error, empty}` when the queue is empty. The items remain in the queue -until `consume_peeked/3` is called. On failure `update_peeked/3` can overwrite a key -in-place within the same transaction to update the embedded attempt counter. +Returns `{ok, [{RawKey, RawBin}]}`, where `RawBin` is the encoded value, or +`{error, empty}` when the queue is empty. The items stay in the queue until +`consume_peeked/3` removes them. If the callback fails, `update_peeked/3` can +overwrite a key in place, in the same transaction, to bump the attempt counter +stored in the item. All three operations must be called within the same transaction so that the read, the callback invocation, and the delete-or-update are atomic. @@ -137,8 +138,8 @@ consume_peeked({Tx, Dir}, KVs, Quid) -> Overwrites the value at `Key` within an existing transaction. Call this within the same transaction as `peek_k/3` when the callback fails, -to update the embedded attempt counter before the transaction commits. The -updated message will be visible to the next consumer. +to update the attempt counter stored in the message before the transaction +commits. The next consumer sees the updated message. """. -endif. -spec update_peeked(dgen_backend:tenant(), dgen_backend:key(), term()) -> ok. diff --git a/src/dgen_registry.erl b/src/dgen_registry.erl index 0f0d1b9..9ff9524 100644 --- a/src/dgen_registry.erl +++ b/src/dgen_registry.erl @@ -3,22 +3,8 @@ -define(DOCATTRS, ?OTP_RELEASE >= 27). -%% The API runs in the caller's process, so under simulation its gen_server calls must be -%% eta_net's: OTP's gen:do_call/4 waits on the wall clock and a real monitor. -include("../include/dgen_eta.hrl"). -%% Test-only (DST, OTP >= 28): turn off supervisor `hibernate_after`, a wall-clock timer -%% that breaks seed determinism (elector_pid/1 calls into the supervisor). --ifdef(DST). --if(?OTP_RELEASE >= 28). --define(SUP_SIM_FLAGS, #{hibernate_after => infinity}). --else. --define(SUP_SIM_FLAGS, #{}). --endif. --else. --define(SUP_SIM_FLAGS, #{}). --endif. - -if(?DOCATTRS). -moduledoc """ OTP-compatible process registry implementing the `{via, dgen_registry, {RegistryName, LogicalName}}` contract. @@ -56,44 +42,58 @@ gen_server:start_link({via, dgen_registry, {my_registry, user_service}}, gen_server:call({via, dgen_registry, {my_registry, user_service}}, ping). ``` -## Process identity - -Starting a registry creates zero atoms. `Name` is supplied by the caller and is -application-controlled, not generated by this module. The top-level supervisor, the -elector, and the connector are never registered under a name — the supervisor is -addressed by the pid `start_link/2,3` returns, and the elector is discovered at -startup (its `$ancestors` process-dictionary entry, then -`supervisor:which_children/1`) and thereafter held as a pid by the member and the -connector. Only the **member** is locally registered, under `Name` itself — the same -atom also names its ETS table (`names_table/1`); no additional atom is derived from -it. This matters for applications that start many independent registries dynamically -(e.g. one per tenant): atoms are never garbage-collected, so deriving additional -atoms per registry (as earlier versions of this module did) would grow the atom table -without bound. See §4.8 of the design doc. +## Process model + +`dgen_registry` is a supervisor that manages a set of children described below. When +your application starts a `dgen_registry`, you provide a `RegistryName`, which must +be an atom. Among your clustered nodes, this atom identifies the grouping of +`dgen_registry` processes into a single clustered entity. When the documentation refers +to "the dgen_registry" it will typically be referring to this cluster of processes, +rather than a singlular Erlang process, unless otherwise specified. + +- `dgen_registry_elector`: A `dgen_server` process that decides which `dgen_registry` + is the leader by using the underlying database for consensus. There can be only + one leader per named registry. +- `dgen_registry_connector`: A `gen_server` process that automatically connects your nodes + together in the Erlang distribution so that they can each rpc to each other. +- `dgen_registry_member`: A `gen_statem` process that stores the actual process map + in its in-memory state, and is responsible for implementation the replication of + that state among the set of leader/followers. + +The `dgen_registry` process is not registered by name in the local process registry. +Instead, you must use the functions in this module or the via contract when using +the registry. + +**Implementation detail:** The `dgen_registry_member` **is** registered in the local +process registry. This is an internal implementation detail and should not be used +outside of `dgen`. ## Consistency model -All writes (`register_name`, `unregister_name`) and consistent reads +**Strongly consistent operations**: All writes (`register_name`, `unregister_name`) and consistent reads (`whereis_name_consistent`) go through the elected leader member process. The leader is the single writer for the name table. -`whereis_name/1` (used by the OTP via-tuple machinery) is a snapshot read +**Eventually consistent operations**: `whereis_name/1` (used by the OTP via-tuple machinery) is a **snapshot read** served lock-free from the local member's `protected` ETS table, in the calling -process — no member-mailbox hop, no network hop, no backend round-trip. The -table is kept in sync by one-way replication casts from the leader. There is -therefore a short window after registration where a remote node's -`whereis_name/1` may still return `undefined`; this is the same -eventual-consistency trade-off as `gproc` global mode. +process. The table is kept in sync by one-way replication casts from the leader. +There is therefore a short window after registration where a remote node's +`whereis_name/1` may still return `undefined`. ## Leadership -The elected leader is the member process on the Erlang node that most -recently committed a backend transaction for the elector queue — i.e., -whoever wins the backend consensus. When leadership changes, the new leader -gathers the freshest names map from every reachable member and broadcasts a -snapshot to all followers on assumption. No distributed lock is used: the -leader-key commit fences the old leader, making the handoff window naturally -quiescent (§5.7 of the design doc). +Leadership election takes advantage of the distributed consensus of the underlying +database, FoundationDB, and is thus simple. The leader is determined by the Erlang +node that processes the "elect leader" message on the `dgen_server`. Since the `dgen_server` +message queue is stored in the database, we are guaranteed that one and only one +Erlang node will successfully dequeue that message from the queue. That winning node is +the registry leader. + +When leadership changes, the new leader gathers state from itself (previously a follower) +and from other current followers before allowing any new operations on the registry. +There is no distributed lock. Instead, the leader-key in the database fences the +old leader. Via the leader-key all participating members agree on what the cluster +is doing. (§5.7 of the design doc). ## Auto-unregistration @@ -101,25 +101,75 @@ The leader monitors every registered Pid. When a monitored process exits, the leader removes the entry and propagates `{name_unregistered}` to all follower members. +In the time period between a process exiting and the registry leader processing +the `'DOWN'`, the registry will still return that process's former mapping. +Once a `'DOWN'` is converted into an unregistry operation, then the process map's +consistency guarantees hold. If you need stronger guarantees on process termination, +you may have the process unregister itself at the opportune time. + ## Metadata and queries -A registration can carry metadata for its lifetime: `register_name/3` / -`set_metadata/2` attach an `index` map (queryable) and an opaque `data` payload; -`get_metadata/1` / `get_metadata_consistent/1` read it back (lock-free / leader-routed, -mirroring `whereis_name/1` / `whereis_name_consistent/1`). `query/2` / -`query_consistent/2` find every registration whose `index` satisfies a conjunction of -exact equalities. See §4.7 of the design doc. +Each process registration can optionally provide metadata, decided by your application. +Process metadata can be used as a simple data store whose lifetime is tied to +the registered process itself. + +The metadata can be specified as an `index` map, which is queryable and/or an opaque `data` +payload, which is not queryable, but is retrievable. + +Refer to the following functions for more detail: + + - `register_name/3` + - `set_metadata/2` + - `get_metadata/1`/`get_metadata_consistent/1` + - `query/2`/`query_consistent/2` + +Also see §4.7 of the design doc. ## Presence -`subscribe/4` / `unsubscribe/2` turn a query into a live feed: a subscription watches -one query and notifies (via a second query) whenever the set of processes matching the -watched query changes — `{dgen_presence, SubId, [{joined | left, Name, Pid}]}`. -Subscriptions are **durable** (stored in the elector's backend state, keyed by an -application id), so they outlive the cluster — built for process-presence systems whose -stakeholders are database entities. See §4.9 of the design doc. +`subscribe/4` / `unsubscribe/2` build upon the queryable metadata. A subscription is +a **durable** query, stored in the elector's backend state, that sends Erlang messages +when metadata inside the registry changes. -*This documentation is LLM-generated. See the AI disclosure in `README.md`.* +Put another way, a subscription watches one query and notifies (via a second query) whenever +the set of processes matching the watched query changes. + + - `{dgen_presence, SubId, [{joined | left, Name, Pid}]}` + +We expect the typical use case of a subscription to be one that is tied to the lifetime of +some database entry of your application, and thus its existence out-lives any in-memory +process registry map. In other words, if your application scales to zero, it can scale back +up and the subscriptions will survive. + +See §4.9 of the design doc. + +### Example + +For example, consider a chat application where a Room is created in the database, and +active WebSocket connections (each an Erlang process) correspond to who is online/present +at a given moment. + +```erlang +% Write a database entity for a Room with primary key `Id` +% ... + +% Store the subscription with the same `Id` (unsubscribe later when the Room is deleted) +ok = dgen_registry:subscribe(chat_registry, Id, + #{in_room => Id}, % <- This query watches for changes + #{in_room => Id} % <- This query finds which processes to notify +). + +% In the WebSocket process +yes = dgen_registry:register_name({chat_registry, ref}, self()). + +dgen_registry:set_metadata( + {chat_registry, self()}, + #{index => #{in_room => Id}} % <- This metadata satisfies the queries above. +). +``` + +Now, all WebSocket processes in the Room are notified of all other WebSocket processes +in the same room via the `dgen_presence` message. """. -endif. @@ -205,17 +255,21 @@ this registry's proactive distribution mesh so it free-rides on connections anot registry maintains (§4.6) — the rest default to a self-sufficient, single-fault-tolerant registry. See `docs/dgen_registry_design.md` (§8 Configuration). -The supervisor itself is **not** registered under a name — hold onto the `pid()` -this returns (e.g. `Supervisor.stop/2` to tear it down) rather than looking it up -later by `Name`, which now names the member, not the supervisor. See the -"Process identity" note above. +The supervisor itself is **not** registered under a name. See the +"Process model" note above. ## `keyspace` — separating identity from the coordination keyspace -`Name` normally plays three roles at once: this member's **identity** (the registered -process name, the ETS table name, and the second element of its `member_id/0`), and -the registry's **keyspace** (the tuid prefixing every durable key — the elector queue, -the leader key, the version key). +The `keyspace` option will typically not be used by your application code. It exists +to allow more than one registry of the same name per Erlang node, which is not advised +outside of dgen's own test suites. + +`Name` is used in the following ways: + + 1. The member's **identity**: the registered process name, the ETS table name, + and the second element of its `member_id/0`. + 2. The registry's **keyspace**: the tenant-unique-identifier (tuid) prefixing + every durable key — the elector queue, the leader key, the version key. The optional `keyspace` option splits the second role out: @@ -223,23 +277,6 @@ The optional `keyspace` option splits the second role out: dgen_registry:start_link(member_a, Tenant, #{keyspace => shared_registry}), dgen_registry:start_link(member_b, Tenant, #{keyspace => shared_registry}). ``` - -Both members now coordinate through *one* registry's durable state — one elector -queue, one leader key, one version key — while keeping distinct identities, so they -elect a leader between themselves and replicate to each other exactly as members on -separate nodes do. It defaults to `Name`, which is the ordinary one-member-per-node -deployment. - -This exists for **simulation**: `member_id/0` is `{node(), Name}`, so without it a -single VM can host at most one member of a given registry and any multi-member test -needs real distribution and real nodes. With it, an N-member cluster runs inside one -BEAM — which is what makes deterministic, seeded, replayable simulation of the -replication protocol possible (see `test/support/sim/`). It is **not** intended for -production deployment: two members of one registry sharing a VM share a fate, so they -are not two holders in the sense of Guarantee 4, and the connector's node-level -backstops (§4.6) cannot distinguish them. - -Note that `delete/2` takes the **keyspace** name, not a member's `Name`. """. -endif. -spec start_link( @@ -263,8 +300,9 @@ start_link(Name, Tenant, Opts) -> Registers `Pid` under `{RegistryName, LogicalName}`. Routes through the local member, which forwards to the leader if needed. -Returns `yes` on success, `no` if the name is already held by a *different* -process or no leader is currently elected. +Returns `yes` on success, or `no` if a *different* live process holds the name +or an opt-in policy refuses it (§8). If no leader is reachable, the call waits +for one, and exits if nothing is decided within `register_timeout` (§3). Registering the **same** pid under the **same** name again is an idempotent `yes` (not a `no`): a caller whose register call timed out — but whose @@ -279,16 +317,6 @@ register_name(Name, Pid) -> -if(?DOCATTRS). -doc """ Removes the registration for `{RegistryName, LogicalName}`. Always returns `ok`. - -Routes through the local member as a **call** (mirroring the register path): the -member removes the entry from its own replica immediately — snapshot reads on this -node see the delete at once — and forwards the removal to the leader as a *tracked* -request. `ok` normally means the leader has committed the removal; if the leader is -unreachable (or answers late), `ok` means the removal is **stashed and re-driven** — -forwarded to the current leader as a pid-guarded retract on the next leadership -change or rejoin snapshot — so an explicit unregister is never silently lost -(Non-goal 5 of the design doc). Cluster-wide visibility remains eventually -consistent (replication is asynchronous). """. -endif. -spec unregister_name({atom(), term()}) -> ok. @@ -307,14 +335,7 @@ unregister_name({RegistryName, LogicalName}) -> -if(?DOCATTRS). -doc """ -Snapshot read — served lock-free from the local member's ETS table. - -Runs entirely in the **calling process** (an `ets:lookup/2` against the member's -`protected` names table), so it never blocks on the member's mailbox, the leader, -or the backend — many callers read concurrently. May be slightly stale on follower -nodes in the brief window between a remote registration and the replication cast -arriving, and returns `undefined` while the member is (re)starting and its table -does not yet exist. +Finds a process by its name in the registry via a snapshot read, using the local member's ETS table. """. -endif. -spec whereis_name({atom(), term()}) -> pid() | undefined. @@ -352,20 +373,13 @@ send(Name, Msg) -> -if(?DOCATTRS). -doc """ -Registers `Pid` under `{RegistryName, LogicalName}` with metadata, in one fenced step. +Registers `Pid` under `{RegistryName, LogicalName}` with metadata. `MetaSpec` is `#{index => map(), data => term()}` (both optional; defaults `#{}` and -`undefined`). Like `register_name/2`, returns `yes` on success or `no` if the name is -already taken, no leader is currently elected, or the leader is unreachable — those -are all *adjudicated* verdicts, answered promptly. +`undefined`). Returns `yes` or `no` with the same meaning as in `register_name/2`. A call **timeout** (the member, or the leader behind it, did not answer within -`register_timeout`, §8 — default 5000 ms) **exits** rather than answering `no`: a -timeout means no verdict exists, and masking it as `no` would tell the caller "the -name is taken" when the registration may in fact have committed. Letting the exit -propagate matches what `global`, `syn`, and other registries do — the caller (or its -supervisor) crashes and retries with fresh state. A missing registry (`noproc`) also -exits — a usage error. +`register_timeout`, §8 — default 5000 ms) **exits** rather than answering `no`. """. -endif. -spec register_name({atom(), term()}, pid(), map()) -> yes | no. @@ -381,10 +395,10 @@ register_name({RegistryName, LogicalName}, Pid, MetaSpec) -> Replaces the metadata of the existing registration `{RegistryName, LogicalName}`. `MetaSpec` is `#{index => map(), data => term()}` (both optional; an omitted field -becomes `#{}` / `undefined` — this is a replace, not a merge). Routes through the leader -and rides the fenced commit. Returns `ok`, `{error, not_registered}` if the name is not -currently bound, or `{error, no_leader}` if no leader is elected (or leadership changed -mid-flight — retry). +becomes `#{}` / `undefined` — this is a replace, not a merge). + +Returns `ok`, `{error, not_registered}` if the name is not currently bound, or +`{error, no_leader}` if no leader is elected. """. -endif. -spec set_metadata({atom(), term()}, map()) -> ok | {error, not_registered | no_leader}. @@ -403,12 +417,10 @@ set_metadata({RegistryName, LogicalName}, MetaSpec) -> -if(?DOCATTRS). -doc """ -Snapshot read of a registration's metadata — served lock-free from the calling process. +Retrieves a registered process's metadata via a snapshot read, using the local member's ETS table. -Like `whereis_name/1`, this is an `ets:lookup/2` against the local member's `protected` -table with no member round-trip. Returns `{ok, #{pid := pid(), index := map(), data := -term()}}` for a registered name, or `undefined` if it is not registered (or the member's -table does not exist yet). May lag a recent change made on another node. +Returns `{ok, #{pid := pid(), index := map(), data := term()}}` for a registered name, +or `undefined` if it is not registered. """. -endif. -spec get_metadata({atom(), term()}) -> @@ -423,10 +435,10 @@ get_metadata({RegistryName, LogicalName}) -> -if(?DOCATTRS). -doc """ -Consistent metadata read routed through the leader (authoritative, never stale). +Retrieves a registered process's metadata via a consistent read. -Returns the same shape as `get_metadata/1`. More expensive (one network hop) but -reflects the leader's committed view. +Returns `{ok, #{pid := pid(), index := map(), data := term()}}` for a registered name, +or `undefined` if it is not registered. """. -endif. -spec get_metadata_consistent({atom(), term()}) -> @@ -440,12 +452,12 @@ get_metadata_consistent({RegistryName, LogicalName}) -> -if(?DOCATTRS). -doc """ -Finds every registration whose indexed metadata satisfies *all* of `Constraints`. +Finds every registration whose indexed metadata satisfies *all* of `Constraints`, via a snapshot read. `Constraints` is `#{attr() => value()}` — a conjunction of exact equalities (AND-equal) matched against each registration's `index` map. Returns a point-in-time list of `#{name, pid, index, data}` matches (possibly empty), or `{error, empty_query}` if -`Constraints` is empty (use a future `list/1` for "everything"). +`Constraints` is empty. This is a **batch-consistent snapshot** read of the local member's replica: it runs on the member's mailbox (unlike the lock-free single-key reads) so it never observes a @@ -467,10 +479,10 @@ query(RegistryName, Constraints) -> -if(?DOCATTRS). -doc """ -Like `query/2`, but routed through the leader for an authoritative (never-stale) result. +Finds every registration whose indexed metadata satisfies *all* of `Constraints`, via a consistent read. -Returns the same shape as `query/2`. More expensive (it serialises on the leader's -mailbox, one network hop from a follower) but reflects the leader's committed view. +Returns a point-in-time list of `#{name, pid, index, data}` matches (possibly empty), +or `{error, empty_query}` if `Constraints` is empty. """. -endif. -spec query_consistent(atom(), map()) -> @@ -490,50 +502,42 @@ query_consistent(RegistryName, Constraints) -> -if(?DOCATTRS). -doc """ -Creates (or updates) a **durable** presence subscription `SubId`: keeps the processes -matching the `Notify` query up to date with the set of processes matching the `Watch` -query. +Creates (or updates) a **durable** presence subscription `SubId`. + +A subscription keeps the processes matching the `Notify` query up to date with the set +of processes matching the `Watch` query. `Watch` and `Notify` are `query/0` values — a `{all, #{attr => value}}` conjunction of exact equalities over registration `index` metadata (a bare map is accepted and read as `{all, Map}`). `SubId` is any application term; a subscribe with an existing `SubId` is an idempotent **upsert** (replaces its queries). -Whenever a committed batch of registrations changes which processes satisfy `Watch` — a -process registers into it, unregisters/dies out of it, or a metadata change moves it in -or out — the registry sends, to **every process currently matching `Notify`**, one -message: +Whenever a committed batch of registrations changes which processes satisfy `Watch`, +the registry sends the following message to **every process currently matching `Notify`**: ```erlang {dgen_presence, SubId, Events} ``` where `Events` is a non-empty list of `presence_event/0` — `{joined, Name, Pid}` / -`{left, Name, Pid}`. Every process that matches `Notify` receives an **initial snapshot** -(the current `Watch` set as a batch of `joined` events) when it enters the notify set — -whether that is at subscribe time or when it registers into `Notify` later — and deltas -thereafter. A viewer therefore always has a complete, continuously-maintained view from -the moment it appears, so the subscription can safely be created before any viewer exists -(e.g. tied to a database entity's lifetime). +`{left, Name, Pid}`. + +When a process is registerd that matches the notify set, it received an **initial snapshot**, +the current `Watch` set as a batch of `joined` events. Thereafter, it receives deltas. +The initial snapshot and the deltas combine to a complete, continuously-maintained view +of the set of "present" processes. ## Durability The subscription is stored in the elector's **durable** backend state, not tied to any -Erlang process: it outlives the cluster. An application can therefore tie a +Erlang process: it outlives the registry cluster. An application can therefore tie a subscription's lifetime to a database entity (subscribe when the entity is created, `unsubscribe/2` when it is deleted) and have presence come back intact after a scale-to-zero — the durable subscription is reloaded and re-fed to the leader on restart, and processes matching `Notify` resume receiving updates as they re-register. Returns `ok`, or `{error, empty_query}` if either query is empty (validated locally). -The write is sent to the elector as a durable **cast** (fire-and-forget) for low latency, -so `ok` means "accepted for durable processing", not "already committed": it commits -asynchronously and `subscriptions/1` reflects it shortly after. Durability still holds — -the change commits in the elector's backend transaction and survives a full cluster -restart — and because a subscription is an idempotent upsert keyed by `SubId`, a write -lost in the narrow window before the elector durably enqueues it is safely recoverable by -re-subscribing (the application owns the source of truth). A missing registry (`noproc`) -exits. +The write is sent to the elector as a durable `dgen_server:cast`. """. -endif. -spec subscribe(atom(), term(), query(), query()) -> ok | {error, empty_query}. @@ -544,54 +548,51 @@ subscribe(RegistryName, SubId, Watch, Notify) -> true -> {error, empty_query}; false -> - elector_cast(RegistryName, {subscribe, SubId, NWatch, NNotify}) + with_elector(RegistryName, fun(Pid) -> + dgen_registry_elector:subscribe(Pid, SubId, NWatch, NNotify) + end) end. -if(?DOCATTRS). -doc """ -Removes the durable presence subscription `SubId` (from `subscribe/4`). Returns `ok` -(idempotent — removing an unknown `SubId` is a no-op). Sent as a durable cast, applied -asynchronously (see `subscribe/4`). A missing registry exits. +Removes the durable presence subscription `SubId` (from `subscribe/4`). Returns `ok`. +Sent as a durable `dgen_server:cast`, applied asynchronously (see `subscribe/4`). """. -endif. -spec unsubscribe(atom(), term()) -> ok. unsubscribe(RegistryName, SubId) -> - elector_cast(RegistryName, {unsubscribe, SubId}). + with_elector(RegistryName, fun(Pid) -> dgen_registry_elector:unsubscribe(Pid, SubId) end). -if(?DOCATTRS). -doc """ Removes **all** durable presence subscriptions from the registry. Returns `ok`. - -Like `unsubscribe/2` but wholesale — a durable cast, applied asynchronously, pushed to -the leader so it drops every subscription's live state. Useful for tearing down presence -without deleting the registry itself (a full `delete/2` also removes them, along with -everything else). """. -endif. -spec unsubscribe_all(atom()) -> ok. unsubscribe_all(RegistryName) -> - elector_cast(RegistryName, unsubscribe_all). + with_elector(RegistryName, fun(Pid) -> dgen_registry_elector:unsubscribe_all(Pid) end). -if(?DOCATTRS). -doc """ Returns the registry's durable presence subscriptions as `#{SubId => {Watch, Notify}}`. - -A read of the elector's committed state (shared cluster-wide and durable), so it -reflects every subscription regardless of which node created it. Returns `#{}` if the -registry cannot currently be reached. """. -endif. -spec subscriptions(atom()) -> #{term() => {query(), query()}}. subscriptions(RegistryName) -> with_elector(RegistryName, #{}, fun(Pid) -> - dgen_server:priority_call(Pid, get_subscriptions) + dgen_registry_elector:priority_get_subscriptions(Pid) end). -%% Fire-and-forget; the elector commits it durably. A missing registry exits. -elector_cast(RegistryName, Msg) -> - case elector_pid(RegistryName) of - undefined -> exit({noproc, {?MODULE, elector, RegistryName}}); - Pid -> dgen_server:cast(Pid, Msg) +with_elector(Name, Default, Fun) -> + case elector_pid(Name) of + undefined -> Default; + Pid -> Fun(Pid) + end. + +with_elector(Name, Fun) -> + case elector_pid(Name) of + undefined -> exit({noproc, {?MODULE, elector, Name}}); + Pid -> Fun(Pid) end. %% --------------------------------------------------------------------------- @@ -602,18 +603,9 @@ elector_cast(RegistryName, Msg) -> -doc """ Removes **all** of the registry's durable backend state. Returns `ok`. -Every key a registry writes — the elector's membership/leader/subscription state and its -message queue, plus the leader-fence and version keys — lives under a single tenant -keyspace prefix derived from `Name`, so this clears exactly that registry's keys (and no -other registry sharing the tenant). Use it to reclaim the durable footprint of a -registry you are done with — e.g. one of many short-lived, per-entity registries in a -shared tenant (see the "Process identity" note on why nothing else is left behind). - **Stop the registry first.** `delete/2` operates directly on the backend and assumes no live processes: a running elector caches its state in memory and would re-write it (and -so re-create the keys) on its next commit. The correct sequence is -`Supervisor.stop(Sup)` (the pid `start_link/2,3` returned), then `delete/2` with the -tenant you started the registry with. +so re-create the keys) on its next commit. This does not delete anything the registered processes themselves own; it only removes the registry's own bookkeeping. @@ -667,14 +659,14 @@ whereis_name_consistent({RegistryName, LogicalName}) -> -endif. -spec get_leader(Name :: atom()) -> member_id() | undefined. get_leader(Name) -> - with_elector(Name, undefined, fun(Pid) -> dgen_server:priority_call(Pid, get_leader) end). + with_elector(Name, undefined, fun(Pid) -> dgen_registry_elector:priority_get_leader(Pid) end). -if(?DOCATTRS). -doc "Returns the current leader epoch. Increments each time a new leader is elected.". -endif. -spec get_epoch(Name :: atom()) -> non_neg_integer(). get_epoch(Name) -> - with_elector(Name, 0, fun(Pid) -> dgen_server:priority_call(Pid, get_epoch) end). + with_elector(Name, 0, fun(Pid) -> dgen_registry_elector:priority_get_epoch(Pid) end). %% Poll interval and per-probe call bound for await_ready/2. -define(READY_POLL_INTERVAL, 50). @@ -682,16 +674,11 @@ get_epoch(Name) -> -if(?DOCATTRS). -doc """ -Is the registry on this node ready to serve registrations *right now*? - -Returns `true` once the local member both knows a leader and has synced registry -state (applied the leader's snapshot, or assumed leadership itself). Returns `false` -while it is still starting/joining, has no elected leader, or is momentarily -unresponsive (a busy member is treated as not-ready rather than blocking the caller). +Returns `true` if the registry on this node is ready to serve registrations. This is a point-in-time check; use `await_ready/2` to block until ready. Note that readiness means registrations are being *served*, not that any particular -registration will return `yes` (a name may already be taken, or leadership may move). +registration will return `yes`. """. -endif. -spec ready(Name :: atom()) -> boolean(). @@ -710,8 +697,8 @@ Reports this node's member's own view of the registry, as a map: - `member_id` — this member's `member_id/0`. - `leader` — the member id it currently believes leads, or `undefined`. -- `is_leader` — whether that belief names itself. -- `epoch` — the leadership epoch it has heard (§4.2). +- `is_leader` — `true` if the member believes itself to be the leader. +- `epoch` — the leadership epoch it's aware of (§4.2). - `synced` — whether it has applied a snapshot or assumed leadership. - `applied_version` — the commit version its replica has applied up to (§4.5). @@ -780,14 +767,7 @@ await_ready_loop(Name, Deadline) -> -endif. -spec get_members(Name :: atom()) -> [member_id()]. get_members(Name) -> - with_elector(Name, [], fun(Pid) -> dgen_server:priority_call(Pid, get_members) end). - -%% Bypasses the member's mailbox (§8); `Default` if the elector is not found. -with_elector(Name, Default, Fun) -> - case elector_pid(Name) of - undefined -> Default; - Pid -> Fun(Pid) - end. + with_elector(Name, [], fun(Pid) -> dgen_registry_elector:priority_get_members(Pid) end). %% --------------------------------------------------------------------------- %% Process identity helpers @@ -796,10 +776,6 @@ with_elector(Name, Default, Fun) -> -if(?DOCATTRS). -doc """ Returns the registered name of the member process for the given registry name. - -Identity today (`member_name(Name) =:= Name`) — the member is registered under -`Name` directly. Kept as a function (rather than inlining `Name` at call sites) so -callers do not depend on that being the case. """. -endif. -spec member_name(atom()) -> atom(). @@ -809,13 +785,6 @@ member_name(Name) -> -if(?DOCATTRS). -doc """ Returns the ETS table name holding the registry's local `Name => Pid` replica. - -Identity today (`names_table(Name) =:= Name`) — the member's names table is -named `Name` directly (ETS `named_table` names and Erlang process-registration -names are separate namespaces, so this does not collide with the member's own -registration). The table is owned and written by the member (a `protected`, -`read_concurrency` table) and read lock-free from any process — see -`whereis_name/1`. """. -endif. -spec names_table(atom()) -> atom(). @@ -826,14 +795,6 @@ names_table(Name) -> -doc """ Returns the current elector `pid()` for registry `Name`, or `undefined` if the member (and so the elector) cannot currently be found. - -The elector has no registered name — it is discovered by locating the member -(registered as `Name`), reading its `$ancestors` process-dictionary entry (set by -`proc_lib` at spawn time) to find the shared supervisor, then reading that -supervisor's children (`supervisor:which_children/1`) for the one with child id -`elector`. This is an all-local, in-memory operation (no network hop); it is what -`get_leader/1`, `get_epoch/1`, and `get_members/1` use internally, and is exported -for introspection. """. -endif. -spec elector_pid(atom()) -> pid() | undefined. @@ -842,7 +803,12 @@ elector_pid(Name) -> undefined -> undefined; MemberPid -> - case ancestors_of(MemberPid) of + Ancestors = + case erlang:process_info(MemberPid, dictionary) of + {dictionary, Dict} -> proplists:get_value('$ancestors', Dict, []); + undefined -> [] + end, + case Ancestors of [Sup | _] -> case lists:keyfind(elector, 1, supervisor:which_children(Sup)) of {elector, ElectorPid, _Type, _Modules} when is_pid(ElectorPid) -> @@ -855,22 +821,17 @@ elector_pid(Name) -> end end. -%% `$ancestors` (no public proc_lib accessor): pids or names, as which_children/1 takes. --spec ancestors_of(pid()) -> [pid() | atom()]. -ancestors_of(Pid) -> - case erlang:process_info(Pid, dictionary) of - {dictionary, Dict} -> proplists:get_value('$ancestors', Dict, []); - undefined -> [] - end. - %% --------------------------------------------------------------------------- %% Supervisor callbacks %% --------------------------------------------------------------------------- init({Name, Tenant, Opts}) -> - %% Elector first (unnamed; the member discovers it via this supervisor). A large - %% consume_k (> 1 also disables inlining) keeps one stable consumer, hence leader. - %% `Keyspace` differs from `Name` only when members of one registry share a VM. + %% Start the elector first. + %% + %% 1. A large consume_k keeps one stable consumer, and therefore a stable leader. + %% 2. consume_k > 1 disables inlining. + %% 3. By default, Keyspace is the same as Name. It cna optionally be changed to + %% to support dgen test suite. Keyspace = maps:get(keyspace, Opts, Name), ElectorSpec = #{ id => elector, @@ -886,7 +847,7 @@ init({Name, Tenant, Opts}) -> modules => [dgen_server, dgen_registry_elector] }, - %% From `Keyspace`, as the elector's: all members fence on the same leader key. + %% Then start the member. It will discover the elector via this supervisor. Tuid = dgen_registry_elector:tuid(Keyspace), MemberSpec = #{ id => member, @@ -906,7 +867,7 @@ init({Name, Tenant, Opts}) -> modules => [dgen_registry_member] }, - %% Started after the member, from which it discovers the elector. + %% Finally start the node connector. ConnectorSpec = #{ id => connector, start => @@ -920,7 +881,12 @@ init({Name, Tenant, Opts}) -> modules => [dgen_registry_connector] }, - %% one_for_all: the others hold the elector's pid, so never run with a stale one. + %% Supervisor config + %% + %% 1. one_for_all: the children discover and store the elector pid, so they must + %% all restart in unison. + %% 2. intensity and period: Sane choices, but they are not tuned. + %% 3. Simulation flags: defined at compile time for DST {ok, {maps:merge(#{strategy => one_for_all, intensity => 5, period => 10}, ?SUP_SIM_FLAGS), [ ElectorSpec, MemberSpec, ConnectorSpec diff --git a/src/dgen_registry/dgen_registry_conflicts.erl b/src/dgen_registry/dgen_registry_conflicts.erl index f8e9a9a..fb1137d 100644 --- a/src/dgen_registry/dgen_registry_conflicts.erl +++ b/src/dgen_registry/dgen_registry_conflicts.erl @@ -2,44 +2,53 @@ -define(DOCATTRS, ?OTP_RELEASE >= 27). +-include("dgen_registry_member.hrl"). + -if(?DOCATTRS). -moduledoc false. -endif. -%% Conflict repair (§5.6; `§N.N` is docs/design/dgen_registry_design.md). Pids are never -%% durable, so if every holder of a binding is lost the name can be re-issued while the -%% original lives on; on reconnection two live pids hold one name. +%% Detects and repairs conflicting name bindings (§5.6). `§N.N` refers to +%% docs/design/dgen_registry_design.md. +%% +%% Pids are never durable. If every member holding a binding is lost, the leader can +%% hand the name out again while the original pid is still alive. When the lost +%% members reconnect, two live pids hold the same name. %% -%% Evidence: a trail of explicit releases, `{Name, Pid} => ms`, recorded at commit or -%% from the broadcast and merged at handoff; a witness binding on it is lag, not a -%% conflict. Keyed by pair, so one release never masks another name. Not pruned while a -%% member is disconnected or within a TTL of a departure: its stale rows may resurface. +%% Evidence. We keep a trail of explicit releases as `{Name, Pid} => ms`. The leader +%% records them at commit, followers record them from the broadcast, and a handoff +%% merges them. If a witness still binds a released pair, that is replication lag, not +%% a conflict. The trail is keyed by name and pid together, so a release of one name +%% never hides a conflict on another. We don't prune it while a member is disconnected +%% or within a TTL of a departure, because that member's stale rows may come back. %% -%% Verdict: the authority (freshest reconstruction) binds a name to a live pid and a -%% witness to a different live pid off the trail. A name the authority lacks is no -%% conflict. Liveness is probed off the leader's loop: conflict_pids/3 lists what to -%% probe, detect/4 decides. +%% Verdict. A conflict exists when the authority binds a name to a live pid and a +%% witness binds it to a different live pid that isn't on the trail. The authority is +%% the freshest reconstruction. If the authority doesn't hold the name at all, there is +%% no conflict. The leader probes liveness outside its own loop. get_conflict_pids/3 +%% lists the pids to probe, and detect/4 makes the decision. %% -%% Remedy: resolve/5 kills the authority and all divergent pids and drops the name, -%% unless `terminate_on_conflict` is off (alarm only) or the per-name, per-term -%% `conflict_kill_budget` is spent (alarm and stop: escalate rather than loop). Kills -%% and alarms are returned as effects; nothing here probes, sends or kills. +%% Remedy. resolve/5 kills the authority pid and every divergent pid, and drops the +%% name. There are two exceptions. With `terminate_on_conflict` off, we only raise an +%% alarm. When the `conflict_kill_budget` for that name is spent, we raise an alarm and +%% stop killing, so a recurring conflict escalates instead of looping. This module +%% returns kills and alarms as effects. It never probes, sends or kills anything itself. %% -%% Prevention: with `reject_when_degraded`, a leader whose reconstruction may lack a -%% committed binding refuses names it does not hold (degraded/5). +%% Prevention. With `reject_when_degraded`, a leader whose reconstruction may be +%% missing a committed binding refuses names it doesn't already hold. See +%% is_reconstruction_degraded/5. %% %% *This documentation is LLM-generated. See the AI disclosure in README.md.* --export([new/0, trail/1, released/3, merge/2, departed/2, prune/4]). --export([conflict_pids/3, detect/4, resolve/5]). --export([degraded/1, set_degraded/2, degraded/5]). +-export([new/0, get_trail/1, record_releases/3, merge/2, record_departure/2, prune/4]). +-export([get_conflict_pids/3, detect/4, resolve/5]). +-export([is_degraded/1, set_degraded/2, is_reconstruction_degraded/5]). -export_type([t/0, trail/0, conflict/0, effect/0]). -type records() :: dgen_registry_replica:records(). -%% Bare-pid keys come from pre-upgrade members during a rolling upgrade. --type trail() :: #{{term(), pid()} | pid() => integer()}. +-type trail() :: #{{term(), pid()} => integer()}. -type conflict() :: {Name :: term(), Authority :: pid(), Divergent :: [pid()]}. @@ -47,17 +56,7 @@ {kill, pid()} | {alarm, conflict(), terminated | detect_only | budget_exhausted}. --record(conflicts, { - trail = #{} :: trail(), - %% Leader-only: name => recent kill times (ms), for `conflict_kill_budget`. - kill_budget = #{} :: #{term() => [integer()]}, - %% Last member departure seen; suspends trail pruning for a TTL. - last_departure = 0 :: integer(), - %% Leader-only: may the last reconstruction lack a committed binding? - degraded = false :: boolean() -}). - --opaque t() :: #conflicts{}. +-type t() :: #conflicts{}. -spec new() -> t(). new() -> #conflicts{}. @@ -66,25 +65,31 @@ new() -> #conflicts{}. %% Evidence: the release trail %% --------------------------------------------------------------------------- --spec trail(t()) -> trail(). -trail(#conflicts{trail = T}) -> T. +-spec get_trail(t()) -> trail(). +get_trail(#conflicts{trail = T}) -> T. -%% Record explicit releases of live pids at `Now`. --spec released([{term(), pid()}], integer(), t()) -> t(). -released([], _Now, C) -> +%% Records explicit releases of live pids at time `Now`. +-spec record_releases([{term(), pid()}], integer(), t()) -> t(). +record_releases([], _Now, C) -> C; -released(Pairs, Now, C = #conflicts{trail = T}) -> +record_releases(Pairs, Now, C = #conflicts{trail = T}) -> C#conflicts{trail = lists:foldl(fun(Pair, Acc) -> Acc#{Pair => Now} end, T, Pairs)}. -%% Merge gathered trails, keeping each entry's latest release time. +%% Merges the trails gathered at handoff, keeping the latest release time for each entry. -spec merge([trail()], t()) -> t(). merge(Trails, C = #conflicts{trail = T}) -> - C#conflicts{trail = merge_trails(T, Trails)}. + Merged = lists:foldl( + fun(Other, Acc) -> maps:merge_with(fun(_K, A, B) -> max(A, B) end, Acc, Other) end, + T, + Trails + ), + C#conflicts{trail = Merged}. --spec departed(integer(), t()) -> t(). -departed(Now, C) -> C#conflicts{last_departure = Now}. +-spec record_departure(integer(), t()) -> t(). +record_departure(Now, C) -> C#conflicts{last_departure = Now}. -%% Trail entries expire only if every member is connected and none departed in the TTL. +%% Expires old trail entries, but only when every member is connected and none has +%% departed within the TTL. Also drops kill-budget timestamps older than the window. -spec prune(integer(), boolean(), dgen_config:config(), t()) -> t(). prune(Now, AllConnected, Config, C = #conflicts{trail = T, kill_budget = KB}) -> TTL = dgen_config:conflict_release_ttl(Config), @@ -105,25 +110,29 @@ prune(Now, AllConnected, Config, C = #conflicts{trail = T, kill_budget = KB}) -> %% Verdict %% --------------------------------------------------------------------------- -%% Every pid a conflict could turn on, to probe off the leader's loop before detect/4. --spec conflict_pids(records(), [records()], trail()) -> [pid()]. -conflict_pids(Authority, Witnesses, Trail) -> +%% Returns every pid a conflict could involve. The leader probes these outside its +%% loop before calling detect/4. +-spec get_conflict_pids(records(), [records()], trail()) -> [pid()]. +get_conflict_pids(Authority, Witnesses, Trail) -> lists:usort( maps:fold( fun(Name, PidSet, Acc) -> - [P || P <- [authority(Name, Authority) | maps:keys(PidSet)], is_pid(P)] ++ Acc + [P || P <- [get_authority_pid(Name, Authority) | maps:keys(PidSet)], is_pid(P)] ++ + Acc end, [], - candidates(Authority, Witnesses, Trail) + get_divergent_bindings(Authority, Witnesses, Trail) ) ). -%% The authority's records may be among the witnesses: an entry agrees with itself. +%% Returns the conflicts among the divergent bindings, using `Alive` to test pids. The +%% authority's records may also appear among the witnesses, which is fine because an +%% entry always agrees with itself. -spec detect(records(), [records()], trail(), fun((pid()) -> boolean())) -> [conflict()]. detect(Authority, Witnesses, Trail, Alive) -> maps:fold( fun(Name, PidSet, Acc) -> - APid = authority(Name, Authority), + APid = get_authority_pid(Name, Authority), Others = [P || P <- maps:keys(PidSet), P =/= APid], case Others =:= [] orelse not (is_pid(APid) andalso Alive(APid)) of true -> @@ -136,11 +145,12 @@ detect(Authority, Witnesses, Trail, Alive) -> end end, [], - candidates(Authority, Witnesses, Trail) + get_divergent_bindings(Authority, Witnesses, Trail) ). -%% Differing witness bindings off the trail: `#{Name => #{Pid => true}}`. -candidates(Authority, Witnesses, Trail) -> +%% Returns the witness bindings that differ from the authority and aren't on the +%% trail, as `#{Name => #{Pid => true}}`. +get_divergent_bindings(Authority, Witnesses, Trail) -> lists:foldl( fun(Map, Acc0) -> maps:fold( @@ -149,7 +159,7 @@ candidates(Authority, Witnesses, Trail) -> {Pid, _, _} -> Acc; _AbsentOrDifferent -> - case on_trail(Name, Pid, Trail) of + case maps:is_key({Name, Pid}, Trail) of true -> Acc; false -> @@ -166,20 +176,18 @@ candidates(Authority, Witnesses, Trail) -> Witnesses ). -authority(Name, Authority) -> +get_authority_pid(Name, Authority) -> case maps:get(Name, Authority, undefined) of {Pid, _Index, _Data} -> Pid; undefined -> undefined end. -on_trail(Name, Pid, Trail) -> - maps:is_key({Name, Pid}, Trail) orelse maps:is_key(Pid, Trail). - %% --------------------------------------------------------------------------- %% Remedy %% --------------------------------------------------------------------------- -%% `Records` only loses names, so with no conflicts it passes through by reference. +%% Applies the remedy to each conflict. Resolving only ever removes names from +%% `Records`, so with no conflicts we return it unchanged without copying. -spec resolve([conflict()], records(), dgen_config:config(), integer(), t()) -> {records(), t(), [effect()]}. resolve([], Records, _Config, _Now, C) -> @@ -188,26 +196,27 @@ resolve(Conflicts, Records, Config, Now, C) -> Terminate = dgen_config:terminate_on_conflict(Config), Budget = dgen_config:conflict_kill_budget(Config), {Records1, C1, EffsRev} = lists:foldl( - fun(Conflict, Acc) -> resolve_one(Conflict, Terminate, Budget, Now, Acc) end, + fun(Conflict, Acc) -> resolve_conflict(Conflict, Terminate, Budget, Now, Acc) end, {Records, C, []}, Conflicts ), {Records1, C1, lists:reverse(EffsRev)}. -resolve_one(Conflict, false, _Budget, _Now, {Records, C, Effs}) -> - %% Detect-only: alarm, and keep the authority's binding. +resolve_conflict(Conflict, false, _Budget, _Now, {Records, C, Effs}) -> + %% Detect-only mode. Raise an alarm and keep the authority's binding. {Records, C, [{alarm, Conflict, detect_only} | Effs]}; -resolve_one( +resolve_conflict( {Name, Authority, Divergent} = Conflict, true, {Count, Window}, Now, {Records, C, Effs} ) -> KB = C#conflicts.kill_budget, Recent = [Ts || Ts <- maps:get(Name, KB, []), Ts >= Now - Window], case length(Recent) < Count of false -> - %% Exhausted: escalate, stop killing this name, keep it served. + %% The budget is spent. Escalate with an alarm, stop killing for this name, and + %% keep serving it. {Records, C, [{alarm, Conflict, budget_exhausted} | Effs]}; true -> - %% A pid that died since detection ignores the kill. + %% A pid that died since detection simply ignores the kill. Kills = [{kill, P} || P <- [Authority | Divergent]], { maps:remove(Name, Records), @@ -220,30 +229,24 @@ resolve_one( %% Prevention %% --------------------------------------------------------------------------- --spec degraded(t()) -> boolean(). -degraded(#conflicts{degraded = D}) -> D. +-spec is_degraded(t()) -> boolean(). +is_degraded(#conflicts{degraded = D}) -> D. -spec set_degraded(boolean(), t()) -> t(). set_degraded(D, C) -> C#conflicts{degraded = D}. -%% Degraded exactly when the reconstruction stopped short of the durable version key -%% (`Frontier`): replicas are stream prefixes, so the freshest at the key is the -%% committed map (TLA+ DegradedFlagSound). With the key unreadable, fall back to -%% reachability: degraded if a member not marked fresh did not answer the gather. --spec degraded( +%% Returns true when the reconstruction stopped short of `Frontier`, the durable +%% version key. Every replica is a prefix of the same stream, so a replica that reached +%% the key holds the committed map (TLA+ DegradedFlagSound). If we couldn't read the +%% key, we fall back to reachability. The reconstruction is degraded if any member not +%% marked fresh failed to answer the gather. +-spec is_reconstruction_degraded( non_neg_integer(), non_neg_integer() | unknown, [term()], [term()], #{term() => term()} ) -> boolean(). -degraded(MaxVersion, Frontier, _OtherIds, _FreshIds, _PeerResults) when is_integer(Frontier) -> +is_reconstruction_degraded(MaxVersion, Frontier, _OtherIds, _FreshIds, _PeerResults) when + is_integer(Frontier) +-> MaxVersion < Frontier; -degraded(_MaxVersion, unknown, OtherIds, FreshIds, PeerResults) -> +is_reconstruction_degraded(_MaxVersion, unknown, OtherIds, FreshIds, PeerResults) -> Unreachable = OtherIds -- maps:keys(PeerResults), lists:any(fun(Id) -> not lists:member(Id, FreshIds) end, Unreachable). - -%% --------------------------------------------------------------------------- -%% Internal -%% --------------------------------------------------------------------------- - -merge_trails(Trail, Others) -> - lists:foldl( - fun(T, Acc) -> maps:merge_with(fun(_K, A, B) -> max(A, B) end, Acc, T) end, Trail, Others - ). diff --git a/src/dgen_registry/dgen_registry_connector.erl b/src/dgen_registry/dgen_registry_connector.erl index 2d59219..57ab1fe 100644 --- a/src/dgen_registry/dgen_registry_connector.erl +++ b/src/dgen_registry/dgen_registry_connector.erl @@ -9,14 +9,17 @@ -moduledoc false. -endif. -%% Node-level connectivity for one registry (sibling of the member and elector): meshes -%% distribution to every member node, reaps members stranded on dropped nodes, and probes -%% leader liveness. Never touches names, commits or leadership; reads the elector -%% off-loop and casts fenced `{member_down}` back. `§N.N` = docs/design/dgen_registry_design.md. +%% Handles node-level connectivity for one registry. It runs alongside the member and +%% the elector. It connects distribution to every member node, reaps members stranded +%% on nodes that dropped, and checks that the leader is still alive. It never touches +%% names, commits or leadership. It reads the elector from spawned processes and casts +%% `{member_down}` back, tagged with the member's token so the elector can ignore a +%% stale one. `§N.N` refers to docs/design/dgen_registry_design.md. %% -%% Modes (§8, §4.6): `self_managed` runs the mesh; `provided_externally` opens no -%% connections but still does the registry-scoped duties (reap, leader probe, epoch nudge -%% on its own `epoch_check` timer) and warns if no leader is seen for a sustained window. +%% There are two modes (§8, §4.6). In `self_managed` mode the connector runs the mesh. +%% In `provided_externally` mode it opens no connections, but it still reaps members, +%% probes the leader, and nudges the member's epoch on its own `epoch_check` timer. It +%% also warns if it sees no leader for a sustained period. %% %% *This documentation is LLM-generated. See the AI disclosure in README.md.* @@ -31,42 +34,54 @@ code_change/3 ]). -%% Backstop; startup and `nodeup` also trigger an immediate pass. +%% Safety timer for meshing the network of nodes. Startup and `nodeup` also trigger an +%% immediate pass. -define(MESH_INTERVAL, 10000). -%% Don't reconnect a node right after its `nodedown`: let the departure settle as a -%% `{member_down}` rather than healing the partition instantly. `nodeup` clears it early. +%% We don't reconnect a node right after its `nodedown`. The departure should settle as +%% a `{member_down}` first, instead of the partition healing instantly. A `nodeup` ends +%% the cooldown early. -define(MESH_DOWN_COOLDOWN, 10000). -%% Backstop reap for a member whose own `{member_down}` was fenced as stale. Retried -%% because a resurrecting `{join}` may land after the first pass. Opens no connection. +%% Backstop reap for a member whose own `{member_down}` the elector rejected as stale. +%% We retry because a `{join}` that brings the member back may land after the first +%% pass. Reaping opens no connection. -define(REAP_INTERVAL, 1000). -define(REAP_ATTEMPTS, 6). -%% Reaps a leader on a node we were *never* connected to (e.g. cold restart onto a dead -%% leader: no `nodedown` ever fires). Gated on `ever_connected` so a distribution-only -%% partition cannot depose the majority's leader via the DB (ping-pong); a lost-but-met -%% leader is the nodedown path's job. Grace lets the async mesh connect first. +%% Reaps a leader on a node we were *never* connected to. This happens on a cold restart +%% onto a dead leader, where no `nodedown` ever fires. We skip nodes in `ever_connected`. +%% Otherwise a partition in distribution alone could let us depose the majority's leader +%% through the database, and the two sides would keep deposing each other. A leader we +%% did connect to and then lost is handled by the `nodedown` path. The grace period lets +%% the mesh connect first. -define(LEADER_PROBE_GRACE, 4000). -define(LEADER_PROBE_INTERVAL, 4000). -%% `provided_externally` only: epoch-nudge / isolation-check tick (replaces the mesh). +%% Only in `provided_externally` mode. This tick nudges the epoch and checks for +%% isolation, in place of the mesh. -define(EPOCH_CHECK_INTERVAL, 10000). -%% ~60 s of no leader: long enough to ride out startup/handoff, short enough to surface. +%% About 60 s without a leader. That is long enough to ride out startup or a handoff, +%% and short enough to surface a real problem. -define(ISOLATION_WARN_TICKS, 6). -record(state, { - %% Member's registered name (= registry name); target of `{durable_epoch, _}`. + %% The member's registered name, which is the registry name. We send + %% `{durable_epoch, _}` to it. name :: atom(), mode = self_managed :: self_managed | provided_externally, - %% `undefined` only until `discover_elector`. + %% `undefined` only until `discover_elector` runs. elector :: pid() | undefined, - %% Node -> nodedown timestamp; mesh skips these (self_managed only). + %% Maps each node to the time of its `nodedown`. The mesh skips these nodes. Only + %% used in `self_managed` mode. recently_down = #{} :: #{node() => integer()}, - %% Never pruned; gates the leader probe. + %% Never pruned. The leader probe skips these nodes. ever_connected = #{} :: #{node() => true}, - %% provided_externally isolation warning: counter and one-shot latch. + + %% `no_leader_ticks` and `isolation_warned` are a safety feature for + %% `provided_externally` mode. We count the ticks without a connectivity leader + %% and log a warning once. no_leader_ticks = 0 :: non_neg_integer(), isolation_warned = false :: boolean() }). @@ -76,7 +91,7 @@ %% --------------------------------------------------------------------------- -if(?DOCATTRS). --doc "Starts the connector for registry `Name` (unregistered; held by the supervisor).". +-doc "Starts the connector for registry `Name`. The process is unregistered, and the supervisor holds its pid.". -endif. -spec start_link(Name :: atom(), Args :: map()) -> gen_server:start_ret(). start_link(Name, Args) -> @@ -89,8 +104,6 @@ start_link(Name, Args) -> init(#{name := Name} = Args) -> net_kernel:monitor_nodes(true), Mode = dgen_config:connectivity(maps:get(config, Args, #{})), - %% Can't resolve the elector here: the supervisor is blocked in start_child for this - %% process, so which_children would deadlock. {ok, #state{ name = Name, @@ -101,7 +114,8 @@ init(#{name := Name} = Args) -> }, {continue, discover_elector}}. -%% The member starts before this connector, so `elector_pid/1` can walk from it. +%% The member starts before this connector, so `elector_pid/1` can find the elector +%% through it. handle_continue(discover_elector, State = #state{name = Name, mode = Mode}) -> Elector = dgen_registry:elector_pid(Name), true = is_pid(Elector), @@ -109,9 +123,9 @@ handle_continue(discover_elector, State = #state{name = Name, mode = Mode}) -> self_managed -> self() ! mesh_connect; provided_externally -> - arm(?EPOCH_CHECK_INTERVAL, epoch_check) + schedule_timer(?EPOCH_CHECK_INTERVAL, epoch_check) end, - arm(?LEADER_PROBE_GRACE, probe_leader), + schedule_timer(?LEADER_PROBE_GRACE, probe_leader), {noreply, State#state{elector = Elector}}. handle_call(_Request, _From, State) -> @@ -120,7 +134,7 @@ handle_call(_Request, _From, State) -> handle_cast(_Msg, State) -> {noreply, State}. -%% The member handles {nodeup, _} separately (re-announces its {join}). +%% The member handles `{nodeup, _}` separately, by re-announcing its `{join}`. handle_info({nodeup, Node}, State = #state{mode = provided_externally}) -> Ever = maps:put(Node, true, State#state.ever_connected), {noreply, State#state{ever_connected = Ever}}; @@ -136,7 +150,7 @@ handle_info({nodedown, Node}, State = #state{recently_down = RD}) -> schedule_reap(Node, ?REAP_ATTEMPTS), {noreply, State#state{recently_down = RD#{Node => erlang:system_time(millisecond)}}}; handle_info(epoch_check, State = #state{elector = Elector, name = Name}) -> - arm(?EPOCH_CHECK_INTERVAL, epoch_check), + schedule_timer(?EPOCH_CHECK_INTERVAL, epoch_check), spawn_epoch_check(Elector, Name, self()), {noreply, State}; handle_info({connectivity_leader, undefined}, State = #state{no_leader_ticks = N}) -> @@ -152,7 +166,7 @@ handle_info({connectivity_leader, undefined}, State = #state{no_leader_ticks = N {noreply, State#state{no_leader_ticks = N1, isolation_warned = Warned1}}; handle_info({connectivity_leader, _Leader}, State) -> {noreply, State#state{no_leader_ticks = 0, isolation_warned = false}}; -%% Stops once the node reconnects (a live member's `{join}` wins). +%% Stops once the node reconnects, because a live member's `{join}` takes precedence. handle_info({reap_down, _Node, 0}, State) -> {noreply, State}; handle_info({reap_down, Node, N}, State = #state{elector = Elector}) -> @@ -165,20 +179,21 @@ handle_info({reap_down, Node, N}, State = #state{elector = Elector}) -> end, {noreply, State}; handle_info(probe_leader, State = #state{elector = Elector, ever_connected = Ever}) -> - arm(?LEADER_PROBE_INTERVAL, probe_leader), + schedule_timer(?LEADER_PROBE_INTERVAL, probe_leader), spawn_probe_leader(Elector, node(), Ever), {noreply, State}; handle_info(mesh_connect, State = #state{elector = Elector, name = Name}) -> - arm(?MESH_INTERVAL, mesh_connect), + schedule_timer(?MESH_INTERVAL, mesh_connect), Now = erlang:system_time(millisecond), RD = maps:filter( fun(_Node, Ts) -> Ts >= Now - ?MESH_DOWN_COOLDOWN end, State#state.recently_down ), spawn_mesh_fetch(Elector, self(), Name), {noreply, State#state{recently_down = RD}}; -%% Suppression is decided here, not in the fetch helper: this message is ordered after -%% any `{nodedown, _}` on our mailbox, so a stale in-flight fetch can't reconnect a node -%% that just dropped (which would heal the partition before `{member_down}` settles). +%% We filter out recently-down nodes here, not in the fetch helper. This message +%% arrives after any `{nodedown, _}` already in our mailbox, so a stale fetch can't +%% reconnect a node that just dropped. That would heal the partition before the +%% `{member_down}` settles. handle_info({mesh_members, MemberNodes}, State) -> Now = erlang:system_time(millisecond), RD = maps:filter( @@ -206,21 +221,22 @@ code_change(_OldVsn, State, _Extra) -> %% --------------------------------------------------------------------------- schedule_reap(Node, N) -> - arm(?REAP_INTERVAL, {reap_down, Node, N}), + schedule_timer(?REAP_INTERVAL, {reap_down, Node, N}), ok. -%% Fenced with each member's current durable token so a genuine rejoin isn't clobbered. +%% Sends `member_down` with each member's current durable token. If the member has +%% genuinely rejoined, its token has changed and the elector ignores the stale one. spawn_reap(Elector, Node) -> - _ = spawn(fun() -> - try dgen_server:priority_call(Elector, get_members) of + spawn_catch_all(fun() -> + case dgen_registry_elector:priority_get_members(Elector) of Members when is_list(Members) -> lists:foreach( fun ({N, _} = MemberId) when N =:= Node -> - Token = dgen_registry_elector:member_token( + Token = dgen_registry_elector:get_member_token( Elector, MemberId, undefined ), - dgen_server:cast(Elector, {member_down, MemberId, Token}); + dgen_registry_elector:member_down(Elector, MemberId, Token); (_) -> ok end, @@ -228,15 +244,12 @@ spawn_reap(Elector, Node) -> ); _ -> ok - catch - _:_ -> ok end - end), - ok. + end). spawn_probe_leader(Elector, SelfNode, EverConnected) -> - _ = spawn(fun() -> - try dgen_server:priority_call(Elector, get_leader) of + spawn_catch_all(fun() -> + case dgen_registry_elector:priority_get_leader(Elector) of {LeaderNode, _} when LeaderNode =/= SelfNode -> case lists:member(LeaderNode, nodes()) orelse @@ -247,47 +260,42 @@ spawn_probe_leader(Elector, SelfNode, EverConnected) -> end; _ -> ok - catch - _:_ -> ok end - end), - ok. + end). -%% Regular `call`, not `priority_call`: it rides the durable queue in order with -%% membership changes, so a just-joined node is seen as soon as its join is processed. +%% Uses a regular `call`, not `priority_call`. The call goes through the durable queue +%% in order with membership changes, so we see a newly joined node as soon as the +%% elector processes its join. spawn_mesh_fetch(Elector, Connector, MemberName) -> - _ = spawn(fun() -> + spawn_catch_all(fun() -> nudge_durable_epoch(Elector, MemberName), - try dgen_server:call(Elector, get_members) of + case dgen_registry_elector:get_members(Elector) of Members when is_list(Members) -> Connector ! {mesh_members, [Node || {Node, _Name} <- Members]}; _ -> ok - catch - _:_ -> ok end - end), - ok. + end). spawn_epoch_check(Elector, MemberName, Connector) -> - _ = spawn(fun() -> + spawn_catch_all(fun() -> nudge_durable_epoch(Elector, MemberName), Leader = - try dgen_server:priority_call(Elector, get_leader) of + try dgen_registry_elector:priority_get_leader(Elector) of L -> L catch - %% Counts toward the isolation window. + %% A failed call counts toward the isolation window. _:_ -> undefined end, Connector ! {connectivity_leader, Leader} - end), - ok. + end). -%% Missed-handoff backstop: if the committed epoch is ahead of the member's, it re-joins. -%% The send must be inside `try` (not after `of`): sending to an unregistered name raises. +%% Backstop for a missed handoff. If the committed epoch is ahead of the member's, the +%% member re-joins. The send must be inside the `try` body, not after `of`, because +%% sending to an unregistered name raises. nudge_durable_epoch(Elector, MemberName) -> try - case dgen_server:priority_call(Elector, get_epoch) of + case dgen_registry_elector:priority_get_epoch(Elector) of DurableEpoch when is_integer(DurableEpoch) -> MemberName ! {durable_epoch, DurableEpoch}, ok; @@ -308,12 +316,22 @@ warn_possible_isolation(Name, Ticks) -> ), ok. -%% Off-loop: the handshake can block on an unreachable node. +%% Runs in a separate process, because the handshake can block on an unreachable node. spawn_mesh_connect(Targets) -> - _ = spawn(fun() -> lists:foreach(fun net_kernel:connect_node/1, Targets) end), + spawn_catch_all(fun() -> lists:foreach(fun net_kernel:connect_node/1, Targets) end). + +spawn_catch_all(Fun) -> + _ = spawn(fun() -> + try + Fun() + catch + _:_ -> + ok + end + end), ok. -%% See `dgen_registry_member:arm/2`. -arm(Delay, Msg) -> +%% See `dgen_registry_member:schedule_timer/2`. +schedule_timer(Delay, Msg) -> _ = ?ETA_LOG({arm, Msg, Delay}), erlang:send_after(Delay, self(), Msg). diff --git a/src/dgen_registry/dgen_registry_elector.erl b/src/dgen_registry/dgen_registry_elector.erl index ef6b6c0..b32bdad 100644 --- a/src/dgen_registry/dgen_registry_elector.erl +++ b/src/dgen_registry/dgen_registry_elector.erl @@ -4,218 +4,285 @@ -define(DOCATTRS, ?OTP_RELEASE >= 27). -include("../../include/dgen_eta.hrl"). --define(SnapshotTimeout, 2000). + +% Actions (aka side effects) taken by the elector +-define(ActionPushLeaderToMembers(MemberId), + dgen_registry_elector_actions:push_leader_to_members(MemberId) +). +-define(ActionPushPresenceToLeader(Msg), + dgen_registry_elector_actions:push_presence_to_leader(Msg) +). -if(?DOCATTRS). -moduledoc false. -endif. -%% `dgen_server` callback tracking registry membership and electing a leader. State -%% is durable and shared by every node of the registry; messages arrive via a durable -%% FIFO queue. `§N.N` = docs/design/dgen_registry_design.md. +%% A `dgen_server` callback that tracks registry membership and elects a leader. Its +%% state is durable and shared by every node of the registry, and messages reach it +%% through a durable FIFO queue. See docs/design/dgen_registry_design.md. %% -%% Leadership is sticky (kept while the incumbent is a member) to avoid thrashing. -%% A membership change routes through the leader as a post-commit action, with no -%% distributed lock (§5.7): committing the new leader key fences the old leader, whose -%% next batch commit then aborts. The leader gathers replicas, resolves conflicts -%% (§5.6) and casts the snapshot to followers from its own process, so per-pair FIFO -%% orders it before any later `{name_registered}`. +%% Leadership is sticky. The current leader keeps the role as long as it is a member, +%% which avoids thrashing. A membership change reaches the leader as a post-commit +%% action, with no distributed lock (§5.7). Committing a new leader key fences the old +%% leader, so its next batch commit aborts. The leader gathers replicas, resolves +%% conflicts (§5.6) and casts the snapshot to followers from its own process. Erlang +%% delivers messages between two processes in order, so each follower gets the +%% snapshot before any later `{name_registered}`. %% -%% Leader key `{Tuid, <<"leader">>}` holds `term_to_binary({MemberId | undefined, -%% Epoch})`: the epoch makes it a real fencing token (no ABA on depose-and-re-elect). +%% The leader key `{Tuid, <<"leader">>}` holds `term_to_binary({MemberId | undefined, +%% Epoch})`. The epoch is what makes it a real fencing token. Without it, deposing a +%% leader and re-electing the same member would leave the key unchanged. %% %% *This documentation is LLM-generated. See the AI disclosure in README.md.* -export([ + join/4, + member_down/3, + get_members/1, + + subscribe/4, + unsubscribe/2, + unsubscribe_all/1, + + priority_get_members/1, + priority_get_leader/1, + priority_get_epoch/1, + priority_get_subscriptions/1, + priority_get_subscriptions/2, + get_member_token/3, + get_leader_db_key/2, + tuid/1, + init/1, handle_cast_tx/3, - handle_call_tx/4, - leader_db_key/2, - tuid/1, - member_token/3 + handle_call_tx/4 ]). -export_type([member_id/0, member_info/0, registry_state/0]). -type member_id() :: {node(), atom()}. -%% `fresh` = never synced any state; an unreachable fresh member holds no bindings, so -%% it is excluded from the leader's `degraded` computation (§5.6). -type member_info() :: #{ - joined_at := integer(), join_token := reference(), fresh => boolean() + % Each member joins with a fresh token. When a member goes down, the token tells + % us whether the DOWN refers to this join or an earlier one. + join_token := reference(), + + % `true` when the member has not yet synced any state. Used in state replication. + fresh => boolean() }. + +% Stored in FDB. Any proposed changes to the state have implications on preexisting registries. -type registry_state() :: #{ + % The registry name name := atom(), + + % The list of members who have joined members := #{member_id() => member_info()}, + + % The authoritative leader. Only `dgen_registry_elector` chooses the leader. leader := member_id() | undefined, + + % Increases with every leadership change, so re-electing the same member still + % changes the leader key. epoch := non_neg_integer(), - %% Durable presence subscriptions (§4.9), `SubId => {Watch, Notify}`; survive a - %% full cluster restart. Source of truth; the leader is fed deltas. + + %% Durable presence subscriptions (§4.9), as `SubId => {Watch, Notify}`. They + %% survive a full cluster restart. This map is the source of truth, and the leader + %% receives changes to it as deltas. subscriptions => #{term() => {term(), term()}} }. %% --------------------------------------------------------------------------- -%% dgen_server callbacks +%% API %% --------------------------------------------------------------------------- +join(Elector, MemberId, Token, Fresh) -> + dgen_server:cast(Elector, {join, MemberId, Token, Fresh}). + +member_down(Elector, MemberId, Token) -> + dgen_server:cast(Elector, {member_down, MemberId, Token}). + +get_members(Elector) -> + dgen_server:call(Elector, get_members). + +subscribe(Elector, SubId, Watch, Notify) -> + dgen_server:cast(Elector, {subscribe, SubId, Watch, Notify}). + +unsubscribe(Elector, SubId) -> + dgen_server:cast(Elector, {unsubscribe, SubId}). + +unsubscribe_all(Elector) -> + dgen_server:cast(Elector, unsubscribe_all). + +priority_get_members(Elector) -> + dgen_server:priority_call(Elector, get_members). + +priority_get_leader(Elector) -> + dgen_server:priority_call(Elector, get_leader). + +priority_get_epoch(Elector) -> + dgen_server:priority_call(Elector, get_epoch). + +priority_get_subscriptions(Elector) -> + dgen_server:priority_call(Elector, get_subscriptions). + +priority_get_subscriptions(Elector, Timeout) -> + dgen_server:priority_call(Elector, get_subscriptions, Timeout). + -if(?DOCATTRS). -doc """ -Initialises the elector state with an empty member map and undefined leader. - -`name` is the co-located member's **identity** — it is what `elect_leader/5` turns -into the local `member_id/0` when preferring the local member for leadership. -`keyspace` is the registry's durable **prefix**, and is what the tuid is derived -from; it defaults to `name`. The two differ only when a registry is started with -`dgen_registry:start_link/3`'s `keyspace` option, which lets several members of one -registry share a VM (see that function's docs). +Returns the token the elector holds for `MemberId` right now. Returns `Default` if +the member is unknown or the read fails. + +This is a priority read, so it skips the durable queue. Callers attach the result to +a `{member_down}`. Because it is the elector's *current* token, a DOWN for a peer that +has since rejoined with a new token is ignored, while a member that is really dead is +still reaped. It has no side effects, so it is safe to call from a spawned helper. The +member's peer-DOWN path and the connector's reap both use it. """. -endif. --spec init(#{name := atom(), keyspace => atom()}) -> - {ok, dgen_server:tuid(), registry_state()}. -init(Args = #{name := Name}) -> - State = #{ - name => Name, members => #{}, leader => undefined, epoch => 0, subscriptions => #{} - }, - {ok, tuid(maps:get(keyspace, Args, Name)), State}. +-spec get_member_token(pid(), member_id(), Default) -> reference() | Default. +get_member_token(Elector, MemberId, Default) -> + try dgen_server:priority_call(Elector, {get_member_token, MemberId}) of + T when is_reference(T) -> T; + _ -> Default + catch + _:_ -> Default + end. -if(?DOCATTRS). -doc """ -The transaction-unit id (backend keyspace prefix) for registry `Name`. +Returns the packed backend key for the leader value. -The single source of truth for the registry's backend key prefix: the elector, the -member (its version-key commit and leader-key fence), and the top-level supervisor -must all use the *same* tuid, so they derive it here rather than each rebuilding the -literal. Exported for that reason. +Exported so callers can set up a backend watch without going through the +elector process. +""". +-endif. +-spec get_leader_db_key(dgen_backend:dir(), dgen_server:tuid()) -> dgen_backend:key(). +get_leader_db_key(Dir, Tuid) -> + B = dgen_config:backend(), + B:dir_pack(Dir, leader_key_tuple(Tuid)). + +-if(?DOCATTRS). +-doc """ +Returns the tenant-unique id for registry `Name`. It is the backend keyspace prefix. + +The elector, the member, and the top-level supervisor must all use the *same* tuid. +The member needs it for its version-key commit and its leader-key fence. So they all +derive it here instead of each rebuilding the literal, which is why it is exported. """. -endif. -spec tuid(atom()) -> dgen_server:tuid(). tuid(Name) -> {<<"dgen_registry.">>, atom_to_binary(Name)}. +%% --------------------------------------------------------------------------- +%% dgen_server callbacks +%% --------------------------------------------------------------------------- + +-if(?DOCATTRS). +-doc """ +Initialises the elector state with an empty member map and an undefined leader. + +`name` is the **identity** of the member on the same node. `elect_leader/5` turns it +into the local `member_id/0` when it prefers the local member for leadership. +`keyspace` is the registry's durable **prefix**, and the tuid is derived from it. It +defaults to `name`. The two differ only when the registry is started with the +`keyspace` option of `dgen_registry:start_link/3`, which lets several members of one +registry share a VM. See that function's docs. +""". +-endif. +-spec init(#{name := atom(), keyspace => atom()}) -> + {ok, dgen_server:tuid(), registry_state()}. +init(Args = #{name := Name}) -> + % Reminder: init/1 is always called, but the initial state is only relevant when there + % exists no state in FDB for this tuid. + State = #{ + name => Name, members => #{}, leader => undefined, epoch => 0, subscriptions => #{} + }, + {ok, tuid(maps:get(keyspace, Args, Name)), State}. + -if(?DOCATTRS). -doc """ Processes membership change messages within a backend transaction. -Handles `{join, MemberId, Token, Fresh}` (a legacy 3-tuple `{join, MemberId, -Token}` is accepted as `Fresh = false`) and `{member_down, MemberId, Token}`. -Returns `{noreply, NewState, Actions}` when the handoff must be routed through the -leader (every join; a leader-changing member_down), else plain -`{noreply, NewState}`. The `Actions` run after the transaction commits — no -distributed lock (§5.7). - -Each `{join}` carries a unique token (a `reference()` generated by the member -process before enqueuing) and a freshness flag (whether the member has ever synced -registry state). The elector stores both in `member_info`. - -A `{member_down, MemberId, Token}` is silently discarded when its token does not -match the stored token for that member — this means the member has rejoined with -a new token since the DOWN was detected, so the message is stale. This prevents -a partition-recovery race where a `{member_down}` enqueued during the disconnect -is processed after the subsequent `{join}` that heals the cluster. +Handles `{join, MemberId, Token, Fresh}` and `{member_down, MemberId, Token}`. +Returns `{noreply, NewState, Actions}` when the handoff must go through the leader. +That is every join, and every member_down that changes the leader. Otherwise it +returns plain `{noreply, NewState}`. The `Actions` run after the transaction commits, +so no distributed lock is needed (§5.7). + +Each `{join}` carries a unique token and a freshness flag. The member process creates +the token with `make_ref()` before enqueuing. The flag says whether the member has +ever synced registry state. The elector stores both in `member_info`. + +We silently drop a `{member_down, MemberId, Token}` when its token doesn't match the +stored token for that member. A mismatch means the member has rejoined with a new +token since the DOWN was detected, so the message is stale. This prevents a race in +partition recovery, where a `{member_down}` enqueued during the disconnect is +processed after the `{join}` that heals the cluster. """. -endif. -spec handle_cast_tx(dgen_server:tx_ctx(), term(), registry_state()) -> dgen_server:noreply_ret(). -%% Rolling upgrade: legacy 3-tuple join, treated as non-fresh (conservative). -handle_cast_tx(TxCtx, {join, MemberId, Token}, State) -> - handle_cast_tx(TxCtx, {join, MemberId, Token, false}, State); handle_cast_tx(TxCtx, {join, MemberId, Token, Fresh}, State) -> #{name := Name, members := Members, leader := OldLeader, epoch := OldEpoch} = State, - MemberInfo = #{ - joined_at => erlang:system_time(millisecond), join_token => Token, fresh => Fresh - }, + MemberInfo = #{join_token => Token, fresh => Fresh}, NewMembers = Members#{MemberId => MemberInfo}, {NewLeader, NewEpoch} = elect_leader(TxCtx, Name, NewMembers, OldLeader, OldEpoch), NewState = State#{members => NewMembers, leader => NewLeader, epoch => NewEpoch}, - %% Every join routes through the leader so the joiner gets the snapshot from the same - %% sender as later name broadcasts (FIFO). - {noreply, NewState, [assume_distribute_action(MemberId)]}; + %% Every join goes through the leader, so the joiner gets its snapshot from the same + %% process that sends later name broadcasts, and receives them in order. + {noreply, NewState, [?ActionPushLeaderToMembers(MemberId)]}; handle_cast_tx(TxCtx, {member_down, MemberId, Token}, State) -> #{name := Name, members := Members, leader := OldLeader, epoch := OldEpoch} = State, case maps:get(MemberId, Members, undefined) of undefined -> {noreply, State}; #{join_token := StoredToken} when StoredToken =/= Token -> - %% Stale: member rejoined with a new token since this DOWN was detected. + %% Stale. The member rejoined with a new token after this DOWN was detected. {noreply, State}; _ -> NewMembers = maps:remove(MemberId, Members), {NewLeader, NewEpoch} = elect_leader(TxCtx, Name, NewMembers, OldLeader, OldEpoch), NewState = State#{members => NewMembers, leader => NewLeader, epoch => NewEpoch}, - case NewLeader =/= OldLeader of + case NewLeader =:= OldLeader of true -> - {noreply, NewState, [assume_distribute_action(undefined)]}; + %% Survivors saw the death through their own monitors, so we + %% don't broadcast. + {noreply, NewState}; false -> - %% Survivors saw the death via their own monitors; no broadcast. - {noreply, NewState} + {noreply, NewState, [?ActionPushLeaderToMembers(undefined)]} end end; -%% Presence writes (§4.9) are casts for latency: `ok` means accepted, not committed. -%% Idempotent upserts by application id, so a write lost before consume is recoverable. +%% Presence writes (§4.9) are casts for latency, so `ok` means accepted, not committed. +%% Each is an idempotent upsert keyed by the application's id, so the caller can simply +%% repeat a write that was lost before the elector consumed it. handle_cast_tx(_TxCtx, {subscribe, SubId, Watch, Notify}, State) -> - Subs = subscriptions_of(State), + Subs = get_subscriptions(State), NewState = State#{subscriptions => Subs#{SubId => {Watch, Notify}}}, - {noreply, NewState, [presence_push_action({subscribe, SubId, Watch, Notify})]}; + {noreply, NewState, [?ActionPushPresenceToLeader({subscribe, SubId, Watch, Notify})]}; handle_cast_tx(_TxCtx, {unsubscribe, SubId}, State) -> - Subs = subscriptions_of(State), + Subs = get_subscriptions(State), NewState = State#{subscriptions => maps:remove(SubId, Subs)}, - {noreply, NewState, [presence_push_action({unsubscribe, SubId})]}; + {noreply, NewState, [?ActionPushPresenceToLeader({unsubscribe, SubId})]}; handle_cast_tx(_TxCtx, unsubscribe_all, State) -> - {noreply, State#{subscriptions => #{}}, [presence_push_action(unsubscribe_all)]}. - -%% Post-commit: an unreachable leader must not fail the action (the change is already -%% committed; members self-correct on the next membership event). -assume_distribute_action(TriggerMember) -> - fun(#{members := All, leader := Leader, epoch := Epoch}) -> - case Leader of - undefined -> - ok; - _ -> - try - %% The leader pulls subscriptions from its co-located elector. - call_to_member( - Leader, - {elector_assume_and_distribute, TriggerMember, maps:keys(All), - member_tokens(All), fresh_ids(All), Epoch} - ) - catch - exit:Reason -> - depose_if_dead(Leader, Reason, All) - end - end - end. + {noreply, State#{subscriptions => #{}}, [?ActionPushPresenceToLeader(unsubscribe_all)]}. -%% Rolling upgrade: durable state may predate `subscriptions`. -subscriptions_of(State) -> +get_subscriptions(State) -> maps:get(subscriptions, State, #{}). -%% `noproc` means the leader's node answered but its registry is gone; leaving it as -%% leader strands the cluster, so force a re-election (fenced with its current token). -%% Not-connected is ambiguous and deliberately not deposed: disambiguating would open a -%% distribution connection and defeat partition detection. Timeouts = slow, left alone. -depose_if_dead(Leader, Reason, All) -> - case is_noproc(Reason) andalso maps:get(Leader, All, undefined) of - #{join_token := Token} -> - dgen_server:cast(self(), {member_down, Leader, Token}); - _ -> - ok - end. - -is_noproc(noproc) -> true; -is_noproc({noproc, _}) -> true; -is_noproc(_) -> false. - %% --------------------------------------------------------------------------- %% handle_call_tx/4 (all elector calls) %% --------------------------------------------------------------------------- -%% dgen_server dispatches every call (including priority reads) to the `_tx` variant -%% when exported, so there is deliberately no `handle_call/3`. +%% When the `_tx` variant is exported, dgen_server sends every call to it, priority +%% reads included. So there is deliberately no `handle_call/3`. -if(?DOCATTRS). -doc """ Handles the elector's read-only priority calls: `get_leader`, `get_members`, `get_epoch`, -`{get_member_token, _}`, and `get_subscriptions` (the durable subscription map, §4.9). -The presence writes are casts — see handle_cast_tx/3. +`{get_member_token, _}`, and `get_subscriptions`, which returns the durable +subscription map (§4.9). The presence writes are casts. See handle_cast_tx/3. """. -endif. -spec handle_call_tx(dgen_server:tx_ctx(), term(), dgen_server:from(), registry_state()) -> @@ -226,8 +293,9 @@ handle_call_tx(_TxCtx, get_members, _From, State) -> {reply, maps:keys(maps:get(members, State, #{})), State}; handle_call_tx(_TxCtx, get_epoch, _From, State) -> {reply, maps:get(epoch, State, 0), State}; -%% Fresh from durable state, so a `{member_down}` is fenced with the token the elector -%% actually holds, not a stale cached one that would get a genuine death discarded (§5.7). +%% Reads the token from durable state, so a `{member_down}` carries the token the +%% elector actually holds. A stale cached token would cause a real death to be +%% discarded (§5.7). handle_call_tx(_TxCtx, {get_member_token, MemberId}, _From, State) -> Token = case maps:get(MemberId, maps:get(members, State, #{}), undefined) of @@ -236,51 +304,15 @@ handle_call_tx(_TxCtx, {get_member_token, MemberId}, _From, State) -> end, {reply, Token, State}; handle_call_tx(_TxCtx, get_subscriptions, _From, State) -> - {reply, subscriptions_of(State), State}. - -%% Fire-and-forget; a dropped push is caught up by the reseed on the next handoff. -presence_push_action(Update) -> - fun(#{leader := Leader, epoch := Epoch}) -> - case Leader of - undefined -> ok; - _ -> push_to_leader(Leader, {presence_update, Update, Epoch}) - end - end. - -%% Same auto-reconnect guard as call_to_member/2. -push_to_leader({Node, Name}, Msg) -> - case dgen_utils:node_reachable(Node) of - true -> gen_statem:cast({Name, Node}, Msg); - false -> ok - end. - --if(?DOCATTRS). --doc """ -Client-side helper: read the token the elector holds for `MemberId` right now, -falling back to `Default` if the member is unknown or the read fails. - -A priority read (bypasses the durable queue). This is the fence for a -`{member_down}`: reporting the elector's *current* token means a peer that has -genuinely rejoined (advancing its token) is not clobbered by a stale DOWN, while a -truly dead member is still reaped. Pure — safe to call from an off-loop helper; the -member's peer-DOWN path and the connector's reap both use it. -""". --endif. --spec member_token(pid(), member_id(), Default) -> reference() | Default. -member_token(Elector, MemberId, Default) -> - try dgen_server:priority_call(Elector, {get_member_token, MemberId}) of - T when is_reference(T) -> T; - _ -> Default - catch - _:_ -> Default - end. + {reply, get_subscriptions(State), State}. %% --------------------------------------------------------------------------- %% Internal helpers %% --------------------------------------------------------------------------- -%% Sticky incumbent, else the local member (the transaction-race winner), else -%% lists:min/1. Epoch increments only when a new leader is chosen. +%% Keeps the current leader if it is still a member. Otherwise it picks the local +%% member, since this node won the transaction race, and failing that the lowest +%% member id. The epoch increments only when a new leader is chosen. -spec elect_leader( dgen_server:tx_ctx(), atom(), @@ -311,33 +343,5 @@ elect_leader(#{td := {Tx, Dir}, tuid := Tuid}, Name, Members, OldLeader, OldEpoc B:set(Tx, B:dir_pack(Dir, leader_key_tuple(Tuid)), term_to_binary({Leader, Epoch})), {Leader, Epoch}. --if(?DOCATTRS). --doc """ -Returns the packed backend key for the leader value. - -Exported so callers can set up a backend watch without going through the -elector process. -""". --endif. --spec leader_db_key(dgen_backend:dir(), dgen_server:tuid()) -> dgen_backend:key(). -leader_db_key(Dir, Tuid) -> - B = dgen_config:backend(), - B:dir_pack(Dir, leader_key_tuple(Tuid)). - leader_key_tuple(Tuid) -> dgen_key:extend(Tuid, <<"leader">>). - -member_tokens(Members) -> - maps:map(fun(_, #{join_token := T}) -> T end, Members). - -fresh_ids(Members) -> - [Id || {Id, Info} <- maps:to_list(Members), maps:get(fresh, Info, false)]. - -call_to_member({Node, Name}, Msg) -> - %% Never let the call auto-reconnect a disconnected node: the resulting nodeup makes - %% the peer re-join with a new token, so its `{member_down}` is discarded as stale - %% and the partition goes undetected (§5.7). - case dgen_utils:node_reachable(Node) of - true -> gen_statem:call({Name, Node}, Msg, ?SnapshotTimeout); - false -> exit({nodedown, Node}) - end. diff --git a/src/dgen_registry/dgen_registry_elector_actions.erl b/src/dgen_registry/dgen_registry_elector_actions.erl new file mode 100644 index 0000000..d95656b --- /dev/null +++ b/src/dgen_registry/dgen_registry_elector_actions.erl @@ -0,0 +1,96 @@ +-module(dgen_registry_elector_actions). + +-define(DOCATTRS, ?OTP_RELEASE >= 27). + +-include("../../include/dgen_eta.hrl"). +-define(SnapshotTimeout, 2000). + +-if(?DOCATTRS). +-moduledoc false. +-endif. + +%% Post-commit actions returned by `dgen_registry_elector:handle_cast_tx/3`. Each one is +%% a fun that takes the committed elector state. `dgen_server` runs it in the elector +%% process after the transaction commits. +%% +%% *This documentation is LLM-generated. See the AI disclosure in README.md.* + +-export([ + push_leader_to_members/1, + push_presence_to_leader/1 +]). + +%% Tells the leader to take over and distribute the snapshot. An unreachable leader +%% must not fail the action. The change is already committed, and members correct +%% themselves on the next membership event. +push_leader_to_members(TriggerMember) -> + fun(#{members := All, leader := Leader, epoch := Epoch}) -> + case Leader of + undefined -> + ok; + _ -> + try + %% The leader reads subscriptions from the elector on its own node. + call_to_member( + Leader, + {elector_assume_and_distribute, TriggerMember, maps:keys(All), + get_member_tokens(All), get_fresh_ids(All), Epoch} + ) + catch + exit:Reason -> + maybe_force_leader_down(Leader, Reason, All) + end + end + end. + +%% Sends a presence change to the leader without waiting. If the push is dropped, the +%% reseed on the next handoff catches the leader up. +push_presence_to_leader(Update) -> + fun(#{leader := Leader, epoch := Epoch}) -> + case Leader of + undefined -> ok; + _ -> push_to_leader(Leader, {presence_update, Update, Epoch}) + end + end. + +%% `noproc` means the leader's node answered but its registry is gone. Leaving it as +%% leader would strand the cluster, so we force a re-election by casting `member_down` +%% with its current token. We deliberately don't depose a leader that is merely not +%% connected. That case is ambiguous, and finding out would open a distribution +%% connection and defeat partition detection. A timeout just means the leader is slow, +%% so we leave it alone too. `self()` is the elector, because actions run in its process. +maybe_force_leader_down(Leader, Reason, All) -> + case is_noproc(Reason) andalso maps:get(Leader, All, undefined) of + #{join_token := Token} -> + % We have to avoid calling `dgen_registry_elector:member_down/3` because + % that would create a circular dependency. + dgen_server:cast(self(), {member_down, Leader, Token}); + _ -> + ok + end. + +is_noproc(noproc) -> true; +is_noproc({noproc, _}) -> true; +is_noproc(_) -> false. + +%% Uses the same guard against auto-reconnect as call_to_member/2. +push_to_leader({Node, Name}, Msg) -> + case dgen_utils:node_reachable(Node) of + true -> gen_statem:cast({Name, Node}, Msg); + false -> ok + end. + +get_member_tokens(Members) -> + maps:map(fun(_, #{join_token := T}) -> T end, Members). + +get_fresh_ids(Members) -> + [Id || {Id, Info} <- maps:to_list(Members), maps:get(fresh, Info, false)]. + +call_to_member({Node, Name}, Msg) -> + %% Never let the call auto-reconnect a disconnected node. The resulting nodeup + %% makes the peer re-join with a new token. Its `{member_down}` is then discarded as + %% stale, and the partition goes undetected (§5.7). + case dgen_utils:node_reachable(Node) of + true -> gen_statem:call({Name, Node}, Msg, ?SnapshotTimeout); + false -> exit({nodedown, Node}) + end. diff --git a/src/dgen_registry/dgen_registry_intents.erl b/src/dgen_registry/dgen_registry_intents.erl index bcf32a1..79e06a0 100644 --- a/src/dgen_registry/dgen_registry_intents.erl +++ b/src/dgen_registry/dgen_registry_intents.erl @@ -2,40 +2,48 @@ -define(DOCATTRS, ?OTP_RELEASE >= 27). +-include("dgen_registry_member.hrl"). + -if(?DOCATTRS). -moduledoc false. -endif. -%% A non-leader member's unfinished client requests: writes only the leader can commit. -%% `§N.N` refers to docs/design/dgen_registry_design.md. +%% Tracks the client requests a non-leader member is holding until the leader +%% commits them. `§N.N` refers to docs/design/dgen_registry_design.md. +%% +%% Each intent is one request. It may owe the caller a reply, owe the leader a +%% change, or both. We store it under the reference we sent with the forward, so +%% the leader's reply can find it. If a reply arrives under a reference we no +%% longer hold, the intent was already answered or re-sent, and we drop the reply. %% -%% An intent owes a reply (a blocked caller's `From`), an effect (a change that must -%% reach the leader), or both. It is a kind in a phase, keyed by its forward's reference; -%% a reply under an unknown reference (superseded or answered) is ignored. +%% An intent moves through these phases: %% -%% kinds phases -%% {register, Name, Pid, Meta} awaiting_leader no reachable leader -%% {unregister, Name, ReleasedPid} {forwarded, Leader} sent; awaiting a reply -%% {set_meta, Name, Index, Data} {awaiting_version, V, L} L committed it at V; -%% awaiting our replica +%% awaiting_leader No leader is reachable yet. +%% {forwarded, L} We sent it to leader L and are waiting for a reply. +%% {awaiting_version, V, E, L} +%% L committed it at version V in epoch E. We are waiting +%% for our replica to reach V in that same epoch. %% -%% Three events move it: on_reply/4 (the leader answered), on_version/2 (our replica -%% advanced), redrive/5 (a leader may have been established). +%% Three events move it forward: on_reply/5 (the leader answered), on_version/2 +%% (our replica advanced), and redrive/5 (leadership may have changed). %% -%% Re-drive follows idempotence, by kind alone: -%% - register: re-routed until its outcome is known. Same pid, same name is idempotent -%% (§5.2), so `no` is always adjudicated, never caused by leadership moving. -%% - unregister: re-driven as a pid-guarded retract, safe against re-registration. -%% - set_meta: a replace, unsafe to re-send under reordering; if its leader no longer -%% leads, the caller gets `{error, no_leader}`. +%% When leadership changes, each kind is re-sent only as far as re-sending is safe: +%% - register: re-sent until we know the outcome. Registering the same pid under +%% the same name twice is harmless (§5.2), so a `no` always means another pid +%% holds the name, never that leadership moved. +%% - unregister: re-sent as a retract that only removes the name if the same pid +%% still holds it, so it can't undo a later registration. +%% - set_meta: replaces the value, so a late duplicate could overwrite a newer +%% one. If its leader has stepped down, the caller gets `{error, no_leader}`. %% -%% Pure: functions return effects for the member to carry out. +%% The functions here have no side effects. They return effects for the member +%% to carry out. %% %% *This documentation is LLM-generated. See the AI disclosure in README.md.* --export([new/0, submit/5, on_reply/4, on_version/2, redrive/5, prune/3]). +-export([new/0, submit/5, on_reply/5, on_version/2, redrive/5, prune/3]). --export_type([t/0, kind/0, route/0, scope/0, effect/0]). +-export_type([t/0, kind/0, phase/0, route/0, scope/0, position/0, effect/0]). -type member_id() :: dgen_registry_elector:member_id(). -type from() :: gen_server:from(). @@ -48,46 +56,28 @@ -type phase() :: awaiting_leader | {forwarded, member_id()} - | {awaiting_version, non_neg_integer(), member_id()}. - --record(intent, { - kind :: kind(), - phase :: phase(), - %% `answered`: the reply was given but the effect is still owed. - from :: from() | answered, - %% Wall-clock ms of the request; compared against the caller timeout. - at :: integer(), - %% Arrival order. Effects are emitted in this order, never map (reference) order, - %% so the message schedule is reproducible under deterministic simulation. - seq :: non_neg_integer() -}). - --record(intents, { - all = #{} :: #{reference() => #intent{}}, - %% Ref => V of the `awaiting_version` subset: a replica advance scans only these. - deferred = #{} :: #{reference() => non_neg_integer()}, - next_seq = 0 :: non_neg_integer() -}). - --opaque t() :: #intents{}. + | {awaiting_version, non_neg_integer(), non_neg_integer(), member_id()}. + +-type t() :: #intents{}. %% Where a request can go right now, as the member sees it. -type route() :: self | {remote, member_id()} | none. -type scope() :: moved | all. +%% Our replica's epoch and applied version. +-type position() :: {Epoch :: non_neg_integer(), Applied :: non_neg_integer()}. + -type effect() :: {reply, from(), term()} - %% Tracked forward to the leader. + %% Forward to the leader and wait for its reply. | {forward, member_id(), reference(), kind()} - %% Untracked pid-guarded clear. + %% Ask the leader to clear the name if this pid still holds it. No reply. | {retract, member_id(), Name :: term(), pid()} - %% We lead: our own commit pipeline. + %% We are the leader, so commit it ourselves. | {enqueue, kind(), from() | answered} - %% A committed `yes` is ahead of our replica: ask the leader for a snapshot. - | {resync, member_id()} - %% A pre-version leader's `yes` (rolling upgrade): the reply binds the row. - | {insert_row, Name :: term(), pid(), Meta :: {map(), term()}}. + %% Our replica is behind a committed `yes`, so ask the leader for a snapshot. + | {resync, member_id()}. -spec new() -> t(). new() -> #intents{}. @@ -96,9 +86,10 @@ new() -> #intents{}. %% Submission %% --------------------------------------------------------------------------- -%% Accept a request on a non-leader. A forwardable unregister/set_meta is tracked and -%% its caller waits for the commit; everything else, including an `answered` -%% unregister the pipeline surrendered, takes the re-drive path. +%% Accepts a request on a non-leader. When a leader is reachable, an unregister or +%% set_meta is forwarded and its caller waits for the commit. Everything else goes +%% through dispatch/4, the same path a re-drive takes. That includes an unregister +%% the pipeline handed back already answered. -spec submit(kind(), from() | answered, integer(), route(), t()) -> {t(), [effect()]}. submit({set_meta, _, _, _}, From, _Now, Route, T) when Route =:= none; Route =:= self -> {T, [{reply, From, {error, no_leader}}]}; @@ -106,10 +97,10 @@ submit(Kind, From, Now, Route, T = #intents{next_seq = Seq}) -> I = #intent{kind = Kind, phase = awaiting_leader, from = From, at = Now, seq = Seq}, T1 = T#intents{next_seq = Seq + 1}, case Route of - %% An `answered` unregister re-drives as a pid-guarded retract instead. + %% An unregister that was already answered goes to dispatch/4 as a retract. {remote, L} when element(1, Kind) =/= register, From =/= answered -> Ref = make_ref(), - {put(Ref, I#intent{phase = {forwarded, L}}, T1), [{forward, L, Ref, Kind}]}; + {put_intent(Ref, I#intent{phase = {forwarded, L}}, T1), [{forward, L, Ref, Kind}]}; _ -> dispatch(make_ref(), I, Route, T1) end. @@ -118,75 +109,108 @@ submit(Kind, From, Now, Route, T = #intents{next_seq = Seq}) -> %% The three events %% --------------------------------------------------------------------------- -%% The leader answered the forward under `Ref`; `Applied` is our replica's version. -%% A register `yes` is version-guarded (§5.5): answered only once our replica holds the -%% committing batch, so the handoff gather can see this member as a second holder. -%% Never write the row here: the same batch may have cleared the name too. --spec on_reply(reference(), term(), non_neg_integer(), t()) -> {t(), [effect()]}. -on_reply(Ref, Reply, Applied, T = #intents{all = All}) -> +%% Handles the leader's reply to the forward sent under `Ref`. We hold a register +%% `yes` until our replica reaches the version that committed it (§5.5). Otherwise a +%% leadership handoff could gather our replica without the name and lose it. We +%% don't write the row here, because the same batch may also have cleared the name. +%% +%% The version only counts in the epoch that committed it. A later leader may have +%% taken over without that commit (a degraded handoff, §5.7), so if our replica has +%% already moved to a later epoch, we send the register again along `Route`. +-spec on_reply(reference(), term(), position(), route(), t()) -> {t(), [effect()]}. +on_reply(Ref, Reply, Pos, Route, T = #intents{all = All}) -> case All of #{Ref := I = #intent{phase = {forwarded, Leader}}} -> - resolve(Ref, I, Leader, Reply, Applied, delete(Ref, T)); + resolve_reply(Ref, I, Leader, Reply, Pos, Route, delete_intent(Ref, T)); _ -> {T, []} end. -resolve(_Ref, #intent{kind = {register, Name, Pid, Meta}, from = From}, _L, Reply, _A, T) when - Reply =:= {register, yes, legacy} --> - %% Pre-version leader: no version to guard on; the reply alone binds the row. - {T, [{reply, From, yes}, {insert_row, Name, Pid, Meta}]}; -resolve(Ref, I = #intent{kind = {register, _, _, _}, from = From}, L, {register, yes, V}, A, T) -> - case V =< A of - true -> {T, [{reply, From, yes}]}; - false -> {put(Ref, I#intent{phase = {awaiting_version, V, L}}, T), [{resync, L}]} +resolve_reply( + Ref, I = #intent{kind = {register, _, _, _}}, L, {register, yes, V, E}, Pos, Route, T +) -> + #intent{from = From} = I, + case Pos of + {E, A} when V =< A -> + {T, [{reply, From, yes}]}; + {CurE, _A} when CurE > E -> + dispatch(make_ref(), I#intent{phase = awaiting_leader}, Route, T); + _ -> + {put_intent(Ref, I#intent{phase = {awaiting_version, V, E, L}}, T), [{resync, L}]} end; -resolve(_Ref, #intent{kind = {register, _, _, _}, from = From}, _L, {register, no, _}, _A, T) -> +resolve_reply( + _Ref, + #intent{kind = {register, _, _, _}, from = From}, + _L, + {register, no, _, _}, + _Pos, + _Route, + T +) -> {T, [{reply, From, no}]}; -resolve(_Ref, #intent{kind = {set_meta, _, _, _}, from = From}, _L, {set_meta, Result}, _A, T) -> +resolve_reply( + _Ref, #intent{kind = {set_meta, _, _, _}, from = From}, _L, {set_meta, Result}, _Pos, _Route, T +) -> {T, [{reply, From, Result}]}; -resolve(_Ref, #intent{kind = {unregister, _, _}, from = From}, _L, unregister, _A, T) -> - {T, reply_if_owed(From, ok)}; -resolve(Ref, I, _Leader, _Mismatched, _Applied, T) -> - %% A reply whose shape does not match the kind it is keyed under: keep waiting. - {put(Ref, I, T), []}. - -%% Release every deferred `yes` our replica now covers. --spec on_version(non_neg_integer(), t()) -> {t(), [effect()]}. -on_version(_Applied, T = #intents{deferred = D}) when map_size(D) =:= 0 -> +resolve_reply( + _Ref, #intent{kind = {unregister, _, _}, from = From}, _L, unregister, _Pos, _Route, T +) -> + {T, maybe_reply(From, ok)}; +resolve_reply(Ref, I, _Leader, _Mismatched, _Pos, _Route, T) -> + %% The reply doesn't match the request's kind, so keep waiting. + {put_intent(Ref, I, T), []}. + +%% Answers every held `yes` whose version our replica has now reached in the same +%% epoch. A held `yes` from an earlier epoch goes back to `awaiting_leader`, because +%% our replica may no longer contain its commit. The caller must re-drive after any +%% change that can move the epoch. +-spec on_version(position(), t()) -> {t(), [effect()]}. +on_version(_Pos, T = #intents{deferred = D}) when map_size(D) =:= 0 -> {T, []}; -on_version(Applied, T = #intents{all = All, deferred = D}) -> - Due = in_order([{Ref, maps:get(Ref, All)} || {Ref, V} <- maps:to_list(D), V =< Applied]), +on_version({Epoch, Applied}, T = #intents{all = All, deferred = D}) -> + Due = sort_by_arrival([ + {Ref, maps:get(Ref, All)} + || {Ref, {V, E}} <- maps:to_list(D), E =:= Epoch, V =< Applied + ]), + Stale = [Ref || {Ref, {_V, E}} <- maps:to_list(D), E < Epoch], + T1 = lists:foldl( + fun(Ref, Acc) -> + put_intent(Ref, (maps:get(Ref, All))#intent{phase = awaiting_leader}, Acc) + end, + T, + Stale + ), lists:foldl( fun({Ref, #intent{from = From}}, {Acc, Effs}) -> - {delete(Ref, Acc), Effs ++ [{reply, From, yes}]} + {delete_intent(Ref, Acc), Effs ++ [{reply, From, yes}]} end, - {T, []}, + {T1, []}, Due ). -%% Re-drive unresolved intents by kind (see header); register/set_meta past `TTL` (the -%% caller timeout) are dropped unreplied. A register awaiting a version from the current -%% leader is kept; one forwarded to it is kept only when `Scope = moved`. With `all` (a -%% node came back) that forward may have died with the link, so it is re-sent; a -%% duplicate is decided idempotently but costs a second commit. +%% Re-sends unresolved intents, following the rules in the module header. +%% Registers and set_metas older than `TTL` (the caller's timeout) are dropped +%% without a reply, since nobody is waiting. A register already waiting on the +%% current leader is left alone. With `Scope = all`, a node has reconnected and +%% a forward to it may have been lost with the old link, so we send it again. A +%% duplicate is harmless but costs an extra commit. -spec redrive(route(), scope(), integer(), non_neg_integer(), t()) -> {t(), [effect()]}. redrive(Route, Scope, Now, TTL, T = #intents{all = All}) -> {T1, EffsRev} = lists:foldl( fun({Ref, I}, {Acc, Effs}) -> - {Acc1, New} = redrive_one(Ref, I, Route, Scope, Now, TTL, delete(Ref, Acc)), + {Acc1, New} = redrive_one(Ref, I, Route, Scope, Now, TTL, delete_intent(Ref, Acc)), {Acc1, lists:reverse(New, Effs)} end, {T, []}, - in_order(maps:to_list(All)) + sort_by_arrival(maps:to_list(All)) ), {T1, lists:reverse(EffsRev)}. redrive_one(Ref, I = #intent{kind = {register, _, _, _}, at = At}, Route, Scope, Now, TTL, T) -> case {Now - At >= TTL, I#intent.phase, Route} of {true, _, _} -> {T, []}; - {false, {awaiting_version, _V, L}, {remote, L}} -> {put(Ref, I, T), []}; - {false, {forwarded, L}, {remote, L}} when Scope =:= moved -> {put(Ref, I, T), []}; + {false, {awaiting_version, _V, _E, L}, {remote, L}} -> {put_intent(Ref, I, T), []}; + {false, {forwarded, L}, {remote, L}} when Scope =:= moved -> {put_intent(Ref, I, T), []}; {false, _, _} -> dispatch(make_ref(), I#intent{phase = awaiting_leader}, Route, T) end; redrive_one(Ref, I = #intent{kind = {unregister, _, _}}, Route, _Scope, _Now, _TTL, T) -> @@ -195,18 +219,19 @@ redrive_one(Ref, I = #intent{kind = {set_meta, _, _, _}}, Route, _Scope, Now, TT #intent{from = From, at = At} = I, case {Now - At >= TTL, I#intent.phase, Route} of {true, _, _} -> {T, []}; - {false, {forwarded, L}, {remote, L}} -> {put(Ref, I, T), []}; + {false, {forwarded, L}, {remote, L}} -> {put_intent(Ref, I, T), []}; {false, _, _} -> {T, [{reply, From, {error, no_leader}}]} end. -%% Drop timed-out register/set_meta intents so a long leaderless window cannot grow -%% the map. Unregisters are kept: their effect is owed whether or not anyone waits. +%% Drops timed-out registers and set_metas so the map can't grow during a long +%% stretch without a leader. Unregisters are kept, because the name must still be +%% removed even when nobody is waiting. -spec prune(integer(), non_neg_integer(), t()) -> t(). prune(Now, TTL, T = #intents{all = All}) -> maps:fold( fun (_Ref, #intent{kind = {unregister, _, _}}, Acc) -> Acc; - (Ref, #intent{at = At}, Acc) when Now - At >= TTL -> delete(Ref, Acc); + (Ref, #intent{at = At}, Acc) when Now - At >= TTL -> delete_intent(Ref, Acc); (_Ref, _I, Acc) -> Acc end, T, @@ -217,36 +242,39 @@ prune(Now, TTL, T = #intents{all = All}) -> %% Internal %% --------------------------------------------------------------------------- -%% Send a register/unregister where `Route` allows, keeping it while it owes something. +%% Sends a register or unregister along `Route`, and keeps the intent while it +%% still owes something. dispatch(_Ref, #intent{kind = Kind = {register, _, _, _}, from = From}, self, T) -> {T, [{enqueue, Kind, From}]}; dispatch(Ref, I = #intent{kind = Kind = {register, _, _, _}}, {remote, L}, T) -> - {put(Ref, I#intent{phase = {forwarded, L}}, T), [{forward, L, Ref, Kind}]}; + {put_intent(Ref, I#intent{phase = {forwarded, L}}, T), [{forward, L, Ref, Kind}]}; dispatch(Ref, I = #intent{kind = {register, _, _, _}}, none, T) -> - {put(Ref, I#intent{phase = awaiting_leader}, T), []}; -%% An unregister's caller is answered `ok` now; the removal follows as a retract. + {put_intent(Ref, I#intent{phase = awaiting_leader}, T), []}; +%% An unregister's caller gets `ok` right away. The removal follows as a retract. dispatch(_Ref, #intent{kind = {unregister, _Name, undefined}, from = From}, _Route, T) -> - %% Nothing was bound locally when it was asked: nothing to retract. - {T, reply_if_owed(From, ok)}; + %% The name wasn't bound locally when the caller asked, so there's nothing to retract. + {T, maybe_reply(From, ok)}; dispatch(_Ref, #intent{kind = Kind = {unregister, _, _}, from = From}, self, T) -> - {T, reply_if_owed(From, ok) ++ [{enqueue, Kind, answered}]}; + {T, maybe_reply(From, ok) ++ [{enqueue, Kind, answered}]}; dispatch(_Ref, #intent{kind = {unregister, Name, Pid}, from = From}, {remote, L}, T) -> - {T, reply_if_owed(From, ok) ++ [{retract, L, Name, Pid}]}; + {T, maybe_reply(From, ok) ++ [{retract, L, Name, Pid}]}; dispatch(Ref, I = #intent{kind = {unregister, _, _}, from = From}, none, T) -> - %% CP: answer now, keep the intent until a leader can take it. - {put(Ref, I#intent{phase = awaiting_leader, from = answered}, T), reply_if_owed(From, ok)}. + %% Answer now, and keep the intent until a leader can take it. + {put_intent(Ref, I#intent{phase = awaiting_leader, from = answered}, T), maybe_reply(From, ok)}. -reply_if_owed(answered, _Value) -> []; -reply_if_owed(From, Value) -> [{reply, From, Value}]. +maybe_reply(answered, _Value) -> []; +maybe_reply(From, Value) -> [{reply, From, Value}]. -put(Ref, I = #intent{phase = {awaiting_version, V, _L}}, T = #intents{all = A, deferred = D}) -> - T#intents{all = A#{Ref => I}, deferred = D#{Ref => V}}; -put(Ref, I, T = #intents{all = A, deferred = D}) -> +put_intent( + Ref, I = #intent{phase = {awaiting_version, V, E, _L}}, T = #intents{all = A, deferred = D} +) -> + T#intents{all = A#{Ref => I}, deferred = D#{Ref => {V, E}}}; +put_intent(Ref, I, T = #intents{all = A, deferred = D}) -> T#intents{all = A#{Ref => I}, deferred = maps:remove(Ref, D)}. -%% `{Ref, Intent}` pairs in arrival order. -in_order(Pairs) -> +%% Sorts `{Ref, Intent}` pairs into arrival order. +sort_by_arrival(Pairs) -> lists:sort(fun({_, A}, {_, B}) -> A#intent.seq =< B#intent.seq end, Pairs). -delete(Ref, T = #intents{all = A, deferred = D}) -> +delete_intent(Ref, T = #intents{all = A, deferred = D}) -> T#intents{all = maps:remove(Ref, A), deferred = maps:remove(Ref, D)}. diff --git a/src/dgen_registry/dgen_registry_member.erl b/src/dgen_registry/dgen_registry_member.erl index b2614b0..5b82c01 100644 --- a/src/dgen_registry/dgen_registry_member.erl +++ b/src/dgen_registry/dgen_registry_member.erl @@ -4,8 +4,9 @@ -define(DOCATTRS, ?OTP_RELEASE >= 27). -include("../../include/dgen_eta.hrl"). +-include("dgen_registry_member.hrl"). -eta_observe( - {state, [ + {data, [ member_id, leader, replica, @@ -19,93 +20,107 @@ -moduledoc false. -endif. -%% Per-node member of a `dgen_registry`: local name cache and consistent read/write -%% proxy. Section references (`§N.N`) point to `docs/design/dgen_registry_design.md`; -%% TLA+ action names to `formal/DgenRegistryReplication.tla`. +%% The per-node member of a `dgen_registry`. It keeps a local cache of the names and +%% proxies consistent reads and writes to the leader. `§N.N` refers to +%% `docs/design/dgen_registry_design.md`, and TLA+ action names refer to +%% `formal/DgenRegistryReplication.tla`. %% %% ## Roles %% -%% - Follower: applies the leader's `{names_batch, …}` / `{apply_names_snapshot, …}` -%% casts to its replica (a batch is one whole group commit, applied all or nothing), -%% and forwards writes and consistent reads to the leader as casts correlated by a -%% `Ref`. The loop never blocks on a call to another member: a blocking forward -%% could deadlock against the leader's replicate-before-ack wait. All replication -%% messages come from the leader process, so per-pair FIFO orders a snapshot before -%% any later batch, and a batch before the reply to a write it carries. -%% - Leader: the sole writer. At handoff it reconstructs the name map from the -%% freshest reachable replica (gathered off the loop; there is no durable name -%% state, §4.4), serves all writes through one group-commit pipeline, monitors -%% registered pids, and replicates each committed batch. A registration is acked -%% `yes` only once two members hold it (§8): a forwarding follower is the second -%% holder for free; a direct registration waits for a follower's `replicate_ack` -%% (degrade-open on timeout unless `strict_replication`). +%% - Follower: applies the leader's `{names_batch, …}` and `{apply_names_snapshot, …}` +%% casts to its replica. Each batch is one whole group commit, and we apply it all +%% or nothing. The follower forwards writes and consistent reads to the leader as +%% casts tagged with a `Ref`. The loop never blocks on a call to another member, +%% because a blocking forward could deadlock against the leader while it waits for +%% replica acks. All replication messages come from the leader process, so +%% per-pair FIFO delivers a snapshot before any later batch, and a batch before the +%% reply to a write it carries. +%% - Leader: the only writer. There is no durable name state (§4.4), so at handoff the +%% leader rebuilds the name map from the freshest replica it can reach. It gathers +%% those replicas off the loop. It serves all writes through one group-commit +%% pipeline, monitors registered pids, and replicates each committed batch. It only +%% answers a registration `yes` once two members hold it (§8). When a follower +%% forwarded the registration, that follower is already the second holder. A +%% direct registration waits for a follower's `replicate_ack`. On timeout the +%% leader answers `yes` anyway (degrade-open), unless `strict_replication` is set. %% -%% With no reachable leader a write is never answered `no`; the caller blocks as an +%% When no leader is reachable, we never answer a write `no`. The caller waits as an %% intent until a leader appears or its own call times out (§3). %% %% ## Where things live %% -%% - `dgen_registry_replica`: the `protected` ETS table (rows `{Name, Pid, Index, Data}`, -%% read lock-free by callers), inverted index, applied version and epoch, and the -%% batch/snapshot admission rules. This process is its only writer; it is not durable. -%% - `dgen_registry_intents`: requests this member accepted but cannot decide, and -%% their re-drive rules on leader change. -%% - `dgen_registry_pipeline`: the leader's commit queue, in-flight commit, pending -%% replicate acks, and the surrender rule for ops it cannot finish. -%% - `dgen_registry_conflicts`: release trail, kill budget, `degraded` flag (§5.6). +%% - `dgen_registry_replica`: the `protected` ETS table, the inverted index, the +%% applied version and epoch, and the rules for accepting a batch or snapshot. Rows +%% are `{Name, Pid, Index, Data}`, and callers read them without locking. This +%% process is the only writer. The table is not durable. +%% - `dgen_registry_intents`: requests this member accepted but can't decide itself, +%% and the rules for re-driving them when the leader changes. +%% - `dgen_registry_pipeline`: the leader's commit queue, the in-flight commit, the +%% pending replica acks, and the surrender rule for ops it can't finish. +%% - `dgen_registry_conflicts`: the release trail, the kill budget and the `degraded` +%% flag (§5.6). %% - `dgen_registry_connector`: keeps the distribution mesh in step with membership. %% - `dgen_registry_elector`: elects the leader and asks this member to assume. %% -%% Multi-key queries run on this member's mailbox, between batch applies, so they see a -%% whole-batch-consistent view (§6); single-key snapshot reads bypass the member. +%% Multi-key queries run in this member's mailbox, between batch applies, so they +%% never see half a batch (§6). Single-key snapshot reads skip the member entirely. %% %% ## States %% -%% A `gen_statem` whose state is a projection of data (`state_of/1`, recomputed after -%% every event by `next/2`), so it cannot drift: +%% This is a `gen_statem` whose state mirrors the `leader` and `assume_ref` fields. +%% Only three events write those fields, and each returns an explicit `next_state`: +%% - an `{elector_assume_and_distribute}` that starts a gather moves to `assuming`. +%% - that gather's `{assume_gathered}` moves to `leader`. +%% - an accepted `{apply_names_snapshot}` moves to the role the snapshot names. +%% Every other event returns `keep_state`. %% -%% searching no leader known. -%% assuming a handoff gather is in flight (`assume_ref` armed). -%% leader this member leads; entering runs `assume_leadership/1`, leaving runs -%% `relinquish_leadership/1`. -%% follower following a known leader. +%% searching No leader is known. +%% assuming A handoff gather is in flight, and `assume_ref` is set. +%% leader This member leads. Entering starts the monitor sweep and seeds the +%% watch sets. Leaving demonitors every registered pid. +%% follower We follow a known leader. %% -%% Gap/resync, leader reachability and the commit pipeline are deliberately data, not -%% state: each is orthogonal to role (a deposed-but-uninformed leader can still gap, -%% as the spec's RecvBcastGap has no leaderView guard). +%% Gap detection and resync, leader reachability and the commit pipeline are kept in +%% data on purpose, not in the state. None of them depends on the role. For example, a +%% leader that was deposed but hasn't heard yet can still see a gap, because the +%% spec's RecvBcastGap has no leaderView guard. %% %% *This documentation is LLM-generated. See the AI disclosure in README.md.* -%% Per-member bound on the handoff gather; a member that misses it is skipped. +%% How long the handoff gather waits for each member. We skip a member that misses it. -define(GATHER_TIMEOUT, 2000). -%% Pause between re-gathers while the assume gather is behind the durable version key -%% (the SafeAssume fence in the TLA+ spec); bounded overall by ?GATHER_TIMEOUT. +%% The pause between re-gathers while the assume gather is behind the durable version +%% key. This is the SafeAssume guard in the TLA+ spec. ?GATHER_TIMEOUT bounds the total +%% wait. -define(ASSUME_CATCHUP_INTERVAL, 25). -%% Best-effort bound on pulling durable subscriptions from the elector at handoff (§4.9). +%% How long we try to pull the durable subscriptions from the elector at handoff +%% (§4.9). This is best-effort. -define(SUBS_PULL_TIMEOUT, 1000). -%% Period of the maintenance pass pruning the release trail and kill budget (§5.6). -%% Trail pruning is suspended while any member is disconnected: it misses unregister -%% broadcasts entirely, so the trail must outlive the partition. +%% How often the maintenance pass prunes the release trail and kill budget (§5.6). +%% We stop pruning the trail while any member is disconnected. That member misses the +%% unregister broadcasts entirely, so the trail has to outlast the partition. -define(PRUNE_INTERVAL, 30000). -%% How long a follower waits after requesting a resync (it observed a gap in the -%% leader's broadcast stream) before it may request again. +%% How long a follower waits after asking for a resync before it may ask again. It asks +%% when it sees a gap in the leader's broadcast stream. -define(RESYNC_RETRY, 2000). -%% Names monitored per `{monitor_sweep, …}` step (see assume_leadership/1). +%% How many names each `{monitor_sweep, …}` step monitors. See the `leader` enter call. -define(MONITOR_SWEEP_CHUNK, 5000). -%% Period of the leader's replication heartbeat: an empty `{names_batch, [], …}` at its -%% applied version. Without it a follower that lost the stream's tail (or its resync -%% request) would not detect the gap until the next write. +%% How often the leader sends its replication heartbeat, an empty +%% `{names_batch, [], …}` at its applied version. Without it, a follower that lost the +%% end of the stream, or lost its resync request, wouldn't notice the gap until the +%% next write. -define(REPLICA_HEARTBEAT_INTERVAL, 5000). -%% Delay before re-enqueueing (or, if deposed, surrendering) the ops of a failed batch, -%% so a persistently failing backend does not spin. +%% How long we wait before re-enqueueing the ops of a failed batch, or surrendering +%% them if we were deposed. The delay keeps a backend that keeps failing from spinning. -define(REQUEUE_DELAY, 1000). -%% Bound on the per-peer aliveness probe used when adjudicating a suspected conflict. +%% How long we wait for each peer to answer a liveness probe while checking a +%% suspected conflict. -define(ALIVE_PROBE_TIMEOUT, 1000). -export([start_link/2]). @@ -116,67 +131,16 @@ terminate/3, code_change/4 ]). +%% State functions. handle_event/4 calls them, not gen_statem. +-export([ + searching/3, + assuming/3, + leader/3, + follower/3 +]). %% Exported for unit tests of the batch planner. -export([plan_batch/4]). -%% Presence subscription query, internal tagged form (see dgen_registry:query/0). --type query() :: {'all', #{term() => term()}}. - -%% Application-supplied presence subscription id (§4.9). --type sub_id() :: term(). - --record(state, { - member_id :: dgen_registry_elector:member_id(), - %% Discovered once after init; the `one_for_all` supervisor restarts us with the - %% elector, so this pid is never stale. - elector :: pid() | undefined, - %% Backend handle for the leader's fenced version-key commit (dgen_registry_names). - tenant :: dgen_backend:tenant(), - tuid :: dgen_server:tuid(), - leader :: dgen_registry_elector:member_id() | undefined, - %% Local replica: ETS rows, inverted index, applied version, epoch. - replica :: dgen_registry_replica:t(), - %% Leader-only (§4.9): `#{SubId => {Watch, Notify}}`, seeded at assume and advanced - %% by `{presence_update, …}`. - subs = #{} :: #{sub_id() => {query(), query()}}, - %% Leader-only: names matching each watch query; diffed per commit for joined/left. - sub_matches = #{} :: #{sub_id() => #{term() => pid()}}, - %% Leader-only: pids matching each notify query; a pid entering it is sent the - %% current watch snapshot. - notify_matches = #{} :: #{sub_id() => #{term() => pid()}}, - %% Peer-member monitors (all members) - members :: #{dgen_registry_elector:member_id() => reference()}, - monitors :: #{reference() => dgen_registry_elector:member_id()}, - %% Registered-process monitors (leader only). No reverse map: each monitor is - %% tagged `{down, Name}`, so its DOWN names the binding. - name_to_ref :: #{term() => reference()}, - %% Requests awaiting a leader's decision; see dgen_registry_intents. - intents = dgen_registry_intents:new() :: dgen_registry_intents:t(), - %% Leader-only write ops not yet answered; see dgen_registry_pipeline. - pipeline = dgen_registry_pipeline:new() :: dgen_registry_pipeline:t(), - %% The epoch in which a fenced commit last made this member re-announce itself - %% (rejoin_once_fenced/1), so it does so at most once per epoch. - fence_rejoined = undefined :: undefined | non_neg_integer(), - %% Token for our `{join, Self, Token}`; refreshed on nodeup so a stale - %% `{member_down, Self, OldToken}` is discarded. - join_token :: reference(), - %% Peer join tokens, echoed in `{member_down, PeerId, Token}` so the elector can - %% discard a stale DOWN. - peer_tokens :: #{dgen_registry_elector:member_id() => reference()}, - %% Outstanding resync request timer; cleared when any snapshot arrives. - resync_timer = undefined :: undefined | reference(), - %% Ever synced (snapshot applied or leadership assumed)? Joins announce - %% `fresh = not synced`; the first sync re-announces with `fresh = false`. - synced = false :: boolean(), - %% Tags the in-flight off-loop handoff gather (§5.7); `{assume_gathered, …}` applies - %% only if it still matches, so a superseded gather is harmless. - assume_ref = undefined :: undefined | reference(), - %% Conflict repair state (§5.6); see dgen_registry_conflicts. - conflicts = dgen_registry_conflicts:new() :: dgen_registry_conflicts:t(), - %% Per-registry tuning options, resolved through dgen_config. - config = #{} :: dgen_config:config() -}). - %% --------------------------------------------------------------------------- %% Public API %% --------------------------------------------------------------------------- @@ -198,22 +162,23 @@ callback_mode() -> init(#{name := Name, tenant := Tenant, tuid := Tuid} = Args) -> Config = maps:get(config, Args, #{}), MemberId = {node(), Name}, - %% Recreated empty on every (re)start (no heir) and rebuilt from the leader's - %% snapshot. + %% The table has no heir, so every start or restart creates it empty. The leader's + %% snapshot fills it. Replica = dgen_registry_replica:new(dgen_registry:names_table(Name)), - %% On `{nodeup, _}` the member re-announces its `{join}` (partition heal). + %% When a partition heals, `{nodeup, _}` makes the member send its `{join}` again. net_kernel:monitor_nodes(true), - %% Resolve the cached telemetry lookup now, while the tree starts: done lazily it is - %% a synchronous `code_server` call inside whichever member emits first, which under - %% `eta_run` simulation blocks on a process the scheduler does not own. - _ = telemetry_available(), - arm(?PRUNE_INTERVAL, prune_released), - arm(?REPLICA_HEARTBEAT_INTERVAL, replica_heartbeat), - %% Announcing needs the elector's pid; `discover_elector` runs as an internal event - %% because `supervisor:which_children/1` would deadlock while the supervisor still - %% waits on this init/1. + %% Look up telemetry now, while the tree starts, and cache the answer. Done lazily, + %% it would be a synchronous `code_server` call inside whichever member emits + %% first. Under `eta_run` simulation that call blocks on a process the scheduler + %% doesn't own. + _ = is_telemetry_available(), + schedule_timer(?PRUNE_INTERVAL, prune_released), + schedule_timer(?REPLICA_HEARTBEAT_INTERVAL, replica_heartbeat), + %% We need the elector's pid to announce ourselves. `discover_elector` runs as an + %% internal event, because `supervisor:which_children/1` would deadlock while the + %% supervisor is still waiting on this init/1. {ok, searching, - #state{ + #data{ member_id = MemberId, elector = undefined, tenant = Tenant, @@ -230,736 +195,743 @@ init(#{name := Name, tenant := Tenant, tuid := Tuid} = Args) -> [{next_event, internal, discover_elector}]}. %% --------------------------------------------------------------------------- -%% handle_event/4 — dispatch, and the derived state -%% --------------------------------------------------------------------------- - -%% -- state enter: the leadership lifecycle, in one place --------------------- - -%% The assume/relinquish pairing lives on the role edges so no transition path can -%% forget half of it. Deliberately not here: redrive_intents (fires on non-edge -%% leadership events too) and fail_pending_acks (stays in do_leader_changed; a -%% re-assumed leader's stale acks are already fenced by BatchRef). A leader entering -%% `assuming` relinquishes at assume start; a death in the gather window is caught by -%% the next sweep's immediate `noproc` DOWN. -handle_event(enter, OldState, leader, Data) when OldState =/= leader -> - {keep_state, assume_leadership(Data)}; -handle_event(enter, leader, NewState, Data) when NewState =/= leader -> - {keep_state, relinquish_leadership(Data)}; -handle_event(enter, _OldState, _NewState, Data) -> - {keep_state, Data}; -%% -- the assume lifecycle --------------------------------------------------- - -%% A continuing leader absorbs a join on the fast path; otherwise a genuine change. -%% The epoch guard stays on Data: the `leader` state does not say which term. -handle_event( - {call, From}, - {elector_assume_and_distribute, MemberId, AllIds, Tokens, FreshIds, Epoch}, - State, - Data = #state{synced = Synced} -) -> - {reply, Reply, Data1} = - case - State =:= leader andalso Synced andalso MemberId =/= undefined andalso - epoch(Data) =:= Epoch - of - true -> assume_fast_path(MemberId, AllIds, Tokens, FreshIds, Epoch, Data); - false -> assume_genuine(MemberId, AllIds, Tokens, FreshIds, Epoch, Data) - end, - next(Data1, [{reply, From, Reply}]); -%% The gather continuation applies only in `assuming` and only for the assume that -%% armed it (the ref); otherwise it was superseded and is dropped. -handle_event( - info, - {assume_gathered, {_, _, _, _, _, Ref}, _, _, _, _, _, _} = Msg, - assuming, - Data = #state{assume_ref = Ref} -) -> - {noreply, Data1} = handle_info(Msg, Data), - next(Data1, []); -handle_event( - info, {assume_gathered, _Ctx, _Self, _Freshest, _MaxV, _Peers, _Subs, _Alive}, _AnyState, Data -) -> - {keep_state, Data}; -handle_event( - info, - {joiner_gathered, MemberId, _AllIds, _Tokens, Epoch, _Result} = Msg, - leader, - Data = #state{members = Members} -) when is_map_key(MemberId, Members) -> - case epoch(Data) =:= Epoch of - true -> - {noreply, Data1} = handle_info(Msg, Data), - next(Data1, []); - false -> - {keep_state, Data} - end; -handle_event(info, {joiner_gathered, _M, _A, _T, _E, _R}, _AnyState, Data) -> - {keep_state, Data}; -%% -- leader-only periodic work --------------------------------------------- - -%% Everyone keeps the timer running so a later election heartbeats immediately. -handle_event(info, replica_heartbeat, leader, Data) -> - broadcast_heartbeat(Data), - arm(?REPLICA_HEARTBEAT_INTERVAL, replica_heartbeat), - {keep_state, Data}; -handle_event(info, replica_heartbeat, _AnyState, Data) -> - arm(?REPLICA_HEARTBEAT_INTERVAL, replica_heartbeat), - {keep_state, Data}; -%% Monitor sweep steps apply only in the term that armed them. -handle_event(info, {monitor_sweep, Epoch, _Cont} = Msg, leader, Data) -> - case epoch(Data) =:= Epoch of - true -> - {noreply, Data1} = handle_info(Msg, Data), - next(Data1, []); - false -> - {keep_state, Data} - end; -handle_event(info, {monitor_sweep, _Epoch, _Cont}, _AnyState, Data) -> - {keep_state, Data}; -%% -- leader-only protocol serves ------------------------------------------- - -%% Only a member that believes it leads answers a resync request. -handle_event(cast, {resync_req, _FollowerId} = Msg, leader, Data) -> - {noreply, Data1} = handle_cast(Msg, Data), - next(Data1, []); -handle_event(cast, {resync_req, _FollowerId}, _AnyState, Data) -> - {keep_state, Data}; -%% A forwarded registration reaching a member mid-assume is postponed until the assume -%% resolves; if superseded, it stays silent and the forwarder re-drives. -handle_event(cast, {register_req, _Ref, _FollowerId, _Name, _Pid, _Meta}, assuming, Data) -> - {keep_state, Data, [postpone]}; -%% A non-leader stays silent; the follower re-drives the unregister intent. -handle_event(cast, {unregister_req, _Ref, _FollowerId, _Name} = Msg, leader, Data) -> - {noreply, Data1} = handle_cast(Msg, Data), - next(Data1, []); -handle_event(cast, {unregister_req, _Ref, _FollowerId, _Name}, _AnyState, Data) -> - {keep_state, Data}; -%% -- the generic shim ------------------------------------------------------- - -handle_event({call, From}, Msg, _State, Data) -> - case handle_call(Msg, From, Data) of - {reply, Reply, Data1} -> next(Data1, [{reply, From, Reply}]); - {noreply, Data1} -> next(Data1, []) - end; -handle_event(cast, Msg, _State, Data) -> - {noreply, Data1} = handle_cast(Msg, Data), - next(Data1, []); -handle_event(info, Msg, _State, Data) -> - {noreply, Data1} = handle_info(Msg, Data), - next(Data1, []); -handle_event(internal, Continue, _State, Data) -> - {noreply, Data1} = handle_continue(Continue, Data), - next(Data1, []). - -%% `assuming` wins: a mid-assume member clears its leader view on purpose. -state_of(#state{assume_ref = Ref}) when Ref =/= undefined -> assuming; -state_of(#state{leader = undefined}) -> searching; -state_of(#state{leader = Self, member_id = Self}) -> leader; -state_of(#state{}) -> follower. - -%% A same-state `next_state` keeps timeouts and fires no enter call, so deriving on -%% every event is harmless. -next(Data, Actions) -> - {next_state, state_of(Data), Data, Actions}. - -epoch(#state{replica = R}) -> dgen_registry_replica:epoch(R). -applied_version(#state{replica = R}) -> dgen_registry_replica:version(R). - -%% --------------------------------------------------------------------------- -%% handle_continue/2 -%% --------------------------------------------------------------------------- - -%% Safe here: the supervisor is no longer blocked on our init/1 (see init/1). -handle_continue( - discover_elector, State = #state{member_id = {_Node, Name} = MemberId, join_token = Token} -) -> - Elector = dgen_registry:elector_pid(Name), - true = is_pid(Elector), - %% `fresh = true`: never synced, so this member cannot be hiding any bindings. - dgen_server:cast(Elector, {join, MemberId, Token, true}), - {noreply, State#state{elector = Elector}}. - -%% --------------------------------------------------------------------------- -%% handle_call/3 +%% handle_event/4 — enter, events shared by every state, then the state functions %% --------------------------------------------------------------------------- -%% Readiness probe (dgen_registry:await_ready/2): a leader is known and state synced. -handle_call(ready, _From, State = #state{leader = Leader, synced = Synced}) -> - {reply, Leader =/= undefined andalso Synced, State}; -%% Status probe (dgen_registry:status/1). Reports this member's own *belief* about -%% leadership, which can lag the elector's committed view during a handoff (§5.1). -handle_call(status, _From, State) -> - #state{member_id = Self, leader = Leader, synced = Synced} = State, - {reply, - #{ - member_id => Self, - leader => Leader, - is_leader => Leader =:= Self, - epoch => epoch(State), - synced => Synced, - applied_version => applied_version(State) - }, - State}; -%% ---- Name registration ---------------------------------------------------- - -%% Routing (leader / follower / no leader) is in route_register/5; the only immediate -%% reply is an adjudicated `no`. -handle_call({register, LogicalName, Pid, Meta}, From, State) -> - {noreply, route_register(LogicalName, Pid, Meta, From, State)}; -%% ---- Name unregistration ---------------------------------------------------- - -handle_call({unregister, LogicalName}, From, State) -> - route_unregister(LogicalName, From, State); -%% ---- Metadata write (set_metadata) ---------------------------------------- - -handle_call( - {set_metadata, LogicalName, Index, Data}, - From, - State = #state{leader = Leader, member_id = Leader} -) -> - {noreply, enqueue_op({set_meta, LogicalName, Index, Data, {meta, From}}, State)}; -%% Follower: forward as an intent. The reply routes back through this member so the -%% leader's `{names_batch}` (FIFO, ahead of it) has updated our row first: -%% read-after-write for a local snapshot read. -handle_call({set_metadata, LogicalName, Index, Data}, From, State) -> - {noreply, submit_intent({set_meta, LogicalName, Index, Data}, From, State)}; -%% ---- Consistent metadata read (leader only) -------------------------------- - -handle_call( - {get_metadata, LogicalName}, - From, - State = #state{leader = Leader, member_id = Leader, replica = R} -) -> - %% Fenced (Guarantee 5): the row is captured now, and a spawned helper verifies - %% leadership against the durable leader key before replying, so a deposed leader - %% cannot answer authoritatively. - reply_fenced(From, dgen_registry_replica:lookup_metadata(R, LogicalName), undefined, State), - {noreply, State}; -handle_call( - {get_metadata, LogicalName}, - From, - State = #state{leader = Leader} -) when Leader =/= undefined -> - %% The leader answers `From` directly; unreachable leader answers now. - case member_reachable(Leader) of - true -> - cast_to_member(Leader, {get_metadata_req, LogicalName, From}), - {noreply, State}; - false -> - {reply, undefined, State} - end; -handle_call({get_metadata, _LogicalName}, _From, State) -> - {reply, undefined, State}; -%% ---- Indexed query (on the mailbox, so batch-consistent, §6) --------------- - -handle_call({query, Constraints}, _From, State = #state{replica = R}) -> - {reply, dgen_registry_replica:query_constraints(R, Constraints), State}; -%% Consistent query: on the leader, fenced like the other consistent reads; otherwise -%% forwarded and answered by the leader directly. -handle_call( - {query_consistent, Constraints}, - From, - State = #state{leader = Leader, member_id = Leader, replica = R} -) -> - reply_fenced(From, dgen_registry_replica:query_constraints(R, Constraints), [], State), - {noreply, State}; -handle_call( - {query_consistent, Constraints}, From, State = #state{leader = Leader} -) when Leader =/= undefined -> - case member_reachable(Leader) of - true -> - cast_to_member(Leader, {query_req, Constraints, From}), - {noreply, State}; - false -> - {reply, [], State} - end; -handle_call({query_consistent, _Constraints}, _From, State) -> - {reply, [], State}; -%% ---- Consistent read (leader only) ---------------------------------------- - -handle_call( - {whereis, LogicalName}, - From, - State = #state{leader = Leader, member_id = Leader} -) -> - %% Fenced, as in the {get_metadata} leader clause. - reply_fenced( - From, dgen_registry_replica:lookup_pid(State#state.replica, LogicalName), undefined, State - ), - {noreply, State}; -handle_call( - {whereis, LogicalName}, - From, - State = #state{leader = Leader} -) when Leader =/= undefined -> - %% The leader replies to `From` directly so this loop never blocks on it. - case member_reachable(Leader) of - true -> - cast_to_member(Leader, {whereis_req, LogicalName, From}), - {noreply, State}; - false -> - {reply, undefined, State} - end; -handle_call({whereis, _LogicalName}, _From, State) -> - {reply, undefined, State}; -%% ---- Snapshot read --------------------------------------------------------- -%% Served lock-free by the caller (`dgen_registry:whereis_name/1`); no handler here. - -%% ---- Handoff gather (any member) ------------------------------------------- - -%% Our records, applied version and release trail, for a new leader's freshest-wins -%% reconstruction (§5.7 / §4.4) and trail merge (§5.6). -handle_call( - get_names_snapshot, - _From, - State = #state{replica = R, conflicts = C} -) -> - %% Shipped as an off-heap binary; see dgen_registry_replica:encode/1. - Released = dgen_registry_conflicts:trail(C), - {reply, {dgen_registry_replica:encode(R), dgen_registry_replica:version(R), Released}, - State}; - -handle_call(_Request, _From, State) -> - {reply, {error, unknown_call}, State}. - -%% ---- Elector calls (dispatched from handle_event) --------------------------- - -%% Bodies of `{elector_assume_and_distribute, …}`. `MemberId` is the joiner that -%% triggered it (`undefined` for a member_down); `AllIds` the full member list. +%% Taking and giving up leadership happens when the role changes, in the enter clauses +%% of the state functions. leader/3 takes it, and non_leader_enter/2 gives it up. +handle_event(enter, OldState, State, Data) -> + apply(?MODULE, State, [enter, OldState, Data]); +%% -- calls ------------------------------------------------------------------ + +%% Answers the readiness probe from dgen_registry:await_ready/2. We are ready once we +%% know a leader and have synced. +handle_event({call, From}, ready, _State, Data) -> + #data{leader = Leader, synced = Synced} = Data, + {keep_state_and_data, [{reply, From, Leader =/= undefined andalso Synced}]}; +%% Answers the status probe from dgen_registry:status/1. It reports what this member +%% *believes* about leadership, which can lag behind the elector during a handoff +%% (§5.1). +handle_event({call, From}, status, _State, Data) -> + #data{ + member_id = Self, + leader = Leader, + synced = Synced, + replica = #replica{epoch = Epoch, version = Applied} + } = Data, + Status = #{ + member_id => Self, + leader => Leader, + is_leader => Leader =:= Self, + epoch => Epoch, + synced => Synced, + applied_version => Applied + }, + {keep_state_and_data, [{reply, From, Status}]}; +%% route_register/5 decides where the registration goes. The only reply sent right +%% away is a `no` from a leader that refuses it. +handle_event({call, From}, {register, LogicalName, Pid, Meta}, _State, Data) -> + {keep_state, route_register(LogicalName, Pid, Meta, From, Data)}; +handle_event({call, From}, {unregister, LogicalName}, _State, Data) -> + {noreply, Data1} = route_unregister(LogicalName, From, Data), + {keep_state, Data1}; +%% Runs an indexed query. It runs in the mailbox, so it never sees half a batch (§6). +handle_event({call, From}, {query, Constraints}, _State, #data{replica = R}) -> + {keep_state_and_data, [{reply, From, dgen_registry_replica:query_constraints(R, Constraints)}]}; +%% Snapshot reads don't come here. The caller reads the table directly, without locks, +%% in `dgen_registry:whereis_name/1`. %% -%% Fast path: a continuing, synced leader for this epoch is authoritative (every -%% follower is a prefix of its stream), so it skips the gather and re-snapshotting -%% the followers; it only monitors and onboards the joiner. -assume_fast_path(MemberId, AllIds, Tokens, FreshIds, Epoch, State = #state{member_id = Self}) -> - State1 = merge_peer_tokens( - Tokens, add_member_monitors(extra_member_ids(MemberId, AllIds, Self), State) - ), - %% A fresh joiner holds no bindings: onboard now. A rejoining previously-synced - %% member may hold divergent bindings (§5.6 partition heal), so gather its replica - %% off the loop and adjudicate it in {joiner_gathered} first (best-effort). - case MemberId =:= Self orelse lists:member(MemberId, FreshIds) of - true -> - {reply, ok, onboard_joiner(MemberId, AllIds, Tokens, State1)}; - false -> - spawn_joiner_gather(MemberId, AllIds, Tokens, Epoch, State1), - {reply, ok, State1} - end. - -assume_genuine(MemberId, AllIds, Tokens, FreshIds, Epoch, State = #state{member_id = Self}) -> - %% Genuine change: the freshest-wins gather is a network fan-out, so it runs off - %% the loop and the elector is answered `ok` now (an inline gather would time out - %% its call and stall queued membership changes). Our own records and version are - %% captured now so a broadcast landing mid-gather cannot skew freshest-wins. - SelfRecords = dgen_registry_replica:records(State#state.replica), - OtherIds = lists:delete(Self, AllIds), - Ref = make_ref(), - spawn_assume_gather( - {MemberId, AllIds, Tokens, FreshIds, Epoch, Ref}, - SelfRecords, - applied_version(State), - OtherIds, - State#state.elector, - self(), - State#state.tenant, - State#state.tuid, - State#state.conflicts - ), - %% No leader for the window: writes block and are re-driven by the continuation - %% rather than forwarded to the old, possibly-dead leader. - {reply, ok, State#state{assume_ref = Ref, leader = undefined}}. - -%% --------------------------------------------------------------------------- -%% handle_cast/2 -%% --------------------------------------------------------------------------- - -%% Leader: a forwarded registration, answered via {register_reply} once committed. -handle_cast( - {register_req, Ref, FollowerId, LogicalName, Pid, Meta}, - State = #state{leader = Leader, member_id = Leader} -) -> - case reject_new_when_degraded(LogicalName, State) of - true -> - cast_to_member(FollowerId, {register_reply, Ref, no}), - {noreply, State}; - false -> - {noreply, enqueue_op({add, LogicalName, Pid, Meta, {forward, FollowerId, Ref}}, State)} - end; -%% Not the leader: stay silent rather than give a non-adjudicated `no` (§3); the -%% follower re-drives its intent. -handle_cast({register_req, _Ref, _FollowerId, _LogicalName, _Pid, _Meta}, State) -> - {noreply, State}; -%% Leader: a forwarded unregister, answered via {unregister_reply} once committed. -handle_cast( - {unregister_req, Ref, FollowerId, LogicalName}, - State = #state{leader = Leader, member_id = Leader, replica = R} -) -> - ReleasedPid = dgen_registry_replica:lookup_pid(R, LogicalName), - {noreply, - enqueue_op( - {remove, LogicalName, ReleasedPid, {forward_unreg, FollowerId, Ref}}, - row_delete(State, LogicalName) - )}; -%% Follower: a forwarded unregister committed (FIFO: after its {names_batch}). -handle_cast({unregister_reply, Ref}, State) -> - {noreply, intent_reply(Ref, unregister, State)}; -%% Leader: a forwarded set_metadata, answered via {set_meta_reply} once committed. -handle_cast( - {set_meta_req, Ref, FollowerId, LogicalName, Index, Data}, - State = #state{leader = Leader, member_id = Leader} -) -> - {noreply, - enqueue_op({set_meta, LogicalName, Index, Data, {forward_meta, FollowerId, Ref}}, State)}; -%% Not the leader (stale belief): reject so the caller retries. -handle_cast({set_meta_req, Ref, FollowerId, _LogicalName, _Index, _Data}, State) -> - cast_to_member(FollowerId, {set_meta_reply, Ref, {error, no_leader}}), - {noreply, State}; -%% Follower: FIFO puts the matching {names_batch} ahead of this, so read-after-write -%% holds for a local get_metadata. -handle_cast({set_meta_reply, Ref, Result}, State) -> - {noreply, intent_reply(Ref, {set_meta, Result}, State)}; -%% Leader: forwarded consistent reads answer the caller's `From` directly, fenced. -handle_cast( - {get_metadata_req, LogicalName, From}, - State = #state{leader = Leader, member_id = Leader, replica = R} -) -> - reply_fenced(From, dgen_registry_replica:lookup_metadata(R, LogicalName), undefined, State), - {noreply, State}; -handle_cast({get_metadata_req, _LogicalName, From}, State) -> - gen_server:reply(From, undefined), - {noreply, State}; -handle_cast( - {query_req, Constraints, From}, - State = #state{leader = Leader, member_id = Leader, replica = R} -) -> - reply_fenced(From, dgen_registry_replica:query_constraints(R, Constraints), [], State), - {noreply, State}; -handle_cast({query_req, _Constraints, From}, State) -> - gen_server:reply(From, []), - {noreply, State}; -handle_cast( - {whereis_req, LogicalName, From}, - State = #state{leader = Leader, member_id = Leader, replica = R} -) -> - reply_fenced(From, dgen_registry_replica:lookup_pid(R, LogicalName), undefined, State), - {noreply, State}; -%% Not the leader (stale belief): `undefined`; the caller can retry. -handle_cast({whereis_req, _LogicalName, From}, State) -> - gen_server:reply(From, undefined), - {noreply, State}; -%% Follower: a forwarded registration's verdict at commit Version. A `yes` is -%% version-guarded (§5.5): answered only once our replica has applied Version, so we -%% are a second holder the handoff gather (§5.7) is guaranteed to see. Normally true -%% (FIFO: the batch precedes the reply); if the batch was gap-refused the ack waits -%% for the resync. See dgen_registry_intents. -handle_cast({register_reply, Ref, Result, Version}, State) -> - {noreply, intent_reply(Ref, {register, Result, Version}, State)}; -%% Rolling upgrade: unversioned reply from a pre-version leader; `legacy`, not 0 (see -%% dgen_registry_intents:on_reply/4). -handle_cast({register_reply, Ref, Result}, State) -> - {noreply, intent_reply(Ref, {register, Result, legacy}, State)}; -%% Follower: confirm we hold a batch's bindings (FIFO: after its {names_batch}). -%% Version-guarded: if we have not applied that version (gap, stale epoch) we must not -%% claim to hold them. -handle_cast( - {replicate_sync, BatchRef, LeaderId, Version}, - State = #state{member_id = Self} -) -> - case applied_version(State) >= Version of +%% Answers the handoff gather, on any member. We return our records, applied version +%% and release trail. A new leader uses them to rebuild the names from the freshest +%% replica (§5.7, §4.4) and to merge trails (§5.6). We also say whether the leader we +%% followed is gone, which tells the new leader our replica can't catch up any further. +handle_event({call, From}, get_names_snapshot, _State, Data) -> + #data{replica = R = #replica{version = Version}, conflicts = #conflicts{trail = Released}} = + Data, + %% We send the records as an off-heap binary. See dgen_registry_replica:encode/1. + Reply = {dgen_registry_replica:encode(R), Version, Released, Data#data.leader_down}, + {keep_state_and_data, [{reply, From, Reply}]}; +%% -- casts ------------------------------------------------------------------ + +%% Follower: the leader committed an unregister we forwarded. FIFO delivers its +%% {names_batch} first. +handle_event(cast, {unregister_reply, Ref}, _State, Data) -> + {keep_state, apply_intent_reply(Ref, unregister, Data)}; +%% Follower: FIFO delivers the matching {names_batch} before this reply, so a local +%% get_metadata right after it sees the new value. +handle_event(cast, {set_meta_reply, Ref, Result}, _State, Data) -> + {keep_state, apply_intent_reply(Ref, {set_meta, Result}, Data)}; +%% Follower: the leader's answer to a registration we forwarded, committed at +%% `Version` in `Epoch`. We only pass a `yes` on once our replica has applied +%% `Version` in that epoch (§5.5). That makes us a second holder the handoff gather +%% (§5.7) is sure to see. Usually FIFO has already delivered the batch. If we refused +%% the batch because of a gap, the `yes` waits for the resync. See +%% dgen_registry_intents. +handle_event(cast, {register_reply, Ref, Result, Version, Epoch}, _State, Data) -> + {keep_state, apply_intent_reply(Ref, {register, Result, Version, Epoch}, Data)}; +%% Follower: the leader refused a registration we forwarded without committing +%% anything, so there is no version. Only a `no` arrives without a version. A `yes` +%% always carries one. +handle_event(cast, {register_reply, Ref, no}, _State, Data) -> + {keep_state, apply_intent_reply(Ref, {register, no, undefined, undefined}, Data)}; +%% Follower: confirms to the leader that we hold a batch's bindings. FIFO delivers its +%% {names_batch} first. If we haven't applied that version, because of a gap or a stale +%% epoch, we must not claim to hold them. +handle_event(cast, {replicate_sync, BatchRef, LeaderId, Version}, _State, Data) -> + #data{member_id = Self, replica = #replica{version = Applied}} = Data, + case Applied >= Version of true -> cast_to_member(LeaderId, {replicate_ack, BatchRef, Self}); false -> ok end, - {noreply, State}; -%% Leader: once `register_replicas` distinct followers ack, answer the batch's direct -%% registrations `yes`. -handle_cast({replicate_ack, BatchRef, FollowerId}, State = #state{pipeline = P}) -> - case dgen_registry_pipeline:ack(BatchRef, FollowerId, P) of + keep_state_and_data; +%% Leader: once `register_replicas` different followers have acked, we answer the +%% batch's direct registrations `yes`. +handle_event(cast, {replicate_ack, BatchRef, FollowerId}, _State, Data = #data{pipeline = P}) -> + case dgen_registry_pipeline:record_ack(BatchRef, FollowerId, P) of {done, Direct, TimerRef, P1} -> erlang:cancel_timer(TimerRef), - deliver_direct(Direct), - {noreply, State#state{pipeline = P1}}; + deliver_direct_yes(Direct), + {keep_state, Data#data{pipeline = P1}}; {waiting, P1} -> - {noreply, State#state{pipeline = P1}} + {keep_state, Data#data{pipeline = P1}} end; -%% Replication broadcast: one message per committed batch. dgen_registry_replica -%% admits it only if contiguous (PrevV), so our replica stays a prefix of the leader's -%% stream; a gap requests a resync. Applying it as a unit keeps "same applied_version -%% implies same content" true (see broadcast_batch/5). -handle_cast({names_batch, Ops, Epoch, PrevV, Version, LeaderId}, State = #state{replica = R}) -> - State1 = +%% The replication broadcast, one message per committed batch. dgen_registry_replica +%% only accepts it if `PrevV` matches our version, so our replica stays a prefix of the +%% leader's stream. On a gap we ask for a resync. Applying the batch as a unit keeps +%% "same applied_version means same content" true. See broadcast_batch/5. +handle_event( + cast, {names_batch, Ops, Epoch, PrevV, Version, LeaderId}, _State, Data = #data{replica = R} +) -> + Data1 = case dgen_registry_replica:offer_batch(Epoch, PrevV, Version, Ops, R) of {applied, R1} -> - intents_advanced(record_released(Ops, cancel_resync(State#state{replica = R1}))); + release_deferred_acks(record_releases(Ops, cancel_resync(Data#data{replica = R1}))); gap -> - request_resync(LeaderId, State); + request_resync(LeaderId, Data); _StaleEpochOrDuplicate -> - State + Data end, - {noreply, State1}; -%% Leader: answer a resync request with a snapshot at our epoch and applied version. -handle_cast( - {resync_req, FollowerId}, - State = #state{leader = Leader, member_id = Leader, replica = R} -) -> - RecordsBin = dgen_registry_replica:encode(R), - cast_to_member( - FollowerId, - {apply_names_snapshot, RecordsBin, Leader, [], #{}, dgen_registry_replica:epoch(R), - dgen_registry_replica:version(R)} - ), - {noreply, State}; -%% A pid-guarded retract (strict_replication fail-closed, §8) forwarded by a member -%% that could not commit it, so a binding answered `no` cannot live on. The pid guard -%% never clobbers a name since re-registered. -handle_cast( - {retract_req, LogicalName, Pid}, State = #state{leader = Leader, member_id = Leader} -) -> - {noreply, enqueue_op({retract, LogicalName, Pid}, State)}; -handle_cast({retract_req, LogicalName, Pid}, State = #state{leader = Leader}) when - Leader =/= undefined --> - cast_to_member(Leader, {retract_req, LogicalName, Pid}), - {noreply, State}; -handle_cast({retract_req, LogicalName, Pid}, State) -> - %% No leader (likely mid-handoff): hold it as an answered unregister intent so - %% it is re-driven rather than lost. - {noreply, submit_intent({unregister, LogicalName, Pid}, answered, State)}; -%% Leadership snapshot from the new leader: replica, leader transition and member -%% monitors apply in one cast, so nothing interleaves. The version re-baselines us. -handle_cast( + {keep_state, Data1}; +%% A snapshot from the new leader. The replica, the leader change and the member +%% monitors all arrive in one cast, so nothing can slip in between them. Its version +%% becomes our new starting point. +handle_event( + cast, {apply_names_snapshot, RecordsPayload, NewLeader, ExtraMembers, Tokens, Epoch, Version}, - State = #state{member_id = Self, leader = OldLeader, replica = R} + _State, + Data ) -> - %% Version-monotonic, not merely epoch-guarded: a late snapshot can never roll the - %% replica backward (the handoff-gather race in the TLA+ spec). A refused one is - %% harmless; gap/resync re-baselines us. + #data{member_id = Self, leader = OldLeader, replica = R} = Data, + %% We check the version as well as the epoch, so a late snapshot can never move + %% the replica backward. The TLA+ spec found this race in the handoff gather. + %% Refusing a snapshot is harmless, because gap detection and resync catch us up. Records = dgen_registry_replica:decode_records(RecordsPayload), case dgen_registry_replica:offer_snapshot(Epoch, Version, Records, R) of {accepted, R1} -> - %% Replica before the leader transition: assume_leadership reads the - %% names back from the table. - State0 = cancel_resync(State#state{replica = R1}), - State1 = do_leader_changed(NewLeader, OldLeader, Self, State0), - %% Supersedes any in-flight assume gather. - State2 = State1#state{assume_ref = undefined}, - State3 = add_member_monitors(ExtraMembers, State2), - State4 = merge_peer_tokens(Tokens, State3), - {noreply, - maybe_start_commit(redrive_intents(moved, intents_advanced(mark_synced(State4))))}; + %% Update the replica before the leader changes, because the `leader` + %% enter call reads the names back from the table. + Data0 = cancel_resync(Data#data{replica = R1}), + %% redrive_intents/2 below re-routes forwarded intents. The state enter + %% calls take or give up leadership. If we just lost leadership, we + %% resolve direct registrations still waiting for replica acks, so their + %% timers don't fire on a follower. + Data1 = + case OldLeader =:= Self andalso NewLeader =/= Self of + true -> fail_pending_acks(Data0#data{leader = NewLeader, leader_down = false}); + false -> Data0#data{leader = NewLeader, leader_down = false} + end, + %% The snapshot replaces any assume gather still in flight. + Data2 = Data1#data{assume_ref = undefined}, + Data3 = add_member_monitors(ExtraMembers, Data2), + Data4 = merge_peer_tokens(Tokens, Data3), + Data5 = maybe_start_commit( + redrive_intents(moved, release_deferred_acks(mark_synced(Data4))) + ), + %% Move to the role the snapshot names. This works from any state, + %% including `assuming`, whose gather the snapshot replaces. + NextState = + case NewLeader of + Self -> leader; + undefined -> searching; + _ -> follower + end, + {next_state, NextState, Data5}; stale -> - {noreply, State} + keep_state_and_data end; -%% Rolling upgrade: legacy fire-and-forget unregister casts from pre-call nodes. -handle_cast( - {unregister, LogicalName}, - State = #state{leader = Leader, member_id = Leader, replica = R} -) -> - %% Capture the bound pid before the optimistic delete, or plan_op/3 could not - %% tell a real unregister from a no-op. - ReleasedPid = dgen_registry_replica:lookup_pid(R, LogicalName), - {noreply, - enqueue_op({remove, LogicalName, ReleasedPid, undefined}, row_delete(State, LogicalName))}; -handle_cast({unregister, LogicalName}, State = #state{leader = Leader}) when - Leader =/= undefined --> - cast_to_member(Leader, {unregister, LogicalName}), - {noreply, row_delete(State, LogicalName)}; -%% Leader fast path onboarding a joiner: an up-to-date follower only monitors it and -%% records its token (so a later DOWN is fenced with the elector's token). -handle_cast({peer_joined, MemberId, Tokens}, State) -> - {noreply, merge_peer_tokens(Tokens, add_member_monitors([MemberId], State))}; -%% Presence subscription delta from the elector (§4.9); a stale push to a deposed -%% leader is dropped (the next leader is reseeded on assume). -handle_cast({presence_update, Update, Epoch}, State = #state{member_id = Self, leader = Self}) -> - case Epoch >= epoch(State) of - true -> {noreply, apply_presence_update(Update, State)}; - false -> {noreply, State} - end; -handle_cast({presence_update, _Update, _Epoch}, State) -> - {noreply, State}; -handle_cast(_, State) -> - {noreply, State}. - -apply_presence_update({subscribe, SubId, Watch, Notify}, State = #state{subs = Subs}) -> - seed_subscription(SubId, State#state{subs = Subs#{SubId => {Watch, Notify}}}); -apply_presence_update( - {unsubscribe, SubId}, State = #state{subs = Subs, sub_matches = SM, notify_matches = NM} -) -> - State#state{ - subs = maps:remove(SubId, Subs), - sub_matches = maps:remove(SubId, SM), - notify_matches = maps:remove(SubId, NM) - }; -apply_presence_update(unsubscribe_all, State) -> - State#state{subs = #{}, sub_matches = #{}, notify_matches = #{}}. - -%% --------------------------------------------------------------------------- -%% handle_info/2 -%% --------------------------------------------------------------------------- - -%% Too few replica acks for these direct registrations in time (§8): degrade open -%% (ack `yes` leader-only, telemetry) or, with `strict_replication`, fail closed -%% (reject and retract). -handle_info({replicate_timeout, BatchRef}, State = #state{pipeline = P, config = Config}) -> - case dgen_registry_pipeline:ack_timeout(BatchRef, P) of +%% Sent by a leader taking the fast path for a joiner. An up-to-date follower only +%% monitors the joiner and records its token. The token lets the elector tell a later +%% DOWN for this join apart from a stale one. +handle_event(cast, {peer_joined, MemberId, Tokens}, _State, Data) -> + {keep_state, merge_peer_tokens(Tokens, add_member_monitors([MemberId], Data))}; +%% -- infos ------------------------------------------------------------------ + +%% Too few replicas acked these direct registrations in time (§8). By default we +%% degrade open: we answer `yes` with only the leader holding the name, and emit +%% telemetry. With `strict_replication` we fail closed: we answer `no` and retract the +%% bindings. +handle_event(info, {replicate_timeout, BatchRef}, _State, Data) -> + #data{pipeline = P, config = Config} = Data, + case dgen_registry_pipeline:handle_ack_timeout(BatchRef, P) of {expired, Direct, Needed, Got, P1} -> - State1 = State#state{pipeline = P1}, + Data1 = Data#data{pipeline = P1}, case dgen_config:strict_replication(Config) of false -> notify_degrade_open(Needed, Got, length(Direct)), - deliver_direct(Direct), - {noreply, State1}; + deliver_direct_yes(Direct), + {keep_state, Data1}; true -> - {noreply, reject_and_retract_direct(Direct, State1)} + {keep_state, reject_and_retract_direct(Direct, Data1)} end; none -> - {noreply, State} + keep_state_and_data end; -handle_info({dgen_transaction, Ref, Reply}, State = #state{pipeline = P}) -> - case dgen_registry_pipeline:finish(Ref, P) of +handle_event(info, {dgen_transaction, Ref, Reply}, _State, Data = #data{pipeline = P}) -> + case dgen_registry_pipeline:finish_commit(Ref, P) of {ok, Plan, MRef, P1} -> - %% The result precedes the worker's DOWN, so flush drops it. + %% The result arrives before the worker's DOWN, so `flush` drops the DOWN. erlang:demonitor(MRef, [flush]), - State1 = commit_result(Reply, Plan, State#state{pipeline = P1}), - {noreply, maybe_start_commit(State1)}; + Data1 = handle_commit_result(Reply, Plan, Data#data{pipeline = P1}), + {keep_state, maybe_start_commit(Data1)}; stale -> - {noreply, State} + keep_state_and_data end; - -%% A registered process died (leader only). The DOWN carries Ref so the plan's -%% ref-match guard ignores a stale DOWN for a name already re-bound. If we no longer -%% lead, the new leader (which monitors the pid itself) drives the removal. -handle_info({{down, LogicalName}, Ref, process, _Pid, _Reason}, State) -> - {noreply, enqueue_op({down, LogicalName, Ref}, State)}; -handle_info({'DOWN', Ref, process, _Pid, _Reason}, State) -> - #state{ +%% A registered process died. Only the leader monitors them. The DOWN carries `Ref`, +%% and the planner ignores it if the name has since been bound again under a new +%% monitor. If we no longer lead, the new leader monitors the pid itself and removes +%% the name. +handle_event(info, {{down, LogicalName}, Ref, process, _Pid, _Reason}, _State, Data) -> + % {down, LogicalName} comes from a custom tag on the monitor: + % + % erlang:monitor(process, Pid, [{tag, {down, Name}}]). + % + {keep_state, enqueue_op({down, LogicalName, Ref}, Data)}; +handle_event(info, {'DOWN', Ref, process, _Pid, _Reason}, _State, Data) -> + #data{ monitors = Monitors, elector = Elector, - member_id = Self - } = State, + member_id = Self, + leader = Leader, + leader_down = LeaderDown + } = Data, case maps:get(Ref, Monitors, undefined) of undefined -> - %% Not a peer monitor: the commit worker, or a late straggler. - case dgen_registry_pipeline:worker_down(Ref, State#state.pipeline) of + %% Not a peer monitor. It is the commit worker or a late leftover. + case dgen_registry_pipeline:handle_worker_down(Ref, Data#data.pipeline) of {ok, Plan, P1} -> - State1 = State#state{pipeline = P1}, - {noreply, maybe_start_commit(surrender(failed, maps:get(ops, Plan), State1))}; + Data1 = Data#data{pipeline = P1}, + {keep_state, maybe_start_commit(surrender(failed, maps:get(ops, Plan), Data1))}; stale -> - {noreply, State} + keep_state_and_data end; Self -> - {noreply, State}; + keep_state_and_data; DeadMemberId -> - %% Fence {member_down} with the elector's *current* token for the peer, not - %% our cached one, which a dropped snapshot may have left stale (a stale - %% token would strand a dead member and block failover, §5.7). The read runs - %% in a helper, never inline: a call to the elector could deadlock against - %% its own `{elector_assume_and_distribute}` call into this member. - Cached = maps:get(DeadMemberId, State#state.peer_tokens, undefined), + %% Send {member_down} with the elector's *current* token for the peer, not + %% our cached one. A dropped snapshot may have left ours stale, and the + %% elector would ignore it. The dead member would then stay listed and + %% block failover (§5.7). A helper reads the token, never this loop, + %% because a call to the elector could deadlock against its own + %% `{elector_assume_and_distribute}` call into this member. + Cached = maps:get(DeadMemberId, Data#data.peer_tokens, undefined), spawn_member_down(Elector, DeadMemberId, Cached), - {noreply, remove_member(DeadMemberId, State)} + %% If it was our leader, nothing more from it can reach us now. + Data1 = Data#data{leader_down = LeaderDown orelse DeadMemberId =:= Leader}, + {keep_state, remove_member(DeadMemberId, Data1)} end; -handle_info({nodeup, _Node}, State = #state{elector = Elector, member_id = Self}) -> - %% We may have been removed during a partition: re-announce with a fresh token - %% so a queued stale {member_down, Self, OldToken} is discarded. +handle_event(info, {nodeup, _Node}, _State, Data) -> + #data{elector = Elector, member_id = Self} = Data, + %% We may have been removed during a partition. We join again with a fresh token, + %% so the elector discards any queued {member_down, Self, OldToken}. NewToken = make_ref(), - dgen_server:cast(Elector, {join, Self, NewToken, not State#state.synced}), - %% Partition heal: if our leader is reachable, re-drive intents now (idempotent) - %% so a held or swallowed unregister is never lost (Non-goal 5), without waiting - %% for the re-onboarding snapshot. - State1 = State#state{join_token = NewToken}, - {noreply, - case intent_route(State1) of - none -> State1; - _Reachable -> redrive_intents(all, State1) - end}; -%% Fast-path continuation for a rejoining (non-fresh) member: adjudicate its replica -%% against our authoritative table (§5.6), merge its trail, then onboard it. A killed -%% authority's DOWN drives the normal cleanup. handle_event/4 checked the epoch. -handle_info( - {joiner_gathered, MemberId, AllIds, Tokens, _Epoch, Result}, - State = #state{member_id = Self, leader = Self, members = Members} + dgen_registry_elector:join(Elector, Self, NewToken, not Data#data.synced), + %% The partition healed. If our leader is reachable, we re-drive intents now + %% instead of waiting for the onboarding snapshot. Re-driving is idempotent, and it + %% makes sure an unregister we were holding, or one that was lost, still happens + %% (Non-goal 5). + Data1 = Data#data{join_token = NewToken}, + case get_intent_route(Data1) of + none -> {keep_state, Data1}; + _Reachable -> {keep_state, redrive_intents(all, Data1)} + end; +%% Periodic maintenance. Prunes the release trail and kill budget (§5.6), and drops +%% stale intents. +handle_event(info, prune_released, _State, Data) -> + #data{conflicts = C, members = Members, config = Config} = Data, + schedule_timer(?PRUNE_INTERVAL, prune_released), + Now = erlang:system_time(millisecond), + AllConnected = lists:all( + fun({Node, _Name}) -> dgen_utils:node_reachable(Node) end, + maps:keys(Members) + ), + C1 = dgen_registry_conflicts:prune(Now, AllConnected, Config, C), + {keep_state, prune_intents(Data#data{conflicts = C1})}; +%% Nobody answered our resync request. We clear the timer so the next gap or heartbeat +%% can ask again. +handle_event(info, resync_timeout, _State, Data) -> + {keep_state, Data#data{resync_timer = undefined}}; +%% The connector saw that the durable election epoch is ahead of ours, so a handoff +%% never reached us. Joining again makes the leader assume and send snapshots again. +handle_event(info, {durable_epoch, DurableEpoch}, _State, Data) when is_integer(DurableEpoch) -> + #data{replica = #replica{epoch = Epoch}} = Data, + case DurableEpoch > Epoch of + true -> + #data{elector = Elector, member_id = Self, join_token = Token, synced = Synced} = + Data, + dgen_registry_elector:join(Elector, Self, Token, not Synced); + false -> + ok + end, + keep_state_and_data; +handle_event(info, {durable_epoch, _DurableEpoch}, _State, _Data) -> + keep_state_and_data; +%% -- internal --------------------------------------------------------------- + +%% This is safe here because the supervisor is no longer waiting on our init/1. +handle_event( + internal, discover_elector, _State, Data = #data{member_id = {_Node, Name} = MemberId} +) -> + #data{join_token = Token} = Data, + Elector = dgen_registry:elector_pid(Name), + true = is_pid(Elector), + %% We pass `fresh = true`. We have never synced, so we can't hold any bindings. + dgen_registry_elector:join(Elector, MemberId, Token, true), + {keep_state, Data#data{elector = Elector}}; +%% -- the current state handles everything else ------------------------------ + +handle_event(EventType, EventContent, State, Data) -> + apply(?MODULE, State, [EventType, EventContent, Data]). + +%% --------------------------------------------------------------------------- +%% State functions +%% --------------------------------------------------------------------------- +%% +%% handle_event/4 calls these for the enter call and for every event it doesn't handle +%% itself. The state mirrors the data. In `leader`, the `leader` field equals +%% `member_id`. In `follower`, it names another member. In `searching` and `assuming` +%% it is `undefined`. We only ever set `assume_ref` while `leader` is `undefined`. +%% +%% Events a state doesn't handle fall through like this: +%% searching, assuming -> leaderless/3 -> non_leader/3 -> unhandled/3 +%% follower -> non_leader/3 -> unhandled/3 +%% leader -> unhandled/3 + +%% -- leader ----------------------------------------------------------------- + +%% Taking and giving up leadership both happen in enter calls, so no path between +%% states can do one without the other. Two things are deliberately not here. +%% redrive_intents/2 also runs on leadership events that don't change the state. +%% fail_pending_acks/1 stays in the apply_names_snapshot handler, because when a leader +%% assumes again, `BatchRef` already makes its old acks harmless. +%% +%% We don't monitor registered processes here all at once. Instead we send ourselves +%% {monitor_sweep, …} messages that each monitor one chunk, tagged with the epoch, and +%% live traffic runs between them. This is safe because monitoring a pid that is +%% already dead or unreachable fires a DOWN right away. A process that dies before its +%% chunk is still caught. We just don't promise to clean it up promptly. +leader(enter, _OldState, Data) -> + #data{replica = #replica{epoch = Epoch}} = Data, + self() ! {monitor_sweep, Epoch, start}, + %% Seed the watch sets so the first commit computes its changes correctly (§4.9). + {keep_state, rebuild_sub_matches(Data)}; +%% If we stay leader in the same epoch, we handle a join on the fast path. Otherwise +%% leadership really is changing. We check the epoch in the data, because the `leader` +%% state doesn't say which term we lead. +leader( + {call, From}, + {elector_assume_and_distribute, MemberId, AllIds, Tokens, FreshIds, Epoch}, + Data +) -> + #data{synced = Synced, replica = #replica{epoch = CurrentEpoch}} = Data, + case Synced andalso MemberId =/= undefined andalso CurrentEpoch =:= Epoch of + true -> + Data1 = assume_leadership_fast_path(MemberId, AllIds, Tokens, FreshIds, Epoch, Data), + {keep_state, Data1, [{reply, From, ok}]}; + false -> + %% Give up leadership and start a gather. See + %% prepare_to_assume_leadership/6. + Data1 = prepare_to_assume_leadership(MemberId, AllIds, Tokens, FreshIds, Epoch, Data), + {next_state, assuming, Data1, [{reply, From, ok}]} + end; +leader({call, From}, {set_metadata, LogicalName, Index, MetaData}, Data) -> + {keep_state, enqueue_op({set_meta, LogicalName, Index, MetaData, {meta, From}}, Data)}; +%% Consistent reads (Guarantee 5). We capture the value now. A spawned helper then +%% checks the durable leader key before it replies, so a deposed leader can't give an +%% authoritative answer. +leader({call, From}, {get_metadata, LogicalName}, Data = #data{replica = R}) -> + reply_fenced(From, dgen_registry_replica:lookup_metadata(R, LogicalName), undefined, Data), + keep_state_and_data; +leader({call, From}, {query_consistent, Constraints}, Data = #data{replica = R}) -> + reply_fenced(From, dgen_registry_replica:query_constraints(R, Constraints), [], Data), + keep_state_and_data; +leader({call, From}, {whereis, LogicalName}, Data = #data{replica = R}) -> + reply_fenced(From, dgen_registry_replica:lookup_pid(R, LogicalName), undefined, Data), + keep_state_and_data; +%% A forwarded registration. We answer with {register_reply} once it commits. +leader(cast, {register_req, Ref, FollowerId, LogicalName, Pid, Meta}, Data) -> + case should_reject_when_degraded(LogicalName, Data) of + true -> + cast_to_member(FollowerId, {register_reply, Ref, no}), + keep_state_and_data; + false -> + {keep_state, + enqueue_op({add, LogicalName, Pid, Meta, {forward, FollowerId, Ref}}, Data)} + end; +%% A forwarded unregister. We answer with {unregister_reply} once it commits. +leader(cast, {unregister_req, Ref, FollowerId, LogicalName}, Data = #data{replica = R}) -> + ReleasedPid = dgen_registry_replica:lookup_pid(R, LogicalName), + {keep_state, + enqueue_op( + {remove, LogicalName, ReleasedPid, {forward_unreg, FollowerId, Ref}}, + delete_row(Data, LogicalName) + )}; +%% A forwarded set_metadata. We answer with {set_meta_reply} once it commits. +leader(cast, {set_meta_req, Ref, FollowerId, LogicalName, Index, MetaData}, Data) -> + {keep_state, + enqueue_op( + {set_meta, LogicalName, Index, MetaData, {forward_meta, FollowerId, Ref}}, Data + )}; +%% A forwarded consistent read. We reply straight to the caller's `From`, after the +%% same leadership check as a local read. +leader(cast, {get_metadata_req, LogicalName, From}, Data = #data{replica = R}) -> + reply_fenced(From, dgen_registry_replica:lookup_metadata(R, LogicalName), undefined, Data), + keep_state_and_data; +leader(cast, {query_req, Constraints, From}, Data = #data{replica = R}) -> + reply_fenced(From, dgen_registry_replica:query_constraints(R, Constraints), [], Data), + keep_state_and_data; +leader(cast, {whereis_req, LogicalName, From}, Data = #data{replica = R}) -> + reply_fenced(From, dgen_registry_replica:lookup_pid(R, LogicalName), undefined, Data), + keep_state_and_data; +%% Answers a resync request with a snapshot at our epoch and applied version. +leader(cast, {resync_req, FollowerId}, Data) -> + #data{leader = Leader, replica = R = #replica{epoch = Epoch, version = Version}} = Data, + RecordsBin = dgen_registry_replica:encode(R), + cast_to_member( + FollowerId, {apply_names_snapshot, RecordsBin, Leader, [], #{}, Epoch, Version} + ), + keep_state_and_data; +%% A retract forwarded by a member that couldn't commit it. With `strict_replication`, +%% a binding we answered `no` must not stay registered (§8), so it gets retracted. The +%% retract only clears the name if the same pid still holds it, so it never removes a +%% later registration. +leader(cast, {retract_req, LogicalName, Pid}, Data) -> + {keep_state, enqueue_op({retract, LogicalName, Pid}, Data)}; +%% A change to the presence subscriptions, pushed by the elector (§4.9). We drop a +%% push from an older epoch. The next leader pulls the full set when it assumes. +leader(cast, {presence_update, Update, Epoch}, Data) -> + #data{replica = #replica{epoch = CurrentEpoch}} = Data, + case Epoch >= CurrentEpoch of + true -> {keep_state, apply_presence_update(Update, Data)}; + false -> keep_state_and_data + end; +%% The second half of the fast path, for a member rejoining with old data. We check +%% its replica for conflicts against our table, which is authoritative (§5.6), merge +%% its trail, and then onboard it. If repair kills a pid we hold, its DOWN cleans up +%% the name as usual. We only act on this in the term that spawned the gather. +leader( + info, + {joiner_gathered, MemberId, AllIds, Tokens, Epoch, Result}, + Data = #data{members = Members} ) when is_map_key(MemberId, Members) -> - State1 = - case Result of - {ok, {Records, _Version, Released}, Alive} -> - %% The resolved map is discarded: repair only kills and alarms. - OwnRecords = dgen_registry_replica:records(State#state.replica), - {_Records, S1} = repair_conflicts(OwnRecords, [Records], [Released], Alive, State), - S1; - error -> - %% Unanswered: onboard anyway; divergence surfaces at the next gather. - State - end, - {noreply, onboard_joiner(MemberId, AllIds, Tokens, State1)}; -%% The off-loop handoff gather returned (only while `assume_ref` matches). The -%% freshest map is the reconstructed state: broadcasts are totally ordered and every -%% replica is a prefix of the stream. MaxVersion re-baselines the followers. -handle_info( + #data{replica = R = #replica{epoch = CurrentEpoch}} = Data, + case CurrentEpoch =:= Epoch of + true -> + Data1 = + case Result of + {ok, {Records, _Version, Released}, Alive} -> + %% We discard the resolved map. Here, repair only kills + %% processes and raises alarms. + OwnRecords = dgen_registry_replica:get_records(R), + {_Records, D1} = repair_conflicts( + OwnRecords, [Records], [Released], Alive, Data + ), + D1; + error -> + %% The joiner didn't answer. We onboard it anyway, and any + %% divergence shows up at the next gather. + Data + end, + {keep_state, cast_data_to_new_member(MemberId, AllIds, Tokens, Data1)}; + false -> + keep_state_and_data + end; +%% One step of the monitor sweep that the `leader` enter call started. Each message +%% handles one chunk, and only in the epoch that started the sweep. We skip a name +%% that is already monitored because it committed after the sweep began. We also skip +%% a name bound to a different pid since its chunk was read. A pid that died before +%% its chunk gets an immediate `noproc` DOWN, so the delay loses nothing. +leader(info, {monitor_sweep, Epoch, Cont0}, Data) -> + #data{replica = R = #replica{epoch = CurrentEpoch}} = Data, + case CurrentEpoch =:= Epoch of + true -> {keep_state, monitor_sweep(Epoch, Cont0, R, Data)}; + false -> keep_state_and_data + end; +%% Retries the ops of a failed batch while we still lead. We drop registrations whose +%% caller has died. +leader(info, {requeue_ops, Ops}, Data) -> + Live = [Op || Op <- Ops, is_caller_alive(Op)], + {keep_state, maybe_start_commit(lists:foldl(fun enqueue_op/2, Data, Live))}; +%% Every member keeps this timer running, so a member that later becomes leader +%% starts sending heartbeats right away. +leader(info, replica_heartbeat, Data) -> + broadcast_heartbeat(Data), + schedule_timer(?REPLICA_HEARTBEAT_INTERVAL, replica_heartbeat), + keep_state_and_data; +leader(EventType, EventContent, Data) -> + unhandled(EventType, EventContent, Data). + +%% -- follower --------------------------------------------------------------- + +follower(enter, OldState, Data) -> + non_leader_enter(OldState, Data); +%% Consistent reads. The leader replies to `From` directly, so this loop never waits +%% on it. If the leader is unreachable, we answer with the default value now. +follower({call, From}, {get_metadata, LogicalName}, Data = #data{leader = Leader}) -> + forward_read(Leader, {get_metadata_req, LogicalName, From}, From, undefined, Data); +follower({call, From}, {query_consistent, Constraints}, Data = #data{leader = Leader}) -> + forward_read(Leader, {query_req, Constraints, From}, From, [], Data); +follower({call, From}, {whereis, LogicalName}, Data = #data{leader = Leader}) -> + forward_read(Leader, {whereis_req, LogicalName, From}, From, undefined, Data); +follower(cast, {retract_req, LogicalName, Pid}, #data{leader = Leader}) -> + cast_to_member(Leader, {retract_req, LogicalName, Pid}), + keep_state_and_data; +follower(EventType, EventContent, Data) -> + non_leader(EventType, EventContent, Data). + +%% -- searching -------------------------------------------------------------- + +searching(enter, OldState, Data) -> + non_leader_enter(OldState, Data); +searching(EventType, EventContent, Data) -> + leaderless(EventType, EventContent, Data). + +%% -- assuming --------------------------------------------------------------- + +%% A leader that moves to `assuming` gives up leadership as the assume starts. If a +%% registered process dies during the gather, the next monitor sweep gets an immediate +%% `noproc` DOWN for it. +assuming(enter, OldState, Data) -> + non_leader_enter(OldState, Data); +%% The handoff gather finished. This clause only matches if `Ref` is still our +%% `assume_ref`. Otherwise a newer assume replaced it, and the result is dropped. The +%% freshest map becomes our state. That is safe because broadcasts are totally ordered +%% and every replica is a prefix of the stream. `MaxVersion` becomes the followers' +%% new starting version. +assuming( + info, {assume_gathered, {MemberId, AllIds, Tokens, FreshIds, Epoch, Ref}, SelfRecords, FreshestRecords, {MaxVersion, Frontier}, PeerResults, Subs, Alive}, - State = #state{member_id = Self, assume_ref = Ref} + Data = #data{assume_ref = Ref} ) -> + #data{member_id = Self} = Data, OtherIds = lists:delete(Self, AllIds), Peers = maps:values(PeerResults), - Degraded = dgen_registry_conflicts:degraded( + Degraded = dgen_registry_conflicts:is_reconstruction_degraded( MaxVersion, Frontier, OtherIds, FreshIds, PeerResults ), - %% Repair the conflicts the gather exposes (§5.6): conflicted names are dropped. - {Records, State0} = repair_conflicts( + %% Repair the conflicts the gather found (§5.6). Names in conflict are dropped. + {Records, Data0} = repair_conflicts( FreshestRecords, [SelfRecords | [Recs || {Recs, _V, _Rel} <- Peers]], [Rel || {_Recs, _V, Rel} <- Peers], Alive, - State + Data ), - %% Clearing assume_ref enters `leader`, whose enter call runs assume_leadership - %% against the reconstructed replica and `Subs` (§4.9). - State2 = State0#state{ - replica = dgen_registry_replica:adopt(Epoch, MaxVersion, Records, State0#state.replica), + %% We lead now. The move to `leader` below runs its enter call against the rebuilt + %% replica and `Subs` (§4.9). + Data2 = Data0#data{ + replica = dgen_registry_replica:adopt(Epoch, MaxVersion, Records, Data0#data.replica), leader = Self, - conflicts = dgen_registry_conflicts:set_degraded(Degraded, State0#state.conflicts), + leader_down = false, + conflicts = dgen_registry_conflicts:set_degraded(Degraded, Data0#data.conflicts), assume_ref = undefined, subs = Subs }, - State3 = add_member_monitors(extra_member_ids(MemberId, AllIds, Self), State2), - State4 = merge_peer_tokens(Tokens, State3), - %% Encode once; every cast shares the refc binary. + Data3 = add_member_monitors(get_extra_member_ids(MemberId, AllIds, Self), Data2), + Data4 = merge_peer_tokens(Tokens, Data3), + %% Encode once. Every cast shares the same refc binary. RecordsBin = dgen_registry_replica:encode_records(Records), lists:foreach( fun(Id) -> cast_to_member( Id, - {apply_names_snapshot, RecordsBin, Self, extra_member_ids(MemberId, AllIds, Id), + {apply_names_snapshot, RecordsBin, Self, get_extra_member_ids(MemberId, AllIds, Id), Tokens, Epoch, MaxVersion} ) end, OtherIds ), %% Re-drive every unresolved intent into our own pipeline. - {noreply, maybe_start_commit(redrive_intents(moved, mark_synced(cancel_resync(State4))))}; -%% Periodic maintenance: prune the release trail and kill budget (§5.6) and stale -%% intents. -handle_info(prune_released, State = #state{conflicts = C, members = Members, config = Config}) -> - arm(?PRUNE_INTERVAL, prune_released), - Now = erlang:system_time(millisecond), - AllConnected = lists:all( - fun({Node, _Name}) -> dgen_utils:node_reachable(Node) end, - maps:keys(Members) - ), - C1 = dgen_registry_conflicts:prune(Now, AllConnected, Config, C), - {noreply, prune_intents(State#state{conflicts = C1})}; -%% Unanswered resync request: clear the guard so the next gap (or heartbeat) retries. -handle_info(resync_timeout, State) -> - {noreply, State#state{resync_timer = undefined}}; -%% Deferred monitor sweep (see assume_leadership/1), one chunk per message. A name -%% already monitored (committed since the sweep began) or re-bound since its chunk was -%% selected is skipped. A pid that died before its chunk fires an immediate `noproc` -%% DOWN, so deferral loses nothing. handle_event/4 checked the epoch. -handle_info( - {monitor_sweep, Epoch, Cont0}, - State = #state{member_id = Self, leader = Self, replica = R} + {next_state, leader, + maybe_start_commit(redrive_intents(moved, mark_synced(cancel_resync(Data4))))}; +%% We postpone a forwarded registration that arrives during an assume until the assume +%% finishes. If another assume replaces ours, we never answer it, and the forwarder +%% re-drives it. +assuming(cast, {register_req, _Ref, _FollowerId, _Name, _Pid, _Meta}, _Data) -> + {keep_state_and_data, [postpone]}; +assuming(EventType, EventContent, Data) -> + leaderless(EventType, EventContent, Data). + +%% -- shared by the non-leader states ---------------------------------------- + +%% Leaving `leader`. We demonitor every registered pid but keep the replica for +%% snapshot reads. We also drop the commit version the pipeline cached, so a later +%% term can't use it as its read version. +non_leader_enter(leader, Data) -> + #data{name_to_ref = NTR, pipeline = P} = Data, + maps:foreach(fun(_LogicalName, Ref) -> erlang:demonitor(Ref, [flush]) end, NTR), + {keep_state, Data#data{name_to_ref = #{}, pipeline = dgen_registry_pipeline:start_new_term(P)}}; +non_leader_enter(_OldState, _Data) -> + keep_state_and_data. + +%% Shared by `searching` and `assuming`, where there is no leader to forward to. +leaderless({call, From}, {get_metadata, _LogicalName}, _Data) -> + {keep_state_and_data, [{reply, From, undefined}]}; +leaderless({call, From}, {query_consistent, _Constraints}, _Data) -> + {keep_state_and_data, [{reply, From, []}]}; +leaderless({call, From}, {whereis, _LogicalName}, _Data) -> + {keep_state_and_data, [{reply, From, undefined}]}; +%% There is no leader, most likely because a handoff is under way. We hold the retract +%% as an unregister intent marked `answered`, so it is re-driven later instead of lost. +leaderless(cast, {retract_req, LogicalName, Pid}, Data) -> + {keep_state, submit_intent({unregister, LogicalName, Pid}, answered, Data)}; +leaderless(EventType, EventContent, Data) -> + non_leader(EventType, EventContent, Data). + +%% Shared by `searching`, `assuming` and `follower`. +non_leader( + {call, From}, {elector_assume_and_distribute, MemberId, AllIds, Tokens, FreshIds, Epoch}, Data +) -> + %% Move to `assuming`. We do this even from `assuming`, because the newer gather + %% replaces ours. + Data1 = prepare_to_assume_leadership(MemberId, AllIds, Tokens, FreshIds, Epoch, Data), + {next_state, assuming, Data1, [{reply, From, ok}]}; +%% Forwards the write as an intent. The reply comes back through this member. FIFO +%% delivers the leader's `{names_batch}` before it, so our row is already updated and +%% a local snapshot read right after sees the new value. +non_leader({call, From}, {set_metadata, LogicalName, Index, MetaData}, Data) -> + {keep_state, submit_intent({set_meta, LogicalName, Index, MetaData}, From, Data)}; +%% The sender thinks we lead, but we don't. We reject it so the caller retries. +non_leader(cast, {set_meta_req, Ref, FollowerId, _LogicalName, _Index, _MetaData}, _Data) -> + cast_to_member(FollowerId, {set_meta_reply, Ref, {error, no_leader}}), + keep_state_and_data; +%% The sender thinks we lead, but we don't. We answer with the empty value, and the +%% caller can retry. +non_leader(cast, {get_metadata_req, _LogicalName, From}, _Data) -> + gen_server:reply(From, undefined), + keep_state_and_data; +non_leader(cast, {query_req, _Constraints, From}, _Data) -> + gen_server:reply(From, []), + keep_state_and_data; +non_leader(cast, {whereis_req, _LogicalName, From}, _Data) -> + gen_server:reply(From, undefined), + keep_state_and_data; +%% We were deposed, so we surrender the ops of the failed batch. See +%% dgen_registry_pipeline:surrender/2. +non_leader(info, {requeue_ops, Ops}, Data) -> + {keep_state, surrender(deposed, Ops, Data)}; +non_leader(info, replica_heartbeat, _Data) -> + schedule_timer(?REPLICA_HEARTBEAT_INTERVAL, replica_heartbeat), + keep_state_and_data; +non_leader(EventType, EventContent, Data) -> + unhandled(EventType, EventContent, Data). + +%% Events the current state ignores. A non-leader doesn't answer leader-only messages +%% such as `register_req`, `unregister_req`, `resync_req` and `presence_update`. Only +%% the leader may decide them (§3), and the forwarder re-drives. Gather results that +%% were replaced and sweep steps from an old epoch also end up here. +unhandled({call, From}, _Request, _Data) -> + {keep_state_and_data, [{reply, From, {error, unknown_call}}]}; +unhandled(_EventType, _EventContent, _Data) -> + keep_state_and_data. + +forward_read(Leader, Req, From, Default, _Data) -> + case is_member_reachable(Leader) of + true -> + cast_to_member(Leader, Req), + keep_state_and_data; + false -> + {keep_state_and_data, [{reply, From, Default}]} + end. + +%% --------------------------------------------------------------------------- +%% Elector calls (dispatched from the state functions) +%% --------------------------------------------------------------------------- + +%% These handle `{elector_assume_and_distribute, …}`. `MemberId` is the member whose +%% join triggered the call, or `undefined` when a member went down. `AllIds` is the +%% full member list. +%% +%% The fast path. A leader that stays leader in this epoch and has synced is +%% authoritative, because every follower holds a prefix of its stream. So it skips the +%% gather and doesn't resend snapshots to the followers. It only monitors and onboards +%% the joiner. +assume_leadership_fast_path( + MemberId, AllIds, Tokens, FreshIds, Epoch, Data = #data{member_id = Self} ) -> - Tab = dgen_registry_replica:tab(R), + Data1 = merge_peer_tokens( + Tokens, add_member_monitors(get_extra_member_ids(MemberId, AllIds, Self), Data) + ), + %% A fresh joiner holds no bindings, so we onboard it now. A member that synced + %% before and is rejoining may hold bindings that disagree with ours, for example + %% after a partition heals (§5.6). We gather its replica off the loop and check it + %% for conflicts in {joiner_gathered} first. This check is best-effort. + case MemberId =:= Self orelse lists:member(MemberId, FreshIds) of + true -> + cast_data_to_new_member(MemberId, AllIds, Tokens, Data1); + false -> + spawn_member_gather(MemberId, AllIds, Tokens, Epoch, Data1), + Data1 + end. + +prepare_to_assume_leadership(MemberId, AllIds, Tokens, FreshIds, Epoch, Data) -> + #data{ + member_id = Self, + replica = R = #replica{version = Applied}, + elector = Elector, + tenant = Tenant, + tuid = Tuid, + conflicts = C + } = Data, + %% Leadership is really changing. The gather asks every peer for its replica, so it + %% runs off the loop and we answer the elector `ok` now. A gather on the loop would + %% time out the elector's call and hold up queued membership changes. We capture + %% our own records and version now, so a broadcast that lands during the gather + %% can't change which replica counts as freshest. + SelfRecords = dgen_registry_replica:get_records(R), + OtherIds = lists:delete(Self, AllIds), + Ref = make_ref(), + spawn_leadership_gather( + {MemberId, AllIds, Tokens, FreshIds, Epoch, Ref}, + SelfRecords, + {Applied, Data#data.leader_down}, + OtherIds, + Elector, + self(), + Tenant, + Tuid, + C + ), + %% We have no leader until the gather finishes. Writes wait as intents and are + %% re-driven once it does, instead of going to the old leader, which may be dead. + Data#data{assume_ref = Ref, leader = undefined, leader_down = false}. + +apply_presence_update({subscribe, SubId, Watch, Notify}, Data = #data{subs = Subs}) -> + seed_subscription(SubId, Data#data{subs = Subs#{SubId => {Watch, Notify}}}); +apply_presence_update({unsubscribe, SubId}, Data) -> + #data{subs = Subs, sub_matches = SM, notify_matches = NM} = Data, + Data#data{ + subs = maps:remove(SubId, Subs), + sub_matches = maps:remove(SubId, SM), + notify_matches = maps:remove(SubId, NM) + }; +apply_presence_update(unsubscribe_all, Data) -> + Data#data{subs = #{}, sub_matches = #{}, notify_matches = #{}}. + +%% Monitors one chunk of the sweep, then sends the next step to ourselves until the +%% table is done. +monitor_sweep(Epoch, Cont0, R, Data) -> + #replica{tab = Tab} = R, Select = case Cont0 of start -> @@ -971,7 +943,7 @@ handle_info( end, case Select of '$end_of_table' -> - {noreply, State}; + Data; {Pairs, Cont1} -> NTR1 = lists:foldl( @@ -984,71 +956,29 @@ handle_info( false -> Acc#{Name => monitor_name(Name, Pid)} end end, - State#state.name_to_ref, + Data#data.name_to_ref, Pairs ), self() ! {monitor_sweep, Epoch, Cont1}, - {noreply, State#state{name_to_ref = NTR1}} - end; -%% Retry a failed batch's ops if we still lead (dropping registrations whose caller -%% died), otherwise surrender them as deposed (dgen_registry_pipeline:surrender/2). -handle_info({requeue_ops, Ops}, State = #state{leader = Self, member_id = Self}) -> - Live = [Op || Op <- Ops, caller_alive(Op)], - {noreply, maybe_start_commit(lists:foldl(fun enqueue_op/2, State, Live))}; -handle_info({requeue_ops, Ops}, State) -> - {noreply, surrender(deposed, Ops, State)}; -%% From the connector: the durable election epoch is ahead of ours, so a handoff never -%% reached us. Re-announcing triggers a fresh assume/fan-out. -handle_info({durable_epoch, DurableEpoch}, State) when is_integer(DurableEpoch) -> - case DurableEpoch > epoch(State) of - true -> - #state{elector = Elector, member_id = Self, join_token = Token, synced = Synced} = - State, - dgen_server:cast(Elector, {join, Self, Token, not Synced}); - false -> - ok - end, - {noreply, State}; -handle_info({durable_epoch, _DurableEpoch}, State) -> - {noreply, State}; -handle_info(_Info, State) -> - {noreply, State}. + Data#data{name_to_ref = NTR1} + end. -terminate(_Reason, _StatemState, _State) -> +terminate(_Reason, _StatemState, _Data) -> net_kernel:monitor_nodes(false), ok. -code_change(_OldVsn, StatemState, State, _Extra) -> - {ok, StatemState, State}. +code_change(_OldVsn, StatemState, Data, _Extra) -> + {ok, StatemState, Data}. %% --------------------------------------------------------------------------- %% Internal helpers %% --------------------------------------------------------------------------- -do_leader_changed(NewLeader, OldLeader, Self, State) -> - %% Forwarded intents are re-routed by the caller's redrive_intents/2; the - %% assume/relinquish halves run in the state enter calls. - if - OldLeader =:= Self, NewLeader =/= Self -> - %% Lost leadership: resolve direct registrations awaiting replica acks - %% so their timers do not fire into a follower. - fail_pending_acks(State#state{leader = NewLeader}); - true -> - State#state{leader = NewLeader} - end. - -%% Registered-process monitors are not created inline but by chunked, epoch-fenced -%% {monitor_sweep, …} self-sends that interleave with live traffic. Safe because a -%% monitor on an already-dead (or unreachable) pid fires an immediate DOWN, so a death -%% before its chunk is caught, never lost; dead-pid cleanup has no promptness bound. -assume_leadership(State) -> - self() ! {monitor_sweep, epoch(State), start}, - %% Seed watch sets so the first commit's deltas are computed correctly (§4.9). - recompute_sub_matches(State). - -%% Rebuild every subscription's watch and notify sets from the replica (on assume -%% only; apply_committed_plan maintains them after). A silent reseed: no snapshots. -recompute_sub_matches(State = #state{subs = Subs, replica = R}) -> +%% Rebuilds the watch and notify sets of every subscription from the replica. This only +%% runs on assume, and apply_committed_plan/3 keeps the sets current after that. It +%% sends no snapshots. +rebuild_sub_matches(Data) -> + #data{subs = Subs, replica = R} = Data, SM = maps:map( fun(_SubId, {Watch, _Notify}) -> dgen_registry_replica:query_match_names(R, Watch) end, Subs @@ -1057,28 +987,17 @@ recompute_sub_matches(State = #state{subs = Subs, replica = R}) -> fun(_SubId, {_Watch, Notify}) -> dgen_registry_replica:query_match_names(R, Notify) end, Subs ), - State#state{sub_matches = SM, notify_matches = NM}. - -%% Demonitor all registered pids; keep the replica for snapshot reads. The pipeline's -%% cached commit version is dropped so it cannot pin a later term's read version. -relinquish_leadership(State = #state{name_to_ref = NTR}) -> - maps:foreach( - fun(_LogicalName, Ref) -> - erlang:demonitor(Ref, [flush]) - end, - NTR - ), - State#state{ - name_to_ref = #{}, pipeline = dgen_registry_pipeline:new_term(State#state.pipeline) - }. - -%% Freshest-wins gather across our replica and every reachable peer. Broadcasts are -%% totally ordered and FIFO and every replica is a prefix of the stream, so the highest -%% applied_version holds the freshest binding for every name, and a version tie implies -%% identical content. Peer snapshots are also returned for conflict detection (§5.6) -%% and the trail merge. Per-peer calls run in parallel. + Data#data{sub_matches = SM, notify_matches = NM}. + +%% Gathers our replica and every reachable peer's, and picks the freshest. Broadcasts +%% are totally ordered and FIFO, and every replica is a prefix of the stream. So the +%% replica with the highest applied_version has the latest binding for every name, and +%% two replicas at the same version have the same content. We also return the peer +%% snapshots for conflict detection (§5.6) and the trail merge, and whether every peer +%% answered with its leader gone. The peer calls run in parallel. gather_maps(SelfNames, SelfVersion, OtherIds) -> - PeerResults = gather_peer_snapshots(OtherIds), + {PeerResults, GoneIds} = gather_peer_snapshots(OtherIds), + PeersSettled = lists:sort(GoneIds) =:= lists:sort(OtherIds), {FreshestNames, MaxVersion} = maps:fold( fun(_Id, {Names, Version, _Released}, {_BestNames, BestVersion} = Best) -> case Version > BestVersion of @@ -1089,42 +1008,42 @@ gather_maps(SelfNames, SelfVersion, OtherIds) -> {SelfNames, SelfVersion}, PeerResults ), - {FreshestNames, MaxVersion, PeerResults}. + {FreshestNames, MaxVersion, PeerResults, PeersSettled}. -%% Fast-path onboarding: ship the joiner our snapshot (with the member list) and tell -%% existing followers to monitor it via {peer_joined}; their replicas are untouched. -onboard_joiner(MemberId, AllIds, Tokens, State = #state{member_id = Self, replica = R}) -> +%% Onboards a joiner on the fast path. We send the joiner our snapshot and the member +%% list. We tell the existing followers to monitor it with {peer_joined}, and leave +%% their replicas alone. +cast_data_to_new_member(MemberId, AllIds, Tokens, Data) -> + #data{member_id = Self, replica = R = #replica{epoch = Epoch, version = Version}} = Data, RecordsBin = dgen_registry_replica:encode(R), cast_to_member( - MemberId, - {apply_names_snapshot, RecordsBin, Self, AllIds, Tokens, dgen_registry_replica:epoch(R), - dgen_registry_replica:version(R)} + MemberId, {apply_names_snapshot, RecordsBin, Self, AllIds, Tokens, Epoch, Version} ), lists:foreach( fun(Id) -> cast_to_member(Id, {peer_joined, MemberId, Tokens}) end, [Id || Id <- AllIds, Id =/= Self, Id =/= MemberId] ), - State. - -%% Fetch a rejoining member's replica and probe conflict pids' liveness off the -%% leader's loop (both are RPCs), reporting {joiner_gathered, …}. Our `protected` table -%% is readable from this helper. -spawn_joiner_gather(MemberId, AllIds, Tokens, Epoch, State) -> - #state{replica = Replica, conflicts = C} = State, + Data. + +%% Fetches a rejoining member's replica and checks whether the pids in conflict are +%% alive, then reports {joiner_gathered, …}. Both steps are RPCs, so they run in a +%% helper instead of on the leader's loop. The helper can read our table because it is +%% `protected`. +spawn_member_gather(MemberId, AllIds, Tokens, Epoch, Data) -> + #data{replica = Replica, conflicts = C} = Data, Owner = self(), _ = spawn(fun() -> Result = - case member_names(MemberId) of - {ok, {Records, _Version, Released} = Snapshot} -> - Trail = dgen_registry_conflicts:trail( - dgen_registry_conflicts:merge([Released], C) - ), - %% Our table dies with us; then there is no one to report to. + case get_peer_snapshot(MemberId) of + {ok, {Records, _Version, Released} = Snapshot, _LeaderGone} -> + #conflicts{trail = Trail} = dgen_registry_conflicts:merge([Released], C), + %% Our table goes away if we die. Then there is no one to report + %% to anyway. Alive = - try dgen_registry_replica:records(Replica) of + try dgen_registry_replica:get_records(Replica) of Own -> - probe_alive( - dgen_registry_conflicts:conflict_pids(Own, [Records], Trail) + check_alive( + dgen_registry_conflicts:get_conflict_pids(Own, [Records], Trail) ) catch error:badarg -> #{} @@ -1137,25 +1056,24 @@ spawn_joiner_gather(MemberId, AllIds, Tokens, Epoch, State) -> end), ok. -%% The full handoff gather (§5.7), off the loop, reporting {assume_gathered, …}. `Ctx` -%% carries the assume payload and the ref matched against `assume_ref`. The durable -%% subscriptions (§4.9) and liveness probes (§5.6) are fetched here too: a synchronous -%% call to the elector from the member loop could deadlock against the elector's own -%% assume call into this member. -spawn_assume_gather(Ctx, SelfRecords, SelfVersion, OtherIds, Elector, Owner, Tenant, Tuid, C) -> +%% Runs the full handoff gather (§5.7) off the loop and reports {assume_gathered, …}. +%% `Ctx` carries the assume arguments and the ref we match against `assume_ref`. The +%% helper also fetches the durable subscriptions (§4.9) and runs the liveness probes +%% (§5.6). A synchronous call to the elector from the member loop could deadlock +%% against the elector's own assume call into this member. +spawn_leadership_gather(Ctx, SelfRecords, SelfState, OtherIds, Elector, Owner, Tenant, Tuid, C) -> _ = spawn(fun() -> - %% gather_caught_up falls back to a self-only reconstruction if the gather - %% throws, so the continuation always fires. + %% If the gather throws, gather_until_caught_up/5 falls back to our own records, + %% so we always send {assume_gathered, …}. {FreshestRecords, MaxVersion, PeerResults, Frontier} = - gather_caught_up(SelfRecords, SelfVersion, OtherIds, Tenant, Tuid), + gather_until_caught_up(SelfRecords, SelfState, OtherIds, Tenant, Tuid), Subs = fetch_subscriptions(Elector), Peers = maps:values(PeerResults), - Merged = dgen_registry_conflicts:trail( - dgen_registry_conflicts:merge([Rel || {_Recs, _V, Rel} <- Peers], C) - ), + #conflicts{trail = Merged} = + dgen_registry_conflicts:merge([Rel || {_Recs, _V, Rel} <- Peers], C), Witnesses = [SelfRecords | [Recs || {Recs, _V, _Rel} <- Peers]], - Alive = probe_alive( - dgen_registry_conflicts:conflict_pids(FreshestRecords, Witnesses, Merged) + Alive = check_alive( + dgen_registry_conflicts:get_conflict_pids(FreshestRecords, Witnesses, Merged) ), Owner ! {assume_gathered, Ctx, SelfRecords, FreshestRecords, {MaxVersion, Frontier}, @@ -1163,24 +1081,37 @@ spawn_assume_gather(Ctx, SelfRecords, SelfVersion, OtherIds, Elector, Owner, Ten end), ok. -%% Re-gather until MaxVersion reaches the durable version key (the SafeAssume fence in -%% the TLA+ spec) or ?GATHER_TIMEOUT. A live member is never ahead of that key, so a -%% lower MaxVersion means the gather raced a committed broadcast still queued in a -%% peer's mailbox; fanning out that stale map would drop an acked registration. Past -%% the deadline we proceed (the continuation flags `degraded`). -gather_caught_up(SelfRecords, SelfVersion, OtherIds, Tenant, Tuid) -> +%% Gathers again until `MaxVersion` reaches the durable version key, or until +%% ?GATHER_TIMEOUT passes. This is the SafeAssume guard in the TLA+ spec. No live +%% member is ever ahead of that key. A lower `MaxVersion` means a committed broadcast +%% was still queued in some peer's mailbox when we gathered. Sending out that stale map +%% would lose a registration we already acked. After the deadline we go ahead anyway, +%% and the {assume_gathered} handler marks the leader `degraded`. +%% +%% We stop early when waiting can't help. The old leader may have died after a commit +%% but before broadcasting it, and then no live member will ever reach the frontier. +%% Erlang delivers a process's messages before its DOWN. So once we and every peer have +%% handled the old leader's DOWN, nothing more from it can arrive, and we go ahead +%% `degraded` without waiting out the deadline. `SelfLeaderGone` is our own answer, +%% taken when we captured our records. +gather_until_caught_up(SelfRecords, {SelfVersion, SelfLeaderGone}, OtherIds, Tenant, Tuid) -> Deadline = erlang:monotonic_time(millisecond) + ?GATHER_TIMEOUT, - gather_caught_up(SelfRecords, SelfVersion, OtherIds, Tenant, Tuid, Deadline). + gather_until_caught_up( + SelfRecords, SelfVersion, SelfLeaderGone, OtherIds, Tenant, Tuid, Deadline + ). -gather_caught_up(SelfRecords, SelfVersion, OtherIds, Tenant, Tuid, Deadline) -> - {FreshestRecords, MaxVersion, PeerResults} = +gather_until_caught_up( + SelfRecords, SelfVersion, SelfLeaderGone, OtherIds, Tenant, Tuid, Deadline +) -> + {FreshestRecords, MaxVersion, PeerResults, PeersSettled} = try gather_maps(SelfRecords, SelfVersion, OtherIds) catch - _:_ -> {SelfRecords, SelfVersion, #{}} + _:_ -> {SelfRecords, SelfVersion, #{}, false} end, - %% Best-effort, on the applied_version scale. `unknown` (backend hiccup, test - %% double) disables the fence rather than wedging the handoff. + %% This read is best-effort, and it returns a value on the applied_version scale. + %% If it fails, for example on a backend hiccup or with a test double, we get + %% `unknown`. That turns the check off rather than stalling the handoff. Frontier = try dgen_registry_names:read_committed_frontier(Tenant, Tuid) @@ -1189,19 +1120,23 @@ gather_caught_up(SelfRecords, SelfVersion, OtherIds, Tenant, Tuid, Deadline) -> end, case Frontier =:= unknown orelse MaxVersion >= Frontier orelse + (SelfLeaderGone andalso PeersSettled) orelse erlang:monotonic_time(millisecond) >= Deadline of true -> {FreshestRecords, MaxVersion, PeerResults, Frontier}; false -> timer:sleep(?ASSUME_CATCHUP_INTERVAL), - gather_caught_up(SelfRecords, SelfVersion, OtherIds, Tenant, Tuid, Deadline) + gather_until_caught_up( + SelfRecords, SelfVersion, SelfLeaderGone, OtherIds, Tenant, Tuid, Deadline + ) end. -%% Best-effort, short-bounded subscription pull from the elector; off-loop callers only. -%% On timeout the empty set is reconciled by a later delta or handoff. +%% Pulls the subscriptions from the elector, with a short timeout. Only call it from +%% off the loop. On timeout we return no subscriptions, and a later update or handoff +%% fills them in. fetch_subscriptions(Elector) when is_pid(Elector) -> - try dgen_server:priority_call(Elector, get_subscriptions, ?SUBS_PULL_TIMEOUT) of + try dgen_registry_elector:priority_get_subscriptions(Elector, ?SUBS_PULL_TIMEOUT) of Subs when is_map(Subs) -> Subs; _ -> #{} catch @@ -1210,32 +1145,37 @@ fetch_subscriptions(Elector) when is_pid(Elector) -> fetch_subscriptions(_Elector) -> #{}. -%% One member_names/1 helper per peer under one deadline; late answers are dropped by -%% the catch-all handle_info clause. +%% Spawns one get_peer_snapshot/1 helper per peer, all under one deadline. Answers +%% that arrive late are dropped by unhandled/3. gather_peer_snapshots([]) -> - #{}; + {#{}, []}; gather_peer_snapshots(OtherIds) -> Ref = make_ref(), Parent = self(), lists:foreach( fun(MemberId) -> - spawn(fun() -> Parent ! {Ref, MemberId, member_names(MemberId)} end) + spawn(fun() -> Parent ! {Ref, MemberId, get_peer_snapshot(MemberId)} end) end, OtherIds ), Deadline = erlang:monotonic_time(millisecond) + ?GATHER_TIMEOUT + 500, - collect_gather(Ref, OtherIds, Deadline, #{}). + collect_gather(Ref, OtherIds, Deadline, {#{}, []}). +%% Collects `{PeerResults, GoneIds}`. `GoneIds` lists the peers whose leader is gone. collect_gather(_Ref, [], _Deadline, Acc) -> Acc; -collect_gather(Ref, Waiting, Deadline, Acc) -> +collect_gather(Ref, Waiting, Deadline, {Results, GoneIds} = Acc) -> Timeout = max(0, Deadline - erlang:monotonic_time(millisecond)), _ = ?ETA_LOG({gather_wait, length(Waiting), Timeout}), receive - {Ref, MemberId, {ok, Snapshot}} -> - collect_gather(Ref, lists:delete(MemberId, Waiting), Deadline, Acc#{ - MemberId => Snapshot - }); + {Ref, MemberId, {ok, Snapshot, LeaderGone}} -> + Results1 = Results#{MemberId => Snapshot}, + GoneIds1 = + case LeaderGone of + true -> [MemberId | GoneIds]; + false -> GoneIds + end, + collect_gather(Ref, lists:delete(MemberId, Waiting), Deadline, {Results1, GoneIds1}); {Ref, MemberId, error} -> collect_gather(Ref, lists:delete(MemberId, Waiting), Deadline, Acc) after Timeout -> @@ -1243,29 +1183,40 @@ collect_gather(Ref, Waiting, Deadline, Acc) -> Acc end. -%% Detect and repair conflicts (§5.6) per dgen_registry_conflicts:resolve/5, carrying -%% out its kills and alarms. Returns the reconstruction with conflicted names dropped. -repair_conflicts(Authority, Witnesses, Trails, Alive, State = #state{conflicts = C0}) -> - C1 = dgen_registry_conflicts:merge(Trails, C0), - Trail = dgen_registry_conflicts:trail(C1), - Conflicts = dgen_registry_conflicts:detect(Authority, Witnesses, Trail, probed(Alive)), +%% Detects and repairs conflicts (§5.6) using dgen_registry_conflicts:resolve/5, and +%% carries out the kills and alarms it asks for. Returns the rebuilt names without the +%% names in conflict. +repair_conflicts(Authority, Witnesses, Trails, Alive, Data = #data{conflicts = C0}) -> + C1 = #conflicts{trail = Trail} = dgen_registry_conflicts:merge(Trails, C0), + %% For a pid we didn't probe, we only check it if it is local, which needs no RPC. A + %% remote one counts as dead, so we never kill it on evidence we didn't gather. + IsAlive = fun(Pid) -> + case Alive of + #{Pid := A} -> A; + _ when node(Pid) =:= node() -> is_process_alive(Pid); + _ -> false + end + end, + Conflicts = dgen_registry_conflicts:detect(Authority, Witnesses, Trail, IsAlive), Now = erlang:system_time(millisecond), {Records, C2, Effects} = - dgen_registry_conflicts:resolve(Conflicts, Authority, State#state.config, Now, C1), + dgen_registry_conflicts:resolve(Conflicts, Authority, Data#data.config, Now, C1), lists:foreach( fun - ({kill, Pid}) -> exit(Pid, kill); + ({kill, Pid}) -> + exit(Pid, kill); ({alarm, {Name, APid, Divergent}, Action}) -> alarm_conflict(Name, APid, Divergent, Action) end, Effects ), - {Records, State#state{conflicts = C2}}. + {Records, Data#data{conflicts = C2}}. -%% Parallel liveness probe; runs in the gather helpers, never on the member's loop. -probe_alive([]) -> +%% Checks in parallel whether each pid is alive. It runs in the gather helpers, never +%% on the member's loop. +check_alive([]) -> #{}; -probe_alive(Pids) -> +check_alive(Pids) -> Ref = make_ref(), Parent = self(), lists:foreach( @@ -1283,32 +1234,21 @@ collect_alive(Ref, Waiting, Deadline, Acc) -> after Timeout -> Acc end. -%% An unprobed pid is checked only if local (no RPC); a remote one counts as dead so -%% it is never killed on evidence we did not gather. -probed(Alive) -> - fun(Pid) -> - case Alive of - #{Pid := A} -> A; - _ when node(Pid) =:= node() -> is_process_alive(Pid); - _ -> false - end - end. - alarm_conflict(Name, Authority, Divergent, Action) -> logger:error( "dgen_registry: name conflict — name=~p authority=~p divergent=~p action=~p. " "Two different live pids held one name (single-fault uniqueness breach, §5.6).", [Name, Authority, Divergent, Action] ), - %% `action` budget_exhausted is the one an operator must act on. + %% An operator must act when `action` is `budget_exhausted`. emit_telemetry( [dgen_registry, conflict], #{divergent_count => length(Divergent)}, #{name => Name, authority => Authority, action => Action} ). -%% An unreachable node counts as not-alive, so a conflict is never escalated on a -%% binding we cannot verify. +%% A pid on an unreachable node counts as dead, so we never act on a conflict over a +%% binding we can't verify. is_pid_alive(Pid) when node(Pid) =:= node() -> is_process_alive(Pid); is_pid_alive(Pid) -> @@ -1325,48 +1265,55 @@ is_pid_alive(Pid) -> end end. -%% Bounded snapshot read of a peer's records, version and trail; `error` if -%% disconnected or slow (its bindings are then unavailable to this gather). -member_names({Node, Name}) -> - case lists:member(Node, nodes()) of +%% Reads a peer's records, version and trail, and whether its leader is gone, with a +%% timeout. Returns `error` if the peer is disconnected or too slow. This gather then +%% goes ahead without its bindings. +get_peer_snapshot({Node, Name}) -> + case dgen_utils:node_reachable(Node) of false -> error; true -> - %% All inside `try`: decode_records can fault on a corrupt payload. + %% Everything is inside the `try`, because decode_records/1 can fail on a + %% corrupt payload. try - {Payload, Version, Released} = + {Payload, Version, Released, LeaderGone} = gen_statem:call({Name, Node}, get_names_snapshot, ?GATHER_TIMEOUT), Names = dgen_registry_replica:decode_records(Payload), - true = is_map(Names) andalso is_integer(Version) andalso is_map(Released), - {ok, {Names, Version, Released}} + true = + is_map(Names) andalso is_integer(Version) andalso is_map(Released) andalso + is_boolean(LeaderGone), + {ok, {Names, Version, Released}, LeaderGone} catch _:_ -> error end end. -%% §5.6 prevention (opt-in `reject_when_degraded`): while degraded, refuse names the -%% leader does not hold; they could be committed bindings the reconstruction missed. -reject_new_when_degraded(Name, #state{conflicts = C, replica = R, config = Config}) -> - dgen_registry_conflicts:degraded(C) andalso dgen_config:reject_when_degraded(Config) andalso +%% Prevention from §5.6, turned on with `reject_when_degraded`. While the leader is +%% degraded, it refuses names it doesn't hold. Such a name could have a committed +%% binding that the rebuild missed. +should_reject_when_degraded(Name, Data) -> + #data{conflicts = #conflicts{degraded = Degraded}, replica = R, config = Config} = Data, + Degraded andalso dgen_config:reject_when_degraded(Config) andalso not dgen_registry_replica:exists(R, Name). -%% With no (reachable) leader a registration is never answered `no` (that would be a -%% false `already_started`); it blocks as an intent until a leader appears. -route_register(Name, Pid, Meta, From, State = #state{leader = Leader, member_id = Self}) -> +%% Without a reachable leader we never answer a registration `no`, because the caller +%% would wrongly see `already_started`. It waits as an intent until a leader appears. +route_register(Name, Pid, Meta, From, Data) -> + #data{leader = Leader, member_id = Self} = Data, if Leader =:= Self -> - enqueue_register(Name, Pid, Meta, From, State); + enqueue_register(Name, Pid, Meta, From, Data); true -> - submit_intent({register, Name, Pid, Meta}, From, State) + submit_intent({register, Name, Pid, Meta}, From, Data) end. -%% Mirrors route_register. The bound pid is captured before the optimistic row_delete -%% so the leader can clear it and record it in the trail (§5.6). Leader: commit, answer -%% `ok` on commit. Follower: forward as a tracked intent, or with no reachable leader -%% answer `ok` now and hold the intent until one is (CP: never lost). -route_unregister( - LogicalName, From, State = #state{leader = Leader, member_id = Self, replica = R} -) -> +%% Works like route_register/5. We read the bound pid before the optimistic row delete, +%% so the leader can clear it and record it in the trail (§5.6). A leader commits the +%% unregister and answers `ok` once it commits. Any other member forwards it as an +%% intent. If no leader is reachable, we answer `ok` now and keep the intent until a +%% leader is reachable, so the unregister is never lost. +route_unregister(LogicalName, From, Data) -> + #data{leader = Leader, member_id = Self, replica = R} = Data, ReleasedPid = dgen_registry_replica:lookup_pid(R, LogicalName), if @@ -1374,148 +1321,163 @@ route_unregister( {noreply, enqueue_op( {remove, LogicalName, ReleasedPid, {unreg, From}}, - row_delete(State, LogicalName) + delete_row(Data, LogicalName) )}; true -> - State1 = row_delete(State, LogicalName), - {noreply, submit_intent({unregister, LogicalName, ReleasedPid}, From, State1)} + Data1 = delete_row(Data, LogicalName), + {noreply, submit_intent({unregister, LogicalName, ReleasedPid}, From, Data1)} end. -enqueue_register(Name, Pid, Meta, From, State) -> - case reject_new_when_degraded(Name, State) of +enqueue_register(Name, Pid, Meta, From, Data) -> + case should_reject_when_degraded(Name, Data) of true -> gen_server:reply(From, no), - State; + Data; false -> - enqueue_op({add, Name, Pid, Meta, {local, From}}, State) + enqueue_op({add, Name, Pid, Meta, {local, From}}, Data) end. %% --------------------------------------------------------------------------- -%% Intents — requests this member cannot decide itself (dgen_registry_intents) +%% Intents: requests this member can't decide itself (dgen_registry_intents) %% --------------------------------------------------------------------------- -intent_route(#state{leader = Self, member_id = Self}) -> +get_intent_route(#data{leader = Self, member_id = Self}) -> self; -intent_route(#state{leader = undefined}) -> +get_intent_route(#data{leader = undefined}) -> none; -intent_route(#state{leader = Leader}) -> - case member_reachable(Leader) of +get_intent_route(#data{leader = Leader}) -> + case is_member_reachable(Leader) of true -> {remote, Leader}; false -> none end. -submit_intent(Kind, From, State = #state{intents = I}) -> +submit_intent(Kind, From, Data = #data{intents = I}) -> Now = erlang:system_time(millisecond), - apply_intents(dgen_registry_intents:submit(Kind, From, Now, intent_route(State), I), State). - -intent_reply(Ref, Reply, State = #state{intents = I}) -> - apply_intents(dgen_registry_intents:on_reply(Ref, Reply, applied_version(State), I), State). - -%% Our replica advanced: release the gap-deferred `yes` acks it now covers. -intents_advanced(State = #state{intents = I}) -> - apply_intents(dgen_registry_intents:on_version(applied_version(State), I), State). - -%% Re-drive unresolved intents: `Scope` is `moved` for a new leadership view, `all` for -%% a restored link (see dgen_registry_intents:redrive/5). -redrive_intents(Scope, State = #state{intents = I, config = Config}) -> + apply_intents(dgen_registry_intents:submit(Kind, From, Now, get_intent_route(Data), I), Data). + +apply_intent_reply(Ref, Reply, Data) -> + #data{intents = I, replica = #replica{epoch = Epoch, version = Applied}} = Data, + Route = get_intent_route(Data), + apply_intents(dgen_registry_intents:on_reply(Ref, Reply, {Epoch, Applied}, Route, I), Data). + +%% Our replica advanced. Releases the held `yes` replies whose version it now covers. +release_deferred_acks(Data) -> + #data{intents = I, replica = #replica{epoch = Epoch, version = Applied}} = Data, + apply_intents(dgen_registry_intents:on_version({Epoch, Applied}, I), Data). + +%% Re-drives unresolved intents. `Scope` is `moved` when the leader changed, and `all` +%% when a connection came back. See dgen_registry_intents:redrive/5. +redrive_intents(Scope, Data) -> + #data{intents = I, config = Config} = Data, Now = erlang:system_time(millisecond), TTL = dgen_config:register_timeout(Config), - Route = intent_route(State), - apply_intents(dgen_registry_intents:redrive(Route, Scope, Now, TTL, I), State). + Route = get_intent_route(Data), + apply_intents(dgen_registry_intents:redrive(Route, Scope, Now, TTL, I), Data). -%% Drop intents whose callers timed out, bounding growth during a long partition. -prune_intents(State = #state{intents = I, config = Config}) -> +%% Drops intents whose callers timed out, so the map can't grow without bound during +%% a long partition. +prune_intents(Data) -> + #data{intents = I, config = Config} = Data, Now = erlang:system_time(millisecond), TTL = dgen_config:register_timeout(Config), - State#state{intents = dgen_registry_intents:prune(Now, TTL, I)}. + Data#data{intents = dgen_registry_intents:prune(Now, TTL, I)}. -apply_intents({Intents, Effects}, State) -> - lists:foldl(fun intent_effect/2, State#state{intents = Intents}, Effects). +apply_intents({Intents, Effects}, Data) -> + lists:foldl(fun apply_intent_effect/2, Data#data{intents = Intents}, Effects). -intent_effect({reply, From, Value}, State) -> +apply_intent_effect({reply, From, Value}, Data) -> gen_server:reply(From, Value), - State; -intent_effect({forward, Leader, Ref, Kind}, State = #state{member_id = Self}) -> - cast_to_member(Leader, forward_msg(Ref, Self, Kind)), - State; -intent_effect({retract, Leader, Name, Pid}, State) -> - forward_retract(Leader, Name, Pid), - State; -intent_effect({enqueue, {register, Name, Pid, Meta}, From}, State) -> - enqueue_register(Name, Pid, Meta, From, State); -intent_effect({enqueue, {unregister, Name, Pid}, answered}, State) -> - enqueue_op({retract, Name, Pid}, State); -intent_effect({resync, Leader}, State) -> - request_resync(Leader, State); -intent_effect({insert_row, Name, Pid, {Index, Data}}, State) -> - row_insert(State, Name, Pid, Index, Data). - -forward_msg(Ref, Self, {register, Name, Pid, Meta}) -> + Data; +apply_intent_effect({forward, Leader, Ref, Kind}, Data = #data{member_id = Self}) -> + cast_to_member(Leader, get_forward_msg(Ref, Self, Kind)), + Data; +%% Re-drives an unregister intent as a retract to a reachable leader. The retract only +%% clears the name if the same pid still holds it. +apply_intent_effect({retract, Leader, Name, Pid}, Data) -> + cast_to_member(Leader, {retract_req, Name, Pid}), + Data; +apply_intent_effect({enqueue, {register, Name, Pid, Meta}, From}, Data) -> + enqueue_register(Name, Pid, Meta, From, Data); +apply_intent_effect({enqueue, {unregister, Name, Pid}, answered}, Data) -> + enqueue_op({retract, Name, Pid}, Data); +apply_intent_effect({resync, Leader}, Data) -> + request_resync(Leader, Data). + +get_forward_msg(Ref, Self, {register, Name, Pid, Meta}) -> {register_req, Ref, Self, Name, Pid, Meta}; -forward_msg(Ref, Self, {unregister, Name, _ReleasedPid}) -> +get_forward_msg(Ref, Self, {unregister, Name, _ReleasedPid}) -> {unregister_req, Ref, Self, Name}; -forward_msg(Ref, Self, {set_meta, Name, Index, Data}) -> - {set_meta_req, Ref, Self, Name, Index, Data}. +get_forward_msg(Ref, Self, {set_meta, Name, Index, MetaData}) -> + {set_meta_req, Ref, Self, Name, Index, MetaData}. -%% Ops arriving during an in-flight commit accumulate and ride the next batch. -enqueue_op(Op, State = #state{pipeline = P}) -> - maybe_start_commit(State#state{pipeline = dgen_registry_pipeline:enqueue(Op, P)}). +%% Ops that arrive while a commit is in flight queue up and go in the next batch. +enqueue_op(Op, Data = #data{pipeline = P}) -> + maybe_start_commit(Data#data{pipeline = dgen_registry_pipeline:enqueue(Op, P)}). %% At most one commit runs at a time, in a dgen_transaction worker off this loop. A -%% member that no longer leads surrenders the queue instead. -maybe_start_commit(State = #state{pipeline = P, leader = Leader, member_id = Self}) -> - case {dgen_registry_pipeline:in_flight(P), dgen_registry_pipeline:num_pending(P)} of +%% member that no longer leads surrenders the queue instead. No timer drives this. +%% enqueue_op/2 starts a commit when none is running, and the worker's +%% `{dgen_transaction, …}` result starts the next one. This is the group commit from +%% §4.3. +maybe_start_commit(Data) -> + #data{ + pipeline = P = #pipeline{committing = Committing, num_pending = NumPending}, + leader = Leader, + member_id = Self + } = Data, + case {Committing =/= undefined, NumPending} of {true, _} -> - State; + Data; {false, 0} -> - State; + Data; {false, _} when Leader =/= Self -> {Ops, P1} = dgen_registry_pipeline:take_pending(P), - surrender(deposed, Ops, State#state{pipeline = P1}); + surrender(deposed, Ops, Data#data{pipeline = P1}); {false, _} -> - start_commit(State) + start_commit(Data) end. -start_commit(State) -> - Epoch = epoch(State), - Tab = dgen_registry_replica:tab(State#state.replica), - #state{ +start_commit(Data) -> + #data{ + replica = #replica{epoch = Epoch, tab = Tab}, member_id = Self, tenant = Tenant, tuid = Tuid, name_to_ref = NTR, pipeline = P, config = Config - } = State, + } = Data, {ThisBatch, P1} = dgen_registry_pipeline:take_batch(dgen_config:commit_batch_size(Config), P), - LastVersion = dgen_registry_pipeline:read_version(P1), - State1 = State#state{pipeline = P1}, - %% Planning is O(batch): bindings resolve lazily (see plan_batch/4). + #pipeline{last_version = LastVersion} = P1, + Data1 = Data#data{pipeline = P1}, + %% Planning costs O(batch), because it looks bindings up only as it needs them. See + %% plan_batch/4. Plan = plan_batch(ThisBatch, Tab, NTR, Epoch), #{dbop := DBOp, replies := RepliesRev} = Plan, case map_size(DBOp) =:= 0 of true -> - %% Only rejections: no commit needed. + %% The batch only has rejections, so there is nothing to commit. send_replies(lists:reverse(RepliesRev)), - maybe_start_commit(State1); + maybe_start_commit(Data1); false -> Ref = make_ref(), - %% Pin last_version as the read version to skip a GRV; if stale, the - %% leader-key read raises too-old and dgen_transaction retries with a fresh - %% one. The worker is monitored so its death still resolves the commit. Its - %% only write is the fenced version bump; the name delta applies in memory. + %% We reuse `last_version` as the read version to skip a GRV. If it is + %% stale, the leader-key read raises too-old, and dgen_transaction retries + %% with a fresh one. We monitor the worker so that its death still ends the + %% commit. The worker only writes the version key, and only if we still + %% lead. The name changes are applied in memory. {ok, {_Pid, MRef}} = dgen_registry_names:start_commit( Tenant, Tuid, Self, #{owner => self(), ref => Ref, epoch => Epoch, read_version => LastVersion} ), - State1#state{pipeline = dgen_registry_pipeline:start(Ref, MRef, Plan, P1)} + Data1#data{pipeline = dgen_registry_pipeline:start_commit(Ref, MRef, Plan, P1)} end. -%% Apply a committed batch: update the replica, (de)monitor pids, replicate, and -%% answer callers. -apply_committed_plan(Plan, Version, State) -> +%% Applies a committed batch. Updates the replica, monitors and demonitors pids, +%% replicates the batch, and answers callers. +apply_committed_plan(Plan, Version, Data) -> #{ dbop := DBOp, names := WNames, @@ -1525,97 +1487,102 @@ apply_committed_plan(Plan, Version, State) -> released := Released, direct_meta := DirectMeta } = Plan, - #state{ + #data{ member_id = Self, name_to_ref = NTR0, members = Members, - replica = R0 - } = State, - PrevVersion = dgen_registry_replica:version(R0), - %% Only DBOp names are touched, so a concurrent optimistic unregister of an - %% untouched name is never clobbered. - State0 = State#state{replica = dgen_registry_replica:commit(Version, DBOp, WNames, WMeta, R0)}, + replica = R0 = #replica{version = PrevVersion, epoch = Epoch} + } = Data, + %% We only touch the names in `DBOp`. An optimistic unregister of some other name + %% that happened meanwhile is left alone. + Data0 = Data#data{replica = dgen_registry_replica:commit(Version, DBOp, WNames, WMeta, R0)}, NTR1 = apply_monitor_ops(DBOp, WNames, NTR0), - %% Broadcast first, so the replicate_sync (FIFO behind it) reaches a follower only - %% after it holds the batch. One message per batch, never per name: a partially - %% received batch would report the full version with partial content, breaking - %% "same version implies same content" that the handoff gather relies on (§5.7). + %% Broadcast first. FIFO then delivers the replicate_sync to a follower only after + %% it holds the batch. We send one message per batch, never one per name. If a + %% follower received only part of a batch, it would report the full version with + %% only part of the content. The handoff gather relies on "same version means same + %% content" (§5.7). broadcast_batch(Members, lists:reverse(BcastsRev), PrevVersion, Version, Self), Now = erlang:system_time(millisecond), - State1 = State0#state{ + Data1 = Data0#data{ name_to_ref = NTR1, - pipeline = dgen_registry_pipeline:committed(Version, State0#state.pipeline), - %% Explicit releases extend the conflict trail (§5.6). - conflicts = dgen_registry_conflicts:released(Released, Now, State0#state.conflicts) + pipeline = dgen_registry_pipeline:set_last_version(Version, Data0#data.pipeline), + %% Explicit releases go into the conflict trail (§5.6). + conflicts = dgen_registry_conflicts:record_releases(Released, Now, Data0#data.conflicts) }, - %% Forwarded registrations and rejections are answered now (a forwarded `yes` is - %% already two-holder); a direct `yes` waits for a follower's replica ack (§8). - {DirectYes, Immediate} = lists:partition(fun is_direct_yes/1, lists:reverse(RepliesRev)), + %% We answer forwarded registrations and rejections now, because the follower that + %% forwarded a `yes` is already the second holder. A direct `yes` waits for a + %% follower's replica ack (§8). + {DirectYes, Immediate} = lists:partition( + fun + ({{local, _From}, yes}) -> true; + (_Other) -> false + end, + lists:reverse(RepliesRev) + ), lists:foreach( - fun({Origin, Result}) -> deliver_committed_reply(Origin, Result, Version) end, Immediate + fun({Origin, Result}) -> deliver_committed_reply(Origin, Result, Version, Epoch) end, + Immediate ), - %% {Name, Pid} lets strict_replication roll a direct binding back. + %% We keep `{Name, Pid}` so `strict_replication` can roll a direct binding back. EnrichedDirect = [ {Origin, Name, Pid} || {Origin, yes} <- DirectYes, {Name, Pid} <- [maps:get(Origin, DirectMeta)] ], - %% Presence notifications (§4.9), computed against the committed replica. - State2 = fire_presence(DBOp, State1), - confirm_direct(EnrichedDirect, Version, State2). - -%% Presence for a committed batch (leader only, §4.9), diffing the changed names per -%% subscription: watch-set changes yield `joined`/`left` for continuing notify targets; -%% a pid newly entering the notify set gets the full watch snapshot. -fire_presence(_DBOp, State = #state{subs = Subs}) when map_size(Subs) =:= 0 -> - State; -fire_presence(DBOp, State = #state{replica = R, subs = Subs}) -> - Tab = dgen_registry_replica:tab(R), + %% Send presence notifications (§4.9), computed against the committed replica. + Data2 = send_presence_events(DBOp, Data1), + await_replica_acks(EnrichedDirect, Version, Data2). + +%% Sends presence events for a committed batch. Only the leader does this (§4.9). For +%% each subscription we diff the changed names. Changes to the watch set become +%% `joined` and `left` events for the notify targets that were already there. A pid +%% that newly enters the notify set gets the full watch snapshot instead. +send_presence_events(_DBOp, Data = #data{subs = Subs}) when map_size(Subs) =:= 0 -> + Data; +send_presence_events(DBOp, Data) -> + #data{replica = #replica{tab = Tab}, subs = Subs} = Data, RealNames = maps:keys(DBOp), maps:fold( fun(SubId, {Watch, Notify}, S) -> diff_subscription(SubId, Watch, Notify, RealNames, Tab, S) end, - State, + Data, Subs ). -%% Seed a (re)subscription: compute both sets and send the watch snapshot to the -%% current notify targets. -seed_subscription( - SubId, - State = #state{ - subs = Subs, replica = R, sub_matches = SM, notify_matches = NM - } -) -> +%% Seeds a new or replaced subscription. Computes both sets and sends the watch +%% snapshot to the current notify targets. +seed_subscription(SubId, Data) -> + #data{subs = Subs, replica = R, sub_matches = SM, notify_matches = NM} = Data, case Subs of #{SubId := {Watch, Notify}} -> WatchMatches = dgen_registry_replica:query_match_names(R, Watch), NotifyMatches = dgen_registry_replica:query_match_names(R, Notify), - send_presence(SubId, snapshot_events(WatchMatches), pidset(NotifyMatches)), - State#state{ + send_presence(SubId, get_snapshot_events(WatchMatches), get_unique_pids(NotifyMatches)), + Data#data{ sub_matches = SM#{SubId => WatchMatches}, notify_matches = NM#{SubId => NotifyMatches} }; _ -> - State + Data end. -%% Diff one subscription and fire its notifications. Done on pid-sets, so a pid already -%% a target under another name gets no duplicate snapshot. -diff_subscription(SubId, Watch, Notify, RealNames, Tab, State) -> - #state{sub_matches = SM, notify_matches = NM} = State, +%% Diffs one subscription and sends its notifications. We compare sets of pids, so a +%% pid that is already a target under another name doesn't get a second snapshot. +diff_subscription(SubId, Watch, Notify, RealNames, Tab, Data) -> + #data{sub_matches = SM, notify_matches = NM} = Data, {WatchEvents, WatchAfter} = diff_query_matches(Watch, RealNames, maps:get(SubId, SM, #{}), Tab), NotifyBefore = maps:get(SubId, NM, #{}), {_NotifyEvents, NotifyAfter} = diff_query_matches(Notify, RealNames, NotifyBefore, Tab), - AfterPids = pidset(NotifyAfter), - NewTargets = AfterPids -- pidset(NotifyBefore), + AfterPids = get_unique_pids(NotifyAfter), + NewTargets = AfterPids -- get_unique_pids(NotifyBefore), ContinuingTargets = AfterPids -- NewTargets, send_presence(SubId, WatchEvents, ContinuingTargets), - send_presence(SubId, snapshot_events(WatchAfter), NewTargets), - State#state{sub_matches = SM#{SubId => WatchAfter}, notify_matches = NM#{SubId => NotifyAfter}}. + send_presence(SubId, get_snapshot_events(WatchAfter), NewTargets), + Data#data{sub_matches = SM#{SubId => WatchAfter}, notify_matches = NM#{SubId => NotifyAfter}}. -%% Diff a query's membership over the changed names: `joined`/`left` events and the -%% updated `#{Name => Pid}` map. +%% Diffs which of the changed names match a query. Returns the `joined` and `left` +%% events and the updated `#{Name => Pid}` map. diff_query_matches(Query, RealNames, Before, Tab) -> lists:foldl( fun(Name, {EvAcc, MatchAcc}) -> @@ -1649,10 +1616,10 @@ diff_query_matches(Query, RealNames, Before, Tab) -> RealNames ). -snapshot_events(Matches) -> +get_snapshot_events(Matches) -> [{joined, Name, Pid} || {Name, Pid} <- maps:to_list(Matches)]. -pidset(Matches) -> +get_unique_pids(Matches) -> lists:usort(maps:values(Matches)). send_presence(_SubId, [], _Pids) -> @@ -1663,51 +1630,47 @@ send_presence(SubId, Events, Pids) -> Msg = {dgen_presence, SubId, Events}, lists:foreach(fun(Pid) -> Pid ! Msg end, Pids). -is_direct_yes({{local, _From}, yes}) -> true; -is_direct_yes(_Other) -> false. - -%% Replicate-before-ack for direct registrations (§5.5, §8): wait for -%% `register_replicas` distinct follower acks (capped by follower count; none needed if -%% there are no followers), or the `strict_replication` policy on timeout. -confirm_direct([], _Version, State) -> - State; -confirm_direct( - Direct, - Version, - State = #state{members = Members, member_id = Self, pipeline = P, config = Config} -) -> +%% Makes direct registrations wait for replication before we ack them (§5.5, §8). We +%% wait for acks from `register_replicas` different followers, capped at the number of +%% followers. With no followers we answer right away. On timeout the +%% `strict_replication` setting decides the answer. +await_replica_acks([], _Version, Data) -> + Data; +await_replica_acks(Direct, Version, Data) -> + #data{members = Members, member_id = Self, pipeline = P, config = Config} = Data, Needed = min(dgen_config:register_replicas(Config), map_size(Members)), case Needed =< 0 of true -> - deliver_direct(Direct), - State; + deliver_direct_yes(Direct), + Data; false -> BatchRef = make_ref(), - %% Carries Version so a gapped follower that lacks the batch never acks. + %% It carries `Version`, so a follower that missed the batch never acks. broadcast_to_peers(Members, {replicate_sync, BatchRef, Self, Version}), - TimerRef = arm( + TimerRef = schedule_timer( dgen_config:replicate_timeout(Config), {replicate_timeout, BatchRef} ), - State#state{ + Data#data{ pipeline = dgen_registry_pipeline:await_acks(BatchRef, Direct, TimerRef, Needed, P) } end. -deliver_direct(Direct) -> +deliver_direct_yes(Direct) -> lists:foreach(fun({Origin, _Name, _Pid}) -> deliver_reply(Origin, yes) end, Direct). -%% strict_replication fail-closed: reject and retract (a pid-guarded, committed clear). -reject_and_retract_direct(Direct, State) -> +%% Fails closed under `strict_replication`. Answers each caller `no` and commits a +%% retract, which only clears the name if the same pid still holds it. +reject_and_retract_direct(Direct, Data) -> lists:foldl( fun({Origin, Name, Pid}, S) -> deliver_reply(Origin, no), enqueue_op({retract, Name, Pid}, S) end, - State, + Data, Direct ). -%% Degrade-open event (§8): telemetry and a log warning. +%% Reports a degrade-open (§8) with telemetry and a log warning. notify_degrade_open(Needed, Got, Count) -> logger:warning( "dgen_registry: degrade-open — acked ~b direct registration(s) leader-only " @@ -1720,18 +1683,18 @@ notify_degrade_open(Needed, Got, Count) -> #{needed => Needed, got => Got} ). -%% `telemetry` is a soft dependency. Availability is resolved once per VM and cached: -%% `code:ensure_loaded/1` on a missing module calls `code_server` every time, which -%% also breaks reproducibility under eta simulation. +%% `telemetry` is an optional dependency. We check once per VM whether it is loaded and +%% cache the answer. `code:ensure_loaded/1` on a missing module calls `code_server` +%% every time, which is slow and also makes runs under eta simulation unreproducible. -define(TELEMETRY_KEY, {?MODULE, telemetry_available}). emit_telemetry(Event, Measurements, Metadata) -> - case telemetry_available() of + case is_telemetry_available() of true -> apply(telemetry, execute, [Event, Measurements, Metadata]); false -> ok end. -telemetry_available() -> +is_telemetry_available() -> case persistent_term:get(?TELEMETRY_KEY, undefined) of undefined -> Available = code:ensure_loaded(telemetry) =:= {module, telemetry}, @@ -1741,8 +1704,8 @@ telemetry_available() -> Available end. -%% Answer a write op's origin: gen_server:reply/2 locally, or a reply cast to the -%% forwarding member. +%% Answers whoever sent a write op. A local caller gets gen_server:reply/2. A +%% forwarding member gets a reply cast. deliver_reply({local, From}, Result) -> gen_server:reply(From, Result); deliver_reply({meta, From}, Result) -> @@ -1756,18 +1719,20 @@ deliver_reply({forward_meta, MemberId, Ref}, Result) -> deliver_reply({forward_unreg, MemberId, Ref}, _Result) -> cast_to_member(MemberId, {unregister_reply, Ref}). -%% A forwarded registration's reply carries the commit version so the follower can -%% version-guard its `yes` (see the {register_reply, _, _, _} handler). -deliver_committed_reply({forward, MemberId, Ref}, Result, Version) -> - cast_to_member(MemberId, {register_reply, Ref, Result, Version}); -deliver_committed_reply(Origin, Result, _Version) -> +%% The reply to a forwarded registration carries the commit version and our epoch. +%% The follower holds a `yes` until its replica reaches that version in that epoch. +%% See the {register_reply, _, _, _, _} handler. +deliver_committed_reply({forward, MemberId, Ref}, Result, Version, Epoch) -> + cast_to_member(MemberId, {register_reply, Ref, Result, Version, Epoch}); +deliver_committed_reply(Origin, Result, _Version, _Epoch) -> deliver_reply(Origin, Result). -%% Fold the ordered ops into a plan: the working overlay, the net durable delta (`dbop`, -%% last write wins), and replies and broadcasts (newest first). `wntr` tracks monitor -%% refs for the `down` ref-match guard; a name bound in this batch carries a -%% `{pending, Pid}` placeholder that never matches a real ref. `names` starts empty: -%% each op resolves its one name via seed_lookup/3, so planning is O(batch). +%% Folds the ops, in order, into a plan. The plan holds the batch overlay, the net +%% change per name in `dbop` where the last write wins, and the replies and broadcasts, +%% newest first. `wntr` tracks monitor refs, so a `down` op only clears a name if its +%% ref still matches. A name bound in this batch gets a `{pending, Pid}` placeholder +%% that never matches a real ref. `names` starts empty. Each op looks up its one name +%% with lookup_in_batch/3, so planning costs O(batch). plan_batch(Ops, Tab, NTR, Epoch) -> Init = #{ tab => Tab, @@ -1779,24 +1744,25 @@ plan_batch(Ops, Tab, NTR, Epoch) -> bcasts => [], released => [], direct_meta => #{}, - %% A plan whose epoch is no longer current is not applied; broadcasts carry it. + %% We don't apply a plan whose epoch is no longer current. Broadcasts carry it. epoch => Epoch, - %% Kept so a failed batch can be surrendered. + %% Kept so we can surrender the ops if the batch fails. ops => Ops }, lists:foldl(fun(Op, Acc) -> plan_op(Op, Epoch, Acc) end, Init, Ops). -%% A name's pid partway through this batch: the batch overlay `WN` first, else ETS. -%% `removed` marks a name cleared earlier in this batch: only unregister deletes from -%% ETS eagerly, so a prior `retract` or `down` is visible only through the overlay. -seed_lookup(WN, Tab, Name) -> +%% Returns a name's pid as of this point in the batch. We check the batch overlay `WN` +%% first, then ETS. `removed` marks a name cleared earlier in this batch. Only +%% unregister deletes from ETS right away, so an earlier `retract` or `down` shows up +%% only in the overlay. +lookup_in_batch(WN, Tab, Name) -> case WN of #{Name := removed} -> undefined; #{Name := Pid} -> Pid; _ -> lookup_name(Tab, Name) end. -plan_op({add, Name, Pid, {Index, Data} = Meta, Origin}, Epoch, Acc) -> +plan_op({add, Name, Pid, {Index, MetaData} = Meta, Origin}, Epoch, Acc) -> #{ tab := Tab, names := WN, @@ -1807,14 +1773,15 @@ plan_op({add, Name, Pid, {Index, Data} = Meta, Origin}, Epoch, Acc) -> bcasts := Bs, direct_meta := DM } = Acc, - case seed_lookup(WN, Tab, Name) of + case lookup_in_batch(WN, Tab, Name) of Taken when is_pid(Taken), Taken =/= Pid -> - %% Held by a different pid: reject (the singleton contract). + %% A different pid holds the name, so we reject. One name, one pid. Acc#{replies := [{Origin, no} | Rs]}; _Free_or_same -> - %% Free, or the same pid: an idempotent `yes` (§5.2), so a caller whose - %% call timed out can re-drive and get a decisive answer. Direct-origin - %% bindings are recorded for strict_replication rollback. + %% The name is free, or the same pid holds it. Either way we answer `yes` + %% (§5.2), so a caller whose call timed out can retry and get a clear + %% answer. We record direct bindings so `strict_replication` can roll them + %% back. DM1 = case Origin of {local, _From} -> DM#{Origin => {Name, Pid}}; @@ -1826,28 +1793,29 @@ plan_op({add, Name, Pid, {Index, Data} = Meta, Origin}, Epoch, Acc) -> wntr := WNTR#{Name => {pending, Pid}}, dbop := DB#{Name => {set, node(Pid)}}, replies := [{Origin, yes} | Rs], - bcasts := [{name_registered, Name, Pid, Index, Data, Epoch} | Bs], + bcasts := [{name_registered, Name, Pid, Index, MetaData, Epoch} | Bs], direct_meta := DM1 } end; -plan_op({set_meta, Name, Index, Data, Origin}, Epoch, Acc) -> +plan_op({set_meta, Name, Index, MetaData, Origin}, Epoch, Acc) -> #{tab := Tab, names := WN, meta := WMeta, dbop := DB, replies := Rs, bcasts := Bs} = Acc, - case seed_lookup(WN, Tab, Name) of + case lookup_in_batch(WN, Tab, Name) of undefined -> Acc#{replies := [{Origin, {error, not_registered}} | Rs]}; Pid -> - %% Keep an in-batch {set,…} so the new pid is still monitored; otherwise - %% {meta,…} leaves the existing monitor alone. + %% If this batch already has a {set, …} for the name, we keep it so the new + %% pid still gets monitored. Otherwise {meta, …} leaves the existing + %% monitor alone. DBVal = case maps:get(Name, DB, undefined) of {set, _} = SetOp -> SetOp; _ -> {meta, node(Pid)} end, Acc#{ - meta := WMeta#{Name => {Index, Data}}, + meta := WMeta#{Name => {Index, MetaData}}, dbop := DB#{Name => DBVal}, replies := [{Origin, ok} | Rs], - bcasts := [{metadata_set, Name, Index, Data, Epoch} | Bs] + bcasts := [{metadata_set, Name, Index, MetaData, Epoch} | Bs] } end; plan_op({remove, Name, ReleasedPid, Origin}, Epoch, Acc) -> @@ -1860,17 +1828,18 @@ plan_op({remove, Name, ReleasedPid, Origin}, Epoch, Acc) -> bcasts := Bs, released := Rel } = Acc, - %% Clear if the name was bound at enqueue (ReleasedPid; the optimistic delete - %% already removed it from ETS) or is bound now in this batch (seed_lookup). The - %% released pair joins the conflict trail (§5.6). The origin gets `ok` either way. - Rs1 = unreg_reply(Origin, Rs), + %% We clear the name if it was bound when the op was enqueued, or if it is bound + %% now in this batch. In the first case `ReleasedPid` is that pid, and the + %% optimistic delete already removed the row from ETS. The released `{Name, Pid}` + %% goes into the conflict trail (§5.6). The caller gets `ok` either way. + Rs1 = maybe_add_unreg_reply(Origin, Rs), WasBound = is_pid(ReleasedPid), - Current = seed_lookup(WN, Tab, Name), - %% Guard against the *current* holder: the optimistic row_delete frees the name at - %% arrival, so a register queued ahead of this remove may already have rebound it. - %% Clearing that new holder would break Guarantee 1. A captured pid clears only if - %% the current holder is that pid or nobody; a capture of `undefined` clears - %% whatever is bound (unregister by name). + Current = lookup_in_batch(WN, Tab, Name), + %% Check the *current* holder. The optimistic row delete frees the name as soon as + %% the unregister arrives, so a register queued ahead of this remove may already + %% have bound it again. Clearing that new holder would break Guarantee 1. If we + %% captured a pid, we only clear the name when that pid or nobody holds it. If we + %% captured `undefined`, we clear whatever is bound, as an unregister by name. Rebound = WasBound andalso is_pid(Current) andalso Current =/= ReleasedPid, case (WasBound orelse Current =/= undefined) andalso not Rebound of true -> @@ -1879,7 +1848,8 @@ plan_op({remove, Name, ReleasedPid, Origin}, Epoch, Acc) -> true -> [{Name, ReleasedPid} | Rel]; false -> Rel end, - %% The released pid rides the broadcast into every member's trail (§5.6). + %% The broadcast carries the released pid, so every member adds it to its + %% trail (§5.6). ReleasedOut = case WasBound of true -> ReleasedPid; @@ -1898,9 +1868,9 @@ plan_op({remove, Name, ReleasedPid, Origin}, Epoch, Acc) -> end; plan_op({retract, Name, Pid}, Epoch, Acc) -> #{tab := Tab, names := WN, wntr := WNTR, dbop := DB, bcasts := Bs, released := Rel} = Acc, - %% Pid-guarded (strict_replication fail-closed): clear only if still this pid's, - %% and trail it so a lagging follower's copy is not mistaken for a conflict. - case seed_lookup(WN, Tab, Name) of + %% We only clear the name if this pid still holds it. We also add it to the trail, + %% so a lagging follower that still has the binding isn't taken for a conflict. + case lookup_in_batch(WN, Tab, Name) of Pid -> Acc#{ names := WN#{Name => removed}, @@ -1916,7 +1886,8 @@ plan_op({down, Name, Ref}, Epoch, Acc) -> #{names := WN, wntr := WNTR, dbop := DB, bcasts := Bs} = Acc, case maps:get(Name, WNTR, undefined) of Ref -> - %% The pid is dead, so nothing is trailed (the trail is for live releases). + %% The pid is dead, so we add nothing to the trail. The trail is only for + %% pids released while alive. Acc#{ names := WN#{Name => removed}, wntr := maps:remove(Name, WNTR), @@ -1924,16 +1895,17 @@ plan_op({down, Name, Ref}, Epoch, Acc) -> bcasts := [{name_unregistered, Name, undefined, Epoch} | Bs] }; _ -> - %% Stale DOWN (re-registered or already removed). + %% A stale DOWN. The name was registered again or already removed. Acc end. -%% A remove with origin `undefined` (legacy cast or retract) has no caller. -unreg_reply(undefined, Rs) -> Rs; -unreg_reply(Origin, Rs) -> [{Origin, ok} | Rs]. +%% A remove whose origin is `undefined` has no caller to answer. That happens when we +%% retry a surrendered remove. +maybe_add_unreg_reply(undefined, Rs) -> Rs; +maybe_add_unreg_reply(Origin, Rs) -> [{Origin, ok} | Rs]. -%% Monitor side effects of the delta: clear demonitors, set re-monitors the new pid, -%% meta leaves the monitor alone. +%% Updates monitors to match the batch. `clear` demonitors, `set` monitors the new pid +%% in place of the old one, and `meta` leaves the monitor alone. apply_monitor_ops(DBOp, WNames, NTR) -> maps:fold( fun @@ -1953,19 +1925,22 @@ apply_monitor_ops(DBOp, WNames, NTR) -> send_replies(Replies) -> lists:foreach(fun({Origin, Reply}) -> deliver_reply(Origin, Reply) end, Replies). -%% Extend the conflict trail (§5.6) from an applied batch. Keyed by {Name, Pid}, so a -%% release under one name never masks a conflict on another name the same pid holds. -record_released(Ops, State = #state{conflicts = C}) -> +%% Adds the releases in an applied batch to the conflict trail (§5.6). The trail is +%% keyed by `{Name, Pid}`, so releasing one name never hides a conflict on another +%% name the same pid holds. +record_releases(Ops, Data = #data{conflicts = C}) -> Pairs = [{Name, Pid} || {name_unregistered, Name, Pid} <- Ops, is_pid(Pid)], Now = erlang:system_time(millisecond), - State#state{conflicts = dgen_registry_conflicts:released(Pairs, Now, C)}. + Data#data{conflicts = dgen_registry_conflicts:record_releases(Pairs, Now, C)}. -%% One `{names_batch, Ops, Epoch, PrevV, Version, LeaderId}` per committed batch; the -%% epoch is carried once on the envelope. Followers use PrevV/Version for gap detection -%% and LeaderId as the resync target. An empty batch sends nothing. +%% Sends one `{names_batch, Ops, Epoch, PrevV, Version, LeaderId}` per committed batch. +%% The epoch appears once, on the message, not on each op. Followers use `PrevV` and +%% `Version` to detect gaps, and ask `LeaderId` for a resync. An empty batch sends +%% nothing. -ifdef(MUTATION_PARTIAL_BATCH). -%% MUTATION (test-only): reintroduces one message per op sharing {PrevV, Version}, so a -%% partially received batch diverges silently. Paired with dgen_registry_replica. +%% MUTATION, for tests only. Sends one message per op, all with the same +%% `{PrevV, Version}`, so a follower that receives only part of a batch diverges +%% without noticing. It pairs with the same mutation in dgen_registry_replica. broadcast_batch(_Members, [], _PrevV, _Version, _Self) -> ok; broadcast_batch(Members, Ops, PrevV, Version, Self) -> @@ -1973,7 +1948,7 @@ broadcast_batch(Members, Ops, PrevV, Version, Self) -> fun(Op) -> broadcast_to_peers( Members, - {names_batch, [strip_bcast_epoch(Op)], bcast_epoch(Op), PrevV, Version, Self} + {names_batch, [strip_bcast_epoch(Op)], get_bcast_epoch(Op), PrevV, Version, Self} ) end, Ops @@ -1982,78 +1957,86 @@ broadcast_batch(Members, Ops, PrevV, Version, Self) -> broadcast_batch(_Members, [], _PrevV, _Version, _Self) -> ok; broadcast_batch(Members, [First | _] = Ops, PrevV, Version, Self) -> - Epoch = bcast_epoch(First), + Epoch = get_bcast_epoch(First), Stripped = [strip_bcast_epoch(Op) || Op <- Ops], broadcast_to_peers(Members, {names_batch, Stripped, Epoch, PrevV, Version, Self}). -endif. -ifdef(MUTATION_QUIET_RESYNC). -%% MUTATION (test-only, `DGEN_MUTATION=quiet_resync`): the heartbeat advertises -%% nothing, so a follower that loses the stream's tail stays diverged while quiet. -broadcast_heartbeat(_State) -> +%% MUTATION, for tests only (`DGEN_MUTATION=quiet_resync`). The heartbeat sends +%% nothing, so a follower that lost the end of the stream stays behind while there +%% are no writes. +broadcast_heartbeat(_Data) -> ok. -else. -%% The leader's periodic empty batch at {Applied, Applied}; see +%% Sends the leader's periodic empty batch at `{Applied, Applied}`. See %% ?REPLICA_HEARTBEAT_INTERVAL. -broadcast_heartbeat(State = #state{members = Members, member_id = Self}) -> - Applied = applied_version(State), - broadcast_to_peers(Members, {names_batch, [], epoch(State), Applied, Applied, Self}), +broadcast_heartbeat(Data) -> + #data{ + members = Members, + member_id = Self, + replica = #replica{epoch = Epoch, version = Applied} + } = Data, + broadcast_to_peers(Members, {names_batch, [], Epoch, Applied, Applied, Self}), ok. -endif. -bcast_epoch({name_registered, _Name, _Pid, _Index, _Data, Epoch}) -> Epoch; -bcast_epoch({metadata_set, _Name, _Index, _Data, Epoch}) -> Epoch; -bcast_epoch({name_unregistered, _Name, _ReleasedPid, Epoch}) -> Epoch. +get_bcast_epoch({name_registered, _Name, _Pid, _Index, _Data, Epoch}) -> Epoch; +get_bcast_epoch({metadata_set, _Name, _Index, _Data, Epoch}) -> Epoch; +get_bcast_epoch({name_unregistered, _Name, _ReleasedPid, Epoch}) -> Epoch. -strip_bcast_epoch({name_registered, Name, Pid, Index, Data, _Epoch}) -> - {name_registered, Name, Pid, Index, Data}; -strip_bcast_epoch({metadata_set, Name, Index, Data, _Epoch}) -> - {metadata_set, Name, Index, Data}; +strip_bcast_epoch({name_registered, Name, Pid, Index, MetaData, _Epoch}) -> + {name_registered, Name, Pid, Index, MetaData}; +strip_bcast_epoch({metadata_set, Name, Index, MetaData, _Epoch}) -> + {metadata_set, Name, Index, MetaData}; strip_bcast_epoch({name_unregistered, Name, ReleasedPid, _Epoch}) -> {name_unregistered, Name, ReleasedPid}. -%% Ask `LeaderId` for a snapshot after a gap, at most once per ?RESYNC_RETRY. -request_resync(LeaderId, State = #state{resync_timer = undefined, member_id = Self}) -> +%% Asks `LeaderId` for a snapshot after a gap, at most once per ?RESYNC_RETRY. +request_resync(LeaderId, Data = #data{resync_timer = undefined}) -> + #data{member_id = Self} = Data, cast_to_member(LeaderId, {resync_req, Self}), - Ref = arm(?RESYNC_RETRY, resync_timeout), - State#state{resync_timer = Ref}; -request_resync(_LeaderId, State) -> - State. - -cancel_resync(State = #state{resync_timer = undefined}) -> - State; -cancel_resync(State = #state{resync_timer = Ref}) -> + Ref = schedule_timer(?RESYNC_RETRY, resync_timeout), + Data#data{resync_timer = Ref}; +request_resync(_LeaderId, Data) -> + Data. + +cancel_resync(Data = #data{resync_timer = undefined}) -> + Data; +cancel_resync(Data = #data{resync_timer = Ref}) -> erlang:cancel_timer(Ref), - State#state{resync_timer = undefined}. - -%% Mark synced, re-announcing once with `fresh = false`. -mark_synced(State = #state{synced = true}) -> - State; -mark_synced(State = #state{elector = Elector, member_id = Self, join_token = Token}) -> - dgen_server:cast(Elector, {join, Self, Token, false}), - State#state{synced = true}. - -%% Connected, so a cast will not be dropped? -member_reachable({Node, _Name}) -> + Data#data{resync_timer = undefined}. + +%% Marks us synced. The first time, we join again with `fresh = false`. +mark_synced(Data = #data{synced = true}) -> + Data; +mark_synced(Data) -> + #data{elector = Elector, member_id = Self, join_token = Token} = Data, + dgen_registry_elector:join(Elector, Self, Token, false), + Data#data{synced = true}. + +%% Returns whether the member's node is connected, so a cast to it won't be dropped. +is_member_reachable({Node, _Name}) -> dgen_utils:node_reachable(Node). -%% Report a peer's death with the elector's current token (fallback `Cached`), from a -%% helper so this loop never blocks on the elector (see the DOWN handler). +%% Reports a peer's death with the elector's current token, falling back to `Cached`. +%% It runs in a helper so this loop never blocks on the elector. See the DOWN handler. spawn_member_down(Elector, MemberId, Cached) -> _ = spawn(fun() -> - Token = dgen_registry_elector:member_token(Elector, MemberId, Cached), - dgen_server:cast(Elector, {member_down, MemberId, Token}) + Token = dgen_registry_elector:get_member_token(Elector, MemberId, Cached), + dgen_registry_elector:member_down(Elector, MemberId, Token) end), ok. -%% Guarantee 5: answer a consistent read only after verifying against the durable -%% leader key that we are still the leader for our epoch; a deposed leader on the -%% minority side of a partition would otherwise serve its frozen replica. `Value` is -%% captured at the mailbox point; the verify runs in a helper so the DB round-trip -%% never blocks the loop. Linearizable because leadership is continuous within an -%% epoch. Any failure answers `Denied` (the CP refusal). -reply_fenced(From, Value, Denied, State = #state{tenant = Tenant, tuid = Tuid, member_id = Self}) -> - Epoch = epoch(State), +%% Guarantee 5. Answers a consistent read only after the durable leader key confirms we +%% still lead in our epoch. Otherwise a deposed leader on the minority side of a +%% partition would serve its frozen replica. We capture `Value` at this point in the +%% mailbox. The check runs in a helper so the database round trip never blocks the +%% loop. The read is linearizable because leadership doesn't change within an epoch. +%% If the check fails for any reason, the caller gets `Denied`. +reply_fenced(From, Value, Denied, Data) -> + #data{tenant = Tenant, tuid = Tuid, member_id = Self, replica = #replica{epoch = Epoch}} = + Data, _ = spawn(fun() -> Reply = @@ -2065,44 +2048,52 @@ reply_fenced(From, Value, Denied, State = #state{tenant = Tenant, tuid = Tuid, m end), ok. -%% Pid-guarded retract to a reachable leader (an unregister intent's re-drive). -forward_retract(Leader, Name, Pid) -> - cast_to_member(Leader, {retract_req, Name, Pid}). - -%% Applied only if committed under the leadership view it was planned in. -commit_result(Reply, Plan, State = #state{leader = Leader, member_id = Self}) -> +%% Applies the commit result, but only if we still lead in the epoch the plan was made +%% in. +handle_commit_result(Reply, Plan, Data) -> + #data{leader = Leader, member_id = Self, replica = #replica{epoch = Epoch}} = Data, case Reply of {committed, Version} when Leader =:= Self -> - case maps:get(epoch, Plan) =:= epoch(State) of + case maps:get(epoch, Plan) =:= Epoch of true -> - apply_committed_plan(Plan, Version, State); + apply_committed_plan(Plan, Version, Data); false -> - %% Older epoch: applying or broadcasting an old-term plan would - %% diverge replicas. Only the version bump was durable, so retry. - surrender(failed, maps:get(ops, Plan), State) + %% The plan is from an older epoch. Applying or broadcasting it + %% would make replicas diverge. Only the version bump was written, + %% so we retry the ops. + surrender(failed, maps:get(ops, Plan), Data) end; {aborted, fenced} -> - %% A newer leader exists and we have not heard: ask now. - surrender(failed, maps:get(ops, Plan), rejoin_once_fenced(State)); + %% A newer leader exists and we haven't heard about it. We ask now. + surrender(failed, maps:get(ops, Plan), maybe_rejoin(Data)); _Deposed_or_error -> - surrender(failed, maps:get(ops, Plan), State) + surrender(failed, maps:get(ops, Plan), Data) end. -%% A fenced commit proves we were deposed unheard: re-announce now rather than wait -%% for the connector's durable-epoch check. Once per epoch, since each join is a -%% durable queue entry and a partitioned member keeps being fenced. -rejoin_once_fenced(State = #state{fence_rejoined = Epoch}) when Epoch =/= undefined -> - case epoch(State) of - Epoch -> State; - _ -> rejoin_once_fenced(State#state{fence_rejoined = undefined}) +%% A commit aborted with `fenced` proves we were deposed without hearing about it. We +%% join again now instead of waiting for the connector's durable epoch check. We do +%% this at most once per epoch. Each join is a durable queue entry, and a partitioned +%% member keeps getting `fenced`. +maybe_rejoin(Data = #data{fence_rejoined = Fenced}) when Fenced =/= undefined -> + #data{replica = #replica{epoch = Epoch}} = Data, + case Epoch of + Fenced -> Data; + _ -> maybe_rejoin(Data#data{fence_rejoined = undefined}) end; -rejoin_once_fenced(State) -> - #state{elector = Elector, member_id = Self, join_token = Token, synced = Synced} = State, - dgen_server:cast(Elector, {join, Self, Token, not Synced}), - State#state{fence_rejoined = epoch(State)}. - -%% Carry out dgen_registry_pipeline:surrender/2: replies, intents, delayed retries. -surrender(Reason, Ops, State) -> +maybe_rejoin(Data) -> + #data{ + elector = Elector, + member_id = Self, + join_token = Token, + synced = Synced, + replica = #replica{epoch = Epoch} + } = Data, + dgen_registry_elector:join(Elector, Self, Token, not Synced), + Data#data{fence_rejoined = Epoch}. + +%% Carries out what dgen_registry_pipeline:surrender/2 returns. That is replies, +%% intents and delayed retries. +surrender(Reason, Ops, Data) -> lists:foldl( fun ({reply, Origin, Value}, S) -> @@ -2111,29 +2102,30 @@ surrender(Reason, Ops, State) -> ({intent, Kind, From}, S) -> submit_intent(Kind, From, S); ({retry, Retry}, S) -> - arm(?REQUEUE_DELAY, {requeue_ops, Retry}), + schedule_timer(?REQUEUE_DELAY, {requeue_ops, Retry}), S end, - State, + Data, dgen_registry_pipeline:surrender(Reason, Ops) ). -%% Only a local direct registration's caller can be checked; anything else is kept. -caller_alive({add, _Name, _Pid, _Meta, {local, {Caller, _Tag}}}) when node(Caller) =:= node() -> +%% We can only check the caller of a local direct registration. Any other op is kept. +is_caller_alive({add, _Name, _Pid, _Meta, {local, {Caller, _Tag}}}) when node(Caller) =:= node() -> is_process_alive(Caller); -caller_alive(_Op) -> +is_caller_alive(_Op) -> true. -%% Logs the arm so a simulated `{clock, Ms}` trace entry is attributable; `?ETA_LOG` -%% is a no-op in production builds. -arm(Delay, Msg) -> +%% Logs an `arm` event, so a `{clock, Ms}` entry in a simulation trace can be traced +%% back to its timer. `?ETA_LOG` does nothing in production builds. +schedule_timer(Delay, Msg) -> _ = ?ETA_LOG({arm, Msg, Delay}), erlang:send_after(Delay, self(), Msg). -%% Losing leadership with direct registrations awaiting acks: resolve them by the -%% replicate-timeout policy (§8). Fail-closed hands the pid-guarded retract to the -%% intents so the caller's `no` cannot leave the binding alive. -fail_pending_acks(State = #state{pipeline = P, config = Config}) -> +%% We lost leadership while direct registrations were waiting for acks. Resolves them +%% the same way as a replicate timeout (§8). When failing closed, we submit the +%% retract as an intent, so a caller told `no` can't leave the binding in place. +fail_pending_acks(Data) -> + #data{pipeline = P, config = Config} = Data, {Batches, P1} = dgen_registry_pipeline:take_acks(P), Strict = dgen_config:strict_replication(Config), lists:foldl( @@ -2142,7 +2134,7 @@ fail_pending_acks(State = #state{pipeline = P, config = Config}) -> case Strict of false -> notify_degrade_open(Needed, Got, length(Direct)), - deliver_direct(Direct), + deliver_direct_yes(Direct), S; true -> lists:foldl( @@ -2155,7 +2147,7 @@ fail_pending_acks(State = #state{pipeline = P, config = Config}) -> ) end end, - State#state{pipeline = P1}, + Data#data{pipeline = P1}, Batches ). @@ -2163,23 +2155,19 @@ fail_pending_acks(State = #state{pipeline = P, config = Config}) -> %% Row writes outside the stream (see dgen_registry_replica) %% --------------------------------------------------------------------------- -%% The optimistic delete on unregister, and the legacy unversioned reply's insert -%% (rolling upgrade). Neither moves the replica's version. -row_delete(S = #state{replica = R}, Name) -> - S#state{replica = dgen_registry_replica:delete(R, Name)}. - -row_insert(S = #state{replica = R}, Name, Pid, Index, Data) -> - S#state{replica = dgen_registry_replica:insert(R, Name, Pid, Index, Data)}. +%% Deletes the row optimistically on unregister. It doesn't change the replica version. +delete_row(S = #data{replica = R}, Name) -> + S#data{replica = dgen_registry_replica:delete(R, Name)}. -%% Bare table, so plan_batch/4 stays testable against a plain ETS table. +%% Takes the bare table, so plan_batch/4 can be tested against a plain ETS table. lookup_name(Tab, Name) -> case ets:lookup(Tab, Name) of [{_Name, Pid, _Index, _Data} | _] -> Pid; [] -> undefined end. -add_member_monitors(MemberIds, State) -> - #state{member_id = Self, members = Members, monitors = Monitors} = State, +add_member_monitors(MemberIds, Data) -> + #data{member_id = Self, members = Members, monitors = Monitors} = Data, {NewMembers, NewMonitors} = lists:foldl( fun(MemberId, {MA, MonA}) -> case MemberId =:= Self orelse maps:is_key(MemberId, MA) of @@ -2194,22 +2182,23 @@ add_member_monitors(MemberIds, State) -> {Members, Monitors}, MemberIds ), - State#state{members = NewMembers, monitors = NewMonitors}. + Data#data{members = NewMembers, monitors = NewMonitors}. -remove_member(MemberId, State = #state{members = Members, monitors = Monitors}) -> +remove_member(MemberId, Data) -> + #data{members = Members, monitors = Monitors} = Data, case maps:get(MemberId, Members, undefined) of undefined -> - State; + Data; Ref -> erlang:demonitor(Ref, [flush]), - %% Drop the peer's token (the caller already read it). The departure time - %% keeps trail pruning suspended for a TTL window (§5.6). - State#state{ + %% Drop the peer's token, which the caller has already read. Recording the + %% departure time pauses trail pruning for a while (§5.6). + Data#data{ members = maps:remove(MemberId, Members), monitors = maps:remove(Ref, Monitors), - peer_tokens = maps:remove(MemberId, State#state.peer_tokens), - conflicts = dgen_registry_conflicts:departed( - erlang:system_time(millisecond), State#state.conflicts + peer_tokens = maps:remove(MemberId, Data#data.peer_tokens), + conflicts = dgen_registry_conflicts:record_departure( + erlang:system_time(millisecond), Data#data.conflicts ) } end. @@ -2223,34 +2212,32 @@ demonitor_name(LogicalName, NTR) -> maps:remove(LogicalName, NTR) end. -%% The one place a registered-process monitor is created; tagged `{down, Name}`. +%% The only place we monitor a registered process. The monitor is tagged +%% `{down, Name}`. monitor_name(Name, Pid) -> erlang:monitor(process, Pid, [{tag, {down, Name}}]). -%% Extra member monitors on a leadership transition: none for member_down; all peers -%% for the joiner itself; just the joiner for everyone else. -extra_member_ids(undefined, _AllIds, _Id) -> []; -extra_member_ids(MemberId, AllIds, MemberId) -> AllIds; -extra_member_ids(MemberId, _AllIds, _Id) -> [MemberId]. +%% Returns which members `Id` should start monitoring on a leadership change. After a +%% member_down there are none. The joiner monitors every member, and everyone else +%% monitors just the joiner. +get_extra_member_ids(undefined, _AllIds, _Id) -> []; +get_extra_member_ids(MemberId, AllIds, MemberId) -> AllIds; +get_extra_member_ids(MemberId, _AllIds, _Id) -> [MemberId]. -%% The elector is authoritative and always sends current tokens, so overwrite. -merge_peer_tokens(Tokens, State = #state{peer_tokens = PeerTokens}) -> - State#state{peer_tokens = maps:merge(PeerTokens, Tokens)}. +%% The elector always sends the current tokens, so its values overwrite ours. +merge_peer_tokens(Tokens, Data = #data{peer_tokens = PeerTokens}) -> + Data#data{peer_tokens = maps:merge(PeerTokens, Tokens)}. broadcast_to_peers(Members, Msg) -> maps:foreach(fun(MemberId, _) -> cast_to_member(MemberId, Msg) end, Members). -%% Every inter-member message goes through here. A cast to a disconnected node would -%% auto-reconnect distribution; during a partition that heals it before either side -%% observes the departure. So drop the cast unless the node is already connected. -%% Under `-ifdef(DST)` `eta_transform` rewrites the cast to `eta_net:cast/2`. -cast_to_member(To, Msg) -> - do_cast_to_member(To, Msg). - -do_cast_to_member({Node, Name}, Msg) when Node =:= node() -> - gen_statem:cast({Name, Node}, Msg); -do_cast_to_member({Node, Name}, Msg) -> - case lists:member(Node, nodes()) of +%% Every message between members goes through here. A cast to a disconnected node +%% would make distribution reconnect on its own. During a partition, that would heal it +%% before either side noticed the other had left. So we drop the cast unless the node +%% is already connected. Under `-ifdef(DST)`, `eta_transform` rewrites the cast to +%% `eta_net:cast/2`. +cast_to_member({Node, Name} = To, Msg) -> + case is_member_reachable(To) of true -> gen_statem:cast({Name, Node}, Msg); false -> ok end. diff --git a/src/dgen_registry/dgen_registry_member.hrl b/src/dgen_registry/dgen_registry_member.hrl new file mode 100644 index 0000000..0aeecfb --- /dev/null +++ b/src/dgen_registry/dgen_registry_member.hrl @@ -0,0 +1,142 @@ +%% Records shared by the dgen_registry modules, so the member can pattern match on +%% fields of its helpers' state. Each record still belongs to one module, which is the +%% only one that writes its fields: +%% `#data{}` dgen_registry_member +%% `#pipeline{}` dgen_registry_pipeline +%% `#conflicts{}` dgen_registry_conflicts +%% `#intent{}`, `#intents{}` dgen_registry_intents +%% `#replica{}` dgen_registry_replica +%% +%% *This documentation is LLM-generated. See the AI disclosure in README.md.* + +-ifndef(DGEN_REGISTRY_MEMBER_HRL). +-define(DGEN_REGISTRY_MEMBER_HRL, true). + +-record(data, { + member_id :: dgen_registry_elector:member_id(), + %% Looked up once after init. The `one_for_all` supervisor restarts us together + %% with the elector, so this pid never goes stale. + elector :: pid() | undefined, + %% The backend handle the leader uses to commit the version key. See + %% dgen_registry_names. + tenant :: dgen_backend:tenant(), + tuid :: dgen_server:tuid(), + leader :: dgen_registry_elector:member_id() | undefined, + %% Whether we have handled the DOWN for `leader`. Erlang delivers a process's + %% messages before its DOWN, so once this is true we have every broadcast that + %% leader sent us. Every write of `leader` resets it. + leader_down = false :: boolean(), + %% The local replica: ETS rows, inverted index, applied version and epoch. + replica :: dgen_registry_replica:t(), + %% Leader only (§4.9). Maps `SubId` to `{Watch, Notify}`. It is loaded at assume + %% and updated by `{presence_update, …}`. + subs = #{} :: #{term() => {dgen_registry_replica:query(), dgen_registry_replica:query()}}, + %% Leader only. The names matching each watch query. Each commit diffs it to find + %% `joined` and `left` events. + sub_matches = #{} :: #{term() => #{term() => pid()}}, + %% Leader only. The pids matching each notify query. A pid that newly matches is + %% sent the current watch snapshot. + notify_matches = #{} :: #{term() => #{term() => pid()}}, + %% Monitors on peer members. Every member keeps these. + members :: #{dgen_registry_elector:member_id() => reference()}, + monitors :: #{reference() => dgen_registry_elector:member_id()}, + %% Monitors on registered processes. Only the leader keeps these. We need no + %% reverse map, because each monitor is tagged `{down, Name}` and its DOWN carries + %% the name. + name_to_ref :: #{term() => reference()}, + %% Requests waiting for a leader to decide them. See dgen_registry_intents. + intents = dgen_registry_intents:new() :: dgen_registry_intents:t(), + %% Leader only. Write ops not yet answered. See dgen_registry_pipeline. + pipeline = dgen_registry_pipeline:new() :: dgen_registry_pipeline:t(), + %% The epoch in which a `fenced` commit last made this member join again. See + %% maybe_rejoin/1. It lets us join again at most once per epoch. + fence_rejoined = undefined :: undefined | non_neg_integer(), + %% The token for our join. We replace it on nodeup, so the elector discards a + %% stale `{member_down, Self, OldToken}`. + join_token :: reference(), + %% The join tokens of our peers. We send them back in + %% `{member_down, PeerId, Token}`, so the elector can discard a stale DOWN. + peer_tokens :: #{dgen_registry_elector:member_id() => reference()}, + %% The timer for an unanswered resync request. Any snapshot clears it. + resync_timer = undefined :: undefined | reference(), + %% Whether we have ever synced, by applying a snapshot or assuming leadership. + %% Joins send `fresh = not synced`. The first sync joins again with + %% `fresh = false`. + synced = false :: boolean(), + %% Tags the handoff gather running off the loop (§5.7). We only apply + %% `{assume_gathered, …}` if its ref still matches, so a replaced gather is + %% harmless. + assume_ref = undefined :: undefined | reference(), + %% Conflict repair state (§5.6). See dgen_registry_conflicts. + conflicts = dgen_registry_conflicts:new() :: dgen_registry_conflicts:t(), + %% Per-registry tuning options, resolved through dgen_config. + config = #{} :: dgen_config:config() +}). + +-record(pipeline, { + pending = queue:new() :: queue:queue(dgen_registry_pipeline:op()), + num_pending = 0 :: non_neg_integer(), + %% We monitor the worker, so if it dies without a result the pipeline doesn't get + %% stuck. + committing = undefined :: undefined | #{ref := reference(), mref := reference(), plan := map()}, + %% Used as the next read version to skip a GRV. If it is stale, the worker + %% retries. + last_version = undefined :: undefined | integer(), + %% Maps `BatchRef` to `{Seq, Direct, TimerRef, Needed, Acked}`. We answer batches + %% in `Seq` order, never in reference order, so simulated runs are reproducible. + acks = #{} :: #{ + reference() => + { + non_neg_integer(), + [dgen_registry_pipeline:direct()], + reference(), + non_neg_integer(), + #{ + dgen_registry_elector:member_id() => true + } + } + }, + next_ack_seq = 0 :: non_neg_integer() +}). + +-record(conflicts, { + trail = #{} :: dgen_registry_conflicts:trail(), + %% Leader only. Maps each name to its recent kill times in ms, for + %% `conflict_kill_budget`. + kill_budget = #{} :: #{term() => [integer()]}, + %% When we last saw a member leave. Trail pruning pauses for a TTL after it. + last_departure = 0 :: integer(), + %% Leader only. True when the last rebuild may have missed a committed binding. + degraded = false :: boolean() +}). + +-record(intent, { + kind :: dgen_registry_intents:kind(), + phase :: dgen_registry_intents:phase(), + %% `answered` means the caller already has its reply, but the change still has to + %% be made. + from :: gen_server:from() | answered, + %% The wall-clock time of the request in ms. We compare it with the caller + %% timeout. + at :: integer(), + %% The arrival order. We emit effects in this order, never in map order, so the + %% message schedule is reproducible under deterministic simulation. + seq :: non_neg_integer() +}). + +-record(intents, { + all = #{} :: #{reference() => #intent{}}, + %% Maps `Ref` to `{V, E}` for the intents in `awaiting_version`. When the replica + %% advances, we only scan these. + deferred = #{} :: #{reference() => {non_neg_integer(), non_neg_integer()}}, + next_seq = 0 :: non_neg_integer() +}). + +-record(replica, { + tab :: ets:tid() | atom(), + index = #{} :: #{term() => #{term() => #{term() => []}}}, + version = 0 :: non_neg_integer(), + epoch = 0 :: non_neg_integer() +}). + +-endif. diff --git a/src/dgen_registry/dgen_registry_names.erl b/src/dgen_registry/dgen_registry_names.erl index 4eb9249..2f9595c 100644 --- a/src/dgen_registry/dgen_registry_names.erl +++ b/src/dgen_registry/dgen_registry_names.erl @@ -6,23 +6,26 @@ -moduledoc false. -endif. -%% Durable fence for the leader-driven commit pipeline (§4.4 of -%% docs/design/dgen_registry_design.md). Name state lives only in members' memory; the -%% DB holds one version key per registry, versionstamp-bumped once per batch. The bump -%% exists solely to make the commit a write transaction, so the resolver actually checks -%% the leader-key read-conflict. +%% Provides the durable fence for the commit pipeline that the leader drives (§4.4 of +%% docs/design/dgen_registry_design.md). Name state lives only in the members' memory. +%% The database holds one version key per registry, and each batch bumps it with a +%% versionstamp. The bump exists only to make the commit a write transaction. Without a +%% write, the resolver would never check the read conflict on the leader key. %% %% ``` %% {Tuid, <<"version">>} -> 10-byte versionstamp (big-endian commit version) %% {Tuid, <<"leader">>} -> term_to_binary({MemberId, Epoch}) (elector-owned fence) %% ``` %% -%% Uniqueness therefore survives any single member loss but not loss of both holders of a -%% binding; that residual is covered by kill-both-on-conflict (§5.6), not the DB. +%% So uniqueness survives the loss of any single member, but not the loss of both +%% members holding a binding. That remaining case is handled by killing both pids on +%% conflict (§5.6), not by the database. %% -%% Fencing on `{Self, Epoch}` (epoch prevents ABA). Pinned read version (leader-era): -%% read-conflict on the leader key without reading it. Fresh GRV or retry: read it, -%% since the GRV may post-date a leadership change a conflict range would miss. +%% We fence on `{Self, Epoch}`. The epoch makes sure a re-elected member doesn't match +%% its own earlier term. When the read version is pinned to one taken during this +%% leader's term, we add a read conflict on the leader key without reading it. With a +%% fresh read version (GRV), or on a retry, we must read the key. The fresh version may +%% come after a leadership change, and a conflict range would not catch that. %% %% *This documentation is LLM-generated. See the AI disclosure in README.md.* @@ -33,19 +36,19 @@ -if(?DOCATTRS). -doc """ -Reads the registry's durable version key — the versionstamp of the most recently -committed batch, decoded as a big-endian integer. +Reads the registry's durable version key. It holds the versionstamp of the most +recently committed batch, decoded as a big-endian integer. -Returns a non-negative integer (`0` if no batch has committed yet). It is a -globally-monotonic FDB commit version, **not** a count of batches. Used for -diagnostics and tests; the registry itself does not read this key — the commit -version returned by the transaction is what stamps the in-memory maps. +Returns a non-negative integer, or `0` if no batch has committed yet. The value is a +globally monotonic FDB commit version, **not** a count of batches. It is used for +diagnostics and tests. The registry itself does not read this key. It stamps the +in-memory maps with the commit version that the transaction returns. """. -endif. -spec read_version(dgen_backend:tenant(), dgen_server:tuid()) -> non_neg_integer(). read_version(Tenant = {_Handle, Dir}, Tuid) -> B = dgen_config:backend(), - Key = version_db_key(Dir, Tuid), + Key = get_version_db_key(Dir, Tuid), dgen_backend:transactional(Tenant, fun({Tx, _Dir}) -> case B:wait(B:get(Tx, Key)) of not_found -> 0; @@ -55,21 +58,22 @@ read_version(Tenant = {_Handle, Dir}, Tuid) -> -if(?DOCATTRS). -doc """ -Reads the durable version key as the **committed frontier on the same scale as a +Reads the durable version key as the **committed frontier, on the same scale as a member's `applied_version`** (`get_committed_version`). -The version key holds a 10-byte FDB versionstamp — an 8-byte commit version -followed by a 2-byte in-transaction batch order — so `read_version/2` returns the -commit version left-shifted 16 bits plus the batch. A member stamps its maps with -the bare 8-byte commit version, so the two are not directly comparable; this -shifts the batch bytes off to recover the commit version. Each registry commit is -its own transaction (a single versionstamp write), so the batch order is always 0, -but the shift is correct regardless. - -Used by the leadership-handoff fence (`dgen_registry_member:gather_caught_up/6`): -a new leader whose gathered `MaxVersion` is behind this frontier has an incomplete -reconstruction and re-gathers before assuming — the fix for the handoff-gather -race (formal/DgenRegistryReplication.tla). Returns `0` if no batch has committed. +The version key holds a 10-byte FDB versionstamp. That is an 8-byte commit version +followed by a 2-byte batch order within the transaction. So `read_version/2` returns +the commit version shifted left 16 bits, plus the batch order. A member stamps its +maps with the bare 8-byte commit version, so the two can't be compared directly. This +function shifts off the batch bytes to recover the commit version. Each registry +commit is its own transaction with a single versionstamp write, so the batch order is +always 0, but the shift is correct either way. + +The leadership-handoff fence uses this in `dgen_registry_member:gather_until_caught_up/7`. +If a new leader's gathered `MaxVersion` is behind this frontier, its reconstruction is +incomplete, and it gathers again before assuming leadership. That is the fix for the +handoff-gather race found in formal/DgenRegistryReplication.tla. Returns `0` if no +batch has committed. """. -endif. -spec read_committed_frontier(dgen_backend:tenant(), dgen_server:tuid()) -> non_neg_integer(). @@ -80,26 +84,32 @@ read_committed_frontier(Tenant, Tuid) -> -doc """ Starts an asynchronous, fenced batch commit in a `dgen_transaction` worker. -The commit's only write is a conflict-free atomic bump of the registry's version -key (see the module doc); its purpose is to fence the leader (a stale leader's -commit aborts) and to assign the batch a globally-monotonic commit version that the -caller uses to stamp its in-memory maps. It runs in its own process so the caller -(the registry leader) is not blocked while it is in flight, and the read version may -be pinned (`read_version`) to skip a GRV. The worker delivers -`{dgen_transaction, Ref, Reply}` to `owner` where `Reply` is `{committed, Version}` -on success, `{aborted, fenced}` if a newer leader has been elected, or -`{error, Reason}` on a non-retryable backend failure. - -The worker is **monitored** (returns `{ok, {Pid, MRef}}`): the caller must handle -its `DOWN` as a commit failure, so a worker that dies without delivering a result -(e.g. it is killed) still resolves the in-flight commit rather than wedging it. - -`Self` is the committing member's id and `epoch` its current leadership epoch — -together the fence subject `{Self, Epoch}`. `Opts` is a map: `owner` (pid, -required), `ref` (correlation token, required), `epoch` (the leader's epoch, -required), `read_version` (a prior committed version to pin, or -omitted/`undefined` for a fresh GRV). On a retryable conflict the worker resets -to a fresh GRV automatically. +The commit's only write is a conflict-free atomic bump of the registry's version key +(see the module doc). It has two purposes. It fences the leader, so a stale leader's +commit aborts. And it gives the batch a globally monotonic commit version, which the +caller uses to stamp its in-memory maps. The commit runs in its own process so the +registry leader isn't blocked while it is in flight. The caller may pin the read +version with `read_version` to skip fetching a fresh one (a GRV). The worker sends +`{dgen_transaction, Ref, Reply}` to `owner`. `Reply` is `{committed, Version}` on +success, `{aborted, fenced}` if a newer leader has been elected, or `{error, Reason}` +on a backend failure that can't be retried. + +The worker is **monitored**, and the function returns `{ok, {Pid, MRef}}`. The caller +must treat the worker's `DOWN` as a commit failure. That way, a worker that dies +without delivering a result, for example because it was killed, still resolves the +in-flight commit instead of leaving it stuck. + +`Self` is the committing member's id and `epoch` is its current leadership epoch. +Together they form `{Self, Epoch}`, the value the fence checks. `Opts` is a map with +these keys: + +- `owner`, required. The pid that receives the result. +- `ref`, required. A correlation token. +- `epoch`, required. The leader's epoch. +- `read_version`, optional. A previously committed version to pin. Omit it or pass + `undefined` to get a fresh GRV. + +On a retryable conflict the worker switches to a fresh GRV automatically. """. -endif. -spec start_commit( @@ -114,43 +124,43 @@ to a fresh GRV automatically. } ) -> {ok, {pid(), reference()}} | {error, term()}. start_commit({Db, Dir}, Tuid, Self, Opts) -> - LeaderKey = dgen_registry_elector:leader_db_key(Dir, Tuid), - VersionKey = version_db_key(Dir, Tuid), + LeaderKey = dgen_registry_elector:get_leader_db_key(Dir, Tuid), + VersionKey = get_version_db_key(Dir, Tuid), ReadVersion = maps:get(read_version, Opts, undefined), Args = #{ self => Self, epoch => maps:get(epoch, Opts), leader_key => LeaderKey, version_key => VersionKey, - fence => fence_mode(ReadVersion) + fence => get_fence_mode(ReadVersion) }, TxOpts = [ {db, Db}, {owner, maps:get(owner, Opts)}, {ref, maps:get(ref, Opts)} - | read_version_opt(ReadVersion) + | get_read_version_opt(ReadVersion) ], dgen_transaction:start_monitor(?MODULE, Args, TxOpts). -read_version_opt(undefined) -> []; -read_version_opt(Version) -> [{read_version, Version}]. +get_read_version_opt(undefined) -> []; +get_read_version_opt(Version) -> [{read_version, Version}]. -fence_mode(undefined) -> read; -fence_mode(_Version) -> conflict. +get_fence_mode(undefined) -> read; +get_fence_mode(_Version) -> conflict. -if(?DOCATTRS). -doc """ -Verifies, against the durable leader key, that `Self` is the fenced leader for -`Epoch` right now. - -Used by the consistent-read path (`whereis_name_consistent/1` and friends): a -member that believes it is leader confirms that belief against the database -before answering, so a deposed leader that has not yet heard about the handoff -(e.g. it is on the minority side of a partition) cannot serve a stale answer as -authoritative (Guarantee 5 of the design doc). Any failure to verify — a -mismatch, a missing key, or an unreachable backend — returns `false` (the CP -refusal). +Checks the durable leader key to verify that `Self` is the leader for `Epoch` right +now. + +The consistent-read path uses this, in `whereis_name_consistent/1` and related +functions. A member that believes it is the leader confirms that against the database +before answering. Otherwise a deposed leader that hasn't heard about the handoff yet, +for example one on the minority side of a partition, could serve a stale answer as +authoritative (Guarantee 5 of the design doc). Any failure to verify returns `false`, +whether it is a mismatch, a missing key, or an unreachable backend. The registry +prefers refusing to answer over answering wrongly. """. -endif. -spec verify_leader( @@ -161,27 +171,23 @@ refusal). ) -> boolean(). verify_leader(Tenant = {_Db, Dir}, Tuid, Self, Epoch) -> B = dgen_config:backend(), - Key = dgen_registry_elector:leader_db_key(Dir, Tuid), + Key = dgen_registry_elector:get_leader_db_key(Dir, Tuid), try dgen_backend:transactional(Tenant, fun({Tx, _Dir}) -> case B:wait(B:get(Tx, Key)) of not_found -> false; - Bin -> leader_matches(binary_to_term(Bin), Self, Epoch) + Bin -> is_fenced_leader(binary_to_term(Bin), Self, Epoch) end end) catch _:_ -> false end. -%% Rolling upgrade: a bare `{Node, Name}` (pre-epoch) is accepted on id alone. Both -%% formats are 2-tuples; the second element's type (integer vs atom) tells them apart. -leader_matches({{_, _} = Leader, LeaderEpoch}, Self, Epoch) when is_integer(LeaderEpoch) -> +is_fenced_leader({{_, _} = Leader, LeaderEpoch}, Self, Epoch) when is_integer(LeaderEpoch) -> Leader =:= Self andalso LeaderEpoch =:= Epoch; -leader_matches({undefined, LeaderEpoch}, _Self, _Epoch) when is_integer(LeaderEpoch) -> +is_fenced_leader({undefined, LeaderEpoch}, _Self, _Epoch) when is_integer(LeaderEpoch) -> false; -leader_matches({_, _} = Leader, Self, _Epoch) -> - Leader =:= Self; -leader_matches(_Other, _Self, _Epoch) -> +is_fenced_leader(_Other, _Self, _Epoch) -> false. %% --------------------------------------------------------------------------- @@ -191,26 +197,27 @@ leader_matches(_Other, _Self, _Epoch) -> init(Args) -> {ok, Args}. -%% Pinned (leader-era) read version: any later leader-key write conflicts us, so no -%% storage read is needed (single round-trip). +%% The read version is pinned to one from this leader's term. Any later write to the +%% leader key conflicts with us, so we don't need to read it, which saves a round trip. handle_begin(Tx, State = #{fence := conflict, leader_key := LeaderKey}) -> B = dgen_config:backend(), B:add_read_conflict_key(Tx, LeaderKey), bump_version(B, Tx, State), {commit, State}; handle_begin(Tx, State = #{fence := read}) -> - fenced_by_read(Tx, State). + fence_by_read(Tx, State). -%% Retry runs on a fresh GRV that may post-date a leadership change: must read. +%% A retry runs on a fresh read version that may come after a leadership change, so we +%% must read the leader key. handle_retry(Tx, State) -> - fenced_by_read(Tx, State). + fence_by_read(Tx, State). handle_committed(Version, _State) -> {ok, Version}. -fenced_by_read(Tx, State = #{self := Self, epoch := Epoch, leader_key := LeaderKey}) -> +fence_by_read(Tx, State = #{self := Self, epoch := Epoch, leader_key := LeaderKey}) -> B = dgen_config:backend(), - case leader_matches(current_leader(B, Tx, LeaderKey), Self, Epoch) of + case is_fenced_leader(get_current_leader(B, Tx, LeaderKey), Self, Epoch) of true -> bump_version(B, Tx, State), {commit, State}; @@ -222,16 +229,17 @@ fenced_by_read(Tx, State = #{self := Self, epoch := Epoch, leader_key := LeaderK %% Internal %% --------------------------------------------------------------------------- -%% Conflict-free on the value, so successive leader commits never conflict on it. +%% Writes a versionstamped value, which never causes a conflict. So successive commits +%% from the leader never conflict on this key. bump_version(B, Tx, #{version_key := VersionKey}) -> B:set_versionstamped_value(Tx, VersionKey, <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>). -current_leader(B, Tx, LeaderKey) -> +get_current_leader(B, Tx, LeaderKey) -> case B:wait(B:get(Tx, LeaderKey)) of not_found -> undefined; Bin -> binary_to_term(Bin) end. -version_db_key(Dir, Tuid) -> +get_version_db_key(Dir, Tuid) -> B = dgen_config:backend(), B:dir_pack(Dir, dgen_key:extend(Tuid, <<"version">>)). diff --git a/src/dgen_registry/dgen_registry_pipeline.erl b/src/dgen_registry/dgen_registry_pipeline.erl index ac00719..09010c8 100644 --- a/src/dgen_registry/dgen_registry_pipeline.erl +++ b/src/dgen_registry/dgen_registry_pipeline.erl @@ -2,49 +2,69 @@ -define(DOCATTRS, ?OTP_RELEASE >= 27). +-include("dgen_registry_member.hrl"). + -if(?DOCATTRS). -moduledoc false. -endif. -%% The leader's write pipeline: every op it has accepted and not yet answered. +%% Tracks the leader's writes, meaning every op it has accepted and not yet answered. %% `§N.N` refers to docs/design/dgen_registry_design.md. %% -%% Phases, one field each: -%% queued `pending` waiting for the next group commit. -%% committing `committing` the single in-flight commit; arrivals ride the next one. -%% awaiting `acks` a committed direct registration waiting for a follower -%% holders copy before its `yes` (§5.5); the follower-side twin is -%% an intent's `awaiting_version` (dgen_registry_intents). -%% The member plans and applies the ops (plan_batch/4, apply_committed_plan/3). +%% An op moves through these phases. Each phase has its own field: +%% +%% queued `pending` Waiting for the next group commit. +%% committing `committing` The one commit in flight. New ops wait for the next one. +%% awaiting `acks` A committed direct registration. Before replying `yes`, +%% holders we wait until a follower holds a copy (§5.5). On the +%% follower side, the matching phase is `awaiting_version` +%% in dgen_registry_intents. +%% +%% The member plans and applies the ops itself, in plan_batch/4 and +%% apply_committed_plan/3. +%% +%% surrender/2 decides what happens to ops the leader can't finish. With `failed`, the +%% commit didn't land, or it landed under an older epoch, so we retry the ops. +%% With `deposed`, we hand each op off according to its kind. The rules match the ones +%% the follower uses for its intents: +%% +%% add, local Becomes a follower intent and is re-sent to the next leader. +%% add, forwarded No reply. The follower that forwarded it re-sends its own intent. +%% set_meta Replies `{error, no_leader}`, on `failed` too. It replaces the +%% value, so re-sending it is unsafe. +%% remove/retract A waiting caller gets `ok`, on `failed` too. The removal then +%% continues as an already-answered unregister intent that only +%% removes the name if the same pid still holds it. +%% down Dropped, because the next leader monitors the pid itself. %% -%% One surrender rule, surrender/2, for ops the leader cannot finish. `failed` (the -%% commit did not land, or landed under a moved view) retries them; `deposed` hands -%% them off by kind, by the same idempotence rule as the follower's intents: -%% add, local becomes a follower intent, re-driven to the next leader. -%% add, forwarded no reply: the forwarding follower re-drives its own intent. -%% set_meta `{error, no_leader}`, also on failed: a replace is unsafe to re-send. -%% remove/retract a waiting caller gets `ok`, also on failed; then an answered, -%% pid-guarded unregister intent. -%% down dropped: the next leader monitors the pid itself. -%% So `no` stays an adjudicated refusal (§3), never "the leader could not finish". +%% So `no` always means the name is taken (§3). It never means the leader couldn't +%% finish. %% -%% Pure: surrender/2 returns effects for the member to carry out. +%% surrender/2 has no side effects. It returns effects for the member to carry out. %% %% *This documentation is LLM-generated. See the AI disclosure in README.md.* --export([new/0, enqueue/2, num_pending/1, in_flight/1, take_batch/2, take_pending/1]). --export([start/4, finish/2, worker_down/2, committed/2, read_version/1, new_term/1]). --export([await_acks/5, ack/3, ack_timeout/2, take_acks/1]). +-export([new/0, enqueue/2, get_num_pending/1, is_in_flight/1, take_batch/2, take_pending/1]). +-export([ + start_commit/4, + finish_commit/2, + handle_worker_down/2, + set_last_version/2, + get_read_version/1, + start_new_term/1 +]). +-export([await_acks/5, record_ack/3, handle_ack_timeout/2, take_acks/1]). -export([surrender/2]). --export_type([t/0, op/0, origin/0, effect/0]). +-export_type([t/0, op/0, origin/0, direct/0, effect/0]). -type member_id() :: dgen_registry_elector:member_id(). -type from() :: gen_server:from(). -%% Where an op came from, so how to answer it: plain variants are direct calls answered -%% via `From`; `forward*` ones get a `{register_reply | set_meta_reply | -%% unregister_reply, Ref, ...}` cast to the forwarding member. +%% Where an op came from, which tells us how to answer it. The plain variants are +%% direct calls, answered through `From`. For the `forward*` variants we cast a +%% `{register_reply | set_meta_reply | unregister_reply, Ref, ...}` to the forwarding +%% member. -type origin() :: {local, from()} | {forward, member_id(), reference()} @@ -53,10 +73,11 @@ | {unreg, from()} | {forward_unreg, member_id(), reference()}. -%% A leader write op. A batch rides one fenced commit and applies in arrival order, so -%% it equals one-at-a-time processing. `remove`'s ReleasedPid is captured before the -%% optimistic local delete; when set it drives the durable clear and feeds the -%% conflict trail (§5.6). `down` is Ref-guarded against a stale DOWN. +%% A write op on the leader. A batch goes through one fenced commit and is applied in +%% arrival order, so the result is the same as processing ops one at a time. For +%% `remove`, we capture `ReleasedPid` before the optimistic local delete. When it is +%% set, it drives the durable clear and is added to the conflict trail (§5.6). `down` +%% carries the monitor reference, so we can ignore a stale DOWN. -type op() :: {add, Name :: term(), pid(), Meta :: {map(), term()}, origin()} | {set_meta, Name :: term(), Index :: map(), Data :: term(), origin()} @@ -64,32 +85,16 @@ | {retract, Name :: term(), pid()} | {down, Name :: term(), reference()}. -%% Awaiting holders: origin plus binding, so a fail-closed timeout can retract it. +%% A registration awaiting holders. We keep the binding along with the origin, so that +%% under strict replication a timed-out wait can retract the binding and refuse the +%% call. -type direct() :: {origin(), Name :: term(), pid()}. --record(pipeline, { - pending = queue:new() :: queue:queue(op()), - num_pending = 0 :: non_neg_integer(), - %% Monitored so a worker dying without a result cannot wedge the pipeline. - committing = undefined :: undefined | #{ref := reference(), mref := reference(), plan := map()}, - %% Pinned as the next read version to skip a GRV; if stale, the worker retries. - last_version = undefined :: undefined | integer(), - %% BatchRef => {Seq, Direct, TimerRef, Needed, Acked}. Answered in Seq order, never - %% reference order, so simulated schedules are reproducible. - acks = #{} :: #{ - reference() => - {non_neg_integer(), [direct()], reference(), non_neg_integer(), #{ - member_id() => true - }} - }, - next_ack_seq = 0 :: non_neg_integer() -}). - --opaque t() :: #pipeline{}. +-type t() :: #pipeline{}. -type effect() :: {reply, origin(), term()} - %% Hand the kind to the follower-side intents (dgen_registry_intents:submit/5). + %% Hand the request to the follower-side intents (dgen_registry_intents:submit/5). | {intent, dgen_registry_intents:kind(), from() | answered} | {retry, [op()]}. @@ -104,14 +109,15 @@ new() -> #pipeline{}. enqueue(Op, P = #pipeline{pending = Q, num_pending = N}) -> P#pipeline{pending = queue:in(Op, Q), num_pending = N + 1}. --spec num_pending(t()) -> non_neg_integer(). -num_pending(#pipeline{num_pending = N}) -> N. +-spec get_num_pending(t()) -> non_neg_integer(). +get_num_pending(#pipeline{num_pending = N}) -> N. --spec in_flight(t()) -> boolean(). -in_flight(#pipeline{committing = C}) -> C =/= undefined. +-spec is_in_flight(t()) -> boolean(). +is_in_flight(#pipeline{committing = C}) -> C =/= undefined. -%% Take up to `Max` oldest ops. The bound keeps a burst (a DOWN flood) from becoming -%% one huge commit whose plan/apply/broadcast would freeze the loop. +%% Takes up to `Max` of the oldest ops. The limit keeps a burst, such as a flood of +%% DOWNs, from turning into one huge commit whose planning, applying and broadcasting +%% would freeze the loop. -spec take_batch(pos_integer(), t()) -> {[op()], t()}. take_batch(Max, P = #pipeline{pending = Q, num_pending = N}) -> Size = min(N, Max), @@ -132,32 +138,34 @@ take_n(Q, N, Acc) -> %% Committing %% --------------------------------------------------------------------------- --spec start(reference(), reference(), map(), t()) -> t(). -start(Ref, MRef, Plan, P = #pipeline{committing = undefined}) -> +-spec start_commit(reference(), reference(), map(), t()) -> t(). +start_commit(Ref, MRef, Plan, P = #pipeline{committing = undefined}) -> P#pipeline{committing = #{ref => Ref, mref => MRef, plan => Plan}}. --spec finish(reference(), t()) -> {ok, map(), reference(), t()} | stale. -finish(Ref, P = #pipeline{committing = #{ref := Ref, mref := MRef, plan := Plan}}) -> +-spec finish_commit(reference(), t()) -> {ok, map(), reference(), t()} | stale. +finish_commit(Ref, P = #pipeline{committing = #{ref := Ref, mref := MRef, plan := Plan}}) -> {ok, Plan, MRef, P#pipeline{committing = undefined}}; -finish(_Ref, _P) -> +finish_commit(_Ref, _P) -> stale. -%% A monitored process died: if it is the commit worker, release the slot. --spec worker_down(reference(), t()) -> {ok, map(), t()} | stale. -worker_down(MRef, P = #pipeline{committing = #{mref := MRef, plan := Plan}}) -> +%% Handles the death of a monitored process. If it was the commit worker, we free the +%% commit slot. +-spec handle_worker_down(reference(), t()) -> {ok, map(), t()} | stale. +handle_worker_down(MRef, P = #pipeline{committing = #{mref := MRef, plan := Plan}}) -> {ok, Plan, P#pipeline{committing = undefined}}; -worker_down(_MRef, _P) -> +handle_worker_down(_MRef, _P) -> stale. --spec committed(integer(), t()) -> t(). -committed(Version, P) -> P#pipeline{last_version = Version}. +-spec set_last_version(integer(), t()) -> t(). +set_last_version(Version, P) -> P#pipeline{last_version = Version}. --spec read_version(t()) -> undefined | integer(). -read_version(#pipeline{last_version = V}) -> V. +-spec get_read_version(t()) -> undefined | integer(). +get_read_version(#pipeline{last_version = V}) -> V. -%% The cached version belongs to the ended term: never pin a later term's read to it. --spec new_term(t()) -> t(). -new_term(P) -> P#pipeline{last_version = undefined}. +%% Clears the cached version, because it belongs to the term that just ended. A later +%% term must never pin its read version to it. +-spec start_new_term(t()) -> t(). +start_new_term(P) -> P#pipeline{last_version = undefined}. %% --------------------------------------------------------------------------- %% Awaiting holders @@ -169,10 +177,11 @@ await_acks(BatchRef, Direct, TimerRef, Needed, P = #pipeline{acks = A, next_ack_ acks = A#{BatchRef => {Seq, Direct, TimerRef, Needed, #{}}}, next_ack_seq = Seq + 1 }. -%% Distinct followers count once; at `Needed` the batch's direct registrations are done. --spec ack(reference(), member_id(), t()) -> +%% Records an ack from `Follower`. Each follower counts once. When `Needed` followers +%% have acked, the batch's direct registrations are done. +-spec record_ack(reference(), member_id(), t()) -> {done, [direct()], reference(), t()} | {waiting, t()}. -ack(BatchRef, Follower, P = #pipeline{acks = A}) -> +record_ack(BatchRef, Follower, P = #pipeline{acks = A}) -> case A of #{BatchRef := {Seq, Direct, TimerRef, Needed, Acked}} -> Acked1 = Acked#{Follower => true}, @@ -187,9 +196,9 @@ ack(BatchRef, Follower, P = #pipeline{acks = A}) -> {waiting, P} end. --spec ack_timeout(reference(), t()) -> +-spec handle_ack_timeout(reference(), t()) -> {expired, [direct()], non_neg_integer(), non_neg_integer(), t()} | none. -ack_timeout(BatchRef, P = #pipeline{acks = A}) -> +handle_ack_timeout(BatchRef, P = #pipeline{acks = A}) -> case maps:take(BatchRef, A) of {{_Seq, Direct, _TimerRef, Needed, Acked}, A1} -> {expired, Direct, Needed, map_size(Acked), P#pipeline{acks = A1}}; @@ -197,7 +206,7 @@ ack_timeout(BatchRef, P = #pipeline{acks = A}) -> none end. -%% Every batch still awaiting holders, in arrival order. +%% Takes every batch still awaiting holders, in arrival order. -spec take_acks(t()) -> {[{[direct()], reference(), non_neg_integer(), non_neg_integer()}], t()}. take_acks(P = #pipeline{acks = A}) -> @@ -211,7 +220,7 @@ take_acks(P = #pipeline{acks = A}) -> %% Exits %% --------------------------------------------------------------------------- -%% Effects come out in op order. +%% Returns the effects in op order. -spec surrender(failed | deposed, [op()]) -> [effect()]. surrender(failed, Ops) -> {Replies, Retry} = lists:foldl( @@ -232,26 +241,27 @@ surrender(failed, Ops) -> _ -> [{retry, lists:reverse(Retry)}] end; surrender(deposed, Ops) -> - lists:flatmap(fun deposed/1, Ops). + lists:flatmap(fun get_deposed_effects/1, Ops). -deposed({add, Name, Pid, Meta, {local, From}}) -> +get_deposed_effects({add, Name, Pid, Meta, {local, From}}) -> [{intent, {register, Name, Pid, Meta}, From}]; -deposed({add, _Name, _Pid, _Meta, _Forwarded}) -> +get_deposed_effects({add, _Name, _Pid, _Meta, _Forwarded}) -> []; -deposed({set_meta, _Name, _Index, _Data, Origin}) -> +get_deposed_effects({set_meta, _Name, _Index, _Data, Origin}) -> [{reply, Origin, {error, no_leader}}]; -deposed({remove, Name, Pid, Origin}) -> +get_deposed_effects({remove, Name, Pid, Origin}) -> Reply = case Origin of undefined -> []; _ -> [{reply, Origin, ok}] end, - Reply ++ retract(Name, Pid); -deposed({retract, Name, Pid}) -> - retract(Name, Pid); -deposed({down, _Name, _Ref}) -> + Reply ++ get_retract_effects(Name, Pid); +get_deposed_effects({retract, Name, Pid}) -> + get_retract_effects(Name, Pid); +get_deposed_effects({down, _Name, _Ref}) -> []. -%% Nothing was bound when an unregister was asked: nothing to retract. -retract(_Name, undefined) -> []; -retract(Name, Pid) -> [{intent, {unregister, Name, Pid}, answered}]. +%% The name wasn't bound when the caller asked to unregister, so there's nothing to +%% retract. +get_retract_effects(_Name, undefined) -> []; +get_retract_effects(Name, Pid) -> [{intent, {unregister, Name, Pid}, answered}]. diff --git a/src/dgen_registry/dgen_registry_replica.erl b/src/dgen_registry/dgen_registry_replica.erl index 2892174..a563901 100644 --- a/src/dgen_registry/dgen_registry_replica.erl +++ b/src/dgen_registry/dgen_registry_replica.erl @@ -2,6 +2,8 @@ -define(DOCATTRS, ?OTP_RELEASE >= 27). +-include("dgen_registry_member.hrl"). + -if(?DOCATTRS). -moduledoc false. -endif. @@ -9,93 +11,109 @@ %% A member's copy of the registry's name table, and the rules for advancing it. %% `§N.N` refers to docs/design/dgen_registry_design.md. %% -%% A replica is a position in the leader's totally ordered stream of committed batches -%% plus the content it implies: `tab` (rows `{Name, Pid, Index, Data}` in a protected -%% ETS table, read lock-free by any process), `index` (attribute -> value -> names, -%% derived from the rows), `version` (commit version applied up to) and `epoch` (the -%% leadership term followed). Commit versions are monotonic across terms, so replicas -%% compare by version alone. +%% A replica is a position in the leader's stream of committed batches, which is +%% totally ordered, together with the content that position implies. It has four +%% fields: +%% +%% `tab` The rows `{Name, Pid, Index, Data}`, in a protected ETS table that any +%% process can read without locking. +%% `index` An inverted index from attribute to value to names, derived from the +%% rows. +%% `version` The commit version we have applied up to. +%% `epoch` The leadership term we are following. +%% +%% Commit versions keep increasing across terms, so we compare replicas by version +%% alone. %% -%% Invariant: two replicas at the same version hold the same content. Freshest-wins -%% handoff reconstruction (§5.7) depends on it; every admission rule protects it. +%% Invariant: two replicas at the same version hold the same content. The handoff +%% rebuilds the table from the freshest replica (§5.7), which only works if this +%% holds. Every rule for accepting a batch or snapshot exists to protect it. +%% +%% These rules mirror formal/DgenRegistryReplication.tla action for action: %% -%% Admission mirrors formal/DgenRegistryReplication.tla action for action: %% offer_batch/5 RecvBcastApply / RecvBcastDrop / RecvBcastGap %% offer_snapshot/4 RecvSnap (with the SafeAssumeSnap guard) -%% An older-epoch batch is dropped; a newer-epoch batch is applied but does not raise -%% `epoch`, while a current-or-newer snapshot does. The asymmetry is the model's and is -%% unreachable: a new leader snapshots each follower before broadcasting to it (FIFO). %% -%% Pure: no sends, timers or replies; results tell the member what happened. +%% We drop a batch from an older epoch. We apply a batch from a newer epoch but don't +%% raise `epoch`, while a snapshot from the current or a newer epoch does raise it. The +%% model has the same asymmetry, and it can't actually happen. A new leader sends each +%% follower a snapshot before broadcasting to it, and messages arrive in order. +%% +%% Nothing here sends messages, sets timers or replies. The return values tell the +%% member what happened. %% %% *This documentation is LLM-generated. See the AI disclosure in README.md.* --export([new/1, tab/1, version/1, epoch/1]). +-export([new/1, get_tab/1, get_version/1, get_epoch/1]). -export([offer_batch/5, offer_snapshot/4]). -export([adopt/4, commit/5, insert/5, delete/2]). --export([lookup_pid/2, lookup_metadata/2, exists/2, records/1, encode/1]). +-export([lookup_pid/2, lookup_metadata/2, exists/2, get_records/1, encode/1]). -export([query/2, query_constraints/2, query_match_names/2, query_satisfies/2]). -export([encode_records/1, decode_records/1]). -export_type([t/0, records/0, bcast_op/0, query/0]). --record(replica, { - tab :: ets:tid() | atom(), - index = #{} :: #{term() => #{term() => #{term() => []}}}, - version = 0 :: non_neg_integer(), - epoch = 0 :: non_neg_integer() -}). +-type t() :: #replica{}. --opaque t() :: #replica{}. - -%% Replica content as a map, as carried by the handoff gather and snapshot fan-out. +%% Replica content as a map. The handoff gather and the snapshot fan-out carry it in +%% this shape. -type records() :: #{term() => {pid(), map(), term()}}. -%% One op of a `{names_batch, …}` broadcast, epoch already stripped to the envelope. +%% One op of a `{names_batch, …}` broadcast. The epoch isn't repeated here, because it +%% travels once in the batch envelope. -type bcast_op() :: {name_registered, term(), pid(), map(), term()} | {metadata_set, term(), map(), term()} | {name_unregistered, term(), pid() | undefined}. -%% A tagged query (§4.9). Only `{all, Constraints}` (AND-equal) exists today. +%% A tagged query (§4.9). Only `{all, Constraints}` exists today. It matches rows +%% where every attribute equals the given value. -type query() :: {all, #{term() => term()}}. %% --------------------------------------------------------------------------- %% Construction and accessors %% --------------------------------------------------------------------------- -%% Created empty on every member (re)start, with no heir: a crashed member's replica -%% is rebuilt from the leader's snapshot. +%% Creates an empty replica. This happens on every member start or restart. The table +%% has no heir, because a crashed member rebuilds its replica from the leader's +%% snapshot. -spec new(atom()) -> t(). new(TabName) -> Tab = ets:new(TabName, [named_table, set, protected, {read_concurrency, true}]), #replica{tab = Tab}. --spec tab(t()) -> ets:tid() | atom(). -tab(#replica{tab = Tab}) -> Tab. +-spec get_tab(t()) -> ets:tid() | atom(). +get_tab(#replica{tab = Tab}) -> Tab. --spec version(t()) -> non_neg_integer(). -version(#replica{version = V}) -> V. +-spec get_version(t()) -> non_neg_integer(). +get_version(#replica{version = V}) -> V. --spec epoch(t()) -> non_neg_integer(). -epoch(#replica{epoch = E}) -> E. +-spec get_epoch(t()) -> non_neg_integer(). +get_epoch(#replica{epoch = E}) -> E. %% --------------------------------------------------------------------------- %% Stream admission %% --------------------------------------------------------------------------- -%% Offer one broadcast batch. Applied only if contiguous (`PrevV` is our version), so -%% the replica stays a prefix of the leader's stream; on `gap` nothing is applied and -%% the caller requests a snapshot. A batch is one message, so a lost batch always -%% leaves a gap. The leader's heartbeat is an empty batch at `{V, V}`. +%% Offers one broadcast batch. We apply it only when `PrevV` equals our version, so the +%% replica stays a prefix of the leader's stream. On `gap` we apply nothing, and the +%% caller requests a snapshot. Each batch is one message, so a lost batch always leaves +%% a gap. The leader's heartbeat is an empty batch at `{V, V}`. +%% +%% A batch from a later epoch is also a gap. It means we missed the new leader's +%% snapshot, and only a snapshot moves us into a new epoch. Applying the batch would +%% let a held `yes` from the old epoch be answered by the new leader's versions, even +%% if the new leader took over without that commit. -spec offer_batch(non_neg_integer(), non_neg_integer(), non_neg_integer(), [bcast_op()], t()) -> {applied, t()} | stale_epoch | duplicate | gap. -%% MUTATION_PARTIAL_BATCH (test-only, never in a release): re-admits per-op messages -%% sharing a version, reintroducing undetectable partially delivered batches. +%% MUTATION_PARTIAL_BATCH is for tests only and never in a release. It accepts +%% separate per-op messages that share a version. That brings back the bug where a +%% partially delivered batch goes undetected. -ifdef(MUTATION_PARTIAL_BATCH). offer_batch(Epoch, PrevV, V, Ops, R = #replica{epoch = CurEpoch, version = Applied}) -> if Epoch < CurEpoch -> stale_epoch; + Epoch > CurEpoch -> gap; PrevV =:= Applied -> {applied, apply_ops(Ops, R#replica{version = max(Applied, V)})}; V =:= Applied -> {applied, apply_ops(Ops, R)}; V =< Applied -> duplicate; @@ -105,15 +123,17 @@ offer_batch(Epoch, PrevV, V, Ops, R = #replica{epoch = CurEpoch, version = Appli offer_batch(Epoch, PrevV, V, Ops, R = #replica{epoch = CurEpoch, version = Applied}) -> if Epoch < CurEpoch -> stale_epoch; + Epoch > CurEpoch -> gap; PrevV =:= Applied -> {applied, apply_ops(Ops, R#replica{version = max(Applied, V)})}; V =< Applied -> duplicate; true -> gap end. -endif. -%% Offer a wholesale snapshot (handoff fan-out, joiner onboarding, resync answer). The -%% version guard stops a snapshot gathered before a batch but delivered after it from -%% rolling the replica back (§5.7). Equal versions are accepted: equal content. +%% Offers a full snapshot. Snapshots come from the handoff fan-out, from onboarding a +%% joiner, and in answer to a resync. The version check stops a snapshot that was +%% gathered before a batch, but delivered after it, from rolling the replica back +%% (§5.7). We accept an equal version, because equal versions mean equal content. -spec offer_snapshot(non_neg_integer(), non_neg_integer(), records(), t()) -> {accepted, t()} | stale. offer_snapshot(Epoch, V, Records, R = #replica{epoch = CurEpoch, version = Applied}) -> @@ -126,9 +146,9 @@ offer_snapshot(Epoch, V, Records, R = #replica{epoch = CurEpoch, version = Appli %% Authoring (the leader's own replica) %% --------------------------------------------------------------------------- -%% Re-baseline unconditionally. The table is read lock-free meanwhile, so new rows are -%% written first and only then are absent names deleted: a reader never sees a bound -%% name go missing. +%% Replaces the replica content unconditionally. Other processes keep reading the +%% table while this runs, so we write the new rows first and only then delete the +%% names that are gone. That way a reader never sees a bound name go missing. -spec adopt(non_neg_integer(), non_neg_integer(), records(), t()) -> t(). adopt(Epoch, V, Records, R = #replica{tab = Tab}) -> ets:insert( @@ -145,11 +165,11 @@ adopt(Epoch, V, Records, R = #replica{tab = Tab}) -> Tab ), lists:foreach(fun(Name) -> ets:delete(Tab, Name) end, Absent), - R#replica{index = index_rebuild(Records), version = V, epoch = Epoch}. + R#replica{index = rebuild_index(Records), version = V, epoch = Epoch}. -%% Apply the leader's own committed batch at `V`. Only names in `DBOp` are touched, so -%% a concurrent optimistic delete is not clobbered, and a metadata-only op never -%% resurrects a row removed between plan and apply. +%% Applies the leader's own committed batch at `V`. We only touch names in `DBOp`, so +%% we don't undo an optimistic delete that happened in the meantime. A metadata-only +%% op never brings back a row that was removed between planning and applying. -spec commit(non_neg_integer(), map(), map(), map(), t()) -> t(). commit(V, DBOp, WNames, WMeta, R) -> R1 = maps:fold( @@ -168,19 +188,20 @@ commit(V, DBOp, WNames, WMeta, R) -> ), R1#replica{version = V}. -%% Row writers outside the stream (the optimistic local delete on unregister, the -%% legacy pre-version reply path's insert). Neither moves the version. +%% These write rows outside the stream, and neither changes the version. The member +%% uses delete/2 for its optimistic local delete on unregister. Tests use insert/5 to +%% build replicas. -spec insert(t(), term(), pid(), map(), term()) -> t(). insert(R = #replica{tab = Tab, index = Inv}, Name, Pid, Index, Data) -> - Old = row_index(Tab, Name), + Old = get_row_index(Tab, Name), ets:insert(Tab, {Name, Pid, Index, Data}), - R#replica{index = index_add(index_remove(Inv, Name, Old), Name, Index)}. + R#replica{index = add_to_index(remove_from_index(Inv, Name, Old), Name, Index)}. -spec delete(t(), term()) -> t(). delete(R = #replica{tab = Tab, index = Inv}, Name) -> - Old = row_index(Tab, Name), + Old = get_row_index(Tab, Name), ets:delete(Tab, Name), - R#replica{index = index_remove(Inv, Name, Old)}. + R#replica{index = remove_from_index(Inv, Name, Old)}. %% --------------------------------------------------------------------------- %% Reads @@ -193,7 +214,7 @@ lookup_pid(#replica{tab = Tab}, Name) -> [] -> undefined end. -%% The public `get_metadata` shape, `{ok, #{pid, index, data}}` or `undefined`. +%% Returns the public `get_metadata` shape, `{ok, #{pid, index, data}}` or `undefined`. -spec lookup_metadata(t(), term()) -> {ok, map()} | undefined. lookup_metadata(#replica{tab = Tab}, Name) -> case ets:lookup(Tab, Name) of @@ -205,9 +226,9 @@ lookup_metadata(#replica{tab = Tab}, Name) -> exists(#replica{tab = Tab}, Name) -> ets:member(Tab, Name). -%% Materialise the content as a records map. --spec records(t()) -> records(). -records(#replica{tab = Tab}) -> +%% Returns the content as a records map. +-spec get_records(t()) -> records(). +get_records(#replica{tab = Tab}) -> ets:foldl( fun({Name, Pid, Index, Data}, Acc) -> Acc#{Name => {Pid, Index, Data}} end, #{}, Tab ). @@ -216,11 +237,12 @@ records(#replica{tab = Tab}) -> %% Snapshot codec %% --------------------------------------------------------------------------- -%% Snapshots ship as one compressed binary: off-heap, never GC-scanned, and shared by -%% every follower in a fan-out. `encode/1` folds ETS straight into a pair list (cheaper -%% than building a map); `encode_records/1` takes a map already in hand. Level 1 -%% because encode time sits on client-visible handoff/resync paths. `decode_records/1` -%% also accepts the bare list/map terms older senders ship. +%% Snapshots travel as one compressed binary. The binary lives off-heap, so the GC +%% never scans it, and every follower in a fan-out shares the same copy. `encode/1` +%% folds the ETS table straight into a list of pairs, which is cheaper than building a +%% map. `encode_records/1` takes a map the caller already has. We use compression +%% level 1 because encoding time adds to handoff and resync latency, which clients +%% see. `decode_records/1` reads either kind of binary. -define(SNAPSHOT_ENCODE_OPTS, [{compressed, 1}]). -spec encode(t()) -> binary(). @@ -234,27 +256,30 @@ encode(#replica{tab = Tab}) -> encode_records(Records) when is_map(Records) -> term_to_binary(Records, ?SNAPSHOT_ENCODE_OPTS). --spec decode_records(binary() | list() | map()) -> records(). -decode_records(Bin) when is_binary(Bin) -> decode_records(binary_to_term(Bin)); -decode_records(List) when is_list(List) -> maps:from_list(List); -decode_records(Map) when is_map(Map) -> Map. +-spec decode_records(binary()) -> records(). +decode_records(Bin) when is_binary(Bin) -> + case binary_to_term(Bin) of + Pairs when is_list(Pairs) -> maps:from_list(Pairs); + Records when is_map(Records) -> Records + end. %% --------------------------------------------------------------------------- %% Queries (§4.7, §4.9) against the inverted index %% --------------------------------------------------------------------------- -%% Run a tagged query, returning `[#{name, pid, index, data}]`. +%% Runs a tagged query and returns `[#{name, pid, index, data}]`. -spec query(t(), query()) -> [map()]. query(R, {all, Constraints}) -> query_constraints(R, Constraints). -%% AND-equal query: intersect posting sets smallest-first, then re-check survivors -%% against their rows so a stale posting cannot yield a false positive. +%% Returns the rows where every attribute in `Constraints` equals its value. We +%% intersect the posting sets, smallest first. Then we re-check each remaining name +%% against its row, so a stale posting can't produce a false match. -spec query_constraints(t(), #{term() => term()}) -> [map()]. query_constraints(_R, Constraints) when map_size(Constraints) =:= 0 -> []; query_constraints(#replica{tab = Tab, index = Inv}, Constraints) -> - Postings = [posting(Inv, Attr, Value) || {Attr, Value} <- maps:to_list(Constraints)], + Postings = [get_posting(Inv, Attr, Value) || {Attr, Value} <- maps:to_list(Constraints)], case lists:any(fun(P) -> map_size(P) =:= 0 end, Postings) of true -> []; @@ -262,16 +287,19 @@ query_constraints(#replica{tab = Tab, index = Inv}, Constraints) -> [Smallest | Rest] = lists:sort( fun(A, B) -> map_size(A) =< map_size(B) end, Postings ), - Names = [N || N <- maps:keys(Smallest), in_all(N, Rest)], + Names = [ + N + || N <- maps:keys(Smallest), lists:all(fun(P) -> maps:is_key(N, P) end, Rest) + ], materialize(Names, Constraints, Tab) end. -%% The membership `#{Name => Pid}` a query currently resolves to. +%% Returns the names and pids, as `#{Name => Pid}`, that a query currently matches. -spec query_match_names(t(), query()) -> #{term() => pid()}. query_match_names(R, Query) -> maps:from_list([{N, P} || #{name := N, pid := P} <- query(R, Query)]). -%% Does `Index` (a row's indexed-attribute map) satisfy the query? +%% Returns whether `Index`, the indexed attributes of a row, satisfies the query. -spec query_satisfies(query(), map()) -> boolean(). query_satisfies({all, Constraints}, Index) -> map_size(Constraints) > 0 andalso satisfies(Constraints, Index). @@ -287,27 +315,28 @@ apply_op({name_registered, Name, Pid, Index, Data}, R) -> insert(R, Name, Pid, Index, Data); apply_op({metadata_set, Name, Index, Data}, R) -> update_meta(R, Name, Index, Data); -%% The released pid matters only to the member's conflict trail (§5.6). +%% Only the member's conflict trail uses the released pid (§5.6). apply_op({name_unregistered, Name, _ReleasedPid}, R) -> delete(R, Name). -%% Keeps the pid. A no-op if the row is absent: never invent postings. +%% Updates the metadata and keeps the pid. If the row is absent we do nothing, so we +%% never add postings for a name that isn't bound. update_meta(R = #replica{tab = Tab, index = Inv}, Name, Index, Data) -> - Old = row_index(Tab, Name), + Old = get_row_index(Tab, Name), case ets:update_element(Tab, Name, [{3, Index}, {4, Data}]) of - true -> R#replica{index = index_add(index_remove(Inv, Name, Old), Name, Index)}; + true -> R#replica{index = add_to_index(remove_from_index(Inv, Name, Old), Name, Index)}; false -> R end. -%% `#{}` if no row, so a writer can retract old postings first. -row_index(Tab, Name) -> +%% Returns the row's index, or `#{}` if there is no row. Writers use it to remove the +%% old postings first. +get_row_index(Tab, Name) -> case ets:lookup(Tab, Name) of [{_N, _Pid, Index, _Data} | _] -> Index; [] -> #{} end. -%% Removal prunes empty value- and attribute-maps. -index_add(Inv, Name, Index) -> +add_to_index(Inv, Name, Index) -> maps:fold( fun(Attr, Value, Acc) -> Vs = maps:get(Attr, Acc, #{}), @@ -318,7 +347,9 @@ index_add(Inv, Name, Index) -> Index ). -index_remove(Inv, Name, Index) -> +%% Removes the postings for `Name`, and prunes any value or attribute map left +%% empty so the index doesn't keep entries for names that are gone. +remove_from_index(Inv, Name, Index) -> maps:fold( fun(Attr, Value, Acc) -> case Acc of @@ -341,22 +372,19 @@ index_remove(Inv, Name, Index) -> Index ). -index_rebuild(Records) -> +rebuild_index(Records) -> maps:fold( - fun(Name, {_Pid, Index, _Data}, Acc) -> index_add(Acc, Name, Index) end, + fun(Name, {_Pid, Index, _Data}, Acc) -> add_to_index(Acc, Name, Index) end, #{}, Records ). -posting(Inv, Attr, Value) -> +get_posting(Inv, Attr, Value) -> case Inv of #{Attr := #{Value := Names}} -> Names; _ -> #{} end. -in_all(Name, Postings) -> - lists:all(fun(P) -> maps:is_key(Name, P) end, Postings). - materialize(Names, Constraints, Tab) -> lists:foldl( fun(Name, Acc) -> diff --git a/src/dgen_server.erl b/src/dgen_server.erl index 8b3cbc4..393e1b4 100644 --- a/src/dgen_server.erl +++ b/src/dgen_server.erl @@ -9,15 +9,15 @@ -moduledoc """ A durable, distributed gen_server backed by a pluggable storage backend. -A dgen_server is an abstract entity composed of durable state and operations on -that state. The state lives in the configured backend (default: FoundationDB) -and the operations are defined by a callback module implementing the -`dgen_server` behaviour. This allows a gen_server to outlive any single Erlang -process, node, or cluster. +A dgen_server is an abstract entity made of durable state and the operations on +that state. The state lives in the configured backend (default: FoundationDB), +and a callback module implementing the `dgen_server` behaviour defines the +operations. This lets a gen_server outlive any single Erlang process, node, or +cluster. -Zero or more Erlang processes may act on a dgen_server at any time. Processes -with `consume` enabled consume messages from the durable queue and invoke -callbacks; processes without it only publish messages. +Zero or more Erlang processes may act on a dgen_server at any time. A process +with `consume` enabled takes messages off the durable queue and invokes the +callbacks. A process without it only publishes messages. ## Options @@ -34,81 +34,82 @@ The following options may be passed via the `Opts` proplist: caller raises `{dead_letter, N}` (for `call` messages), and the optional `handle_dead_letter/2` callback is invoked. `infinity` (the default) disables dead-lettering entirely. -- `consume_k` (default `1`) - the maximum number of queued messages a consumer - processes per transaction (the batch size). Each consume cycle peeks up to - `consume_k` messages, processes them one at a time while carrying the durable - state in memory, and commits the whole batch in a **single** transaction (the - state is read once and written once). It then immediately re-arms itself, so a - consumer that has work keeps draining batches. - - Raising `consume_k` amortises the fixed per-transaction cost (read version, - state read/write, commit) across more messages, increasing throughput when the - per-message callback work is small. The cost is larger transactions and a - coarser unit of progress (a batch is all-or-nothing; on a conflict the whole - batch retries). Because a busy consumer keeps draining, a larger `consume_k` - also tends to keep a **single** node as the active consumer, which is useful - when the callback's identity matters — for example an election-style callback - where you want to minimise churn in who processes successive messages. +- `consume_k` (default `1`) - the batch size, meaning the maximum number of + queued messages a consumer processes per transaction. Each consume cycle peeks + up to `consume_k` messages and processes them one at a time, carrying the + durable state in memory. It then commits the whole batch in a **single** + transaction, so the state is read once and written once. The consumer then + starts the next cycle right away, so a consumer that has work keeps draining + batches. + + Raising `consume_k` spreads the fixed cost of a transaction (read version, + state read and write, commit) across more messages. That raises throughput when + each callback does little work. The cost is larger transactions and a coarser + unit of progress. A batch is all-or-nothing, so a conflict retries the whole + batch. Because a busy consumer keeps draining, a larger `consume_k` also tends + to keep a **single** node as the active consumer. That helps when it matters + which node runs the callback, for example in an election-style callback where + you want the same node to keep processing successive messages. See "consume_k and inlining" below for how `consume_k > 1` changes call handling. - `lock_timeout` (default `infinity`) - maximum number of milliseconds a distributed lock may be held before other consumers treat it as stale. When a callback returns `{lock, State}`, `dgen_server` sets a timestamped lock key - in the backend that pauses all other consumers while `handle_locked/4` runs. - Under normal operation the lock holder clears the key itself before - `handle_locked` returns. If the holder is killed (SIGKILL, VM abort) the - lock persists until another consumer detects staleness: it re-checks - `lock_timeout` ms after the lock was set and clears it if that deadline has - passed. `infinity` (the default) disables stale-lock detection entirely — - a dead holder will permanently block all consumers. - - Busting is a time-based guess that the holder is dead, and nothing revokes a - holder that is merely slow, so the guess can be wrong. The lock is **fenced** - against that: each acquisition stamps the key with a unique token, and a holder - may release the lock, and commit the result of `handle_locked/4`, only while the - key still carries its token. A holder busted mid-run is therefore refused at - commit rather than overwriting what its successor has since written, and it - cannot release a successor's lock. - - A fenced section is not lost. The message that took the lock goes back on the - queue in the same transaction that refuses the commit, and the process exits with - `{lock_fenced, requeued}`; whichever consumer holds the lock next retries it. - Because `handle_locked/4` runs outside a transaction, its **external side effects - already happened and will happen again on the retry** — it must be safe to run - twice. - - Since a wrong guess now costs a retry rather than silent data loss, set this for - the recovery time you want from a dead holder rather than above the worst-case - `handle_locked` duration. + in the backend. The key pauses all other consumers while `handle_locked/4` + runs. Normally the lock holder clears the key itself once `handle_locked` + returns. If the holder is killed (SIGKILL, VM abort), the lock stays until + another consumer notices it is stale. That consumer re-checks the lock + `lock_timeout` ms after it was set and clears it if the deadline has passed. + We call this busting the lock. `infinity` (the default) turns off stale-lock + detection entirely, so a dead holder blocks all consumers forever. + + Busting is a guess, based only on time, that the holder is dead. Nothing stops + a holder that is merely slow, so the guess can be wrong. The lock is + **fenced** against that. Each acquisition stamps the key with a unique token, + and a holder may release the lock or commit the result of `handle_locked/4` + only while the key still carries its token. So if a holder is busted mid-run, + its commit is refused instead of overwriting what its successor has written + since, and it cannot release the successor's lock. + + A fenced section is not lost. The transaction that refuses the commit also + puts the message that took the lock back on the queue, and the process exits + with `{lock_fenced, requeued}`. Whichever consumer holds the lock next retries + the message. Because `handle_locked/4` runs outside a transaction, its + **external side effects have already happened and will happen again on the + retry**. It must be safe to run twice. + + A wrong guess now costs a retry rather than silent data loss. So set this to + how quickly you want to recover from a dead holder, rather than above the + worst-case `handle_locked` duration. ## consume_k and inlining A message can reach a callback by two routes: -1. **Through the durable queue.** `cast/2` and (normally) `call/2` enqueue the - message; a consumer later peeks it as part of a `consume_k`-sized batch and - processes it inside the batch's single transaction. This is the ordered, +1. **Through the durable queue.** `cast/2`, and usually `call/2`, enqueue the + message. A consumer later peeks it as part of a `consume_k`-sized batch and + processes it inside that batch's single transaction. This is the ordered, durable path, and the only path when `consume_k > 1`. -2. **Inline.** As a latency optimisation, when `consume_k =:= 1` a `call/2` whose - queue is currently empty and unlocked is processed *immediately*, in the - caller's own request transaction, instead of being enqueued and waited on. - The result is identical to processing it through the queue; it just skips the - enqueue/await round-trip. - -When **`consume_k > 1`, inlining is disabled**: every `call/2` goes through the -queue and the batched consume loop, so `consume_k` is always in effect. Use this -when you want all processing funnelled through the batched, single-consumer loop — -for example to keep a stable consumer identity (an election-style callback that -would otherwise see successive messages committed by different nodes). With the -default `consume_k =:= 1`, inlining is enabled and a contention-free `call/2` is -served on the fast path. - -Independently of `consume_k`, `priority_call/2` and `priority_cast/2` **always** -bypass the queue (and locks) and are handled immediately. They are an explicit -escape hatch for urgent, usually read-only work; they trade away ordering with -respect to queued messages, so prefer `call/2`/`cast/2` for anything that must be -ordered with the rest of the stream. +2. **Inline.** To save latency when `consume_k =:= 1`, a `call/2` that finds + its queue empty and unlocked is processed *immediately*, in the caller's own + request transaction, instead of being enqueued and waited on. The result is + the same as going through the queue. It just skips the round-trip of + enqueueing and waiting. + +When **`consume_k > 1`, inlining is disabled**. Every `call/2` goes through the +queue and the batched consume loop, so `consume_k` always applies. Use this +when you want all processing to go through the batched, single-consumer loop, +for example to keep the same node as consumer. An election-style callback would +otherwise see successive messages committed by different nodes. With the +default `consume_k =:= 1`, inlining is enabled and a `call/2` with no +contention takes the fast path. + +Whatever `consume_k` is, `priority_call/2` and `priority_cast/2` **always** +bypass the queue and any lock, and are handled immediately. They are an +explicit escape hatch for urgent, usually read-only work. They give up ordering +relative to queued messages, so prefer `call/2` and `cast/2` for anything that +must stay in order with the rest of the stream. ## Callbacks @@ -121,25 +122,24 @@ ordered with the rest of the stream. Receives `(Msg, AttemptCount)`. Return value is ignored. Useful for custom alerting or metrics. -`Actions` is a list of 1-arity funs executed after the transaction commits. The -argument is the module state as committed by that transaction — note that under -batched consumption (`consume_k > 1`) all of a batch's actions run after the -batch's single commit and each receives the state as of the **end of the batch**, -not the state at the message that produced the action. +`Actions` is a list of 1-arity funs that run after the transaction commits. +Each one receives the module state as that transaction committed it. Under +batched consumption (`consume_k > 1`), all of a batch's actions run after its +single commit. Each receives the state as of the **end of the batch**, not the +state right after the message that produced the action. ## Module State -The module that implements the `dgen_server` behaviour may define any term to -serve as the State. `dgen_server` will encode this state for writing to the -database. We encourage you to structure your state to fit in with this -encoding scheme, which will yield performance benefits. +The module that implements the `dgen_server` behaviour may use any term as its +State. `dgen_server` encodes this state to write it to the database. If you +shape your state to fit the encodings below, updates get cheaper. - **term** (fallback) - `term_to_binary`, chunked into 100KB values. -- **assigns map** - map with all atom keys; each entry stored at a - separate DB key using `atom_to_binary(Key)` in the path. -- **component list** - list of maps where every item has an atom `id` - key with a binary value; each item stored separately, ordered by a - fractional index embedded in the DB key. +- **assigns map** - a map whose keys are all atoms. Each entry is stored at its + own DB key, with `atom_to_binary(Key)` in the path. +- **component list** - a list of maps where every item has an atom `id` key + with a binary value. Each item is stored separately, ordered by a fractional + index embedded in the DB key. The encoding is applied recursively. For example, an assigns map whose value is a component list will nest both encodings in the key path. @@ -174,16 +174,16 @@ is a component list will nest both encodings in the key path. -type noreply_ret() :: {noreply, state()} | {noreply, state(), [action()]}. -type stop_ret() :: {stop, term(), state()} | {stop, term(), state(), [action()]}. -%% Passed as the first argument to `_tx` callback variants. -%% `td` is the current backend transaction+directory pair; `tuid` is the server's -%% tenant-unique identifier. Both may be used to read or write arbitrary keys -%% within the same atomic transaction as the callback. +%% Passed as the first argument to the `_tx` callback variants. +%% `td` is the current backend transaction and directory pair. `tuid` is the +%% server's tenant-unique identifier. A callback can use them to read or write +%% any keys in the same atomic transaction it runs in. -type tx_ctx() :: #{td := dgen_backend:tenant(), tuid := tuid()}. %% Passed as the first argument to `handle_locked/4`. -%% `db` is the DB-level tenant (not a transaction); `tuid` is the server's -%% tenant-unique identifier. Use `dgen_backend:transactional/2` to open -%% explicit transactions within the locked section. +%% `db` is the DB-level tenant, not a transaction. `tuid` is the server's +%% tenant-unique identifier. Use `dgen_backend:transactional/2` to open +%% transactions inside the locked section. -type db_ctx() :: #{db := dgen_backend:tenant(), tuid := tuid()}. -callback init(Args :: term()) -> init_ret(). @@ -265,15 +265,15 @@ is a component list will nest both encodings in the key path. dead_letter_threshold = infinity :: pos_integer() | infinity, consume_k = 1 :: pos_integer(), lock_timeout = infinity :: pos_integer() | infinity, - %% Token of the lock this process currently holds, set by set_lock/2 within the - %% transaction that acquired it. Identifies *which* acquisition is ours, so - %% releasing and committing can tell our own lock from a successor's. + %% The token of the lock this process holds. set_lock/2 sets it in the + %% transaction that takes the lock. It tells us *which* acquisition is ours, + %% so that releasing and committing can tell our lock from a successor's. %% See clear_lock_owned/2 and owns_lock/2. lock_token :: undefined | binary(), - %% The queue envelope of the message that took the lock, kept so a fenced - %% locked section can put it back rather than drop it (see consume_locked/5). - %% `undefined` for locks taken on the priority and inline paths, which have no - %% queued message behind them. + %% The queue envelope of the message that took the lock. We keep it so that + %% a fenced locked section can put the message back instead of dropping it + %% (see consume_locked/5). It is `undefined` for locks taken on the priority + %% and inline paths, because no queued message stands behind them. locked_envelope :: undefined | term() }). @@ -531,14 +531,13 @@ inline_or_push(Td, Request, LocalFrom, WatchTo, Options, Tuid, State = #state{co Push = fun(Attempts) -> dgen:push_call(Td, Tuid, get_quid(Tuid), Request, WatchTo, Options, Attempts) end, - %% Inline handling — process the call immediately in this transaction instead of - %% enqueuing it and waiting for the consume loop — is attempted only when - %% consume_k =:= 1, and then only if the queue is empty and unlocked. With - %% consume_k > 1 the server is configured for batched, single-consumer processing, - %% so inlining is disabled entirely: every call goes through the durable queue and - %% the consume loop, keeping consume_k always in effect. See "consume_k and - %% inlining" in the moduledoc. (`priority_call` always bypasses the queue, - %% independent of consume_k.) + %% Inlining means processing the call right here in this transaction, instead + %% of enqueueing it and waiting for the consume loop. We try it only when + %% `consume_k =:= 1`, and then only if the queue is empty and unlocked. With + %% `consume_k > 1` the server is set up for batched, single-consumer + %% processing, so we never inline. Every call goes through the durable queue + %% and the consume loop, so `consume_k` always applies. See "consume_k and + %% inlining" in the moduledoc. `priority_call` bypasses the queue regardless. InlineEligible = K =:= 1 andalso dgen_queue:length(Td, get_quid(Tuid)) =:= 0, case InlineEligible andalso not is_locked(Td, State) of true -> @@ -610,9 +609,9 @@ handle_info({Ref, ready}, State = #state{watch = ?FUTURE(Ref)}) -> handle_info(consume_after_penalty, State) -> handle_cast(consume, State); handle_info(recheck_lock, State) -> - %% Fired by the timer scheduled in handle_consume when the lock was not yet - %% stale. Re-enter the consume loop so check_lock is re-evaluated; if - %% the lock holder died without clearing the lock, it will now be stale. + %% handle_consume schedules this timer when it finds a lock that is not yet + %% stale. We re-enter the consume loop so check_lock runs again. If the + %% holder died without clearing the lock, the lock is now stale. handle_cast(consume, State); handle_info(Info, State = #state{tenant = Tenant}) -> Result = dgen_backend:transactional(Tenant, fun(Td) -> @@ -628,10 +627,10 @@ terminate(_Reason, _State) -> code_change(_OldVsn, State, _Extra) -> {ok, State}. -%% Dispatches a callback, preferring the `_tx` variant (which receives a -%% `tx_ctx()` as its first argument) when the callback module exports it. +%% Dispatches a callback. If the callback module exports the `_tx` variant, we +%% call that one, with a `tx_ctx()` as its first argument. %% -%% Resolution order for a callback named `handle_foo/N`: +%% For a callback named `handle_foo/N`, we look for these in order: %% 1. `handle_foo_tx/(N+1)` — receives `#{td, tuid}` prepended to Args %% 2. `handle_foo/N` — standard variant, no tx context invoke_tx_callback(Td, Callback, Args, State = #state{mod = Mod, tuid = Tuid}) -> @@ -671,7 +670,8 @@ dispatch_callback(_Td, ModState, Mod, Fn, Args, _State) -> CallbackResult = erlang:apply(Mod, Fn, Args ++ [ModState]), {ok, ModState, CallbackResult}. -%% Like invoke_tx_callback/4 but uses CurrentModState directly instead of reading from FDB. +%% Like invoke_tx_callback/4, but uses `CurrentModState` instead of reading the +%% state from the backend. invoke_tx_callback_batch( Td, Callback, Args, CurrentModState, State = #state{mod = Mod, tuid = Tuid} ) -> @@ -812,10 +812,10 @@ handle_callback_result( {_, State1} = set_mod_state(Td, OrigModState, ModState, State), {{stop, Reason, Actions}, ModState, State1}. -%% Batch variant of handle_callback_result: does NOT write mod state to the backend. -%% consume_batch is responsible for a single set_mod_state call at each exit point. -%% The lock case still sets the lock key within the transaction, as it must be -%% atomic with the mod state write and queue consume. +%% The batch variant of handle_callback_result. It does NOT write the mod state +%% to the backend. Instead, consume_batch calls set_mod_state once at each exit +%% point. The lock case still sets the lock key in this transaction, because it +%% must commit atomically with the mod state write and the dequeue. handle_callback_result_batch(_Td, _EventType, _Msg, {noreply, ModState}, _OrigModState, State) -> {{noreply, []}, ModState, State}; handle_callback_result_batch( @@ -876,14 +876,15 @@ handle_consume(Tenant, K, Tuid, State = #state{dead_letter_threshold = Threshold clear_lock(Td, State), consume_queued(Td, K, Quid, Threshold, State); {live, Remaining} -> - %% Lock is held but not yet stale. Set a queue watch - %% for the fast path (lock cleared normally fires - %% notify → push key changes → watch fires). Also - %% schedule a timer so that if the lock holder is - %% killed without clearing the lock and no new messages - %% arrive, we still re-evaluate staleness after the - %% remaining timeout — guaranteeing recovery even on a - %% perfectly quiet queue. + %% The lock is held but not yet stale. We set a queue + %% watch for the fast path. When the holder clears the + %% lock normally, it notifies the queue, the push key + %% changes, and the watch fires. We also schedule a + %% timer for when the remaining timeout runs out. If + %% the holder is killed without clearing the lock and + %% no new messages arrive, the timer still makes us + %% re-check staleness, so we recover even on a + %% completely quiet queue. Watch = dgen_queue:watch_push(Td, Quid), Action = fun(_) -> case Remaining of @@ -962,14 +963,15 @@ consume_queued(Td, K, Quid, Threshold, State) -> end end. -%% Processes peeked KVs one at a time, carrying mod state in memory. -%% Mod state is read once (in consume_queued) and written once at each exit point, -%% avoiding repeated FDB reads/writes due to set_versionstamped_value not being -%% visible in read-your-own-writes within the same transaction. +%% Processes the peeked KVs one at a time, carrying the mod state in memory. +%% consume_queued reads the mod state once, and we write it once at each exit +%% point. We can't read it back after each message, because a value written with +%% set_versionstamped_value is not visible to later reads in the same +%% transaction. %% %% Parameters: -%% 8th — CurrentModState: mod state produced by the last successfully invoked callback -%% 9th — InitModState: mod state at the start of the batch, used for diff-based writes +%% 8th, CurrentModState: the mod state returned by the last callback that succeeded +%% 9th, InitModState: the mod state at the start of the batch, used to write a diff consume_batch(Td, [], Quid, _Threshold, State, AccActions, AccKVs, CurrentModState, InitModState) -> {_, State1} = set_mod_state(Td, InitModState, CurrentModState, State), dgen_queue:consume_peeked(Td, lists:reverse(AccKVs), Quid), @@ -1014,8 +1016,8 @@ consume_batch( dgen_queue:consume_peeked(Td, AllKVs, Quid), PriorActions = lists:append(lists:reverse(AccActions)), %% This message leaves the queue here, in the transaction that - %% takes the lock, so keep its envelope: if the locked section - %% is later fenced, that is the only way to put it back. + %% takes the lock. We keep its envelope, because if the locked + %% section is later fenced, that is the only way to put it back. State3 = State2#state{cache_misses = 0, locked_envelope = Envelope}, case PriorActions of [] -> {{lock, EventType, Msg}, ModState, State3}; @@ -1033,8 +1035,8 @@ consume_batch( [] -> ok; _ -> - % Write mod state for successfully processed prior messages - % before committing their dequeues. + % Writes the mod state from the earlier messages that + % succeeded, before committing their dequeues. set_mod_state(Td, InitModState, CurrentModState, State), dgen_queue:consume_peeked(Td, lists:reverse(AccKVs), Quid) end, @@ -1114,12 +1116,13 @@ consume_locked(EventType, Msg, ModState, IsLocalReply, State = #state{tenant = T erlang:error(Reason); {ok, OrigModState, CallbackResult, State1} -> dgen_backend:transactional(Tenant, fun(Td) -> - %% Fence. handle_locked/4 ran outside any transaction, so an - %% arbitrary amount of time has passed since the lock was taken - %% and another consumer may have judged this one dead and busted - %% it. CallbackResult was computed from the state read *before* - %% the lock, so committing it now would overwrite whatever the - %% successor has since done, silently. Refuse instead. + %% The fence. handle_locked/4 ran outside any transaction, so + %% any amount of time may have passed since we took the lock. + %% Another consumer may have decided we were dead and busted + %% the lock. `CallbackResult` came from the state we read + %% *before* the lock, so committing it now would silently + %% overwrite whatever the successor has done since. We refuse + %% instead. case owns_lock(Td, State1) of false -> fence(Td, State1); @@ -1158,28 +1161,28 @@ consume_locked(EventType, Msg, ModState, IsLocalReply, State = #state{tenant = T case Result of {fenced, Requeued} -> - %% Raised only after the fencing transaction has committed, so the - %% requeue is durable before this process goes down. Crashing is the - %% same disposition a failing callback gets: the mod state was not - %% written, no actions run, and the in-memory cache dies with the - %% process rather than being carried forward past a missed write. + %% We raise only after the fencing transaction has committed, so the + %% requeue is durable before this process goes down. A failing + %% callback is handled the same way. The mod state was not written + %% and no actions run. The in-memory cache dies with the process, so + %% it is never carried forward past a write that didn't happen. erlang:error({lock_fenced, Requeued}); _ -> finalize(Result) end. -%% A fenced locked section: commit nothing of the callback's result, but put the -%% message that took the lock back on the queue in the same transaction, so the -%% refusal costs a retry rather than the message. The retry is picked up by -%% whichever consumer holds the lock now. +%% Handles a fenced locked section. We commit none of the callback's result, but +%% we put the message that took the lock back on the queue in the same +%% transaction. That way the refusal costs a retry instead of the message. +%% Whichever consumer holds the lock now picks up the retry. %% -%% The envelope is re-pushed verbatim, so a call keeps its reply slot (the waiting -%% caller is never told; it simply waits for the retry) and the attempt counter is -%% carried over unchanged — being fenced is not the message's fault and must not -%% walk it toward the dead-letter queue. +%% We re-push the envelope unchanged. A call keeps its reply slot, so the waiting +%% caller never hears about the fence and just waits for the retry. The attempt +%% counter also stays the same. Being fenced is not the message's fault, so it +%% must not move the message toward the dead-letter queue. %% -%% Priority and inline locks have no queued message (`locked_envelope` is -%% undefined); for those the caller receives the exit. +%% Priority and inline locks have no queued message, so `locked_envelope` is +%% `undefined`. For those, the caller receives the exit. fence(_Td, #state{locked_envelope = undefined}) -> {fenced, not_requeued}; fence(Td, #state{locked_envelope = Envelope, tuid = Tuid}) -> @@ -1202,7 +1205,7 @@ set_reply(Td, From, Reply) -> set_raise(Td, From, Class, Reason) -> write_reply_slot(Td, From, {raise, Class, Reason}). -% Skip writing if the caller timed out and cleared the reply keys. +% Skips the write if the caller timed out and cleared the reply keys. write_reply_slot({Tx, Dir}, From, Value) -> B = dgen_config:backend(), ReplySentinelKey = dgen_mod_state_codec:term_first_key(Dir, From), @@ -1228,13 +1231,15 @@ resolve_version(State = #state{mod_state_cache = {VF, ModStateResult}}) -> resolve_version(State) -> State. -%% A lock records the time it was set — so staleness can be computed — and a token -%% unique to this acquisition. The token is what makes releasing safe: a holder -%% busted mid-run must not clear the lock its successor now holds (clear_lock_owned/2). +%% Sets the lock key. The lock records the time it was set, so other consumers +%% can tell when it is stale. It also records a token unique to this acquisition. +%% The token is what makes releasing safe, because a holder busted mid-run must +%% not clear the lock its successor now holds (see clear_lock_owned/2). %% -%% Runs inside the acquiring transaction, so on a commit conflict the whole closure -%% re-runs and mints a fresh token; only the committed attempt's State survives, so -%% the token in the returned State is always the one that actually landed. +%% This runs inside the transaction that takes the lock. On a commit conflict the +%% whole closure runs again and mints a fresh token. Only the State from the +%% attempt that committed survives, so the returned token is always the one that +%% actually landed. set_lock({Tx, Dir}, State = #state{tuid = Tuid}) -> B = dgen_config:backend(), Token = crypto:strong_rand_bytes(8), @@ -1242,22 +1247,25 @@ set_lock({Tx, Dir}, State = #state{tuid = Tuid}) -> B:set(Tx, B:dir_pack(Dir, get_lock_key(Tuid)), Value), State#state{lock_token = Token}. -%% Single read of the lock key — returns not_locked | stale | {live, infinity | Ms}. -%% Used by handle_consume to decide in one pass whether to clear, sleep, or proceed. +%% Reads the lock key once and returns not_locked | stale | {live, infinity | Ms}. +%% handle_consume uses it to decide in one pass whether to clear the lock, wait, +%% or proceed. check_lock({Tx, Dir}, #state{tuid = Tuid, lock_timeout = Timeout}) -> B = dgen_config:backend(), case B:wait(B:get(Tx, B:dir_pack(Dir, get_lock_key(Tuid)))) of not_found -> not_locked; <<>> -> - %% backward compat: v0.2.0 used empty binary (no timestamp) — treat - %% as live with no timeout so we never falsely clear a legitimate lock. + %% Backward compatibility. v0.2.0 wrote an empty binary with no + %% timestamp. We treat it as live with no timeout, so we never clear + %% a legitimate lock by mistake. {live, infinity}; Value -> - %% v0.3.0+ stores the set-time as a millisecond timestamp so we can - %% compute both staleness and the remaining wait in one decode; v0.4.1+ - %% pairs it with an acquisition token. Both shapes are read, so a node - %% running the older writer stays interoperable during an upgrade. + %% v0.3.0 and later store the time the lock was set, in milliseconds, + %% so one decode gives us both staleness and the remaining wait. + %% v0.4.1 and later pair the timestamp with an acquisition token. We + %% read both shapes, so a node still running the older writer keeps + %% working during an upgrade. case (catch binary_to_term(Value)) of {Ts, _Token} when is_integer(Ts) -> lock_liveness(Ts, Timeout); @@ -1276,10 +1284,10 @@ lock_liveness(Ts, Timeout) -> _ -> {live, Timeout - Elapsed} end. -%% Unconditional release, used when busting a lock judged stale. Safe there -%% without a token check because check_lock/2 read the key in the same -%% transaction: the read conflict fences this against a concurrent acquisition, -%% so the clear can only commit if the value judged stale is still the one there. +%% Releases the lock unconditionally. We use it to bust a lock judged stale. That +%% is safe without a token check, because check_lock/2 read the key in the same +%% transaction. If another consumer takes the lock concurrently, that read +%% conflicts, so the clear only commits if the stale value is still there. clear_lock(Td = {Tx, Dir}, #state{tuid = Tuid}) -> B = dgen_config:backend(), LockKey = B:dir_pack(Dir, get_lock_key(Tuid)), @@ -1287,29 +1295,29 @@ clear_lock(Td = {Tx, Dir}, #state{tuid = Tuid}) -> B:clear_range(Tx, LockKey, LockEnd), dgen_queue:notify(Td, get_quid(Tuid)). -%% Release for a holder letting go of a lock it acquired itself, which — unlike -%% busting — happens in a *later* transaction than the one that took the lock. -%% In between, a consumer that judged this holder stale may have busted it and -%% taken the lock for its own locked section. Clearing unconditionally would then -%% release a lock this process does not hold, admitting a third consumer with no -%% timeout elapsed and no fault: one late holder would unlock an arbitrarily long -%% chain of successors. So clear only a lock still carrying our own token. +%% Releases a lock this process took itself. Unlike busting, this happens in a +%% *later* transaction than the one that took the lock. In between, a consumer +%% that judged us stale may have busted the lock and taken it for its own locked +%% section. If we cleared unconditionally, we would release a lock we don't hold +%% and let a third consumer in, with no timeout elapsed and no fault. One late +%% holder could then unlock a long chain of successors. So we only clear a lock +%% that still carries our own token. %% -%% Declining to clear costs nothing. A lock that really is this holder's leftover -%% still carries its token and is cleared here; one that does not is a successor's -%% live lock, which that successor will release, or the staleness path will bust. +%% Declining to clear costs nothing. If the lock really is ours, it still carries +%% our token and we clear it here. If it doesn't, it is a successor's live lock. +%% That successor will release it, or the staleness check will bust it. clear_lock_owned(Td, State) -> case owns_lock(Td, State) of true -> clear_lock(Td, State); false -> ok end. -%% Does the lock key still carry the token this process acquired? +%% Returns whether the lock key still carries the token this process acquired. %% -%% The read is what makes fencing work: performed inside the transaction that is -%% about to write, it puts the lock key in that transaction's conflict set, so a -%% bust racing the commit forces a retry rather than slipping between the check -%% and the write. There is no window. +%% This read is what makes fencing work. We do it inside the transaction that is +%% about to write, so the lock key joins that transaction's conflict set. If a +%% bust races the commit, the transaction retries. The bust can't slip in between +%% the check and the write. owns_lock({Tx, Dir}, #state{tuid = Tuid, lock_token = Token}) when Token =/= undefined -> B = dgen_config:backend(), case B:wait(B:get(Tx, B:dir_pack(Dir, get_lock_key(Tuid)))) of @@ -1333,7 +1341,7 @@ is_locked(Td, State) -> get_quid(Tuple) -> dgen_key:extend(Tuple, <<"q">>). -%% Arm a timer, and say so. See `dgen_registry_member:arm/2`. +%% Arms a timer and logs an `arm` event for simulation traces. See `dgen_registry_member:schedule_timer/2`. arm(Delay, Msg) -> _ = ?ETA_LOG({arm, Msg, Delay}), erlang:send_after(Delay, self(), Msg). diff --git a/src/dgen_transaction.erl b/src/dgen_transaction.erl index c57c903..87cb8a3 100644 --- a/src/dgen_transaction.erl +++ b/src/dgen_transaction.erl @@ -7,17 +7,17 @@ -moduledoc """ Behaviour that wraps the lifecycle of a single FDB transaction in a process. -`dgen_backend:transactional/2` runs a closure and auto-retries it: simple, but -the transaction is anonymous and the caller's process blocks for the whole -commit. `dgen_transaction` instead gives the transaction **its own process**. -The owning process creates the transaction, runs a callback module's body -against it, decides when to commit, drives the retry loop, and delivers the -outcome asynchronously. Two things this buys us: - -1. **More control over the transaction lifecycle.** For example, a dgen_transaction - can be written using a previous transaction's GRV instead of paying for - a fresh GRV, then falling back to a fresh GRV automatically on retry - (the backend's `on_error/2` resets the read version). +`dgen_backend:transactional/2` runs a closure and retries it automatically. +That is simple, but the caller never holds the transaction, and the caller's +process blocks for the whole commit. `dgen_transaction` instead gives the +transaction **its own process**. That process creates the transaction, runs the +body from a callback module against it, decides when to commit, drives the +retry loop, and delivers the outcome asynchronously. This buys us two things: + +1. **More control over the transaction lifecycle.** For example, a + dgen_transaction can reuse the read version (GRV) of a previous transaction + instead of paying for a fresh one. On retry it falls back to a fresh GRV + automatically, because the backend's `on_error/2` resets the read version. 2. **Non-blocking commit.** A calling process can hand a write to a `dgen_transaction` worker and keep receiving messages from its queue while the commit is in flight, learning the result via a message. @@ -49,34 +49,35 @@ When the transaction reaches a terminal state the worker sends the owner {dgen_transaction, Ref, Reply} ``` -where `Ref` is the caller-supplied (or generated) correlation token and `Reply` -is one of: +where `Ref` is the correlation token, either supplied by the caller or +generated, and `Reply` is one of: - `{committed, Result}` — `Result` is whatever `handle_committed/2` returned. -- `{aborted, Reason}` — the module chose not to commit / not to retry. +- `{aborted, Reason}` — the module chose not to commit, or not to retry. - `{error, Reason}` — a non-retryable backend error, retry-limit exhaustion, or the worker terminating abnormally. ## Callbacks -- `init/1` — set up callback state. -- `handle_begin/2` — issue the transaction body (reads for fencing + writes) - against the live `Tx`; replayed on every retry unless `handle_retry/2` is - provided. Returns a directive (noreply, commit, etc). +- `init/1` — sets up the callback state. +- `handle_begin/2` — issues the transaction body against the live `Tx`. That is + the reads that guard the commit against conflicts, plus the writes. It is + replayed on every retry unless the module provides `handle_retry/2`. Returns + a directive such as `noreply` or `commit`. - `handle_committed/2` — called with the committed version after a successful - commit; returns `{ok, Result}`. + commit. Returns `{ok, Result}`. Optional: -- `handle_cast/3`, `handle_call/4` — interactive operations against the live - transaction before commit (the process-model the caller drives directly). -- `handle_retry/2` — re-issue the body on retry (defaults to `handle_begin/2`). -- `handle_conflict/2` — veto or allow a retry on a retryable error - (defaults to `retry`). It governs retryable errors raised **either** by the - commit **or** while issuing the body (e.g. a read at a stale pinned read - version) — both route through `handle_conflict/2` → `on_error/2` → replay, so a - too-old pinned read falls back to a fresh GRV rather than failing the worker. -- `terminate/2` — cleanup; receives the terminal `Reply`. +- `handle_cast/3`, `handle_call/4` — let the caller drive the live transaction + directly with messages before it commits. +- `handle_retry/2` — re-issues the body on retry. Defaults to `handle_begin/2`. +- `handle_conflict/2` — allows or vetoes a retry after a retryable error. + Defaults to `retry`. It sees retryable errors from the commit **and** from + issuing the body, such as a read at a pinned read version that has grown too + old. Both go through `handle_conflict/2` → `on_error/2` → replay. So a stale + pinned read version falls back to a fresh GRV instead of failing the worker. +- `terminate/2` — cleans up. Receives the terminal `Reply`. *This documentation is LLM-generated. See the AI disclosure in `README.md`.* """. @@ -107,9 +108,10 @@ Optional: -export_type([directive/0]). -%% The transaction handle is passed to the body callbacks; everything else the -%% body needs (directory, tuid, the batch being committed, …) lives in the -%% callback module's own state. dgen_transaction itself is directory-agnostic. +%% The body callbacks receive the transaction handle. Everything else the body +%% needs, such as the directory, the tuid or the batch being committed, lives in +%% the callback module's own state. dgen_transaction itself knows nothing about +%% directories. -callback init(Args :: term()) -> {ok, cb_state()} | {stop, Reason :: term()}. -callback handle_begin(Tx :: dgen_backend:tx(), State :: cb_state()) -> directive(). -callback handle_committed(CommittedVersion :: integer(), State :: cb_state()) -> @@ -146,8 +148,8 @@ Optional: ref :: term(), retries = 0 :: non_neg_integer(), max_retries :: non_neg_integer() | infinity, - %% Set once the terminal Reply has been delivered to the owner, so the - %% gen_server terminate/2 does not deliver a second (failure) Reply. + %% Set once the owner has the terminal Reply, so that the gen_server + %% terminate/2 does not send it a second, failure Reply. done = false :: boolean() }). @@ -167,10 +169,12 @@ Optional: -doc """ Starts (and links) a transaction worker for `Module`. -`Opts` must contain `{db, Db}`. Optional keys: `read_version` -(`undefined` | `V`, default `undefined`), `owner` (default the caller), -`ref` (correlation token, default a fresh `make_ref/0`), `max_retries` -(default `infinity`). +`Opts` must contain `{db, Db}`. The optional keys are: + +- `read_version`: `undefined` or a version `V`. Defaults to `undefined`. +- `owner`: defaults to the caller. +- `ref`: the correlation token. Defaults to a fresh `make_ref/0`. +- `max_retries`: defaults to `infinity`. """. -endif. -spec start_link(module(), term(), opts()) -> {ok, pid()} | {error, term()}. @@ -196,11 +200,11 @@ start_monitor(Module, Args, Opts) -> -doc """ Runs a transaction worker synchronously and returns its `Reply`. -Convenience wrapper around `gen_server:start_monitor/3` (unlinked, started and -monitored atomically): blocks until the worker delivers its terminal `Reply`, or -returns `{error, {crashed, Reason}}` if the worker dies first. Intended for -callers that want the explicit-lifecycle features (cached GRV, module-controlled -retry) without managing the worker. +Starts the worker with `gen_server:start_monitor/3`, which monitors it +atomically without linking. Blocks until the worker delivers its terminal +`Reply`, or returns `{error, {crashed, Reason}}` if the worker dies first. Use +it when you want the lifecycle control, such as a cached GRV or retries decided +by the module, without managing the worker yourself. """. -endif. -spec run(module(), term(), opts()) -> term(). @@ -280,7 +284,7 @@ handle_info(_Info, St) -> terminate(_Reason, #st{done = true}) -> ok; terminate(Reason, St = #st{done = false}) -> - %% Abnormal exit before a terminal Reply was delivered — tell the owner. + %% We are exiting abnormally before the owner got a terminal Reply, so tell it. _ = finish(St, {error, {terminated, Reason}}), ok. @@ -291,11 +295,12 @@ code_change(_OldVsn, St, _Extra) -> %% Directive handling %% --------------------------------------------------------------------------- -%% Run the transaction body (handle_begin first, handle_retry/handle_begin on a -%% replay) and act on its directive. The body invocation is wrapped so that a -%% retryable FDB error raised *inside it* (e.g. a read at a stale pinned read -%% version) routes through the same retry machinery as a commit conflict — -%% retries are the callback module's decision (handle_conflict/2), not a crash. +%% Runs the transaction body and acts on its directive. The first attempt runs +%% handle_begin, and a replay runs handle_retry or handle_begin. We catch errors +%% from the body so that a retryable FDB error raised inside it, such as a read at +%% a stale pinned read version, goes through the same retry path as a commit +%% conflict. The callback module decides whether to retry in handle_conflict/2, +%% instead of the worker crashing. run_body(St = #st{mod = M, retries = N}) -> BodyResult = try @@ -347,8 +352,8 @@ do_commit(St = #st{b = B, tx = Tx, mod = M, cb = Cb}) -> error:Reason -> handle_fdb_error(Reason, St) end. -%% A backend error from the body or the commit: a non-FDB error is fatal; an FDB -%% error code is offered to the module for a retry decision. +%% Handles a backend error from the body or the commit. A non-FDB error is fatal. +%% For an FDB error code, we let the module decide whether to retry. handle_fdb_error(Reason, St = #st{b = B}) -> case B:error_code(Reason) of error -> @@ -380,7 +385,8 @@ retry_commit(Code, St = #st{retries = N, max_retries = Max}) when retry_commit(Code, St = #st{b = B, tx = Tx, retries = N}) -> try B:wait(B:on_error(Tx, Code)) of _ -> - %% Transaction reset for retry (read version cleared → fresh GRV). + %% on_error reset the transaction and cleared its read version, so + %% the retry gets a fresh GRV. run_body(St#st{retries = N + 1}) catch error:Reason -> @@ -400,7 +406,7 @@ apply_read_version(_B, _Tx, undefined) -> apply_read_version(B, Tx, Version) -> B:set_read_version(Tx, Version). -%% Deliver the terminal Reply to the owner exactly once and mark the worker done. +%% Delivers the terminal Reply to the owner exactly once and marks the worker done. finish(St = #st{done = true}, _Reply) -> St; finish(St = #st{owner = Owner, ref = Ref, mod = M, cb = Cb}, Reply) -> diff --git a/src/dgen_utils.erl b/src/dgen_utils.erl index 77a80ed..bd176a5 100644 --- a/src/dgen_utils.erl +++ b/src/dgen_utils.erl @@ -1,8 +1,9 @@ -module(dgen_utils). -%% Small, dependency-free helpers shared across dgen modules (the BEAM/distribution -%% analogue of dgen_key's key helpers). Keep this module general: anything here -%% should be useful to more than one caller and carry no dgen-specific state. +%% Small, dependency-free helpers for the BEAM and distribution, shared across dgen +%% modules. It plays the same role for them that `dgen_key` plays for keys. Keep this +%% module general. Anything here should be useful to more than one caller and hold +%% no dgen-specific state. %% %% *This documentation is LLM-generated. See the AI disclosure in README.md.* @@ -12,35 +13,34 @@ -if(?DOCATTRS). -doc """ -Is `Node` usable for a synchronous message right now — the local node, or a -currently-connected one? +Returns whether we can send `Node` a message right now. That is true for the local +node and for any node that is currently connected. -This is the predicate the registry uses everywhere it must decide whether a cast -or call to a peer will actually be delivered (as opposed to silently triggering an -automatic distribution *reconnect*, which would heal a partition before it can be -observed — see `dgen_registry_member:cast_to_member/2` and -`dgen_registry_elector:call_to_member/2`). The local node is trivially reachable; -a remote node is reachable iff it is in `nodes()`. +The registry uses this check wherever it must decide whether a cast or call to a +peer will really be delivered. Sending to a disconnected node would instead start an +automatic distribution reconnect, which can heal a partition before the registry +notices it. See `dgen_registry_member:cast_to_member/2` and +`dgen_registry_elector:call_to_member/2`. A remote node is reachable only if it is +in `nodes()`. """. -endif. -spec node_reachable(node()) -> boolean(). node_reachable(Node) -> Node =:= node() orelse lists:member(Node, nodes()). -%% Wall-clock sleep and monotonic read that stay real under simulation. +%% A sleep and a monotonic clock read that stay on real time under simulation. %% -%% This module deliberately carries no `dgen_eta.hrl` include, so calls routed -%% through here are exempt from the eta transform. That exemption is the entire -%% point: a *startup* wait like `dgen_registry:await_ready/2` runs in whatever -%% process is bringing the system up — under `eta_run` that is the driver, the one -%% process whose transformed sleep arms a virtual deadline nothing will ever reach -%% (`eta_time:sleep/1`'s own boundary rule). Readiness polling is a wall-clock -%% concern by definition: it happens before the schedule owns the system, or on a -%% real node where there is no schedule at all. +%% This module deliberately doesn't include `dgen_eta.hrl`, so the eta transform +%% leaves calls through here alone. That is the whole point. A startup wait such as +%% `dgen_registry:await_ready/2` runs in whichever process is bringing the system +%% up. Under `eta_run` that is the driver. If the driver's sleep were transformed, +%% it would wait for a virtual deadline that never arrives, by the boundary rule in +%% `eta_time:sleep/1`. Readiness polling is about wall-clock time anyway. It happens +%% either before the scheduler takes over, or on a real node with no scheduler. %% -%% Do NOT reach for these inside a member/elector/connector code path that runs -%% during a simulation — there they are a real-time leak of exactly the class the -%% transform exists to remove. +%% Do NOT use these in member, elector or connector code that runs during a +%% simulation. There they would leak real time into the simulation, which is +%% exactly what the transform exists to prevent. -spec real_sleep(non_neg_integer()) -> ok. real_sleep(Ms) -> timer:sleep(Ms). diff --git a/test/dgen_registry_cluster_test.exs b/test/dgen_registry_cluster_test.exs index 4268258..9d12cf5 100644 --- a/test/dgen_registry_cluster_test.exs +++ b/test/dgen_registry_cluster_test.exs @@ -111,7 +111,7 @@ defmodule DGen.RegistryClusterTest do # The peer member's current applied version (via the same snapshot call the # handoff gather uses), so a test can craft version-contiguous broadcasts. defp peer_applied_version(peer_node, peer_member) do - {_records, version, _released} = + {_records, version, _released, _leader_down} = :erpc.call(peer_node, GenServer, :call, [peer_member, :get_names_snapshot]) version @@ -1149,7 +1149,7 @@ defmodule DGen.RegistryClusterTest do member = :dgen_registry.member_name(reg) epoch = :dgen_registry.get_epoch(reg) leader = :dgen_registry.get_leader(reg) - {_records, v, _released} = GenServer.call(member, :get_names_snapshot) + {_records, v, _released, _leader_down} = GenServer.call(member, :get_names_snapshot) GenServer.cast( member, @@ -1408,7 +1408,7 @@ defmodule DGen.RegistryClusterTest do assert :undefined == remote_whereis(peer_node, reg, :gap_ghost) # The resync snapshot re-baselines the peer to the leader's version... - {_records, leader_v, _released} = + {_records, leader_v, _released, _leader_down} = GenServer.call(:dgen_registry.member_name(reg), :get_names_snapshot) assert eventually(fn -> diff --git a/test/dgen_registry_conflicts_test.exs b/test/dgen_registry_conflicts_test.exs index faff91b..b9712bd 100644 --- a/test/dgen_registry_conflicts_test.exs +++ b/test/dgen_registry_conflicts_test.exs @@ -6,7 +6,7 @@ defmodule DGen.RegistryConflictsTest do returns the kills and alarms for the member to carry out. These tests pin which gathered states are a conflict, what the remedy does with one (and when it holds back), when the release trail may expire, and the version-defined - `degraded/5`. + `is_reconstruction_degraded/5`. """ use ExUnit.Case, async: true @@ -99,21 +99,6 @@ defmodule DGen.RegistryConflictsTest do ) end - test "a legacy bare-pid trail entry (rolling upgrade) still suppresses" do - p1 = spawn_live() - p2 = spawn_live() - # A pre-upgrade member's gathered trail keys by pid alone; the detector - # honours that coarser shape until the mixed entries age out. - released = %{p2 => System.system_time(:millisecond)} - - assert [] = - detect( - %{n: rec(p1)}, - [%{n: rec(p1)}, %{n: rec(p2)}], - released - ) - end - test "a dead divergent pid is not a conflict" do p1 = spawn_live() p2 = spawn(fn -> :ok end) @@ -171,9 +156,9 @@ defmodule DGen.RegistryConflictsTest do end # The leader adjudicates on its own loop, so liveness is probed ahead of time in - # the gather helpers (conflict_pids/3 says which pids) and handed in as data + # the gather helpers (get_conflict_pids/3 says which pids) and handed in as data # (detect/4). No probe — no RPC — runs during adjudication. - test "conflict_pids/3 names exactly the pids adjudication would probe" do + test "get_conflict_pids/3 names exactly the pids adjudication would probe" do [p1, p2, p3, p4] = for _ <- 1..4, do: spawn_live() fresh = %{n: rec(p1), agreed: rec(p3)} @@ -181,7 +166,7 @@ defmodule DGen.RegistryConflictsTest do released = %{{:trailed, p4} => System.system_time(:millisecond)} assert Enum.sort([p1, p2]) == - :dgen_registry_conflicts.conflict_pids(fresh, witnesses, released) + :dgen_registry_conflicts.get_conflict_pids(fresh, witnesses, released) end test "detect/4 decides on the liveness it is given, not by probing" do @@ -251,46 +236,47 @@ defmodule DGen.RegistryConflictsTest do test "merging keeps each entry's latest release time" do p = spawn_live() - c = Conflicts.released([{:n, p}], 10, Conflicts.new()) + c = Conflicts.record_releases([{:n, p}], 10, Conflicts.new()) c = Conflicts.merge([%{{:n, p} => 5}, %{{:m, p} => 7}], c) - assert Conflicts.trail(c) == %{{:n, p} => 10, {:m, p} => 7} + assert Conflicts.get_trail(c) == %{{:n, p} => 10, {:m, p} => 7} end test "entries expire after the TTL only while every member is connected" do p = spawn_live() - c = Conflicts.released([{:n, p}], 0, Conflicts.new()) + c = Conflicts.record_releases([{:n, p}], 0, Conflicts.new()) - assert Conflicts.trail(Conflicts.prune(5_000, false, @ttl_config, c)) != %{} - assert Conflicts.trail(Conflicts.prune(5_000, true, @ttl_config, c)) == %{} + assert Conflicts.get_trail(Conflicts.prune(5_000, false, @ttl_config, c)) != %{} + assert Conflicts.get_trail(Conflicts.prune(5_000, true, @ttl_config, c)) == %{} end test "entries outlive a member departure by a TTL" do p = spawn_live() c = - Conflicts.released([{:n, p}], 0, Conflicts.new()) |> then(&Conflicts.departed(4_500, &1)) + Conflicts.record_releases([{:n, p}], 0, Conflicts.new()) + |> then(&Conflicts.record_departure(4_500, &1)) - assert Conflicts.trail(Conflicts.prune(5_000, true, @ttl_config, c)) != %{} - assert Conflicts.trail(Conflicts.prune(6_000, true, @ttl_config, c)) == %{} + assert Conflicts.get_trail(Conflicts.prune(5_000, true, @ttl_config, c)) != %{} + assert Conflicts.get_trail(Conflicts.prune(6_000, true, @ttl_config, c)) == %{} end end - describe "degraded/5" do + describe "is_reconstruction_degraded/5" do test "is exactly a reconstruction below the durable version key" do - assert Conflicts.degraded(9, 10, [:a], [], %{}) - refute Conflicts.degraded(10, 10, [:a], [], %{}) + assert Conflicts.is_reconstruction_degraded(9, 10, [:a], [], %{}) + refute Conflicts.is_reconstruction_degraded(10, 10, [:a], [], %{}) end test "ignores who answered the gather when the key is known" do # :b never answered and was never synced — still not degraded. - refute Conflicts.degraded(10, 10, [:a, :b], [], %{a: :snapshot}) + refute Conflicts.is_reconstruction_degraded(10, 10, [:a, :b], [], %{a: :snapshot}) end test "falls back to reachability when the key could not be read" do - assert Conflicts.degraded(10, :unknown, [:a, :b], [], %{a: :snapshot}) + assert Conflicts.is_reconstruction_degraded(10, :unknown, [:a, :b], [], %{a: :snapshot}) # A member marked fresh (never synced) holds nothing to miss. - refute Conflicts.degraded(10, :unknown, [:a, :b], [:b], %{a: :snapshot}) + refute Conflicts.is_reconstruction_degraded(10, :unknown, [:a, :b], [:b], %{a: :snapshot}) end end end diff --git a/test/dgen_registry_guarantees_test.exs b/test/dgen_registry_guarantees_test.exs index b9f742b..9f2580d 100644 --- a/test/dgen_registry_guarantees_test.exs +++ b/test/dgen_registry_guarantees_test.exs @@ -63,7 +63,7 @@ defmodule DGen.RegistryGuaranteesTest do # The member's current applied version, read through the same snapshot call the # handoff gather uses, so tests can craft contiguous (or gapped) broadcasts. defp applied_version(member) do - {_records, version, _released} = GenServer.call(member, :get_names_snapshot) + {_records, version, _released, _leader_down} = GenServer.call(member, :get_names_snapshot) version end diff --git a/test/dgen_registry_intents_test.exs b/test/dgen_registry_intents_test.exs index 4359ca5..4d25968 100644 --- a/test/dgen_registry_intents_test.exs +++ b/test/dgen_registry_intents_test.exs @@ -4,7 +4,7 @@ defmodule DGen.RegistryIntentsTest do `dgen_registry_intents` is pure: every call returns the new intents and the effects for the member to carry out. These tests drive it through its three - events (`on_reply/4`, `on_version/2`, `redrive/5`) and pin the re-drive rule: + events (`on_reply/5`, `on_version/2`, `redrive/5`) and pin the re-drive rule: re-drive what is idempotent (register, unregister-as-retract), and answer `{:error, :no_leader}` for what is not (set_meta). """ @@ -63,14 +63,15 @@ defmodule DGen.RegistryIntentsTest do end end - describe "on_reply/4 — a register's verdict" do + describe "on_reply/5 — a register's verdict" do setup %{pid: pid, from: from} do {t, effects} = Intents.submit(register(pid), from, 0, {:remote, @l1}, Intents.new()) {:ok, t: t, ref: forwarded_ref(effects)} end test "yes at or behind our version is answered", %{t: t, ref: ref, from: from} do - assert {_t, [{:reply, ^from, :yes}]} = Intents.on_reply(ref, {:register, :yes, 10}, 10, t) + assert {_t, [{:reply, ^from, :yes}]} = + Intents.on_reply(ref, {:register, :yes, 10, 1}, {1, 10}, {:remote, @l1}, t) end test "yes ahead of our version waits for the replica and asks for a resync", %{ @@ -78,24 +79,54 @@ defmodule DGen.RegistryIntentsTest do ref: ref, from: from } do - {t, effects} = Intents.on_reply(ref, {:register, :yes, 20}, 10, t) + {t, effects} = Intents.on_reply(ref, {:register, :yes, 20, 1}, {1, 10}, {:remote, @l1}, t) assert effects == [{:resync, @l1}] - assert {_t, []} = Intents.on_version(15, t) - assert {_t, [{:reply, ^from, :yes}]} = Intents.on_version(20, t) + assert {_t, []} = Intents.on_version({1, 15}, t) + assert {_t, [{:reply, ^from, :yes}]} = Intents.on_version({1, 20}, t) end - test "no is answered no", %{t: t, ref: ref, from: from} do - assert {_t, [{:reply, ^from, :no}]} = Intents.on_reply(ref, {:register, :no, 10}, 10, t) + test "a held yes is re-sent once our replica moves to a later epoch", %{ + t: t, + ref: ref, + from: from + } do + {t, _} = Intents.on_reply(ref, {:register, :yes, 20, 1}, {1, 10}, {:remote, @l1}, t) + + # A later leader's snapshot passes the version, but it may lack the commit. + assert {t, []} = Intents.on_version({2, 30}, t) + + assert {_t, [{:forward, @l2, _ref, _kind}]} = + Intents.redrive({:remote, @l2}, :moved, 0, @ttl, t) + + refute_received {:reply, ^from, _} + end + + test "a yes from an epoch we have already left is re-sent, not answered", %{t: t, ref: ref} do + assert {_t, [{:forward, @l2, _ref, _kind}]} = + Intents.on_reply(ref, {:register, :yes, 10, 1}, {2, 30}, {:remote, @l2}, t) + end + + test "a yes from an epoch we haven't reached waits for that epoch", %{ + t: t, + ref: ref, + from: from + } do + {t, [{:resync, @l1}]} = + Intents.on_reply(ref, {:register, :yes, 10, 2}, {1, 30}, {:remote, @l1}, t) + + assert {t, []} = Intents.on_version({1, 40}, t) + assert {_t, [{:reply, ^from, :yes}]} = Intents.on_version({2, 10}, t) end - test "a legacy yes binds the row itself", %{t: t, ref: ref, from: from, pid: pid} do - assert {_t, [{:reply, ^from, :yes}, {:insert_row, :name, ^pid, _}]} = - Intents.on_reply(ref, {:register, :yes, :legacy}, 0, t) + test "no is answered no", %{t: t, ref: ref, from: from} do + assert {_t, [{:reply, ^from, :no}]} = + Intents.on_reply(ref, {:register, :no, 10, 1}, {1, 10}, {:remote, @l1}, t) end test "a reply for an unknown reference is ignored", %{t: t} do - assert {^t, []} = Intents.on_reply(make_ref(), {:register, :yes, 10}, 10, t) + assert {^t, []} = + Intents.on_reply(make_ref(), {:register, :yes, 10, 1}, {1, 10}, {:remote, @l1}, t) end end @@ -112,7 +143,8 @@ defmodule DGen.RegistryIntentsTest do assert new_ref != old_ref # The deposed leader's late answer no longer finds anyone to answer. - assert {_t, []} = Intents.on_reply(old_ref, {:register, :no, 10}, 10, t) + assert {_t, []} = + Intents.on_reply(old_ref, {:register, :no, 10, 1}, {1, 10}, {:remote, @l1}, t) end test "a forward to the leader that is still current is kept on a new view", %{ @@ -125,7 +157,9 @@ defmodule DGen.RegistryIntentsTest do # A snapshot from the same leader: it received the forward (or postponed it # through its assume window), so re-sending would only commit it twice. assert {t, []} = Intents.redrive({:remote, @l1}, :moved, 0, @ttl, t) - assert {_t, [{:reply, ^from, :yes}]} = Intents.on_reply(ref, {:register, :yes, 1}, 1, t) + + assert {_t, [{:reply, ^from, :yes}]} = + Intents.on_reply(ref, {:register, :yes, 1, 1}, {1, 1}, {:remote, @l1}, t) end test "a restored link re-sends a forward to the same leader (the cast may be lost)", %{ @@ -143,7 +177,15 @@ defmodule DGen.RegistryIntentsTest do from: from } do {t, effects} = Intents.submit(register(pid), from, 0, {:remote, @l1}, Intents.new()) - {t, _} = Intents.on_reply(forwarded_ref(effects), {:register, :yes, 20}, 10, t) + + {t, _} = + Intents.on_reply( + forwarded_ref(effects), + {:register, :yes, 20, 1}, + {1, 10}, + {:remote, @l1}, + t + ) # Same leader: still waiting on our replica, nothing to send. assert {_t, []} = Intents.redrive({:remote, @l1}, :moved, 0, @ttl, t) diff --git a/test/dgen_registry_pipeline_test.exs b/test/dgen_registry_pipeline_test.exs index 2186ee5..3d2949b 100644 --- a/test/dgen_registry_pipeline_test.exs +++ b/test/dgen_registry_pipeline_test.exs @@ -24,31 +24,31 @@ defmodule DGen.RegistryPipelineTest do {batch, p} = Pipeline.take_batch(2, p) assert batch == [{:retract, 1, pid}, {:retract, 2, pid}] - assert Pipeline.num_pending(p) == 1 + assert Pipeline.get_num_pending(p) == 1 ref = make_ref() mref = make_ref() - p = Pipeline.start(ref, mref, %{ops: batch}, p) - assert Pipeline.in_flight(p) + p = Pipeline.start_commit(ref, mref, %{ops: batch}, p) + assert Pipeline.is_in_flight(p) - assert :stale = Pipeline.finish(make_ref(), p) - assert {:ok, %{ops: ^batch}, ^mref, p} = Pipeline.finish(ref, p) - refute Pipeline.in_flight(p) + assert :stale = Pipeline.finish_commit(make_ref(), p) + assert {:ok, %{ops: ^batch}, ^mref, p} = Pipeline.finish_commit(ref, p) + refute Pipeline.is_in_flight(p) end test "a dead worker releases the slot too", %{pid: pid} do mref = make_ref() - p = Pipeline.start(make_ref(), mref, %{ops: [{:retract, :n, pid}]}, Pipeline.new()) + p = Pipeline.start_commit(make_ref(), mref, %{ops: [{:retract, :n, pid}]}, Pipeline.new()) - assert :stale = Pipeline.worker_down(make_ref(), p) - assert {:ok, %{ops: [_]}, p} = Pipeline.worker_down(mref, p) - refute Pipeline.in_flight(p) + assert :stale = Pipeline.handle_worker_down(make_ref(), p) + assert {:ok, %{ops: [_]}, p} = Pipeline.handle_worker_down(mref, p) + refute Pipeline.is_in_flight(p) end test "the cached read version is dropped at the end of a term" do - p = Pipeline.committed(42, Pipeline.new()) - assert Pipeline.read_version(p) == 42 - assert Pipeline.read_version(Pipeline.new_term(p)) == :undefined + p = Pipeline.set_last_version(42, Pipeline.new()) + assert Pipeline.get_read_version(p) == 42 + assert Pipeline.get_read_version(Pipeline.start_new_term(p)) == :undefined end end @@ -59,18 +59,18 @@ defmodule DGen.RegistryPipelineTest do timer = make_ref() p = Pipeline.await_acks(batch, direct, timer, 2, Pipeline.new()) - assert {:waiting, p} = Pipeline.ack(batch, @follower, p) - assert {:waiting, p} = Pipeline.ack(batch, @follower, p) - assert {:done, ^direct, ^timer, p} = Pipeline.ack(batch, {:n3@host, :reg}, p) - assert :none = Pipeline.ack_timeout(batch, p) + assert {:waiting, p} = Pipeline.record_ack(batch, @follower, p) + assert {:waiting, p} = Pipeline.record_ack(batch, @follower, p) + assert {:done, ^direct, ^timer, p} = Pipeline.record_ack(batch, {:n3@host, :reg}, p) + assert :none = Pipeline.handle_ack_timeout(batch, p) end test "a timeout reports how many confirmed", %{pid: pid, from: from} do batch = make_ref() p = Pipeline.await_acks(batch, [{{:local, from}, :n, pid}], make_ref(), 2, Pipeline.new()) - {:waiting, p} = Pipeline.ack(batch, @follower, p) + {:waiting, p} = Pipeline.record_ack(batch, @follower, p) - assert {:expired, [_], 2, 1, _p} = Pipeline.ack_timeout(batch, p) + assert {:expired, [_], 2, 1, _p} = Pipeline.handle_ack_timeout(batch, p) end test "losing leadership takes every batch, in arrival order", %{pid: pid, from: from} do diff --git a/test/dgen_registry_replica_test.exs b/test/dgen_registry_replica_test.exs index 4249e2f..e30776c 100644 --- a/test/dgen_registry_replica_test.exs +++ b/test/dgen_registry_replica_test.exs @@ -22,7 +22,7 @@ defmodule DGen.RegistryReplicaTest do defp reg(name, pid, index \\ %{}), do: {:name_registered, name, pid, index, nil} defp unreg(name, pid), do: {:name_unregistered, name, pid} - defp pids(r), do: Map.new(Replica.records(r), fn {n, {p, _, _}} -> {n, p} end) + defp pids(r), do: Map.new(Replica.get_records(r), fn {n, {p, _, _}} -> {n, p} end) setup do {:ok, @@ -34,7 +34,7 @@ defmodule DGen.RegistryReplicaTest do r = replica(1, 10) assert {:applied, r} = Replica.offer_batch(1, 10, 20, [reg(:a, p1), reg(:b, p2)], r) - assert Replica.version(r) == 20 + assert Replica.get_version(r) == 20 assert pids(r) == %{a: p1, b: p2} end @@ -45,19 +45,21 @@ defmodule DGen.RegistryReplicaTest do assert pids(r) == %{} end - test "a newer epoch's contiguous batch applies but does not raise the epoch", %{p1: p1} do + test "a newer epoch's batch is a gap, even when contiguous", %{p1: p1} do + # Only the new leader's snapshot moves us into its epoch. Applying its batch + # would let a held `yes` from our epoch be released by the new leader's + # versions. r = replica(1, 10) - assert {:applied, r} = Replica.offer_batch(2, 10, 20, [reg(:a, p1)], r) - assert Replica.epoch(r) == 1 - assert Replica.version(r) == 20 + assert :gap = Replica.offer_batch(2, 10, 20, [reg(:a, p1)], r) + assert :gap = Replica.offer_batch(2, 10, 10, [], r) end test "the heartbeat (an empty batch at {V, V}) is a no-op on a caught-up replica" do r = replica(1, 10) assert {:applied, r} = Replica.offer_batch(1, 10, 10, [], r) - assert Replica.version(r) == 10 + assert Replica.get_version(r) == 10 end test "metadata changes keep the inverted index in step", %{p1: p1} do @@ -92,7 +94,7 @@ defmodule DGen.RegistryReplicaTest do assert :gap = Replica.offer_batch(1, 15, 20, [reg(:a, p1)], r) assert pids(r) == %{} - assert Replica.version(r) == 10 + assert Replica.get_version(r) == 10 end test "a heartbeat ahead of us reveals a lost tail" do @@ -108,14 +110,14 @@ defmodule DGen.RegistryReplicaTest do assert {:accepted, r} = Replica.offer_snapshot(1, 20, %{b: {p2, %{}, nil}}, r) assert pids(r) == %{b: p2} - assert Replica.version(r) == 20 + assert Replica.get_version(r) == 20 end test "a newer epoch's snapshot sets the epoch", %{p1: p1} do r = replica(1, 10) assert {:accepted, r} = Replica.offer_snapshot(3, 10, %{a: {p1, %{}, nil}}, r) - assert Replica.epoch(r) == 3 + assert Replica.get_epoch(r) == 3 end test "a snapshot behind our version is refused — no rollback", %{p1: p1, p2: p2} do @@ -143,7 +145,7 @@ defmodule DGen.RegistryReplicaTest do assert pids(r) == %{a: p2, c: p2} assert [] = Replica.query(r, {:all, %{k: 1}}) assert [%{name: :a}] = Replica.query(r, {:all, %{k: 2}}) - assert {Replica.epoch(r), Replica.version(r)} == {2, 20} + assert {Replica.get_epoch(r), Replica.get_version(r)} == {2, 20} end end @@ -153,7 +155,7 @@ defmodule DGen.RegistryReplicaTest do r = Replica.commit(20, %{a: :clear}, %{}, %{}, r) assert pids(r) == %{b: p2} - assert Replica.version(r) == 20 + assert Replica.get_version(r) == 20 end test "a metadata-only op never resurrects an optimistically deleted row", %{p1: p1} do diff --git a/test/dgen_registry_sim_test.exs b/test/dgen_registry_sim_test.exs index 16f0d5c..28a9b24 100644 --- a/test/dgen_registry_sim_test.exs +++ b/test/dgen_registry_sim_test.exs @@ -19,7 +19,7 @@ defmodule DGen.RegistrySimTest do run is reproducible at the fault level rather than bit-for-bit. That is enough to re-run a seed and usually see the same failure, and enough to read the trace and understand it — it is not enough to guarantee it. Getting the rest requires - virtualising the backend and the commit worker; see `test/support/sim/README.md`. + virtualising the backend and the commit worker; see `test/support/sim/dgen_registry_simulation.md`. Only inter-member protocol traffic is faulted. The elector's durable membership/election queue is untouched, so elections still make progress and the @@ -662,8 +662,8 @@ defmodule DGen.RegistrySimTest do # a version discontinuity that the existing resync repairs. When it was one # message per name a member could hold a strict subset while reporting the # batch's version — undetectable, because gap detection compares versions and the - # versions matched. Finding 1 in README.md has the mechanism; these tests pin the - # fix from both directions. + # versions matched. Finding 1 in dgen_registry_simulation.md has the mechanism; these + # tests pin the fix from both directions. # --------------------------------------------------------------------------- describe "batch atomicity" do @@ -1034,7 +1034,7 @@ defmodule DGen.RegistrySimTest do describe "membership join" do @tag timeout: 120_000 test "a member joining mid-workload under loss onboards fully", %{tenant: tenant} do - # The continuing-leader join fast path (`onboard_joiner`/`{peer_joined}`) + # The continuing-leader join fast path (`cast_data_to_new_member`/`{peer_joined}`) # is explicitly outside the TLA+ model's scope, and until this test it was # exercised only by the `:cluster` suite (excluded on `dgen_mem`) — so a # regression here was visible to no formal layer at all. The join runs diff --git a/test/dgen_registry_test.exs b/test/dgen_registry_test.exs index 7750b50..e34a7e7 100644 --- a/test/dgen_registry_test.exs +++ b/test/dgen_registry_test.exs @@ -695,11 +695,11 @@ defmodule DGen.RegistryTest do # Snapshot codec (pure, unit-tested in isolation) # # encode/1 folds the replica's ETS table straight into a pair list inside the blob - # (no records-map materialization); decode_records/1 must normalize every wire - # shape — this build's list-in-binary and map-in-binary, plus both pre-binary - # legacy forms — to the one map callers use. A shape the decoder silently - # mishandled would surface as a corrupt replica after a resync or handoff, so - # the tolerance is pinned here rather than trusted. + # (no records-map materialization); decode_records/1 must normalize both blob + # shapes — list-in-binary (encode/1) and map-in-binary (encode_records/1) — to the + # one map callers use. A shape the decoder silently mishandled would surface as a + # corrupt replica after a resync or handoff, so the tolerance is pinned here + # rather than trusted. # --------------------------------------------------------------------------- describe "snapshot codec (dgen_registry_replica encode/1, decode_records/1)" do @@ -732,24 +732,6 @@ defmodule DGen.RegistryTest do :dgen_registry_replica.encode_records(records) ) end - - test "legacy wire shapes still normalize (rolling upgrade)" do - p1 = spawn_live() - - # Pre-binary-era apply_names_snapshot: an on-heap [{name, record}] list. - assert %{a: {^p1, %{}, nil}} = - :dgen_registry_replica.decode_records([{:a, {p1, %{}, nil}}]) - - # Pre-binary-era get_names_snapshot: a bare records map. - assert %{a: {^p1, %{}, nil}} = - :dgen_registry_replica.decode_records(%{a: {p1, %{}, nil}}) - - # Pre-list-era binary: a map inside the blob (what an old node still sends). - assert %{a: {^p1, %{}, nil}} = - :dgen_registry_replica.decode_records( - :erlang.term_to_binary(%{a: {p1, %{}, nil}}, [:compressed]) - ) - end end # --------------------------------------------------------------------------- @@ -758,7 +740,7 @@ defmodule DGen.RegistryTest do # A group commit's plan is built *before* anything lands in the member's ETS # table, so an op must still see an earlier op's not-yet-committed decision if # both touch the same name within one batch. plan_batch resolves this lazily - # (seed_lookup/3: the batch's own overlay first, a point ETS lookup otherwise) + # (lookup_in_batch/3: the batch's own overlay first, a point ETS lookup otherwise) # rather than needing a full pre-seeded map of the registry. These tests pin # that overlay directly — in particular the `removed` marker it uses to mean # "an earlier op in this batch explicitly cleared this name", which matters @@ -1033,10 +1015,15 @@ defmodule DGen.RegistryTest do # The member's current applied version, read through the same snapshot call the # handoff gather uses, so tests can craft contiguous (or gapped) broadcasts. defp applied_version(member) do - {_records, version, _released} = GenServer.call(member, :get_names_snapshot) + {_records, version, _released, _leader_down} = GenServer.call(member, :get_names_snapshot) version end + defp leader_down?(member) do + {_records, _version, _released, leader_down} = GenServer.call(member, :get_names_snapshot) + leader_down + end + # A replication broadcast in the shape the leader actually sends: one # `{:names_batch, Ops, Epoch, PrevV, Version, LeaderId}` message per committed # batch, with the batch's ops carried inside it. A batch is delivered whole or @@ -1207,7 +1194,8 @@ defmodule DGen.RegistryTest do # behind what the member has already applied. GenServer.cast( member, - {:apply_names_snapshot, %{}, leader, [], %{}, epoch + 1, v - 1} + {:apply_names_snapshot, :dgen_registry_replica.encode_records(%{}), leader, [], %{}, + epoch + 1, v - 1} ) # Barrier: ensure the member has processed (and rejected) the stale snapshot. @@ -1235,7 +1223,9 @@ defmodule DGen.RegistryTest do # guard does not block a forward snapshot. GenServer.cast( member, - {:apply_names_snapshot, %{new_name: {new_pid, %{}, %{}}}, leader, [], %{}, epoch, v + 1} + {:apply_names_snapshot, + :dgen_registry_replica.encode_records(%{new_name: {new_pid, %{}, %{}}}), leader, [], %{}, + epoch, v + 1} ) :sys.get_state(member) @@ -1295,7 +1285,12 @@ defmodule DGen.RegistryTest do end defp follow(member, leader, epoch, version) do - GenServer.cast(member, {:apply_names_snapshot, %{}, leader, [], %{}, epoch, version}) + GenServer.cast( + member, + {:apply_names_snapshot, :dgen_registry_replica.encode_records(%{}), leader, [], %{}, + epoch, version} + ) + :sys.get_state(member) end @@ -1336,8 +1331,8 @@ defmodule DGen.RegistryTest do assert Task.yield(caller, 100) == nil # Leader 1's late `no` finds no one to answer; leader 2's `yes` decides it. - GenServer.cast(member, {:register_reply, ref1, :no, v}) - GenServer.cast(member, {:register_reply, ref2, :yes, v}) + GenServer.cast(member, {:register_reply, ref1, :no, v, epoch + 1}) + GenServer.cast(member, {:register_reply, ref2, :yes, v, epoch + 2}) assert :yes == Task.await(caller) end @@ -1359,7 +1354,45 @@ defmodule DGen.RegistryTest do follow(member, l1, epoch + 1, v) refute_receive {:"$gen_cast", {:register_req, _, _, _, _, _}}, 100 - GenServer.cast(member, {:register_reply, ref1, :yes, v}) + GenServer.cast(member, {:register_reply, ref1, :yes, v, epoch + 1}) + assert :yes == Task.await(caller) + end + + test "a held yes is not answered by a later leader that lacks its commit", %{reg: reg} do + member = :dgen_registry.member_name(reg) + pid = spawn(fn -> Process.sleep(:infinity) end) + on_exit(fn -> Process.exit(pid, :kill) end) + + epoch = :dgen_registry.get_epoch(reg) + v = applied_version(member) + l1 = fake_leader() + follow(member, l1, epoch + 1, v) + + caller = Task.async(fn -> :dgen_registry.register_name({reg, :lost}, pid) end) + {ref1, :lost, ^pid} = await_forward() + + # Leader 1 commits at v + 1 and answers, but dies before its batch reaches us. + # The member holds the `yes` and asks leader 1 for a resync. + GenServer.cast(member, {:register_reply, ref1, :yes, v + 1, epoch + 1}) + assert_receive {:"$gen_cast", {:resync_req, _}} + Process.unregister(elem(l1, 1)) + + # Leader 2 took over at v without that commit, and its snapshot to us was lost. + # Its next batch starts at our version, but it comes from a later epoch, so it + # is a gap and must not release the held `yes`. + l2 = fake_leader() + GenServer.cast(member, names_batch([], epoch + 2, v, v + 2, l2)) + :sys.get_state(member) + assert Task.yield(caller, 100) == nil, "the held yes was answered by leader 2's batch" + + # Leader 2's resync snapshot doesn't hold the name, so the register is re-sent. + follow(member, l2, epoch + 2, v + 2) + {ref2, :lost, ^pid} = await_forward() + assert ref2 != ref1 + assert Task.yield(caller, 0) == nil + + GenServer.cast(member, {:register_reply, ref2, :yes, v + 3, epoch + 2}) + GenServer.cast(member, names_batch([], epoch + 2, v + 2, v + 3, l2)) assert :yes == Task.await(caller) end @@ -1380,7 +1413,7 @@ defmodule DGen.RegistryTest do :sys.get_state(member) refute_receive {:"$gen_cast", {:register_reply, ^ref, _}}, 200 - refute_receive {:"$gen_cast", {:register_reply, ^ref, _, _}} + refute_receive {:"$gen_cast", {:register_reply, ^ref, _, _, _}} end end @@ -1462,8 +1495,9 @@ defmodule DGen.RegistryTest do GenServer.cast( member, - {:apply_names_snapshot, %{held: {held, %{}, :undefined}}, {node(), winner}, [], %{}, - epoch + 2, v} + {:apply_names_snapshot, + :dgen_registry_replica.encode_records(%{held: {held, %{}, :undefined}}), + {node(), winner}, [], %{}, epoch + 2, v} ) assert_receive {:"$gen_cast", {:retract_req, :held, ^held}}, @@ -1479,7 +1513,7 @@ defmodule DGen.RegistryTest do "the in-flight registration was not re-sent to the new leader" assert Task.yield(register, 0) == nil, "the in-flight registration was answered early" - GenServer.cast(member, {:register_reply, ref, :yes, v}) + GenServer.cast(member, {:register_reply, ref, :yes, v, epoch + 2}) assert :yes == Task.await(register) end end @@ -1513,10 +1547,19 @@ defmodule DGen.RegistryTest do on_exit(fn -> Process.exit(fake, :kill) end) fake_name = :"degr_fake_#{:erlang.unique_integer([:positive])}" Process.register(fake, fake_name) - GenServer.cast(member, {:apply_names_snapshot, %{}, {node(), fake_name}, [], %{}, epoch, v}) + + # The member monitors the fake leader, as a follower monitors a real one. + fake_id = {node(), fake_name} + + GenServer.cast( + member, + {:apply_names_snapshot, :dgen_registry_replica.encode_records(%{}), fake_id, [fake_id], + %{}, epoch, v} + ) + {:follower, _} = :sys.get_state(member) - %{reg: reg, member: member, self_id: self_id, epoch: epoch} + %{reg: reg, member: member, self_id: self_id, epoch: epoch, fake: fake} end defp reassume(member, self_id, all_ids, epoch) do @@ -1567,7 +1610,36 @@ defmodule DGen.RegistryTest do assert_receive {:dgen_transaction, ^ref, {:committed, _}}, 5_000 # The gather retries toward the key until ?GATHER_TIMEOUT, then assumes degraded. - reassume(member, self_id, [self_id], epoch) + {elapsed_us, _} = :timer.tc(fn -> reassume(member, self_id, [self_id], epoch) end) + # The fake leader is alive, so a late broadcast could still come. We wait it out. + assert elapsed_us >= 2_000_000 + + pid = spawn(fn -> Process.sleep(:infinity) end) + on_exit(fn -> Process.exit(pid, :kill) end) + assert :no == :dgen_registry.register_name({reg, :new_name}, pid) + end + + test "a leader that died before broadcasting its last commit doesn't stall the handoff", + %{reg: reg, member: member, self_id: self_id, epoch: epoch, tenant: tenant, fake: fake} do + # The same missing commit, but now its leader is dead and we have handled its + # DOWN. Nothing more can arrive from it, so the gather stops waiting at once. + tuid = {"dgen_registry.", Atom.to_string(reg)} + ref = make_ref() + + {:ok, _} = + :dgen_registry_names.start_commit(tenant, tuid, self_id, %{ + owner: self(), + ref: ref, + epoch: epoch + }) + + assert_receive {:dgen_transaction, ^ref, {:committed, _}}, 5_000 + + Process.exit(fake, :kill) + assert eventually(fn -> leader_down?(member) end, 5_000) + + {elapsed_us, _} = :timer.tc(fn -> reassume(member, self_id, [self_id], epoch) end) + assert elapsed_us < 1_000_000 pid = spawn(fn -> Process.sleep(:infinity) end) on_exit(fn -> Process.exit(pid, :kill) end) diff --git a/test/support/sim/cluster.ex b/test/support/sim/cluster.ex index d9c207a..591b0f1 100644 --- a/test/support/sim/cluster.ex +++ b/test/support/sim/cluster.ex @@ -33,7 +33,7 @@ defmodule DGen.Sim.Cluster do table: `nodes()` does not list it and `dgen_utils:node_reachable/1` therefore answers `true` for every member id, since every member id's node component is the one real `node()`. So the *member*-level reactions to a node event are - exercised here (`handle_info({nodeup, _})`'s rejoin and unregister re-drive, + exercised here (the `{nodeup, _}` handler's rejoin and unregister re-drive, the peer-monitor DOWN that drives `{member_down}`), and the connector's reachability-keyed backstops — the `{nodedown, Node}` reap and the leader probe — are not: they filter member ids by node and no member id names a @@ -374,8 +374,8 @@ defmodule DGen.Sim.Cluster do published |> Map.delete(:replica) |> Map.merge(%{ - epoch: :dgen_registry_replica.epoch(replica), - applied_version: :dgen_registry_replica.version(replica) + epoch: :dgen_registry_replica.get_epoch(replica), + applied_version: :dgen_registry_replica.get_version(replica) }) other -> @@ -582,7 +582,7 @@ defmodule DGen.Sim.Cluster do which no other fault or lifecycle helper reaches. Returns `{cluster, name}`. On a healthy cluster this exercises the continuing-leader **fast path** - (`onboard_joiner` / `{peer_joined}` in `dgen_registry_member.erl`): the + (`cast_data_to_new_member` / `{peer_joined}` in `dgen_registry_member.erl`): the leader snapshots only the joiner and tells existing followers to just monitor it — no gather, no re-assume, epoch unchanged. That path is explicitly OUT of the TLA+ model's scope (its handoff premise is "the gather @@ -711,7 +711,7 @@ defmodule DGen.Sim.Cluster do keeps a timeout here meaning "did not converge *despite* traffic". (That quiescent-divergence window is itself worth knowing about, and is written - up in `test/support/sim/README.md`.) + up in `test/support/sim/dgen_registry_simulation.md`.) """ def converge(%__MODULE__{} = c, timeout \\ 15_000) do :ok = heal_everything(c.net) diff --git a/test/support/sim/README.md b/test/support/sim/dgen_registry_simulation.md similarity index 98% rename from test/support/sim/README.md rename to test/support/sim/dgen_registry_simulation.md index 5183a66..51bdf29 100644 --- a/test/support/sim/README.md +++ b/test/support/sim/dgen_registry_simulation.md @@ -95,7 +95,7 @@ spec's `DropMsg`. Injecting reordering would only produce false positives. **A lost message comes with a link signal.** In real distribution, messages are lost because a link failed, and the failure sends `nodedown`/`nodeup` to both -ends. The registry relies on those signals: `handle_info({nodeup, _})` re-joins +ends. The registry relies on those signals: the member's `{nodeup, _}` handler re-joins and re-drives stashed unregisters. So `Cluster.converge/2` heals through `eta_net:heal_partition/3` with a `nodeup` signal. A fault model that leaves out these signals produces failures that look like product bugs but aren't. @@ -169,7 +169,7 @@ number. helper. 4. **`dgen` and `dgen_registry` weren't built with the eta transform.** Their `gen_server:call`s escaped the scheduler. This was a simulation-only gap. -5. **`telemetry_available/0` called `code_server` from inside a member.** It is +5. **`is_telemetry_available/0` called `code_server` from inside a member.** It is now resolved in `init/1`. 6. **`set_metadata/2` and `unregister_name/1` ignored `register_timeout`.** 7. **One unregister freed a name twice.** An optimistic delete plus an diff --git a/test/support/sim/invariants.ex b/test/support/sim/invariants.ex index 7ebdbd2..e2060a0 100644 --- a/test/support/sim/invariants.ex +++ b/test/support/sim/invariants.ex @@ -79,9 +79,8 @@ defmodule DGen.Sim.Invariants do first. It does not hold continuously, and not because of a bug: members write their replica *optimistically*, outside the replicated stream, without advancing `applied_version`. `route_unregister/3` deletes the row locally the moment the - caller asks, before the leader has committed anything, and a follower's - `handle_register_reply/4` inserts the row when the leader's `yes` arrives. Both - leave that member holding different content from its peers *at the same version* + caller asks, before the leader has committed anything. That leaves the member + holding different content from its peers *at the same version* for as long as the batch takes to come back around. So a mid-flight difference here is ordinary speculation, not divergence. What diff --git a/test/support/sim/registry_harness.ex b/test/support/sim/registry_harness.ex index 4ca608d..94d496d 100644 --- a/test/support/sim/registry_harness.ex +++ b/test/support/sim/registry_harness.ex @@ -63,7 +63,7 @@ defmodule DGen.Sim.RegistryHarness do machine; the last one only showed up with several simulation suites running at once, because what it turned on was how much real time a run took. - 1. **`dgen_registry_member:telemetry_available/0`.** `code:ensure_loaded/1` + 1. **`dgen_registry_member:is_telemetry_available/0`.** `code:ensure_loaded/1` short-circuits on a loaded module but not on a missing one, so the first telemetry event in a VM was a synchronous call into `code_server` — a process the scheduler does not own — from inside a member. Resolved in `init/1` now,