diff --git a/.github/workflows/formal.yml b/.github/workflows/formal.yml index 7803703..f08b669 100644 --- a/.github/workflows/formal.yml +++ b/.github/workflows/formal.yml @@ -47,6 +47,12 @@ jobs: # satisfiable by a protocol that never acks anyone. - cfg: DgenRegistryReplicationAckReachable expect: fail + # The gather's deadline modeled: the version-defined `degraded` flag is + # sound (DegradedFlagSound), and its canary proves the path is reached. + - cfg: DgenRegistryReplicationDegradedAssume + expect: pass + - cfg: DgenRegistryReplicationDegradedReachable + expect: fail steps: - uses: actions/checkout@v3 - uses: actions/setup-java@v4 diff --git a/CHANGELOG.md b/CHANGELOG.md index 43e0728..f0c82a1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,24 @@ # Changelog +## v0.4.3 (TBD) + +### Bug fixes + +- When a follower forwards a registration and the leader changes at the same + time, the registration is now re-driven to the new leader rather than answered + with `no`. +- A buffered unregister is no longer lost when the leader changes mid-commit. +- A partition heal no longer stalls the leader on conflict-detection liveness + probes; they run off its loop. +- `whereis_name/1` and `get_metadata/1` no longer briefly read a bound name as + `undefined` while a member re-baselines its replica (a resync or handoff). + +### Enhancements + +- `dgen_registery_member`'s internal implementation is refactored into + `dgen_registry_replica`, `dgen_registry_intents`, `dgen_registry_pipeline`, + and `dgen_registry_conflicts`. + ## v0.4.2 (2026-10-04) ### Enahancements diff --git a/README.md b/README.md index 532bc16..1902773 100644 --- a/README.md +++ b/README.md @@ -20,13 +20,12 @@ in theory, be substituted, but for now FDB is the only implemented backend. - **`dgen_server`** — the `gen_server` programming model (same callbacks, same return shapes), backed by durable state and a durable message queue instead of process memory. Any number of processes, on any number of nodes, can - consume from the same server concurrently. See the - [dgen_server design doc](docs/design/dgen_server_design.md). + consume from the same server concurrently, and via strict serializability + of FoundationDB, `gen_server` semantics are preserved. - **`dgen_registry`** — an OTP-compatible process registry: give a running process a logical name and find or message it by name from anywhere in the cluster, via the standard `{via, dgen_registry, {RegistryName, LogicalName}}` - contract. See the - [dgen_registry design doc](docs/design/dgen_registry_design.md). + contract. Both are built on the same idea: push state and coordination into a strongly-consistent backend, so the OTP programming model you already know @@ -74,17 +73,15 @@ end configuration. - [formal/README.md](formal/README.md) — the TLA+ model of `dgen_registry`'s replication protocol. -- [`eta`](https://hexdocs.pm/eta) — `dgen_registry` is partally - tested with Deterministic Simulation Testing, to assist in verification - of the actual implementation of the TLA+ model, including its invariants. - The framework for driving the simulation was extracted to a separate - library for general use. +- [test/support/sim/README.md](test/support/sim/README.md) — deterministic + simulation testing of `dgen_registry` against the same invariants, driven by + [`eta`](https://hexdocs.pm/eta). - API reference and getting-started guides: . - [CHANGELOG.md](CHANGELOG.md) — release notes. ## AI full disclosure - - This software is developed with strong assistance from LLMs and with humans + - This software is developed with strong assistance from LLMs and with humans leading the ideas, testing, and debugging. We say this openly because it shaped how the project was built. If you are not happy with AI-developed code, this software is not for you. This disclosure was adopted from [antirez/ds4](https://github.com/antirez/ds4). diff --git a/docs/design/dgen_registry_design.md b/docs/design/dgen_registry_design.md index abbf9af..f491c63 100644 --- a/docs/design/dgen_registry_design.md +++ b/docs/design/dgen_registry_design.md @@ -1,29 +1,21 @@ # dgen_registry Design -`dgen_registry` is an OTP-compatible **process registry**: it lets you give a -running process a logical name and then find or message that process by name from -anywhere in an Erlang cluster. It implements the standard -`{via, dgen_registry, {RegistryName, LogicalName}}` contract, so ordinary OTP -behaviours (`gen_server`, `gen_statem`, `gen_event`, …) can be registered and -addressed by name with no special API. - -Its defining choice is that it leans on a **strongly-consistent database** for -coordination. The default and reference backend is FoundationDB. That single -decision is what gives `dgen_registry` its particular personality, summarised up -front so you can decide quickly whether it fits: - -- **It is a CP system.** When the network splits, the side that cannot reach the - database stops accepting registrations rather than risk handing the same name to - two processes. It chooses consistency over availability. -- **It is a *singleton* registry.** A logical name maps to at most one live process - cluster-wide, and the registry will, if it must, **terminate** a process to keep - that true. Register only processes that can withstand being forcibly killed — for - example, supervised and restart-safe, or transient by design. -- **It stores almost nothing in the database.** Process identifiers (pids) are never - written to durable storage. A registry of a million processes keeps roughly two - keys for its name table — independent of how many names are registered. (The - membership/leader-election machinery underneath keeps its own small set of keys, - also independent of the name count; see §4.4.) +`dgen_registry` is an OTP-compatible **process registry**: give a running process a +logical name, then find or message it by name from anywhere in an Erlang cluster. It +implements the standard `{via, dgen_registry, {RegistryName, LogicalName}}` contract, +so `gen_server`, `gen_statem`, `gen_event` and friends can be registered and addressed +by name with no special API. + +It coordinates through a **strongly-consistent database** (FoundationDB is the default +and reference backend). That choice defines it: + +- **It is CP.** The side of a partition that cannot reach the database stops accepting + registrations rather than risk giving one name to two processes (§5.3). +- **It is a *singleton* registry.** A name maps to at most one live process + cluster-wide, and the registry will **terminate** processes to keep that true, so + register only processes that can withstand being killed (§5.6). +- **It stores almost nothing durably.** Pids are never written to the database; the + durable footprint does not grow with the number of registered names (§4.4). This document is the source of truth for what the registry does. The [Guarantees](#6-guarantees) and @@ -31,34 +23,17 @@ This document is the source of truth for what the registry does. The behaviour that contradicts a guarantee is a bug, and behaviour listed as a non-goal is intentional. -It is written to be read top to bottom. The early sections assume only that you -know what an Erlang process and a `gen_server` are; the later sections get more -precise about distributed-systems behaviour. - **This document is LLM generated.** ---- - ## 1. The one-minute model -Picture a cluster of Erlang nodes that all want to agree on a shared phone book: -"the name `user_service` belongs to *this* process." `dgen_registry` keeps that -phone book consistent. - -- On each node there is a small **member** process that holds a local copy of the - phone book (a `name → pid` map). -- One member is the **leader**. All changes — registering a name, unregistering it, - cleaning up after a process dies — go through the leader, which applies them in - order and then tells the other members. -- The leader is chosen using the database, so every node agrees on who the leader - is, and a stale leader is automatically prevented from making changes. -- Looking up a name is usually a local, lock-free ETS read that runs in the calling - process — fast, with no network hop and no round-trip to the member — and there is - also a slower, always-authoritative lookup when you need it. - -Everything below is an elaboration of that picture. - ---- +- Each node runs a **member** process holding a local copy of the `name → pid` table. +- One member is the **leader**. Every change — register, unregister, cleanup after a + process dies — goes through it; it applies changes in order and tells the others. +- The leader is chosen through the database, so every node agrees who leads, and a + stale leader cannot make changes. +- Lookups are usually a local, lock-free ETS read; an authoritative lookup is available + when you need it. ## 2. Quick start @@ -75,10 +50,8 @@ Everything below is an elaboration of that picture. gen_server:call({via, dgen_registry, {my_registry, user_service}}, ping). ``` -Run `dgen_registry:start_link/2` on **every** node that should participate. Each -call starts the local machinery and joins the node to the named registry. The -`Tenant` identifies the backend database and keyspace the registry coordinates -through; every node must point at the same one. +Run `dgen_registry:start_link/2` on **every** participating node, all pointing at the +same `Tenant` (the backend database and keyspace). The direct API mirrors the via-tuple: @@ -88,8 +61,8 @@ dgen_registry:whereis_name({my_registry, user_service}), %% -> pid() | u dgen_registry:unregister_name({my_registry, user_service}). %% -> ok ``` -A registration can also carry **metadata** — attributes that live exactly as long as -the registration and can be searched: +A registration can carry searchable **metadata** that lives exactly as long as the +registration ([§4.7](#47-metadata-and-queries)): ```erlang dgen_registry:register_name({my_registry, user_service}, Pid, @@ -98,258 +71,146 @@ dgen_registry:get_metadata({my_registry, user_service}), %% -> {ok, #{pid, ind dgen_registry:query(my_registry, #{region => us_east}). %% -> [#{name, pid, index, data}, ...] ``` -See [§4.7 Metadata and queries](#47-metadata-and-queries). - ---- - ## 3. Core concepts -**Logical names.** A name is any Erlang term, scoped to a registry. `{orders, 42}`, -`"worker-7"`, and `user_service` are all valid names. Names live only in memory and -mean nothing across a full cluster restart (see [Non-goals](#7-non-goals-and-explicit-non-guarantees)). +**Logical names.** Any Erlang term, scoped to a registry (`{orders, 42}`, `"worker-7"`, +`user_service`). Names live only in memory and do not survive a full cluster restart +(Non-goal 1). -**Registries are independent namespaces.** You can run many registries at once -(`orders`, `sessions`, …). Each has its own leader, its own membership, its own -name table, and its own slice of the database. They share nothing. +**Registries are independent namespaces.** Each registry (`orders`, `sessions`, …) has +its own leader, membership, name table and database keys. They share nothing. -**Two kinds of read.** Looking a name up comes in two flavours, and choosing -between them is the main thing an application author decides: +**Two kinds of read.** -| Function | Where it reads from | Speed | Freshness | +| Function | Reads from | Cost | Freshness | |---|---|---|---| -| `whereis_name/1` | the local member's ETS table, read in the calling process | fast, no network hop, no member round-trip | *snapshot* — may briefly lag a recent change made on another node | -| `whereis_name_consistent/1` | the current leader, with the answer **fence-verified** against the database | one network hop plus a database check | *authoritative* — reflects the leader's committed view, and a deposed leader cannot serve it (§5.1) | - -`whereis_name/1` is what OTP's routing uses internally, so a `gen_server:call` -through a via-tuple takes the fast path: it does a lock-free `ets:lookup/2` against -the local member's `protected` names table directly in the caller, so many processes -can resolve names concurrently without ever queuing behind the member's mailbox. - -**Registration blocks until decided; `no` is always adjudicated.** -`register_name/2` answers `yes` if the name is now yours, or `no` **only** for an -*adjudicated* refusal — the name is held by a different live process (or the opt-in -`reject_when_degraded` prevention mode, §8). It never returns `no` merely because the -registry is not ready: when there is no reachable leader (none elected yet at startup, -a brief no-leader window during a handoff, or the leader is unreachable in a -partition) the call **blocks** and is re-driven the moment a leader is established, so -it succeeds if one appears within `register_timeout`. Returning `no` there would tell -OTP's via machinery "already started" for a name that is not taken — a wrong decisive -answer. - -If no leader appears in time, the call **exits** on its `register_timeout` (§8, -default 5 s) rather than inventing a verdict — the same posture `global`, `syn`, and -`gproc`'s distributed mode take; the supervisor restarts the caller with fresh state. -One consequence worth knowing: after such a timeout (or a `no` raced with a leader -failover) the name *may* be durably bound to the pid you tried to register — a retry -then answers `yes` (re-registering the same pid under the same name is idempotent, -§5.2), and if the caller instead abandons the pid, its exit clears the binding -automatically. - -**Automatic cleanup.** The leader watches every registered process with -`erlang:monitor/2`. When a registered process exits, its name is unregistered -automatically — you do not need to call `unregister_name/1` yourself. - -**Metadata rides along with a registration.** Beyond a bare pid, a registration can -carry an opaque data payload and a set of indexed attributes searchable with a query. -Both live exactly as long as the registration — see -[§4.7 Metadata and queries](#47-metadata-and-queries). - ---- +| `whereis_name/1` | the local member's `protected` ETS table, read in the calling process | no network hop, no mailbox | *snapshot* — may briefly lag a change made on another node | +| `whereis_name_consistent/1` | the current leader, fence-verified against the database (§5.1) | one hop plus a database read | *authoritative* | + +OTP's via routing uses `whereis_name/1`, so a `gen_server:call` through a via-tuple +takes the fast path and never queues behind the member's mailbox. + +**Registration blocks until decided; `no` is always adjudicated.** `register_name/2` +answers `yes` if the name is now yours, and `no` **only** when the name is held by a +different live process or under an opt-in §8 policy (`reject_when_degraded`, or +`strict_replication` failing closed). With no reachable leader (none elected yet, a +handoff window, a partition) the call **blocks** and is re-driven once one is +established; a registration whose leader was deposed, or whose commit failed, is +retried or re-sent to the next leader. A `no` there would tell OTP's via machinery +"already started" for a name that is not taken. + +If nothing decides in time the call **exits** on `register_timeout` (§8, default 5 s), +as `global`, `syn` and distributed `gproc` do. The name *may* then be bound to the pid +you tried to register: a retry answers `yes` (§5.2), and if the pid is abandoned its +exit clears the binding. + +**Automatic cleanup.** The leader monitors every registered process; when one exits, +its name is unregistered automatically. + +**A returned pid is a candidate, not a liveness guarantee.** Any pid the registry hands +back — from a lookup, a query, or a presence notification — may have died by the time +you use it. ## 4. How it works ### 4.1 Three processes per node -Starting a registry on a node creates three processes under a local supervisor: +Starting a registry on a node creates three processes under a local `one_for_all` +supervisor: | Process | Role | |---|---| -| **member** | Holds the local `name → pid` replica in a `protected` ETS table that it alone writes and any process reads lock-free. Serves reads. When it is the leader, it is the sole writer for the name table. | -| **elector** | Tracks which nodes are members and decides which member is the leader, coordinating through the database. | -| **connector** | Keeps Erlang distribution in step with that membership: proactively connects to every member node, reaps stranded ones, and runs the leader-liveness probe. Pure node-level connectivity plumbing, with no names/leadership state of its own. Its proactive-mesh half can be turned off per registry (`connectivity => provided_externally`, §4.6, §8) so a registry free-rides on connections another registry maintains. | +| **member** | Holds the local `name → pid` replica in a `protected` ETS table that it alone writes and anyone reads. Serves reads. When leader, it is the sole writer of the name table. | +| **elector** | Tracks membership and decides the leader, coordinating through the database. | +| **connector** | Keeps Erlang distribution in step with membership (§4.6): connects to member nodes, reaps stranded members, probes leader liveness. | -Application code only ever talks to the member (directly or through the -via-tuple). The elector and connector work behind the scenes. +Application code only talks to the member (directly or via the via-tuple). ### 4.2 Leadership comes from the database -The elector does not run a heartbeat or a bespoke election protocol. Membership -changes (a node joining, a node leaving) are submitted as entries on a durable -queue. Each entry is consumed and applied exactly once — the database serialises the -queue so the entries are processed one at a time, in a single global order, no matter -which node happens to consume a given one. The resulting decision (the member set and -who leads) is committed to the database, where every node reads the same answer. There -is no log for each node to replay independently and no opportunity for two nodes to -reach different conclusions. - -Leadership is **sticky**: the current leader keeps the role as long as it remains a -member, which avoids needless churn when nodes come and go. If the leader departs, -the next membership change elects a replacement deterministically. - -Each leadership term is stamped with an increasing **epoch** number. Messages from -an older epoch are ignored, so a message from a deposed leader cannot disturb the -current one. The epoch is also written into the durable leadership record itself, -which makes the fence (§5.1) a true monotonic fencing token rather than an identity -check: even a leader that is deposed and later *re-elected* cannot complete a write -it started in its earlier term. +There is no heartbeat or bespoke election protocol. Membership changes (join, leave) +are entries on a durable queue that the database serialises: each is applied exactly +once, in one global order, whichever node consumes it. The resulting member set and +leader are committed to the database, so every node reads the same answer. -### 4.3 The leader is the sole writer +Leadership is **sticky**: the leader keeps the role while it remains a member. If it +departs, the next membership change deterministically elects a replacement. -Every change to the name table — `register`, `unregister`, and the automatic -unregister triggered by a monitored process dying — is funnelled to the leader. -The leader applies changes through its own `gen_server` mailbox, one at a time, so -they are totally ordered without any additional locking. A follower that receives a -registration **forwards** it to the leader and is told the verdict. +Each leadership term carries an increasing **epoch**, also written into the durable +leadership record. Messages from an older epoch are ignored, and the epoch makes the +fence of §5.1 a monotonic fencing token. -Crucially, no member ever blocks its own message loop waiting on another member. A -follower forwarding a registration does not sit and wait; it hands the request off -and replies to the caller when the answer comes back. This keeps the system free of -the circular waits that would otherwise be possible when the leader, in turn, needs -something from a follower. +### 4.3 The leader is the sole writer + +Every change to the name table — register, unregister, and auto-unregister on process +death — is applied by the leader through its own mailbox, one at a time, so changes are +totally ordered without locks. A follower **forwards** registrations to the leader and +relays the verdict. No member ever blocks its message loop waiting on another member, +which rules out circular waits. ### 4.4 What the database actually stores -The database's job here is **coordination, not storage**. Pids are local to a node -and to a process lifetime; a pid written down and read back later could point at an -unrelated, reused process, so pids are never persisted. The name → pid mapping lives -entirely in the members' memory — each member keeps its replica in a local ETS table -that it recreates, empty, whenever it (re)starts. - -As a result the **name table** costs only about **two keys** in the database, -regardless of how many names are registered: - -- a **leader key**, recording the current leader *and its epoch* (the fencing - token, §5.1), and -- a **version key**, rewritten once per batch of committed changes with a - monotonically increasing commit version. - -The version key exists to make the leader's write a real database transaction -(so the consistency machinery in the next section engages) and to give each batch a -globally-ordered version number. Neither key grows with the number of registered -names. - -Those two are the registry's *own* keys. Separately, the membership and -leader-election layer (the elector) coordinates through a durable, database-serialised -queue, and that queue keeps its own keys: a small amount of consumer/state bookkeeping -plus one entry per *in-flight* membership change (a node joining or leaving). Those -keys are bounded by the size of the cluster and its churn — they are consumed and -cleared as membership settles — and, like the name-table keys, **do not grow with the -number of registered names**. So the "two keys" figure is the name table's footprint -specifically; the whole registry's durable footprint is "two, plus the election -queue's small, name-count-independent overhead." The property that matters either way: -nothing the registry writes to the database scales with how many processes you -register. +The database is for **coordination, not storage**. A pid read back later could name an +unrelated, reused process, so pids are never persisted; the name table lives only in +the members' ETS replicas, recreated empty whenever a member (re)starts. -### 4.5 Replication +The name table costs **two keys**, regardless of how many names are registered: -After the leader applies a batch of changes it broadcasts them to the other members, -which update their local maps. Combined with the forwarding path, this is what keeps -every node's phone book current. +- a **leader key** — the current leader and its epoch (the fencing token, §5.1); +- a **version key** — rewritten once per batch of committed changes with a + monotonically increasing commit version. It makes each leader write a real + transaction (so the fence engages) and gives every batch a global order. -A batch is broadcast as **one message**, carrying its version *and the version of the -batch before it*, so a member applies a batch only when it is **contiguous** with what -it already holds. A member that missed a batch — a message dropped while it was briefly -disconnected — detects the gap, stops applying, and asks the leader for a fresh -snapshot instead of silently continuing with a hole in its replica. Every member's -replica is therefore always a *prefix* of the leader's ordered stream, which is what -makes the handoff reconstruction in §5.7 sound. The details of *how current* a -replica is, and what survives a failure, are the subject of the next section. +The elector's state (membership queue, §4.2; subscriptions, §4.9) is bounded by cluster +size, churn and subscription count. Nothing scales with the number of registered names. + +### 4.5 Replication + +After committing a batch, the leader broadcasts it to the other members as one message +carrying its version and the previous batch's version. A member applies a batch only +if it is **contiguous** with what it holds; on a gap (e.g. a dropped message) it stops +and asks the leader for a fresh snapshot. Every replica is therefore a *prefix* of the +leader's ordered stream — the property the handoff reconstruction (§5.7) relies on. ### 4.6 Joining, leaving, and connectivity -Membership is **dynamic and self-describing**. A node joins simply by starting the -registry; it leaves by stopping it (or by failing). There is no node list to maintain -and no separate cluster-formation step — the membership lives in the database, and -because that record is read transactionally, **every node sees the same membership**, -no matter which node happened to process a given join or leave. - -The registry also keeps Erlang's distribution mesh in step with that membership. Each -member periodically — and at startup, and whenever a node becomes reachable — reads the -authoritative member set and opens an Erlang connection to every member node it is not -already connected to. So you do **not** have to wire the cluster together yourself with -a discovery library or manual `net_adm:ping`: once a node's join is committed, the -other members connect to it, and a brand-new node (which at first exists only as a row -in the database) connects outward to the members it finds there. That outbound -connection is what draws it into the cluster and triggers its first state sync. - -The one thing the registry cannot do for you is make two nodes *capable* of connecting: -they must share an Erlang cookie and be able to resolve and reach each other on the -network. Given that, the registry guarantees the connections themselves — see -[Guarantees](#6-guarantees). +A node joins by starting the registry and leaves by stopping it (or failing). There is +no node list and no cluster-formation step: membership lives in the database and is +read transactionally, so every node sees the same membership. + +The registry also builds the Erlang distribution mesh. At startup, periodically, and +whenever a node becomes reachable, each member reads the member set and connects to any +member node it is not connected to. You do not need a discovery library or +`net_adm:ping`; a new node connects outward to the members it finds in the database, +which triggers its first state sync. Nodes must still be *able* to connect: shared +cookie, resolvable and reachable on the network. #### Delegating connectivity (`connectivity => provided_externally`) -The proactive mesh just described is a **node-level** concern — Erlang distribution is a -single mesh of TCP links shared by everything on the node — but each registry runs its -own copy of it. That is exactly what you want for one registry. It becomes wasteful when -a single node hosts **many** registries at once, which is a first-class use case here: -§4.8 goes to some length so that an application can start many independent registries -dynamically (one per tenant, say) without leaking atoms. On such a node, every registry's -connector independently subscribes to node events, keeps its own (identical) node-state, -runs its own mesh timer, and — most expensively — performs its own periodic **durable -`get_members` read** and its own `connect_node` sweep. Those reads and sweeps scale with -the *number of registries on the node*, not with the size of the cluster, even though the -connections they produce are one shared mesh. - -`connectivity => provided_externally` lets you collapse that duplication by hand. A -registry started in this mode runs its connector with the **proactive mesh switched -off**: it opens no distribution connections and does no periodic member-set read. It -relies on the fact that distribution links are node-global — once *any* registry on the -node has connected node A to node B, every process on A can reach B, including a member of -a registry that never dialled the connection itself. - -The intended shape is a **provider/consumer split**: - -- One **system registry** runs in the default `self_managed` mode and is started on - **every** node in the cluster. Its mesh converges the full distribution graph. -- Many **tenant registries** run in `provided_externally` mode and free-ride on the - links the system registry maintains. They pay none of the mesh cost. - -This is, in effect, a manually-designated node-level connectivity singleton: rather than -introduce a dedicated per-node process (which would have to hold a set of elector pids and -track their liveness — reintroducing the stale-pid problem §4.8's `one_for_all` design -eliminates), you elect an existing, always-present registry to play that role. - -**The contract, and its footgun.** Delegation is only safe if the provider's member set -spans a **node-superset** of every consumer's member set — i.e. the system registry is -running on every node any tenant registry runs on. If that is violated — a tenant member -lands on a node the system registry does not cover — nothing connects that node, and it is -**silently isolated**: it cannot reach a leader, so it answers the CP write refusals of -§5.3 with no obvious cause. This is the price of the flexibility, and it is the deployer's -responsibility, not the registry's. As a cheap backstop, a `provided_externally` connector -that observes **no elected leader for a sustained window** (~60 s) logs a warning naming -this contract, so a misconfiguration surfaces in the logs rather than as a mute refusal. - -**What is *not* delegated.** Only the mesh is node-level and therefore shareable. The -connector's other duties are **registry-scoped** — they read *this* registry's elector and -act on *this* registry's leader and members — and stay active in both modes: - -- the **stranded-member reap** (a `nodedown` backstop that reaps a member the elector - still lists on a departed node), -- the **leader-liveness probe** (reaps a leader recovered from durable state onto a node - this node has never connected — the cold-restart failover, §5.3), and -- the **durable-epoch nudge** (prompts the member to re-join on a missed handoff). - -These cannot be offloaded to the provider registry, which holds no handle on a tenant -registry's elector. Disabling *them* would not save shared work; it would just delete a -registry's own liveness backstops — so `connectivity` deliberately governs the mesh only. -(Because the epoch nudge normally piggybacks on the mesh's member-set read, a -`provided_externally` connector re-homes it onto a small dedicated timer.) - -Any value other than `provided_externally` — including an unrecognised one — resolves to -`self_managed`, so a typo fails *safe*, toward a self-sufficient registry rather than a -silently isolated one. +Distribution links are node-global, but each registry runs its own mesh (a periodic +member-set read and connect sweep), so on a node hosting many registries that work +scales with the number of registries. A registry started with +`connectivity => provided_externally` turns its mesh off and free-rides on links +another registry maintains. The intended shape is one **system registry** in the +default `self_managed` mode on **every** node, plus many `provided_externally` tenant +registries. + +**Contract.** The provider must run on a **node-superset** of every consumer's nodes. +Otherwise a tenant member on an uncovered node is **silently isolated**: it cannot +reach a leader, so its writes block and time out (§3). This is the deployer's +responsibility; as a backstop, a `provided_externally` registry with no elected leader +for ~60 s logs a warning naming the contract. + +Only the mesh is delegated: the stranded-member reap, the leader-liveness probe (§5.3) +and the durable-epoch nudge (§5.7) stay active. Any value other than +`provided_externally`, including a typo, resolves to `self_managed`. ### 4.7 Metadata and queries -A registration is more than a bare pid: it can carry **metadata** that lives for -exactly the registration's lifetime, in two flavours — +A registration can carry metadata that lives exactly as long as the registration: -- **Indexed (`index`)** — a map of attributes the registry indexes, so a **query** - can find every registration matching a set of exact-equality clauses. -- **Non-indexed (`data`)** — an opaque payload, stored and returned verbatim. It is - never interpreted or searched; think of it as a small key/value store whose entries - live as long as the process does. +- **`index`** — a map of attributes the registry indexes for **queries**. +- **`data`** — an opaque payload, stored and returned verbatim, never searched. ```erlang %% Register with metadata in one fenced step (both fields optional; default #{} / undefined). @@ -368,118 +229,45 @@ dgen_registry:query(my_registry, #{role => worker}), %% local snapsh dgen_registry:query_consistent(my_registry, #{role => worker}). %% leader-authoritative ``` -**Storage.** Metadata is not a side table: each member's local replica row is -`{Name, Pid, Index, Data}` — the same ETS row `whereis_name/1` reads, widened. There is -no separate lookup for a plain `whereis_name/1`; it is exactly as cheap as before. Each -member additionally keeps an **inverted index** (`attribute → value → set of names`) in -its own process state, derived from the rows and rebuilt on a handoff — it exists purely -to make a query a few map lookups and a set intersection instead of a table scan. - -**Three kinds of read, not two.** `get_metadata/1` is a third lock-free, caller-side -read alongside `whereis_name/1` — a single `ets:lookup/2` of the full row, with no -member round-trip. `get_metadata_consistent/1` is its leader-routed, authoritative -counterpart, mirroring `whereis_name_consistent/1`. - -**Queries are different: they run *on* the member.** Reading many keys at once (a -query's whole point) makes a caller-side ETS read the wrong tool: the member applies a -committed batch row by row, so a lock-free reader could observe that batch -*half-applied* — some names in the batch updated, others not yet. Routing the query -through the member's own mailbox avoids this: because the member is single-threaded, it -answers a query message strictly *between* batch applications, so the result is always -a **whole-batch-consistent** snapshot — reflecting one committed batch fully, or not at -all, never a mix. This costs a message to the member (unlike the lock-free single-key -reads), which is the deliberate trade: queries are far rarer than routing lookups, and -the extra consistency is worth a mailbox hop. `query_consistent/2` applies the same -mailbox-serialised resolution, but on the leader, for an authoritative answer. - -**Query semantics.** `query/2` and `query_consistent/2` take `Constraints :: -#{attr() => value()}` and return every registration whose `index` map satisfies *every* -clause (AND of exact equalities — no ranges, `OR`, negation, or ordering). An empty -`Constraints` map is rejected (`{error, empty_query}`) rather than silently meaning "all -registrations." A clause naming an attribute that no current registration's `index` -carries simply matches nothing — there is no declared schema of "indexed attributes" to -check a clause against, so an unmatched attribute yields an empty result via the same -AND-equal logic as any other non-matching clause, not a schema error. `data` is never -queryable — only `index` is; a query cannot reference it. A returned pid may already -have died by the time it is used, the same eventual-liveness caveat as any registry -lookup — treat matches as candidates, not liveness guarantees. - -**How writes ride the pipeline.** Metadata introduces no new durable state and no new -fence: `register_name/3` and `set_metadata/2` are batch ops on the same -leader-as-sole-writer, fenced group-commit pipeline as `register_name/2` (§4.3–§4.4). -`set_metadata/2` on a name that is not currently registered is rejected -(`{error, not_registered}`); a leader that has lost leadership before answering a -buffered `set_metadata/2` rejects it with `{error, no_leader}` rather than silently -dropping it, so the caller knows to retry. Replication and read-after-write follow the -same shape as registration: a follower that forwarded a `set_metadata/2` is answered -only after the leader's replication broadcast has updated its own row, so a subsequent -local `get_metadata/1` on that node reflects the write. - -**Lifetime and handoff.** Metadata's lifetime is exactly the registration's — the same -`DOWN`/unregister path that clears a name's row clears its metadata and index entries -with it, no separate cleanup mechanism. On a leadership handoff, metadata travels with -the binding through the same freshest-wins gather (§5.7) that reconstructs names: the -new leader's reconstructed rows carry whichever holder's metadata came with the -freshest binding, and every member rebuilds its inverted index from its reconstructed -rows. The pid-uniqueness conflict detector (§5.6) is unaffected — it adjudicates pids, -and metadata is simply carried along on the winning binding. +**Reads.** Metadata lives in the same replica row `whereis_name/1` reads. +`get_metadata/1` is a lock-free caller-side read like `whereis_name/1`; +`get_metadata_consistent/1` is its authoritative counterpart. Queries use a per-member +inverted index and run **through the member's mailbox** (on the leader for +`query_consistent/2`): the member applies a batch row by row, so a lock-free multi-key +read could see it half-applied, whereas a mailbox-serialised query sees whole batches. + +**Query semantics.** `Constraints :: #{attr() => value()}` matches every registration +whose `index` satisfies *every* clause (AND of exact equalities; no ranges, `OR`, +negation or ordering). An empty map is rejected with `{error, empty_query}`. An +attribute no registration carries simply matches nothing — there is no schema. `data` +is never queryable. + +**Writes.** `register_name/3` and `set_metadata/2` ride the same fenced, sole-writer +commit pipeline as `register_name/2` (§4.3); metadata adds no durable state. +`set_metadata/2` on an unregistered name returns `{error, not_registered}`; if the leader +loses leadership before answering it, the call returns `{error, no_leader}` so the +caller can retry. A follower that forwarded a `set_metadata/2` answers only after its own +row is updated, so a following local `get_metadata/1` on that node sees the write. + +**Lifetime and handoff.** The unregister/`DOWN` path that clears a name clears its +metadata and index entries. On handoff, metadata travels with the binding through the +reconstruction of §5.7; conflict detection (§5.6) adjudicates pids only. ### 4.8 Process identity -Starting a registry creates **zero atoms**. `Name` is supplied by the caller to -`start_link/2,3` — it is application-controlled, not something `dgen_registry` -generates — and the module never derives any further atom from it: nothing else is -registered under a name. This matters for an application that starts many independent -registries dynamically — one per tenant, say — since Erlang atoms are never -garbage-collected: if `dgen_registry` derived even one additional atom per registry (an -`{elector, Name}`-style name, for instance), that would mean an unbounded, -permanently-leaked atom for every tenant the application ever creates. Whatever atom -budget an application spends on `Name` itself is its own to manage. - -Concretely: - -- **The supervisor** returned by `start_link/2,3` is not registered under any name; - callers hold its pid directly (e.g. to `Supervisor.stop/2` it later). -- **The member** is registered as `Name` itself — the same atom the caller passed in, - reused rather than combined into a derived name. -- **The member's ETS names-table** is also named `Name`. ETS `named_table` names and - Erlang process-registration names (`erlang:register/2`) are separate namespaces, so - the same atom names both the process and its table with no collision. -- **The elector** has no registered name at all. It is found, when needed, by walking - from the member: reading the member's `$ancestors` process-dictionary entry (set by - `proc_lib` at spawn time — there is no public accessor for it, so this is read - straight out of the dictionary, the same technique tools like `recon` use) yields - its supervisor, and `supervisor:which_children/1` on that supervisor yields the - elector's current pid (`dgen_registry:elector_pid/1`). This is a local, in-memory - lookup — no network hop, no registry of its own to maintain. `get_leader/1`, - `get_epoch/1`, and `get_members/1` use it to reach the elector directly, the same - lock-free-ish bypass-the-member's-mailbox path they always have. - -**Why this lookup is deferred.** The member cannot resolve the elector inside its own -`init/1`: the supervisor starts its children synchronously, one at a time, and is still -blocked inside `supervisor:start_child/2` waiting for the member's `init/1` to return -when the member is starting — calling `supervisor:which_children/1` on it from there -would deadlock. Instead `init/1` returns `{ok, State, {continue, discover_elector}}`, -which lets the supervisor unblock immediately; `handle_continue/2` then performs the -lookup and the elector-dependent startup steps (joining, connecting the mesh) right -after, before any other message is processed. - -**Why `one_for_all`.** The member resolves the elector's pid once, at startup, and -never re-resolves it. Under `one_for_one`, an elector crash restarts only the elector, -leaving every member holding a stale pid pointing at the dead process — recoverable -only by adding a monitor and a re-discovery path. The supervisor's strategy is -`one_for_all` instead: elector and member are one unit, and a crash of either restarts -both together, so the member's cached elector pid is never stale by construction. There -is no case where a live member can be holding a pid for an elector that is no longer -the one paired with it. +Starting a registry creates **zero atoms**, so an application can start many registries +dynamically (one per tenant, say) without leaking atoms. Only the caller-supplied `Name` +is used: the member is registered as `Name` and its ETS table is named `Name`; the +supervisor and elector are unregistered (the elector is found through the member's +supervisor). The supervisor is `one_for_all`, so the member's cached elector pid is +never stale. ### 4.9 Presence: watch/notify subscriptions -Metadata queries (§4.7) answer *"who matches this right now?"* as a one-shot read. A -**presence subscription** turns that into a live feed: a caller registers a **watch -query** and a **notify query** under an application-chosen id, and the registry pushes a -message to the notify targets whenever the set of processes satisfying the watch query -changes. +A **presence subscription** turns a query (§4.7) into a live feed: under an +application-chosen id, a caller registers a **watch query** and a **notify query**, and +the registry pushes a message to the notify targets whenever the set of processes +matching the watch query changes. ```erlang %% Keep every process matching `notify` up to date with the set matching `watch`. @@ -492,359 +280,176 @@ ok = dgen_registry:subscribe(my_registry, dgen_registry:unsubscribe(my_registry, {presence, RoomId}). ``` -A `query/0` is `{all, #{attr => value}}` (a conjunction of exact equalities — the -`query/2` semantics), or a bare map read as `{all, Map}`. The type is **tagged so it can -grow** (ranges, disjunction, predicates) without breaking existing subscriptions. - -**Durable, keyed by an application id.** A subscription is **not** tied to any Erlang -process or to the cluster's lifetime. It is stored in the **elector's durable -`dgen_server` state** (§4.2) — the same backend-backed, cluster-shared state that holds -the member set and the leader — keyed by an application-supplied `SubId`. Because that -state survives a full cluster restart (a new elector loads the existing state rather than -re-initialising it), an application can model its presence stakeholders as database -entities: `subscribe` when an entity is created, `unsubscribe` when it is deleted, and a -`subscribe` with an existing `SubId` is an idempotent **upsert**. A system can scale to -zero and come back with exactly the subscriptions it had — the notify targets simply -resume receiving updates as they re-register. This is why subscriptions live on the -elector and not, say, as ordinary registry rows (which are pid-scoped and evaporate with -the VM): durability is the whole point. - -**Manipulated only through the elector.** The public API (`subscribe/4`, -`unsubscribe/2`, `unsubscribe_all/1`, `subscriptions/1`) talks to the elector, not the -member. A write is a durable `dgen_server` **cast** (`handle_cast_tx`) — fire-and-forget -for latency, like the membership changes — so `ok` means "accepted for durable -processing", and the change commits (and `subscriptions/1`, a priority read, reflects it) -shortly after. Durability holds regardless, and because a subscription is an idempotent -upsert keyed by its id, a write lost in the narrow pre-enqueue window is recoverable by -re-subscribing. There is no way for an external caller to poke the leader's presence -state directly. - -**Teardown.** Because every durable key a registry writes — the elector's state and -queue, the leader fence, the version key — lives under a single tenant keyspace prefix -derived from the registry name, the whole footprint is reclaimed by one range clear: -`delete/2(Name, Tenant)` (call it after stopping the supervisor; a live elector caches -its state in memory and would re-create the keys on its next commit). This matters for an -application that creates many short-lived, per-entity registries in a shared tenant — -without it, each deleted registry would leak its keys. `unsubscribe_all/1` is the -narrower operation that clears just the subscriptions while the registry keeps running. - -**How a change is detected.** Notifications are computed on the **leader**, at the same -point a committed batch is applied to its replica (the group commit of §4.5). The batch's -durable delta already names every registration that changed; each changed name's -*post-batch* state is tested against every subscription's watch query and compared to -that subscription's previously-matching set. A name that starts matching is a `joined`, -one that stops is a `left`, and a rebind of a matching name to a different pid is a `left` -of the old plus a `joined` of the new. This makes the feed correct for **every** way the -watch set can move — a register, an unregister, an auto-unregister on process death, or a -`set_metadata` that shifts a row in or out of the query. - -**Initial snapshot for every viewer, whenever it appears.** A notify target learns who is -already present via an **initial snapshot** — the current watch set delivered as a batch -of `joined` events. Crucially this fires not only at `subscribe` time but whenever a -process *enters* the notify set: the notify side is tracked symmetrically to the watch -side (`notify_matches` mirrors `sub_matches`), so a process that registers into the -notify query *after* the subscription was created still receives the full current watch -set, then deltas thereafter. This matters because a durable subscription routinely -outlives — or predates — its viewers: the subscription can be created (tied to a database -entity) before any process that will consume it exists. A viewer that is already a notify -target when the watch set changes instead gets the ordinary `joined`/`left` delta. - -**How the leader gets the subscriptions.** The elector is the source of truth; the leader -holds a working copy: - -- *Live change* — a `subscribe`/`unsubscribe` cast commits in the elector's consume - transaction, whose post-commit action pushes the delta (`{presence_update, …}`, - epoch-stamped) to the current leader, which applies it and, for a new subscription, - seeds its watch membership and fires the initial snapshot. -- *Handoff* — a newly-assuming leader **pulls** the full durable set from its co-located - elector (a local priority read — the elector shares the member's supervisor) at the - moment it becomes leader, and computes each watch set against the reconstructed replica. - Pulling at assume time (rather than trusting a set captured earlier) closes the race - where a subscribe commits during a handoff window: anything committed before the pull is - read, and anything after pushes a delta the now-established leader applies. This pull is - also the backstop that reconciles any live delta a transiently unreachable leader missed - — every leadership change re-reads the truth. A follower holds no presence state at all. - -The leader's working state is three maps: `subs` (`SubId → {watch, notify}`, seeded on -assume and advanced by deltas) plus a leader-only current **watch** membership and -current **notify** membership per subscription (both recomputed on assume, advanced per -commit — the watch one drives the `joined`/`left` deltas, the notify one detects a new -viewer to send the initial snapshot to). - -**Consistency.** The feed inherits the model of §5: changes are pushed in commit order, -each subscriber sees a prefix of the true sequence of watch-set transitions, and delivery -is asynchronous (a notify target on another node sees a change slightly after the leader -commits it). A viewer's first message is its initial snapshot (delivered when it enters -the notify set), and every message after that is a delta — so a viewer, from the moment -it appears, always has a complete and then continuously-maintained view of the watch set, -regardless of whether it registered before or after the subscription. As everywhere else, -a delivered `Pid` is a candidate, not a liveness guarantee: it may already have died when -the message arrives. - ---- +A `query/0` is `{all, #{attr => value}}` (the `query/2` semantics) or a bare map read as +`{all, Map}`. The type is tagged so it can grow without breaking existing subscriptions. -## 5. Consistency and failure model +**Durable, keyed by an application id.** Subscriptions live in the elector's durable +state (§4.2), are tied to no process, and survive a full cluster restart. `subscribe` +with an existing `SubId` is an idempotent **upsert**, so applications can model +subscriptions as database entities and scale to zero and back with them intact. + +**API.** `subscribe/4`, `unsubscribe/2`, `unsubscribe_all/1` and `subscriptions/1` go +to the elector. Writes are asynchronous: `ok` means accepted for durable processing; the +change commits (and shows in `subscriptions/1`) shortly after. Re-subscribing recovers a +write lost before it was enqueued. + +**Teardown.** A registry's durable keys share one keyspace prefix; `delete/2(Name, +Tenant)`, called after stopping the supervisor, removes them all in one range clear. +`unsubscribe_all/1` clears only the subscriptions of a running registry. -This section states precisely what holds and what does not. It is the heart of the -document. +**Change detection.** As the leader applies each committed batch it tests every changed +row against each watch query: a name that starts matching is `joined`, one that stops is +`left`, and a rebind to a new pid is both. This covers register, unregister, process +death and `set_metadata`. + +**Initial snapshot.** Whenever a process *enters* the notify set — at `subscribe` time +or later — it first receives the current watch set as `joined` events, then deltas. A +subscription can therefore predate its viewers. + +**Leader state.** Committed subscription changes are pushed to the current leader; a +newly-assuming leader pulls the full durable set and recomputes every watch set, which +also reconciles any change it missed. Followers hold no presence state. + +**Consistency.** Delivery is asynchronous and in commit order: each viewer sees an +initial snapshot, then a prefix of the true sequence of watch-set transitions. Delivered +pids are candidates (§3). + +## 5. Consistency and failure model ### 5.1 Fencing: a stale leader cannot write — or serve consistent reads -When leadership moves, the new leader's identity **and epoch** are committed to the -database. From that instant, the old leader is **fenced**: its attempts to commit a -change conflict with the leadership record and are rejected by the database. It -physically cannot register or unregister a name any more. Because the record carries -the epoch, the fence is a monotonic fencing token: a member that is deposed and later -re-elected holds a *new* epoch, so even its own in-flight writes from the earlier -term are rejected — there is no ABA window in which an old term's write can land in a -new term. - -This is what rules out the classic split-brain failure where two nodes each believe -they are in charge and both hand out the same name. There is at most one leader that -can successfully write, enforced by the database's transaction conflict detection -rather than by hope or timing. - -Consistent **reads** are fenced too. A deposed leader that has not yet heard about -the handoff (the minority side of a partition believes it still leads until told -otherwise) must not serve its frozen replica as authoritative — so every -`*_consistent` answer is released only after the answering member re-verifies its -leadership (id and epoch) against the durable record. A verification that fails, or -cannot reach the database, answers the not-registered/empty value instead: the same -CP refusal the write path makes. This is what lets Guarantee 5 say "authoritative" -without a timing asterisk; the cost is one database read per consistent read, which -is the deliberate price of the word. +When leadership moves, the new leader's identity and epoch are committed to the +database. From then on the old leader is **fenced**: its commits conflict with the +leadership record and the database rejects them. Because the record carries the epoch, +a member deposed and later re-elected cannot land a write from its earlier term either. +At most one leader can write, enforced by transaction conflict detection rather than +timing — so there is no split brain. + +Consistent **reads** are fenced too. A deposed leader may not yet know it (e.g. on the +minority side of a partition), so every `*_consistent` answer is released only after +the member re-verifies its leadership (id and epoch) against the durable record. If +verification fails or cannot reach the database, it answers not-registered/empty. The +cost is one database read per consistent read. ### 5.2 Linearizable registration -Because all registrations flow through a single leader that is fenced against -impostors, and that leader processes them one at a time, registration is -**linearizable**: there is a single, well-defined order of accept/reject decisions, -and once `register_name/2` returns `yes`, every subsequent consistent read observes -that registration (until it is changed). +All registrations flow through a single fenced leader that processes them one at a +time, so registration is **linearizable**: there is one order of decisions, and once +`register_name/2` returns `yes`, every later consistent read observes it until it +changes. -Registration is also **idempotent for the same holder**: registering a name to the -pid that already holds it returns `yes`, not `no`. Only a *different* live pid is -rejected. This is what makes a registration safe to redrive — a caller whose -`register_name` call timed out (§3) or raced a failover, but whose registration -actually committed, retries and gets a decisive `yes` for its own binding rather than -a `no` it would have to disambiguate. +Registration is **idempotent for the same holder**: registering a name to the pid that +already holds it returns `yes`. This makes redriving safe — a caller that timed out +(§3) but whose registration committed gets `yes` on retry. ### 5.3 Partition behaviour: CP -If the network splits, only the side that can still reach the database and holds the -leadership can commit changes. The other side cannot register or unregister names — -its writes are fenced or simply cannot reach the database. - -This is the deliberate CP trade-off: during a partition the disconnected side -sacrifices **availability** (of writes) to preserve **consistency** (no two -processes share a name). Reads of the local snapshot still work on both sides, but -may be stale on the disconnected side. - -One partition shape deserves its own note: an **Erlang-distribution-only** -partition, where the mesh between nodes is severed but the database stays reachable -from both sides. Both sides then observe each other's departure and briefly contend -over the single durable leadership record; the contention settles — leadership is -sticky and the record is one — after which exactly one side holds leadership and -commits, while the other side's members answer the CP refusals (`no` for writes, -not-registered for consistent reads, §5.1) and periodically re-announce themselves -until the mesh heals. The leader-liveness probe deliberately reaps only a leader on -a node the probing member has *never* been connected to (the cold-restart recovery -case), so an isolated member cannot depose a healthy leader through the shared -database. On heal, the rejoin adjudication of §5.6 repairs any name that was -reissued across the split. +Only the side that reaches the database and holds leadership can commit. On the other +side, registrations block and time out (§3), an unregister answers `ok` and is held +until a leader can take it (Non-goal 5), and consistent reads answer not-registered +(§5.1); local snapshot reads keep working but may be stale. Availability of writes is +traded for consistency. + +In an **Erlang-distribution-only** partition (mesh severed, database reachable from +both sides) the sides briefly contend over the single leadership record; then one side +leads and the other blocks as above until the mesh heals. The leader-liveness probe +only reaps a leader on a node the prober has *never* connected to (cold-restart +recovery), so an isolated member cannot depose a healthy leader. On heal, §5.6 repairs +any name reissued across the split. ### 5.4 Uniqueness is single-fault tolerant -The central guarantee is **uniqueness**: at most one live process is registered -under a given name at a time. The honest, precise statement is that this holds -across the failure of **any one member at a time**. +**Uniqueness** — at most one live process per name — holds across the failure of **any +one member at a time**. Pids are not durable, so the only record that a name is taken +is the members' in-memory replicas. Every acknowledged registration is held by at least +two members (§5.5), so losing one member never loses that knowledge; losing *both* +holders at once can, after which the name could be reissued. The registry detects and +repairs that case (§5.6). -Why "one"? Because pids are not durable, the only record that a name is taken is the -in-memory ETS replicas of the members that know about it. The registry's replication keeps -every acknowledged registration on **at least two members** (see below), so losing -any single member never loses the knowledge that a name is taken. Losing *both* -holders simultaneously — a genuinely concurrent multi-node failure — can lose that -knowledge, after which the name could be handed out again. +### 5.5 Addressability and the two-holder rule -The registry does not pretend this cannot happen. Instead it **detects and repairs** -it: see [Conflict resolution](#56-conflict-resolution-by-termination). +**Addressability** asks whether a registered name can still be *found* after a failure. +When `register_name/2` returns `yes`, the binding is normally held by **two members**: -### 5.5 Addressability and the two-holder rule +- A registration **forwarded** from a follower is held by the leader and that follower. + The follower answers `yes` only once its replica has applied the batch's commit + version (deferring if it is mid-resync), so the second copy is always visible to the + handoff reconstruction (§5.7). +- A registration made **on the leader's node** waits for `register_replicas` follower + confirmations (§8) before `yes`. If none arrive within `replicate_timeout`, the + default policy **degrades open** and answers `yes` with a single holder (emitting a + warning and telemetry event); `strict_replication` fails it closed instead. -"Uniqueness" is about not giving a name to two processes. **Addressability** is the -separate question of whether a name you registered can still be *found* after a -failure. - -When `register_name/2` returns `yes`, the binding is held by **two members**, so it -survives any single node going down: - -- A registration **forwarded** from a follower is held by both the leader and that - follower as soon as the follower records it — two holders for free. The - follower's ack is **version-guarded**: the leader's reply carries the batch's - commit version, and the follower answers `yes` only once its replica has applied - up to that version (normally already true — the replication broadcast precedes - the reply). A follower whose replica is momentarily *gapped* (it missed a batch - and is awaiting a resync) defers the ack until the resync lands, so the second - holder is always visible to the handoff reconstruction in §5.7 — without the - guard, a leader crash in that window could silently drop an acked binding whose - only surviving copy sat at a version the freshest-wins gather ignores. -- A registration made **directly on the leader's node** is, at first, held only by - the leader. The leader therefore waits for a follower to confirm it has a copy - before answering `yes`. - -Two honest caveats: - -- **Under load the direct path can degrade.** If no follower confirms within a - timeout, the leader by default answers `yes` anyway with a single holder, to stay - responsive. This is configurable (see [Configuration](#8-configuration)); you can - require more confirmations, or require the registry to fail the registration - instead of proceeding with one holder. -- **A pid cannot outlive its own node.** If the node *hosting the process* dies, the - process is gone regardless, and its name is released. The two-holder rule protects - the *routing entry* against the loss of a node that does **not** host the process - (typically the leader) — it does not resurrect a dead process. +The rule protects the *routing entry* against losing a node that does not host the +process (typically the leader). If the process's own node dies, the process is gone and +its name is released. ### 5.6 Conflict resolution by termination -Because uniqueness is single-fault tolerant rather than absolute, the registry needs -a way to recover if reality ever diverges — for example, a process that held a name -on a node that was unreachable long enough to be dropped, and the name was reissued -elsewhere. When such a node returns, two different live processes could briefly claim -the same name. - -When the registry observes this — two *different, live* processes for one name — it -resolves it decisively: it **terminates both** processes, clears the name, and raises -an alarm (a log line and, when the optional `telemetry` library is present, a -`[dgen_registry, conflict]` event). Supervised processes then restart and re-register -cleanly under the single leader. A per-name budget bounds how often this can fire -before the situation is escalated to an operator instead of looping. - -Detection runs at both moments a divergent replica can surface. At every -**leadership-change handoff**, the gather collects every reachable member's replica -and adjudicates divergences before the new leader fans out its snapshot (§5.7). And -when a previously-synced member **rejoins a continuing leader** — the common shape -of a partition heal, where the surviving side kept its leader and no handoff -happens — the leader first gathers the rejoiner's replica (one bounded round-trip, -off its message loop) and adjudicates it against its own authoritative table before -onboarding. A fresh member (never synced) provably holds no bindings and skips the -check entirely, so a healthy scale-up pays nothing. One honest limitation either -way: repair needs a *witness* — some replica still reporting the stale binding. A -binding no surviving replica reports (a multi-fault loss, Non-goal 2) leaves nothing -to detect; its process has simply lost its registration. - -Detection must not misfire on mere staleness: a member that lagged behind (or was -briefly disconnected) may still report a binding that was *legitimately* unregistered -while its process lives on — that is lag, not a conflict, and killing for it would be -wrong. The registry therefore keeps a **release trail**: every explicit unregister of -a live process is remembered — keyed by the **name-and-pid pair**, so releasing a pid -under one of its names never masks a genuine conflict on *another* name the same -process also holds — for a configurable window (`conflict_release_ttl`, §8) on -*every* member. The leader records it at commit, followers record it from the -replication broadcast, and a leadership handoff merges every reachable member's -trail into the new leader's, so the discriminator survives leadership changes. Trail -entries are only expired while the cluster is fully connected **and** no member has -departed within the last `conflict_release_ttl`: a member dropped from the set -during a partition is no longer "a disconnected member", but its stale rows can -still surface when it rejoins, and the trail must outlive that window too. (A member -gone longer than the TTL is outside the trail's protection horizon; its rejoin is -adjudicated with whatever trail remains, under the kill budget.) - -One scope note on the kill budget below: it is tracked by the **current leader** -and, unlike the trail, is not merged across a handoff — a leadership change starts -the new leader with a fresh budget for every name. The `conflict_kill_budget` bound -is therefore per name *per leader term*; a conflict that regenerates across -failovers is bounded per term, not globally. - -This is why the registry is described as a **singleton** registry, and why it carries -an explicit contract: +If a process held a name on a node unreachable long enough to be dropped, and the name +was reissued elsewhere, two live processes can claim the name when the node returns. +On observing two *different, live* pids for one name, the registry **terminates both**, +clears the name, and raises an alarm (a log line and, if `telemetry` is present, a +`[dgen_registry, conflict]` event). Supervised processes restart and re-register under +the single leader. Killing both, rather than picking a winner, is deliberate. + +Detection runs at every **leadership handoff**, against the gathered replicas (§5.7), +and when a previously-synced member **rejoins a continuing leader** (the usual partition +heal), against the rejoiner's replica; a never-synced member is skipped. Repair needs a +*witness*: a binding no surviving replica reports cannot be detected (Non-goal 2). + +**Staleness is not conflict.** A lagging member may still report a binding that was +legitimately unregistered while its process lives on. Every member therefore keeps a +**release trail** of explicit unregisters of live processes, keyed by name-and-pid, for +`conflict_release_ttl` (§8); it is merged across handoffs. Entries expire only while +the cluster is fully connected and no member has departed within the last TTL. A member +gone longer than the TTL is adjudicated with whatever trail remains. + +**Kill budget.** `conflict_kill_budget` (§8) bounds terminations per name before +escalating to an operator. It is tracked by the current leader and reset on handoff, so +the bound is per name *per leader term*. + +This is the registry's explicit contract: > The registry may forcibly terminate a registered process to enforce uniqueness. > Register only processes that can withstand being forcibly killed — for example, > supervised and restart-safe, or transient by design. -This behaviour is intentional and is **not** a bug. It can be turned off (reducing -the registry to detect-and-alarm), and there is an optional, conservative mode that -prevents the reissue in the first place at the cost of refusing some registrations -during a partition — see [Configuration](#8-configuration). +Termination can be turned off (`terminate_on_conflict`, leaving detect-and-alarm), and +`reject_when_degraded` prevents the reissue in the first place at some cost in +availability — see [Configuration](#8-configuration). ### 5.7 Leadership handoff -When the leader changes, the new leader rebuilds its name table by asking every -reachable member for its current map — the requests fan out in parallel, bounded by a -single overall timeout — and taking the most up-to-date answer. It does not read -names from the database (there are none to read). Because the previous leader is -already fenced, this handoff needs no global pause: the moment leadership commits, -the old leader can no longer interfere. - -That gather is a **network** fan-out, so the new leader runs it **off its message -loop**: it answers the elector immediately and finishes assuming when the gather -returns. This is what keeps a handoff from freezing the new leader — an inline -multi-second gather would stall every request and every further membership change -queued behind it, which under heavy churn (many nodes joining and leaving) compounds -into a cluster that stops responding. During the brief window before the -reconstruction completes the new leader is not yet serving as leader, so writes on it -*block* (the no-leader path, §3) and are re-driven once it assumes — never served -against a half-built table. A newer membership change that arrives mid-gather simply -supersedes the in-flight one. - -Reconstruction takes the freshest reachable member's map as the answer. Because -broadcasts are totally ordered *and every member's replica is a gap-checked prefix of -that order* (§4.5 — a member that missed a batch resyncs rather than advancing past -the hole), the freshest map already holds the current binding for every name a -*reachable* member knows, and two members reporting the same version necessarily hold -the same map — so a name that is two-holder survives any single member being -unreachable (its other holder is in the gather). A binding that -was held *only* by an unreachable member (a degrade-open single holder, or a -multi-fault loss) is absent from the rebuilt table and is **not** restored when that -member later returns — its name is treated as free, and re-registration is the -application's responsibility (§5.4). The one thing that *is* repaired automatically is -a resulting **conflict**: if such a name was reissued while the holder was away, the -two live claimants are reconciled by termination (§5.6) when the member returns. - -Two subtleties make that freshest-wins claim watertight rather than merely -usual-case. A reachable member's report can be *transiently* behind an -already-committed batch still in flight to it from the deposed leader — a -broadcast cast queued but not yet processed, since Erlang gives no ordering -guarantee between messages sent by different processes (the deposed leader's -broadcast and the gather's own request are independent senders, and the -network is free to deliver either first). Left unhandled, a gather that races -ahead of that in-flight broadcast would reconstruct a map missing a binding -its second holder was about to record — and then, worse, its resulting -snapshot would overwrite that follower once the broadcast finally landed, -silently erasing the very copy that made the registration two-holder in the -first place. Two guards close this: - -- The gather is **fenced against the durable version key** (§4.4): a live - member can never be ahead of it, so the assuming leader compares its - gathered maximum version against the version key and re-gathers — bounded, - short retries — until it catches up, rather than finalizing on a report it - knows is stale. Past the bound it proceeds anyway (a genuinely unreachable - holder, not a race, is the ordinary degraded case and is backstopped by - §5.6). -- Any wholesale re-baseline a member applies — a handoff snapshot or a resync - answer — is **version-monotonic**: a member accepts it only if its version - is at least as fresh as what the member already holds. A snapshot computed - before a batch was gathered but delivered after the member independently - applied that batch is therefore rejected rather than obeyed, so a handoff - can never roll a member's replica backward. - -Together these mean the handoff reconstruction is sound not just in the -common case where every reachable member has already drained its mailbox, but -under the interleaving where it hasn't. This is exactly the class of -correctness argument formal methods are suited to; see -[`formal/README.md`](../../formal/README.md) for the TLA+ model that checks it -(and the counterexample it produces if either guard above is disabled). - -Two smaller handoff properties are worth stating. First, a handoff always reaches its -new leader eventually: the membership change and the new leadership record commit -atomically, and if the notification that follows is lost (the notifying process -crashed, or could not reach the new leader), every member periodically compares the -epoch it has heard against the committed one and re-announces itself on a mismatch — -which re-drives the handoff. Second, direct registrations that were awaiting their -replica confirmations when leadership was lost are resolved immediately under the -same policy as a replication timeout (§8), rather than being left to dangle. - ---- +A new leader rebuilds its name table by gathering every reachable member's replica in +parallel and taking the freshest. The old leader is already fenced, so no global pause +is needed. Until the gather completes, writes block (§3) and are re-driven once the new +leader assumes. + +Because replicas are gap-checked prefixes of one ordered stream (§4.5), the freshest +replica holds the current binding for every name any reachable member knows, so a +two-holder binding survives any single unreachable member. A binding held *only* by an +unreachable member (a degrade-open single holder, or a multi-fault loss) is absent from +the rebuilt table and is not restored when that member returns; if the name was reissued +meanwhile, §5.6 resolves the conflict. + +Two guards keep this sound when a committed broadcast from the deposed leader is still +in flight to a member during the gather: + +- The gather is **fenced against the durable version key** (§4.4): no live member can be + ahead of it, so the new leader re-gathers (bounded retries) until its freshest replica + reaches the key. Past the bound it proceeds, and the handoff is **degraded** — it + stopped short of the version key and may be missing committed bindings + (`reject_when_degraded`, §8, acts on this). +- Every wholesale re-baseline (handoff snapshot or resync) is **version-monotonic**: a + member rejects one older than what it holds, so a handoff never rolls a replica back. + +The TLA+ model in [`formal/README.md`](../../formal/README.md) checks this argument. + +A handoff always reaches its new leader eventually: every member periodically compares +the epoch it has heard with the committed one and re-announces itself on a mismatch. +Direct registrations awaiting replica confirmations when leadership is lost are resolved +under the replication timeout policy (§8). ## 6. Guarantees @@ -852,27 +457,23 @@ These are normative. Behaviour that violates one of these is a defect. 1. **Singleton uniqueness (single-fault).** At most one live process is registered under a given name at a time, preserved across the failure of any single member. - A genuine simultaneous loss of both holders of a binding is outside this bound and - is repaired reactively (§5.6), never silently tolerated. -2. **No split-brain writes.** At most one leader can commit changes. A leader that - has lost leadership cannot register, unregister, or otherwise mutate the name - table. There is no merge step and no dual-accept window. -3. **Linearizable registration.** Successful registrations have a single global - order; after `register_name/2` returns `yes`, every later - `whereis_name_consistent/1` reflects it until it changes. -4. **Two-holder durability of acknowledged registrations.** A registration - acknowledged `yes` is held by at least two members and survives any single node - failure — *except* under the degrade-open replication policy (which is the - **default**) when a replication timeout occurred, in which case it may have one - holder. Operators who want the two-holder property unconditionally must opt into - `strict_replication`; the policy spectrum is documented in - [Configuration](#8-configuration). -5. **Authoritative consistent reads.** `whereis_name_consistent/1` reflects the - current leader's committed view of the name. This holds even across an - unnoticed deposition: every consistent answer is fence-verified against the - durable leadership record before release (§5.1), so a deposed leader that has - not yet heard about the handoff answers the CP refusal (not-registered) rather - than a stale value presented as authoritative. + A simultaneous loss of both holders of a binding is outside this bound and is + repaired reactively (§5.6), never silently tolerated. +2. **No split-brain writes.** At most one leader can commit changes. A leader that has + lost leadership cannot register, unregister, or otherwise mutate the name table. + There is no merge step and no dual-accept window. +3. **Linearizable registration.** Successful registrations have a single global order; + after `register_name/2` returns `yes`, every later `whereis_name_consistent/1` + reflects it until it changes. +4. **Two-holder durability of acknowledged registrations.** A registration acknowledged + `yes` is held by at least two members and survives any single node failure — + *except* under the default degrade-open policy after a replication timeout, when it + may have one holder. `strict_replication` makes the property unconditional + ([Configuration](#8-configuration)). +5. **Authoritative consistent reads.** `whereis_name_consistent/1` reflects the current + leader's committed view of the name, even across an unnoticed deposition: every + answer is fence-verified against the durable leadership record (§5.1), so a deposed + leader answers not-registered rather than a stale value. 6. **Automatic unregistration.** When a registered process exits, its name is unregistered without an explicit call. 7. **CP under partition.** The side of a partition that cannot reach the database (or @@ -881,32 +482,23 @@ These are normative. Behaviour that violates one of these is a defect. 8. **Registry isolation.** Distinct registries do not share leadership, membership, name tables, or database keys. 9. **Consistent membership and convergent connectivity.** Every node observes the same - committed member set (the membership lives in the database and is read - transactionally). Given nodes that are *able* to connect (shared cookie, network - reachability), the registry establishes the Erlang connections itself: after a - node's join is committed, the cluster converges so that `nodes()` on each member - returns every other reachable member — without an external discovery mechanism. (It - cannot connect nodes that are misconfigured or unreachable, and an unreachable - member is, by definition, not in `nodes()` until it becomes reachable.) This is the - default (`self_managed`) behaviour; a registry configured - `connectivity => provided_externally` (§4.6, §8) deliberately delegates the - connecting to another registry on the node and does not establish connections - itself — the convergence guarantee then rests on that provider. -10. **Metadata lifetime is tied to the registration.** A registration's metadata (both - `index` and `data`) and its index entries are present exactly while the - registration is, and are removed automatically on unregister or process exit — - the same path, no separate cleanup step (§4.7). + committed member set. Given nodes able to connect (shared cookie, network + reachability), after a node's join commits the cluster converges so that `nodes()` on + each member returns every other reachable member, with no external discovery. In + `connectivity => provided_externally` mode (§4.6, §8) the registry does not connect + nodes itself, and convergence rests on the provider registry. +10. **Metadata lifetime is tied to the registration.** A registration's `index`, `data` + and index entries exist exactly while the registration does and are removed on + unregister or process exit, by the same path (§4.7). 11. **Metadata writes are fenced and linearizable.** `set_metadata/2` rides the same fenced, single-leader commit pipeline as `register_name/2,3`; a leader that has lost leadership cannot apply or broadcast a metadata change. -12. **Metadata reads mirror name reads.** `get_metadata/1` is a lock-free, caller-side - snapshot read with the same freshness characteristics as `whereis_name/1`; - `get_metadata_consistent/1` is leader-authoritative, mirroring - `whereis_name_consistent/1`. -13. **Queries are batch-consistent.** `query/2` and `query_consistent/2` never observe - a half-applied group-commit batch — the result reflects one committed batch fully, - or not at all (§4.7). Both are still *snapshots* (eventually consistent locally, or - authoritative on the leader), and a returned pid may already have died. +12. **Metadata reads mirror name reads.** `get_metadata/1` is a lock-free snapshot read + with the freshness of `whereis_name/1`; `get_metadata_consistent/1` is + leader-authoritative, mirroring `whereis_name_consistent/1`. +13. **Queries are batch-consistent.** `query/2` and `query_consistent/2` never observe a + half-applied commit batch (§4.7). Both are still snapshots, and a returned pid may + already have died. ## 7. Non-goals and explicit non-guarantees @@ -914,64 +506,46 @@ These are intentional. Relying on the opposite is relying on something the regis does not promise. 1. **Durability of names across a full restart.** Pids are never persisted. If every - member of a registry is down at once, all of its names are gone; applications must - re-register on startup. A full cluster restart starts with an empty registry. + member of a registry is down at once, its names are gone and applications must + re-register on startup. 2. **Multi-fault uniqueness.** The simultaneous loss of both holders of a binding may - lose the registration and can, transiently, allow the name to be reissued. This is - detected and repaired (§5.6) *when a surviving replica still reports the old - binding*; a loss with no surviving witness leaves nothing to detect — the - displaced process has simply lost its registration and is not terminated. It is - not prevented — unless the optional prevention mode is enabled, which trades - availability for it. -3. **Availability during a partition.** The disconnected/minority side cannot - register names. This is a consequence of the CP choice, not a defect. -4. **Strong freshness from `whereis_name/1`.** The snapshot read may lag a recent - change made elsewhere by a short replication interval. Use - `whereis_name_consistent/1` when you need the authoritative answer. -5. **Synchronous cluster-wide unregistration.** `unregister_name/1` routes to the - leader as a tracked call and normally returns `ok` after the removal has - committed — but it does **not** block until every node has observed it; - cluster-wide visibility remains eventually consistent (replication is - asynchronous). Eventual propagation is actively driven at every hop: an - unregister whose commit fails, whose leader is deposed mid-flight, whose forward - was dropped, or that was accepted while no leader was reachable is stashed and - re-driven — re-enqueued on the leader, or handed to the current leader as a - pid-guarded clear on the next snapshot or leadership event — rather than - silently dropped. Explicit unregisters are rare, so the tracked round-trip - costs nothing that matters. -6. **Preservation of a process the registry must terminate.** The registry may - terminate a registered process to enforce uniqueness (§5.6). A process that cannot - withstand being forcibly killed — one that is neither restart-safe nor transient by - design — is not a good fit, and its termination is by design, not a bug. -7. **Automatic re-registration after failure.** The registry does not re-register a - process on the application's behalf after a multi-fault loss; detecting the loss - and re-registering (typically via a supervisor) is the application's - responsibility. -8. **Rich queries.** `query/2` and `query_consistent/2` support only a conjunction of - exact-equality clauses over indexed attributes — no ranges, `OR`, negation, - prefix/substring matching, ordering, pagination, or joins (§4.7). -9. **Querying non-indexed metadata.** `data` is opaque and returned only via - `get_metadata/1` / `get_metadata_consistent/1`; it is never matched by a query, and - there is no post-filtering of query results on it. -10. **Durability of metadata across a full restart.** Like names, metadata is - in-memory only; a full cluster restart loses it, and applications re-set metadata - when they re-register (same as item 1, above). -11. **Uniqueness or constraints on indexed values.** Many registrations may share the - same indexed attribute/value pair; the index is a multimap, and the registry does - not enforce a "unique attribute" constraint. -12. **Per-pid (cross-name) metadata.** Metadata is attached per-registration, not - per-pid: a process registered under several names has independent metadata for - each name. - ---- + lose the registration and can transiently allow the name to be reissued. This is + detected and repaired (§5.6) when a surviving replica still reports the old binding; + with no surviving witness the displaced process simply loses its registration and is + not terminated. It is not prevented unless `reject_when_degraded` is enabled, which + trades availability for it. +3. **Availability during a partition.** The disconnected/minority side cannot register + names (§5.3). +4. **Strong freshness from `whereis_name/1`.** The snapshot read may briefly lag a change + made elsewhere. Use `whereis_name_consistent/1` for the authoritative answer. +5. **Synchronous cluster-wide unregistration.** `unregister_name/1` normally returns `ok` + after the removal has committed on the leader, but does not wait for every node to + observe it; replication is asynchronous. Propagation is still guaranteed: an + unregister whose commit fails, whose leader is deposed, whose forward is dropped, or + that arrives with no reachable leader is re-driven to the current leader rather than + dropped. +6. **Preservation of a process the registry must terminate.** The registry may terminate + a registered process to enforce uniqueness (§5.6). A process that is neither + restart-safe nor transient by design is not a good fit, and its termination is by + design. +7. **Automatic re-registration after failure.** After a multi-fault loss, detecting the + loss and re-registering (typically via a supervisor) is the application's job. +8. **Rich queries.** Queries support only a conjunction of exact-equality clauses over + indexed attributes — no ranges, `OR`, negation, prefix/substring matching, ordering, + pagination, or joins (§4.7). +9. **Querying non-indexed metadata.** `data` is returned only by `get_metadata/1` / + `get_metadata_consistent/1`; it is never matched by or used to filter a query. +10. **Durability of metadata across a full restart.** Metadata is in-memory only, like + names (item 1); applications re-set it when they re-register. +11. **Uniqueness or constraints on indexed values.** Many registrations may share an + indexed attribute/value pair; there is no unique-attribute constraint. +12. **Per-pid (cross-name) metadata.** Metadata is per registration: a process + registered under several names has independent metadata for each. ## 8. Configuration -Sensible defaults make the registry single-fault tolerant out of the box; the knobs -exist to move along the availability/strictness spectrum. - -Configuration is **per registry**. Each registry is started independently, and its -tuning is supplied as an options map at start time: +Defaults make the registry single-fault tolerant; the knobs move it along the +availability/strictness spectrum. Configuration is **per registry**, passed at start: ```erlang dgen_registry:start_link(orders, Tenant, #{ @@ -980,107 +554,51 @@ dgen_registry:start_link(orders, Tenant, #{ }). ``` -Because the settings are scoped to a single registry, one registry can be strict -while another degrades open — they do not share configuration. Any key left unset -falls back to the `dgen` application environment (a convenient way to set a global -default for every registry), and then to the built-in default below. +An unset key falls back to the `dgen` application environment (a global default), then +to the built-in default below. | Setting | Default | Effect | |---|---|---| -| `register_timeout` | `5000` ms | Caller-side bound on how long a **leader-routed write** waits before giving up — `register_name/2,3`, `unregister_name/1`, and `set_metadata/2` alike. | -| `register_replicas` | `1` | How many follower copies a *direct* registration waits for before `yes`. Bounded by the number of followers. Higher values widen durability at the cost of registration latency. | -| `commit_batch_size` | `5000` | Max write ops the leader coalesces into one group commit; the rest ride the next commit. Bounds the *inline* per-commit work (plan, replica apply, broadcast fan-out are all O(batch)), so a burst — e.g. a departing node's flood of `DOWN`s — is split across bounded commits and the leader loop stays responsive between them. Latency/throughput only, never correctness. Lower to cap the burst; raise to coalesce more aggressively. | +| `register_timeout` | `5000` ms | How long a leader-routed write (`register_name/2,3`, `unregister_name/1`, `set_metadata/2`) waits before the caller exits (§3). | +| `register_replicas` | `1` | Follower copies a *direct* registration waits for before `yes` (§5.5), bounded by the number of followers. Higher widens durability at the cost of latency. | +| `commit_batch_size` | `5000` | Max write ops coalesced into one group commit; the rest ride the next. Bounds per-commit work so a burst (e.g. a departing node's flood of `DOWN`s) keeps the leader responsive. Affects latency/throughput only, never correctness. | | `replicate_timeout` | `1000` ms | How long the leader waits for those confirmations before applying the timeout policy. | -| `strict_replication` | `false` | Timeout policy. `false`: **degrade open** — acknowledge `yes` with whatever holders exist (possibly one) to stay responsive. `true`: **fail closed** — reject the registration and retract the binding so a caller never sees an unreplicated `yes`. The retraction survives a leadership change: if the rejecting leader is deposed before its retraction commits, the pid-guarded retract is handed to the new leader, so a `no` cannot silently leave the binding alive. | -| `terminate_on_conflict` | `true` | Whether a detected uniqueness conflict (§5.6) terminates the conflicting processes, or only detects and alarms. | -| `conflict_kill_budget` | `{3, 60000}` | At most *N* terminations per name per window (ms) before escalating to an operator instead of looping. Tracked by the current leader only and not merged across a handoff, so the bound is per name *per leader term* (§5.6). | -| `conflict_release_ttl` | `600000` ms | How long an explicitly-released live name/pid pair stays in the conflict-detector trail (§5.6) — the discriminator that keeps a lagging replica from being mistaken for a conflict. Expiry is suspended while any member is disconnected *and* for one TTL after any member departs, so neither a partition nor a dropped member's rejoin within the window can outlive the trail. Entries are tiny; a generous window is cheap. | -| `reject_when_degraded` | `false` | Optional prevention (§5.6). When `true`, a leader whose last handoff could not reach every member that *could be holding bindings* refuses to register a name it does not already hold — preventing a reissue rather than repairing it. A brand-new joiner that has not yet connected (it exists only as a row in the database, §4.6) holds nothing and does not count, so a healthy scale-up does not trip this. Deliberately blunt otherwise: it cannot distinguish a partition from a deliberate scale-down, so it is off by default. | -| `connectivity` | `self_managed` | Who maintains the Erlang-distribution mesh for this registry (§4.6). `self_managed` (default): the registry runs its own proactive mesh — self-sufficient. `provided_externally`: the mesh is **off** (no `connect_node`, no periodic member-set read); the registry free-rides on links another registry maintains, for a node hosting many registries. Requires that a `self_managed` provider registry span a node-superset of this registry's nodes, or the node is silently isolated (§4.6) — a sustained no-leader window logs a warning. Governs the mesh only; the registry-scoped backstops (leader-liveness probe, stranded-member reap, durable-epoch nudge) stay active. Any value other than `provided_externally` resolves to `self_managed` (typos fail safe). | - -Observability: when a direct registration degrades open, the registry emits an event -through the optional `telemetry` library (if present) and logs a warning, so the -weakening of the two-holder rule under load is visible. - ---- +| `strict_replication` | `false` | Timeout policy. `false`: **degrade open** — answer `yes` with whatever holders exist (possibly one), with a warning and telemetry event. `true`: **fail closed** — answer `no` and retract the binding; the retraction is handed to the next leader if this one is deposed first, so the binding cannot survive. | +| `terminate_on_conflict` | `true` | Whether a detected conflict (§5.6) terminates the claimants, or only detects and alarms. | +| `conflict_kill_budget` | `{3, 60000}` | At most *N* terminations per name per window (ms) before escalating to an operator. Per name *per leader term* (§5.6). | +| `conflict_release_ttl` | `600000` ms | How long an explicitly released live name/pid pair stays in the release trail (§5.6). Expiry pauses while any member is disconnected and for one TTL after any member departs. Entries are tiny. | +| `reject_when_degraded` | `false` | Optional prevention (§5.6). When `true`, a leader whose last handoff was **degraded** (reconstruction stopped short of the durable version key, §5.7) answers `no` to registering a name it does not already hold. Merely-behind or never-synced members do not cause degradation, so normal scale-up/down does not trip it; losing a binding's only holders does. Trades availability for prevention. | +| `connectivity` | `self_managed` | Who maintains the distribution mesh (§4.6). `self_managed`: this registry runs its own. `provided_externally`: mesh off; relies on a `self_managed` registry spanning a node-superset of this one's nodes, else the node is silently isolated (a sustained no-leader window logs a warning). Registry-scoped backstops stay active. Any other value resolves to `self_managed`. | ## 9. Comparison with `global`, `gproc`, and Elixir's `Registry` -Erlang ships with a global registry, `gproc` is a widely-used third-party one, and -Elixir's `Registry` is the standard local one. They make different, reasonable -trade-offs; `dgen_registry` is not a replacement so much as a different point in the -design space. The summaries below aim to be accurate and even-handed — the right -choice depends on what you are optimising for. - -A quick orientation: Elixir's `Registry` solves the *single-node* problem, while -`global`, `gproc` (global mode), and `dgen_registry` all solve the *cluster-wide* -problem in different ways. +Elixir's `Registry` solves the single-node problem; `global`, `gproc` (global mode) and +`dgen_registry` solve the cluster-wide one in different ways. Each is a reasonable +trade-off. ### Elixir's `Registry` (standard library, single node) -Elixir's `Registry` is a fast, ETS-backed, **single-node** process registry. It is -not a distributed registry and is not Erlang's `global` — it does not span an Erlang -cluster; each node has its own independent `Registry`. Within that one node it is -excellent: it supports unique and duplicate keys, is partitioned internally for high -concurrency, integrates with the `{:via, Registry, …}` tuple, and adds value-style -metadata and dispatch. - -The contrast with `dgen_registry` is simply **scope**. If your naming need is -local to a node, `Registry` is the natural, dependency-free choice and there is no -reason to reach for anything heavier. When you need *one* owner of a name across a -*cluster* — the problem `dgen_registry` exists for — `Registry` does not address it on -its own; cluster-wide registration is typically layered on top of a separate -distribution mechanism. `dgen_registry` provides that cluster-wide guarantee directly. +A fast, partitioned, ETS-backed **single-node** registry with unique and duplicate keys, +via-tuple support, values and dispatch. If your naming need is local to a node it is the +natural, dependency-free choice. It does not provide one owner of a name across a +cluster, which is what `dgen_registry` is for. ### `global` (OTP, built in) -`global` is part of the Erlang/OTP distribution and needs no external dependency. It -maintains a single cluster-wide namespace, replicated in ETS on every node, and -serialises registrations with a distributed locking protocol. During a network -partition both sides remain **available** for registration, and when the partition -heals `global` reconciles name clashes using a resolver function you can supply -(with a sensible default). - -The essential contrast is the partition model. `global` is **AP-leaning**: it stays -available on both sides of a split and reconciles afterward, which is excellent when -availability matters more than never having two owners momentarily. -`dgen_registry` is **CP**: it refuses writes on the disconnected side so that two -owners never arise, accepting reduced availability during a split. If you want a -zero-dependency, always-available registry and can define how to merge after a split, -`global` is a natural fit. If you want the system to guarantee a single owner even -through a partition, and you already run (or can run) a consistent database, -`dgen_registry` is aimed at that. +Built into OTP with no dependency: one cluster-wide namespace replicated on every node, +serialised by distributed locks. It is **AP-leaning** — both sides of a partition keep +accepting registrations, and clashes are reconciled on heal by a resolver function. +Choose it for a zero-dependency, always-available registry when you can define the +merge; choose `dgen_registry` to guarantee a single owner through a partition, if you +can run a consistent database. ### `gproc` (third-party) -`gproc` is a popular, feature-rich library. Beyond name → pid registration it offers -properties, counters, aggregated counters, and an extended `ets:select`-style -selection API, in both a very fast single-node local mode and a distributed global -mode. Its global mode elects a leader among the Erlang nodes themselves to coordinate -updates. - -The contrast here is **scope, coordination source, and membership model**. `gproc` -covers far more than naming — counters, aggregated counters, and a general selection -API (match specs, guards, ranges) are outside `dgen_registry`'s scope entirely. -`dgen_registry` does have per-registration metadata and an AND-equal query over it -(§4.7), but that is a narrow, lifetime-tied convenience layered on the name table, not -a general secondary-index or counter system — if you need `gproc`'s breadth, especially -within a single node, it is hard to beat, and it runs entirely in memory with **no -external dependency or on-disk state**, a real operational advantage. Its distributed -mode coordinates by electing a leader among the participating Erlang nodes. - -`dgen_registry` is narrowly a cluster name registry, and it externalises coordination -to a database rather than electing a coordinator purely within the Erlang cluster. -That externalisation is the whole point of `dgen_registry`'s consistency story, and -also its main cost: it requires a backend database to run, which `gproc` does not. -What the externalisation buys, beyond consistency, is a **dynamic membership model**: -nodes join and leave simply by starting or stopping the registry, the database -serialises the resulting membership changes, and a stale leader is fenced -automatically — so growing, shrinking, or rolling a cluster needs no pre-declared node -list and causes minimal disruption. If your priority is a self-contained, in-memory -system with the richest feature set, `gproc` is compelling; if it is a strict -cluster-wide singleton with low-friction, dynamic membership, `dgen_registry` is aimed -at that. +A feature-rich, fully in-memory library — names, properties, counters, aggregated +counters, `ets:select`-style queries — with a fast local mode and a global mode that +elects a leader among the Erlang nodes. Its breadth is outside `dgen_registry`'s scope, +and it needs no external database. `dgen_registry` pays for that dependency with strict +consistency and dynamic membership: no node list, stale leaders fenced automatically. ### At a glance @@ -1089,31 +607,22 @@ at that. | Cluster-wide | No (single node) | Yes | Yes | Yes | | Ships with | Elixir stdlib | OTP | No (library) | No (library) | | External dependency | None | None | None | A CP database backend | -| On-disk state on the BEAM nodes | None | None | None | None (durable state lives in the DB) | | Coordination source | N/A (local) | Distributed locks | Leader among nodes | Strongly-consistent database | | Partition stance | N/A (local) | Stays available, merges on heal | Available, leader-coordinated | Refuses minority writes (CP) | | Membership model | Single node | Dynamic (node connections) | Leader election among nodes | Dynamic join/leave via the DB; no node list | | Scope | Names (+ duplicate keys, dispatch) | Names | Names, properties, counters, rich queries | Names + lifetime-tied metadata | | Metadata / secondary index | Values (no query) | No | Properties, counters, `ets:select`-style queries | `index`/`data` per registration; AND-equal query over `index` | -| Pids in durable storage | No | No | No | No | | Authoritative read | Local (single node) | — | — | `whereis_name_consistent/1` (`get_metadata_consistent/1`, `query_consistent/2`) | | Enforces single owner via termination | No | No | No | Yes (configurable) | | Isolated namespaces | Yes (each `Registry`) | No (one namespace) | Via classes/scopes | Yes (independent registries) | ---- - ## 10. When `dgen_registry` is (and isn't) a good fit -**Good fit when** you need a strict cluster-wide singleton — exactly one live process -per name — and you would rather a name be briefly unavailable than briefly owned -twice; when your registered processes can withstand being forcibly killed (restart-safe -or transient by design); when you already operate, or are willing to operate, a -consistent database such as FoundationDB; and when any per-registration lookup beyond -plain name → pid is a simple AND-equal search over a handful of attributes (§4.7), not -a rich query workload. - -**Reconsider when** you need writes to remain available on every side of a network -partition; when you cannot run an external database; when a registered process must -never be killed by the registry; or when you need counters, aggregated counters, or a -general selection API (ranges, match specs) rather than plain name → pid lookup with an -AND-equal metadata query, for which a purpose-built library is a better tool. +**Good fit** when you need a strict cluster-wide singleton and prefer a name briefly +unavailable to briefly owned twice; your registered processes can withstand being +killed; you can operate a consistent database such as FoundationDB; and lookups beyond +name → pid are simple AND-equal searches (§4.7). + +**Reconsider** when writes must stay available on every side of a partition, you cannot +run an external database, a registered process must never be killed by the registry, or +you need counters or a general selection API. diff --git a/docs/design/dgen_server_design.md b/docs/design/dgen_server_design.md index f824d4d..56edaee 100644 --- a/docs/design/dgen_server_design.md +++ b/docs/design/dgen_server_design.md @@ -1,39 +1,13 @@ # dgen_server Design -`dgen_server` brings the `gen_server` programming model to a distributed system. -It uses the same callback interface — `init/1`, `handle_call/3`, `handle_cast/2` -— but unlike a `gen_server`, it is not a single Erlang process. A `dgen_server` -is a virtual entity that exists as long as its state does in the configured -backend (default: FoundationDB), independent of any running process, node, or -cluster. - -Its defining choice is that state and message queue both live in a -strongly-consistent database instead of process memory. That single decision is -what gives `dgen_server` its particular personality, summarised up front so you -can decide quickly whether it fits: - -- **It behaves like an ordinary `gen_server` to write.** Same callbacks, same - return shapes, same mental model. No new programming model to learn. -- **It is durable and highly available.** State and queued messages survive the - process, the node, and even a full cluster restart. Zero or more Erlang - processes may act on a given `dgen_server` at any time — any number of - consumers, on any number of nodes, can consume from the same message queue - concurrently, with serialization and exactly-once delivery (under normal - operation) guaranteed by the backend. -- **Callbacks run inside a database transaction and must be pure.** - `handle_call/3`, `handle_cast/2`, and `handle_info/2` execute within a backend - transaction (subject to its size/time limits) and must not perform side - effects — no I/O, no messaging, nothing that isn't safely repeatable if the - transaction retries. Side effects are pushed to a post-commit **Actions** - list instead (§4.3). - -This document is the source of truth for what `dgen_server` does. The -[Guarantees](#6-guarantees) section is normative: behaviour that contradicts a -guarantee is a bug. - -It is written to be read top to bottom. The early sections assume only that you -know what a `gen_server` is; the later sections get more precise about the -transaction model and failure behaviour. +`dgen_server` brings the `gen_server` programming model (`init/1`, +`handle_call/3`, `handle_cast/2`, ...) to a distributed system. A `dgen_server` +is not an Erlang process: it is durable state plus a durable message queue in a +strongly-consistent backend (default: FoundationDB), and it exists as long as +that state does, independent of any process, node, or cluster. + +The [Guarantees](#6-guarantees) section is normative: behaviour that contradicts +it is a bug. **This document is LLM generated.** @@ -41,32 +15,17 @@ transaction model and failure behaviour. ## 1. The one-minute model -I love `gen_server`. There are only two things stopping most of us from writing -an entire app with them: - -1. **Durability** — the state is lost when the process goes down. -2. **High availability** — the functionality is unavailable when the process - goes down. - -`dgen_server` solves both by moving state and the message queue into the -backend database and decoupling them from any single process: - -- A `dgen_server` has no single Erlang-process representation. It is durable - state plus a durable, ordered message queue, both living in the database. -- Zero or more **consumer** processes — on any node — pull messages off that - queue, run your callback, and commit the result in one database transaction: - the state is read once, the callback runs, and the new state (plus the - reply, if any) is written back, all atomically. -- **Standard** messages (`call`, `cast`) go through the durable queue and are - processed with strict serializability. **Priority** messages - (`priority_call`, `priority_cast`, `handle_info`) skip the queue and run - immediately — an escape hatch for urgent work, at the cost of ordering. -- If a consumer crashes mid-transaction, nothing commits: the message stays in - the queue (or is lost, for the non-durable priority paths — see - [§5](#5-crash-and-failure-behaviour)) and is retried by the next consumer - that picks it up. - -Everything below is an elaboration of that picture. +- Zero or more **consumer** processes, on any node, pull messages off the queue, + run your callback, and commit the new state (and reply, if any) in one + database transaction. +- **Standard** messages (`call`, `cast`) go through the queue and are processed + with strict serializability. **Priority** messages (`priority_call`, + `priority_cast`, `handle_info`) skip the queue and run immediately, at the + cost of ordering. +- Callbacks run inside that transaction and must be pure; side effects go in + post-commit **Actions** (§4.3) or `handle_locked/4` (§4.4). +- If a consumer crashes mid-transaction nothing commits; a queued message is + retried by the next consumer (priority messages are lost, §5). --- @@ -76,9 +35,6 @@ Everything below is an elaboration of that picture. ### Erlang -The simplest distributed server is just a regular `gen_server` with the -`dgen_server` behaviour: - ```erlang -module(counter). -behavior(dgen_server). @@ -86,51 +42,32 @@ The simplest distributed server is just a regular `gen_server` with the -export([start/1, increment/1, value/1]). -export([init/1, handle_call/3]). -start(Tenant) -> - dgen_server:start(?MODULE, [], [{tenant, Tenant}]). - -increment(Pid) -> - dgen_server:call(Pid, increment). - -value(Pid) -> - dgen_server:call(Pid, value). +start(Tenant) -> dgen_server:start(?MODULE, [], [{tenant, Tenant}]). +increment(Pid) -> dgen_server:call(Pid, increment). +value(Pid) -> dgen_server:call(Pid, value). -init([]) -> - {ok, 0}. +init([]) -> {ok, 0}. -handle_call(increment, _From, State) -> - {reply, ok, State + 1}; -handle_call(value, _From, State) -> - {reply, State, State}. +handle_call(increment, _From, State) -> {reply, ok, State + 1}; +handle_call(value, _From, State) -> {reply, State, State}. ``` -Start it inside a FoundationDB directory, and the state persists across -restarts: - ```erlang Tenant = dgen_erlfdb:sandbox_open(<<"demo">>, <<"counter">>), {ok, Pid} = counter:start(Tenant), -counter:increment(Pid), -counter:increment(Pid), -2 = counter:value(Pid), - -%% Restart the process +counter:increment(Pid), counter:increment(Pid), dgen_server:stop(Pid), {ok, Pid2} = counter:start(Tenant), -2 = counter:value(Pid2). %% State persisted! +2 = counter:value(Pid2). %% State persisted ``` ### Elixir -The simplest distributed server is just a regular `GenServer` with -`use DGen.Server`: - ```elixir defmodule Counter do use DGen.Server def start(tenant), do: DGen.Server.start(__MODULE__, [], tenant: tenant) - def increment(pid), do: DGen.Server.cast(pid, :increment) def value(pid), do: DGen.Server.call(pid, :value) @@ -145,20 +82,13 @@ defmodule Counter do end ``` -Start it inside a FoundationDB directory, and the state persists across -restarts: - ```elixir tenant = :dgen_erlfdb.sandbox_open("demo", "counter") {:ok, pid} = Counter.start(tenant) -Counter.increment(pid) -Counter.increment(pid) -2 = Counter.value(pid) - -# Restart the process +Counter.increment(pid); Counter.increment(pid) GenServer.stop(pid) {:ok, pid2} = Counter.start(tenant) -2 = Counter.value(pid2) # State persisted! +2 = Counter.value(pid2) # State persisted ``` @@ -167,45 +97,22 @@ GenServer.stop(pid) ## 3. Core concepts -**backend.** A `dgen` backend implementation interfaces with a database. The -provided backend is for FoundationDB using `erlfdb`. Other compatible backends -can, in principle, be contributed. - -**tenant.** A pairing of a database handle and a directory handle, as provided -by the backend, that defines a subspace of the keyset partitioned for some -purpose as defined by the developer. Every `dgen_server` is started against a -tenant. - -**key-tuple.** A tuple that is to be encoded into a binary for storage as a key -in a key-value pair inside a tenant subspace. Any key-tuple may be further -extended by inserting a new item at the end of the tuple; in that case the -original key-tuple becomes a prefix key-tuple, a container for other -key-values via range operations. - -**tuid.** Short for tenant-unique identifier — a key-tuple that uniquely -identifies some resource in a tenant. `init/1` may return `{ok, Tuid, State}` -to choose it explicitly. - -**message-queue.** Each `dgen_server` has a durable queue of messages from -which it processes requests, held entirely in the backend. - -**call-request** / **cast-request.** A call-request is an operation put on the -message-queue that expects a response; a cast-request does not. - -**priority-request.** A call or cast request that ignores the message-queue -and is handled immediately by a `dgen_server` process. Use with caution — this -breaks ordering guarantees with respect to the queued stream. Useful for -urgent, usually read-only work ("snapshot reads"). - -**dgen_server, dgen_server process, dgen_server consumer.** The `dgen_server` -itself is the abstract entity — its state is one or more key-values in the -tenant, and its functionality is defined by the callback module. It has no -singular representation in the Erlang VM. A **dgen_server process** is any -Erlang process capable of pushing requests onto the message-queue. A -**dgen_server consumer** is a dgen_server process that is *also* capable of -consuming items from the message-queue and performing the operations they -describe. There can be zero, one, or many consumers for a given `dgen_server` -at once. +- **backend**: the database interface; the provided one is FoundationDB via + `erlfdb`. +- **tenant**: a database handle plus a directory handle defining a key + subspace. Every `dgen_server` is started against one. +- **key-tuple**: a tuple encoded as a key in a tenant; extending it makes the + original a prefix containing other keys. +- **tuid**: the key-tuple identifying a `dgen_server` in its tenant. `init/1` + may return `{ok, Tuid, State}` to choose it. +- **message-queue**: the durable queue a `dgen_server` processes requests from. +- **call-request / cast-request**: a queued operation that does / does not + expect a response. +- **priority-request**: a call or cast handled immediately, ignoring the queue + and its ordering; for urgent, usually read-only work. +- **dgen_server process / consumer**: any process that can push requests onto + the queue / one that also processes them. A `dgen_server` may have zero, one, + or many consumers. --- @@ -213,380 +120,197 @@ at once. ### 4.1 Message processing paths -`dgen_server` provides different message paths with different guarantees: - | | Standard (`call`, `cast`) | Priority (`priority_call`, `priority_cast`, `handle_info`) | |---|---|---| -| Durable queue | goes through it | skips it entirely | -| Ordering | strictly serialized with the rest of the queued stream | none — races ahead of queued messages | -| Transaction | executes within a database transaction (subject to backend transaction limits) | executes within a database transaction (subject to backend transaction limits) | -| Side effects in the callback | not permitted — callbacks must be pure with respect to external systems | not permitted — same rule | -| The lock (§4.4) | respected — paused while locked | ignored — always executes, even while locked | - -Priority messages are an explicit escape hatch for urgent work that can't wait -behind the queue; standard messages are the default, ordered, durable path. +| Durable queue | goes through it | skips it | +| Ordering | strictly serialized with the queued stream | none; races ahead of queued messages | +| Transaction | runs inside one (backend limits apply) | runs inside one (backend limits apply) | +| Side effects in the callback | not permitted | not permitted | +| The lock (§4.4) | respected; paused while locked | ignored; always executes | ### 4.2 `consume_k` and inlining `consume_k` (default `1`) is the maximum number of queued messages a consumer -processes per transaction — the batch size. Each consume cycle peeks up to -`consume_k` messages, processes them one at a time while carrying the durable -state in memory, and commits the whole batch in a **single** transaction (the -state is read once and written once). It then immediately re-arms itself, so a -consumer that has work keeps draining batches. - -Raising `consume_k` amortises the fixed per-transaction cost (read version, -state read/write, commit) across more messages, increasing throughput when the -per-message callback work is small. The cost is larger transactions and a -coarser unit of progress — a batch is all-or-nothing, so a conflict retries -the *whole* batch. Because a busy consumer keeps draining, a larger `consume_k` -also tends to keep a **single** node as the active consumer, which is useful -when the callback's identity matters — for example an election-style callback -where you want to minimise churn in who processes successive messages. - -A message can reach a callback by two routes: - -1. **Through the durable queue.** `cast/2` and (normally) `call/2` enqueue the - message; a consumer later peeks it as part of a `consume_k`-sized batch and - processes it inside the batch's single transaction. This is the ordered, - durable path, and the *only* path when `consume_k > 1`. -2. **Inline.** As a latency optimisation, when `consume_k =:= 1` a `call/2` - whose queue is currently empty and unlocked is processed *immediately*, in - the caller's own request transaction, instead of being enqueued and waited - on. The result is identical to processing it through the queue — it just - skips the enqueue/await round-trip. - -When **`consume_k > 1`, inlining is disabled**: every `call/2` goes through -the queue and the batched consume loop, so `consume_k` is always in effect. -Use this when you want all processing funnelled through the batched, -single-consumer loop. With the default `consume_k =:= 1`, inlining is enabled -and a contention-free `call/2` is served on the fast path. - -Independently of `consume_k`, `priority_call/2` and `priority_cast/2` -**always** bypass the queue (and locks) and are handled immediately, per §4.1. +processes per transaction. A consume cycle peeks up to `consume_k` messages, +processes them in order with the state carried in memory, commits the batch in +**one** transaction (state read once, written once), and immediately re-arms. + +A larger `consume_k` amortises per-transaction cost and raises throughput when +callbacks are cheap, at the price of larger transactions and all-or-nothing +batches (a conflict retries the whole batch). A busy consumer keeps draining, +so it also tends to keep a **single** node as the active consumer, which helps +when consumer identity matters (e.g. election-style callbacks). + +**Inlining.** When `consume_k =:= 1`, a `call/2` whose queue is empty and +unlocked is processed immediately in the caller's own transaction, with the +same result as going through the queue. When **`consume_k > 1`, inlining is +disabled** and every `call/2` goes through the queue. + +`priority_call/2` and `priority_cast/2` always bypass the queue and the lock +(§4.1). ### 4.3 Actions Callbacks may return `{reply, Reply, State, Actions}` or -`{noreply, State, Actions}`, where `Actions` is a list of 1-arity functions. -These functions: +`{noreply, State, Actions}`, where `Actions` is a list of 1-arity functions +that: -- Execute after the transaction commits. -- Receive the committed `State` as their argument, but cannot modify it. -- Are the correct place for side effects: logging, telemetry, publishing to - external systems. -- Can return `halt` to stop processing actions, or any other value to - continue. +- run after the transaction commits; +- receive the committed `State` but cannot modify it; +- are the place for side effects (logging, telemetry, external publishing); +- may return `halt` to stop running the remaining actions. -Under batched consumption (`consume_k > 1`), all of a batch's actions run -after the batch's single commit, and **each** receives the state as of the -**end of the batch** — not the state at the specific message that produced -the action. +With `consume_k > 1`, a batch's actions all run after its single commit and +each receives the state as of the **end of the batch**. ### 4.4 Locking -A callback may return `{lock, State}` instead of `{noreply, State}` or -`{reply, ...}` to enter locked mode. This is useful when a state change -requires synchronous post-commit coordination before the next queue message -can be safely processed — long-running operations that would exceed -transaction time limits, such as calling external APIs or performing extended -computations. +A callback may return `{lock, State}` to enter locked mode, for work that must +finish before the next queued message is processed but cannot run inside a +transaction (external API calls, long computations). -When locked: +While locked: - Standard `call` and `cast` messages are queued but not processed. -- Priority messages and `handle_info` continue to execute (§4.1). -- The `handle_locked/4` callback is invoked **outside** a transaction: - - Not subject to backend transaction limits. - - Side effects are permitted. - - Can modify state, which is written back to the database. - -`lock_timeout` (default `infinity`) bounds how long a lock may be held before -other consumers treat it as stale. When a callback returns `{lock, State}`, -`dgen_server` sets a timestamped lock key in the backend that pauses other -consumers while `handle_locked/4` runs. Under normal operation the lock holder -clears the key itself before `handle_locked` returns. If the holder is killed -(SIGKILL, VM abort) the lock persists until another consumer detects -staleness: it re-checks `lock_timeout` ms after the lock was set and clears it -if that deadline has passed. `infinity` (the default) disables stale-lock -detection entirely — a dead holder will permanently block all consumers. Set -this to a value safely larger than the worst-case `handle_locked` duration for -your callback. - -See §4.7 for the exact commit/coordinate/release sequence. +- Priority messages and `handle_info` still execute (§4.1). +- `handle_locked/4` runs **outside** a transaction: no transaction limits, side + effects permitted, and the state it returns is written back. + +`lock_timeout` (default `infinity`) bounds how long a lock may be held. The +holder normally clears the lock when `handle_locked/4` returns. If the holder +is killed (SIGKILL, VM abort), another consumer clears the lock once +`lock_timeout` ms have passed since it was set. With `infinity`, a dead holder +blocks all consumers permanently. Set it safely above the worst-case +`handle_locked/4` duration. + +Clearing a stale lock is a guess that the holder is dead, so the lock is +fenced (§4.7): a busted holder's result is not committed, its message is +requeued, and it exits with `{lock_fenced, requeued}`. (A priority or inline +lock has no queued message: it exits with `{lock_fenced, not_requeued}`, and the +caller receives the exit.) Its side effects have already happened, so +**`handle_locked/4` must be safe to run twice**. ### 4.5 Persisted state -State is persisted to the key-value store using a structured encoding scheme -that optimises for partial updates. Three encoding types are supported: - -1. **Assigns map**: maps with all atom keys are split across separate keys, - one per entry. No ordering guarantees. - - #{ - mykey => <<"my value">>, - otherkey => 42 - } +State is encoded to optimise partial updates: -2. **Components list**: lists where every item is a map with an atom `id` key - containing a binary value. Each item is stored separately with ordering - maintained via fractional indexing in the storage key. +1. **Assigns map**: a map with all atom keys; one key per entry, unordered. - [ - #{id => "item1", value => 1}, - #{id => "item2", value => 2} - ] + #{mykey => <<"my value">>, otherkey => 42} -3. **Term**: all other terms use `term_to_binary` and are chunked into 100KB - values. +2. **Components list**: a list of maps each with an atom `id` key holding a + binary; one key per item, ordered by fractional indexing. - {this, is <<"some">>, term, 4.5, %{3 => 2}} + [#{id => "item1", value => 1}, #{id => "item2", value => 2}] -4. **Nesting**: the encoder handles nested structures recursively. For - example, an assigns map containing a components list nests both encodings - in the key path. +3. **Term**: anything else, via `term_to_binary`, chunked into 100KB values. - #{ - mykey => <<"my value">>, - mylist => [ - #{id => "item1", value => 1}, - #{id => "item2", value => 2} - ] - } +4. **Nesting**: encodings nest recursively, e.g. a components list inside an + assigns map. -When writing updates, diffs are generated by comparing old and new state: - -- **Assigns map**: only changed entries are written; removed entries are - cleared. -- **Components list**: only changed items are written; ordering changes - update fractional indices. -- **Term**: full rewrite (no diffing). - -If the encoding type changes between updates, the old keys are cleared and the -new encoding is written in full. +On update, assigns maps and components lists write only changed entries +(clearing removed ones; reordering updates fractional indices); terms are +rewritten in full. If the encoding type changes, the old keys are cleared and +the new encoding written in full. ### 4.6 Caching -Each consumer process can maintain an in-memory cache of the state paired with -its versionstamp. On subsequent messages, if the cached versionstamp matches -the current database version, the state is reused without a read operation. -This eliminates redundant reads when processing multiple messages in -sequence. - -The cache is invalidated when the process detects that another consumer has -modified the state. +A consumer may cache the state with its versionstamp and skip the read when the +versionstamp still matches the database. The cache is invalidated when another +consumer modifies the state. ### 4.7 Request flows -**Call request flow:** - -1. Some Erlang process calls `dgen_server:call/3` (the calling entity). -2. A call-request is pushed onto the message-queue, along with the from-key. -3. The from-key and watch are returned to the calling entity. -4. One of the dgen_server consumers consumes the call-request. -5. The consumer retrieves the state. -6. The consumer calls the `handle_call/3` function on the module that - implements the dgen_server behaviour. -7. The callback returns a new state and an optional list of side-effects. -8. The consumer updates the state. -9. The consumer checks whether the reply-sentinel-key still exists (the - caller may have timed out and cleared it). If present, the consumer writes - `{reply, Reply}` as a chunked term under the from-key; if absent, the write - is skipped. -10. The consumer commits the transaction. -11. (concurrent with 12) The consumer executes the side-effects. -12. (concurrent with 11) The calling entity receives the watch notification, - reads the chunked reply via `get_range`, and clears the reply keys. - -On timeout, the calling entity clears the reply keys to prevent durable key -leaks. The callback still runs and state still mutates (just like -`gen_server` — a timed-out call still executes), only the orphan reply key is -eliminated. - -**Cast request flow:** - -1. A cast-request is pushed onto the message-queue. -2. One of the dgen_server consumers consumes the cast-request. -3. The consumer retrieves the state. -4. The consumer calls the `handle_cast/2` function on the module that - implements the dgen_server behaviour. -5. The callback returns a new state and an optional list of side-effects. -6. The consumer updates the state. -7. The consumer commits the transaction. -8. The consumer executes the side-effects. - -**Lock flow:** - -1. The callback returns `{lock, NewState}`. -2. The consumer commits `NewState` to the DB, dequeues the message, **and** - writes a lock key (`{Tuid, <<"k">>}`, holding `{Timestamp, Token}`) — all in - the same transaction. `Token` is unique to this acquisition; the consumer - retains it along with the message's queue envelope. -3. Any other consumer that tries to consume from the queue reads - `is_locked = true` and parks itself on a DB watch instead of processing. -4. The consumer that set the lock calls `handle_locked/4` with the **same** - event type and message that triggered the lock, plus the committed - `NewState`. This is the synchronous coordination window. -5. The consumer commits the result of `handle_locked/4` in a **second** - transaction, conditional on the lock key still carrying its `Token`. If it - does not, the consumer was busted while it ran: nothing is written, the - message from step 2 is pushed back onto the queue in that same transaction, - and the consumer exits with `{lock_fenced, Requeued}`. -6. After `handle_locked/4` returns (regardless of its return value), the lock - key is cleared and the queue watch is notified in an `after` block — - always, even on exception. The clear is conditional on the key still - carrying this consumer's `Token`: if it does not, this consumer was busted - while it ran and the lock now belongs to a successor, so it is left alone. -7. Parked consumers wake up, see `is_locked = false`, and resume normal queue - consumption. - -The lock therefore guarantees that between committing a state change and -resuming queue consumption, one designated node performs a synchronous -coordination step with no risk of another node racing ahead. +**Call:** the caller enqueues the request with its from-key and watches the +reply-sentinel-key. A consumer runs `handle_call/3`, writes the new state and, +if the sentinel still exists, the reply; then commits. The consumer then runs +the actions while the caller reads and clears the reply. A timed-out caller +clears its reply keys so none leak; as with `gen_server`, the callback still +runs and the state still changes. + +**Cast:** enqueue; a consumer runs `handle_cast/2`, commits, then runs the +actions. + +**Lock:** + +1. The callback returns `{lock, NewState}`. In one transaction the consumer + commits `NewState`, dequeues the message, and writes a lock key holding + `{Timestamp, Token}` (`Token` unique to this acquisition). +2. Other consumers see the lock and wait on a DB watch. +3. The holder calls `handle_locked/4` with the triggering event, message, and + committed `NewState`. +4. The holder commits the result in a second transaction, only if the lock key + still carries its `Token`. Otherwise nothing is written, the message is + requeued in that transaction, and the holder exits with + `{lock_fenced, requeued}` (`not_requeued` for a priority or inline lock). +5. Whatever `handle_locked/4` returns, even on exception, the lock key is + cleared (only if it still carries this holder's `Token`) and the queue watch + notified, so waiting consumers resume. ### 4.8 Internal terms -Reference material for reading the implementation: - -- **key-bin**: a key-tuple encoded into a binary using the tenant subspace. -- **waiting-key**: a prefix key-tuple that contains all entities waiting on - the result of some call-request. -- **from-key**: a key-tuple that uniquely identifies a single entity waiting - on the result of some call-request. Includes a system-time timestamp - (seconds) so abandoned call keys can be garbage-collected using a - time-based heuristic. Structure: `{WaitingKey..., Timestamp, - term_to_binary(Ref)}`. -- **reply-sentinel-key**: the first chunk key of the reply term under the - from-key. The reply is stored using chunked term encoding - (`{From, <<"t">>, 0}`, `{From, <<"t">>, 1}`, ...) so replies can exceed the - backend's single-value size limit. The backend watch is placed on the - reply-sentinel-key (chunk 0). The client reads the reply via `get_range` and - clears it via `clear_range`. -- **quid**: unique tuple identifier for the queue. It is a prefix key-tuple - that contains all key-values for the message-queue. -- **item-key**: a key-tuple that identifies an item in the queue (a - call-request or cast-request). -- **push-key** / **pop-key**: key-tuples tracking the number of pushes onto, - and pops from, the queue. +- **from-key**: `{WaitingKey..., Timestamp, term_to_binary(Ref)}` identifies + one waiting caller under the **waiting-key** prefix; the timestamp lets + abandoned keys be garbage-collected. +- **reply-sentinel-key**: chunk 0 of the reply, stored as `{From, <<"t">>, N}` + so replies may exceed the backend value-size limit; the caller watches it. +- **quid**: prefix of all queue keys: **item-keys** (queued requests) and the + **push-key / pop-key** counters. --- ## 5. Crash and failure behaviour -**Key guarantee:** standard `call` and `cast` messages are processed -**exactly-once** under normal operation. In the event of a crash before the -transaction commits, the message is retried — so in crash scenarios, delivery -is **at-least-once**. When `dead_letter_threshold` is set, retries are -bounded by that limit. Design your callbacks to be idempotent when possible. - -**During `init/1`:** - -- If the first `init/1` crashes, the gen_server process exits before any - state is persisted. -- When restarted, `init/1` runs again from scratch. -- No durable state exists yet, so there's nothing to recover. - -**During transactional callbacks (`handle_call`, `handle_cast`, -`handle_info`):** - -- The database transaction is automatically aborted — no state changes are - committed. -- For `call` and `cast`: the message remains in the durable queue and will be - retried by the next consumer. If `dead_letter_threshold` is set, each - failed attempt increments a counter embedded in the message envelope; once - the counter reaches the threshold, the message is moved to the dead-letter - queue instead of being retried (see below). -- For `priority_call` and `priority_cast`: the message is lost (it never - entered the queue). -- For `handle_info`: the Erlang message is lost (info messages are not - durable). -- State remains unchanged from before the callback was invoked. - -**Dead-letter queue:** - -A *poison message* is a queue message that consistently crashes consumers. To -enable bounded retries, set `dead_letter_threshold` to a positive integer. -Each message envelope carries an attempt counter; when the counter reaches -the threshold, the message is moved to the dead-letter queue (DLQ) in -FoundationDB rather than being retried: - -- For `call` messages, the blocked caller raises `{dead_letter, N}` where `N` - is the attempt count. -- The optional `handle_dead_letter/2` callback is invoked with the original - message and attempt count, if defined. -- A warning is logged. - -Dead-lettering is **disabled by default** (`dead_letter_threshold: infinity`). -Enable it with the start option: - - - -### Erlang - -```erlang -dgen_server:start(MyMod, [], [{tenant, Tenant}, {dead_letter_threshold, 3}]) -``` - -### Elixir +Standard `call` and `cast` messages are processed **exactly-once** under normal +operation and **at-least-once** if a consumer crashes before commit (bounded by +`dead_letter_threshold` when set). Make callbacks idempotent where possible. -```elixir -DGen.Server.start_link(MyMod, [], tenant: tenant, dead_letter_threshold: 3) -``` +**During `init/1`:** nothing is persisted yet; on restart `init/1` runs again +from scratch. - +**During `handle_call`, `handle_cast`, `handle_info`:** the transaction aborts +and state is unchanged. A `call`/`cast` stays in the queue and is retried by +the next consumer (subject to `dead_letter_threshold`). A `priority_call`, +`priority_cast`, or `handle_info` message is lost. -**Coordinating with the supervisor's restart intensity:** +**Dead-letter queue.** A *poison message* consistently crashes consumers. With +`dead_letter_threshold` set to a positive integer, each failed attempt +increments a counter in the message envelope; at the threshold the message is +moved to the dead-letter queue (DLQ) instead of retried, and: -Each consumer crash counts as one restart from the supervisor's perspective. -OTP supervisors enforce a *restart intensity* — a maximum number of restarts -allowed within a sliding time window (`max_restarts` / `max_seconds`, -defaulting to `3` restarts in `5` seconds). If the supervisor reaches this -limit before `dead_letter_threshold` is hit, the supervisor itself terminates -rather than the message being dead-lettered. +- a blocked `call` caller raises `{dead_letter, N}` (`N` = attempt count); +- the optional `handle_dead_letter/2` callback receives the message and count; +- a warning is logged. -To ensure dead-lettering takes effect, configure the supervisor so that it -tolerates at least `dead_letter_threshold` restarts within the window. A -practical approach is to set `max_restarts >= dead_letter_threshold` with a -`max_seconds` value long enough to cover the expected crash-restart cycle -time: +Dead-lettering is **disabled by default** (`infinity`): ### Erlang ```erlang -%% Allow up to 5 restarts in 60 seconds — enough headroom for a threshold of 3. -{ok, _} = supervisor:start_link({local, my_sup}, my_sup, []), - -%% In the supervisor's init/1: -SupFlags = #{strategy => one_for_one, intensity => 5, period => 60}, +dgen_server:start(MyMod, [], [{tenant, Tenant}, {dead_letter_threshold, 3}]) ``` ### Elixir ```elixir -# Allow up to 5 restarts in 60 seconds — enough headroom for a threshold of 3. -Supervisor.start_link(children, - strategy: :one_for_one, - max_restarts: 5, - max_seconds: 60 -) +DGen.Server.start_link(MyMod, [], tenant: tenant, dead_letter_threshold: 3) ``` -With `dead_letter_threshold: infinity` (the default), poison messages produce -an unbounded crash loop. The supervisor will eventually exhaust its restart -intensity and terminate, which is standard OTP crash-loop behavior. Set a -finite threshold to bound the loop and keep the supervisor alive. - -**Inspecting and managing the dead-letter queue:** +**Supervisor restart intensity.** Each consumer crash is a supervisor restart. +If the supervisor's intensity limit (default 3 restarts in 5 seconds) is hit +before `dead_letter_threshold`, the supervisor terminates instead. Set +`intensity`/`max_restarts >= dead_letter_threshold` with a period long enough +to cover the crash-restart cycle, e.g. 5 restarts in 60 seconds for a threshold +of 3. With `infinity`, a poison message crash-loops until the supervisor gives +up. -Dead-lettered messages accumulate in FoundationDB and are not automatically -cleared. An operator can inspect and manage them from a remote shell using -functions in `dgen_queue`. The `Quid` for a server is obtained with -`dgen_server:get_quid/1` (Erlang) or `:dgen_server.get_quid/1` (Elixir), -passing the `tuid` the server was started with. +**Managing the DLQ.** Dead-lettered messages are never cleared automatically. +Operators manage them with `dgen_queue`, using the `Quid` from +`dgen_server:get_quid(Tuid)`: @@ -594,21 +318,10 @@ passing the `tuid` the server was started with. ```erlang Quid = dgen_server:get_quid(Tuid), - -%% Inspect — returns [{Key, Envelope, AttemptCount, TimestampMs}] -Entries = dgen_queue:peek_dlq(Tenant, Quid), - -%% Count without decoding +Entries = dgen_queue:peek_dlq(Tenant, Quid), %% [{Key, Envelope, Attempts, TsMs}] dgen_queue:dlq_length(Tenant, Quid), - -%% Requeue one entry (resets attempt count to 0, atomic with DLQ delete) -{Key, _Envelope, _N, _Ts} = hd(Entries), dgen_queue:requeue_dlq_entry(Tenant, Quid, Key), %% ok | {error, not_found} - -%% Discard one entry permanently dgen_queue:delete_dlq_entry(Tenant, Key), - -%% Discard all entries for the queue dgen_queue:purge_dlq(Tenant, Quid). ``` @@ -616,62 +329,32 @@ dgen_queue:purge_dlq(Tenant, Quid). ```elixir quid = :dgen_server.get_quid(tuid) - -# Inspect — returns [{key, envelope, attempt_count, timestamp_ms}] -entries = :dgen_queue.peek_dlq(tenant, quid) - -# Count without decoding +entries = :dgen_queue.peek_dlq(tenant, quid) # [{key, envelope, attempts, ts_ms}] :dgen_queue.dlq_length(tenant, quid) - -# Requeue one entry (resets attempt count to 0, atomic with DLQ delete) -{key, _envelope, _n, _ts} = hd(entries) :dgen_queue.requeue_dlq_entry(tenant, quid, key) # :ok | {:error, :not_found} - -# Discard one entry permanently :dgen_queue.delete_dlq_entry(tenant, key) - -# Discard all entries for the queue :dgen_queue.purge_dlq(tenant, quid) ``` -`requeue_dlq_entry/3` is atomic: it pushes the envelope back onto the main -queue with the attempt count reset to zero and deletes the DLQ entry in a -single FDB transaction. If the root cause of the crashes has been fixed and -the server has been redeployed, requeueing allows the message to be retried -from a clean state. - -**During `handle_locked`:** - -- `handle_locked` executes outside a transaction, so previous state changes - have already been persisted. -- If the crash is an Erlang/Elixir throw, then the lock is cleared before the - process exits. -- If the crash is a system disruption such as SegFault, OOM, or sudden power - loss, the lock is not cleared and the dgen_server is deadlocked. Manual - intervention is required to clear the lock. -- In either case, the triggering message has been consumed from the queue, so - it will not be retried. - -**During action execution:** - -- Actions run after the transaction commits, so state changes are already - persisted. -- If an action crashes, the state update succeeds but remaining actions are - not executed. -- The message has been consumed from the queue and will not be retried. - -**Supervisor restart:** - -- When a dgen_server is restarted by a supervisor, it reads existing state - from the database. -- If state exists, `init/1` is called, but the initial state is ignored. The - server resumes with the persisted state. -- The process immediately begins consuming any queued messages, if it's - configured to do so. -- Multiple processes can safely consume from the same queue; they coordinate - via database transactions. +`requeue_dlq_entry/3` atomically moves the envelope back to the main queue with +its attempt count reset to zero, for retry after the cause is fixed. + +**During `handle_locked`:** the state committed with the lock is already +persisted and the triggering message already dequeued. On an exception the lock +is cleared before the process exits and the message is not retried. If the +process dies without clearing it (SIGKILL, OOM, power loss), the lock blocks +the queue until `lock_timeout` expires (forever with `infinity`, requiring +manual intervention). If the holder is fenced, its message is requeued and +retried (§4.4). + +**During actions:** state is already committed and the message consumed. A +crashing action stops the remaining actions; nothing is retried. + +**Supervisor restart:** if durable state exists, `init/1` is called but its +state is ignored; the process resumes from the persisted state and consumes +(if configured to) alongside any other consumers. --- @@ -680,60 +363,52 @@ from a clean state. These are normative. Behaviour that violates one of these is a defect. 1. **Durability.** State and queued messages persist in the backend, - independent of the lifetime of any Erlang process, node, or cluster. + independent of any Erlang process, node, or cluster. 2. **Delivery.** Standard `call`/`cast` messages are processed exactly-once under normal operation. If a consumer crashes before its transaction - commits, the message is retried by another consumer — at-least-once - delivery in that scenario, bounded by `dead_letter_threshold` when - configured (§5). + commits, the message is retried by another consumer: at-least-once in that + case, bounded by `dead_letter_threshold` when configured (§5). 3. **Serializability.** Standard messages are committed with strict - serializability via the durable queue: the state read and write for a - given batch happen inside one transaction. + serializability via the durable queue: a batch's state read and write happen + in one transaction. 4. **Priority messages trade ordering for immediacy.** `priority_call/2`, - `priority_cast/2`, and `handle_info/2` always bypass the queue and the - lock, so they are not ordered with respect to the queued stream. + `priority_cast/2`, and `handle_info/2` always bypass the queue and the lock, + so they are not ordered with respect to the queued stream. 5. **Actions run exactly once per committed batch,** strictly after that - batch's commit, and only once the state they reference is durable. + batch's commit, once the state they reference is durable. 6. **Lock exclusivity.** While a `dgen_server` is locked, only the lock - holder's `handle_locked/4`, priority messages, and `handle_info/2` - execute; other standard `call`/`cast` messages wait until the lock clears. -7. **Callback purity.** `handle_call/3`, `handle_cast/2`, and - `handle_info/2` execute inside a database transaction and must not - perform side effects — those belong in the `Actions` list (§4.3) or in - `handle_locked/4` (§4.4), the two paths that run outside a transaction. + holder's `handle_locked/4`, priority messages, and `handle_info/2` execute; + other standard `call`/`cast` messages wait until the lock clears. +7. **Callback purity.** `handle_call/3`, `handle_cast/2`, and `handle_info/2` + run inside a database transaction and must not perform side effects; those + belong in `Actions` (§4.3) or `handle_locked/4` (§4.4), which run outside a + transaction. --- ## 7. Configuration -Options passed via the `Opts` proplist to `start/3`, `start/4`, `start_link/3`, -or `start_link/4`: +Options in the `Opts` proplist of `start/3,4` and `start_link/3,4`: | Option | Default | Meaning | |---|---|---| | `tenant` | — (required) | `{DbHandle, Dir}` pair identifying the backend subspace. | -| `consume` | `true` | Whether this process consumes messages from the queue, or only publishes them. | -| `reset` | `false` | When `true`, re-initialise the durable state even if it already exists. | -| `dead_letter_threshold` | `infinity` | Number of consecutive processing failures before a message is moved to the dead-letter queue instead of retried (§5). | -| `consume_k` | `1` | Maximum number of queued messages processed per transaction (§4.2). | -| `lock_timeout` | `infinity` | Maximum milliseconds a distributed lock may be held before other consumers treat it as stale (§4.4). | +| `consume` | `true` | Whether this process consumes from the queue, or only publishes. | +| `reset` | `false` | When `true`, re-initialise the durable state even if it exists. | +| `dead_letter_threshold` | `infinity` | Consecutive processing failures before a message is moved to the dead-letter queue instead of retried (§5). | +| `consume_k` | `1` | Maximum queued messages processed per transaction (§4.2). | +| `lock_timeout` | `infinity` | Maximum milliseconds a lock may be held before other consumers treat it as stale (§4.4). | --- ## 8. When `dgen_server` is (and isn't) a good fit -**Good fit when** you want the plain `gen_server` programming model but need -the state — and any in-flight messages — to survive process, node, or cluster -death; when a database round-trip per message is an acceptable cost for that -durability and multi-node availability; and when your side effects can be -deferred to the post-commit `Actions` list, or performed inside the atomic -`handle_locked/4` window, rather than issued inline during a callback. - -**Reconsider when** you need sub-millisecond, purely in-memory latency with no -network or database hop — a plain `gen_server` is faster for state that -doesn't need to survive a restart; when your callback needs to perform -ordinary, non-deferrable side effects directly inside `handle_call`/ -`handle_cast` (`dgen_server` requires those to go through `Actions` or -`handle_locked/4`, since callbacks themselves run inside a database -transaction); or when you cannot run, or do not want to operate, a +**Good fit when** you want the `gen_server` model but need state and in-flight +messages to survive process, node, or cluster death; a database round-trip per +message is acceptable; and side effects can be deferred to `Actions` or done in +`handle_locked/4`. + +**Reconsider when** you need purely in-memory latency (a plain `gen_server` is +faster for state that need not survive a restart); your callbacks need +non-deferrable side effects inline; or you cannot or do not want to operate a strongly-consistent external database. diff --git a/formal/DgenRegistryReplication.cfg b/formal/DgenRegistryReplication.cfg index 8d6d4a8..edd80b1 100644 --- a/formal/DgenRegistryReplication.cfg +++ b/formal/DgenRegistryReplication.cfg @@ -20,17 +20,19 @@ CONSTANTS Heartbeat = FALSE SafeAssumeGather = TRUE SafeAssumeSnap = TRUE + DegradedAssume = FALSE INVARIANTS TypeOK PrefixConsistency UniqueBinding DurableAcked LeaderEpochUnique + DegradedFlagSound CONSTRAINT ChanBound SYMMETRY Symm \* The main model: the FIXED protocol (both SafeAssume* guards TRUE) checked at -\* MaxChanLen = 2 — the channel depth the handoff-gather race needs. All five +\* MaxChanLen = 2 — the channel depth the handoff-gather race needs. All six \* invariants hold exhaustively. This is the proof the fix closes the race the \* pre-fix protocol admitted; DgenRegistryReplicationHandoffRace.cfg is the same \* bounds with both guards FALSE and fails, and the two *Only configs flip each diff --git a/formal/DgenRegistryReplication.tla b/formal/DgenRegistryReplication.tla index f5565f8..c16bd96 100644 --- a/formal/DgenRegistryReplication.tla +++ b/formal/DgenRegistryReplication.tla @@ -33,6 +33,8 @@ CONSTANTS \* durable version key; FALSE = the pre-fix stale gather SafeAssumeSnap, \* TRUE = snapshot apply is version-monotonic; FALSE = the \* pre-fix epoch-only guard (a stale snapshot can rewind) + DegradedAssume, \* TRUE = the gather may also give up waiting for the durable + \* version key (the code's deadline) and assume degraded NoPid, NoMember \* model values ASSUME NoPid \notin Pids @@ -56,17 +58,22 @@ VARIABLES epoch, \* [Members -> Nat]: last epoch learned (set by snapshots/assume) leaderView, \* [Members -> Members \cup {NoMember}]: believed leader; \* leaderView[m] = m <=> m believes it is the leader - deferred, \* [Members -> SUBSET (Names \X Pids \X Nat)]: deferred_yes + deferred, \* [Members -> SUBSET (Names \X Pids \X Nat)]: intents awaiting_version pendingAcks, \* [Members -> SUBSET (Names \X Pids \X Nat)]: leader-side \* direct registrations awaiting a replicate_ack resyncReq, \* [Members -> Members \cup {NoMember}]: outstanding resync \* target (the gapped stream's sender), NoMember = none + \* ---- ghost: what each member's last assume knew ---- + degraded, \* [Members -> BOOLEAN]: the code's `degraded` flag, defined as + \* "gathered max version < durable version key" (§5.6) + complete, \* [Members -> BOOLEAN]: the reconstruction it adopted was the + \* true committed map, histMap[dbVersion] \* ---- the network ---- chan \* [Members -> [Members -> Seq(Msg)]]: per-ordered-pair FIFO vars == <> + deferred, pendingAcks, resyncReq, chan, degraded, complete>> (***************************************************************************) (* Basic definitions *) @@ -115,6 +122,8 @@ Init == /\ pendingAcks = [m \in Members |-> {}] /\ resyncReq = [m \in Members |-> NoMember] /\ chan = [s \in Members |-> [r \in Members |-> <<>>]] + /\ degraded = [m \in Members |-> FALSE] + /\ complete = [m \in Members |-> TRUE] (***************************************************************************) (* Elections and handoff *) @@ -137,9 +146,11 @@ Elect(m) == \* Modeled atomically; the gather reaches every live member (see module \* header). Freshest-wins: adopt the max-applied_version replica (version \* tie => identical content, which PrefixConsistency asserts). Deferred -\* acks are rejected and pending direct acks resolved-as-rejected, matching -\* do_leader_changed/reject_forwards; then snapshots fan out to every other -\* live member from this process (FIFO with later broadcasts). +\* acks are dropped and pending direct acks resolved-as-rejected; then snapshots +\* fan out to every other live member from this process (FIFO with later +\* broadcasts). (The code re-drives a dropped deferred ack as a fresh forward — +\* dgen_registry_intents:redrive/5 — which is just another RegisterForward here, +\* so dropping it is the sound abstraction for the safety properties.) \* THE FIX, first half (SafeAssumeGather): the assuming leader must not reconstruct \* from a gather that is behind the durable version key. `maxV` is the freshest \* applied_version any live member reports; `dbVersion` is the committed frontier @@ -152,15 +163,21 @@ Elect(m) == \* race (see formal/README.md). In the real code this is the new leader comparing \* its gathered MaxVersion against the durable version key and retrying/waiting the \* gather until caught up (bounded, then assume-degraded — the availability -\* tradeoff, out of this safety model's scope, analogous to DegradeOpen). +\* tradeoff, analogous to DegradeOpen). With DegradedAssume that deadline is +\* modeled too: the assume may proceed behind the version key at any point. It +\* then records the code's `degraded` flag (maxV < dbVersion) and, as ghost +\* state, whether the adopted map really was the committed one — which +\* DegradedFlagSound checks the flag against. AssumeGather(m) == /\ alive[m] /\ dbLeader = m /\ epoch[m] < dbEpoch /\ LET maxV == MaxOf({appliedVer[x] : x \in Live}) - IN /\ (SafeAssumeGather => maxV >= dbVersion) + IN /\ (SafeAssumeGather => (maxV >= dbVersion \/ DegradedAssume)) + /\ degraded' = [degraded EXCEPT ![m] = (maxV < dbVersion)] /\ \E src \in Live : /\ appliedVer[src] = maxV + /\ complete' = [complete EXCEPT ![m] = (rep[src] = histMap[dbVersion])] /\ rep' = [rep EXCEPT ![m] = rep[src]] /\ appliedVer' = [appliedVer EXCEPT ![m] = maxV] /\ LET snap == [type |-> "snap", rep |-> rep[src], @@ -246,18 +263,18 @@ RegisterDirect(l, n, p) == leaderView, deferred, resyncReq>> (***************************************************************************) -(* Broadcast delivery — mirrors apply_bcast/6 case for case *) +(* Broadcast delivery — mirrors dgen_registry_replica:offer_batch/5 *) (***************************************************************************) \* Contiguous (prev = applied): apply the row, advance applied_version -\* (bump_applied), flush any deferred acks the advance releases +\* (offer_batch's `applied`), flush any deferred acks the advance releases \* (flush_deferred), cancel an outstanding resync. Matches the shipped -\* apply_bcast/6 case for case: since a batch ships as ONE message per commit +\* offer_batch/5 case for case: since a batch ships as ONE message per commit \* version, `PrevV =:= Applied` is the only apply clause — a re-delivered \* duplicate falls to the `V =< Applied` drop below, exactly as in the code \* (the old `m.ver = appliedVer[f]` apply disjunct modeled the pre-fix -\* multi-message batches and was dead in every config; see formal/README.md, -\* "A closed abstraction gap"). +\* multi-message batches and was dead in every config; see the sim README, +\* finding 1). RecvBcastApply(s, f) == /\ alive[f] /\ HeadIs(s, f, "bcast") @@ -340,7 +357,7 @@ HeartbeatBcast(l) == alive, rep, appliedVer, epoch, leaderView, deferred, pendingAcks, resyncReq>> -\* Receipt mirrors apply_bcast/6 on an empty batch with PrevV = Version: +\* Receipt mirrors offer_batch/5 on an empty batch with PrevV = Version: \* caught-up (ver = applied) takes the apply clause, which applies no rows and \* re-bumps to the same version — observably just cancel_resync (flush_deferred \* releases nothing: every deferred ack at or below applied was flushed when @@ -369,7 +386,7 @@ RecvHeartbeat(s, f) == \* on a leadership change (an unknown-Ref reply is ignored). Otherwise: \* applied >= ver -> ack now (the broadcast/snapshot already delivered the \* row; the follower is a version-visible second holder); -\* gapped + guard -> defer (deferred_yes), released by flush_deferred; +\* gapped + guard -> defer (awaiting_version), released by on_version; \* gapped, no guard-> the pre-guard bug: optimistic row write + immediate \* ack. The row exists but at a version the freshest-wins \* gather ignores — the silent-loss window. @@ -466,8 +483,8 @@ DegradeTimeout(l) == \* frontier >= any follower's applied_version), and a legitimate resync only ever \* moves a gapped follower FORWARD, so the guard rejects only genuinely stale \* snapshots. In the real code this is the `Version >= CurrentVersion` conjunct -\* beside `Epoch >= CurrentEpoch` in handle_cast({apply_names_snapshot, ...}) -\* (dgen_registry_member.erl) — shipped. +\* beside `Epoch >= CurrentEpoch` in dgen_registry_replica:offer_snapshot/4 +\* (dgen_registry_replica.erl) — shipped. RecvSnap(s, f) == /\ alive[f] /\ HeadIs(s, f, "snap") @@ -550,9 +567,10 @@ DropMsg(s, r) == (* Next / Spec *) (***************************************************************************) -Next == +\* Every action but AssumeGather leaves the assume-time ghost state alone. +Step == \/ \E m \in Members : - \/ Elect(m) \/ AssumeGather(m) \/ Crash(m) \/ DegradeTimeout(m) + \/ Elect(m) \/ Crash(m) \/ DegradeTimeout(m) \/ HeartbeatBcast(m) \/ \E l \in Members, n \in Names, p \in Pids : \/ RegisterDirect(l, n, p) @@ -563,6 +581,10 @@ Next == \/ RecvReply(s, r) \/ RecvSync(s, r) \/ RecvAck(s, r) \/ RecvSnap(s, r) \/ ServeResync(s, r) \/ DropMsg(s, r) +Next == + \/ \E m \in Members : AssumeGather(m) + \/ Step /\ UNCHANGED <> + Spec == Init /\ [][Next]_vars (***************************************************************************) @@ -584,6 +606,8 @@ TypeOK == /\ resyncReq \in [Members -> Members \cup {NoMember}] /\ acked \subseteq Regs /\ crashes \in 0..MaxCrashes + /\ degraded \in [Members -> BOOLEAN] + /\ complete \in [Members -> BOOLEAN] /\ \A s \in Members : \A r \in Members : \A i \in 1..Len(chan[s][r]) : chan[s][r][i] \in Msg @@ -612,6 +636,21 @@ Holders(a) == DurableAcked == \A a \in acked : Cardinality(Holders(a)) >= 1 + MaxCrashes - crashes +\* The `degraded` flag is sound (§5.6): an assume that did NOT flag itself +\* degraded — its gathered max version reached the durable version key — adopted +\* exactly the committed map, so no committed binding is missing from it. This +\* is what lets `reject_when_degraded` key on versions instead of on which members +\* answered the gather: by PrefixConsistency the freshest replica at the key IS +\* the committed map. (The converse is not claimed: a degraded assume may still +\* happen to hold everything.) +DegradedFlagSound == + \A m \in Members : ~degraded[m] => complete[m] + +\* NOT an invariant — the reachability canary for DegradedFlagSound, which a model +\* that never assumes degraded would satisfy vacuously. Asserted as an +\* INVARIANT by the DegradedReachable config, which MUST FAIL. +NoDegradedAssume == \A m \in Members : ~degraded[m] + \* Fencing sanity: two members may both believe they lead only under \* different epochs (§5.1's monotonic fencing token). LeaderEpochUnique == diff --git a/formal/DgenRegistryReplicationAckReachable.cfg b/formal/DgenRegistryReplicationAckReachable.cfg index 5a56c2e..a142298 100644 --- a/formal/DgenRegistryReplicationAckReachable.cfg +++ b/formal/DgenRegistryReplicationAckReachable.cfg @@ -20,6 +20,7 @@ CONSTANTS Heartbeat = FALSE SafeAssumeGather = TRUE SafeAssumeSnap = TRUE + DegradedAssume = FALSE INVARIANTS NoAcks CONSTRAINT ChanBound diff --git a/formal/DgenRegistryReplicationDegradeOpen.cfg b/formal/DgenRegistryReplicationDegradeOpen.cfg index cdbca89..99ae8c8 100644 --- a/formal/DgenRegistryReplicationDegradeOpen.cfg +++ b/formal/DgenRegistryReplicationDegradeOpen.cfg @@ -20,6 +20,7 @@ CONSTANTS Heartbeat = TRUE SafeAssumeGather = TRUE SafeAssumeSnap = TRUE + DegradedAssume = FALSE INVARIANTS DurableAcked CONSTRAINT ChanBound diff --git a/formal/DgenRegistryReplicationDegradeOpenSafety.cfg b/formal/DgenRegistryReplicationDegradeOpenSafety.cfg index 917b3aa..3fa1c47 100644 --- a/formal/DgenRegistryReplicationDegradeOpenSafety.cfg +++ b/formal/DgenRegistryReplicationDegradeOpenSafety.cfg @@ -20,6 +20,7 @@ CONSTANTS Heartbeat = FALSE SafeAssumeGather = TRUE SafeAssumeSnap = TRUE + DegradedAssume = FALSE INVARIANTS TypeOK PrefixConsistency diff --git a/formal/DgenRegistryReplicationDegradedAssume.cfg b/formal/DgenRegistryReplicationDegradedAssume.cfg new file mode 100644 index 0000000..013bb0c --- /dev/null +++ b/formal/DgenRegistryReplicationDegradedAssume.cfg @@ -0,0 +1,41 @@ +SPECIFICATION Spec +CONSTANTS + m1 = m1 + m2 = m2 + m3 = m3 + Members = {m1, m2, m3} + n1 = n1 + Names = {n1} + p1 = p1 + p2 = p2 + Pids = {p1, p2} + NoPid = NoPid + NoMember = NoMember + MaxVersion = 3 + MaxEpoch = 2 + MaxCrashes = 1 + MaxChanLen = 2 + VersionGuardedAck = TRUE + DegradeOpen = FALSE + Heartbeat = FALSE + SafeAssumeGather = TRUE + SafeAssumeSnap = TRUE + DegradedAssume = TRUE +INVARIANTS + TypeOK + PrefixConsistency + LeaderEpochUnique + DegradedFlagSound +CONSTRAINT ChanBound +SYMMETRY Symm + +\* The handoff gather's deadline, modeled: an assume may give up waiting for the +\* durable version key and proceed degraded (DegradedAssume = TRUE), as the code +\* does after ?GATHER_TIMEOUT. Checks that the `degraded` flag, defined on +\* versions alone (gathered max < durable key), is sound: an assume that did not +\* flag itself adopted exactly the committed map (DegradedFlagSound). +\* +\* UniqueBinding and DurableAcked are deliberately absent: a degraded assume CAN +\* lose a binding — that is what the flag reports, and what §5.6 repairs and +\* `reject_when_degraded` guards. DgenRegistryReplicationDegradedReachable.cfg +\* is this config's canary: it proves degraded assumes are actually reached. diff --git a/formal/DgenRegistryReplicationDegradedReachable.cfg b/formal/DgenRegistryReplicationDegradedReachable.cfg new file mode 100644 index 0000000..7faca3d --- /dev/null +++ b/formal/DgenRegistryReplicationDegradedReachable.cfg @@ -0,0 +1,32 @@ +SPECIFICATION Spec +CONSTANTS + m1 = m1 + m2 = m2 + m3 = m3 + Members = {m1, m2, m3} + n1 = n1 + Names = {n1} + p1 = p1 + p2 = p2 + Pids = {p1, p2} + NoPid = NoPid + NoMember = NoMember + MaxVersion = 3 + MaxEpoch = 2 + MaxCrashes = 1 + MaxChanLen = 2 + VersionGuardedAck = TRUE + DegradeOpen = FALSE + Heartbeat = FALSE + SafeAssumeGather = TRUE + SafeAssumeSnap = TRUE + DegradedAssume = TRUE +INVARIANTS + NoDegradedAssume +CONSTRAINT ChanBound +SYMMETRY Symm + +\* NOT a mutation -- a reachability canary, and it must FAIL. DegradedFlagSound +\* is vacuously true of a model that never assumes degraded; this config asserts +\* that no assume ever is, under DgenRegistryReplicationDegradedAssume.cfg's +\* bounds. TLC finding a counterexample proves the degraded path is exercised. diff --git a/formal/DgenRegistryReplicationHandoffGatherOnly.cfg b/formal/DgenRegistryReplicationHandoffGatherOnly.cfg index 07b98b5..bc3e40a 100644 --- a/formal/DgenRegistryReplicationHandoffGatherOnly.cfg +++ b/formal/DgenRegistryReplicationHandoffGatherOnly.cfg @@ -20,6 +20,7 @@ CONSTANTS Heartbeat = TRUE SafeAssumeGather = FALSE SafeAssumeSnap = TRUE + DegradedAssume = FALSE INVARIANTS DurableAcked UniqueBinding diff --git a/formal/DgenRegistryReplicationHandoffRace.cfg b/formal/DgenRegistryReplicationHandoffRace.cfg index 0648e4e..4e5616e 100644 --- a/formal/DgenRegistryReplicationHandoffRace.cfg +++ b/formal/DgenRegistryReplicationHandoffRace.cfg @@ -20,6 +20,7 @@ CONSTANTS Heartbeat = TRUE SafeAssumeGather = FALSE SafeAssumeSnap = FALSE + DegradedAssume = FALSE INVARIANTS DurableAcked UniqueBinding diff --git a/formal/DgenRegistryReplicationHandoffRaceUnique.cfg b/formal/DgenRegistryReplicationHandoffRaceUnique.cfg index 7f1cf47..91dc9d1 100644 --- a/formal/DgenRegistryReplicationHandoffRaceUnique.cfg +++ b/formal/DgenRegistryReplicationHandoffRaceUnique.cfg @@ -20,6 +20,7 @@ CONSTANTS Heartbeat = TRUE SafeAssumeGather = FALSE SafeAssumeSnap = FALSE + DegradedAssume = FALSE INVARIANTS UniqueBinding CONSTRAINT ChanBound diff --git a/formal/DgenRegistryReplicationHeartbeat.cfg b/formal/DgenRegistryReplicationHeartbeat.cfg index eaaa6cd..8b76326 100644 --- a/formal/DgenRegistryReplicationHeartbeat.cfg +++ b/formal/DgenRegistryReplicationHeartbeat.cfg @@ -20,6 +20,7 @@ CONSTANTS Heartbeat = TRUE SafeAssumeGather = TRUE SafeAssumeSnap = TRUE + DegradedAssume = FALSE INVARIANTS TypeOK PrefixConsistency diff --git a/formal/DgenRegistryReplicationNoGuard.cfg b/formal/DgenRegistryReplicationNoGuard.cfg index da5e876..dbad23b 100644 --- a/formal/DgenRegistryReplicationNoGuard.cfg +++ b/formal/DgenRegistryReplicationNoGuard.cfg @@ -20,6 +20,7 @@ CONSTANTS Heartbeat = TRUE SafeAssumeGather = TRUE SafeAssumeSnap = TRUE + DegradedAssume = FALSE INVARIANTS UniqueBinding DurableAcked diff --git a/formal/DgenRegistryReplicationSnapMonotonicOnly.cfg b/formal/DgenRegistryReplicationSnapMonotonicOnly.cfg index 5e73839..aa386ab 100644 --- a/formal/DgenRegistryReplicationSnapMonotonicOnly.cfg +++ b/formal/DgenRegistryReplicationSnapMonotonicOnly.cfg @@ -20,6 +20,7 @@ CONSTANTS Heartbeat = TRUE SafeAssumeGather = TRUE SafeAssumeSnap = FALSE + DegradedAssume = FALSE INVARIANTS DurableAcked UniqueBinding diff --git a/formal/README.md b/formal/README.md index 390cb3a..8d4fe9a 100644 --- a/formal/README.md +++ b/formal/README.md @@ -1,85 +1,46 @@ # dgen_registry Formal Model -## Why this exists - -The replication in [`src/dgen_registry_member.erl`](../src/dgen_registry_member.erl) -(design: [`docs/design/dgen_registry_design.md`](../docs/design/dgen_registry_design.md) -§4.5, §5) is a bespoke message-passing protocol whose correctness rests on -interleaving-sensitive arguments: FIFO ordering between broadcasts and replies, -gap detection keeping every replica a prefix of the leader's stream, -version-guarded acks, epoch fencing, and a freshest-wins handoff gather. That -is exactly the class of system **TLA+** was built for (it has the same -heritage in Raft, Paxos, and FoundationDB itself), and its model checker -**TLC** explores every interleaving of a bounded instance exhaustively — the -kind of proof a test suite, which only samples interleavings, cannot give. - -This model exists to check the protocol's safety properties against every -reachable interleaving of a small cluster, not just the ones a test happens to -schedule. It found a genuine race during development (a leadership-handoff -gather could race an in-flight broadcast and silently lose an acknowledged -registration) — see `docs/design/dgen_registry_design.md` §5.7 for how the -shipped protocol closes it, and "Mutation and pre-fix configs" below for how -this model keeps the counterexample reproducible. +A TLA+ model of the replication protocol in +[`src/dgen_registry_member.erl`](../src/dgen_registry_member.erl) (design: +[`docs/design/dgen_registry_design.md`](../docs/design/dgen_registry_design.md) +§4.5, §5). TLC checks its safety properties against every interleaving of a +small bounded cluster. Tests can only sample interleavings; the +[simulation harness](../test/support/sim/README.md) covers the other side: it checks +that the real code implements this protocol. ## What it models -One module, `DgenRegistryReplication.tla`, at the abstraction level of the -`dgen_registry_member` moduledoc: +One module, `DgenRegistryReplication.tla`: -- **Fenced commit** — the leader is sole writer; a commit succeeds only if - the durable leader key + epoch still match. -- **Version-stamped broadcasts** — one per commit version, carrying `{Epoch, - PrevVersion, Version}`; followers apply only *contiguous* broadcasts. The code - matches this: a group commit ships as a single `{names_batch, Ops, …}` message - (see "A closed abstraction gap" below). -- **Resync** — a gapped follower requests a snapshot from its stream's +- **Fenced commit**: the leader is the only writer. A commit succeeds only if + the durable leader key and epoch still match. +- **Version-stamped broadcasts**: one per commit version, carrying `{Epoch, + PrevVersion, Version}`. Followers apply only contiguous broadcasts. +- **Resync**: a follower with a gap requests a snapshot from its stream's sender. -- **Forwarded registrations, version-guarded acks** — a follower answers its - caller only once its replica has applied the batch (`deferred_yes` when - gapped). -- **Direct registrations, replicate-before-ack** — the leader waits for a - follower to confirm before answering `yes` (with a configurable - degrade-open timeout). -- **Leadership handoff** — the new leader gathers the freshest live replica - and fans out a snapshot; fenced against the durable version key and - version-monotonic on apply (§5.7). The two guards are separate constants - (`SafeAssumeGather`, `SafeAssumeSnap`) so each can be mutated alone — and - each is independently necessary (both half-mutation configs fail). -- **The replication heartbeat** — the leader's periodic empty batch stamped - `{Applied, Applied}`, the traffic-independent gap reveal (sim README, - finding 2). Modeled with at most one in flight per leader (the 5s interval - dwarfs delivery; unconstrained bursts are only a state-space multiplier — - see `NoHbInFlight`). A deposed-but-uninformed leader still passes the - `leader = Self` guard, so the stale-heartbeat case is reachable and its - harmlessness is checked rather than assumed. Verified with every invariant - in its own config (`DgenRegistryReplicationHeartbeat.cfg`, `MaxVersion = 2` - — full-depth-with-heartbeats exceeds 60M states, so the main config keeps - full depth with `Heartbeat = FALSE` and this one keeps the mechanism). -- **Failures** — member crash (single-fault, bounded by `MaxCrashes`) and - message loss with Erlang's signal-ordering semantics (per-pair FIFO of - delivered messages, arbitrary loss). - -Deliberately out of scope: unregister/retract/`DOWN` cleanup and the §5.6 -conflict trail + kill budget; metadata/queries/presence (they ride the same -pipeline and add no new replication mechanism); membership joins and partial -gathers (the model assumes the handoff gather reaches every live member), -including the continuing-leader **join fast path** (`onboard_joiner` / -`{peer_joined}` in `dgen_registry_member.erl`), whose soundness leans on -`PrefixConsistency` proved here for handoffs but *assumed* for joins; -`register_replicas > 1` (the model's `RecvAck` resolves a direct registration -on the first distinct follower ack, i.e. `register_replicas = 1`; the code's -distinct-follower counting beyond that is unmodeled); and -Erlang-distribution-only partitions that remove live nodes from the member -set. See the module header in `DgenRegistryReplication.tla` for the full -scope note. - -The scope boundary is not hypothetical: the register/unregister concurrency -this model excludes is exactly where the simulation harness's ack-history -invariant found a real double-free (sim README, finding 7 — one unregister -freeing a name twice, Guarantees 1 and 4 both violated, no fault required). -That is the two methods covering for each other as designed; an unregister -extension to this model (histMap carrying deletions) is the natural v2 if -that machinery keeps changing. +- **Forwarded registrations, version-guarded acks**: a follower answers its + caller only after its replica has applied the batch. +- **Direct registrations, replicate-before-ack**: the leader waits for a + follower to confirm before answering `yes`. The degrade-open timeout is a + constant. +- **Leadership handoff** (§5.7): the new leader gathers the freshest live + replica and fans out a snapshot. Two separate constants guard it: the + durable-version-key fence on the gather (`SafeAssumeGather`) and the + version-monotonic snapshot apply (`SafeAssumeSnap`). +- **Replication heartbeat**: the leader's periodic empty batch stamped + `{Applied, Applied}`, which reveals a gap even when there is no traffic. At + most one is in flight per leader. A deposed leader that hasn't been told yet + can still send one, and that case is checked. +- **Failures**: member crash (bounded by `MaxCrashes`) and message loss with + Erlang's signal ordering (per-pair FIFO, arbitrary loss). + +**Deliberately out of scope**: unregister, retract and `DOWN` cleanup; the +§5.6 conflict trail and kill budget; metadata, queries and presence; membership +joins and partial gathers, including the continuing-leader join fast path +(`assume_fast_path/6` / `onboard_joiner` / `{peer_joined}`); `register_replicas +> 1` (`RecvAck` resolves on the first follower ack); Erlang-distribution-only +partitions. The simulation harness covers register/unregister concurrency. The +module header in the `.tla` has the full scope note. ## Checked properties @@ -90,182 +51,74 @@ that machinery keeps changing. | `LeaderEpochUnique` | two members may both believe they lead only under different epochs | §4.2, §5.1 | | `UniqueBinding` | at most one pid is ever acked `yes` per name | Guarantee 1 | | `DurableAcked` | every acked registration has enough version-visible holders to survive `MaxCrashes` | Guarantee 4, §5.5 | +| `DegradedFlagSound` | an assume that did not flag itself `degraded` (gathered max version reached the durable version key) adopted exactly the committed map | §5.6, §8 `reject_when_degraded` | -All five hold exhaustively in the main config (`DgenRegistryReplication.cfg`, -`MaxChanLen = 2`, ~10.7M distinct states, ~2.5 min on 10 cores) and in the -heartbeat config (`DgenRegistryReplicationHeartbeat.cfg`, `MaxVersion = 2` -with `Heartbeat = TRUE`, ~17.2M distinct states, ~5 min). - -Two configs qualify what a green main run means: - -- **`DgenRegistryReplicationAckReachable.cfg`** (expected FAIL) is the - vacuity canary: every invariant above is trivially satisfied by a protocol - that never acks anyone, so this config asserts `NoAcks` (`acked = {}`) as an - invariant under the main constants and TLC's counterexample is machine proof - that acks are reachable through the unmutated guards. If it ever passes, the - model has gone quiet and green runs mean much less. -- **`DgenRegistryReplicationDegradeOpenSafety.cfg`** (expected PASS) - characterizes the DEFAULT deployment. `strict_replication` defaults to - `false` (`dgen_config.erl`), so production runs degrade-open — the very - configuration the DegradeOpen mutation proves loses `DurableAcked`. This - config checks the other four invariants under `DegradeOpen = TRUE`: the - default mode keeps prefix consistency, uniqueness and epoch fencing, and - gives up exactly two-holder durability (verified: ~21.9M distinct states, - no violation). "All five hold" above is a claim about the fail-closed - setting only — but "does degrade-open also break uniqueness?" now has a - machine-checked answer: within these bounds, no. - -## How to run locally - -```sh -formal/check.sh DgenRegistryReplication # must pass — all invariants, full depth -formal/check.sh DgenRegistryReplicationHeartbeat # must pass — all invariants with the heartbeat on -formal/check.sh DgenRegistryReplicationDegradeOpenSafety # must pass — the default mode's residual guarantees -formal/check.sh DgenRegistryReplicationHandoffRace fail # pre-fix protocol — must fail -formal/check.sh DgenRegistryReplicationHandoffRaceUnique fail # its UniqueBinding half — must fail -formal/check.sh DgenRegistryReplicationHandoffGatherOnly fail # gather fence alone off — must fail -formal/check.sh DgenRegistryReplicationSnapMonotonicOnly fail # snapshot guard alone off — must fail -formal/check.sh DgenRegistryReplicationNoGuard fail # mutation — must fail -formal/check.sh DgenRegistryReplicationDegradeOpen fail # mutation — must fail -formal/check.sh DgenRegistryReplicationAckReachable fail # vacuity canary — must fail -``` - -Requires Java 11+ and nothing else: the TLC jar is **checked in** at -`formal/vendor/tla2tools-v1.8.0.jar` and `check.sh` uses it automatically. (It -used to be downloaded from the tlaplus GitHub release and pinned by sha256 — -which broke twice, because the project republishes assets in place on the same -tag; a vendored jar is immutable in git and hermetic.) To use a different TLC -locally, drop it at `formal/tla2tools.jar` (gitignored, takes precedence) or -set `TLA2TOOLS_JAR` — results here were cross-checked on a 2026 build, which -additionally writes `*_TTrace_*` counterexample files beside the spec; those -and `formal/states/` (TLC scratch) stay gitignored, and `check.sh` gives each -run a private temp metadir so concurrent invocations are safe. The three -passing configs take 2.5-5.5 minutes each on a 10-core machine (timings -below); every expected-fail config finishes in seconds. CI -(`.github/workflows/formal.yml`) runs all ten on every push/PR against the -vendored jar. +"All invariants hold" is a claim about the fail-closed setting +(`strict_replication = true`). The default is degrade-open. See +DegradeOpenSafety below for what it still guarantees. -## Mutation and pre-fix configs +## Configs -Six configs deliberately weaken the model to confirm TLC actually catches -what a guard is meant to prevent — all expected to fail (the seventh -expected-fail config, AckReachable, is the vacuity canary described above, -not a mutation): +Each config is `formal/.cfg`. Passing configs take minutes. Failing ones +take seconds. -- **`DgenRegistryReplicationNoGuard.cfg`** (`VersionGuardedAck = FALSE`): the - pre-`deferred_yes` shape — a follower acks its client optimistically before - its replica has applied the row, so a crash right after can leave zero - version-visible holders. This exact optimistic-insert-and-ack shape is - still live code as the rolling-upgrade legacy-reply clause in - `handle_register_reply/4`, so during a mixed-version rollout the verified - guard is off for replies from a pre-version leader — a residual this model - proves matters. -- **`DgenRegistryReplicationDegradeOpen.cfg`** (`DegradeOpen = TRUE`): the - documented `strict_replication = false` caveat — a direct registration's - `yes` can fire off the leader-only degrade-open timeout before any - follower confirms, leaving one holder with no crash spent. Remember this is - the *default* configuration; DegradeOpenSafety above records what it keeps. -- **`DgenRegistryReplicationHandoffRace.cfg`** (both `SafeAssume*` FALSE): the - pre-fix handoff path — every other constant is the shipped default. This - reproduces the race described in design doc §5.7: without the - durable-version-key fence on the assuming leader's gather and the - version-monotonic snapshot apply, TLC finds a counterexample where a single - node crash silently loses an already-acked registration (`DurableAcked`). - One caveat the code carries: `gather_caught_up` falls back to - `DurableVersion = 0` if the frontier read errors, so in a - backend-failure-during-handoff window the shipped code briefly *is* this - mutation — the fence degrades to pre-fix behavior, never worse. -- **`DgenRegistryReplicationHandoffRaceUnique.cfg`** (both FALSE, - `UniqueBinding` alone): TLC stops at the first violation, and in the config - above BFS reaches `DurableAcked` first — so the race's second consequence, - the freed name re-issued to a second pid, was long claimed but never - demonstrated. This config demonstrates it. -- **`DgenRegistryReplicationHandoffGatherOnly.cfg`** / - **`DgenRegistryReplicationSnapMonotonicOnly.cfg`**: each half of the §5.7 - fix flipped by itself. Both fail, so neither guard shadows the other: a - stale gather loses acked rows even with monotonic snapshot applies (the - assuming leader rewinds *itself*), and a late-delivered legitimate snapshot - rewinds a follower even with every gather fenced. - -## A closed abstraction gap: one broadcast per commit version - -`RegisterForward`/`RegisterDirect` each commit **exactly one** name at a version, so -in this model a version is always a single broadcast — and `RecvBcastApply`'s -`m.ver = appliedVer[f]` disjunct is reachable only for a duplicate. - -The code did not match that. The leader group-commits up to `commit_batch_size` -(default 5000) ops into one version and used to broadcast **one message per changed -name**, all sharing it. There, `m.ver = appliedVer[f]` was the clause applying the -2nd..Nth message of a batch — so `applied_version` advanced on a batch's *first* -message, and a member receiving a strict subset reported the full version while -holding only part of it. `PrefixConsistency`, and the "version tie ⇒ identical -content" property `AssumeGather`'s freshest-wins relies on, were being verified here -under an assumption the implementation did not satisfy. - -The simulation harness in [`test/support/sim/`](../test/support/sim/README.md) found -this against the real code (finding 1 there). It was closed **on the code side**: a -batch now ships as a single `{names_batch, Ops, …}` message, so one commit version is -one broadcast, exactly as modelled, and the surviving `V =< Applied` clause is purely -the duplicate guard this spec always assumed it was. - -Worth recording as the general lesson: the gap was not in either artefact taken -alone. TLC verified the spec exhaustively and the tests passed; what neither could -see was that they were describing different protocols. That seam is what the -`Spec ↔ code map` below exists to hold shut, and it is worth re-deriving rather than -trusting whenever either side changes. - -A coda to that closure: the spec long kept the `m.ver = appliedVer[f]` apply -disjunct that only the pre-fix multi-message batches could reach. It has now been -removed (a re-delivered duplicate falls to the `V <= Applied` drop, exactly as in -the code) — and the main config's distinct-state count is **bit-identical** -before and after (10,656,697), which is TLC's way of confirming the disjunct was -dead in every reachable state rather than someone's way of hoping it was. +| Config | Expect | What it shows | +|---|---|---| +| `DgenRegistryReplication` | pass | all six invariants, full depth, heartbeat off | +| `DgenRegistryReplicationHeartbeat` | pass | the first five with the heartbeat on (`MaxVersion = 2`) | +| `DgenRegistryReplicationDegradeOpenSafety` | pass | the default degrade-open mode keeps everything except `DurableAcked` | +| `DgenRegistryReplicationDegradedAssume` | pass | with the gather deadline modeled, `DegradedFlagSound` holds: the flag never misses an incomplete reconstruction | +| `DgenRegistryReplicationHandoffRace` | fail | the pre-§5.7 handoff (both `SafeAssume*` off): one crash loses an acked registration (`DurableAcked`) | +| `DgenRegistryReplicationHandoffRaceUnique` | fail | the same race checked against `UniqueBinding` alone: the freed name is re-issued to a second pid | +| `DgenRegistryReplicationHandoffGatherOnly` | fail | the gather fence alone off: it is needed on its own | +| `DgenRegistryReplicationSnapMonotonicOnly` | fail | the snapshot guard alone off: it is needed on its own | +| `DgenRegistryReplicationNoGuard` | fail | optimistic follower acks (`VersionGuardedAck = FALSE`) lose `DurableAcked`. The code's versionless `{register_reply, Ref, Result}` rolling-upgrade clause has this shape | +| `DgenRegistryReplicationDegradeOpen` | fail | degrade-open (`strict_replication = false`) loses `DurableAcked` | +| `DgenRegistryReplicationAckReachable` | fail | vacuity canary: `NoAcks` fails, so acks are actually reachable | +| `DgenRegistryReplicationDegradedReachable` | fail | vacuity canary: `NoDegradedAssume` fails, so degraded assumes are actually reachable | + +If a vacuity canary ever passes, the model has stopped exercising its +behaviour, and a green run on the other configs means much less. ## Spec ↔ code map | Spec | Code | |---|---| | `Elect` | `dgen_registry_elector` committing a membership/leadership change (abstracted to one durable write) | -| `AssumeGather` (incl. the `SafeAssumeGather` version-key fence) | the `assuming` statem state: `assume_genuine/6` → `spawn_assume_gather` → `gather_caught_up/6` → the `{assume_gathered}` continuation, valid only in `assuming` for its arming ref. The continuing-leader join fast path (`assume_fast_path/6` / `onboard_joiner` / `{peer_joined}`) dispatches in the `leader` state and is OUT OF SCOPE here (see above) | -| `HeartbeatBcast` / `RecvHeartbeat` | `handle_info(replica_heartbeat, …)` leader clause → `broadcast_heartbeat/1` (`{names_batch, [], E, V, V, Self}`) / the empty batch through `apply_bcast/6`'s three-way split | +| `AssumeGather` (incl. the `SafeAssumeGather` version-key fence) | the `assuming` statem state: `assume_genuine/6` → `spawn_assume_gather` → `gather_caught_up/6` → the `{assume_gathered}` continuation, valid only in `assuming` for its arming ref. The continuing-leader join fast path (`assume_fast_path/6` / `onboard_joiner` / `{peer_joined}`) dispatches in the `leader` state and is out of scope | +| `HeartbeatBcast` / `RecvHeartbeat` | `handle_event(info, replica_heartbeat, leader, …)` → `broadcast_heartbeat/1` (`{names_batch, [], E, V, V, Self}`) / the empty batch through `dgen_registry_replica:offer_batch/5` | | `CanCommit`'s `dbLeader/dbEpoch` conjuncts | the fenced version-key bump in `dgen_registry_names:start_commit/4` (§5.1) | | `RegisterForward` | follower `route_register` forward → leader `{register_req}` → group commit → `broadcast_batch/5` + `{register_reply, Ref, yes, Version}` | -| `RegisterDirect` + `RecvSync`/`RecvAck` | leader-local `route_register` → `pending_acks` / `{replicate_sync}` / `{replicate_ack}` | +| `RegisterDirect` + `RecvSync`/`RecvAck` | leader-local `route_register` → `dgen_registry_pipeline:await_acks/5` / `{replicate_sync}` / `{replicate_ack}` → `dgen_registry_pipeline:ack/3` | | `DegradeTimeout` | `handle_info({replicate_timeout, _})` with `strict_replication = false` | -| `RecvBcastApply/Drop/Gap` | `apply_bcast/6` (the three-way case split; one `{names_batch, …}` per commit version) | -| `RecvReply` | `handle_register_reply/4` incl. `deferred_yes`; flushes are `bump_applied/2` → `flush_deferred/1` | -| `RecvSnap` (incl. the `SafeAssumeSnap` version-monotonic guard) | `handle_cast({apply_names_snapshot, ...})` — the `Epoch >= CurrentEpoch andalso Version >= CurrentVersion` guard | +| `RecvBcastApply/Drop/Gap` | `dgen_registry_replica:offer_batch/5` → `applied` / `stale_epoch`+`duplicate` / `gap` (one `{names_batch, …}` per commit version). The member's `{names_batch}` handler reacts to the verdict | +| `RecvReply` | `dgen_registry_intents:on_reply/4` incl. the `awaiting_version` phase. Flushes are `dgen_registry_intents:on_version/2`, called from the `{names_batch}` handler's `applied` branch and on an accepted snapshot. On a leadership change the code re-drives a deferred ack as a fresh forward (`redrive/5`) where the spec drops it: a fresh `RegisterForward` in spec terms | +| `RecvSnap` (incl. the `SafeAssumeSnap` version-monotonic guard) | `dgen_registry_replica:offer_snapshot/4` (the `Epoch >= CurEpoch andalso V >= Applied` guard), called from `handle_cast({apply_names_snapshot, ...})` | | `ServeResync` / `resyncReq` | `{resync_req}` handler / `request_resync/2` | | `Crash` | member/node death; the ETS replica dies with the process | | `DropMsg` | casts lost on an Erlang-distribution disconnect (signal-ordering semantics) | -Keep this in sync when either side changes. +The member's `gen_statem` states (`searching`/`assuming`/`leader`/`follower`) +correspond to what `leaderView[m]`/`epoch[m]`/mid-assume encode in the spec. +Gap/resync, leader reachability and the commit pipeline are per-member +variables on both sides. -Since the member became a `gen_statem`, its states realize this spec's implicit -modes directly — `searching`/`assuming`/`leader`/`follower` are what -`leaderView[m]`/`epoch[m]`/mid-assume encode here — so the map above can name -states rather than ref-guarded continuations. The spec itself needed zero -changes for that port; dimensions the member deliberately keeps as data rather -than state (gap/resync, leader reachability, the commit pipeline) are exactly -the ones this model treats as per-member variables rather than modes. +Keep this table in sync whenever either side changes. If the two drift, TLC +keeps passing and the tests keep passing, but they no longer describe the same +protocol. -## Files +## Running +```sh +formal/check.sh DgenRegistryReplication # expect pass +formal/check.sh DgenRegistryReplicationNoGuard fail # expect fail ``` -formal/ - DgenRegistryReplication.tla the spec (the §5.7 fix lives behind SafeAssumeGather/SafeAssumeSnap) - DgenRegistryReplication.cfg main model, full depth — must pass (all invariants) - DgenRegistryReplicationHeartbeat.cfg the heartbeat, all invariants at MaxVersion 2 — must pass - DgenRegistryReplicationDegradeOpenSafety.cfg the DEFAULT mode's residual guarantees — must pass - DgenRegistryReplicationHandoffRace.cfg pre-fix protocol (both guards off) — must fail - DgenRegistryReplicationHandoffRaceUnique.cfg its UniqueBinding half — must fail - DgenRegistryReplicationHandoffGatherOnly.cfg gather fence alone off — must fail - DgenRegistryReplicationSnapMonotonicOnly.cfg snapshot guard alone off — must fail - DgenRegistryReplicationNoGuard.cfg mutation (pre-deferred_yes) — must fail - DgenRegistryReplicationDegradeOpen.cfg mutation (degrade-open loses DurableAcked) — must fail - DgenRegistryReplicationAckReachable.cfg vacuity canary (NoAcks) — must fail - vendor/tla2tools-v1.8.0.jar the checked-in TLC (see "How to run locally") - check.sh runner, local + CI (per-run metadir; concurrency-safe) - README.md this file -.github/workflows/formal.yml CI workflow (all ten configs + the DST jobs) -``` + +`check.sh [pass|fail]` runs TLC and checks the result against the +expectation. It needs Java 11+ and nothing else. It uses the TLC jar checked in +at `formal/vendor/tla2tools-v1.8.0.jar`. To use a different TLC, set +`TLA2TOOLS_JAR` or put the jar at `formal/tla2tools.jar` (gitignored). Each run +gets its own temporary metadir, so concurrent runs are safe. + +CI (`.github/workflows/formal.yml`) runs every config in the table above with +its expected result on every push and PR. diff --git a/src/dgen_registry.erl b/src/dgen_registry.erl index 45ac51f..0f0d1b9 100644 --- a/src/dgen_registry.erl +++ b/src/dgen_registry.erl @@ -3,40 +3,12 @@ -define(DOCATTRS, ?OTP_RELEASE >= 27). -%% The transform, for a module that is mostly a supervisor and an API surface. -%% -%% What matters is *where this code runs*. `register_name/2`, `unregister_name/1`, -%% `whereis_name/1`, `query/2` and the rest execute in the **caller's** process, and -%% under simulation the caller is a process the scheduler owns. So every -%% `gen_server:call/2,3` here has to be the one `eta_net` provides rather than -%% OTP's -- not for the fault injection, but for the clock and the schedule. -%% -%% Left untransformed, such a call goes into `gen:do_call/4`, which is OTP code no -%% transform reaches, and takes two real-time dependencies with it: a -%% `receive ... after Timeout` on the wall clock, and a real `erlang:monitor` whose -%% `DOWN` the VM delivers when it likes rather than when the schedule says. Both -%% put a scheduled process's progress outside the schedule, and neither shows up in -%% `eta_run:audit/1` -- the run simply stops being a function of its seed. Found -%% exactly that way: clients parked in `gen:do_call/4` at the step where two runs -%% of one seed first disagreed. +%% The API runs in the caller's process, so under simulation its gen_server calls must be +%% eta_net's: OTP's gen:do_call/4 waits on the wall clock and a real monitor. -include("../include/dgen_eta.hrl"). -%% Supervisor flags that exist only to keep this tree schedulable under simulation. -%% -%% The simulation declares these supervisors to `eta_sched` — it has to, because -%% `elector_pid/1` below asks a supervisor for its children, so every -%% `get_leader/1`, `get_epoch/1` and `get_members/1` is a call into one. A process -%% the scheduler owns must have no real timers, and on OTP 28 a supervisor does: -%% `hibernate_after` arrived with a 1000ms default, implemented as a `gen_server` -%% timeout, and `supervisor` is OTP code no parse transform reaches. So every -%% supervisor armed a wall-clock timer that dropped a message into a scheduled -%% process's mailbox whenever real time said so. It moved with machine load — six -%% simulation suites running at once made it fail about one run in three, while -%% one at a time looked clean — and it was the last thing keeping `mix dst` from -%% being a function of its seed. -%% -%% Guarded twice over, so a release build is byte-for-byte what it was: `DST` is -%% set only by the test profile, and the key does not exist before OTP 28. +%% Test-only (DST, OTP >= 28): turn off supervisor `hibernate_after`, a wall-clock timer +%% that breaks seed determinism (elector_pid/1 calls into the supervisor). -ifdef(DST). -if(?OTP_RELEASE >= 28). -define(SUP_SIM_FLAGS, #{hibernate_after => infinity}). @@ -202,17 +174,10 @@ stakeholders are database entities. See §4.9 of the design doc. %% Per-registry tuning options passed to `start_link/3`. -type registry_opts() :: #{atom() => term()}. -%% A query over registrations' indexed metadata, used by the presence subscriptions -%% (`subscribe/4`). Extensible by design — only the AND-equal form exists today: -%% `{all, #{attr() => value()}}` matches a registration whose `index` map contains -%% *all* of the given exact equalities (the same semantics as `query/2`). A bare -%% constraints map is accepted anywhere a query is and is treated as `{all, Map}`. -%% Future query kinds (ranges, disjunction, …) slot in as new tagged variants without -%% changing existing call sites. +%% A metadata query (`subscribe/4`): AND-equal `{all, Map}`, or a bare Map meaning that. -type query() :: {'all', #{term() => term()}} | #{term() => term()}. -%% A presence change delivered to a subscription's notify targets: a name entered -%% (`joined`) or left (`left`) the watch query's result set. See `subscribe/4`. +%% A name entered (`joined`) or left (`left`) a watch query's result set. -type presence_event() :: {joined | left, Name :: term(), Pid :: pid()}. %% --------------------------------------------------------------------------- @@ -332,15 +297,11 @@ unregister_name({RegistryName, LogicalName}) -> gen_statem:call( member_name(RegistryName), {unregister, LogicalName}, - %% Same caller-side bound as the other leader-routed writes — see - %% set_metadata/2. + %% Same caller-side bound as set_metadata/2. dgen_config:register_timeout(#{}) ) catch - %% noproc: the registry is stopped/absent — nothing to unregister, `ok` is - %% truthful. timeout: the removal is already stashed on the member and will - %% be re-driven; the public contract stays `ok` (the via machinery ignores - %% the return value, and unlike register there is no wrong verdict to mask). + %% noproc: nothing to unregister; timeout: the member re-drives it. `ok` either way. exit:_ -> ok end. @@ -362,8 +323,7 @@ whereis_name({RegistryName, LogicalName}) -> [{_Name, Pid, _Index, _Data} | _] -> Pid; [] -> undefined catch - %% The table does not exist yet (member starting / restarting after a - %% crash) — treat as not-registered rather than crashing the caller. + %% No table yet (member (re)starting): not registered. error:badarg -> undefined end. @@ -388,11 +348,7 @@ send(Name, Msg) -> %% --------------------------------------------------------------------------- %% Registration metadata %% --------------------------------------------------------------------------- -%% Metadata is attached to a registration (keyed by name) and lives exactly as long as -%% the registration does — it is removed automatically when the process exits or is -%% unregistered. A `MetaSpec` is `#{index => #{attr() => value()}, data => term()}` -%% (both optional): `index` is the queryable metadata, `data` an opaque payload returned -%% verbatim. See §4.7 of the design doc. +%% Metadata lives exactly as long as its registration (§4.7). -if(?DOCATTRS). -doc """ @@ -438,10 +394,7 @@ set_metadata({RegistryName, LogicalName}, MetaSpec) -> gen_statem:call( member_name(RegistryName), {set_metadata, LogicalName, Index, Data}, - %% Bounded by the same caller-side knob as register_name/2,3 rather than - %% `gen_server:call/2`'s hidden 5s default: this is a leader-routed write - %% on the same pipeline, so an operator who tunes how long a write may - %% block should not find one of them ignoring the setting. + %% register_name/2,3's knob, not call/2's hidden 5s: same write pipeline. dgen_config:register_timeout(#{}) ) catch @@ -634,10 +587,7 @@ subscriptions(RegistryName) -> dgen_server:priority_call(Pid, get_subscriptions) end). -%% Route a durable subscription write to the elector as a fire-and-forget cast (handled -%% by dgen_registry_elector:handle_cast_tx/3 — enqueued durably and committed in the -%% consume transaction). A missing registry is a usage error and exits, mirroring the -%% register path; a cast to a live elector that then dies is a harmless no-op. +%% Fire-and-forget; the elector commits it durably. A missing registry exits. elector_cast(RegistryName, Msg) -> case elector_pid(RegistryName) of undefined -> exit({noproc, {?MODULE, elector, RegistryName}}); @@ -679,14 +629,11 @@ delete(Name, Tenant) -> end), ok. -%% Normalise a public query into the internal tagged form: a bare constraints map is -%% read as `{all, Map}` (AND-equal); an already-tagged query passes through. +%% A bare constraints map means `{all, Map}`. -spec normalize_query(query()) -> {'all', map()}. normalize_query({all, C} = Q) when is_map(C) -> Q; normalize_query(C) when is_map(C) -> {all, C}. -%% Parse a public MetaSpec (`#{index => …, data => …}`, both optional) into the internal -%% `{Index, Data}` pair, defaulting to the empty metadata. -spec meta_of(map()) -> {map(), term()}. meta_of(MetaSpec) -> {maps:get(index, MetaSpec, #{}), maps:get(data, MetaSpec, undefined)}. @@ -753,8 +700,7 @@ ready(Name) -> Ready when is_boolean(Ready) -> Ready; _ -> false catch - %% Member starting/restarting (noproc), or busy past the probe bound - %% (timeout) — not ready yet. + %% noproc (starting) or timeout (busy): not ready yet. exit:_ -> false end. @@ -813,16 +759,8 @@ await_ready(Name, Timeout) -> Deadline = dgen_utils:real_monotonic_ms() + Timeout, await_ready_loop(Name, Deadline). -%% The poll sleeps and reads the clock through `dgen_utils`, which the eta -%% transform does not touch, and that is load-bearing: this module carries the -%% transform (see the include at the top), so a bare `timer:sleep/1` here becomes -%% `eta_time:sleep/1` — and under `eta_run` this loop runs in the *driver* while -%% the cluster starts, the one process whose transformed sleep arms a virtual -%% deadline nothing will ever reach (`eta_time:sleep/1`'s own boundary rule). -%% Observed as the whole run wedging in `await_ready_loop` whenever a member was -%% not ready on the very first poll. Readiness polling is a wall-clock concern: -%% it runs before the schedule owns the system (or on a real node, where there is -%% no schedule at all), exactly like the harness's own `await_quiescent/2`. +%% Sleeps and reads the clock via dgen_utils, untouched by the eta transform: under +%% eta_run this loop runs in the driver, where a transformed sleep would never wake. await_ready_loop(Name, Deadline) -> case ready(Name) of true -> @@ -844,11 +782,7 @@ await_ready_loop(Name, Deadline) -> get_members(Name) -> with_elector(Name, [], fun(Pid) -> dgen_server:priority_call(Pid, get_members) end). -%% Resolve the elector via `elector_pid/1` and apply `Fun` to it, or return `Default` -%% if the member (and so the elector) cannot currently be found — the registry may not -%% have started yet, or may be mid-restart. Keeps get_leader/get_epoch/get_members -%% bypassing the member's mailbox entirely, as before (§8): only an elector lookup is -%% added, not a member round-trip. +%% Bypasses the member's mailbox (§8); `Default` if the elector is not found. with_elector(Name, Default, Fun) -> case elector_pid(Name) of undefined -> Default; @@ -921,12 +855,7 @@ elector_pid(Name) -> end end. -%% Reads Pid's `$ancestors` process-dictionary entry: the list of supervisor/starter -%% identifiers `proc_lib` records for a process at spawn time, immediate parent -%% first. There is no public `proc_lib` accessor for this (it is conventionally read -%% straight out of the dictionary — the same technique tools like `recon` use); each -%% entry is a pid, or a registered name if that ancestor had one, either of which -%% `supervisor:which_children/1` accepts directly. +%% `$ancestors` (no public proc_lib accessor): pids or names, as which_children/1 takes. -spec ancestors_of(pid()) -> [pid() | atom()]. ancestors_of(Pid) -> case erlang:process_info(Pid, dictionary) of @@ -939,22 +868,9 @@ ancestors_of(Pid) -> %% --------------------------------------------------------------------------- init({Name, Tenant, Opts}) -> - %% Elector must start first: the member discovers it (via this supervisor, - %% see dgen_registry_member's `discover_elector` handle_continue) and casts - %% `{join, MemberId}` to it right after. A large consume_k is chosen because the - %% in-transaction work is very small, and we want to keep a single node as the - %% sole consumer as much as possible, to avoid leadership churn. consume_k > 1 - %% also disables dgen_server inlining, so every elector call rides the batched - %% consume loop rather than being served ad hoc on the receiving node — keeping - %% the consumer (hence the leader) stable. - %% - %% The elector is deliberately unnamed (`dgen_server:start_link/3`, no `Reg`): - %% it has no registered name at all, local or otherwise. The member finds it by - %% pid via this supervisor instead — see the "Process identity" moduledoc note. - %% `Name` is this member's identity; `Keyspace` is the durable prefix every - %% member of the registry coordinates through. They are the same atom in the - %% ordinary one-member-per-node case; `keyspace` splits them so several members - %% of one registry can share a VM (see start_link/3's docs). + %% Elector first (unnamed; the member discovers it via this supervisor). A large + %% consume_k (> 1 also disables inlining) keeps one stable consumer, hence leader. + %% `Keyspace` differs from `Name` only when members of one registry share a VM. Keyspace = maps:get(keyspace, Opts, Name), ElectorSpec = #{ id => elector, @@ -970,11 +886,7 @@ init({Name, Tenant, Opts}) -> modules => [dgen_server, dgen_registry_elector] }, - %% Tuid matches the elector's (both derive it from dgen_registry_elector:tuid/1) - %% so the member can compute the leader key and the per-registry version key - %% directly, against the same keyspace the elector's fence writes. Derived from - %% `Keyspace`, not `Name`: every member of a registry must fence against the same - %% leader key even when their identities differ. + %% From `Keyspace`, as the elector's: all members fence on the same leader key. Tuid = dgen_registry_elector:tuid(Keyspace), MemberSpec = #{ id => member, @@ -994,14 +906,7 @@ init({Name, Tenant, Opts}) -> modules => [dgen_registry_member] }, - %% The connector keeps Erlang distribution in step with the DB-backed membership - %% (the proactive mesh, the backstop reap, the leader-liveness probe) — the - %% node-level connectivity plumbing, split out of the member so each has one job. - %% It is unnamed and holds only the elector pid; it is started **after** the member - %% so it can discover the elector by walking from the (already-registered) member, - %% exactly as the member does. `Opts` is threaded through so the connector can - %% resolve its `connectivity` mode (§8) — `provided_externally` turns off the - %% proactive mesh. See dgen_registry_connector. + %% Started after the member, from which it discovers the elector. ConnectorSpec = #{ id => connector, start => @@ -1015,13 +920,7 @@ init({Name, Tenant, Opts}) -> modules => [dgen_registry_connector] }, - %% one_for_all: the elector, member, and connector are one unit — the member and - %% connector each hold a pid reference to the elector, discovered once at startup - %% (see the moduledoc's "Process identity" note). If any crashes, restart all - %% together so neither runs with a stale elector reference; there is no - %% cross-restart pid-staleness case to handle at the application level. Order - %% matters: elector first (both others discover it), then member (registers its - %% name), then connector (walks from the member to the elector). + %% one_for_all: the others hold the elector's pid, so never run with a stale one. {ok, {maps:merge(#{strategy => one_for_all, intensity => 5, period => 10}, ?SUP_SIM_FLAGS), [ ElectorSpec, MemberSpec, ConnectorSpec diff --git a/src/dgen_registry/dgen_registry_conflicts.erl b/src/dgen_registry/dgen_registry_conflicts.erl new file mode 100644 index 0000000..f8e9a9a --- /dev/null +++ b/src/dgen_registry/dgen_registry_conflicts.erl @@ -0,0 +1,249 @@ +-module(dgen_registry_conflicts). + +-define(DOCATTRS, ?OTP_RELEASE >= 27). + +-if(?DOCATTRS). +-moduledoc false. +-endif. + +%% Conflict repair (§5.6; `§N.N` is docs/design/dgen_registry_design.md). Pids are never +%% durable, so if every holder of a binding is lost the name can be re-issued while the +%% original lives on; on reconnection two live pids hold one name. +%% +%% Evidence: a trail of explicit releases, `{Name, Pid} => ms`, recorded at commit or +%% from the broadcast and merged at handoff; a witness binding on it is lag, not a +%% conflict. Keyed by pair, so one release never masks another name. Not pruned while a +%% member is disconnected or within a TTL of a departure: its stale rows may resurface. +%% +%% Verdict: the authority (freshest reconstruction) binds a name to a live pid and a +%% witness to a different live pid off the trail. A name the authority lacks is no +%% conflict. Liveness is probed off the leader's loop: conflict_pids/3 lists what to +%% probe, detect/4 decides. +%% +%% Remedy: resolve/5 kills the authority and all divergent pids and drops the name, +%% unless `terminate_on_conflict` is off (alarm only) or the per-name, per-term +%% `conflict_kill_budget` is spent (alarm and stop: escalate rather than loop). Kills +%% and alarms are returned as effects; nothing here probes, sends or kills. +%% +%% Prevention: with `reject_when_degraded`, a leader whose reconstruction may lack a +%% committed binding refuses names it does not hold (degraded/5). +%% +%% *This documentation is LLM-generated. See the AI disclosure in README.md.* + +-export([new/0, trail/1, released/3, merge/2, departed/2, prune/4]). +-export([conflict_pids/3, detect/4, resolve/5]). +-export([degraded/1, set_degraded/2, degraded/5]). + +-export_type([t/0, trail/0, conflict/0, effect/0]). + +-type records() :: dgen_registry_replica:records(). + +%% Bare-pid keys come from pre-upgrade members during a rolling upgrade. +-type trail() :: #{{term(), pid()} | pid() => integer()}. + +-type conflict() :: {Name :: term(), Authority :: pid(), Divergent :: [pid()]}. + +-type effect() :: + {kill, pid()} + | {alarm, conflict(), terminated | detect_only | budget_exhausted}. + +-record(conflicts, { + trail = #{} :: trail(), + %% Leader-only: name => recent kill times (ms), for `conflict_kill_budget`. + kill_budget = #{} :: #{term() => [integer()]}, + %% Last member departure seen; suspends trail pruning for a TTL. + last_departure = 0 :: integer(), + %% Leader-only: may the last reconstruction lack a committed binding? + degraded = false :: boolean() +}). + +-opaque t() :: #conflicts{}. + +-spec new() -> t(). +new() -> #conflicts{}. + +%% --------------------------------------------------------------------------- +%% Evidence: the release trail +%% --------------------------------------------------------------------------- + +-spec trail(t()) -> trail(). +trail(#conflicts{trail = T}) -> T. + +%% Record explicit releases of live pids at `Now`. +-spec released([{term(), pid()}], integer(), t()) -> t(). +released([], _Now, C) -> + C; +released(Pairs, Now, C = #conflicts{trail = T}) -> + C#conflicts{trail = lists:foldl(fun(Pair, Acc) -> Acc#{Pair => Now} end, T, Pairs)}. + +%% Merge gathered trails, keeping each entry's latest release time. +-spec merge([trail()], t()) -> t(). +merge(Trails, C = #conflicts{trail = T}) -> + C#conflicts{trail = merge_trails(T, Trails)}. + +-spec departed(integer(), t()) -> t(). +departed(Now, C) -> C#conflicts{last_departure = Now}. + +%% Trail entries expire only if every member is connected and none departed in the TTL. +-spec prune(integer(), boolean(), dgen_config:config(), t()) -> t(). +prune(Now, AllConnected, Config, C = #conflicts{trail = T, kill_budget = KB}) -> + TTL = dgen_config:conflict_release_ttl(Config), + RecentDeparture = Now - C#conflicts.last_departure < TTL, + T1 = + case AllConnected andalso not RecentDeparture of + true -> maps:filter(fun(_Key, Ts) -> Ts >= Now - TTL end, T); + false -> T + end, + {_Count, Window} = dgen_config:conflict_kill_budget(Config), + KB1 = maps:filter( + fun(_Name, Tss) -> Tss =/= [] end, + maps:map(fun(_Name, Tss) -> [Ts || Ts <- Tss, Ts >= Now - Window] end, KB) + ), + C#conflicts{trail = T1, kill_budget = KB1}. + +%% --------------------------------------------------------------------------- +%% Verdict +%% --------------------------------------------------------------------------- + +%% Every pid a conflict could turn on, to probe off the leader's loop before detect/4. +-spec conflict_pids(records(), [records()], trail()) -> [pid()]. +conflict_pids(Authority, Witnesses, Trail) -> + lists:usort( + maps:fold( + fun(Name, PidSet, Acc) -> + [P || P <- [authority(Name, Authority) | maps:keys(PidSet)], is_pid(P)] ++ Acc + end, + [], + candidates(Authority, Witnesses, Trail) + ) + ). + +%% The authority's records may be among the witnesses: an entry agrees with itself. +-spec detect(records(), [records()], trail(), fun((pid()) -> boolean())) -> [conflict()]. +detect(Authority, Witnesses, Trail, Alive) -> + maps:fold( + fun(Name, PidSet, Acc) -> + APid = authority(Name, Authority), + Others = [P || P <- maps:keys(PidSet), P =/= APid], + case Others =:= [] orelse not (is_pid(APid) andalso Alive(APid)) of + true -> + Acc; + false -> + case [P || P <- Others, Alive(P)] of + [] -> Acc; + Divergent -> [{Name, APid, Divergent} | Acc] + end + end + end, + [], + candidates(Authority, Witnesses, Trail) + ). + +%% Differing witness bindings off the trail: `#{Name => #{Pid => true}}`. +candidates(Authority, Witnesses, Trail) -> + lists:foldl( + fun(Map, Acc0) -> + maps:fold( + fun(Name, {Pid, _Index, _Data}, Acc) -> + case maps:get(Name, Authority, undefined) of + {Pid, _, _} -> + Acc; + _AbsentOrDifferent -> + case on_trail(Name, Pid, Trail) of + true -> + Acc; + false -> + Pids = maps:get(Name, Acc, #{}), + Acc#{Name => Pids#{Pid => true}} + end + end + end, + Acc0, + Map + ) + end, + #{}, + Witnesses + ). + +authority(Name, Authority) -> + case maps:get(Name, Authority, undefined) of + {Pid, _Index, _Data} -> Pid; + undefined -> undefined + end. + +on_trail(Name, Pid, Trail) -> + maps:is_key({Name, Pid}, Trail) orelse maps:is_key(Pid, Trail). + +%% --------------------------------------------------------------------------- +%% Remedy +%% --------------------------------------------------------------------------- + +%% `Records` only loses names, so with no conflicts it passes through by reference. +-spec resolve([conflict()], records(), dgen_config:config(), integer(), t()) -> + {records(), t(), [effect()]}. +resolve([], Records, _Config, _Now, C) -> + {Records, C, []}; +resolve(Conflicts, Records, Config, Now, C) -> + Terminate = dgen_config:terminate_on_conflict(Config), + Budget = dgen_config:conflict_kill_budget(Config), + {Records1, C1, EffsRev} = lists:foldl( + fun(Conflict, Acc) -> resolve_one(Conflict, Terminate, Budget, Now, Acc) end, + {Records, C, []}, + Conflicts + ), + {Records1, C1, lists:reverse(EffsRev)}. + +resolve_one(Conflict, false, _Budget, _Now, {Records, C, Effs}) -> + %% Detect-only: alarm, and keep the authority's binding. + {Records, C, [{alarm, Conflict, detect_only} | Effs]}; +resolve_one( + {Name, Authority, Divergent} = Conflict, true, {Count, Window}, Now, {Records, C, Effs} +) -> + KB = C#conflicts.kill_budget, + Recent = [Ts || Ts <- maps:get(Name, KB, []), Ts >= Now - Window], + case length(Recent) < Count of + false -> + %% Exhausted: escalate, stop killing this name, keep it served. + {Records, C, [{alarm, Conflict, budget_exhausted} | Effs]}; + true -> + %% A pid that died since detection ignores the kill. + Kills = [{kill, P} || P <- [Authority | Divergent]], + { + maps:remove(Name, Records), + C#conflicts{kill_budget = KB#{Name => [Now | Recent]}}, + [{alarm, Conflict, terminated} | lists:reverse(Kills, Effs)] + } + end. + +%% --------------------------------------------------------------------------- +%% Prevention +%% --------------------------------------------------------------------------- + +-spec degraded(t()) -> boolean(). +degraded(#conflicts{degraded = D}) -> D. + +-spec set_degraded(boolean(), t()) -> t(). +set_degraded(D, C) -> C#conflicts{degraded = D}. + +%% Degraded exactly when the reconstruction stopped short of the durable version key +%% (`Frontier`): replicas are stream prefixes, so the freshest at the key is the +%% committed map (TLA+ DegradedFlagSound). With the key unreadable, fall back to +%% reachability: degraded if a member not marked fresh did not answer the gather. +-spec degraded( + non_neg_integer(), non_neg_integer() | unknown, [term()], [term()], #{term() => term()} +) -> boolean(). +degraded(MaxVersion, Frontier, _OtherIds, _FreshIds, _PeerResults) when is_integer(Frontier) -> + MaxVersion < Frontier; +degraded(_MaxVersion, unknown, OtherIds, FreshIds, PeerResults) -> + Unreachable = OtherIds -- maps:keys(PeerResults), + lists:any(fun(Id) -> not lists:member(Id, FreshIds) end, Unreachable). + +%% --------------------------------------------------------------------------- +%% Internal +%% --------------------------------------------------------------------------- + +merge_trails(Trail, Others) -> + lists:foldl( + fun(T, Acc) -> maps:merge_with(fun(_K, A, B) -> max(A, B) end, Acc, T) end, Trail, Others + ). diff --git a/src/dgen_registry/dgen_registry_connector.erl b/src/dgen_registry/dgen_registry_connector.erl new file mode 100644 index 0000000..2d59219 --- /dev/null +++ b/src/dgen_registry/dgen_registry_connector.erl @@ -0,0 +1,319 @@ +-module(dgen_registry_connector). +-behaviour(gen_server). + +-define(DOCATTRS, ?OTP_RELEASE >= 27). + +-include("../../include/dgen_eta.hrl"). + +-if(?DOCATTRS). +-moduledoc false. +-endif. + +%% Node-level connectivity for one registry (sibling of the member and elector): meshes +%% distribution to every member node, reaps members stranded on dropped nodes, and probes +%% leader liveness. Never touches names, commits or leadership; reads the elector +%% off-loop and casts fenced `{member_down}` back. `§N.N` = docs/design/dgen_registry_design.md. +%% +%% Modes (§8, §4.6): `self_managed` runs the mesh; `provided_externally` opens no +%% connections but still does the registry-scoped duties (reap, leader probe, epoch nudge +%% on its own `epoch_check` timer) and warns if no leader is seen for a sustained window. +%% +%% *This documentation is LLM-generated. See the AI disclosure in README.md.* + +-export([start_link/2]). +-export([ + init/1, + handle_continue/2, + handle_call/3, + handle_cast/2, + handle_info/2, + terminate/2, + code_change/3 +]). + +%% Backstop; startup and `nodeup` also trigger an immediate pass. +-define(MESH_INTERVAL, 10000). + +%% Don't reconnect a node right after its `nodedown`: let the departure settle as a +%% `{member_down}` rather than healing the partition instantly. `nodeup` clears it early. +-define(MESH_DOWN_COOLDOWN, 10000). + +%% Backstop reap for a member whose own `{member_down}` was fenced as stale. Retried +%% because a resurrecting `{join}` may land after the first pass. Opens no connection. +-define(REAP_INTERVAL, 1000). +-define(REAP_ATTEMPTS, 6). + +%% Reaps a leader on a node we were *never* connected to (e.g. cold restart onto a dead +%% leader: no `nodedown` ever fires). Gated on `ever_connected` so a distribution-only +%% partition cannot depose the majority's leader via the DB (ping-pong); a lost-but-met +%% leader is the nodedown path's job. Grace lets the async mesh connect first. +-define(LEADER_PROBE_GRACE, 4000). +-define(LEADER_PROBE_INTERVAL, 4000). + +%% `provided_externally` only: epoch-nudge / isolation-check tick (replaces the mesh). +-define(EPOCH_CHECK_INTERVAL, 10000). + +%% ~60 s of no leader: long enough to ride out startup/handoff, short enough to surface. +-define(ISOLATION_WARN_TICKS, 6). + +-record(state, { + %% Member's registered name (= registry name); target of `{durable_epoch, _}`. + name :: atom(), + mode = self_managed :: self_managed | provided_externally, + %% `undefined` only until `discover_elector`. + elector :: pid() | undefined, + %% Node -> nodedown timestamp; mesh skips these (self_managed only). + recently_down = #{} :: #{node() => integer()}, + %% Never pruned; gates the leader probe. + ever_connected = #{} :: #{node() => true}, + %% provided_externally isolation warning: counter and one-shot latch. + no_leader_ticks = 0 :: non_neg_integer(), + isolation_warned = false :: boolean() +}). + +%% --------------------------------------------------------------------------- +%% Public API +%% --------------------------------------------------------------------------- + +-if(?DOCATTRS). +-doc "Starts the connector for registry `Name` (unregistered; held by the supervisor).". +-endif. +-spec start_link(Name :: atom(), Args :: map()) -> gen_server:start_ret(). +start_link(Name, Args) -> + gen_server:start_link(?MODULE, Args#{name => Name}, []). + +%% --------------------------------------------------------------------------- +%% gen_server callbacks +%% --------------------------------------------------------------------------- + +init(#{name := Name} = Args) -> + net_kernel:monitor_nodes(true), + Mode = dgen_config:connectivity(maps:get(config, Args, #{})), + %% Can't resolve the elector here: the supervisor is blocked in start_child for this + %% process, so which_children would deadlock. + {ok, + #state{ + name = Name, + mode = Mode, + elector = undefined, + recently_down = #{}, + ever_connected = maps:from_keys(nodes(), true) + }, + {continue, discover_elector}}. + +%% The member starts before this connector, so `elector_pid/1` can walk from it. +handle_continue(discover_elector, State = #state{name = Name, mode = Mode}) -> + Elector = dgen_registry:elector_pid(Name), + true = is_pid(Elector), + case Mode of + self_managed -> + self() ! mesh_connect; + provided_externally -> + arm(?EPOCH_CHECK_INTERVAL, epoch_check) + end, + arm(?LEADER_PROBE_GRACE, probe_leader), + {noreply, State#state{elector = Elector}}. + +handle_call(_Request, _From, State) -> + {reply, {error, unknown_call}, State}. + +handle_cast(_Msg, State) -> + {noreply, State}. + +%% The member handles {nodeup, _} separately (re-announces its {join}). +handle_info({nodeup, Node}, State = #state{mode = provided_externally}) -> + Ever = maps:put(Node, true, State#state.ever_connected), + {noreply, State#state{ever_connected = Ever}}; +handle_info({nodeup, Node}, State) -> + RD = maps:remove(Node, State#state.recently_down), + Ever = maps:put(Node, true, State#state.ever_connected), + self() ! mesh_connect, + {noreply, State#state{recently_down = RD, ever_connected = Ever}}; +handle_info({nodedown, Node}, State = #state{mode = provided_externally}) -> + schedule_reap(Node, ?REAP_ATTEMPTS), + {noreply, State}; +handle_info({nodedown, Node}, State = #state{recently_down = RD}) -> + schedule_reap(Node, ?REAP_ATTEMPTS), + {noreply, State#state{recently_down = RD#{Node => erlang:system_time(millisecond)}}}; +handle_info(epoch_check, State = #state{elector = Elector, name = Name}) -> + arm(?EPOCH_CHECK_INTERVAL, epoch_check), + spawn_epoch_check(Elector, Name, self()), + {noreply, State}; +handle_info({connectivity_leader, undefined}, State = #state{no_leader_ticks = N}) -> + N1 = N + 1, + Warned1 = + case N1 >= ?ISOLATION_WARN_TICKS andalso not State#state.isolation_warned of + true -> + warn_possible_isolation(State#state.name, N1), + true; + false -> + State#state.isolation_warned + end, + {noreply, State#state{no_leader_ticks = N1, isolation_warned = Warned1}}; +handle_info({connectivity_leader, _Leader}, State) -> + {noreply, State#state{no_leader_ticks = 0, isolation_warned = false}}; +%% Stops once the node reconnects (a live member's `{join}` wins). +handle_info({reap_down, _Node, 0}, State) -> + {noreply, State}; +handle_info({reap_down, Node, N}, State = #state{elector = Elector}) -> + case dgen_utils:node_reachable(Node) of + true -> + ok; + false -> + spawn_reap(Elector, Node), + schedule_reap(Node, N - 1) + end, + {noreply, State}; +handle_info(probe_leader, State = #state{elector = Elector, ever_connected = Ever}) -> + arm(?LEADER_PROBE_INTERVAL, probe_leader), + spawn_probe_leader(Elector, node(), Ever), + {noreply, State}; +handle_info(mesh_connect, State = #state{elector = Elector, name = Name}) -> + arm(?MESH_INTERVAL, mesh_connect), + Now = erlang:system_time(millisecond), + RD = maps:filter( + fun(_Node, Ts) -> Ts >= Now - ?MESH_DOWN_COOLDOWN end, State#state.recently_down + ), + spawn_mesh_fetch(Elector, self(), Name), + {noreply, State#state{recently_down = RD}}; +%% Suppression is decided here, not in the fetch helper: this message is ordered after +%% any `{nodedown, _}` on our mailbox, so a stale in-flight fetch can't reconnect a node +%% that just dropped (which would heal the partition before `{member_down}` settles). +handle_info({mesh_members, MemberNodes}, State) -> + Now = erlang:system_time(millisecond), + RD = maps:filter( + fun(_Node, Ts) -> Ts >= Now - ?MESH_DOWN_COOLDOWN end, State#state.recently_down + ), + SelfNode = node(), + Targets = [ + Node + || Node <- MemberNodes, Node =/= SelfNode, not maps:is_key(Node, RD) + ], + spawn_mesh_connect(Targets), + {noreply, State#state{recently_down = RD}}; +handle_info(_Info, State) -> + {noreply, State}. + +terminate(_Reason, _State) -> + net_kernel:monitor_nodes(false), + ok. + +code_change(_OldVsn, State, _Extra) -> + {ok, State}. + +%% --------------------------------------------------------------------------- +%% Internal helpers +%% --------------------------------------------------------------------------- + +schedule_reap(Node, N) -> + arm(?REAP_INTERVAL, {reap_down, Node, N}), + ok. + +%% Fenced with each member's current durable token so a genuine rejoin isn't clobbered. +spawn_reap(Elector, Node) -> + _ = spawn(fun() -> + try dgen_server:priority_call(Elector, get_members) of + Members when is_list(Members) -> + lists:foreach( + fun + ({N, _} = MemberId) when N =:= Node -> + Token = dgen_registry_elector:member_token( + Elector, MemberId, undefined + ), + dgen_server:cast(Elector, {member_down, MemberId, Token}); + (_) -> + ok + end, + Members + ); + _ -> + ok + catch + _:_ -> ok + end + end), + ok. + +spawn_probe_leader(Elector, SelfNode, EverConnected) -> + _ = spawn(fun() -> + try dgen_server:priority_call(Elector, get_leader) of + {LeaderNode, _} when LeaderNode =/= SelfNode -> + case + lists:member(LeaderNode, nodes()) orelse + maps:is_key(LeaderNode, EverConnected) + of + true -> ok; + false -> spawn_reap(Elector, LeaderNode) + end; + _ -> + ok + catch + _:_ -> ok + end + end), + ok. + +%% Regular `call`, not `priority_call`: it rides the durable queue in order with +%% membership changes, so a just-joined node is seen as soon as its join is processed. +spawn_mesh_fetch(Elector, Connector, MemberName) -> + _ = spawn(fun() -> + nudge_durable_epoch(Elector, MemberName), + try dgen_server:call(Elector, get_members) of + Members when is_list(Members) -> + Connector ! {mesh_members, [Node || {Node, _Name} <- Members]}; + _ -> + ok + catch + _:_ -> ok + end + end), + ok. + +spawn_epoch_check(Elector, MemberName, Connector) -> + _ = spawn(fun() -> + nudge_durable_epoch(Elector, MemberName), + Leader = + try dgen_server:priority_call(Elector, get_leader) of + L -> L + catch + %% Counts toward the isolation window. + _:_ -> undefined + end, + Connector ! {connectivity_leader, Leader} + end), + ok. + +%% Missed-handoff backstop: if the committed epoch is ahead of the member's, it re-joins. +%% The send must be inside `try` (not after `of`): sending to an unregistered name raises. +nudge_durable_epoch(Elector, MemberName) -> + try + case dgen_server:priority_call(Elector, get_epoch) of + DurableEpoch when is_integer(DurableEpoch) -> + MemberName ! {durable_epoch, DurableEpoch}, + ok; + _ -> + ok + end + catch + _:_ -> ok + end. + +warn_possible_isolation(Name, Ticks) -> + logger:warning( + "dgen_registry ~p: connectivity=provided_externally but no leader has been " + "observed for ~b checks (~b ms). Distribution connectivity for this registry is " + "delegated (§4.6) — verify a self_managed provider registry spans this node, or " + "this node may be isolated and refusing writes (CP, §5.3).", + [Name, Ticks, Ticks * ?EPOCH_CHECK_INTERVAL] + ), + ok. + +%% Off-loop: the handshake can block on an unreachable node. +spawn_mesh_connect(Targets) -> + _ = spawn(fun() -> lists:foreach(fun net_kernel:connect_node/1, Targets) end), + ok. + +%% See `dgen_registry_member:arm/2`. +arm(Delay, Msg) -> + _ = ?ETA_LOG({arm, Msg, Delay}), + erlang:send_after(Delay, self(), Msg). diff --git a/src/dgen_registry/dgen_registry_elector.erl b/src/dgen_registry/dgen_registry_elector.erl new file mode 100644 index 0000000..ef6b6c0 --- /dev/null +++ b/src/dgen_registry/dgen_registry_elector.erl @@ -0,0 +1,343 @@ +-module(dgen_registry_elector). +-behaviour(dgen_server). + +-define(DOCATTRS, ?OTP_RELEASE >= 27). + +-include("../../include/dgen_eta.hrl"). +-define(SnapshotTimeout, 2000). + +-if(?DOCATTRS). +-moduledoc false. +-endif. + +%% `dgen_server` callback tracking registry membership and electing a leader. State +%% is durable and shared by every node of the registry; messages arrive via a durable +%% FIFO queue. `§N.N` = docs/design/dgen_registry_design.md. +%% +%% Leadership is sticky (kept while the incumbent is a member) to avoid thrashing. +%% A membership change routes through the leader as a post-commit action, with no +%% distributed lock (§5.7): committing the new leader key fences the old leader, whose +%% next batch commit then aborts. The leader gathers replicas, resolves conflicts +%% (§5.6) and casts the snapshot to followers from its own process, so per-pair FIFO +%% orders it before any later `{name_registered}`. +%% +%% Leader key `{Tuid, <<"leader">>}` holds `term_to_binary({MemberId | undefined, +%% Epoch})`: the epoch makes it a real fencing token (no ABA on depose-and-re-elect). +%% +%% *This documentation is LLM-generated. See the AI disclosure in README.md.* + +-export([ + init/1, + handle_cast_tx/3, + handle_call_tx/4, + leader_db_key/2, + tuid/1, + member_token/3 +]). + +-export_type([member_id/0, member_info/0, registry_state/0]). + +-type member_id() :: {node(), atom()}. +%% `fresh` = never synced any state; an unreachable fresh member holds no bindings, so +%% it is excluded from the leader's `degraded` computation (§5.6). +-type member_info() :: #{ + joined_at := integer(), join_token := reference(), fresh => boolean() +}. +-type registry_state() :: #{ + name := atom(), + members := #{member_id() => member_info()}, + leader := member_id() | undefined, + epoch := non_neg_integer(), + %% Durable presence subscriptions (§4.9), `SubId => {Watch, Notify}`; survive a + %% full cluster restart. Source of truth; the leader is fed deltas. + subscriptions => #{term() => {term(), term()}} +}. + +%% --------------------------------------------------------------------------- +%% dgen_server callbacks +%% --------------------------------------------------------------------------- + +-if(?DOCATTRS). +-doc """ +Initialises the elector state with an empty member map and undefined leader. + +`name` is the co-located member's **identity** — it is what `elect_leader/5` turns +into the local `member_id/0` when preferring the local member for leadership. +`keyspace` is the registry's durable **prefix**, and is what the tuid is derived +from; it defaults to `name`. The two differ only when a registry is started with +`dgen_registry:start_link/3`'s `keyspace` option, which lets several members of one +registry share a VM (see that function's docs). +""". +-endif. +-spec init(#{name := atom(), keyspace => atom()}) -> + {ok, dgen_server:tuid(), registry_state()}. +init(Args = #{name := Name}) -> + State = #{ + name => Name, members => #{}, leader => undefined, epoch => 0, subscriptions => #{} + }, + {ok, tuid(maps:get(keyspace, Args, Name)), State}. + +-if(?DOCATTRS). +-doc """ +The transaction-unit id (backend keyspace prefix) for registry `Name`. + +The single source of truth for the registry's backend key prefix: the elector, the +member (its version-key commit and leader-key fence), and the top-level supervisor +must all use the *same* tuid, so they derive it here rather than each rebuilding the +literal. Exported for that reason. +""". +-endif. +-spec tuid(atom()) -> dgen_server:tuid(). +tuid(Name) -> + {<<"dgen_registry.">>, atom_to_binary(Name)}. + +-if(?DOCATTRS). +-doc """ +Processes membership change messages within a backend transaction. + +Handles `{join, MemberId, Token, Fresh}` (a legacy 3-tuple `{join, MemberId, +Token}` is accepted as `Fresh = false`) and `{member_down, MemberId, Token}`. +Returns `{noreply, NewState, Actions}` when the handoff must be routed through the +leader (every join; a leader-changing member_down), else plain +`{noreply, NewState}`. The `Actions` run after the transaction commits — no +distributed lock (§5.7). + +Each `{join}` carries a unique token (a `reference()` generated by the member +process before enqueuing) and a freshness flag (whether the member has ever synced +registry state). The elector stores both in `member_info`. + +A `{member_down, MemberId, Token}` is silently discarded when its token does not +match the stored token for that member — this means the member has rejoined with +a new token since the DOWN was detected, so the message is stale. This prevents +a partition-recovery race where a `{member_down}` enqueued during the disconnect +is processed after the subsequent `{join}` that heals the cluster. +""". +-endif. +-spec handle_cast_tx(dgen_server:tx_ctx(), term(), registry_state()) -> + dgen_server:noreply_ret(). + +%% Rolling upgrade: legacy 3-tuple join, treated as non-fresh (conservative). +handle_cast_tx(TxCtx, {join, MemberId, Token}, State) -> + handle_cast_tx(TxCtx, {join, MemberId, Token, false}, State); +handle_cast_tx(TxCtx, {join, MemberId, Token, Fresh}, State) -> + #{name := Name, members := Members, leader := OldLeader, epoch := OldEpoch} = State, + MemberInfo = #{ + joined_at => erlang:system_time(millisecond), join_token => Token, fresh => Fresh + }, + NewMembers = Members#{MemberId => MemberInfo}, + {NewLeader, NewEpoch} = elect_leader(TxCtx, Name, NewMembers, OldLeader, OldEpoch), + NewState = State#{members => NewMembers, leader => NewLeader, epoch => NewEpoch}, + %% Every join routes through the leader so the joiner gets the snapshot from the same + %% sender as later name broadcasts (FIFO). + {noreply, NewState, [assume_distribute_action(MemberId)]}; +handle_cast_tx(TxCtx, {member_down, MemberId, Token}, State) -> + #{name := Name, members := Members, leader := OldLeader, epoch := OldEpoch} = State, + case maps:get(MemberId, Members, undefined) of + undefined -> + {noreply, State}; + #{join_token := StoredToken} when StoredToken =/= Token -> + %% Stale: member rejoined with a new token since this DOWN was detected. + {noreply, State}; + _ -> + NewMembers = maps:remove(MemberId, Members), + {NewLeader, NewEpoch} = elect_leader(TxCtx, Name, NewMembers, OldLeader, OldEpoch), + NewState = State#{members => NewMembers, leader => NewLeader, epoch => NewEpoch}, + case NewLeader =/= OldLeader of + true -> + {noreply, NewState, [assume_distribute_action(undefined)]}; + false -> + %% Survivors saw the death via their own monitors; no broadcast. + {noreply, NewState} + end + end; +%% Presence writes (§4.9) are casts for latency: `ok` means accepted, not committed. +%% Idempotent upserts by application id, so a write lost before consume is recoverable. +handle_cast_tx(_TxCtx, {subscribe, SubId, Watch, Notify}, State) -> + Subs = subscriptions_of(State), + NewState = State#{subscriptions => Subs#{SubId => {Watch, Notify}}}, + {noreply, NewState, [presence_push_action({subscribe, SubId, Watch, Notify})]}; +handle_cast_tx(_TxCtx, {unsubscribe, SubId}, State) -> + Subs = subscriptions_of(State), + NewState = State#{subscriptions => maps:remove(SubId, Subs)}, + {noreply, NewState, [presence_push_action({unsubscribe, SubId})]}; +handle_cast_tx(_TxCtx, unsubscribe_all, State) -> + {noreply, State#{subscriptions => #{}}, [presence_push_action(unsubscribe_all)]}. + +%% Post-commit: an unreachable leader must not fail the action (the change is already +%% committed; members self-correct on the next membership event). +assume_distribute_action(TriggerMember) -> + fun(#{members := All, leader := Leader, epoch := Epoch}) -> + case Leader of + undefined -> + ok; + _ -> + try + %% The leader pulls subscriptions from its co-located elector. + call_to_member( + Leader, + {elector_assume_and_distribute, TriggerMember, maps:keys(All), + member_tokens(All), fresh_ids(All), Epoch} + ) + catch + exit:Reason -> + depose_if_dead(Leader, Reason, All) + end + end + end. + +%% Rolling upgrade: durable state may predate `subscriptions`. +subscriptions_of(State) -> + maps:get(subscriptions, State, #{}). + +%% `noproc` means the leader's node answered but its registry is gone; leaving it as +%% leader strands the cluster, so force a re-election (fenced with its current token). +%% Not-connected is ambiguous and deliberately not deposed: disambiguating would open a +%% distribution connection and defeat partition detection. Timeouts = slow, left alone. +depose_if_dead(Leader, Reason, All) -> + case is_noproc(Reason) andalso maps:get(Leader, All, undefined) of + #{join_token := Token} -> + dgen_server:cast(self(), {member_down, Leader, Token}); + _ -> + ok + end. + +is_noproc(noproc) -> true; +is_noproc({noproc, _}) -> true; +is_noproc(_) -> false. + +%% --------------------------------------------------------------------------- +%% handle_call_tx/4 (all elector calls) +%% --------------------------------------------------------------------------- + +%% dgen_server dispatches every call (including priority reads) to the `_tx` variant +%% when exported, so there is deliberately no `handle_call/3`. +-if(?DOCATTRS). +-doc """ +Handles the elector's read-only priority calls: `get_leader`, `get_members`, `get_epoch`, +`{get_member_token, _}`, and `get_subscriptions` (the durable subscription map, §4.9). +The presence writes are casts — see handle_cast_tx/3. +""". +-endif. +-spec handle_call_tx(dgen_server:tx_ctx(), term(), dgen_server:from(), registry_state()) -> + dgen_server:reply_ret(). +handle_call_tx(_TxCtx, get_leader, _From, State) -> + {reply, maps:get(leader, State, undefined), State}; +handle_call_tx(_TxCtx, get_members, _From, State) -> + {reply, maps:keys(maps:get(members, State, #{})), State}; +handle_call_tx(_TxCtx, get_epoch, _From, State) -> + {reply, maps:get(epoch, State, 0), State}; +%% Fresh from durable state, so a `{member_down}` is fenced with the token the elector +%% actually holds, not a stale cached one that would get a genuine death discarded (§5.7). +handle_call_tx(_TxCtx, {get_member_token, MemberId}, _From, State) -> + Token = + case maps:get(MemberId, maps:get(members, State, #{}), undefined) of + #{join_token := T} -> T; + _ -> undefined + end, + {reply, Token, State}; +handle_call_tx(_TxCtx, get_subscriptions, _From, State) -> + {reply, subscriptions_of(State), State}. + +%% Fire-and-forget; a dropped push is caught up by the reseed on the next handoff. +presence_push_action(Update) -> + fun(#{leader := Leader, epoch := Epoch}) -> + case Leader of + undefined -> ok; + _ -> push_to_leader(Leader, {presence_update, Update, Epoch}) + end + end. + +%% Same auto-reconnect guard as call_to_member/2. +push_to_leader({Node, Name}, Msg) -> + case dgen_utils:node_reachable(Node) of + true -> gen_statem:cast({Name, Node}, Msg); + false -> ok + end. + +-if(?DOCATTRS). +-doc """ +Client-side helper: read the token the elector holds for `MemberId` right now, +falling back to `Default` if the member is unknown or the read fails. + +A priority read (bypasses the durable queue). This is the fence for a +`{member_down}`: reporting the elector's *current* token means a peer that has +genuinely rejoined (advancing its token) is not clobbered by a stale DOWN, while a +truly dead member is still reaped. Pure — safe to call from an off-loop helper; the +member's peer-DOWN path and the connector's reap both use it. +""". +-endif. +-spec member_token(pid(), member_id(), Default) -> reference() | Default. +member_token(Elector, MemberId, Default) -> + try dgen_server:priority_call(Elector, {get_member_token, MemberId}) of + T when is_reference(T) -> T; + _ -> Default + catch + _:_ -> Default + end. + +%% --------------------------------------------------------------------------- +%% Internal helpers +%% --------------------------------------------------------------------------- + +%% Sticky incumbent, else the local member (the transaction-race winner), else +%% lists:min/1. Epoch increments only when a new leader is chosen. +-spec elect_leader( + dgen_server:tx_ctx(), + atom(), + #{member_id() => member_info()}, + member_id() | undefined, + non_neg_integer() +) -> {member_id() | undefined, non_neg_integer()}. +elect_leader(#{td := {Tx, Dir}, tuid := Tuid}, Name, Members, OldLeader, OldEpoch) -> + {Leader, Epoch} = + case OldLeader =/= undefined andalso maps:is_key(OldLeader, Members) of + true -> + {OldLeader, OldEpoch}; + false -> + LocalId = {node(), dgen_registry:member_name(Name)}, + NewLeader = + case maps:is_key(LocalId, Members) of + true -> + LocalId; + false -> + case maps:keys(Members) of + [] -> undefined; + Ids -> lists:min(Ids) + end + end, + {NewLeader, OldEpoch + 1} + end, + B = dgen_config:backend(), + B:set(Tx, B:dir_pack(Dir, leader_key_tuple(Tuid)), term_to_binary({Leader, Epoch})), + {Leader, Epoch}. + +-if(?DOCATTRS). +-doc """ +Returns the packed backend key for the leader value. + +Exported so callers can set up a backend watch without going through the +elector process. +""". +-endif. +-spec leader_db_key(dgen_backend:dir(), dgen_server:tuid()) -> dgen_backend:key(). +leader_db_key(Dir, Tuid) -> + B = dgen_config:backend(), + B:dir_pack(Dir, leader_key_tuple(Tuid)). + +leader_key_tuple(Tuid) -> + dgen_key:extend(Tuid, <<"leader">>). + +member_tokens(Members) -> + maps:map(fun(_, #{join_token := T}) -> T end, Members). + +fresh_ids(Members) -> + [Id || {Id, Info} <- maps:to_list(Members), maps:get(fresh, Info, false)]. + +call_to_member({Node, Name}, Msg) -> + %% Never let the call auto-reconnect a disconnected node: the resulting nodeup makes + %% the peer re-join with a new token, so its `{member_down}` is discarded as stale + %% and the partition goes undetected (§5.7). + case dgen_utils:node_reachable(Node) of + true -> gen_statem:call({Name, Node}, Msg, ?SnapshotTimeout); + false -> exit({nodedown, Node}) + end. diff --git a/src/dgen_registry/dgen_registry_intents.erl b/src/dgen_registry/dgen_registry_intents.erl new file mode 100644 index 0000000..bcf32a1 --- /dev/null +++ b/src/dgen_registry/dgen_registry_intents.erl @@ -0,0 +1,252 @@ +-module(dgen_registry_intents). + +-define(DOCATTRS, ?OTP_RELEASE >= 27). + +-if(?DOCATTRS). +-moduledoc false. +-endif. + +%% A non-leader member's unfinished client requests: writes only the leader can commit. +%% `§N.N` refers to docs/design/dgen_registry_design.md. +%% +%% An intent owes a reply (a blocked caller's `From`), an effect (a change that must +%% reach the leader), or both. It is a kind in a phase, keyed by its forward's reference; +%% a reply under an unknown reference (superseded or answered) is ignored. +%% +%% kinds phases +%% {register, Name, Pid, Meta} awaiting_leader no reachable leader +%% {unregister, Name, ReleasedPid} {forwarded, Leader} sent; awaiting a reply +%% {set_meta, Name, Index, Data} {awaiting_version, V, L} L committed it at V; +%% awaiting our replica +%% +%% Three events move it: on_reply/4 (the leader answered), on_version/2 (our replica +%% advanced), redrive/5 (a leader may have been established). +%% +%% Re-drive follows idempotence, by kind alone: +%% - register: re-routed until its outcome is known. Same pid, same name is idempotent +%% (§5.2), so `no` is always adjudicated, never caused by leadership moving. +%% - unregister: re-driven as a pid-guarded retract, safe against re-registration. +%% - set_meta: a replace, unsafe to re-send under reordering; if its leader no longer +%% leads, the caller gets `{error, no_leader}`. +%% +%% Pure: functions return effects for the member to carry out. +%% +%% *This documentation is LLM-generated. See the AI disclosure in README.md.* + +-export([new/0, submit/5, on_reply/4, on_version/2, redrive/5, prune/3]). + +-export_type([t/0, kind/0, route/0, scope/0, effect/0]). + +-type member_id() :: dgen_registry_elector:member_id(). +-type from() :: gen_server:from(). + +-type kind() :: + {register, Name :: term(), pid(), Meta :: {map(), term()}} + | {unregister, Name :: term(), ReleasedPid :: pid() | undefined} + | {set_meta, Name :: term(), Index :: map(), Data :: term()}. + +-type phase() :: + awaiting_leader + | {forwarded, member_id()} + | {awaiting_version, non_neg_integer(), member_id()}. + +-record(intent, { + kind :: kind(), + phase :: phase(), + %% `answered`: the reply was given but the effect is still owed. + from :: from() | answered, + %% Wall-clock ms of the request; compared against the caller timeout. + at :: integer(), + %% Arrival order. Effects are emitted in this order, never map (reference) order, + %% so the message schedule is reproducible under deterministic simulation. + seq :: non_neg_integer() +}). + +-record(intents, { + all = #{} :: #{reference() => #intent{}}, + %% Ref => V of the `awaiting_version` subset: a replica advance scans only these. + deferred = #{} :: #{reference() => non_neg_integer()}, + next_seq = 0 :: non_neg_integer() +}). + +-opaque t() :: #intents{}. + +%% Where a request can go right now, as the member sees it. +-type route() :: self | {remote, member_id()} | none. + +-type scope() :: moved | all. + +-type effect() :: + {reply, from(), term()} + %% Tracked forward to the leader. + | {forward, member_id(), reference(), kind()} + %% Untracked pid-guarded clear. + | {retract, member_id(), Name :: term(), pid()} + %% We lead: our own commit pipeline. + | {enqueue, kind(), from() | answered} + %% A committed `yes` is ahead of our replica: ask the leader for a snapshot. + | {resync, member_id()} + %% A pre-version leader's `yes` (rolling upgrade): the reply binds the row. + | {insert_row, Name :: term(), pid(), Meta :: {map(), term()}}. + +-spec new() -> t(). +new() -> #intents{}. + +%% --------------------------------------------------------------------------- +%% Submission +%% --------------------------------------------------------------------------- + +%% Accept a request on a non-leader. A forwardable unregister/set_meta is tracked and +%% its caller waits for the commit; everything else, including an `answered` +%% unregister the pipeline surrendered, takes the re-drive path. +-spec submit(kind(), from() | answered, integer(), route(), t()) -> {t(), [effect()]}. +submit({set_meta, _, _, _}, From, _Now, Route, T) when Route =:= none; Route =:= self -> + {T, [{reply, From, {error, no_leader}}]}; +submit(Kind, From, Now, Route, T = #intents{next_seq = Seq}) -> + I = #intent{kind = Kind, phase = awaiting_leader, from = From, at = Now, seq = Seq}, + T1 = T#intents{next_seq = Seq + 1}, + case Route of + %% An `answered` unregister re-drives as a pid-guarded retract instead. + {remote, L} when element(1, Kind) =/= register, From =/= answered -> + Ref = make_ref(), + {put(Ref, I#intent{phase = {forwarded, L}}, T1), [{forward, L, Ref, Kind}]}; + _ -> + dispatch(make_ref(), I, Route, T1) + end. + +%% --------------------------------------------------------------------------- +%% The three events +%% --------------------------------------------------------------------------- + +%% The leader answered the forward under `Ref`; `Applied` is our replica's version. +%% A register `yes` is version-guarded (§5.5): answered only once our replica holds the +%% committing batch, so the handoff gather can see this member as a second holder. +%% Never write the row here: the same batch may have cleared the name too. +-spec on_reply(reference(), term(), non_neg_integer(), t()) -> {t(), [effect()]}. +on_reply(Ref, Reply, Applied, T = #intents{all = All}) -> + case All of + #{Ref := I = #intent{phase = {forwarded, Leader}}} -> + resolve(Ref, I, Leader, Reply, Applied, delete(Ref, T)); + _ -> + {T, []} + end. + +resolve(_Ref, #intent{kind = {register, Name, Pid, Meta}, from = From}, _L, Reply, _A, T) when + Reply =:= {register, yes, legacy} +-> + %% Pre-version leader: no version to guard on; the reply alone binds the row. + {T, [{reply, From, yes}, {insert_row, Name, Pid, Meta}]}; +resolve(Ref, I = #intent{kind = {register, _, _, _}, from = From}, L, {register, yes, V}, A, T) -> + case V =< A of + true -> {T, [{reply, From, yes}]}; + false -> {put(Ref, I#intent{phase = {awaiting_version, V, L}}, T), [{resync, L}]} + end; +resolve(_Ref, #intent{kind = {register, _, _, _}, from = From}, _L, {register, no, _}, _A, T) -> + {T, [{reply, From, no}]}; +resolve(_Ref, #intent{kind = {set_meta, _, _, _}, from = From}, _L, {set_meta, Result}, _A, T) -> + {T, [{reply, From, Result}]}; +resolve(_Ref, #intent{kind = {unregister, _, _}, from = From}, _L, unregister, _A, T) -> + {T, reply_if_owed(From, ok)}; +resolve(Ref, I, _Leader, _Mismatched, _Applied, T) -> + %% A reply whose shape does not match the kind it is keyed under: keep waiting. + {put(Ref, I, T), []}. + +%% Release every deferred `yes` our replica now covers. +-spec on_version(non_neg_integer(), t()) -> {t(), [effect()]}. +on_version(_Applied, T = #intents{deferred = D}) when map_size(D) =:= 0 -> + {T, []}; +on_version(Applied, T = #intents{all = All, deferred = D}) -> + Due = in_order([{Ref, maps:get(Ref, All)} || {Ref, V} <- maps:to_list(D), V =< Applied]), + lists:foldl( + fun({Ref, #intent{from = From}}, {Acc, Effs}) -> + {delete(Ref, Acc), Effs ++ [{reply, From, yes}]} + end, + {T, []}, + Due + ). + +%% Re-drive unresolved intents by kind (see header); register/set_meta past `TTL` (the +%% caller timeout) are dropped unreplied. A register awaiting a version from the current +%% leader is kept; one forwarded to it is kept only when `Scope = moved`. With `all` (a +%% node came back) that forward may have died with the link, so it is re-sent; a +%% duplicate is decided idempotently but costs a second commit. +-spec redrive(route(), scope(), integer(), non_neg_integer(), t()) -> {t(), [effect()]}. +redrive(Route, Scope, Now, TTL, T = #intents{all = All}) -> + {T1, EffsRev} = lists:foldl( + fun({Ref, I}, {Acc, Effs}) -> + {Acc1, New} = redrive_one(Ref, I, Route, Scope, Now, TTL, delete(Ref, Acc)), + {Acc1, lists:reverse(New, Effs)} + end, + {T, []}, + in_order(maps:to_list(All)) + ), + {T1, lists:reverse(EffsRev)}. + +redrive_one(Ref, I = #intent{kind = {register, _, _, _}, at = At}, Route, Scope, Now, TTL, T) -> + case {Now - At >= TTL, I#intent.phase, Route} of + {true, _, _} -> {T, []}; + {false, {awaiting_version, _V, L}, {remote, L}} -> {put(Ref, I, T), []}; + {false, {forwarded, L}, {remote, L}} when Scope =:= moved -> {put(Ref, I, T), []}; + {false, _, _} -> dispatch(make_ref(), I#intent{phase = awaiting_leader}, Route, T) + end; +redrive_one(Ref, I = #intent{kind = {unregister, _, _}}, Route, _Scope, _Now, _TTL, T) -> + dispatch(Ref, I, Route, T); +redrive_one(Ref, I = #intent{kind = {set_meta, _, _, _}}, Route, _Scope, Now, TTL, T) -> + #intent{from = From, at = At} = I, + case {Now - At >= TTL, I#intent.phase, Route} of + {true, _, _} -> {T, []}; + {false, {forwarded, L}, {remote, L}} -> {put(Ref, I, T), []}; + {false, _, _} -> {T, [{reply, From, {error, no_leader}}]} + end. + +%% Drop timed-out register/set_meta intents so a long leaderless window cannot grow +%% the map. Unregisters are kept: their effect is owed whether or not anyone waits. +-spec prune(integer(), non_neg_integer(), t()) -> t(). +prune(Now, TTL, T = #intents{all = All}) -> + maps:fold( + fun + (_Ref, #intent{kind = {unregister, _, _}}, Acc) -> Acc; + (Ref, #intent{at = At}, Acc) when Now - At >= TTL -> delete(Ref, Acc); + (_Ref, _I, Acc) -> Acc + end, + T, + All + ). + +%% --------------------------------------------------------------------------- +%% Internal +%% --------------------------------------------------------------------------- + +%% Send a register/unregister where `Route` allows, keeping it while it owes something. +dispatch(_Ref, #intent{kind = Kind = {register, _, _, _}, from = From}, self, T) -> + {T, [{enqueue, Kind, From}]}; +dispatch(Ref, I = #intent{kind = Kind = {register, _, _, _}}, {remote, L}, T) -> + {put(Ref, I#intent{phase = {forwarded, L}}, T), [{forward, L, Ref, Kind}]}; +dispatch(Ref, I = #intent{kind = {register, _, _, _}}, none, T) -> + {put(Ref, I#intent{phase = awaiting_leader}, T), []}; +%% An unregister's caller is answered `ok` now; the removal follows as a retract. +dispatch(_Ref, #intent{kind = {unregister, _Name, undefined}, from = From}, _Route, T) -> + %% Nothing was bound locally when it was asked: nothing to retract. + {T, reply_if_owed(From, ok)}; +dispatch(_Ref, #intent{kind = Kind = {unregister, _, _}, from = From}, self, T) -> + {T, reply_if_owed(From, ok) ++ [{enqueue, Kind, answered}]}; +dispatch(_Ref, #intent{kind = {unregister, Name, Pid}, from = From}, {remote, L}, T) -> + {T, reply_if_owed(From, ok) ++ [{retract, L, Name, Pid}]}; +dispatch(Ref, I = #intent{kind = {unregister, _, _}, from = From}, none, T) -> + %% CP: answer now, keep the intent until a leader can take it. + {put(Ref, I#intent{phase = awaiting_leader, from = answered}, T), reply_if_owed(From, ok)}. + +reply_if_owed(answered, _Value) -> []; +reply_if_owed(From, Value) -> [{reply, From, Value}]. + +put(Ref, I = #intent{phase = {awaiting_version, V, _L}}, T = #intents{all = A, deferred = D}) -> + T#intents{all = A#{Ref => I}, deferred = D#{Ref => V}}; +put(Ref, I, T = #intents{all = A, deferred = D}) -> + T#intents{all = A#{Ref => I}, deferred = maps:remove(Ref, D)}. + +%% `{Ref, Intent}` pairs in arrival order. +in_order(Pairs) -> + lists:sort(fun({_, A}, {_, B}) -> A#intent.seq =< B#intent.seq end, Pairs). + +delete(Ref, T = #intents{all = A, deferred = D}) -> + T#intents{all = maps:remove(Ref, A), deferred = maps:remove(Ref, D)}. diff --git a/src/dgen_registry/dgen_registry_member.erl b/src/dgen_registry/dgen_registry_member.erl new file mode 100644 index 0000000..b2614b0 --- /dev/null +++ b/src/dgen_registry/dgen_registry_member.erl @@ -0,0 +1,2256 @@ +-module(dgen_registry_member). +-behaviour(gen_statem). + +-define(DOCATTRS, ?OTP_RELEASE >= 27). + +-include("../../include/dgen_eta.hrl"). +-eta_observe( + {state, [ + member_id, + leader, + replica, + synced, + intents, + pipeline + ]} +). + +-if(?DOCATTRS). +-moduledoc false. +-endif. + +%% Per-node member of a `dgen_registry`: local name cache and consistent read/write +%% proxy. Section references (`§N.N`) point to `docs/design/dgen_registry_design.md`; +%% TLA+ action names to `formal/DgenRegistryReplication.tla`. +%% +%% ## Roles +%% +%% - Follower: applies the leader's `{names_batch, …}` / `{apply_names_snapshot, …}` +%% casts to its replica (a batch is one whole group commit, applied all or nothing), +%% and forwards writes and consistent reads to the leader as casts correlated by a +%% `Ref`. The loop never blocks on a call to another member: a blocking forward +%% could deadlock against the leader's replicate-before-ack wait. All replication +%% messages come from the leader process, so per-pair FIFO orders a snapshot before +%% any later batch, and a batch before the reply to a write it carries. +%% - Leader: the sole writer. At handoff it reconstructs the name map from the +%% freshest reachable replica (gathered off the loop; there is no durable name +%% state, §4.4), serves all writes through one group-commit pipeline, monitors +%% registered pids, and replicates each committed batch. A registration is acked +%% `yes` only once two members hold it (§8): a forwarding follower is the second +%% holder for free; a direct registration waits for a follower's `replicate_ack` +%% (degrade-open on timeout unless `strict_replication`). +%% +%% With no reachable leader a write is never answered `no`; the caller blocks as an +%% intent until a leader appears or its own call times out (§3). +%% +%% ## Where things live +%% +%% - `dgen_registry_replica`: the `protected` ETS table (rows `{Name, Pid, Index, Data}`, +%% read lock-free by callers), inverted index, applied version and epoch, and the +%% batch/snapshot admission rules. This process is its only writer; it is not durable. +%% - `dgen_registry_intents`: requests this member accepted but cannot decide, and +%% their re-drive rules on leader change. +%% - `dgen_registry_pipeline`: the leader's commit queue, in-flight commit, pending +%% replicate acks, and the surrender rule for ops it cannot finish. +%% - `dgen_registry_conflicts`: release trail, kill budget, `degraded` flag (§5.6). +%% - `dgen_registry_connector`: keeps the distribution mesh in step with membership. +%% - `dgen_registry_elector`: elects the leader and asks this member to assume. +%% +%% Multi-key queries run on this member's mailbox, between batch applies, so they see a +%% whole-batch-consistent view (§6); single-key snapshot reads bypass the member. +%% +%% ## States +%% +%% A `gen_statem` whose state is a projection of data (`state_of/1`, recomputed after +%% every event by `next/2`), so it cannot drift: +%% +%% searching no leader known. +%% assuming a handoff gather is in flight (`assume_ref` armed). +%% leader this member leads; entering runs `assume_leadership/1`, leaving runs +%% `relinquish_leadership/1`. +%% follower following a known leader. +%% +%% Gap/resync, leader reachability and the commit pipeline are deliberately data, not +%% state: each is orthogonal to role (a deposed-but-uninformed leader can still gap, +%% as the spec's RecvBcastGap has no leaderView guard). +%% +%% *This documentation is LLM-generated. See the AI disclosure in README.md.* + +%% Per-member bound on the handoff gather; a member that misses it is skipped. +-define(GATHER_TIMEOUT, 2000). +%% Pause between re-gathers while the assume gather is behind the durable version key +%% (the SafeAssume fence in the TLA+ spec); bounded overall by ?GATHER_TIMEOUT. +-define(ASSUME_CATCHUP_INTERVAL, 25). +%% Best-effort bound on pulling durable subscriptions from the elector at handoff (§4.9). +-define(SUBS_PULL_TIMEOUT, 1000). + +%% Period of the maintenance pass pruning the release trail and kill budget (§5.6). +%% Trail pruning is suspended while any member is disconnected: it misses unregister +%% broadcasts entirely, so the trail must outlive the partition. +-define(PRUNE_INTERVAL, 30000). + +%% How long a follower waits after requesting a resync (it observed a gap in the +%% leader's broadcast stream) before it may request again. +-define(RESYNC_RETRY, 2000). + +%% Names monitored per `{monitor_sweep, …}` step (see assume_leadership/1). +-define(MONITOR_SWEEP_CHUNK, 5000). + +%% Period of the leader's replication heartbeat: an empty `{names_batch, [], …}` at its +%% applied version. Without it a follower that lost the stream's tail (or its resync +%% request) would not detect the gap until the next write. +-define(REPLICA_HEARTBEAT_INTERVAL, 5000). + +%% Delay before re-enqueueing (or, if deposed, surrendering) the ops of a failed batch, +%% so a persistently failing backend does not spin. +-define(REQUEUE_DELAY, 1000). + +%% Bound on the per-peer aliveness probe used when adjudicating a suspected conflict. +-define(ALIVE_PROBE_TIMEOUT, 1000). + +-export([start_link/2]). +-export([ + init/1, + callback_mode/0, + handle_event/4, + terminate/3, + code_change/4 +]). +%% Exported for unit tests of the batch planner. +-export([plan_batch/4]). + +%% Presence subscription query, internal tagged form (see dgen_registry:query/0). +-type query() :: {'all', #{term() => term()}}. + +%% Application-supplied presence subscription id (§4.9). +-type sub_id() :: term(). + +-record(state, { + member_id :: dgen_registry_elector:member_id(), + %% Discovered once after init; the `one_for_all` supervisor restarts us with the + %% elector, so this pid is never stale. + elector :: pid() | undefined, + %% Backend handle for the leader's fenced version-key commit (dgen_registry_names). + tenant :: dgen_backend:tenant(), + tuid :: dgen_server:tuid(), + leader :: dgen_registry_elector:member_id() | undefined, + %% Local replica: ETS rows, inverted index, applied version, epoch. + replica :: dgen_registry_replica:t(), + %% Leader-only (§4.9): `#{SubId => {Watch, Notify}}`, seeded at assume and advanced + %% by `{presence_update, …}`. + subs = #{} :: #{sub_id() => {query(), query()}}, + %% Leader-only: names matching each watch query; diffed per commit for joined/left. + sub_matches = #{} :: #{sub_id() => #{term() => pid()}}, + %% Leader-only: pids matching each notify query; a pid entering it is sent the + %% current watch snapshot. + notify_matches = #{} :: #{sub_id() => #{term() => pid()}}, + %% Peer-member monitors (all members) + members :: #{dgen_registry_elector:member_id() => reference()}, + monitors :: #{reference() => dgen_registry_elector:member_id()}, + %% Registered-process monitors (leader only). No reverse map: each monitor is + %% tagged `{down, Name}`, so its DOWN names the binding. + name_to_ref :: #{term() => reference()}, + %% Requests awaiting a leader's decision; see dgen_registry_intents. + intents = dgen_registry_intents:new() :: dgen_registry_intents:t(), + %% Leader-only write ops not yet answered; see dgen_registry_pipeline. + pipeline = dgen_registry_pipeline:new() :: dgen_registry_pipeline:t(), + %% The epoch in which a fenced commit last made this member re-announce itself + %% (rejoin_once_fenced/1), so it does so at most once per epoch. + fence_rejoined = undefined :: undefined | non_neg_integer(), + %% Token for our `{join, Self, Token}`; refreshed on nodeup so a stale + %% `{member_down, Self, OldToken}` is discarded. + join_token :: reference(), + %% Peer join tokens, echoed in `{member_down, PeerId, Token}` so the elector can + %% discard a stale DOWN. + peer_tokens :: #{dgen_registry_elector:member_id() => reference()}, + %% Outstanding resync request timer; cleared when any snapshot arrives. + resync_timer = undefined :: undefined | reference(), + %% Ever synced (snapshot applied or leadership assumed)? Joins announce + %% `fresh = not synced`; the first sync re-announces with `fresh = false`. + synced = false :: boolean(), + %% Tags the in-flight off-loop handoff gather (§5.7); `{assume_gathered, …}` applies + %% only if it still matches, so a superseded gather is harmless. + assume_ref = undefined :: undefined | reference(), + %% Conflict repair state (§5.6); see dgen_registry_conflicts. + conflicts = dgen_registry_conflicts:new() :: dgen_registry_conflicts:t(), + %% Per-registry tuning options, resolved through dgen_config. + config = #{} :: dgen_config:config() +}). + +%% --------------------------------------------------------------------------- +%% Public API +%% --------------------------------------------------------------------------- + +-if(?DOCATTRS). +-doc "Starts the member process registered as `Name`.". +-endif. +-spec start_link(Name :: atom(), Args :: map()) -> gen_statem:start_ret(). +start_link(Name, Args) -> + gen_statem:start_link({local, Name}, ?MODULE, Args, []). + +%% --------------------------------------------------------------------------- +%% gen_statem callbacks +%% --------------------------------------------------------------------------- + +callback_mode() -> + [handle_event_function, state_enter]. + +init(#{name := Name, tenant := Tenant, tuid := Tuid} = Args) -> + Config = maps:get(config, Args, #{}), + MemberId = {node(), Name}, + %% Recreated empty on every (re)start (no heir) and rebuilt from the leader's + %% snapshot. + Replica = dgen_registry_replica:new(dgen_registry:names_table(Name)), + %% On `{nodeup, _}` the member re-announces its `{join}` (partition heal). + net_kernel:monitor_nodes(true), + %% Resolve the cached telemetry lookup now, while the tree starts: done lazily it is + %% a synchronous `code_server` call inside whichever member emits first, which under + %% `eta_run` simulation blocks on a process the scheduler does not own. + _ = telemetry_available(), + arm(?PRUNE_INTERVAL, prune_released), + arm(?REPLICA_HEARTBEAT_INTERVAL, replica_heartbeat), + %% Announcing needs the elector's pid; `discover_elector` runs as an internal event + %% because `supervisor:which_children/1` would deadlock while the supervisor still + %% waits on this init/1. + {ok, searching, + #state{ + member_id = MemberId, + elector = undefined, + tenant = Tenant, + tuid = Tuid, + leader = undefined, + replica = Replica, + members = #{}, + monitors = #{}, + name_to_ref = #{}, + join_token = make_ref(), + peer_tokens = #{}, + config = Config + }, + [{next_event, internal, discover_elector}]}. + +%% --------------------------------------------------------------------------- +%% handle_event/4 — dispatch, and the derived state +%% --------------------------------------------------------------------------- + +%% -- state enter: the leadership lifecycle, in one place --------------------- + +%% The assume/relinquish pairing lives on the role edges so no transition path can +%% forget half of it. Deliberately not here: redrive_intents (fires on non-edge +%% leadership events too) and fail_pending_acks (stays in do_leader_changed; a +%% re-assumed leader's stale acks are already fenced by BatchRef). A leader entering +%% `assuming` relinquishes at assume start; a death in the gather window is caught by +%% the next sweep's immediate `noproc` DOWN. +handle_event(enter, OldState, leader, Data) when OldState =/= leader -> + {keep_state, assume_leadership(Data)}; +handle_event(enter, leader, NewState, Data) when NewState =/= leader -> + {keep_state, relinquish_leadership(Data)}; +handle_event(enter, _OldState, _NewState, Data) -> + {keep_state, Data}; +%% -- the assume lifecycle --------------------------------------------------- + +%% A continuing leader absorbs a join on the fast path; otherwise a genuine change. +%% The epoch guard stays on Data: the `leader` state does not say which term. +handle_event( + {call, From}, + {elector_assume_and_distribute, MemberId, AllIds, Tokens, FreshIds, Epoch}, + State, + Data = #state{synced = Synced} +) -> + {reply, Reply, Data1} = + case + State =:= leader andalso Synced andalso MemberId =/= undefined andalso + epoch(Data) =:= Epoch + of + true -> assume_fast_path(MemberId, AllIds, Tokens, FreshIds, Epoch, Data); + false -> assume_genuine(MemberId, AllIds, Tokens, FreshIds, Epoch, Data) + end, + next(Data1, [{reply, From, Reply}]); +%% The gather continuation applies only in `assuming` and only for the assume that +%% armed it (the ref); otherwise it was superseded and is dropped. +handle_event( + info, + {assume_gathered, {_, _, _, _, _, Ref}, _, _, _, _, _, _} = Msg, + assuming, + Data = #state{assume_ref = Ref} +) -> + {noreply, Data1} = handle_info(Msg, Data), + next(Data1, []); +handle_event( + info, {assume_gathered, _Ctx, _Self, _Freshest, _MaxV, _Peers, _Subs, _Alive}, _AnyState, Data +) -> + {keep_state, Data}; +handle_event( + info, + {joiner_gathered, MemberId, _AllIds, _Tokens, Epoch, _Result} = Msg, + leader, + Data = #state{members = Members} +) when is_map_key(MemberId, Members) -> + case epoch(Data) =:= Epoch of + true -> + {noreply, Data1} = handle_info(Msg, Data), + next(Data1, []); + false -> + {keep_state, Data} + end; +handle_event(info, {joiner_gathered, _M, _A, _T, _E, _R}, _AnyState, Data) -> + {keep_state, Data}; +%% -- leader-only periodic work --------------------------------------------- + +%% Everyone keeps the timer running so a later election heartbeats immediately. +handle_event(info, replica_heartbeat, leader, Data) -> + broadcast_heartbeat(Data), + arm(?REPLICA_HEARTBEAT_INTERVAL, replica_heartbeat), + {keep_state, Data}; +handle_event(info, replica_heartbeat, _AnyState, Data) -> + arm(?REPLICA_HEARTBEAT_INTERVAL, replica_heartbeat), + {keep_state, Data}; +%% Monitor sweep steps apply only in the term that armed them. +handle_event(info, {monitor_sweep, Epoch, _Cont} = Msg, leader, Data) -> + case epoch(Data) =:= Epoch of + true -> + {noreply, Data1} = handle_info(Msg, Data), + next(Data1, []); + false -> + {keep_state, Data} + end; +handle_event(info, {monitor_sweep, _Epoch, _Cont}, _AnyState, Data) -> + {keep_state, Data}; +%% -- leader-only protocol serves ------------------------------------------- + +%% Only a member that believes it leads answers a resync request. +handle_event(cast, {resync_req, _FollowerId} = Msg, leader, Data) -> + {noreply, Data1} = handle_cast(Msg, Data), + next(Data1, []); +handle_event(cast, {resync_req, _FollowerId}, _AnyState, Data) -> + {keep_state, Data}; +%% A forwarded registration reaching a member mid-assume is postponed until the assume +%% resolves; if superseded, it stays silent and the forwarder re-drives. +handle_event(cast, {register_req, _Ref, _FollowerId, _Name, _Pid, _Meta}, assuming, Data) -> + {keep_state, Data, [postpone]}; +%% A non-leader stays silent; the follower re-drives the unregister intent. +handle_event(cast, {unregister_req, _Ref, _FollowerId, _Name} = Msg, leader, Data) -> + {noreply, Data1} = handle_cast(Msg, Data), + next(Data1, []); +handle_event(cast, {unregister_req, _Ref, _FollowerId, _Name}, _AnyState, Data) -> + {keep_state, Data}; +%% -- the generic shim ------------------------------------------------------- + +handle_event({call, From}, Msg, _State, Data) -> + case handle_call(Msg, From, Data) of + {reply, Reply, Data1} -> next(Data1, [{reply, From, Reply}]); + {noreply, Data1} -> next(Data1, []) + end; +handle_event(cast, Msg, _State, Data) -> + {noreply, Data1} = handle_cast(Msg, Data), + next(Data1, []); +handle_event(info, Msg, _State, Data) -> + {noreply, Data1} = handle_info(Msg, Data), + next(Data1, []); +handle_event(internal, Continue, _State, Data) -> + {noreply, Data1} = handle_continue(Continue, Data), + next(Data1, []). + +%% `assuming` wins: a mid-assume member clears its leader view on purpose. +state_of(#state{assume_ref = Ref}) when Ref =/= undefined -> assuming; +state_of(#state{leader = undefined}) -> searching; +state_of(#state{leader = Self, member_id = Self}) -> leader; +state_of(#state{}) -> follower. + +%% A same-state `next_state` keeps timeouts and fires no enter call, so deriving on +%% every event is harmless. +next(Data, Actions) -> + {next_state, state_of(Data), Data, Actions}. + +epoch(#state{replica = R}) -> dgen_registry_replica:epoch(R). +applied_version(#state{replica = R}) -> dgen_registry_replica:version(R). + +%% --------------------------------------------------------------------------- +%% handle_continue/2 +%% --------------------------------------------------------------------------- + +%% Safe here: the supervisor is no longer blocked on our init/1 (see init/1). +handle_continue( + discover_elector, State = #state{member_id = {_Node, Name} = MemberId, join_token = Token} +) -> + Elector = dgen_registry:elector_pid(Name), + true = is_pid(Elector), + %% `fresh = true`: never synced, so this member cannot be hiding any bindings. + dgen_server:cast(Elector, {join, MemberId, Token, true}), + {noreply, State#state{elector = Elector}}. + +%% --------------------------------------------------------------------------- +%% handle_call/3 +%% --------------------------------------------------------------------------- + +%% Readiness probe (dgen_registry:await_ready/2): a leader is known and state synced. +handle_call(ready, _From, State = #state{leader = Leader, synced = Synced}) -> + {reply, Leader =/= undefined andalso Synced, State}; +%% Status probe (dgen_registry:status/1). Reports this member's own *belief* about +%% leadership, which can lag the elector's committed view during a handoff (§5.1). +handle_call(status, _From, State) -> + #state{member_id = Self, leader = Leader, synced = Synced} = State, + {reply, + #{ + member_id => Self, + leader => Leader, + is_leader => Leader =:= Self, + epoch => epoch(State), + synced => Synced, + applied_version => applied_version(State) + }, + State}; +%% ---- Name registration ---------------------------------------------------- + +%% Routing (leader / follower / no leader) is in route_register/5; the only immediate +%% reply is an adjudicated `no`. +handle_call({register, LogicalName, Pid, Meta}, From, State) -> + {noreply, route_register(LogicalName, Pid, Meta, From, State)}; +%% ---- Name unregistration ---------------------------------------------------- + +handle_call({unregister, LogicalName}, From, State) -> + route_unregister(LogicalName, From, State); +%% ---- Metadata write (set_metadata) ---------------------------------------- + +handle_call( + {set_metadata, LogicalName, Index, Data}, + From, + State = #state{leader = Leader, member_id = Leader} +) -> + {noreply, enqueue_op({set_meta, LogicalName, Index, Data, {meta, From}}, State)}; +%% Follower: forward as an intent. The reply routes back through this member so the +%% leader's `{names_batch}` (FIFO, ahead of it) has updated our row first: +%% read-after-write for a local snapshot read. +handle_call({set_metadata, LogicalName, Index, Data}, From, State) -> + {noreply, submit_intent({set_meta, LogicalName, Index, Data}, From, State)}; +%% ---- Consistent metadata read (leader only) -------------------------------- + +handle_call( + {get_metadata, LogicalName}, + From, + State = #state{leader = Leader, member_id = Leader, replica = R} +) -> + %% Fenced (Guarantee 5): the row is captured now, and a spawned helper verifies + %% leadership against the durable leader key before replying, so a deposed leader + %% cannot answer authoritatively. + reply_fenced(From, dgen_registry_replica:lookup_metadata(R, LogicalName), undefined, State), + {noreply, State}; +handle_call( + {get_metadata, LogicalName}, + From, + State = #state{leader = Leader} +) when Leader =/= undefined -> + %% The leader answers `From` directly; unreachable leader answers now. + case member_reachable(Leader) of + true -> + cast_to_member(Leader, {get_metadata_req, LogicalName, From}), + {noreply, State}; + false -> + {reply, undefined, State} + end; +handle_call({get_metadata, _LogicalName}, _From, State) -> + {reply, undefined, State}; +%% ---- Indexed query (on the mailbox, so batch-consistent, §6) --------------- + +handle_call({query, Constraints}, _From, State = #state{replica = R}) -> + {reply, dgen_registry_replica:query_constraints(R, Constraints), State}; +%% Consistent query: on the leader, fenced like the other consistent reads; otherwise +%% forwarded and answered by the leader directly. +handle_call( + {query_consistent, Constraints}, + From, + State = #state{leader = Leader, member_id = Leader, replica = R} +) -> + reply_fenced(From, dgen_registry_replica:query_constraints(R, Constraints), [], State), + {noreply, State}; +handle_call( + {query_consistent, Constraints}, From, State = #state{leader = Leader} +) when Leader =/= undefined -> + case member_reachable(Leader) of + true -> + cast_to_member(Leader, {query_req, Constraints, From}), + {noreply, State}; + false -> + {reply, [], State} + end; +handle_call({query_consistent, _Constraints}, _From, State) -> + {reply, [], State}; +%% ---- Consistent read (leader only) ---------------------------------------- + +handle_call( + {whereis, LogicalName}, + From, + State = #state{leader = Leader, member_id = Leader} +) -> + %% Fenced, as in the {get_metadata} leader clause. + reply_fenced( + From, dgen_registry_replica:lookup_pid(State#state.replica, LogicalName), undefined, State + ), + {noreply, State}; +handle_call( + {whereis, LogicalName}, + From, + State = #state{leader = Leader} +) when Leader =/= undefined -> + %% The leader replies to `From` directly so this loop never blocks on it. + case member_reachable(Leader) of + true -> + cast_to_member(Leader, {whereis_req, LogicalName, From}), + {noreply, State}; + false -> + {reply, undefined, State} + end; +handle_call({whereis, _LogicalName}, _From, State) -> + {reply, undefined, State}; +%% ---- Snapshot read --------------------------------------------------------- +%% Served lock-free by the caller (`dgen_registry:whereis_name/1`); no handler here. + +%% ---- Handoff gather (any member) ------------------------------------------- + +%% Our records, applied version and release trail, for a new leader's freshest-wins +%% reconstruction (§5.7 / §4.4) and trail merge (§5.6). +handle_call( + get_names_snapshot, + _From, + State = #state{replica = R, conflicts = C} +) -> + %% Shipped as an off-heap binary; see dgen_registry_replica:encode/1. + Released = dgen_registry_conflicts:trail(C), + {reply, {dgen_registry_replica:encode(R), dgen_registry_replica:version(R), Released}, + State}; + +handle_call(_Request, _From, State) -> + {reply, {error, unknown_call}, State}. + +%% ---- Elector calls (dispatched from handle_event) --------------------------- + +%% Bodies of `{elector_assume_and_distribute, …}`. `MemberId` is the joiner that +%% triggered it (`undefined` for a member_down); `AllIds` the full member list. +%% +%% Fast path: a continuing, synced leader for this epoch is authoritative (every +%% follower is a prefix of its stream), so it skips the gather and re-snapshotting +%% the followers; it only monitors and onboards the joiner. +assume_fast_path(MemberId, AllIds, Tokens, FreshIds, Epoch, State = #state{member_id = Self}) -> + State1 = merge_peer_tokens( + Tokens, add_member_monitors(extra_member_ids(MemberId, AllIds, Self), State) + ), + %% A fresh joiner holds no bindings: onboard now. A rejoining previously-synced + %% member may hold divergent bindings (§5.6 partition heal), so gather its replica + %% off the loop and adjudicate it in {joiner_gathered} first (best-effort). + case MemberId =:= Self orelse lists:member(MemberId, FreshIds) of + true -> + {reply, ok, onboard_joiner(MemberId, AllIds, Tokens, State1)}; + false -> + spawn_joiner_gather(MemberId, AllIds, Tokens, Epoch, State1), + {reply, ok, State1} + end. + +assume_genuine(MemberId, AllIds, Tokens, FreshIds, Epoch, State = #state{member_id = Self}) -> + %% Genuine change: the freshest-wins gather is a network fan-out, so it runs off + %% the loop and the elector is answered `ok` now (an inline gather would time out + %% its call and stall queued membership changes). Our own records and version are + %% captured now so a broadcast landing mid-gather cannot skew freshest-wins. + SelfRecords = dgen_registry_replica:records(State#state.replica), + OtherIds = lists:delete(Self, AllIds), + Ref = make_ref(), + spawn_assume_gather( + {MemberId, AllIds, Tokens, FreshIds, Epoch, Ref}, + SelfRecords, + applied_version(State), + OtherIds, + State#state.elector, + self(), + State#state.tenant, + State#state.tuid, + State#state.conflicts + ), + %% No leader for the window: writes block and are re-driven by the continuation + %% rather than forwarded to the old, possibly-dead leader. + {reply, ok, State#state{assume_ref = Ref, leader = undefined}}. + +%% --------------------------------------------------------------------------- +%% handle_cast/2 +%% --------------------------------------------------------------------------- + +%% Leader: a forwarded registration, answered via {register_reply} once committed. +handle_cast( + {register_req, Ref, FollowerId, LogicalName, Pid, Meta}, + State = #state{leader = Leader, member_id = Leader} +) -> + case reject_new_when_degraded(LogicalName, State) of + true -> + cast_to_member(FollowerId, {register_reply, Ref, no}), + {noreply, State}; + false -> + {noreply, enqueue_op({add, LogicalName, Pid, Meta, {forward, FollowerId, Ref}}, State)} + end; +%% Not the leader: stay silent rather than give a non-adjudicated `no` (§3); the +%% follower re-drives its intent. +handle_cast({register_req, _Ref, _FollowerId, _LogicalName, _Pid, _Meta}, State) -> + {noreply, State}; +%% Leader: a forwarded unregister, answered via {unregister_reply} once committed. +handle_cast( + {unregister_req, Ref, FollowerId, LogicalName}, + State = #state{leader = Leader, member_id = Leader, replica = R} +) -> + ReleasedPid = dgen_registry_replica:lookup_pid(R, LogicalName), + {noreply, + enqueue_op( + {remove, LogicalName, ReleasedPid, {forward_unreg, FollowerId, Ref}}, + row_delete(State, LogicalName) + )}; +%% Follower: a forwarded unregister committed (FIFO: after its {names_batch}). +handle_cast({unregister_reply, Ref}, State) -> + {noreply, intent_reply(Ref, unregister, State)}; +%% Leader: a forwarded set_metadata, answered via {set_meta_reply} once committed. +handle_cast( + {set_meta_req, Ref, FollowerId, LogicalName, Index, Data}, + State = #state{leader = Leader, member_id = Leader} +) -> + {noreply, + enqueue_op({set_meta, LogicalName, Index, Data, {forward_meta, FollowerId, Ref}}, State)}; +%% Not the leader (stale belief): reject so the caller retries. +handle_cast({set_meta_req, Ref, FollowerId, _LogicalName, _Index, _Data}, State) -> + cast_to_member(FollowerId, {set_meta_reply, Ref, {error, no_leader}}), + {noreply, State}; +%% Follower: FIFO puts the matching {names_batch} ahead of this, so read-after-write +%% holds for a local get_metadata. +handle_cast({set_meta_reply, Ref, Result}, State) -> + {noreply, intent_reply(Ref, {set_meta, Result}, State)}; +%% Leader: forwarded consistent reads answer the caller's `From` directly, fenced. +handle_cast( + {get_metadata_req, LogicalName, From}, + State = #state{leader = Leader, member_id = Leader, replica = R} +) -> + reply_fenced(From, dgen_registry_replica:lookup_metadata(R, LogicalName), undefined, State), + {noreply, State}; +handle_cast({get_metadata_req, _LogicalName, From}, State) -> + gen_server:reply(From, undefined), + {noreply, State}; +handle_cast( + {query_req, Constraints, From}, + State = #state{leader = Leader, member_id = Leader, replica = R} +) -> + reply_fenced(From, dgen_registry_replica:query_constraints(R, Constraints), [], State), + {noreply, State}; +handle_cast({query_req, _Constraints, From}, State) -> + gen_server:reply(From, []), + {noreply, State}; +handle_cast( + {whereis_req, LogicalName, From}, + State = #state{leader = Leader, member_id = Leader, replica = R} +) -> + reply_fenced(From, dgen_registry_replica:lookup_pid(R, LogicalName), undefined, State), + {noreply, State}; +%% Not the leader (stale belief): `undefined`; the caller can retry. +handle_cast({whereis_req, _LogicalName, From}, State) -> + gen_server:reply(From, undefined), + {noreply, State}; +%% Follower: a forwarded registration's verdict at commit Version. A `yes` is +%% version-guarded (§5.5): answered only once our replica has applied Version, so we +%% are a second holder the handoff gather (§5.7) is guaranteed to see. Normally true +%% (FIFO: the batch precedes the reply); if the batch was gap-refused the ack waits +%% for the resync. See dgen_registry_intents. +handle_cast({register_reply, Ref, Result, Version}, State) -> + {noreply, intent_reply(Ref, {register, Result, Version}, State)}; +%% Rolling upgrade: unversioned reply from a pre-version leader; `legacy`, not 0 (see +%% dgen_registry_intents:on_reply/4). +handle_cast({register_reply, Ref, Result}, State) -> + {noreply, intent_reply(Ref, {register, Result, legacy}, State)}; +%% Follower: confirm we hold a batch's bindings (FIFO: after its {names_batch}). +%% Version-guarded: if we have not applied that version (gap, stale epoch) we must not +%% claim to hold them. +handle_cast( + {replicate_sync, BatchRef, LeaderId, Version}, + State = #state{member_id = Self} +) -> + case applied_version(State) >= Version of + true -> cast_to_member(LeaderId, {replicate_ack, BatchRef, Self}); + false -> ok + end, + {noreply, State}; +%% Leader: once `register_replicas` distinct followers ack, answer the batch's direct +%% registrations `yes`. +handle_cast({replicate_ack, BatchRef, FollowerId}, State = #state{pipeline = P}) -> + case dgen_registry_pipeline:ack(BatchRef, FollowerId, P) of + {done, Direct, TimerRef, P1} -> + erlang:cancel_timer(TimerRef), + deliver_direct(Direct), + {noreply, State#state{pipeline = P1}}; + {waiting, P1} -> + {noreply, State#state{pipeline = P1}} + end; +%% Replication broadcast: one message per committed batch. dgen_registry_replica +%% admits it only if contiguous (PrevV), so our replica stays a prefix of the leader's +%% stream; a gap requests a resync. Applying it as a unit keeps "same applied_version +%% implies same content" true (see broadcast_batch/5). +handle_cast({names_batch, Ops, Epoch, PrevV, Version, LeaderId}, State = #state{replica = R}) -> + State1 = + case dgen_registry_replica:offer_batch(Epoch, PrevV, Version, Ops, R) of + {applied, R1} -> + intents_advanced(record_released(Ops, cancel_resync(State#state{replica = R1}))); + gap -> + request_resync(LeaderId, State); + _StaleEpochOrDuplicate -> + State + end, + {noreply, State1}; +%% Leader: answer a resync request with a snapshot at our epoch and applied version. +handle_cast( + {resync_req, FollowerId}, + State = #state{leader = Leader, member_id = Leader, replica = R} +) -> + RecordsBin = dgen_registry_replica:encode(R), + cast_to_member( + FollowerId, + {apply_names_snapshot, RecordsBin, Leader, [], #{}, dgen_registry_replica:epoch(R), + dgen_registry_replica:version(R)} + ), + {noreply, State}; +%% A pid-guarded retract (strict_replication fail-closed, §8) forwarded by a member +%% that could not commit it, so a binding answered `no` cannot live on. The pid guard +%% never clobbers a name since re-registered. +handle_cast( + {retract_req, LogicalName, Pid}, State = #state{leader = Leader, member_id = Leader} +) -> + {noreply, enqueue_op({retract, LogicalName, Pid}, State)}; +handle_cast({retract_req, LogicalName, Pid}, State = #state{leader = Leader}) when + Leader =/= undefined +-> + cast_to_member(Leader, {retract_req, LogicalName, Pid}), + {noreply, State}; +handle_cast({retract_req, LogicalName, Pid}, State) -> + %% No leader (likely mid-handoff): hold it as an answered unregister intent so + %% it is re-driven rather than lost. + {noreply, submit_intent({unregister, LogicalName, Pid}, answered, State)}; +%% Leadership snapshot from the new leader: replica, leader transition and member +%% monitors apply in one cast, so nothing interleaves. The version re-baselines us. +handle_cast( + {apply_names_snapshot, RecordsPayload, NewLeader, ExtraMembers, Tokens, Epoch, Version}, + State = #state{member_id = Self, leader = OldLeader, replica = R} +) -> + %% Version-monotonic, not merely epoch-guarded: a late snapshot can never roll the + %% replica backward (the handoff-gather race in the TLA+ spec). A refused one is + %% harmless; gap/resync re-baselines us. + Records = dgen_registry_replica:decode_records(RecordsPayload), + case dgen_registry_replica:offer_snapshot(Epoch, Version, Records, R) of + {accepted, R1} -> + %% Replica before the leader transition: assume_leadership reads the + %% names back from the table. + State0 = cancel_resync(State#state{replica = R1}), + State1 = do_leader_changed(NewLeader, OldLeader, Self, State0), + %% Supersedes any in-flight assume gather. + State2 = State1#state{assume_ref = undefined}, + State3 = add_member_monitors(ExtraMembers, State2), + State4 = merge_peer_tokens(Tokens, State3), + {noreply, + maybe_start_commit(redrive_intents(moved, intents_advanced(mark_synced(State4))))}; + stale -> + {noreply, State} + end; +%% Rolling upgrade: legacy fire-and-forget unregister casts from pre-call nodes. +handle_cast( + {unregister, LogicalName}, + State = #state{leader = Leader, member_id = Leader, replica = R} +) -> + %% Capture the bound pid before the optimistic delete, or plan_op/3 could not + %% tell a real unregister from a no-op. + ReleasedPid = dgen_registry_replica:lookup_pid(R, LogicalName), + {noreply, + enqueue_op({remove, LogicalName, ReleasedPid, undefined}, row_delete(State, LogicalName))}; +handle_cast({unregister, LogicalName}, State = #state{leader = Leader}) when + Leader =/= undefined +-> + cast_to_member(Leader, {unregister, LogicalName}), + {noreply, row_delete(State, LogicalName)}; +%% Leader fast path onboarding a joiner: an up-to-date follower only monitors it and +%% records its token (so a later DOWN is fenced with the elector's token). +handle_cast({peer_joined, MemberId, Tokens}, State) -> + {noreply, merge_peer_tokens(Tokens, add_member_monitors([MemberId], State))}; +%% Presence subscription delta from the elector (§4.9); a stale push to a deposed +%% leader is dropped (the next leader is reseeded on assume). +handle_cast({presence_update, Update, Epoch}, State = #state{member_id = Self, leader = Self}) -> + case Epoch >= epoch(State) of + true -> {noreply, apply_presence_update(Update, State)}; + false -> {noreply, State} + end; +handle_cast({presence_update, _Update, _Epoch}, State) -> + {noreply, State}; +handle_cast(_, State) -> + {noreply, State}. + +apply_presence_update({subscribe, SubId, Watch, Notify}, State = #state{subs = Subs}) -> + seed_subscription(SubId, State#state{subs = Subs#{SubId => {Watch, Notify}}}); +apply_presence_update( + {unsubscribe, SubId}, State = #state{subs = Subs, sub_matches = SM, notify_matches = NM} +) -> + State#state{ + subs = maps:remove(SubId, Subs), + sub_matches = maps:remove(SubId, SM), + notify_matches = maps:remove(SubId, NM) + }; +apply_presence_update(unsubscribe_all, State) -> + State#state{subs = #{}, sub_matches = #{}, notify_matches = #{}}. + +%% --------------------------------------------------------------------------- +%% handle_info/2 +%% --------------------------------------------------------------------------- + +%% Too few replica acks for these direct registrations in time (§8): degrade open +%% (ack `yes` leader-only, telemetry) or, with `strict_replication`, fail closed +%% (reject and retract). +handle_info({replicate_timeout, BatchRef}, State = #state{pipeline = P, config = Config}) -> + case dgen_registry_pipeline:ack_timeout(BatchRef, P) of + {expired, Direct, Needed, Got, P1} -> + State1 = State#state{pipeline = P1}, + case dgen_config:strict_replication(Config) of + false -> + notify_degrade_open(Needed, Got, length(Direct)), + deliver_direct(Direct), + {noreply, State1}; + true -> + {noreply, reject_and_retract_direct(Direct, State1)} + end; + none -> + {noreply, State} + end; +handle_info({dgen_transaction, Ref, Reply}, State = #state{pipeline = P}) -> + case dgen_registry_pipeline:finish(Ref, P) of + {ok, Plan, MRef, P1} -> + %% The result precedes the worker's DOWN, so flush drops it. + erlang:demonitor(MRef, [flush]), + State1 = commit_result(Reply, Plan, State#state{pipeline = P1}), + {noreply, maybe_start_commit(State1)}; + stale -> + {noreply, State} + end; + +%% A registered process died (leader only). The DOWN carries Ref so the plan's +%% ref-match guard ignores a stale DOWN for a name already re-bound. If we no longer +%% lead, the new leader (which monitors the pid itself) drives the removal. +handle_info({{down, LogicalName}, Ref, process, _Pid, _Reason}, State) -> + {noreply, enqueue_op({down, LogicalName, Ref}, State)}; +handle_info({'DOWN', Ref, process, _Pid, _Reason}, State) -> + #state{ + monitors = Monitors, + elector = Elector, + member_id = Self + } = State, + case maps:get(Ref, Monitors, undefined) of + undefined -> + %% Not a peer monitor: the commit worker, or a late straggler. + case dgen_registry_pipeline:worker_down(Ref, State#state.pipeline) of + {ok, Plan, P1} -> + State1 = State#state{pipeline = P1}, + {noreply, maybe_start_commit(surrender(failed, maps:get(ops, Plan), State1))}; + stale -> + {noreply, State} + end; + Self -> + {noreply, State}; + DeadMemberId -> + %% Fence {member_down} with the elector's *current* token for the peer, not + %% our cached one, which a dropped snapshot may have left stale (a stale + %% token would strand a dead member and block failover, §5.7). The read runs + %% in a helper, never inline: a call to the elector could deadlock against + %% its own `{elector_assume_and_distribute}` call into this member. + Cached = maps:get(DeadMemberId, State#state.peer_tokens, undefined), + spawn_member_down(Elector, DeadMemberId, Cached), + {noreply, remove_member(DeadMemberId, State)} + end; +handle_info({nodeup, _Node}, State = #state{elector = Elector, member_id = Self}) -> + %% We may have been removed during a partition: re-announce with a fresh token + %% so a queued stale {member_down, Self, OldToken} is discarded. + NewToken = make_ref(), + dgen_server:cast(Elector, {join, Self, NewToken, not State#state.synced}), + %% Partition heal: if our leader is reachable, re-drive intents now (idempotent) + %% so a held or swallowed unregister is never lost (Non-goal 5), without waiting + %% for the re-onboarding snapshot. + State1 = State#state{join_token = NewToken}, + {noreply, + case intent_route(State1) of + none -> State1; + _Reachable -> redrive_intents(all, State1) + end}; +%% Fast-path continuation for a rejoining (non-fresh) member: adjudicate its replica +%% against our authoritative table (§5.6), merge its trail, then onboard it. A killed +%% authority's DOWN drives the normal cleanup. handle_event/4 checked the epoch. +handle_info( + {joiner_gathered, MemberId, AllIds, Tokens, _Epoch, Result}, + State = #state{member_id = Self, leader = Self, members = Members} +) when is_map_key(MemberId, Members) -> + State1 = + case Result of + {ok, {Records, _Version, Released}, Alive} -> + %% The resolved map is discarded: repair only kills and alarms. + OwnRecords = dgen_registry_replica:records(State#state.replica), + {_Records, S1} = repair_conflicts(OwnRecords, [Records], [Released], Alive, State), + S1; + error -> + %% Unanswered: onboard anyway; divergence surfaces at the next gather. + State + end, + {noreply, onboard_joiner(MemberId, AllIds, Tokens, State1)}; +%% The off-loop handoff gather returned (only while `assume_ref` matches). The +%% freshest map is the reconstructed state: broadcasts are totally ordered and every +%% replica is a prefix of the stream. MaxVersion re-baselines the followers. +handle_info( + {assume_gathered, {MemberId, AllIds, Tokens, FreshIds, Epoch, Ref}, SelfRecords, + FreshestRecords, {MaxVersion, Frontier}, PeerResults, Subs, Alive}, + State = #state{member_id = Self, assume_ref = Ref} +) -> + OtherIds = lists:delete(Self, AllIds), + Peers = maps:values(PeerResults), + Degraded = dgen_registry_conflicts:degraded( + MaxVersion, Frontier, OtherIds, FreshIds, PeerResults + ), + %% Repair the conflicts the gather exposes (§5.6): conflicted names are dropped. + {Records, State0} = repair_conflicts( + FreshestRecords, + [SelfRecords | [Recs || {Recs, _V, _Rel} <- Peers]], + [Rel || {_Recs, _V, Rel} <- Peers], + Alive, + State + ), + %% Clearing assume_ref enters `leader`, whose enter call runs assume_leadership + %% against the reconstructed replica and `Subs` (§4.9). + State2 = State0#state{ + replica = dgen_registry_replica:adopt(Epoch, MaxVersion, Records, State0#state.replica), + leader = Self, + conflicts = dgen_registry_conflicts:set_degraded(Degraded, State0#state.conflicts), + assume_ref = undefined, + subs = Subs + }, + State3 = add_member_monitors(extra_member_ids(MemberId, AllIds, Self), State2), + State4 = merge_peer_tokens(Tokens, State3), + %% Encode once; every cast shares the refc binary. + RecordsBin = dgen_registry_replica:encode_records(Records), + lists:foreach( + fun(Id) -> + cast_to_member( + Id, + {apply_names_snapshot, RecordsBin, Self, extra_member_ids(MemberId, AllIds, Id), + Tokens, Epoch, MaxVersion} + ) + end, + OtherIds + ), + %% Re-drive every unresolved intent into our own pipeline. + {noreply, maybe_start_commit(redrive_intents(moved, mark_synced(cancel_resync(State4))))}; +%% Periodic maintenance: prune the release trail and kill budget (§5.6) and stale +%% intents. +handle_info(prune_released, State = #state{conflicts = C, members = Members, config = Config}) -> + arm(?PRUNE_INTERVAL, prune_released), + Now = erlang:system_time(millisecond), + AllConnected = lists:all( + fun({Node, _Name}) -> dgen_utils:node_reachable(Node) end, + maps:keys(Members) + ), + C1 = dgen_registry_conflicts:prune(Now, AllConnected, Config, C), + {noreply, prune_intents(State#state{conflicts = C1})}; +%% Unanswered resync request: clear the guard so the next gap (or heartbeat) retries. +handle_info(resync_timeout, State) -> + {noreply, State#state{resync_timer = undefined}}; +%% Deferred monitor sweep (see assume_leadership/1), one chunk per message. A name +%% already monitored (committed since the sweep began) or re-bound since its chunk was +%% selected is skipped. A pid that died before its chunk fires an immediate `noproc` +%% DOWN, so deferral loses nothing. handle_event/4 checked the epoch. +handle_info( + {monitor_sweep, Epoch, Cont0}, + State = #state{member_id = Self, leader = Self, replica = R} +) -> + Tab = dgen_registry_replica:tab(R), + Select = + case Cont0 of + start -> + ets:select( + Tab, [{{'$1', '$2', '_', '_'}, [], [{{'$1', '$2'}}]}], ?MONITOR_SWEEP_CHUNK + ); + _ -> + ets:select(Cont0) + end, + case Select of + '$end_of_table' -> + {noreply, State}; + {Pairs, Cont1} -> + NTR1 = + lists:foldl( + fun({Name, Pid}, Acc) -> + case + is_map_key(Name, Acc) orelse + dgen_registry_replica:lookup_pid(R, Name) =/= Pid + of + true -> Acc; + false -> Acc#{Name => monitor_name(Name, Pid)} + end + end, + State#state.name_to_ref, + Pairs + ), + self() ! {monitor_sweep, Epoch, Cont1}, + {noreply, State#state{name_to_ref = NTR1}} + end; +%% Retry a failed batch's ops if we still lead (dropping registrations whose caller +%% died), otherwise surrender them as deposed (dgen_registry_pipeline:surrender/2). +handle_info({requeue_ops, Ops}, State = #state{leader = Self, member_id = Self}) -> + Live = [Op || Op <- Ops, caller_alive(Op)], + {noreply, maybe_start_commit(lists:foldl(fun enqueue_op/2, State, Live))}; +handle_info({requeue_ops, Ops}, State) -> + {noreply, surrender(deposed, Ops, State)}; +%% From the connector: the durable election epoch is ahead of ours, so a handoff never +%% reached us. Re-announcing triggers a fresh assume/fan-out. +handle_info({durable_epoch, DurableEpoch}, State) when is_integer(DurableEpoch) -> + case DurableEpoch > epoch(State) of + true -> + #state{elector = Elector, member_id = Self, join_token = Token, synced = Synced} = + State, + dgen_server:cast(Elector, {join, Self, Token, not Synced}); + false -> + ok + end, + {noreply, State}; +handle_info({durable_epoch, _DurableEpoch}, State) -> + {noreply, State}; +handle_info(_Info, State) -> + {noreply, State}. + +terminate(_Reason, _StatemState, _State) -> + net_kernel:monitor_nodes(false), + ok. + +code_change(_OldVsn, StatemState, State, _Extra) -> + {ok, StatemState, State}. + +%% --------------------------------------------------------------------------- +%% Internal helpers +%% --------------------------------------------------------------------------- + +do_leader_changed(NewLeader, OldLeader, Self, State) -> + %% Forwarded intents are re-routed by the caller's redrive_intents/2; the + %% assume/relinquish halves run in the state enter calls. + if + OldLeader =:= Self, NewLeader =/= Self -> + %% Lost leadership: resolve direct registrations awaiting replica acks + %% so their timers do not fire into a follower. + fail_pending_acks(State#state{leader = NewLeader}); + true -> + State#state{leader = NewLeader} + end. + +%% Registered-process monitors are not created inline but by chunked, epoch-fenced +%% {monitor_sweep, …} self-sends that interleave with live traffic. Safe because a +%% monitor on an already-dead (or unreachable) pid fires an immediate DOWN, so a death +%% before its chunk is caught, never lost; dead-pid cleanup has no promptness bound. +assume_leadership(State) -> + self() ! {monitor_sweep, epoch(State), start}, + %% Seed watch sets so the first commit's deltas are computed correctly (§4.9). + recompute_sub_matches(State). + +%% Rebuild every subscription's watch and notify sets from the replica (on assume +%% only; apply_committed_plan maintains them after). A silent reseed: no snapshots. +recompute_sub_matches(State = #state{subs = Subs, replica = R}) -> + SM = maps:map( + fun(_SubId, {Watch, _Notify}) -> dgen_registry_replica:query_match_names(R, Watch) end, + Subs + ), + NM = maps:map( + fun(_SubId, {_Watch, Notify}) -> dgen_registry_replica:query_match_names(R, Notify) end, + Subs + ), + State#state{sub_matches = SM, notify_matches = NM}. + +%% Demonitor all registered pids; keep the replica for snapshot reads. The pipeline's +%% cached commit version is dropped so it cannot pin a later term's read version. +relinquish_leadership(State = #state{name_to_ref = NTR}) -> + maps:foreach( + fun(_LogicalName, Ref) -> + erlang:demonitor(Ref, [flush]) + end, + NTR + ), + State#state{ + name_to_ref = #{}, pipeline = dgen_registry_pipeline:new_term(State#state.pipeline) + }. + +%% Freshest-wins gather across our replica and every reachable peer. Broadcasts are +%% totally ordered and FIFO and every replica is a prefix of the stream, so the highest +%% applied_version holds the freshest binding for every name, and a version tie implies +%% identical content. Peer snapshots are also returned for conflict detection (§5.6) +%% and the trail merge. Per-peer calls run in parallel. +gather_maps(SelfNames, SelfVersion, OtherIds) -> + PeerResults = gather_peer_snapshots(OtherIds), + {FreshestNames, MaxVersion} = maps:fold( + fun(_Id, {Names, Version, _Released}, {_BestNames, BestVersion} = Best) -> + case Version > BestVersion of + true -> {Names, Version}; + false -> Best + end + end, + {SelfNames, SelfVersion}, + PeerResults + ), + {FreshestNames, MaxVersion, PeerResults}. + +%% Fast-path onboarding: ship the joiner our snapshot (with the member list) and tell +%% existing followers to monitor it via {peer_joined}; their replicas are untouched. +onboard_joiner(MemberId, AllIds, Tokens, State = #state{member_id = Self, replica = R}) -> + RecordsBin = dgen_registry_replica:encode(R), + cast_to_member( + MemberId, + {apply_names_snapshot, RecordsBin, Self, AllIds, Tokens, dgen_registry_replica:epoch(R), + dgen_registry_replica:version(R)} + ), + lists:foreach( + fun(Id) -> cast_to_member(Id, {peer_joined, MemberId, Tokens}) end, + [Id || Id <- AllIds, Id =/= Self, Id =/= MemberId] + ), + State. + +%% Fetch a rejoining member's replica and probe conflict pids' liveness off the +%% leader's loop (both are RPCs), reporting {joiner_gathered, …}. Our `protected` table +%% is readable from this helper. +spawn_joiner_gather(MemberId, AllIds, Tokens, Epoch, State) -> + #state{replica = Replica, conflicts = C} = State, + Owner = self(), + _ = spawn(fun() -> + Result = + case member_names(MemberId) of + {ok, {Records, _Version, Released} = Snapshot} -> + Trail = dgen_registry_conflicts:trail( + dgen_registry_conflicts:merge([Released], C) + ), + %% Our table dies with us; then there is no one to report to. + Alive = + try dgen_registry_replica:records(Replica) of + Own -> + probe_alive( + dgen_registry_conflicts:conflict_pids(Own, [Records], Trail) + ) + catch + error:badarg -> #{} + end, + {ok, Snapshot, Alive}; + error -> + error + end, + Owner ! {joiner_gathered, MemberId, AllIds, Tokens, Epoch, Result} + end), + ok. + +%% The full handoff gather (§5.7), off the loop, reporting {assume_gathered, …}. `Ctx` +%% carries the assume payload and the ref matched against `assume_ref`. The durable +%% subscriptions (§4.9) and liveness probes (§5.6) are fetched here too: a synchronous +%% call to the elector from the member loop could deadlock against the elector's own +%% assume call into this member. +spawn_assume_gather(Ctx, SelfRecords, SelfVersion, OtherIds, Elector, Owner, Tenant, Tuid, C) -> + _ = spawn(fun() -> + %% gather_caught_up falls back to a self-only reconstruction if the gather + %% throws, so the continuation always fires. + {FreshestRecords, MaxVersion, PeerResults, Frontier} = + gather_caught_up(SelfRecords, SelfVersion, OtherIds, Tenant, Tuid), + Subs = fetch_subscriptions(Elector), + Peers = maps:values(PeerResults), + Merged = dgen_registry_conflicts:trail( + dgen_registry_conflicts:merge([Rel || {_Recs, _V, Rel} <- Peers], C) + ), + Witnesses = [SelfRecords | [Recs || {Recs, _V, _Rel} <- Peers]], + Alive = probe_alive( + dgen_registry_conflicts:conflict_pids(FreshestRecords, Witnesses, Merged) + ), + Owner ! + {assume_gathered, Ctx, SelfRecords, FreshestRecords, {MaxVersion, Frontier}, + PeerResults, Subs, Alive} + end), + ok. + +%% Re-gather until MaxVersion reaches the durable version key (the SafeAssume fence in +%% the TLA+ spec) or ?GATHER_TIMEOUT. A live member is never ahead of that key, so a +%% lower MaxVersion means the gather raced a committed broadcast still queued in a +%% peer's mailbox; fanning out that stale map would drop an acked registration. Past +%% the deadline we proceed (the continuation flags `degraded`). +gather_caught_up(SelfRecords, SelfVersion, OtherIds, Tenant, Tuid) -> + Deadline = erlang:monotonic_time(millisecond) + ?GATHER_TIMEOUT, + gather_caught_up(SelfRecords, SelfVersion, OtherIds, Tenant, Tuid, Deadline). + +gather_caught_up(SelfRecords, SelfVersion, OtherIds, Tenant, Tuid, Deadline) -> + {FreshestRecords, MaxVersion, PeerResults} = + try + gather_maps(SelfRecords, SelfVersion, OtherIds) + catch + _:_ -> {SelfRecords, SelfVersion, #{}} + end, + %% Best-effort, on the applied_version scale. `unknown` (backend hiccup, test + %% double) disables the fence rather than wedging the handoff. + Frontier = + try + dgen_registry_names:read_committed_frontier(Tenant, Tuid) + catch + _:_ -> unknown + end, + case + Frontier =:= unknown orelse MaxVersion >= Frontier orelse + erlang:monotonic_time(millisecond) >= Deadline + of + true -> + {FreshestRecords, MaxVersion, PeerResults, Frontier}; + false -> + timer:sleep(?ASSUME_CATCHUP_INTERVAL), + gather_caught_up(SelfRecords, SelfVersion, OtherIds, Tenant, Tuid, Deadline) + end. + +%% Best-effort, short-bounded subscription pull from the elector; off-loop callers only. +%% On timeout the empty set is reconciled by a later delta or handoff. +fetch_subscriptions(Elector) when is_pid(Elector) -> + try dgen_server:priority_call(Elector, get_subscriptions, ?SUBS_PULL_TIMEOUT) of + Subs when is_map(Subs) -> Subs; + _ -> #{} + catch + _:_ -> #{} + end; +fetch_subscriptions(_Elector) -> + #{}. + +%% One member_names/1 helper per peer under one deadline; late answers are dropped by +%% the catch-all handle_info clause. +gather_peer_snapshots([]) -> + #{}; +gather_peer_snapshots(OtherIds) -> + Ref = make_ref(), + Parent = self(), + lists:foreach( + fun(MemberId) -> + spawn(fun() -> Parent ! {Ref, MemberId, member_names(MemberId)} end) + end, + OtherIds + ), + Deadline = erlang:monotonic_time(millisecond) + ?GATHER_TIMEOUT + 500, + collect_gather(Ref, OtherIds, Deadline, #{}). + +collect_gather(_Ref, [], _Deadline, Acc) -> + Acc; +collect_gather(Ref, Waiting, Deadline, Acc) -> + Timeout = max(0, Deadline - erlang:monotonic_time(millisecond)), + _ = ?ETA_LOG({gather_wait, length(Waiting), Timeout}), + receive + {Ref, MemberId, {ok, Snapshot}} -> + collect_gather(Ref, lists:delete(MemberId, Waiting), Deadline, Acc#{ + MemberId => Snapshot + }); + {Ref, MemberId, error} -> + collect_gather(Ref, lists:delete(MemberId, Waiting), Deadline, Acc) + after Timeout -> + _ = ?ETA_LOG({gather_timed_out, length(Waiting)}), + Acc + end. + +%% Detect and repair conflicts (§5.6) per dgen_registry_conflicts:resolve/5, carrying +%% out its kills and alarms. Returns the reconstruction with conflicted names dropped. +repair_conflicts(Authority, Witnesses, Trails, Alive, State = #state{conflicts = C0}) -> + C1 = dgen_registry_conflicts:merge(Trails, C0), + Trail = dgen_registry_conflicts:trail(C1), + Conflicts = dgen_registry_conflicts:detect(Authority, Witnesses, Trail, probed(Alive)), + Now = erlang:system_time(millisecond), + {Records, C2, Effects} = + dgen_registry_conflicts:resolve(Conflicts, Authority, State#state.config, Now, C1), + lists:foreach( + fun + ({kill, Pid}) -> exit(Pid, kill); + ({alarm, {Name, APid, Divergent}, Action}) -> + alarm_conflict(Name, APid, Divergent, Action) + end, + Effects + ), + {Records, State#state{conflicts = C2}}. + +%% Parallel liveness probe; runs in the gather helpers, never on the member's loop. +probe_alive([]) -> + #{}; +probe_alive(Pids) -> + Ref = make_ref(), + Parent = self(), + lists:foreach( + fun(Pid) -> spawn(fun() -> Parent ! {Ref, Pid, is_pid_alive(Pid)} end) end, Pids + ), + Deadline = erlang:monotonic_time(millisecond) + ?ALIVE_PROBE_TIMEOUT + 500, + collect_alive(Ref, length(Pids), Deadline, #{}). + +collect_alive(_Ref, 0, _Deadline, Acc) -> + Acc; +collect_alive(Ref, Waiting, Deadline, Acc) -> + Timeout = max(0, Deadline - erlang:monotonic_time(millisecond)), + receive + {Ref, Pid, Alive} -> collect_alive(Ref, Waiting - 1, Deadline, Acc#{Pid => Alive}) + after Timeout -> Acc + end. + +%% An unprobed pid is checked only if local (no RPC); a remote one counts as dead so +%% it is never killed on evidence we did not gather. +probed(Alive) -> + fun(Pid) -> + case Alive of + #{Pid := A} -> A; + _ when node(Pid) =:= node() -> is_process_alive(Pid); + _ -> false + end + end. + +alarm_conflict(Name, Authority, Divergent, Action) -> + logger:error( + "dgen_registry: name conflict — name=~p authority=~p divergent=~p action=~p. " + "Two different live pids held one name (single-fault uniqueness breach, §5.6).", + [Name, Authority, Divergent, Action] + ), + %% `action` budget_exhausted is the one an operator must act on. + emit_telemetry( + [dgen_registry, conflict], + #{divergent_count => length(Divergent)}, + #{name => Name, authority => Authority, action => Action} + ). + +%% An unreachable node counts as not-alive, so a conflict is never escalated on a +%% binding we cannot verify. +is_pid_alive(Pid) when node(Pid) =:= node() -> + is_process_alive(Pid); +is_pid_alive(Pid) -> + Node = node(Pid), + case lists:member(Node, nodes()) of + false -> + false; + true -> + try erpc:call(Node, erlang, is_process_alive, [Pid], ?ALIVE_PROBE_TIMEOUT) of + Alive when is_boolean(Alive) -> Alive; + _ -> false + catch + _:_ -> false + end + end. + +%% Bounded snapshot read of a peer's records, version and trail; `error` if +%% disconnected or slow (its bindings are then unavailable to this gather). +member_names({Node, Name}) -> + case lists:member(Node, nodes()) of + false -> + error; + true -> + %% All inside `try`: decode_records can fault on a corrupt payload. + try + {Payload, Version, Released} = + gen_statem:call({Name, Node}, get_names_snapshot, ?GATHER_TIMEOUT), + Names = dgen_registry_replica:decode_records(Payload), + true = is_map(Names) andalso is_integer(Version) andalso is_map(Released), + {ok, {Names, Version, Released}} + catch + _:_ -> error + end + end. + +%% §5.6 prevention (opt-in `reject_when_degraded`): while degraded, refuse names the +%% leader does not hold; they could be committed bindings the reconstruction missed. +reject_new_when_degraded(Name, #state{conflicts = C, replica = R, config = Config}) -> + dgen_registry_conflicts:degraded(C) andalso dgen_config:reject_when_degraded(Config) andalso + not dgen_registry_replica:exists(R, Name). + +%% With no (reachable) leader a registration is never answered `no` (that would be a +%% false `already_started`); it blocks as an intent until a leader appears. +route_register(Name, Pid, Meta, From, State = #state{leader = Leader, member_id = Self}) -> + if + Leader =:= Self -> + enqueue_register(Name, Pid, Meta, From, State); + true -> + submit_intent({register, Name, Pid, Meta}, From, State) + end. + +%% Mirrors route_register. The bound pid is captured before the optimistic row_delete +%% so the leader can clear it and record it in the trail (§5.6). Leader: commit, answer +%% `ok` on commit. Follower: forward as a tracked intent, or with no reachable leader +%% answer `ok` now and hold the intent until one is (CP: never lost). +route_unregister( + LogicalName, From, State = #state{leader = Leader, member_id = Self, replica = R} +) -> + ReleasedPid = dgen_registry_replica:lookup_pid(R, LogicalName), + + if + Leader =:= Self -> + {noreply, + enqueue_op( + {remove, LogicalName, ReleasedPid, {unreg, From}}, + row_delete(State, LogicalName) + )}; + true -> + State1 = row_delete(State, LogicalName), + {noreply, submit_intent({unregister, LogicalName, ReleasedPid}, From, State1)} + end. + +enqueue_register(Name, Pid, Meta, From, State) -> + case reject_new_when_degraded(Name, State) of + true -> + gen_server:reply(From, no), + State; + false -> + enqueue_op({add, Name, Pid, Meta, {local, From}}, State) + end. + +%% --------------------------------------------------------------------------- +%% Intents — requests this member cannot decide itself (dgen_registry_intents) +%% --------------------------------------------------------------------------- + +intent_route(#state{leader = Self, member_id = Self}) -> + self; +intent_route(#state{leader = undefined}) -> + none; +intent_route(#state{leader = Leader}) -> + case member_reachable(Leader) of + true -> {remote, Leader}; + false -> none + end. + +submit_intent(Kind, From, State = #state{intents = I}) -> + Now = erlang:system_time(millisecond), + apply_intents(dgen_registry_intents:submit(Kind, From, Now, intent_route(State), I), State). + +intent_reply(Ref, Reply, State = #state{intents = I}) -> + apply_intents(dgen_registry_intents:on_reply(Ref, Reply, applied_version(State), I), State). + +%% Our replica advanced: release the gap-deferred `yes` acks it now covers. +intents_advanced(State = #state{intents = I}) -> + apply_intents(dgen_registry_intents:on_version(applied_version(State), I), State). + +%% Re-drive unresolved intents: `Scope` is `moved` for a new leadership view, `all` for +%% a restored link (see dgen_registry_intents:redrive/5). +redrive_intents(Scope, State = #state{intents = I, config = Config}) -> + Now = erlang:system_time(millisecond), + TTL = dgen_config:register_timeout(Config), + Route = intent_route(State), + apply_intents(dgen_registry_intents:redrive(Route, Scope, Now, TTL, I), State). + +%% Drop intents whose callers timed out, bounding growth during a long partition. +prune_intents(State = #state{intents = I, config = Config}) -> + Now = erlang:system_time(millisecond), + TTL = dgen_config:register_timeout(Config), + State#state{intents = dgen_registry_intents:prune(Now, TTL, I)}. + +apply_intents({Intents, Effects}, State) -> + lists:foldl(fun intent_effect/2, State#state{intents = Intents}, Effects). + +intent_effect({reply, From, Value}, State) -> + gen_server:reply(From, Value), + State; +intent_effect({forward, Leader, Ref, Kind}, State = #state{member_id = Self}) -> + cast_to_member(Leader, forward_msg(Ref, Self, Kind)), + State; +intent_effect({retract, Leader, Name, Pid}, State) -> + forward_retract(Leader, Name, Pid), + State; +intent_effect({enqueue, {register, Name, Pid, Meta}, From}, State) -> + enqueue_register(Name, Pid, Meta, From, State); +intent_effect({enqueue, {unregister, Name, Pid}, answered}, State) -> + enqueue_op({retract, Name, Pid}, State); +intent_effect({resync, Leader}, State) -> + request_resync(Leader, State); +intent_effect({insert_row, Name, Pid, {Index, Data}}, State) -> + row_insert(State, Name, Pid, Index, Data). + +forward_msg(Ref, Self, {register, Name, Pid, Meta}) -> + {register_req, Ref, Self, Name, Pid, Meta}; +forward_msg(Ref, Self, {unregister, Name, _ReleasedPid}) -> + {unregister_req, Ref, Self, Name}; +forward_msg(Ref, Self, {set_meta, Name, Index, Data}) -> + {set_meta_req, Ref, Self, Name, Index, Data}. + +%% Ops arriving during an in-flight commit accumulate and ride the next batch. +enqueue_op(Op, State = #state{pipeline = P}) -> + maybe_start_commit(State#state{pipeline = dgen_registry_pipeline:enqueue(Op, P)}). + +%% At most one commit runs at a time, in a dgen_transaction worker off this loop. A +%% member that no longer leads surrenders the queue instead. +maybe_start_commit(State = #state{pipeline = P, leader = Leader, member_id = Self}) -> + case {dgen_registry_pipeline:in_flight(P), dgen_registry_pipeline:num_pending(P)} of + {true, _} -> + State; + {false, 0} -> + State; + {false, _} when Leader =/= Self -> + {Ops, P1} = dgen_registry_pipeline:take_pending(P), + surrender(deposed, Ops, State#state{pipeline = P1}); + {false, _} -> + start_commit(State) + end. + +start_commit(State) -> + Epoch = epoch(State), + Tab = dgen_registry_replica:tab(State#state.replica), + #state{ + member_id = Self, + tenant = Tenant, + tuid = Tuid, + name_to_ref = NTR, + pipeline = P, + config = Config + } = State, + {ThisBatch, P1} = dgen_registry_pipeline:take_batch(dgen_config:commit_batch_size(Config), P), + LastVersion = dgen_registry_pipeline:read_version(P1), + State1 = State#state{pipeline = P1}, + %% Planning is O(batch): bindings resolve lazily (see plan_batch/4). + Plan = plan_batch(ThisBatch, Tab, NTR, Epoch), + #{dbop := DBOp, replies := RepliesRev} = Plan, + case map_size(DBOp) =:= 0 of + true -> + %% Only rejections: no commit needed. + send_replies(lists:reverse(RepliesRev)), + maybe_start_commit(State1); + false -> + Ref = make_ref(), + %% Pin last_version as the read version to skip a GRV; if stale, the + %% leader-key read raises too-old and dgen_transaction retries with a fresh + %% one. The worker is monitored so its death still resolves the commit. Its + %% only write is the fenced version bump; the name delta applies in memory. + {ok, {_Pid, MRef}} = dgen_registry_names:start_commit( + Tenant, + Tuid, + Self, + #{owner => self(), ref => Ref, epoch => Epoch, read_version => LastVersion} + ), + State1#state{pipeline = dgen_registry_pipeline:start(Ref, MRef, Plan, P1)} + end. + +%% Apply a committed batch: update the replica, (de)monitor pids, replicate, and +%% answer callers. +apply_committed_plan(Plan, Version, State) -> + #{ + dbop := DBOp, + names := WNames, + meta := WMeta, + replies := RepliesRev, + bcasts := BcastsRev, + released := Released, + direct_meta := DirectMeta + } = Plan, + #state{ + member_id = Self, + name_to_ref = NTR0, + members = Members, + replica = R0 + } = State, + PrevVersion = dgen_registry_replica:version(R0), + %% Only DBOp names are touched, so a concurrent optimistic unregister of an + %% untouched name is never clobbered. + State0 = State#state{replica = dgen_registry_replica:commit(Version, DBOp, WNames, WMeta, R0)}, + NTR1 = apply_monitor_ops(DBOp, WNames, NTR0), + %% Broadcast first, so the replicate_sync (FIFO behind it) reaches a follower only + %% after it holds the batch. One message per batch, never per name: a partially + %% received batch would report the full version with partial content, breaking + %% "same version implies same content" that the handoff gather relies on (§5.7). + broadcast_batch(Members, lists:reverse(BcastsRev), PrevVersion, Version, Self), + Now = erlang:system_time(millisecond), + State1 = State0#state{ + name_to_ref = NTR1, + pipeline = dgen_registry_pipeline:committed(Version, State0#state.pipeline), + %% Explicit releases extend the conflict trail (§5.6). + conflicts = dgen_registry_conflicts:released(Released, Now, State0#state.conflicts) + }, + %% Forwarded registrations and rejections are answered now (a forwarded `yes` is + %% already two-holder); a direct `yes` waits for a follower's replica ack (§8). + {DirectYes, Immediate} = lists:partition(fun is_direct_yes/1, lists:reverse(RepliesRev)), + lists:foreach( + fun({Origin, Result}) -> deliver_committed_reply(Origin, Result, Version) end, Immediate + ), + %% {Name, Pid} lets strict_replication roll a direct binding back. + EnrichedDirect = [ + {Origin, Name, Pid} + || {Origin, yes} <- DirectYes, {Name, Pid} <- [maps:get(Origin, DirectMeta)] + ], + %% Presence notifications (§4.9), computed against the committed replica. + State2 = fire_presence(DBOp, State1), + confirm_direct(EnrichedDirect, Version, State2). + +%% Presence for a committed batch (leader only, §4.9), diffing the changed names per +%% subscription: watch-set changes yield `joined`/`left` for continuing notify targets; +%% a pid newly entering the notify set gets the full watch snapshot. +fire_presence(_DBOp, State = #state{subs = Subs}) when map_size(Subs) =:= 0 -> + State; +fire_presence(DBOp, State = #state{replica = R, subs = Subs}) -> + Tab = dgen_registry_replica:tab(R), + RealNames = maps:keys(DBOp), + maps:fold( + fun(SubId, {Watch, Notify}, S) -> + diff_subscription(SubId, Watch, Notify, RealNames, Tab, S) + end, + State, + Subs + ). + +%% Seed a (re)subscription: compute both sets and send the watch snapshot to the +%% current notify targets. +seed_subscription( + SubId, + State = #state{ + subs = Subs, replica = R, sub_matches = SM, notify_matches = NM + } +) -> + case Subs of + #{SubId := {Watch, Notify}} -> + WatchMatches = dgen_registry_replica:query_match_names(R, Watch), + NotifyMatches = dgen_registry_replica:query_match_names(R, Notify), + send_presence(SubId, snapshot_events(WatchMatches), pidset(NotifyMatches)), + State#state{ + sub_matches = SM#{SubId => WatchMatches}, + notify_matches = NM#{SubId => NotifyMatches} + }; + _ -> + State + end. + +%% Diff one subscription and fire its notifications. Done on pid-sets, so a pid already +%% a target under another name gets no duplicate snapshot. +diff_subscription(SubId, Watch, Notify, RealNames, Tab, State) -> + #state{sub_matches = SM, notify_matches = NM} = State, + {WatchEvents, WatchAfter} = diff_query_matches(Watch, RealNames, maps:get(SubId, SM, #{}), Tab), + NotifyBefore = maps:get(SubId, NM, #{}), + {_NotifyEvents, NotifyAfter} = diff_query_matches(Notify, RealNames, NotifyBefore, Tab), + AfterPids = pidset(NotifyAfter), + NewTargets = AfterPids -- pidset(NotifyBefore), + ContinuingTargets = AfterPids -- NewTargets, + send_presence(SubId, WatchEvents, ContinuingTargets), + send_presence(SubId, snapshot_events(WatchAfter), NewTargets), + State#state{sub_matches = SM#{SubId => WatchAfter}, notify_matches = NM#{SubId => NotifyAfter}}. + +%% Diff a query's membership over the changed names: `joined`/`left` events and the +%% updated `#{Name => Pid}` map. +diff_query_matches(Query, RealNames, Before, Tab) -> + lists:foldl( + fun(Name, {EvAcc, MatchAcc}) -> + WasPid = maps:get(Name, Before, undefined), + NowMatch = + case ets:lookup(Tab, Name) of + [{_N, RowPid, Index, _Data} | _] -> + case dgen_registry_replica:query_satisfies(Query, Index) of + true -> {true, RowPid}; + false -> false + end; + [] -> + false + end, + case {WasPid, NowMatch} of + {undefined, {true, Pid}} -> + {[{joined, Name, Pid} | EvAcc], MatchAcc#{Name => Pid}}; + {undefined, false} -> + {EvAcc, MatchAcc}; + {WasPid, {true, WasPid}} -> + {EvAcc, MatchAcc}; + {OldPid, {true, NewPid}} -> + {[{joined, Name, NewPid}, {left, Name, OldPid} | EvAcc], MatchAcc#{ + Name => NewPid + }}; + {OldPid, false} -> + {[{left, Name, OldPid} | EvAcc], maps:remove(Name, MatchAcc)} + end + end, + {[], Before}, + RealNames + ). + +snapshot_events(Matches) -> + [{joined, Name, Pid} || {Name, Pid} <- maps:to_list(Matches)]. + +pidset(Matches) -> + lists:usort(maps:values(Matches)). + +send_presence(_SubId, [], _Pids) -> + ok; +send_presence(_SubId, _Events, []) -> + ok; +send_presence(SubId, Events, Pids) -> + Msg = {dgen_presence, SubId, Events}, + lists:foreach(fun(Pid) -> Pid ! Msg end, Pids). + +is_direct_yes({{local, _From}, yes}) -> true; +is_direct_yes(_Other) -> false. + +%% Replicate-before-ack for direct registrations (§5.5, §8): wait for +%% `register_replicas` distinct follower acks (capped by follower count; none needed if +%% there are no followers), or the `strict_replication` policy on timeout. +confirm_direct([], _Version, State) -> + State; +confirm_direct( + Direct, + Version, + State = #state{members = Members, member_id = Self, pipeline = P, config = Config} +) -> + Needed = min(dgen_config:register_replicas(Config), map_size(Members)), + case Needed =< 0 of + true -> + deliver_direct(Direct), + State; + false -> + BatchRef = make_ref(), + %% Carries Version so a gapped follower that lacks the batch never acks. + broadcast_to_peers(Members, {replicate_sync, BatchRef, Self, Version}), + TimerRef = arm( + dgen_config:replicate_timeout(Config), {replicate_timeout, BatchRef} + ), + State#state{ + pipeline = dgen_registry_pipeline:await_acks(BatchRef, Direct, TimerRef, Needed, P) + } + end. + +deliver_direct(Direct) -> + lists:foreach(fun({Origin, _Name, _Pid}) -> deliver_reply(Origin, yes) end, Direct). + +%% strict_replication fail-closed: reject and retract (a pid-guarded, committed clear). +reject_and_retract_direct(Direct, State) -> + lists:foldl( + fun({Origin, Name, Pid}, S) -> + deliver_reply(Origin, no), + enqueue_op({retract, Name, Pid}, S) + end, + State, + Direct + ). + +%% Degrade-open event (§8): telemetry and a log warning. +notify_degrade_open(Needed, Got, Count) -> + logger:warning( + "dgen_registry: degrade-open — acked ~b direct registration(s) leader-only " + "(needed ~b replica ack(s), got ~b) on replicate timeout (§8).", + [Count, Needed, Got] + ), + emit_telemetry( + [dgen_registry, register, degrade_open], + #{count => Count}, + #{needed => Needed, got => Got} + ). + +%% `telemetry` is a soft dependency. Availability is resolved once per VM and cached: +%% `code:ensure_loaded/1` on a missing module calls `code_server` every time, which +%% also breaks reproducibility under eta simulation. +-define(TELEMETRY_KEY, {?MODULE, telemetry_available}). + +emit_telemetry(Event, Measurements, Metadata) -> + case telemetry_available() of + true -> apply(telemetry, execute, [Event, Measurements, Metadata]); + false -> ok + end. + +telemetry_available() -> + case persistent_term:get(?TELEMETRY_KEY, undefined) of + undefined -> + Available = code:ensure_loaded(telemetry) =:= {module, telemetry}, + persistent_term:put(?TELEMETRY_KEY, Available), + Available; + Available -> + Available + end. + +%% Answer a write op's origin: gen_server:reply/2 locally, or a reply cast to the +%% forwarding member. +deliver_reply({local, From}, Result) -> + gen_server:reply(From, Result); +deliver_reply({meta, From}, Result) -> + gen_server:reply(From, Result); +deliver_reply({unreg, From}, Result) -> + gen_server:reply(From, Result); +deliver_reply({forward, MemberId, Ref}, Result) -> + cast_to_member(MemberId, {register_reply, Ref, Result}); +deliver_reply({forward_meta, MemberId, Ref}, Result) -> + cast_to_member(MemberId, {set_meta_reply, Ref, Result}); +deliver_reply({forward_unreg, MemberId, Ref}, _Result) -> + cast_to_member(MemberId, {unregister_reply, Ref}). + +%% A forwarded registration's reply carries the commit version so the follower can +%% version-guard its `yes` (see the {register_reply, _, _, _} handler). +deliver_committed_reply({forward, MemberId, Ref}, Result, Version) -> + cast_to_member(MemberId, {register_reply, Ref, Result, Version}); +deliver_committed_reply(Origin, Result, _Version) -> + deliver_reply(Origin, Result). + +%% Fold the ordered ops into a plan: the working overlay, the net durable delta (`dbop`, +%% last write wins), and replies and broadcasts (newest first). `wntr` tracks monitor +%% refs for the `down` ref-match guard; a name bound in this batch carries a +%% `{pending, Pid}` placeholder that never matches a real ref. `names` starts empty: +%% each op resolves its one name via seed_lookup/3, so planning is O(batch). +plan_batch(Ops, Tab, NTR, Epoch) -> + Init = #{ + tab => Tab, + names => #{}, + meta => #{}, + wntr => NTR, + dbop => #{}, + replies => [], + bcasts => [], + released => [], + direct_meta => #{}, + %% A plan whose epoch is no longer current is not applied; broadcasts carry it. + epoch => Epoch, + %% Kept so a failed batch can be surrendered. + ops => Ops + }, + lists:foldl(fun(Op, Acc) -> plan_op(Op, Epoch, Acc) end, Init, Ops). + +%% A name's pid partway through this batch: the batch overlay `WN` first, else ETS. +%% `removed` marks a name cleared earlier in this batch: only unregister deletes from +%% ETS eagerly, so a prior `retract` or `down` is visible only through the overlay. +seed_lookup(WN, Tab, Name) -> + case WN of + #{Name := removed} -> undefined; + #{Name := Pid} -> Pid; + _ -> lookup_name(Tab, Name) + end. + +plan_op({add, Name, Pid, {Index, Data} = Meta, Origin}, Epoch, Acc) -> + #{ + tab := Tab, + names := WN, + meta := WMeta, + wntr := WNTR, + dbop := DB, + replies := Rs, + bcasts := Bs, + direct_meta := DM + } = Acc, + case seed_lookup(WN, Tab, Name) of + Taken when is_pid(Taken), Taken =/= Pid -> + %% Held by a different pid: reject (the singleton contract). + Acc#{replies := [{Origin, no} | Rs]}; + _Free_or_same -> + %% Free, or the same pid: an idempotent `yes` (§5.2), so a caller whose + %% call timed out can re-drive and get a decisive answer. Direct-origin + %% bindings are recorded for strict_replication rollback. + DM1 = + case Origin of + {local, _From} -> DM#{Origin => {Name, Pid}}; + _ -> DM + end, + Acc#{ + names := WN#{Name => Pid}, + meta := WMeta#{Name => Meta}, + wntr := WNTR#{Name => {pending, Pid}}, + dbop := DB#{Name => {set, node(Pid)}}, + replies := [{Origin, yes} | Rs], + bcasts := [{name_registered, Name, Pid, Index, Data, Epoch} | Bs], + direct_meta := DM1 + } + end; +plan_op({set_meta, Name, Index, Data, Origin}, Epoch, Acc) -> + #{tab := Tab, names := WN, meta := WMeta, dbop := DB, replies := Rs, bcasts := Bs} = Acc, + case seed_lookup(WN, Tab, Name) of + undefined -> + Acc#{replies := [{Origin, {error, not_registered}} | Rs]}; + Pid -> + %% Keep an in-batch {set,…} so the new pid is still monitored; otherwise + %% {meta,…} leaves the existing monitor alone. + DBVal = + case maps:get(Name, DB, undefined) of + {set, _} = SetOp -> SetOp; + _ -> {meta, node(Pid)} + end, + Acc#{ + meta := WMeta#{Name => {Index, Data}}, + dbop := DB#{Name => DBVal}, + replies := [{Origin, ok} | Rs], + bcasts := [{metadata_set, Name, Index, Data, Epoch} | Bs] + } + end; +plan_op({remove, Name, ReleasedPid, Origin}, Epoch, Acc) -> + #{ + tab := Tab, + names := WN, + wntr := WNTR, + dbop := DB, + replies := Rs, + bcasts := Bs, + released := Rel + } = Acc, + %% Clear if the name was bound at enqueue (ReleasedPid; the optimistic delete + %% already removed it from ETS) or is bound now in this batch (seed_lookup). The + %% released pair joins the conflict trail (§5.6). The origin gets `ok` either way. + Rs1 = unreg_reply(Origin, Rs), + WasBound = is_pid(ReleasedPid), + Current = seed_lookup(WN, Tab, Name), + %% Guard against the *current* holder: the optimistic row_delete frees the name at + %% arrival, so a register queued ahead of this remove may already have rebound it. + %% Clearing that new holder would break Guarantee 1. A captured pid clears only if + %% the current holder is that pid or nobody; a capture of `undefined` clears + %% whatever is bound (unregister by name). + Rebound = WasBound andalso is_pid(Current) andalso Current =/= ReleasedPid, + case (WasBound orelse Current =/= undefined) andalso not Rebound of + true -> + Rel1 = + case WasBound of + true -> [{Name, ReleasedPid} | Rel]; + false -> Rel + end, + %% The released pid rides the broadcast into every member's trail (§5.6). + ReleasedOut = + case WasBound of + true -> ReleasedPid; + false -> undefined + end, + Acc#{ + names := WN#{Name => removed}, + wntr := maps:remove(Name, WNTR), + dbop := DB#{Name => clear}, + replies := Rs1, + bcasts := [{name_unregistered, Name, ReleasedOut, Epoch} | Bs], + released := Rel1 + }; + false -> + Acc#{replies := Rs1} + end; +plan_op({retract, Name, Pid}, Epoch, Acc) -> + #{tab := Tab, names := WN, wntr := WNTR, dbop := DB, bcasts := Bs, released := Rel} = Acc, + %% Pid-guarded (strict_replication fail-closed): clear only if still this pid's, + %% and trail it so a lagging follower's copy is not mistaken for a conflict. + case seed_lookup(WN, Tab, Name) of + Pid -> + Acc#{ + names := WN#{Name => removed}, + wntr := maps:remove(Name, WNTR), + dbop := DB#{Name => clear}, + bcasts := [{name_unregistered, Name, Pid, Epoch} | Bs], + released := [{Name, Pid} | Rel] + }; + _ -> + Acc + end; +plan_op({down, Name, Ref}, Epoch, Acc) -> + #{names := WN, wntr := WNTR, dbop := DB, bcasts := Bs} = Acc, + case maps:get(Name, WNTR, undefined) of + Ref -> + %% The pid is dead, so nothing is trailed (the trail is for live releases). + Acc#{ + names := WN#{Name => removed}, + wntr := maps:remove(Name, WNTR), + dbop := DB#{Name => clear}, + bcasts := [{name_unregistered, Name, undefined, Epoch} | Bs] + }; + _ -> + %% Stale DOWN (re-registered or already removed). + Acc + end. + +%% A remove with origin `undefined` (legacy cast or retract) has no caller. +unreg_reply(undefined, Rs) -> Rs; +unreg_reply(Origin, Rs) -> [{Origin, ok} | Rs]. + +%% Monitor side effects of the delta: clear demonitors, set re-monitors the new pid, +%% meta leaves the monitor alone. +apply_monitor_ops(DBOp, WNames, NTR) -> + maps:fold( + fun + (_Name, {meta, _PidNode}, Acc) -> + Acc; + (Name, clear, NTRacc) -> + demonitor_name(Name, NTRacc); + (Name, {set, _PidNode}, NTRacc) -> + NTR1 = demonitor_name(Name, NTRacc), + Pid = maps:get(Name, WNames), + NTR1#{Name => monitor_name(Name, Pid)} + end, + NTR, + DBOp + ). + +send_replies(Replies) -> + lists:foreach(fun({Origin, Reply}) -> deliver_reply(Origin, Reply) end, Replies). + +%% Extend the conflict trail (§5.6) from an applied batch. Keyed by {Name, Pid}, so a +%% release under one name never masks a conflict on another name the same pid holds. +record_released(Ops, State = #state{conflicts = C}) -> + Pairs = [{Name, Pid} || {name_unregistered, Name, Pid} <- Ops, is_pid(Pid)], + Now = erlang:system_time(millisecond), + State#state{conflicts = dgen_registry_conflicts:released(Pairs, Now, C)}. + +%% One `{names_batch, Ops, Epoch, PrevV, Version, LeaderId}` per committed batch; the +%% epoch is carried once on the envelope. Followers use PrevV/Version for gap detection +%% and LeaderId as the resync target. An empty batch sends nothing. +-ifdef(MUTATION_PARTIAL_BATCH). +%% MUTATION (test-only): reintroduces one message per op sharing {PrevV, Version}, so a +%% partially received batch diverges silently. Paired with dgen_registry_replica. +broadcast_batch(_Members, [], _PrevV, _Version, _Self) -> + ok; +broadcast_batch(Members, Ops, PrevV, Version, Self) -> + lists:foreach( + fun(Op) -> + broadcast_to_peers( + Members, + {names_batch, [strip_bcast_epoch(Op)], bcast_epoch(Op), PrevV, Version, Self} + ) + end, + Ops + ). +-else. +broadcast_batch(_Members, [], _PrevV, _Version, _Self) -> + ok; +broadcast_batch(Members, [First | _] = Ops, PrevV, Version, Self) -> + Epoch = bcast_epoch(First), + Stripped = [strip_bcast_epoch(Op) || Op <- Ops], + broadcast_to_peers(Members, {names_batch, Stripped, Epoch, PrevV, Version, Self}). +-endif. + +-ifdef(MUTATION_QUIET_RESYNC). +%% MUTATION (test-only, `DGEN_MUTATION=quiet_resync`): the heartbeat advertises +%% nothing, so a follower that loses the stream's tail stays diverged while quiet. +broadcast_heartbeat(_State) -> + ok. +-else. +%% The leader's periodic empty batch at {Applied, Applied}; see +%% ?REPLICA_HEARTBEAT_INTERVAL. +broadcast_heartbeat(State = #state{members = Members, member_id = Self}) -> + Applied = applied_version(State), + broadcast_to_peers(Members, {names_batch, [], epoch(State), Applied, Applied, Self}), + ok. +-endif. + +bcast_epoch({name_registered, _Name, _Pid, _Index, _Data, Epoch}) -> Epoch; +bcast_epoch({metadata_set, _Name, _Index, _Data, Epoch}) -> Epoch; +bcast_epoch({name_unregistered, _Name, _ReleasedPid, Epoch}) -> Epoch. + +strip_bcast_epoch({name_registered, Name, Pid, Index, Data, _Epoch}) -> + {name_registered, Name, Pid, Index, Data}; +strip_bcast_epoch({metadata_set, Name, Index, Data, _Epoch}) -> + {metadata_set, Name, Index, Data}; +strip_bcast_epoch({name_unregistered, Name, ReleasedPid, _Epoch}) -> + {name_unregistered, Name, ReleasedPid}. + +%% Ask `LeaderId` for a snapshot after a gap, at most once per ?RESYNC_RETRY. +request_resync(LeaderId, State = #state{resync_timer = undefined, member_id = Self}) -> + cast_to_member(LeaderId, {resync_req, Self}), + Ref = arm(?RESYNC_RETRY, resync_timeout), + State#state{resync_timer = Ref}; +request_resync(_LeaderId, State) -> + State. + +cancel_resync(State = #state{resync_timer = undefined}) -> + State; +cancel_resync(State = #state{resync_timer = Ref}) -> + erlang:cancel_timer(Ref), + State#state{resync_timer = undefined}. + +%% Mark synced, re-announcing once with `fresh = false`. +mark_synced(State = #state{synced = true}) -> + State; +mark_synced(State = #state{elector = Elector, member_id = Self, join_token = Token}) -> + dgen_server:cast(Elector, {join, Self, Token, false}), + State#state{synced = true}. + +%% Connected, so a cast will not be dropped? +member_reachable({Node, _Name}) -> + dgen_utils:node_reachable(Node). + +%% Report a peer's death with the elector's current token (fallback `Cached`), from a +%% helper so this loop never blocks on the elector (see the DOWN handler). +spawn_member_down(Elector, MemberId, Cached) -> + _ = spawn(fun() -> + Token = dgen_registry_elector:member_token(Elector, MemberId, Cached), + dgen_server:cast(Elector, {member_down, MemberId, Token}) + end), + ok. + +%% Guarantee 5: answer a consistent read only after verifying against the durable +%% leader key that we are still the leader for our epoch; a deposed leader on the +%% minority side of a partition would otherwise serve its frozen replica. `Value` is +%% captured at the mailbox point; the verify runs in a helper so the DB round-trip +%% never blocks the loop. Linearizable because leadership is continuous within an +%% epoch. Any failure answers `Denied` (the CP refusal). +reply_fenced(From, Value, Denied, State = #state{tenant = Tenant, tuid = Tuid, member_id = Self}) -> + Epoch = epoch(State), + + _ = spawn(fun() -> + Reply = + case dgen_registry_names:verify_leader(Tenant, Tuid, Self, Epoch) of + true -> Value; + false -> Denied + end, + gen_server:reply(From, Reply) + end), + ok. + +%% Pid-guarded retract to a reachable leader (an unregister intent's re-drive). +forward_retract(Leader, Name, Pid) -> + cast_to_member(Leader, {retract_req, Name, Pid}). + +%% Applied only if committed under the leadership view it was planned in. +commit_result(Reply, Plan, State = #state{leader = Leader, member_id = Self}) -> + case Reply of + {committed, Version} when Leader =:= Self -> + case maps:get(epoch, Plan) =:= epoch(State) of + true -> + apply_committed_plan(Plan, Version, State); + false -> + %% Older epoch: applying or broadcasting an old-term plan would + %% diverge replicas. Only the version bump was durable, so retry. + surrender(failed, maps:get(ops, Plan), State) + end; + {aborted, fenced} -> + %% A newer leader exists and we have not heard: ask now. + surrender(failed, maps:get(ops, Plan), rejoin_once_fenced(State)); + _Deposed_or_error -> + surrender(failed, maps:get(ops, Plan), State) + end. + +%% A fenced commit proves we were deposed unheard: re-announce now rather than wait +%% for the connector's durable-epoch check. Once per epoch, since each join is a +%% durable queue entry and a partitioned member keeps being fenced. +rejoin_once_fenced(State = #state{fence_rejoined = Epoch}) when Epoch =/= undefined -> + case epoch(State) of + Epoch -> State; + _ -> rejoin_once_fenced(State#state{fence_rejoined = undefined}) + end; +rejoin_once_fenced(State) -> + #state{elector = Elector, member_id = Self, join_token = Token, synced = Synced} = State, + dgen_server:cast(Elector, {join, Self, Token, not Synced}), + State#state{fence_rejoined = epoch(State)}. + +%% Carry out dgen_registry_pipeline:surrender/2: replies, intents, delayed retries. +surrender(Reason, Ops, State) -> + lists:foldl( + fun + ({reply, Origin, Value}, S) -> + deliver_reply(Origin, Value), + S; + ({intent, Kind, From}, S) -> + submit_intent(Kind, From, S); + ({retry, Retry}, S) -> + arm(?REQUEUE_DELAY, {requeue_ops, Retry}), + S + end, + State, + dgen_registry_pipeline:surrender(Reason, Ops) + ). + +%% Only a local direct registration's caller can be checked; anything else is kept. +caller_alive({add, _Name, _Pid, _Meta, {local, {Caller, _Tag}}}) when node(Caller) =:= node() -> + is_process_alive(Caller); +caller_alive(_Op) -> + true. + +%% Logs the arm so a simulated `{clock, Ms}` trace entry is attributable; `?ETA_LOG` +%% is a no-op in production builds. +arm(Delay, Msg) -> + _ = ?ETA_LOG({arm, Msg, Delay}), + erlang:send_after(Delay, self(), Msg). + +%% Losing leadership with direct registrations awaiting acks: resolve them by the +%% replicate-timeout policy (§8). Fail-closed hands the pid-guarded retract to the +%% intents so the caller's `no` cannot leave the binding alive. +fail_pending_acks(State = #state{pipeline = P, config = Config}) -> + {Batches, P1} = dgen_registry_pipeline:take_acks(P), + Strict = dgen_config:strict_replication(Config), + lists:foldl( + fun({Direct, TimerRef, Needed, Got}, S) -> + erlang:cancel_timer(TimerRef), + case Strict of + false -> + notify_degrade_open(Needed, Got, length(Direct)), + deliver_direct(Direct), + S; + true -> + lists:foldl( + fun({Origin, Name, Pid}, S1) -> + deliver_reply(Origin, no), + submit_intent({unregister, Name, Pid}, answered, S1) + end, + S, + Direct + ) + end + end, + State#state{pipeline = P1}, + Batches + ). + +%% --------------------------------------------------------------------------- +%% Row writes outside the stream (see dgen_registry_replica) +%% --------------------------------------------------------------------------- + +%% The optimistic delete on unregister, and the legacy unversioned reply's insert +%% (rolling upgrade). Neither moves the replica's version. +row_delete(S = #state{replica = R}, Name) -> + S#state{replica = dgen_registry_replica:delete(R, Name)}. + +row_insert(S = #state{replica = R}, Name, Pid, Index, Data) -> + S#state{replica = dgen_registry_replica:insert(R, Name, Pid, Index, Data)}. + +%% Bare table, so plan_batch/4 stays testable against a plain ETS table. +lookup_name(Tab, Name) -> + case ets:lookup(Tab, Name) of + [{_Name, Pid, _Index, _Data} | _] -> Pid; + [] -> undefined + end. + +add_member_monitors(MemberIds, State) -> + #state{member_id = Self, members = Members, monitors = Monitors} = State, + {NewMembers, NewMonitors} = lists:foldl( + fun(MemberId, {MA, MonA}) -> + case MemberId =:= Self orelse maps:is_key(MemberId, MA) of + true -> + {MA, MonA}; + false -> + {Node, Name} = MemberId, + Ref = erlang:monitor(process, {Name, Node}), + {MA#{MemberId => Ref}, MonA#{Ref => MemberId}} + end + end, + {Members, Monitors}, + MemberIds + ), + State#state{members = NewMembers, monitors = NewMonitors}. + +remove_member(MemberId, State = #state{members = Members, monitors = Monitors}) -> + case maps:get(MemberId, Members, undefined) of + undefined -> + State; + Ref -> + erlang:demonitor(Ref, [flush]), + %% Drop the peer's token (the caller already read it). The departure time + %% keeps trail pruning suspended for a TTL window (§5.6). + State#state{ + members = maps:remove(MemberId, Members), + monitors = maps:remove(Ref, Monitors), + peer_tokens = maps:remove(MemberId, State#state.peer_tokens), + conflicts = dgen_registry_conflicts:departed( + erlang:system_time(millisecond), State#state.conflicts + ) + } + end. + +demonitor_name(LogicalName, NTR) -> + case maps:get(LogicalName, NTR, undefined) of + undefined -> + NTR; + OldRef -> + erlang:demonitor(OldRef, [flush]), + maps:remove(LogicalName, NTR) + end. + +%% The one place a registered-process monitor is created; tagged `{down, Name}`. +monitor_name(Name, Pid) -> + erlang:monitor(process, Pid, [{tag, {down, Name}}]). + +%% Extra member monitors on a leadership transition: none for member_down; all peers +%% for the joiner itself; just the joiner for everyone else. +extra_member_ids(undefined, _AllIds, _Id) -> []; +extra_member_ids(MemberId, AllIds, MemberId) -> AllIds; +extra_member_ids(MemberId, _AllIds, _Id) -> [MemberId]. + +%% The elector is authoritative and always sends current tokens, so overwrite. +merge_peer_tokens(Tokens, State = #state{peer_tokens = PeerTokens}) -> + State#state{peer_tokens = maps:merge(PeerTokens, Tokens)}. + +broadcast_to_peers(Members, Msg) -> + maps:foreach(fun(MemberId, _) -> cast_to_member(MemberId, Msg) end, Members). + +%% Every inter-member message goes through here. A cast to a disconnected node would +%% auto-reconnect distribution; during a partition that heals it before either side +%% observes the departure. So drop the cast unless the node is already connected. +%% Under `-ifdef(DST)` `eta_transform` rewrites the cast to `eta_net:cast/2`. +cast_to_member(To, Msg) -> + do_cast_to_member(To, Msg). + +do_cast_to_member({Node, Name}, Msg) when Node =:= node() -> + gen_statem:cast({Name, Node}, Msg); +do_cast_to_member({Node, Name}, Msg) -> + case lists:member(Node, nodes()) of + true -> gen_statem:cast({Name, Node}, Msg); + false -> ok + end. diff --git a/src/dgen_registry_names.erl b/src/dgen_registry/dgen_registry_names.erl similarity index 56% rename from src/dgen_registry_names.erl rename to src/dgen_registry/dgen_registry_names.erl index f14b700..4eb9249 100644 --- a/src/dgen_registry_names.erl +++ b/src/dgen_registry/dgen_registry_names.erl @@ -6,84 +6,29 @@ -moduledoc false. -endif. -%% Durable fence for `dgen_registry`'s leader-driven commit pipeline. -%% -%% Section references (e.g. `§4.4`) point to `docs/dgen_registry_design.md`, the -%% registry's design-and-guarantees document. -%% -%% Historically this module stored one **occupancy key per registered name** -%% (`{Tuid, <<"names">>, Name} -> {Epoch, PidNode}`) as the authoritative taken-set. -%% That has been collapsed to a **single key per registry**: a monotonic version at -%% `{Tuid, <<"version">>}`, rewritten once per committed batch with a conflict-free -%% versionstamp (`set_versionstamped_value`), so its value advances with FDB's -%% commit-version order. See §4.4 of `docs/dgen_registry_design.md`. -%% -%% The bump exists *solely* to turn the fenced commit into a write transaction so the -%% leader-key read-conflict is actually resolved by the FDB resolver (a transaction -%% with conflict ranges but no mutations commits trivially without consulting the -%% resolver). The registry's name→pid state lives entirely in the members' in-memory -%% maps, reconstructed on a handoff by the cross-member gather -%% (`dgen_registry_member:gather_maps/3`); the DB no longer stores name state. -%% -%% ## Key layout +%% Durable fence for the leader-driven commit pipeline (§4.4 of +%% docs/design/dgen_registry_design.md). Name state lives only in members' memory; the +%% DB holds one version key per registry, versionstamp-bumped once per batch. The bump +%% exists solely to make the commit a write transaction, so the resolver actually checks +%% the leader-key read-conflict. %% %% ``` -%% {Tuid, <<"version">>} -> 10-byte versionstamp (big-endian commit version) %% the only data key -%% {Tuid, <<"leader">>} -> term_to_binary({MemberId, Epoch}) %% the fence (elector-owned) +%% {Tuid, <<"version">>} -> 10-byte versionstamp (big-endian commit version) +%% {Tuid, <<"leader">>} -> term_to_binary({MemberId, Epoch}) (elector-owned fence) %% ``` %% -%% So a registry's *name table* holds ~2 DB keys for N processes, independent of N. -%% (The elector's membership/leader-election queue keeps its own small set of keys, -%% bounded by cluster size and churn — also independent of N. See §4.4.) -%% -%% ## Guarantee change: unconditional → single-fault uniqueness -%% -%% With a per-name occupancy set, the DB was the authoritative taken-set: a name could -%% never be re-issued while its key existed, regardless of how many members were lost -%% simultaneously. Collapsing to a single version key removes that backstop. Uniqueness -%% now rests on the in-memory maps and the replication invariants that keep every live -%% binding on at least two holders — the leader plus one follower (forwarded -%% registrations are two-holder by construction; a direct registration waits for a -%% replica ack before its caller is told `yes`; see `dgen_registry_member`). -%% -%% The registry therefore tolerates the loss of **any single member** without -%% re-issuing a live name, but **not** the simultaneous loss of both holders of a -%% binding (nor a degrade-open residual, where a direct ack proceeded on a replication -%% timeout). Closing that remaining gap is the job of the kill-both-on-conflict -%% termination backstop (§5.6 of the design doc) — not the DB. -%% -%% ## Fencing -%% -%% A commit is fenced against a concurrent leadership change on the leader key. The -%% fence compares **both the member id and the epoch** — `{Self, Epoch}` — so it is a -%% monotonic fencing token, not an identity check: a member deposed and later -%% re-elected holds a *new* epoch, and a commit it started under the old epoch can -%% never pass the fence in the new term (the classic ABA hazard of id-only fences). -%% Legacy values (a bare MemberId written by older versions) are accepted on id alone -%% until the next election rewrites the key. Two equivalent mechanisms are used, -%% both safe: -%% -%% - **Read** (the first commit after assuming leadership; every retry): read the -%% leader key inside the transaction and proceed only if it still names `{Self, -%% Epoch}`. The -%% read adds a read-conflict, so a concurrent leadership change conflicts the commit; -%% the retry re-reads, sees the new leader, and aborts with `fenced`. -%% - **Write-conflict only** (steady state, when the read version is pinned to a prior -%% committed version): add a read-conflict on the leader key *without* reading it. A -%% pinned version is necessarily a leader-era version (we only commit while leader), -%% so any later leader-key write conflicts the commit — fencing with no storage read, -%% a single-round-trip commit. The version bump makes this a write transaction, so -%% the conflict is actually resolved. A retry falls back to a read, because its fresh -%% GRV may already post-date the change, which a conflict range would miss. +%% Uniqueness therefore survives any single member loss but not loss of both holders of a +%% binding; that residual is covered by kill-both-on-conflict (§5.6), not the DB. %% -%% A stale leader therefore can never commit. +%% Fencing on `{Self, Epoch}` (epoch prevents ABA). Pinned read version (leader-era): +%% read-conflict on the leader key without reading it. Fresh GRV or retry: read it, +%% since the GRV may post-date a leadership change a conflict range would miss. %% %% *This documentation is LLM-generated. See the AI disclosure in README.md.* -behaviour(dgen_transaction). -export([read_version/2, read_committed_frontier/2, start_commit/4, verify_leader/4]). -%% dgen_transaction callbacks (the async, non-blocking, cached-GRV commit path). -export([init/1, handle_begin/2, handle_retry/2, handle_committed/2]). -if(?DOCATTRS). @@ -177,9 +122,6 @@ start_commit({Db, Dir}, Tuid, Self, Opts) -> epoch => maps:get(epoch, Opts), leader_key => LeaderKey, version_key => VersionKey, - %% How to fence the first attempt (see the Fencing section): a pinned read - %% version is leader-era, so a leader-key write-conflict fully fences with - %% no read; a fresh GRV must read the leader key. fence => fence_mode(ReadVersion) }, TxOpts = @@ -231,11 +173,8 @@ verify_leader(Tenant = {_Db, Dir}, Tuid, Self, Epoch) -> _:_ -> false end. -%% Does a decoded leader-key value name `{Self, Epoch}`? The current format is -%% `{MemberId, Epoch}`; a bare `{Node, Name}` member id is the legacy (pre-epoch) -%% format, accepted on id alone until the next election rewrites the key. Note a -%% member id is itself a 2-tuple, so the formats are distinguished by the second -%% element's type (an integer epoch vs. a registry-name atom). +%% Rolling upgrade: a bare `{Node, Name}` (pre-epoch) is accepted on id alone. Both +%% formats are 2-tuples; the second element's type (integer vs atom) tells them apart. leader_matches({{_, _} = Leader, LeaderEpoch}, Self, Epoch) when is_integer(LeaderEpoch) -> Leader =:= Self andalso LeaderEpoch =:= Epoch; leader_matches({undefined, LeaderEpoch}, _Self, _Epoch) when is_integer(LeaderEpoch) -> @@ -252,13 +191,8 @@ leader_matches(_Other, _Self, _Epoch) -> init(Args) -> {ok, Args}. -%% First attempt. When the read version is pinned (leader-era), fence with a -%% write-conflict on the leader key *without* reading it — no storage read, so the -%% happy-path commit is a single round-trip: a leadership change wrote the leader -%% key after our read version, conflicting the commit (which then retries and -%% reads, below). With a fresh GRV there is no such guarantee — the version may -%% already post-date an unheard-of leadership change a conflict range would miss — -%% so read the leader key instead. +%% Pinned (leader-era) read version: any later leader-key write conflicts us, so no +%% storage read is needed (single round-trip). handle_begin(Tx, State = #{fence := conflict, leader_key := LeaderKey}) -> B = dgen_config:backend(), B:add_read_conflict_key(Tx, LeaderKey), @@ -267,19 +201,13 @@ handle_begin(Tx, State = #{fence := conflict, leader_key := LeaderKey}) -> handle_begin(Tx, State = #{fence := read}) -> fenced_by_read(Tx, State). -%% A retry runs after on_error reset the read version to a fresh GRV, which may -%% post-date a leadership change — so always read the leader key to detect a new -%% leader and abort, rather than a conflict range that would miss it. +%% Retry runs on a fresh GRV that may post-date a leadership change: must read. handle_retry(Tx, State) -> fenced_by_read(Tx, State). handle_committed(Version, _State) -> {ok, Version}. -%% Read the leader key (the read adds the read-conflict that fences against a -%% concurrent leadership change) and bump the version only while still the leader -%% *for this epoch* — a depose-and-re-elect of the same member carries a newer -%% epoch, so a commit planned in the older term is fenced (no ABA). fenced_by_read(Tx, State = #{self := Self, epoch := Epoch, leader_key := LeaderKey}) -> B = dgen_config:backend(), case leader_matches(current_leader(B, Tx, LeaderKey), Self, Epoch) of @@ -294,12 +222,7 @@ fenced_by_read(Tx, State = #{self := Self, epoch := Epoch, leader_key := LeaderK %% Internal %% --------------------------------------------------------------------------- -%% The single write of a fenced commit: a versionstamp on the version key so the -%% value monotonically advances with FDB's commit-version ordering. This turns -%% the transaction into a write transaction (so the leader-key read-conflict is -%% resolved by the resolver) and advances the durable batch counter. -%% `set_versionstamped_value` is conflict-free on the value, so the version key -%% itself never causes a (retryable) conflict between successive leader commits. +%% Conflict-free on the value, so successive leader commits never conflict on it. bump_version(B, Tx, #{version_key := VersionKey}) -> B:set_versionstamped_value(Tx, VersionKey, <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>). diff --git a/src/dgen_registry/dgen_registry_pipeline.erl b/src/dgen_registry/dgen_registry_pipeline.erl new file mode 100644 index 0000000..ac00719 --- /dev/null +++ b/src/dgen_registry/dgen_registry_pipeline.erl @@ -0,0 +1,257 @@ +-module(dgen_registry_pipeline). + +-define(DOCATTRS, ?OTP_RELEASE >= 27). + +-if(?DOCATTRS). +-moduledoc false. +-endif. + +%% The leader's write pipeline: every op it has accepted and not yet answered. +%% `§N.N` refers to docs/design/dgen_registry_design.md. +%% +%% Phases, one field each: +%% queued `pending` waiting for the next group commit. +%% committing `committing` the single in-flight commit; arrivals ride the next one. +%% awaiting `acks` a committed direct registration waiting for a follower +%% holders copy before its `yes` (§5.5); the follower-side twin is +%% an intent's `awaiting_version` (dgen_registry_intents). +%% The member plans and applies the ops (plan_batch/4, apply_committed_plan/3). +%% +%% One surrender rule, surrender/2, for ops the leader cannot finish. `failed` (the +%% commit did not land, or landed under a moved view) retries them; `deposed` hands +%% them off by kind, by the same idempotence rule as the follower's intents: +%% add, local becomes a follower intent, re-driven to the next leader. +%% add, forwarded no reply: the forwarding follower re-drives its own intent. +%% set_meta `{error, no_leader}`, also on failed: a replace is unsafe to re-send. +%% remove/retract a waiting caller gets `ok`, also on failed; then an answered, +%% pid-guarded unregister intent. +%% down dropped: the next leader monitors the pid itself. +%% So `no` stays an adjudicated refusal (§3), never "the leader could not finish". +%% +%% Pure: surrender/2 returns effects for the member to carry out. +%% +%% *This documentation is LLM-generated. See the AI disclosure in README.md.* + +-export([new/0, enqueue/2, num_pending/1, in_flight/1, take_batch/2, take_pending/1]). +-export([start/4, finish/2, worker_down/2, committed/2, read_version/1, new_term/1]). +-export([await_acks/5, ack/3, ack_timeout/2, take_acks/1]). +-export([surrender/2]). + +-export_type([t/0, op/0, origin/0, effect/0]). + +-type member_id() :: dgen_registry_elector:member_id(). +-type from() :: gen_server:from(). + +%% Where an op came from, so how to answer it: plain variants are direct calls answered +%% via `From`; `forward*` ones get a `{register_reply | set_meta_reply | +%% unregister_reply, Ref, ...}` cast to the forwarding member. +-type origin() :: + {local, from()} + | {forward, member_id(), reference()} + | {meta, from()} + | {forward_meta, member_id(), reference()} + | {unreg, from()} + | {forward_unreg, member_id(), reference()}. + +%% A leader write op. A batch rides one fenced commit and applies in arrival order, so +%% it equals one-at-a-time processing. `remove`'s ReleasedPid is captured before the +%% optimistic local delete; when set it drives the durable clear and feeds the +%% conflict trail (§5.6). `down` is Ref-guarded against a stale DOWN. +-type op() :: + {add, Name :: term(), pid(), Meta :: {map(), term()}, origin()} + | {set_meta, Name :: term(), Index :: map(), Data :: term(), origin()} + | {remove, Name :: term(), ReleasedPid :: pid() | undefined, origin() | undefined} + | {retract, Name :: term(), pid()} + | {down, Name :: term(), reference()}. + +%% Awaiting holders: origin plus binding, so a fail-closed timeout can retract it. +-type direct() :: {origin(), Name :: term(), pid()}. + +-record(pipeline, { + pending = queue:new() :: queue:queue(op()), + num_pending = 0 :: non_neg_integer(), + %% Monitored so a worker dying without a result cannot wedge the pipeline. + committing = undefined :: undefined | #{ref := reference(), mref := reference(), plan := map()}, + %% Pinned as the next read version to skip a GRV; if stale, the worker retries. + last_version = undefined :: undefined | integer(), + %% BatchRef => {Seq, Direct, TimerRef, Needed, Acked}. Answered in Seq order, never + %% reference order, so simulated schedules are reproducible. + acks = #{} :: #{ + reference() => + {non_neg_integer(), [direct()], reference(), non_neg_integer(), #{ + member_id() => true + }} + }, + next_ack_seq = 0 :: non_neg_integer() +}). + +-opaque t() :: #pipeline{}. + +-type effect() :: + {reply, origin(), term()} + %% Hand the kind to the follower-side intents (dgen_registry_intents:submit/5). + | {intent, dgen_registry_intents:kind(), from() | answered} + | {retry, [op()]}. + +-spec new() -> t(). +new() -> #pipeline{}. + +%% --------------------------------------------------------------------------- +%% Queued +%% --------------------------------------------------------------------------- + +-spec enqueue(op(), t()) -> t(). +enqueue(Op, P = #pipeline{pending = Q, num_pending = N}) -> + P#pipeline{pending = queue:in(Op, Q), num_pending = N + 1}. + +-spec num_pending(t()) -> non_neg_integer(). +num_pending(#pipeline{num_pending = N}) -> N. + +-spec in_flight(t()) -> boolean(). +in_flight(#pipeline{committing = C}) -> C =/= undefined. + +%% Take up to `Max` oldest ops. The bound keeps a burst (a DOWN flood) from becoming +%% one huge commit whose plan/apply/broadcast would freeze the loop. +-spec take_batch(pos_integer(), t()) -> {[op()], t()}. +take_batch(Max, P = #pipeline{pending = Q, num_pending = N}) -> + Size = min(N, Max), + {Ops, Rest} = take_n(Q, Size, []), + {Ops, P#pipeline{pending = Rest, num_pending = N - Size}}. + +-spec take_pending(t()) -> {[op()], t()}. +take_pending(P = #pipeline{pending = Q}) -> + {queue:to_list(Q), P#pipeline{pending = queue:new(), num_pending = 0}}. + +take_n(Q, 0, Acc) -> + {lists:reverse(Acc), Q}; +take_n(Q, N, Acc) -> + {{value, Op}, Q1} = queue:out(Q), + take_n(Q1, N - 1, [Op | Acc]). + +%% --------------------------------------------------------------------------- +%% Committing +%% --------------------------------------------------------------------------- + +-spec start(reference(), reference(), map(), t()) -> t(). +start(Ref, MRef, Plan, P = #pipeline{committing = undefined}) -> + P#pipeline{committing = #{ref => Ref, mref => MRef, plan => Plan}}. + +-spec finish(reference(), t()) -> {ok, map(), reference(), t()} | stale. +finish(Ref, P = #pipeline{committing = #{ref := Ref, mref := MRef, plan := Plan}}) -> + {ok, Plan, MRef, P#pipeline{committing = undefined}}; +finish(_Ref, _P) -> + stale. + +%% A monitored process died: if it is the commit worker, release the slot. +-spec worker_down(reference(), t()) -> {ok, map(), t()} | stale. +worker_down(MRef, P = #pipeline{committing = #{mref := MRef, plan := Plan}}) -> + {ok, Plan, P#pipeline{committing = undefined}}; +worker_down(_MRef, _P) -> + stale. + +-spec committed(integer(), t()) -> t(). +committed(Version, P) -> P#pipeline{last_version = Version}. + +-spec read_version(t()) -> undefined | integer(). +read_version(#pipeline{last_version = V}) -> V. + +%% The cached version belongs to the ended term: never pin a later term's read to it. +-spec new_term(t()) -> t(). +new_term(P) -> P#pipeline{last_version = undefined}. + +%% --------------------------------------------------------------------------- +%% Awaiting holders +%% --------------------------------------------------------------------------- + +-spec await_acks(reference(), [direct()], reference(), non_neg_integer(), t()) -> t(). +await_acks(BatchRef, Direct, TimerRef, Needed, P = #pipeline{acks = A, next_ack_seq = Seq}) -> + P#pipeline{ + acks = A#{BatchRef => {Seq, Direct, TimerRef, Needed, #{}}}, next_ack_seq = Seq + 1 + }. + +%% Distinct followers count once; at `Needed` the batch's direct registrations are done. +-spec ack(reference(), member_id(), t()) -> + {done, [direct()], reference(), t()} | {waiting, t()}. +ack(BatchRef, Follower, P = #pipeline{acks = A}) -> + case A of + #{BatchRef := {Seq, Direct, TimerRef, Needed, Acked}} -> + Acked1 = Acked#{Follower => true}, + case map_size(Acked1) >= Needed of + true -> + {done, Direct, TimerRef, P#pipeline{acks = maps:remove(BatchRef, A)}}; + false -> + Entry = {Seq, Direct, TimerRef, Needed, Acked1}, + {waiting, P#pipeline{acks = A#{BatchRef => Entry}}} + end; + _ -> + {waiting, P} + end. + +-spec ack_timeout(reference(), t()) -> + {expired, [direct()], non_neg_integer(), non_neg_integer(), t()} | none. +ack_timeout(BatchRef, P = #pipeline{acks = A}) -> + case maps:take(BatchRef, A) of + {{_Seq, Direct, _TimerRef, Needed, Acked}, A1} -> + {expired, Direct, Needed, map_size(Acked), P#pipeline{acks = A1}}; + error -> + none + end. + +%% Every batch still awaiting holders, in arrival order. +-spec take_acks(t()) -> + {[{[direct()], reference(), non_neg_integer(), non_neg_integer()}], t()}. +take_acks(P = #pipeline{acks = A}) -> + Batches = [ + {Direct, TimerRef, Needed, map_size(Acked)} + || {_Seq, Direct, TimerRef, Needed, Acked} <- lists:sort(maps:values(A)) + ], + {Batches, P#pipeline{acks = #{}}}. + +%% --------------------------------------------------------------------------- +%% Exits +%% --------------------------------------------------------------------------- + +%% Effects come out in op order. +-spec surrender(failed | deposed, [op()]) -> [effect()]. +surrender(failed, Ops) -> + {Replies, Retry} = lists:foldl( + fun + ({set_meta, _, _, _, Origin}, {Rs, Rt}) -> + {[{reply, Origin, {error, no_leader}} | Rs], Rt}; + ({remove, Name, Pid, Origin}, {Rs, Rt}) when Origin =/= undefined -> + {[{reply, Origin, ok} | Rs], [{remove, Name, Pid, undefined} | Rt]}; + (Op, {Rs, Rt}) -> + {Rs, [Op | Rt]} + end, + {[], []}, + Ops + ), + lists:reverse(Replies) ++ + case Retry of + [] -> []; + _ -> [{retry, lists:reverse(Retry)}] + end; +surrender(deposed, Ops) -> + lists:flatmap(fun deposed/1, Ops). + +deposed({add, Name, Pid, Meta, {local, From}}) -> + [{intent, {register, Name, Pid, Meta}, From}]; +deposed({add, _Name, _Pid, _Meta, _Forwarded}) -> + []; +deposed({set_meta, _Name, _Index, _Data, Origin}) -> + [{reply, Origin, {error, no_leader}}]; +deposed({remove, Name, Pid, Origin}) -> + Reply = + case Origin of + undefined -> []; + _ -> [{reply, Origin, ok}] + end, + Reply ++ retract(Name, Pid); +deposed({retract, Name, Pid}) -> + retract(Name, Pid); +deposed({down, _Name, _Ref}) -> + []. + +%% Nothing was bound when an unregister was asked: nothing to retract. +retract(_Name, undefined) -> []; +retract(Name, Pid) -> [{intent, {unregister, Name, Pid}, answered}]. diff --git a/src/dgen_registry/dgen_registry_replica.erl b/src/dgen_registry/dgen_registry_replica.erl new file mode 100644 index 0000000..2892174 --- /dev/null +++ b/src/dgen_registry/dgen_registry_replica.erl @@ -0,0 +1,382 @@ +-module(dgen_registry_replica). + +-define(DOCATTRS, ?OTP_RELEASE >= 27). + +-if(?DOCATTRS). +-moduledoc false. +-endif. + +%% A member's copy of the registry's name table, and the rules for advancing it. +%% `§N.N` refers to docs/design/dgen_registry_design.md. +%% +%% A replica is a position in the leader's totally ordered stream of committed batches +%% plus the content it implies: `tab` (rows `{Name, Pid, Index, Data}` in a protected +%% ETS table, read lock-free by any process), `index` (attribute -> value -> names, +%% derived from the rows), `version` (commit version applied up to) and `epoch` (the +%% leadership term followed). Commit versions are monotonic across terms, so replicas +%% compare by version alone. +%% +%% Invariant: two replicas at the same version hold the same content. Freshest-wins +%% handoff reconstruction (§5.7) depends on it; every admission rule protects it. +%% +%% Admission mirrors formal/DgenRegistryReplication.tla action for action: +%% offer_batch/5 RecvBcastApply / RecvBcastDrop / RecvBcastGap +%% offer_snapshot/4 RecvSnap (with the SafeAssumeSnap guard) +%% An older-epoch batch is dropped; a newer-epoch batch is applied but does not raise +%% `epoch`, while a current-or-newer snapshot does. The asymmetry is the model's and is +%% unreachable: a new leader snapshots each follower before broadcasting to it (FIFO). +%% +%% Pure: no sends, timers or replies; results tell the member what happened. +%% +%% *This documentation is LLM-generated. See the AI disclosure in README.md.* + +-export([new/1, tab/1, version/1, epoch/1]). +-export([offer_batch/5, offer_snapshot/4]). +-export([adopt/4, commit/5, insert/5, delete/2]). +-export([lookup_pid/2, lookup_metadata/2, exists/2, records/1, encode/1]). +-export([query/2, query_constraints/2, query_match_names/2, query_satisfies/2]). +-export([encode_records/1, decode_records/1]). + +-export_type([t/0, records/0, bcast_op/0, query/0]). + +-record(replica, { + tab :: ets:tid() | atom(), + index = #{} :: #{term() => #{term() => #{term() => []}}}, + version = 0 :: non_neg_integer(), + epoch = 0 :: non_neg_integer() +}). + +-opaque t() :: #replica{}. + +%% Replica content as a map, as carried by the handoff gather and snapshot fan-out. +-type records() :: #{term() => {pid(), map(), term()}}. + +%% One op of a `{names_batch, …}` broadcast, epoch already stripped to the envelope. +-type bcast_op() :: + {name_registered, term(), pid(), map(), term()} + | {metadata_set, term(), map(), term()} + | {name_unregistered, term(), pid() | undefined}. + +%% A tagged query (§4.9). Only `{all, Constraints}` (AND-equal) exists today. +-type query() :: {all, #{term() => term()}}. + +%% --------------------------------------------------------------------------- +%% Construction and accessors +%% --------------------------------------------------------------------------- + +%% Created empty on every member (re)start, with no heir: a crashed member's replica +%% is rebuilt from the leader's snapshot. +-spec new(atom()) -> t(). +new(TabName) -> + Tab = ets:new(TabName, [named_table, set, protected, {read_concurrency, true}]), + #replica{tab = Tab}. + +-spec tab(t()) -> ets:tid() | atom(). +tab(#replica{tab = Tab}) -> Tab. + +-spec version(t()) -> non_neg_integer(). +version(#replica{version = V}) -> V. + +-spec epoch(t()) -> non_neg_integer(). +epoch(#replica{epoch = E}) -> E. + +%% --------------------------------------------------------------------------- +%% Stream admission +%% --------------------------------------------------------------------------- + +%% Offer one broadcast batch. Applied only if contiguous (`PrevV` is our version), so +%% the replica stays a prefix of the leader's stream; on `gap` nothing is applied and +%% the caller requests a snapshot. A batch is one message, so a lost batch always +%% leaves a gap. The leader's heartbeat is an empty batch at `{V, V}`. +-spec offer_batch(non_neg_integer(), non_neg_integer(), non_neg_integer(), [bcast_op()], t()) -> + {applied, t()} | stale_epoch | duplicate | gap. +%% MUTATION_PARTIAL_BATCH (test-only, never in a release): re-admits per-op messages +%% sharing a version, reintroducing undetectable partially delivered batches. +-ifdef(MUTATION_PARTIAL_BATCH). +offer_batch(Epoch, PrevV, V, Ops, R = #replica{epoch = CurEpoch, version = Applied}) -> + if + Epoch < CurEpoch -> stale_epoch; + PrevV =:= Applied -> {applied, apply_ops(Ops, R#replica{version = max(Applied, V)})}; + V =:= Applied -> {applied, apply_ops(Ops, R)}; + V =< Applied -> duplicate; + true -> gap + end. +-else. +offer_batch(Epoch, PrevV, V, Ops, R = #replica{epoch = CurEpoch, version = Applied}) -> + if + Epoch < CurEpoch -> stale_epoch; + PrevV =:= Applied -> {applied, apply_ops(Ops, R#replica{version = max(Applied, V)})}; + V =< Applied -> duplicate; + true -> gap + end. +-endif. + +%% Offer a wholesale snapshot (handoff fan-out, joiner onboarding, resync answer). The +%% version guard stops a snapshot gathered before a batch but delivered after it from +%% rolling the replica back (§5.7). Equal versions are accepted: equal content. +-spec offer_snapshot(non_neg_integer(), non_neg_integer(), records(), t()) -> + {accepted, t()} | stale. +offer_snapshot(Epoch, V, Records, R = #replica{epoch = CurEpoch, version = Applied}) -> + case Epoch >= CurEpoch andalso V >= Applied of + true -> {accepted, adopt(Epoch, V, Records, R)}; + false -> stale + end. + +%% --------------------------------------------------------------------------- +%% Authoring (the leader's own replica) +%% --------------------------------------------------------------------------- + +%% Re-baseline unconditionally. The table is read lock-free meanwhile, so new rows are +%% written first and only then are absent names deleted: a reader never sees a bound +%% name go missing. +-spec adopt(non_neg_integer(), non_neg_integer(), records(), t()) -> t(). +adopt(Epoch, V, Records, R = #replica{tab = Tab}) -> + ets:insert( + Tab, [{Name, Pid, Index, Data} || {Name, {Pid, Index, Data}} <- maps:to_list(Records)] + ), + Absent = ets:foldl( + fun({Name, _Pid, _Index, _Data}, Acc) -> + case is_map_key(Name, Records) of + true -> Acc; + false -> [Name | Acc] + end + end, + [], + Tab + ), + lists:foreach(fun(Name) -> ets:delete(Tab, Name) end, Absent), + R#replica{index = index_rebuild(Records), version = V, epoch = Epoch}. + +%% Apply the leader's own committed batch at `V`. Only names in `DBOp` are touched, so +%% a concurrent optimistic delete is not clobbered, and a metadata-only op never +%% resurrects a row removed between plan and apply. +-spec commit(non_neg_integer(), map(), map(), map(), t()) -> t(). +commit(V, DBOp, WNames, WMeta, R) -> + R1 = maps:fold( + fun + (Name, clear, Acc) -> + delete(Acc, Name); + (Name, {set, _PidNode}, Acc) -> + {Index, Data} = maps:get(Name, WMeta), + insert(Acc, Name, maps:get(Name, WNames), Index, Data); + (Name, {meta, _PidNode}, Acc) -> + {Index, Data} = maps:get(Name, WMeta), + update_meta(Acc, Name, Index, Data) + end, + R, + DBOp + ), + R1#replica{version = V}. + +%% Row writers outside the stream (the optimistic local delete on unregister, the +%% legacy pre-version reply path's insert). Neither moves the version. +-spec insert(t(), term(), pid(), map(), term()) -> t(). +insert(R = #replica{tab = Tab, index = Inv}, Name, Pid, Index, Data) -> + Old = row_index(Tab, Name), + ets:insert(Tab, {Name, Pid, Index, Data}), + R#replica{index = index_add(index_remove(Inv, Name, Old), Name, Index)}. + +-spec delete(t(), term()) -> t(). +delete(R = #replica{tab = Tab, index = Inv}, Name) -> + Old = row_index(Tab, Name), + ets:delete(Tab, Name), + R#replica{index = index_remove(Inv, Name, Old)}. + +%% --------------------------------------------------------------------------- +%% Reads +%% --------------------------------------------------------------------------- + +-spec lookup_pid(t(), term()) -> pid() | undefined. +lookup_pid(#replica{tab = Tab}, Name) -> + case ets:lookup(Tab, Name) of + [{_Name, Pid, _Index, _Data} | _] -> Pid; + [] -> undefined + end. + +%% The public `get_metadata` shape, `{ok, #{pid, index, data}}` or `undefined`. +-spec lookup_metadata(t(), term()) -> {ok, map()} | undefined. +lookup_metadata(#replica{tab = Tab}, Name) -> + case ets:lookup(Tab, Name) of + [{_Name, Pid, Index, Data} | _] -> {ok, #{pid => Pid, index => Index, data => Data}}; + [] -> undefined + end. + +-spec exists(t(), term()) -> boolean(). +exists(#replica{tab = Tab}, Name) -> + ets:member(Tab, Name). + +%% Materialise the content as a records map. +-spec records(t()) -> records(). +records(#replica{tab = Tab}) -> + ets:foldl( + fun({Name, Pid, Index, Data}, Acc) -> Acc#{Name => {Pid, Index, Data}} end, #{}, Tab + ). + +%% --------------------------------------------------------------------------- +%% Snapshot codec +%% --------------------------------------------------------------------------- + +%% Snapshots ship as one compressed binary: off-heap, never GC-scanned, and shared by +%% every follower in a fan-out. `encode/1` folds ETS straight into a pair list (cheaper +%% than building a map); `encode_records/1` takes a map already in hand. Level 1 +%% because encode time sits on client-visible handoff/resync paths. `decode_records/1` +%% also accepts the bare list/map terms older senders ship. +-define(SNAPSHOT_ENCODE_OPTS, [{compressed, 1}]). + +-spec encode(t()) -> binary(). +encode(#replica{tab = Tab}) -> + Pairs = ets:foldl( + fun({Name, Pid, Index, Data}, Acc) -> [{Name, {Pid, Index, Data}} | Acc] end, [], Tab + ), + term_to_binary(Pairs, ?SNAPSHOT_ENCODE_OPTS). + +-spec encode_records(records()) -> binary(). +encode_records(Records) when is_map(Records) -> + term_to_binary(Records, ?SNAPSHOT_ENCODE_OPTS). + +-spec decode_records(binary() | list() | map()) -> records(). +decode_records(Bin) when is_binary(Bin) -> decode_records(binary_to_term(Bin)); +decode_records(List) when is_list(List) -> maps:from_list(List); +decode_records(Map) when is_map(Map) -> Map. + +%% --------------------------------------------------------------------------- +%% Queries (§4.7, §4.9) against the inverted index +%% --------------------------------------------------------------------------- + +%% Run a tagged query, returning `[#{name, pid, index, data}]`. +-spec query(t(), query()) -> [map()]. +query(R, {all, Constraints}) -> + query_constraints(R, Constraints). + +%% AND-equal query: intersect posting sets smallest-first, then re-check survivors +%% against their rows so a stale posting cannot yield a false positive. +-spec query_constraints(t(), #{term() => term()}) -> [map()]. +query_constraints(_R, Constraints) when map_size(Constraints) =:= 0 -> + []; +query_constraints(#replica{tab = Tab, index = Inv}, Constraints) -> + Postings = [posting(Inv, Attr, Value) || {Attr, Value} <- maps:to_list(Constraints)], + case lists:any(fun(P) -> map_size(P) =:= 0 end, Postings) of + true -> + []; + false -> + [Smallest | Rest] = lists:sort( + fun(A, B) -> map_size(A) =< map_size(B) end, Postings + ), + Names = [N || N <- maps:keys(Smallest), in_all(N, Rest)], + materialize(Names, Constraints, Tab) + end. + +%% The membership `#{Name => Pid}` a query currently resolves to. +-spec query_match_names(t(), query()) -> #{term() => pid()}. +query_match_names(R, Query) -> + maps:from_list([{N, P} || #{name := N, pid := P} <- query(R, Query)]). + +%% Does `Index` (a row's indexed-attribute map) satisfy the query? +-spec query_satisfies(query(), map()) -> boolean(). +query_satisfies({all, Constraints}, Index) -> + map_size(Constraints) > 0 andalso satisfies(Constraints, Index). + +%% --------------------------------------------------------------------------- +%% Internal +%% --------------------------------------------------------------------------- + +apply_ops(Ops, R) -> + lists:foldl(fun apply_op/2, R, Ops). + +apply_op({name_registered, Name, Pid, Index, Data}, R) -> + insert(R, Name, Pid, Index, Data); +apply_op({metadata_set, Name, Index, Data}, R) -> + update_meta(R, Name, Index, Data); +%% The released pid matters only to the member's conflict trail (§5.6). +apply_op({name_unregistered, Name, _ReleasedPid}, R) -> + delete(R, Name). + +%% Keeps the pid. A no-op if the row is absent: never invent postings. +update_meta(R = #replica{tab = Tab, index = Inv}, Name, Index, Data) -> + Old = row_index(Tab, Name), + case ets:update_element(Tab, Name, [{3, Index}, {4, Data}]) of + true -> R#replica{index = index_add(index_remove(Inv, Name, Old), Name, Index)}; + false -> R + end. + +%% `#{}` if no row, so a writer can retract old postings first. +row_index(Tab, Name) -> + case ets:lookup(Tab, Name) of + [{_N, _Pid, Index, _Data} | _] -> Index; + [] -> #{} + end. + +%% Removal prunes empty value- and attribute-maps. +index_add(Inv, Name, Index) -> + maps:fold( + fun(Attr, Value, Acc) -> + Vs = maps:get(Attr, Acc, #{}), + Names = maps:get(Value, Vs, #{}), + Acc#{Attr => Vs#{Value => Names#{Name => []}}} + end, + Inv, + Index + ). + +index_remove(Inv, Name, Index) -> + maps:fold( + fun(Attr, Value, Acc) -> + case Acc of + #{Attr := #{Value := Names} = Vs} -> + Names1 = maps:remove(Name, Names), + Vs1 = + case map_size(Names1) of + 0 -> maps:remove(Value, Vs); + _ -> Vs#{Value => Names1} + end, + case map_size(Vs1) of + 0 -> maps:remove(Attr, Acc); + _ -> Acc#{Attr => Vs1} + end; + _ -> + Acc + end + end, + Inv, + Index + ). + +index_rebuild(Records) -> + maps:fold( + fun(Name, {_Pid, Index, _Data}, Acc) -> index_add(Acc, Name, Index) end, + #{}, + Records + ). + +posting(Inv, Attr, Value) -> + case Inv of + #{Attr := #{Value := Names}} -> Names; + _ -> #{} + end. + +in_all(Name, Postings) -> + lists:all(fun(P) -> maps:is_key(Name, P) end, Postings). + +materialize(Names, Constraints, Tab) -> + lists:foldl( + fun(Name, Acc) -> + case ets:lookup(Tab, Name) of + [{_N, Pid, Index, Data} | _] -> + case satisfies(Constraints, Index) of + true -> [#{name => Name, pid => Pid, index => Index, data => Data} | Acc]; + false -> Acc + end; + [] -> + Acc + end + end, + [], + Names + ). + +satisfies(Constraints, Index) -> + maps:fold( + fun(Attr, Value, Ok) -> Ok andalso maps:get(Attr, Index, '$absent') =:= Value end, + true, + Constraints + ). diff --git a/src/dgen_registry_connector.erl b/src/dgen_registry_connector.erl deleted file mode 100644 index d5b46d8..0000000 --- a/src/dgen_registry_connector.erl +++ /dev/null @@ -1,522 +0,0 @@ --module(dgen_registry_connector). --behaviour(gen_server). - --define(DOCATTRS, ?OTP_RELEASE >= 27). - --include("../include/dgen_eta.hrl"). - --if(?DOCATTRS). --moduledoc false. --endif. - -%% Keeps Erlang distribution in step with the registry's database-backed membership, -%% on behalf of `dgen_registry_member`. One connector runs per registry, a sibling of -%% the member and elector under the registry supervisor. -%% -%% Section references (e.g. `§5.7`) point to `docs/dgen_registry_design.md`. -%% -%% ## Why a separate process -%% -%% This is purely the **node-level connectivity** concern — proactively opening -%% distribution connections to every member node, and reaping nodes that have gone -%% away — which is disjoint from the member's names/commit/leadership state. Its own -%% state is just `recently_down` (mesh suppression) and `ever_connected` (the -%% leader-probe gate); it never touches the names table, the commit pipeline, or -%% leadership. Splitting it out keeps the member focused on registration and keeps -%% this plumbing (and its subtle partition-detection ordering) in one small module. -%% -%% It reads the authoritative member set / leader / tokens from the **elector** -%% (off-loop, via short-lived helpers) and casts `{member_down, …}` back to the -%% elector; the resulting membership change flows to the member through the elector's -%% normal post-commit assume/fan-out, exactly as before. The one message it sends the -%% member directly is `{durable_epoch, E}` (a missed-handoff nudge, piggybacked on the -%% mesh fetch). -%% -%% ## Node-level vs member-level liveness -%% -%% There are two independent liveness mechanisms and this module owns one of them: -%% -%% - **Node-level** (here): `net_kernel:monitor_nodes/1` → `nodeup`/`nodedown`, plus -%% the periodic mesh, the backstop reap, and the leader-liveness probe. Drives -%% connect/disconnect and fenced `{member_down}` reaping. -%% - **Member-level** (in `dgen_registry_member`): an `erlang:monitor` on each peer -%% *member process*, whose DOWN reports a fenced `{member_down}` and feeds the -%% conflict trail. Untouched by this module. -%% -%% Both the member and this connector independently subscribe to -%% `net_kernel:monitor_nodes/1` (multiple subscribers each receive the events): the -%% member reacts to `{nodeup, _}` by re-announcing its own `{join}` (membership -%% identity); this connector reacts by clearing suppression and re-meshing. -%% -%% ## Connectivity modes (`connectivity`, §8) -%% -%% The connector runs in one of two modes, resolved once at `init/1` from the -%% registry's `connectivity` option (`dgen_config:connectivity/1`): -%% -%% - **`self_managed`** (default) — the full behaviour described above: the proactive -%% mesh reads the authoritative member set on a timer and connects to every member -%% node, so this registry is self-sufficient for distribution convergence. -%% -%% - **`provided_externally`** — the **mesh is off**. The connector opens no -%% distribution connections and never reads the member set on a timer; it assumes the -%% links it needs are established by something else on the node (typically another, -%% `self_managed`, "system" registry whose member set spans a superset of this -%% registry's nodes — distribution links are node-global, so a tenant registry -%% free-rides on them). What it still does is exactly the set of duties that are -%% **registry-scoped and cannot be delegated**: the stranded-member reap (`nodedown` -%% → fenced `{member_down}`), the leader-liveness probe, and the `{durable_epoch, _}` -%% nudge. Because the epoch nudge normally piggybacks on the mesh fetch, in this mode -%% it is re-homed onto its own `epoch_check` timer. The connector still subscribes to -%% `net_kernel:monitor_nodes/1` — `nodedown` drives the reap and `nodeup` feeds the -%% probe's `ever_connected` gate — it simply never *initiates* a connection. -%% -%% This mode trades a guarantee for a saving: it sheds the per-registry mesh work -%% (the periodic durable `get_members` read and the redundant `connect_node` calls), -%% which is the bulk of a co-located registry's connectivity overhead, in exchange for -%% the deployer's promise that connectivity is provided. If that promise is broken — -%% a member on a node the provider does not span — the node is silently isolated and -%% answers the CP write refusals (§5.3). As a cheap backstop against that footgun, a -%% `provided_externally` connector that observes **no elected leader for a sustained -%% window** logs a warning naming the delegated-connectivity contract, rather than -%% failing quietly. See §4.6 of the design doc for the full rationale and contract. -%% -%% *This documentation is LLM-generated. See the AI disclosure in README.md.* - --export([start_link/2]). --export([ - init/1, - handle_continue/2, - handle_call/3, - handle_cast/2, - handle_info/2, - terminate/2, - code_change/3 -]). - -%% How often the connector proactively connects Erlang distribution to every other -%% member node, so the cluster converges to a full mesh and `nodes()` returns all -%% members (see the "Membership and connectivity" guarantee in the design doc). A -%% backstop; startup and `nodeup` also trigger an immediate pass. --define(MESH_INTERVAL, 10000). - -%% After observing a node's `nodedown`, the mesh does *not* immediately reconnect it: -%% it waits this long so the departure can be recorded as a `{member_down}` and settle, -%% rather than fighting the partition machinery by healing a drop the instant it -%% happens. A genuine departure leaves the member set within the window (and so is not -%% reconnected at all); a `{nodeup, _}` clears the suppression early. --define(MESH_DOWN_COOLDOWN, 10000). - -%% After a node drops, reap any registry member still stranded on that (still- -%% disconnected) node from the elector's set — a backstop for the case where a -%% member's own monitor `{member_down}` was fenced away as stale (e.g. a `{join}` the -%% dead member enqueued on a nodeup just before its node vanished re-added it with a -%% token never seen here). Each pass reads the elector and reports `member_down` for -%% such members; it opens no distribution connection, so it cannot heal the very -%% partition it is reacting to. Retried a few times because the resurrecting `{join}` -%% may not be processed until after the first pass; it stops early once the node -%% reconnects, leaves the set, or the attempts are exhausted. --define(REAP_INTERVAL, 1000). --define(REAP_ATTEMPTS, 6). - -%% Leader-liveness probe. The normal failover trigger is `nodedown`, which only fires -%% for a node this connector was *connected to* and then lost. A leader recovered from -%% durable elector state whose node was never connected to — e.g. a whole-cluster cold -%% restart where the previously-elected node is gone — produces no `nodedown`, so the -%% dead leader is never reaped and every write forwards to it forever. This periodic -%% probe closes that gap: if the elected leader's node is unreachable, it reaps that -%% node through the same fenced `member_down` path a `nodedown` uses. -%% -%% Crucially, the probe reaps **only a leader whose node this connector has never been -%% connected to** (`ever_connected`). A leader we HAVE met and then lost is the -%% nodedown/monitor path's job: reaping it here would let a member on the minority side -%% of a *distribution-only* partition (mesh severed, database still reachable) depose -%% the majority's healthy leader through the shared database every probe interval — a -%% leadership ping-pong. With the gate, the isolated side settles into the CP refusals -%% (§5.3) instead. The first pass is delayed a grace window so a leader that is merely -%% *not connected yet* (the mesh connect is async) is given time to become reachable. --define(LEADER_PROBE_GRACE, 4000). --define(LEADER_PROBE_INTERVAL, 4000). - -%% `provided_externally` only (§8): how often the connector nudges its member with the -%% committed election epoch (the missed-handoff backstop), re-homed here because the -%% nudge normally piggybacks on the — now absent — mesh fetch. The same tick drives the -%% isolation-warning check below. Matches the mesh cadence it replaces. --define(EPOCH_CHECK_INTERVAL, 10000). - -%% `provided_externally` only: after this many consecutive `epoch_check` ticks with no -%% elected leader, log a warning that connectivity may not be provided (the footgun of -%% delegating the mesh to a provider that does not span this node). At -%% ?EPOCH_CHECK_INTERVAL this is ~60 s — long enough that ordinary startup and handoff -%% windows do not trip it, short enough to surface a real isolation quickly. --define(ISOLATION_WARN_TICKS, 6). - --record(state, { - %% The member's registered name — also this registry's name, and the target of the - %% `{durable_epoch, _}` nudge. Self node is always `node()`. - name :: atom(), - %% Connectivity mode (§8), resolved once at init from `dgen_config:connectivity/1`. - %% `self_managed` runs the proactive mesh; `provided_externally` turns it off and - %% runs the epoch-nudge/isolation-warning timer instead. See the moduledoc. - mode = self_managed :: self_managed | provided_externally, - %% The elector pid, discovered once via the shared supervisor in `discover_elector` - %% (like the member). `undefined` only between init/1 and that continuation. - elector :: pid() | undefined, - %% Nodes whose `nodedown` we saw within the last ?MESH_DOWN_COOLDOWN ms → timestamp. - %% The proactive mesh skips these so it does not reconnect a node the instant it - %% drops (letting the departure settle as a {member_down}); cleared on {nodeup}. - %% Unused in `provided_externally` mode (no mesh consults it). - recently_down = #{} :: #{node() => integer()}, - %% Nodes connected to at least once in this connector's lifetime — seeded from - %% nodes() at init, extended on every {nodeup, _}, never pruned. Gates the - %% leader-liveness probe (see ?LEADER_PROBE_GRACE). - ever_connected = #{} :: #{node() => true}, - %% `provided_externally` only: consecutive `epoch_check` ticks observed with no - %% elected leader, and whether the isolation warning has already been logged (a - %% latch, reset when a leader is next seen). See ?ISOLATION_WARN_TICKS. - no_leader_ticks = 0 :: non_neg_integer(), - isolation_warned = false :: boolean() -}). - -%% --------------------------------------------------------------------------- -%% Public API -%% --------------------------------------------------------------------------- - --if(?DOCATTRS). --doc "Starts the connector for registry `Name` (unregistered; held by the supervisor).". --endif. --spec start_link(Name :: atom(), Args :: map()) -> gen_server:start_ret(). -start_link(Name, Args) -> - gen_server:start_link(?MODULE, Args#{name => Name}, []). - -%% --------------------------------------------------------------------------- -%% gen_server callbacks -%% --------------------------------------------------------------------------- - -init(#{name := Name} = Args) -> - net_kernel:monitor_nodes(true), - Mode = dgen_config:connectivity(maps:get(config, Args, #{})), - %% The elector pid cannot be resolved from init/1: the supervisor is still - %% synchronously inside start_child for *this* process, so which_children would - %% deadlock. Defer to discover_elector, which also kicks off the first mesh pass - %% (self_managed) or the epoch-nudge timer (provided_externally). - {ok, - #state{ - name = Name, - mode = Mode, - elector = undefined, - recently_down = #{}, - %% A node connected at connector start counts as "ever connected". - ever_connected = maps:from_keys(nodes(), true) - }, - {continue, discover_elector}}. - -%% Resolve the elector via the shared supervisor (safe here — the supervisor is no -%% longer blocked on this process's init/1) and start the connectivity loops. The -%% member is started before this connector (see the supervisor child order), so it is -%% already registered and `dgen_registry:elector_pid/1` can walk from it to the elector. -handle_continue(discover_elector, State = #state{name = Name, mode = Mode}) -> - Elector = dgen_registry:elector_pid(Name), - true = is_pid(Elector), - case Mode of - self_managed -> - %% Proactively connect distribution to every member node, now and - %% periodically, so the cluster converges to a full mesh (see mesh_connect). - self() ! mesh_connect; - provided_externally -> - %% No mesh: connectivity is provided elsewhere (§4.6). Drive only the - %% epoch nudge (normally piggybacked on the mesh fetch) and the isolation - %% warning on the epoch tick. - arm(?EPOCH_CHECK_INTERVAL, epoch_check) - end, - %% Backstop the nodedown-driven failover for a leader we never connected to (see - %% probe_leader). Delayed so a genuinely-reachable leader is meshed first. Active - %% in both modes — it is a registry-scoped duty, not part of the mesh. - arm(?LEADER_PROBE_GRACE, probe_leader), - {noreply, State#state{elector = Elector}}. - -handle_call(_Request, _From, State) -> - {reply, {error, unknown_call}, State}. - -handle_cast(_Msg, State) -> - {noreply, State}. - -%% A node became reachable: record it as ever-connected (the probe gate) always, and — -%% in `self_managed` mode — clear any mesh suppression for it and re-mesh so we also -%% reconnect to any other members we lost touch with. In `provided_externally` mode -%% there is no mesh to drive; the ever_connected update (for the leader probe) is the -%% only reaction. The member independently handles {nodeup, _} to re-announce its own -%% {join}; that is its concern, not ours. -handle_info({nodeup, Node}, State = #state{mode = provided_externally}) -> - Ever = maps:put(Node, true, State#state.ever_connected), - {noreply, State#state{ever_connected = Ever}}; -handle_info({nodeup, Node}, State) -> - RD = maps:remove(Node, State#state.recently_down), - Ever = maps:put(Node, true, State#state.ever_connected), - self() ! mesh_connect, - {noreply, State#state{recently_down = RD, ever_connected = Ever}}; -%% A node dropped: start the backstop reap (both modes — it is registry-scoped). In -%% `self_managed` mode also remember it briefly so the mesh does not immediately -%% reconnect it (let the departure settle as a {member_down}); `provided_externally` -%% opens no connections, so it keeps no suppression state. -handle_info({nodedown, Node}, State = #state{mode = provided_externally}) -> - schedule_reap(Node, ?REAP_ATTEMPTS), - {noreply, State}; -handle_info({nodedown, Node}, State = #state{recently_down = RD}) -> - schedule_reap(Node, ?REAP_ATTEMPTS), - {noreply, State#state{recently_down = RD#{Node => erlang:system_time(millisecond)}}}; -%% `provided_externally` only: the mesh's epoch nudge is re-homed here. Read the -%% committed epoch (nudge the member on a missed handoff) and the current leader (drive -%% the isolation warning), both off the loop; the leader observation is reported back as -%% `{connectivity_leader, _}`. Re-arm the tick. -handle_info(epoch_check, State = #state{elector = Elector, name = Name}) -> - arm(?EPOCH_CHECK_INTERVAL, epoch_check), - spawn_epoch_check(Elector, Name, self()), - {noreply, State}; -%% Isolation warning (provided_externally): a leader observation from spawn_epoch_check. -%% No leader for ?ISOLATION_WARN_TICKS consecutive ticks latches a one-shot warning that -%% names the delegated-connectivity contract (§4.6) — the cheap backstop for a member on -%% a node the external provider does not span. Seeing a leader resets the counter and -%% re-arms the latch. -handle_info({connectivity_leader, undefined}, State = #state{no_leader_ticks = N}) -> - N1 = N + 1, - Warned1 = - case N1 >= ?ISOLATION_WARN_TICKS andalso not State#state.isolation_warned of - true -> - warn_possible_isolation(State#state.name, N1), - true; - false -> - State#state.isolation_warned - end, - {noreply, State#state{no_leader_ticks = N1, isolation_warned = Warned1}}; -handle_info({connectivity_leader, _Leader}, State) -> - {noreply, State#state{no_leader_ticks = 0, isolation_warned = false}}; -%% Backstop reap: while `Node` is still disconnected, report `member_down` for any -%% member the elector still lists on it, then re-arm for a few attempts. Stops as soon -%% as the node reconnects (a live member re-announces and its `{join}` wins) or the -%% attempts run out. Reads the elector and casts only — never opens a connection. -handle_info({reap_down, _Node, 0}, State) -> - {noreply, State}; -handle_info({reap_down, Node, N}, State = #state{elector = Elector}) -> - case dgen_utils:node_reachable(Node) of - true -> - ok; - false -> - spawn_reap(Elector, Node), - schedule_reap(Node, N - 1) - end, - {noreply, State}; -%% Periodic leader-liveness probe (see ?LEADER_PROBE_GRACE): if the elector's leader -%% sits on a node that is neither ours, nor connected, nor ever-connected, reap it so a -%% reachable member re-elects. Off the loop — spawn_probe_leader reads the elector and -%% casts only, opening no connection. -handle_info(probe_leader, State = #state{elector = Elector, ever_connected = Ever}) -> - arm(?LEADER_PROBE_INTERVAL, probe_leader), - spawn_probe_leader(Elector, node(), Ever), - {noreply, State}; -%% Converge Erlang distribution to a full mesh: read the authoritative member set from -%% the elector (a DB-backed read, identical on every node) and connect to every member -%% node not currently suppressed. The read runs in a short-lived helper so the -%% (potentially slow, durable-queue) `get_members` call never blocks this loop; the -%% helper reports the set back as `{mesh_members, _}`, and the suppression decision + -%% connect are made *there* — on this connector's mailbox, strictly ordered after any -%% `{nodedown, _}` — so an in-flight fetch cannot reconnect a node that dropped while it -%% was running. A successful connect fires `nodeup`, driving the member's rejoin. -handle_info(mesh_connect, State = #state{elector = Elector, name = Name}) -> - arm(?MESH_INTERVAL, mesh_connect), - Now = erlang:system_time(millisecond), - RD = maps:filter( - fun(_Node, Ts) -> Ts >= Now - ?MESH_DOWN_COOLDOWN end, State#state.recently_down - ), - spawn_mesh_fetch(Elector, self(), Name), - {noreply, State#state{recently_down = RD}}; -%% The mesh fetch helper reports the authoritative member node list. Deciding -%% suppression *here* (not in the helper, at spawn time) is what makes partition -%% detection reliable: a fetch spawned just before a node dropped carries no knowledge -%% of the drop, and computing suppression from its stale snapshot would reconnect the -%% very node being removed — healing the partition before the `{member_down}` can -%% settle. Because this message is on the connector's mailbox, it is strictly ordered -%% after the `{nodedown, Node}` the drop enqueued, so `recently_down` already reflects -%% it. The `connect_node` handshakes run off-process (they can block). -handle_info({mesh_members, MemberNodes}, State) -> - Now = erlang:system_time(millisecond), - RD = maps:filter( - fun(_Node, Ts) -> Ts >= Now - ?MESH_DOWN_COOLDOWN end, State#state.recently_down - ), - SelfNode = node(), - Targets = [ - Node - || Node <- MemberNodes, Node =/= SelfNode, not maps:is_key(Node, RD) - ], - spawn_mesh_connect(Targets), - {noreply, State#state{recently_down = RD}}; -handle_info(_Info, State) -> - {noreply, State}. - -terminate(_Reason, _State) -> - net_kernel:monitor_nodes(false), - ok. - -code_change(_OldVsn, State, _Extra) -> - {ok, State}. - -%% --------------------------------------------------------------------------- -%% Internal helpers -%% --------------------------------------------------------------------------- - -schedule_reap(Node, N) -> - arm(?REAP_INTERVAL, {reap_down, Node, N}), - ok. - -%% Report `member_down` for every member the elector currently lists on `Node`, fenced -%% with each member's current durable token (so a member that has genuinely rejoined is -%% not clobbered). Runs off the loop; reads the elector and casts only. -spawn_reap(Elector, Node) -> - _ = spawn(fun() -> - try dgen_server:priority_call(Elector, get_members) of - Members when is_list(Members) -> - lists:foreach( - fun - ({N, _} = MemberId) when N =:= Node -> - Token = dgen_registry_elector:member_token( - Elector, MemberId, undefined - ), - dgen_server:cast(Elector, {member_down, MemberId, Token}); - (_) -> - ok - end, - Members - ); - _ -> - ok - catch - _:_ -> ok - end - end), - ok. - -%% Read the elector's current leader; if it is a *different* node that is not connected -%% AND never connected to (the cold-restart recovery case), reap every member stranded -%% there (fenced `member_down`, via spawn_reap), triggering re-election. A leader that -%% is ourselves, unset, on a connected node, or on a node met before is left alone. -%% Runs off the loop and never opens a connection. -spawn_probe_leader(Elector, SelfNode, EverConnected) -> - _ = spawn(fun() -> - try dgen_server:priority_call(Elector, get_leader) of - {LeaderNode, _} when LeaderNode =/= SelfNode -> - case - lists:member(LeaderNode, nodes()) orelse - maps:is_key(LeaderNode, EverConnected) - of - true -> ok; - false -> spawn_reap(Elector, LeaderNode) - end; - _ -> - ok - catch - _:_ -> ok - end - end), - ok. - -%% Read the authoritative member set from the elector and report it back to the -%% connector as `{mesh_members, MemberNodes}`, in a short-lived helper so the -%% (potentially slow, durable-queue) read never blocks the loop. The member set is -%% read via a regular `call` (not `priority_call`): it rides the elector's durable -%% queue in order with the membership changes, so the mesh sees a just-joined node as -%% soon as its join is processed. The set is authoritative — the same on every node — -%% including a brand-new member that has only committed its join to the database. -%% -%% Piggybacks a durable-epoch check: read the committed election epoch (a priority -%% read) and send it to the **member** as `{durable_epoch, E}`. If it is ahead of what -%% the member has heard, a leadership handoff never reached this node — the member -%% re-joins to trigger a fresh assume/fan-out. The suppression decision is deliberately -%% *not* made here — the connector applies it on its own mailbox (see mesh_members) — so -%% a fetch spawned before a node dropped cannot reconnect it with a stale snapshot. -spawn_mesh_fetch(Elector, Connector, MemberName) -> - _ = spawn(fun() -> - nudge_durable_epoch(Elector, MemberName), - try dgen_server:call(Elector, get_members) of - Members when is_list(Members) -> - Connector ! {mesh_members, [Node || {Node, _Name} <- Members]}; - _ -> - ok - catch - _:_ -> ok - end - end), - ok. - -%% `provided_externally` counterpart to spawn_mesh_fetch's epoch half. With no mesh to -%% ride, the epoch nudge gets its own off-loop pass, and the same pass reads the current -%% leader and reports it back as `{connectivity_leader, _}` so the connector can drive -%% the isolation warning. Reads only (priority calls); opens no connection. -spawn_epoch_check(Elector, MemberName, Connector) -> - _ = spawn(fun() -> - nudge_durable_epoch(Elector, MemberName), - Leader = - try dgen_server:priority_call(Elector, get_leader) of - L -> L - catch - %% Treat an unreachable/erroring elector as "no leader observed" — it - %% counts toward the isolation window rather than being swallowed. - _:_ -> undefined - end, - Connector ! {connectivity_leader, Leader} - end), - ok. - -%% Read the committed election epoch (a priority read) and nudge the member with it: if -%% it is ahead of what the member has heard, a leadership handoff never reached this node -%% and the member re-joins to trigger a fresh assume/fan-out. Shared by the mesh fetch -%% (self_managed) and the epoch check (provided_externally). -%% -%% `try ... catch` rather than `try ... of ... catch`, and the difference is the -%% whole point: an `of` body sits *outside* the protected expression, so the send -%% below was never covered by the handler that was plainly written to cover it. -%% The member is addressed by registered name, and sending to a name with no -%% process raises -- so a nudge landing while the member is gone (its tree -%% restarting under `one_for_all`, or its node lost) crashed the mesh-fetch helper -%% this runs in, instead of being ignored like every other failure on this path. -%% Everything here is best-effort; the epoch is re-read on the next ?MESH_INTERVAL. -%% -%% Found by node-fault injection in the simulation suite, which widens the window -%% enough to hit it -- see `test/support/sim/README.md`. -nudge_durable_epoch(Elector, MemberName) -> - try - case dgen_server:priority_call(Elector, get_epoch) of - DurableEpoch when is_integer(DurableEpoch) -> - MemberName ! {durable_epoch, DurableEpoch}, - ok; - _ -> - ok - end - catch - _:_ -> ok - end. - -%% Log the delegated-connectivity isolation warning (provided_externally, §4.6). A -%% latched one-shot: fired once when no leader has been seen for ?ISOLATION_WARN_TICKS -%% consecutive ticks, re-armed when a leader reappears. -warn_possible_isolation(Name, Ticks) -> - logger:warning( - "dgen_registry ~p: connectivity=provided_externally but no leader has been " - "observed for ~b checks (~b ms). Distribution connectivity for this registry is " - "delegated (§4.6) — verify a self_managed provider registry spans this node, or " - "this node may be isolated and refusing writes (CP, §5.3).", - [Name, Ticks, Ticks * ?EPOCH_CHECK_INTERVAL] - ), - ok. - -%% Connect Erlang distribution to each target member node, off the loop (the handshake -%% can block on an unreachable node). `connect_node/1` is idempotent and a node that -%% cannot be reached simply returns `false`; either way the next mesh pass retries. -%% Targets are already filtered against the current recently-down suppression. -spawn_mesh_connect(Targets) -> - _ = spawn(fun() -> lists:foreach(fun net_kernel:connect_node/1, Targets) end), - ok. - -%% Arm a timer, and say so. See `dgen_registry_member:arm/2` — the same reasoning -%% applies here, and this module arms most of the periodic work in a run. -arm(Delay, Msg) -> - _ = ?ETA_LOG({arm, Msg, Delay}), - erlang:send_after(Delay, self(), Msg). diff --git a/src/dgen_registry_elector.erl b/src/dgen_registry_elector.erl deleted file mode 100644 index 53014a8..0000000 --- a/src/dgen_registry_elector.erl +++ /dev/null @@ -1,471 +0,0 @@ --module(dgen_registry_elector). --behaviour(dgen_server). - --define(DOCATTRS, ?OTP_RELEASE >= 27). - --include("../include/dgen_eta.hrl"). --define(SnapshotTimeout, 2000). - --if(?DOCATTRS). --moduledoc false. --endif. - -%% `dgen_server` callback module that tracks registry membership and elects a leader. -%% -%% Section references (e.g. `§5.7`) point to `docs/dgen_registry_design.md`, the -%% registry's design-and-guarantees document. -%% -%% The elector's state (member map + current leader) is stored durably in the -%% backend and is shared across every node that runs the same named registry. -%% Messages arrive via a durable FIFO queue, so membership changes are processed -%% one at a time, serialised through the backend. -%% -%% ## Leader election -%% -%% The incumbent leader is kept as long as it remains a member — leadership -%% only changes when the incumbent leaves or no leader has been elected yet. -%% This prevents thrashing when a non-leader node happens to win a backend -%% transaction race. -%% -%% When a new leader must be chosen (no valid incumbent), the node that wins -%% the backend transaction race is preferred: if `{node(), MemberName}` is a -%% current member, that node becomes leader. If not (transient window during -%% startup), `lists:min/1` over live member IDs is used as a deterministic -%% fallback. -%% -%% ## Replication on a membership change -%% -%% A membership change (`{join}` always; `{member_down}` when the leader itself -%% departs) routes through the current leader so it reconstructs and fans out the -%% names snapshot. This runs as a **post-commit action** (the `Actions` list -%% returned from `handle_cast_tx`, executed by `dgen_server` after the membership -%% transaction commits) — **no distributed lock**: -%% -%% 1. **Commit** — the backend transaction commits the new member set and leader key -%% atomically. This is the serialization point: the moment the new leader key is -%% committed, the *old* leader is fenced — its next batch commit conflicts on the -%% leader key and aborts (`dgen_registry_names`), so it can no longer accept a -%% registration. The handoff window is therefore naturally quiescent; the -%% cross-consumer lock the registry formerly used is unnecessary (§5.7 of the -%% design doc). -%% 2. **Leader assumption** — the action calls the leader via -%% `{elector_assume_and_distribute, TriggerMember, AllIds, Tokens, FreshIds, Epoch}`. The -%% leader reconstructs the names map itself by gathering every reachable member's -%% replica (`gather_maps/3`) and taking the freshest — no snapshot is passed, no -%% occupancy read, and no synchronous old-leader hand-off is needed. It resolves any -%% conflicts (§5.6), stores the epoch, sets up `erlang:monitor/2` for every -%% registered pid, and casts `{apply_names_snapshot, ..., Epoch}` to every follower -%% from its own process. The call is wrapped in `try/catch`: an unreachable leader -%% does not fail the action — the membership change is already committed and -%% affected members self-correct on the next membership event. -%% 3. **Follower sync** — each follower receives `{apply_names_snapshot}`, -%% overwrites its names map, and updates its leader field. Because these -%% casts originate from the same process as subsequent `{name_registered}` -%% broadcasts, Erlang's per-pair FIFO guarantee ensures every follower -%% sees the snapshot before any registration that post-dates the transition. -%% -%% ## Leader key in the backend -%% -%% ``` -%% Key path: {Tuid, <<"leader">>} -%% Value: term_to_binary({MemberId | undefined, Epoch}) -%% ``` -%% -%% The value carries the **epoch alongside the member id** so the fence is a real -%% fencing token, not just an identity check: a leader that is deposed and later -%% re-elected gets a *higher* epoch, so a commit it started in its earlier term can -%% never slip past the fence during its later term (the ABA case). Values written by -%% older versions (a bare MemberId) are tolerated by the fence readers in -%% `dgen_registry_names` until the next election rewrites the key. -%% -%% *This documentation is LLM-generated. See the AI disclosure in README.md.* - --export([ - init/1, - handle_cast_tx/3, - handle_call_tx/4, - leader_db_key/2, - tuid/1, - member_token/3 -]). - --export_type([member_id/0, member_info/0, registry_state/0]). - --type member_id() :: {node(), atom()}. -%% `fresh` records whether the member had ever synced any registry state when it -%% announced itself (`true` = a brand-new/empty member). A fresh member holds no -%% bindings by definition, so a leader whose handoff gather cannot reach it is not -%% actually missing anything — it is excluded from the `degraded` computation -%% (§5.6 prevention). The member re-announces with `fresh = false` after its first -%% snapshot, so the flag converges. --type member_info() :: #{ - joined_at := integer(), join_token := reference(), fresh => boolean() -}. --type registry_state() :: #{ - name := atom(), - members := #{member_id() => member_info()}, - leader := member_id() | undefined, - epoch := non_neg_integer(), - %% Durable presence subscriptions (§4.9): `SubId => {Watch, Notify}`, keyed by an - %% application-supplied id. Stored in the elector's *durable* dgen_server state, so - %% they outlive the Erlang cluster entirely — an application can tie a subscription - %% to a database entity's lifetime and have it survive a scale-to-zero and come back - %% intact. The current leader is fed this set (a delta on each change, the full map - %% on a handoff) and computes the notifications; the elector is the source of truth. - subscriptions => #{term() => {term(), term()}} -}. - -%% --------------------------------------------------------------------------- -%% dgen_server callbacks -%% --------------------------------------------------------------------------- - --if(?DOCATTRS). --doc """ -Initialises the elector state with an empty member map and undefined leader. - -`name` is the co-located member's **identity** — it is what `elect_leader/5` turns -into the local `member_id/0` when preferring the local member for leadership. -`keyspace` is the registry's durable **prefix**, and is what the tuid is derived -from; it defaults to `name`. The two differ only when a registry is started with -`dgen_registry:start_link/3`'s `keyspace` option, which lets several members of one -registry share a VM (see that function's docs). -""". --endif. --spec init(#{name := atom(), keyspace => atom()}) -> - {ok, dgen_server:tuid(), registry_state()}. -init(Args = #{name := Name}) -> - State = #{ - name => Name, members => #{}, leader => undefined, epoch => 0, subscriptions => #{} - }, - {ok, tuid(maps:get(keyspace, Args, Name)), State}. - --if(?DOCATTRS). --doc """ -The transaction-unit id (backend keyspace prefix) for registry `Name`. - -The single source of truth for the registry's backend key prefix: the elector, the -member (its version-key commit and leader-key fence), and the top-level supervisor -must all use the *same* tuid, so they derive it here rather than each rebuilding the -literal. Exported for that reason. -""". --endif. --spec tuid(atom()) -> dgen_server:tuid(). -tuid(Name) -> - {<<"dgen_registry.">>, atom_to_binary(Name)}. - --if(?DOCATTRS). --doc """ -Processes membership change messages within a backend transaction. - -Handles `{join, MemberId, Token, Fresh}` (a legacy 3-tuple `{join, MemberId, -Token}` is accepted as `Fresh = false`) and `{member_down, MemberId, Token}`. -Returns `{noreply, NewState, Actions}` when the handoff must be routed through the -leader (every join; a leader-changing member_down), else plain -`{noreply, NewState}`. The `Actions` run after the transaction commits — no -distributed lock (§5.7). - -Each `{join}` carries a unique token (a `reference()` generated by the member -process before enqueuing) and a freshness flag (whether the member has ever synced -registry state). The elector stores both in `member_info`. - -A `{member_down, MemberId, Token}` is silently discarded when its token does not -match the stored token for that member — this means the member has rejoined with -a new token since the DOWN was detected, so the message is stale. This prevents -a partition-recovery race where a `{member_down}` enqueued during the disconnect -is processed after the subsequent `{join}` that heals the cluster. -""". --endif. --spec handle_cast_tx(dgen_server:tx_ctx(), term(), registry_state()) -> - dgen_server:noreply_ret(). - -handle_cast_tx(TxCtx, {join, MemberId, Token}, State) -> - %% Legacy 3-tuple join (pre-`fresh`): treat as non-fresh — the conservative - %% choice, preserving the old degraded computation for that member. - handle_cast_tx(TxCtx, {join, MemberId, Token, false}, State); -handle_cast_tx(TxCtx, {join, MemberId, Token, Fresh}, State) -> - #{name := Name, members := Members, leader := OldLeader, epoch := OldEpoch} = State, - MemberInfo = #{ - joined_at => erlang:system_time(millisecond), join_token => Token, fresh => Fresh - }, - NewMembers = Members#{MemberId => MemberInfo}, - {NewLeader, NewEpoch} = elect_leader(TxCtx, Name, NewMembers, OldLeader, OldEpoch), - NewState = State#{members => NewMembers, leader => NewLeader, epoch => NewEpoch}, - %% A join always routes through the leader so it gathers + distributes the names - %% snapshot (delivering it to the joiner, and, on a leadership change, having the - %% new leader assume). All follower messages then share one sender (FIFO with - %% subsequent name broadcasts). This runs as a post-commit action — the - %% leader-key commit fences the old leader, so the handoff window is naturally - %% quiescent and no distributed lock is needed (§5.7). - {noreply, NewState, [assume_distribute_action(MemberId)]}; -handle_cast_tx(TxCtx, {member_down, MemberId, Token}, State) -> - #{name := Name, members := Members, leader := OldLeader, epoch := OldEpoch} = State, - case maps:get(MemberId, Members, undefined) of - undefined -> - %% Already removed — idempotent. - {noreply, State}; - #{join_token := StoredToken} when StoredToken =/= Token -> - %% Stale: member rejoined with a new token since this DOWN was detected. - {noreply, State}; - _ -> - NewMembers = maps:remove(MemberId, Members), - {NewLeader, NewEpoch} = elect_leader(TxCtx, Name, NewMembers, OldLeader, OldEpoch), - NewState = State#{members => NewMembers, leader => NewLeader, epoch => NewEpoch}, - case NewLeader =/= OldLeader of - true -> - %% Leadership changed (the leader itself went down) — the new - %% leader must gather + distribute. Post-commit action; the commit - %% fences the old leader, so no distributed lock is needed (§5.7). - {noreply, NewState, [assume_distribute_action(undefined)]}; - false -> - %% Leader unchanged — each surviving member detected the death - %% via its own erlang:monitor DOWN signal; no broadcast needed. - {noreply, NewState} - end - end; -%% Presence subscription writes (§4.9). Cast, not call: the public API fires these -%% fire-and-forget for latency, so `ok` means "accepted for durable processing", not yet -%% "committed" — the same posture as the membership casts above. Durability still holds -%% (the change commits in this consume transaction and survives a full cluster restart), -%% and because subscriptions are idempotent upserts keyed by an application-supplied id, -%% a write lost in the pre-consume window is recoverable by re-subscribing. Each returns -%% a post-commit action that pushes the change to the current leader. -handle_cast_tx(_TxCtx, {subscribe, SubId, Watch, Notify}, State) -> - Subs = subscriptions_of(State), - NewState = State#{subscriptions => Subs#{SubId => {Watch, Notify}}}, - {noreply, NewState, [presence_push_action({subscribe, SubId, Watch, Notify})]}; -handle_cast_tx(_TxCtx, {unsubscribe, SubId}, State) -> - Subs = subscriptions_of(State), - NewState = State#{subscriptions => maps:remove(SubId, Subs)}, - {noreply, NewState, [presence_push_action({unsubscribe, SubId})]}; -handle_cast_tx(_TxCtx, unsubscribe_all, State) -> - {noreply, State#{subscriptions => #{}}, [presence_push_action(unsubscribe_all)]}. - -%% Build the post-commit action that routes a membership change through the current -%% leader: it calls `{elector_assume_and_distribute, …}` so the leader gathers every -%% reachable member's names map, resolves conflicts (§5.6), and fans the snapshot out -%% to all followers. Runs after the membership transaction commits (see the -%% `dgen_server` `Actions` contract). `TriggerMember` is the joining member, or -%% `undefined` for a member_down. A `try/catch` keeps an unreachable leader from -%% failing the action — the membership change is already committed and affected -%% members self-correct on the next membership event. -assume_distribute_action(TriggerMember) -> - fun(#{members := All, leader := Leader, epoch := Epoch}) -> - case Leader of - undefined -> - ok; - _ -> - try - %% A newly-assuming leader seeds its presence state (§4.9) by pulling - %% the durable subscription set from its *co-located* elector when it - %% finishes assuming — so we do not thread it through here. - call_to_member( - Leader, - {elector_assume_and_distribute, TriggerMember, maps:keys(All), - member_tokens(All), fresh_ids(All), Epoch} - ) - catch - exit:Reason -> - depose_if_dead(Leader, Reason, All) - end - end - end. - -%% Read the durable subscription map, tolerating durable state written by a -%% pre-presence version (no `subscriptions` key). -subscriptions_of(State) -> - maps:get(subscriptions, State, #{}). - -%% The assume call to the leader failed. If it failed with `noproc` — we *reached* the -%% leader's node but found no member process registered there — the leader's registry is -%% gone (e.g. a graceful supervisor stop) while its node stays up. Leaving it as leader -%% would strand the cluster: a dead leader distributes nothing, and this can persist when -%% a `{join, Leader, NewToken}` a member enqueued on a nodeup is processed *after* the -%% member died, resurrecting it as leader. So enqueue a `{member_down}` fenced with the -%% leader's current token (from `All`, so a genuine concurrent re-join is not clobbered) -%% to force a re-election. -%% -%% Only `noproc` is treated as death: it means the node answered. A *not-connected* -%% failure is deliberately NOT deposed here — it is ambiguous (a live leader simply not -%% yet meshed to whichever node consumed this change would be wrongly unseated), and any -%% liveness probe that could disambiguate would have to open a distribution connection, -%% which would defeat partition detection elsewhere (see dgen_registry_member's mesh -%% suppression). A member on a genuinely dead *node* is instead removed by its peers' -%% own monitor DOWNs (fenced with the elector's current token — see the member's DOWN -%% handler). A plain call timeout on a connected node is a live-but-slow leader and is -%% left alone. -depose_if_dead(Leader, Reason, All) -> - case is_noproc(Reason) andalso maps:get(Leader, All, undefined) of - #{join_token := Token} -> - dgen_server:cast(self(), {member_down, Leader, Token}); - _ -> - ok - end. - -is_noproc(noproc) -> true; -is_noproc({noproc, _}) -> true; -is_noproc(_) -> false. - -%% --------------------------------------------------------------------------- -%% handle_call_tx/4 (all elector calls) -%% --------------------------------------------------------------------------- - -%% dgen_server dispatches *every* call — priority reads (`priority_call`, which still -%% bypass the durable queue and locks) and durable queued calls alike — to the `_tx` -%% variant when a module exports it (see dgen_server:invoke_tx_callback/4). The elector -%% only ever receives priority reads here (the presence *writes* are casts, handled by -%% handle_cast_tx), so all clauses just reply. (There is deliberately no `handle_call/3`: -%% exporting it too would be dead code, since the `_tx` variant always wins the dispatch.) --if(?DOCATTRS). --doc """ -Handles the elector's read-only priority calls: `get_leader`, `get_members`, `get_epoch`, -`{get_member_token, _}`, and `get_subscriptions` (the durable subscription map, §4.9). -The presence writes are casts — see handle_cast_tx/3. -""". --endif. --spec handle_call_tx(dgen_server:tx_ctx(), term(), dgen_server:from(), registry_state()) -> - dgen_server:reply_ret(). -handle_call_tx(_TxCtx, get_leader, _From, State) -> - {reply, maps:get(leader, State, undefined), State}; -handle_call_tx(_TxCtx, get_members, _From, State) -> - {reply, maps:keys(maps:get(members, State, #{})), State}; -handle_call_tx(_TxCtx, get_epoch, _From, State) -> - {reply, maps:get(epoch, State, 0), State}; -%% Return the current stored join token for a member (or `undefined` if it is not a -%% member). Read fresh from the durable elector state, so a member reporting a peer's -%% death can fence its `{member_down}` with the token the elector *actually* holds — not -%% a locally cached one that a dropped snapshot may have left stale. Without this, a -%% member whose peer re-announced a fresh token (on a nodeup) that never reached this -%% node would fence with the old token, and a genuine death would be discarded as stale, -%% stranding the dead member in the set (§5.7). -handle_call_tx(_TxCtx, {get_member_token, MemberId}, _From, State) -> - Token = - case maps:get(MemberId, maps:get(members, State, #{}), undefined) of - #{join_token := T} -> T; - _ -> undefined - end, - {reply, Token, State}; -%% Read the durable subscription map (§4.9) — reflects the elector's committed set. -handle_call_tx(_TxCtx, get_subscriptions, _From, State) -> - {reply, subscriptions_of(State), State}. - -%% Post-commit action that pushes a subscription change to the current leader as an -%% epoch-stamped `{presence_update, …}` cast. Fire-and-forget: the leader applies it -%% if the epoch is current, and a drop (unreachable leader) is caught up by the reseed -%% on the next handoff. No leader yet → nothing to push; the reseed covers it. -presence_push_action(Update) -> - fun(#{leader := Leader, epoch := Epoch}) -> - case Leader of - undefined -> ok; - _ -> push_to_leader(Leader, {presence_update, Update, Epoch}) - end - end. - -%% Cast to the leader member, guarding against an automatic distribution reconnect to a -%% disconnected node (the same hazard call_to_member/2 guards against — see §5.7). -push_to_leader({Node, Name}, Msg) -> - case dgen_utils:node_reachable(Node) of - true -> gen_statem:cast({Name, Node}, Msg); - false -> ok - end. - --if(?DOCATTRS). --doc """ -Client-side helper: read the token the elector holds for `MemberId` right now, -falling back to `Default` if the member is unknown or the read fails. - -A priority read (bypasses the durable queue). This is the fence for a -`{member_down}`: reporting the elector's *current* token means a peer that has -genuinely rejoined (advancing its token) is not clobbered by a stale DOWN, while a -truly dead member is still reaped. Pure — safe to call from an off-loop helper; the -member's peer-DOWN path and the connector's reap both use it. -""". --endif. --spec member_token(pid(), member_id(), Default) -> reference() | Default. -member_token(Elector, MemberId, Default) -> - try dgen_server:priority_call(Elector, {get_member_token, MemberId}) of - T when is_reference(T) -> T; - _ -> Default - catch - _:_ -> Default - end. - -%% --------------------------------------------------------------------------- -%% Internal helpers -%% --------------------------------------------------------------------------- - -%% Prefer the incumbent leader if it is still a member (sticky leadership). -%% This prevents thrashing when a non-leader node wins a queue transaction. -%% Falls back to the local node, then lists:min/1, when the incumbent is gone. -%% Returns {Leader, Epoch}. Epoch increments only when a new leader is chosen. -%% -%% The leader key's value is `{Leader, Epoch}` — the epoch makes the fence a -%% monotonic fencing token (see the "Leader key in the backend" moduledoc note): a -%% commit started under an older epoch of the *same* member cannot pass the fence -%% after a depose-and-re-elect, because the re-election carries a higher epoch. --spec elect_leader( - dgen_server:tx_ctx(), - atom(), - #{member_id() => member_info()}, - member_id() | undefined, - non_neg_integer() -) -> {member_id() | undefined, non_neg_integer()}. -elect_leader(#{td := {Tx, Dir}, tuid := Tuid}, Name, Members, OldLeader, OldEpoch) -> - {Leader, Epoch} = - case OldLeader =/= undefined andalso maps:is_key(OldLeader, Members) of - true -> - {OldLeader, OldEpoch}; - false -> - LocalId = {node(), dgen_registry:member_name(Name)}, - NewLeader = - case maps:is_key(LocalId, Members) of - true -> - LocalId; - false -> - case maps:keys(Members) of - [] -> undefined; - Ids -> lists:min(Ids) - end - end, - {NewLeader, OldEpoch + 1} - end, - B = dgen_config:backend(), - B:set(Tx, B:dir_pack(Dir, leader_key_tuple(Tuid)), term_to_binary({Leader, Epoch})), - {Leader, Epoch}. - --if(?DOCATTRS). --doc """ -Returns the packed backend key for the leader value. - -Exported so callers can set up a backend watch without going through the -elector process. -""". --endif. --spec leader_db_key(dgen_backend:dir(), dgen_server:tuid()) -> dgen_backend:key(). -leader_db_key(Dir, Tuid) -> - B = dgen_config:backend(), - B:dir_pack(Dir, leader_key_tuple(Tuid)). - -leader_key_tuple(Tuid) -> - dgen_key:extend(Tuid, <<"leader">>). - -%% Extract {MemberId => Token} from the members map for passing to member processes. -member_tokens(Members) -> - maps:map(fun(_, #{join_token := T}) -> T end, Members). - -%% Member ids currently marked fresh (never-synced), for the leader's `degraded` -%% computation — an unreachable fresh member cannot be holding bindings. -fresh_ids(Members) -> - [Id || {Id, Info} <- maps:to_list(Members), maps:get(fresh, Info, false)]. - -call_to_member({Node, Name}, Msg) -> - %% Guard against automatic distribution reconnect: if the target node is - %% not currently connected, exit immediately rather than letting - %% gen_server:call trigger a reconnect. An unintended reconnect during the - %% handoff fires {nodeup} on both sides, causing the peer to re-join with a new - %% token — the old {member_down} is then discarded as stale and the partition is - %% never detected. (This guard is independent of the now-removed distributed - %% lock; keep it regardless — §5.7.) - case dgen_utils:node_reachable(Node) of - true -> gen_statem:call({Name, Node}, Msg, ?SnapshotTimeout); - false -> exit({nodedown, Node}) - end. diff --git a/src/dgen_registry_member.erl b/src/dgen_registry_member.erl deleted file mode 100644 index 106344c..0000000 --- a/src/dgen_registry_member.erl +++ /dev/null @@ -1,4200 +0,0 @@ --module(dgen_registry_member). --behaviour(gen_statem). - --define(DOCATTRS, ?OTP_RELEASE >= 27). - --include("../include/dgen_eta.hrl"). --eta_observe( - {state, [ - member_id, - leader, - epoch, - synced, - applied_version, - pending_forwards, - pending_unregs, - deferred_yes, - committing, - num_pending - ]} -). - --if(?DOCATTRS). --moduledoc false. --endif. - -%% Local name cache and consistent read/write proxy for `dgen_registry`. -%% -%% Section references (e.g. `§5.6`) point to `docs/dgen_registry_design.md`, the -%% registry's design-and-guarantees document. -%% -%% Each node participating in a named registry runs one member process. -%% The member has two roles depending on whether it is the current leader. -%% -%% ## Storage -%% -%% Both leader and follower keep their local replica in a **named, `protected` ETS -%% table** (`dgen_registry:names_table/1`), owned and written **only** by this member -%% process — it is the single writer, so per-row updates are atomic and a row is never -%% observed torn. Each row is `{Name, Pid, Index, Data}`: the bound pid plus its -%% metadata (`Index`, the queryable map; `Data`, the opaque payload — both `#{}` / -%% `undefined` for a plain registration; see §4.7 of the design doc). The -%% table is the source of truth for the local replica; there is no parallel in-process -%% map. It is **not** durable: pids are node-local and process-lifetime-scoped, have no -%% meaning after a restart, and are never written to the backend. The member recreates -%% an empty table on (re)start — there is no heir — so a snapshot read in the gap before -%% the member is up (or while it is restarting after a crash) returns `undefined`. -%% -%% Consistent reads and writes go through the leader. Single-key snapshot reads -%% (`dgen_registry:whereis_name/1`, `get_metadata/1`) are served **lock-free from the -%% calling process** by reading the row directly (`ets:lookup/2`), without contacting the -%% member at all — many callers read concurrently and never queue behind the member's -%% mailbox. Multi-key indexed queries (`query/2`, `query_consistent/2`) instead run -%% **on the member's mailbox** against an in-state inverted index (`inv_index`): being -%% single-threaded, the member answers a query *between* batch applies, so it sees a -%% whole-batch-consistent snapshot rather than risking a half-applied group commit (§6). -%% -%% ## Follower role -%% -%% Keeps its local names replica (ETS) in sync by receiving `{names_batch, …}` and -%% `{apply_names_snapshot, …}` casts from the leader (one-way replication). A -%% `{names_batch, Ops, …}` carries one committed group commit whole — its -%% registrations, unregisters and metadata changes together — so a batch is applied -%% in full or not at all; see broadcast_batch/5 for why that atomicity is required -%% rather than merely tidy. All follower messages come from the leader process, so -%% Erlang's per-pair FIFO guarantee ensures followers always see a snapshot before -%% any batch broadcast that post-dates it. -%% -%% ## Membership and connectivity -%% -%% Keeping the Erlang-distribution mesh in step with registry membership — the -%% proactive `net_kernel:connect_node/1` pass, the backstop reap of stranded members, -%% and the leader-liveness probe — lives in the sibling `dgen_registry_connector`, not -%% here. So once a node's join is committed the connector connects to it and `nodes()` -%% converges to every reachable member; the application only has to make the nodes -%% *able* to connect (shared cookie, reachable hostnames), not wire up the mesh itself. -%% -%% The member still subscribes to `nodeup`/`nodedown` via `net_kernel:monitor_nodes/1`. -%% On `{nodeup, Node}` it does two things: it re-announces itself to the elector -%% (`{join, Self}`), and it promptly re-drives any unregisters it stashed while the -%% leader was unreachable (`redrive_pending_unregs`, Non-goal 5). The re-announce -%% handles an Erlang-level partition that caused both sides to remove each other from the -%% member set via `{member_down}` while the DB stayed healthy: once distribution -%% reconnects, both sides re-join and the elector reconstitutes the cluster without a -%% restart. The stashed-unregister re-drive means the removal reaches the (now-reachable) -%% leader immediately, rather than waiting for the re-onboarding snapshot. (The connector subscribes to node -%% events independently for its mesh bookkeeping.) The connector also nudges this -%% member with `{durable_epoch, E}` when the committed election epoch has moved ahead of -%% what the member has heard — a missed handoff — prompting a re-join. -%% -%% ## Registration blocks until a leader exists -%% -%% A registration is only ever answered `no` for an *adjudicated* refusal — the name is -%% held by a different live pid (or the opt-in `reject_when_degraded` prevention mode). -%% When there is simply **no reachable leader** (none elected yet at startup, a brief -%% no-leader window during a handoff, or the elected leader is unreachable in a -%% partition), the registration is **not** answered `no` — that would be a false -%% `already_started` to OTP's via machinery. Instead the caller is *blocked*: the -%% request is stashed in `pending_registers` and re-driven (`redrive_registers/1`) the -%% moment a leader is established (assuming leadership, or applying a leader's snapshot), -%% so it succeeds if a leader appears within the caller's `register_timeout` and -%% otherwise the caller's own `gen_server:call` times out (§3 of the design doc — a -%% timeout exits rather than returning a verdict). The stash is pruned by wall-clock -%% age so a prolonged no-leader window (a partition) cannot grow it without bound. All -%% register routing is in `route_register/5`. -%% -%% Forwards `{register, …}` and `{unregister, …}` to the leader. The register -%% forward is **asynchronous**: the follower stashes the caller's `From` (with the -%% registration's metadata) under a `Ref`, casts `{register_req, Ref, Self, Name, Pid, -%% Meta}` to the leader, and replies `{noreply}` — it never blocks on the leader. When -%% the leader's `{register_reply, Ref, yes|no, Version}` arrives, the follower (on -%% `yes`) writes the registration's row (pid + metadata) into its own table and then -%% answers the caller; on `no` the table is left unchanged. The `yes` ack is -%% **version-guarded**: it is answered only once the follower's replica has applied up -%% to the batch's commit `Version` — normally already true, since the batch's -%% `{names_batch}` broadcast precedes the reply (FIFO). If that broadcast was -%% gap-refused (the follower missed an earlier batch and awaits a resync), the ack is -%% *deferred* until the resync lands (`deferred_yes` / `flush_deferred`), so the -%% "second holder" is always version-visible to the freshest-wins handoff gather — -%% otherwise a single leader crash could silently drop an acked binding whose only -%% surviving copy sat in a gapped replica the gather's version comparison ignores. -%% -%% An `{unregister, …}` is likewise a tracked, call-shaped forward (`unregister_req` / -%% `unregister_reply`): the follower deletes its row optimistically, stashes the -%% caller under a `Ref`, and — if the reply never comes (dropped cast, deposed -%% leader) or no leader was reachable at all — re-drives the removal as a -%% pid-guarded retract on the next snapshot/leadership event (`redrive_unregs`), so -%% an explicit unregister is never silently lost. Explicit unregisters are rare, so -%% the extra round-trip costs nothing that matters. -%% -%% A `set_metadata` forward is also asynchronous: the follower stashes `From` under a -%% `Ref`, casts `{set_meta_req, Ref, Self, Name, Index, Data}`, and answers the caller -%% when the leader's `{set_meta_reply, Ref, _}` arrives. No optimistic update is needed — -%% routing the reply back through this member means the leader's `{names_batch, …}` -%% broadcast (FIFO, ahead of the reply) has updated the follower's row before the caller -%% is answered, preserving read-after-write for a subsequent local `get_metadata`. -%% -%% This async forward is required for deadlock-freedom: no member ever blocks its -%% loop on a call to another member on the registration path. (A blocking forward -%% would deadlock once the leader awaits a follower's replication ack for a direct -%% registration — every follower could be blocked forwarding while the leader is -%% blocked awaiting one of them.) The follower's `yes`-time `names` update also makes -%% it the binding's **confirmed second holder**: by the time the caller sees `yes`, -%% both the leader and the forwarding follower hold the binding, so it survives -%% either node failing — at no extra round-trip. -%% -%% Forwards still in flight when leadership changes are rejected (`no`); the caller -%% retries against the new leader. -%% -%% A **consistent read** (`whereis_name_consistent`) on a follower forwards the same -%% way: the follower casts `{whereis_req, Name, From}` to the leader and replies -%% `{noreply}`, and the leader answers the caller's `From` directly from its -%% authoritative map. The follower never blocks its loop on the read, so a consistent -%% read cannot head-of-line block its other messages. If the leader's node is -%% unreachable, the follower answers `undefined` immediately rather than hang the -%% caller. -%% -%% ## The state machine -%% -%% This member is a `gen_statem` (handle_event_function + state_enter) with four -%% states, each realizing one of the modes the TLA+ model -%% (formal/DgenRegistryReplication.tla) treats implicitly through -%% `leaderView[m]`/`epoch[m]`: -%% -%% searching no leader known (including before the first sync). -%% assuming a handoff gather is in flight (`assume_ref` armed); the -%% `{assume_gathered}` continuation is valid only here, so a -%% superseded gather is a visible state mismatch, not a -%% ref-compare. -%% leader this member believes it leads and is synced. -%% follower it follows a known leader. -%% -%% The state is a DERIVED projection (`state_of/1` over `leader` / `member_id` / -%% `assume_ref`), recomputed after every event by `next/2`: Data stays the -%% single source of truth, so the projection cannot drift. The leadership -%% lifecycle rides the role edges as state-enter calls — entering `leader` runs -%% `assume_leadership/1`, leaving it runs `relinquish_leadership/1` — so no -%% transition path can forget half of the pair. -%% -%% Three dimensions are DELIBERATELY data, not state, each because it proved -%% orthogonal to role: -%% -%% - gap/resync: `request_resync` can fire on a deposed-but-uninformed leader -%% (the spec's RecvBcastGap has no leaderView guard), so a gapped state -%% would lie about the role. -%% - leader reachability: `route_register`/`route_unregister` stash when the -%% KNOWN leader is unreachable, which is a property of the link, not of -%% this member's role — the reason the pending_registers/pending_unregs -%% stashes were kept over gen_statem's `postpone` (that, and the CP -%% contract: an unreachable-leader unregister answers `ok` NOW and keeps -%% the intent, where postpone would defer the reply; and stash entries also -%% originate internally from batch-salvage paths, which are not events). -%% - the commit pipeline (`committing`): ops must accumulate during an -%% in-flight commit, never be postponed. -%% -%% ## Leader role -%% -%% Assumed when the elector calls `{elector_assume_and_distribute, …}`. On a genuine -%% leadership change the member reconstructs its names map by gathering the freshest of -%% every reachable member's replica — the freshest map *is* the reconstructed state -%% (there is no durable taken-set to reconcile against, §4.4), then distributes -%% `{apply_names_snapshot}` casts to all followers from its own process (same sender as -%% future `{names_batch}` broadcasts — see elector moduledoc for the FIFO ordering -%% guarantee). Monitors for the registered pids are established by a chunked, -%% deferred sweep off the client-visible window (see `{monitor_sweep, …}` / -%% assume_leadership/1) — safe because a monitor on an already-dead pid fires an -%% immediate `noproc` DOWN, so a death inside the window is caught at establishment -%% rather than lost. Any stale Pid entries are removed when their DOWN signals -%% arrive. -%% -%% The gather is a **network** fan-out (an RPC per peer, bounded by ?GATHER_TIMEOUT), so -%% it runs **off the member's loop**: the elector's assume call is answered `ok` -%% immediately, and the `{assume_gathered, …}` continuation finishes assuming when the -%% gather returns (`spawn_assume_gather` → the assume_gathered handler, correlated by -%% `assume_ref`). This keeps the member responsive during a handoff — under churn an -%% inline multi-second gather would freeze the loop, time out the elector's call, and -%% stall every membership change queued behind it. During the (sub-?GATHER_TIMEOUT) -%% window the member is not yet leader/synced, so writes block (no reachable leader) and -%% are re-driven by the continuation, never served against a half-built map. -%% -%% The leader is the sole writer for the name table. It: -%% -%% - Serves all writes through one **group-commit buffer** — registrations -%% (`{register}` calls), unregisters (`{unregister}` casts), and auto-unregisters -%% (monitored-process `DOWN`s). Each is parked in the pending buffer; at most one -%% commit runs at a time, so ops that arrive while a commit is in flight accumulate -%% and ride the next batch (`dgen_registry_names:start_commit/4`, whose only write is -%% a fenced bump of the per-registry version key — the name delta is applied in -%% memory). The commit runs in a `dgen_transaction` worker so this -%% process keeps serving snapshot reads while it is in flight, and the read version -%% is pinned to the last committed version to skip a GRV (a stale pin falls back to -%% a fresh GRV in the worker). Ops are applied in arrival order, so a burst of -%% `DOWN`s (a peer node dying) or registrations (a node booting) coalesces into few -%% commits while preserving the one-at-a-time outcome. -%% When the worker reports back, each registration is answered through its origin -%% (a direct call via `gen_server:reply/2`, a forwarded one via a `{register_reply}` -%% cast to the forwarding follower), pids are monitored, and the batch is replicated as one -%% `{names_batch, …}` broadcast. A name already held by a *different* live -%% pid is rejected (`no`); re-registering the *same* pid under the same name is an -%% idempotent `yes` (see plan_op/3); a `DOWN` whose ref no longer matches the current -%% binding is ignored; a fenced commit (leadership lost) rejects the whole batch. -%% - Handles `{whereis, LogicalName}` calls: consistent read from local map. -%% - Handles `{unregister, LogicalName}` calls (and legacy casts): updates the map, -%% demonitors, replicates the batch, and answers the tracked -%% caller `ok` once the removal has committed. -%% - Monitors every registered Pid. When one dies, removes from the map -%% and replicates the batch to followers. -%% -%% On relinquishing leadership the member demonitors all registered Pids and -%% clears the leader-only state. The names replica (ETS) is kept intact (it still -%% serves caller-side snapshot reads). -%% -%% ## Replication and the two-holder invariant -%% -%% A registration is acked `yes` only once **two members** hold the binding, so it -%% survives any single node loss. The cost is asymmetric and maps onto the topology: -%% -%% - A **forwarded** registration (the common case — every follower forwards) is held -%% by the leader *and* the forwarding follower as soon as the follower applies its -%% optimistic update, so it is two-holder for free, with no extra round-trip. -%% - A **direct** registration (one originating on the leader's own node) has only the -%% leader as a holder, so the leader does replicate-before-ack: it broadcasts the -%% binding, then waits for ≥1 follower to confirm a copy (`replicate_sync` → -%% `replicate_ack`) before replying `yes`. Only the leader — 1 of m nodes — pays this, -%% which matches the forward asymmetry: the other m-1 nodes forward and get the second -%% holder for free. -%% -%% ### Caveats — this guarantee is deliberately modest, and under review -%% -%% The two-holder invariant is a pragmatic middle ground, not a strong durability -%% guarantee. It deliberately avoids both persisting pids (which cannot be done -%% meaningfully — a pid is node- and lifetime-scoped, and a stale pid could name a -%% reused process) and forcing every caller to re-register on failover (onerous for an -%% arbitrary process). The trade-offs: -%% -%% - **Single-fault only.** Losing both holders (e.g. the leader and the one follower -%% that acked) loses the binding. With the per-name occupancy set dropped (§4.4), the -%% DB no longer marks the name taken, so this now costs **both** addressability *and* -%% uniqueness: a lost name may be re-issued. Uniqueness is thus single-fault, matching -%% addressability; the kill-both-on-conflict termination backstop (§5.6) is what -%% re-converges if a re-issue ever races a survivor. -%% - **The replica is ephemeral and arbitrarily placed.** The "second holder" is just an -%% in-memory copy in *some* follower's map — whichever acked first, unrelated to where -%% the pid lives. Survival depends on that particular node staying up. -%% - **Degrade-open weakens it under load.** If fewer than `register_replicas` acks -%% arrive within `replicate_timeout`, the leader (by default) acks `yes` leader-only -%% (n=1) to preserve liveness — emitting telemetry. So the invariant is really "N -%% holders, or one under timeout." Set `strict_replication = true` to fail-closed -%% instead (reject + retract the binding); set `register_replicas > 1` for more holders -%% (bounded by the follower count). See §8 and `dgen_config`. -%% - **Pids cannot outlive their node.** If the pid's *own* node dies the pid is dead and -%% the name should be released anyway (its monitor fires). So the two-holder invariant -%% really protects the *routing entry* against loss of a node that does **not** host -%% the pid (typically the leader) — narrower than "registrations are replicated". -%% - **Whether n=2 is the right point is open.** A different point on the spectrum (a -%% re-registration contract, durable routing hints, or higher replication) may prove -%% better; the current choice is not considered final. -%% -%% ## Failure model -%% -%% Name-to-pid mappings are intentionally not stored in the backend — and, since -%% §4.4, neither is the taken-set: the only durable keys are the leader key (fence) -%% and a per-registry version counter. As a result: -%% -%% - **Leader crash**: the new leader reconstructs its map from the freshest of -%% every reachable member's replica (`gather_maps/3`). A binding the dead -%% leader committed to its map but had not yet replicated to any survivor is -%% silently lost — and, with no durable taken-set, its name may be re-issued -%% (single-fault uniqueness). A caller that received `yes` from `register_name/2` -%% may therefore find the name absent, or reassigned, after a leader failover; -%% re-registration after detecting the loss is the caller's responsibility, and -%% the termination backstop (§5.6) re-converges a re-issue that races a survivor. -%% -%% - **Full cluster restart**: all registered names are lost. Applications -%% must re-register on startup. -%% -%% *This documentation is LLM-generated. See the AI disclosure in README.md.* - -%% (The group-commit batch size, the replicate-before-ack target, and the timeout -%% are configurable per registry — see dgen_config:commit_batch_size/1, -%% register_replicas/1, replicate_timeout/1, strict_replication/1.) - -%% Per-member bound on the handoff gather: a member that does not return its names -%% map within this is skipped (its bindings are unavailable for this gather; a -%% binding only that member held is then lost — single-fault uniqueness). --define(GATHER_TIMEOUT, 2000). -%% How long the off-loop assume gather keeps re-gathering while its reconstruction is -%% behind the durable version key (the committed frontier), and how long it pauses -%% between attempts. This closes the handoff-gather race proven in -%% formal/DgenRegistryReplication.tla (the SafeAssume fence): a gather can momentarily -%% race an in-flight broadcast still queued in a peer's mailbox and reconstruct a stale -%% map; the peer applies that broadcast within milliseconds, so a brief re-gather -%% catches up. Bounded overall by ?GATHER_TIMEOUT (past it the missing version's holder -%% is genuinely unreachable — a real degrade — so we proceed with what we have). --define(ASSUME_CATCHUP_INTERVAL, 25). -%% Upper bound on the off-loop pull of durable subscriptions from the co-located elector -%% during a handoff (§4.9). Short and best-effort: a busy elector falls back to the -%% empty set, reconciled by a later delta / the next handoff. --define(SUBS_PULL_TIMEOUT, 1000). - -%% How often the periodic maintenance pass runs: it prunes `recently_released` -%% (the conflict-detector trail, §5.6 — TTL from `dgen_config:conflict_release_ttl/1`) -%% and the per-name `kill_budget` timestamps. Trail pruning is *suspended* while any -%% current member is disconnected: a disconnected member misses unregister broadcasts -%% entirely (they are dropped, not delayed), so its stale rows can surface at a rejoin -%% gather arbitrarily later than any broadcast lag — the trail must outlive the -%% partition, not just the lag. The trail itself is also replicated (unregister -%% broadcasts carry the released pid) and merged across members at a handoff gather, -%% so it survives leadership changes. --define(PRUNE_INTERVAL, 30000). - -%% How long a follower waits after requesting a resync (it observed a gap in the -%% leader's broadcast stream) before it may request again. --define(RESYNC_RETRY, 2000). - -%% Names monitored per {monitor_sweep, …} step when a new leader establishes its -%% registered-process monitors (see assume_leadership/1). ~0.7us per monitor, so a -%% chunk costs a few ms of loop time — small enough to interleave with live traffic, -%% large enough that a 200k-name sweep is ~40 messages. --define(MONITOR_SWEEP_CHUNK, 5000). - -%% How often the leader broadcasts an **empty** batch stamped at its current applied -%% version — a replication heartbeat. -%% -%% Gap detection is otherwise entirely traffic-driven: a follower learns it missed a -%% batch only when a *later* batch arrives whose PrevVersion does not match its -%% replica (`apply_bcast/6`), or when a forwarded register reply arrives ahead of it. -%% So a follower that loses the *tail* of the stream — the last batch before writes -%% stop — has nothing left to reveal the gap, and holds a stale replica for as long -%% as the cluster stays quiescent. Losing the `resync_req` (or the snapshot -%% answering it) has the same shape: `resync_timeout` clears the once-per-window -%% guard, but nothing re-requests without new traffic. -%% -%% The heartbeat closes both. It is an ordinary `{names_batch, [], …}` with -%% `PrevVersion = Version = the leader's applied_version`, so it needs no new -%% handling: a caught-up follower matches `PrevV =:= applied_version` and applies -%% nothing, while a follower that is behind matches neither that nor `V =< Applied` -%% and so takes the existing gap branch and resyncs. Cost is one small cast per -%% follower per interval, independent of the number of registered names. --define(REPLICA_HEARTBEAT_INTERVAL, 5000). - -%% Delay before re-driving the destructive ops (removes/retracts/downs) of a batch -%% that failed to commit — a backend error, a fence, or a worker death. Registrations -%% are simply rejected (callers retry), but a lost unregister/retract would leave a -%% permanent replica divergence, so those are re-enqueued (still leader) or forwarded -%% to the current leader (deposed) after this delay. --define(REQUEUE_DELAY, 1000). - -%% Bound on the per-peer aliveness probe used when adjudicating a suspected conflict. --define(ALIVE_PROBE_TIMEOUT, 1000). - --export([start_link/2]). -%% gen_statem callbacks. The former gen_server handlers (handle_call/3, -%% handle_cast/2, handle_info/2, handle_continue/2) live on as internal -%% functions dispatched by handle_event/4 — phase 1 of the statem port keeps -%% their bodies verbatim so the behaviour swap is isolated from any state -%% redesign; the real states arrive in phase 2. --export([ - init/1, - callback_mode/0, - handle_event/4, - terminate/3, - code_change/4 -]). -%% Exported for unit testing the §5.6 conflict-detection predicate in isolation. --export([detect_conflicts/3]). -%% Exported for unit testing the snapshot codec in isolation — every wire shape -%% decode_records/1 tolerates, and encode_table/1's table-to-blob equivalence. --export([encode_table/1, encode_records/1, decode_records/1]). -%% Exported for unit testing the group-commit batch planner in isolation — in -%% particular, the batch-local overlay (`seed_lookup/3`, the `removed` marker) that lets -%% a later op in the same batch see an earlier op's not-yet-committed decision without a -%% full-registry seed map. Takes a real ETS table (the same `{Name, Pid, Index, Data}` -%% row shape as the member's names table) so a test can pre-populate the pre-batch state. --export([plan_batch/4]). - -%% A registration's metadata: an indexed map (queryable, a later increment) and an -%% opaque data payload (returned verbatim). A plain register_name/2 uses the empty -%% default `{#{}, undefined}`. See §4.7 of the design doc. --type meta() :: {Index :: map(), Data :: term()}. - -%% A presence subscription's watch/notify queries, in the internal tagged form (see -%% dgen_registry:query/0). Extensible: only `{all, Constraints}` (AND-equal) exists today. --type query() :: {'all', #{term() => term()}}. - -%% An application-supplied presence subscription id (§4.9). The durable subscriptions -%% live in the elector's dgen_server state; the leader is fed the set and computes the -%% notifications. Keyed by this id so an application can tie a subscription to a -%% database entity and re-address it (unsubscribe, re-subscribe) by the same stable key -%% across a full cluster restart. --type sub_id() :: term(). - -%% Where a write op came from, and how to answer it once committed: -%% - {local, From}: a direct register/2,3 call to this leader; answered -%% with gen_server:reply/2. -%% - {forward, MemberId, Ref}: a registration forwarded by a follower; answered -%% with a {register_reply, Ref, _} cast to that member, -%% which then answers its own caller. -%% - {meta, From}: a set_metadata call made directly on this leader; -%% answered with gen_server:reply/2. -%% - {forward_meta, MemberId, Ref}: a set_metadata forwarded by a follower; answered -%% with a {set_meta_reply, Ref, _} cast to that member, -%% which then answers its own caller (after the FIFO-ordered -%% {names_batch} broadcast has updated the follower's row). -%% - {unreg, From}: an unregister_name call made directly on this leader; -%% answered `ok` with gen_server:reply/2 once committed. -%% - {forward_unreg, MemberId, Ref}: an unregister forwarded by a follower; answered -%% with an {unregister_reply, Ref} cast to that member, -%% which then answers its own caller `ok` (after the -%% FIFO-ordered {names_batch} broadcast). --type origin() :: - {local, gen_server:from()} - | {forward, dgen_registry_elector:member_id(), reference()} - | {meta, gen_server:from()} - | {forward_meta, dgen_registry_elector:member_id(), reference()} - | {unreg, gen_server:from()} - | {forward_unreg, dgen_registry_elector:member_id(), reference()}. - -%% A leader write op awaiting the next group commit. All ride one fenced commit (a -%% single version-key bump) and are applied to the local replica in arrival order, so -%% the batched outcome equals sequential one-at-a-time processing: -%% - add: a registration (with its metadata); the origin gets the deferred yes/no. -%% - set_meta: replace an existing registration's metadata; the origin ({meta, From}) -%% gets `ok`, or `{error, not_registered}` if the name is not bound. -%% - remove: an unregister (by name). ReleasedPid is the pid that was bound when -%% the unregister was enqueued (or `undefined` if the name was unbound), -%% captured before the leader removes it from the replica optimistically. -%% A non-`undefined` ReleasedPid both drives the durable clear and is -%% recorded in `recently_released` (the conflict-detector trail, §5.6). -%% Origin (or `undefined` for a legacy cast / salvaged re-drive) is -%% answered `ok` once the batch commits. -%% - down: an auto-unregister from a monitored process exit; Ref-guarded so a -%% stale DOWN for an already-re-registered name is ignored. The pid is -%% dead, so it is *not* recorded in `recently_released`. --type batch_op() :: - {add, LogicalName :: term(), Pid :: pid(), Meta :: meta(), Origin :: origin()} - | {set_meta, LogicalName :: term(), Index :: map(), Data :: term(), Origin :: origin()} - | {remove, LogicalName :: term(), ReleasedPid :: pid() | undefined, - Origin :: origin() | undefined} - | {retract, LogicalName :: term(), Pid :: pid()} - | {down, LogicalName :: term(), Ref :: reference()}. - -%% A follower's in-flight forwarded call, keyed in `pending_forwards` by the `Ref` the -%% leader echoes in its reply. One map holds all three kinds so there is a single home -%% for "calls I forwarded and am awaiting a verdict on", and a single resolution when -%% leadership moves. The kinds still differ in how they resolve — a register applies -%% (or gap-defers) the optimistic row on `yes`; a set_meta just answers; an unregister -%% is re-driven rather than rejected on a leadership change (see reject_forwards / -%% redrive_unregs): -%% - {register, Name, Pid, Meta, From}: awaiting {register_reply}; on `yes` the row is -%% written (making this member the second holder) and From -%% answered — version-guarded, see handle_register_reply. -%% - {set_meta, From}: awaiting {set_meta_reply}; From gets the leader's result. -%% - {unregister, Name, Pid, From}: awaiting {unregister_reply}; From gets `ok`. Pid is -%% the released pid (or `undefined` if unbound locally) so a -%% re-drive can pid-guard the retract. --type pending_forward() :: - {register, term(), pid(), meta(), gen_server:from()} - | {set_meta, gen_server:from()} - | {unregister, term(), pid() | undefined, gen_server:from()}. - --record(state, { - member_id :: dgen_registry_elector:member_id(), - %% The elector has no registered name (see dgen_registry's "Process identity" - %% moduledoc note); this is its pid, discovered once via the shared supervisor in - %% the `discover_elector` handle_continue right after init, and never re-resolved - %% afterward — the supervisor is `one_for_all`, so if the elector ever dies this - %% member is torn down and restarted alongside it (a fresh discovery on the next - %% start), never left holding a stale reference. `undefined` only for the brief - %% window between init/1 returning and discover_elector running. - elector :: pid() | undefined, - %% Backend handle and tuid, used by the leader to drive the fenced version-key - %% commit (a single key per registry — no per-name state) and to fence on the - %% leader key. See dgen_registry_names. - tenant :: dgen_backend:tenant(), - tuid :: dgen_server:tuid(), - leader :: dgen_registry_elector:member_id() | undefined, - %% Named, `protected` ETS table holding the local replica as `{Name, Pid, Index, - %% Data}` rows (pid + metadata). The member is its sole writer; any process reads it - %% lock-free (`dgen_registry:whereis_name/1`, `get_metadata/1`). Authoritative on the - %% leader; replicated snapshot on followers. Never written to durable storage — Pids - %% are ephemeral. On a handoff the freshest member's table is the sole reconstruction - %% source (no durable taken-set). - names_tab :: ets:tid() | atom(), - %% Inverted index over the rows' `Index` (queryable) metadata, kept in this member's - %% process state on *every* member (so `query/2` is a local read): `#{attr() => - %% #{value() => #{Name => []}}}` — attribute → value → set of names. Derived state, - %% maintained incrementally alongside every row write (see the `row_*` helpers) and - %% rebuilt wholesale on a handoff / snapshot apply. AND-equal queries intersect the - %% per-clause posting sets (§4.7 of the design doc). - inv_index = #{} :: #{term() => #{term() => #{term() => []}}}, - %% Leader-only presence state (§4.9): the durable subscription set the elector feeds - %% this member — `#{SubId => {Watch, Notify}}`. Seeded wholesale from the - %% `{elector_assume_and_distribute}` payload when this member assumes leadership, and - %% advanced by `{presence_update, …}` deltas the elector pushes as subscriptions are - %% created/removed. A follower does not track it (only the leader fires - %% notifications); it is reseeded from the elector on the next assume. - subs = #{} :: #{sub_id() => {query(), query()}}, - %% Leader-only: the current watch-set membership of each subscription — - %% `#{SubId => #{Name => Pid}}` — i.e. the names that satisfy the subscription's - %% watch query right now. Recomputed wholesale from the replica on assuming - %% leadership (recompute_sub_matches) and advanced incrementally per commit - %% (apply_committed_plan): comparing a batch's post-state matches against this - %% pre-state is what yields the `joined`/`left` deltas. - sub_matches = #{} :: #{sub_id() => #{term() => pid()}}, - %% Leader-only: the current *notify*-set membership of each subscription, the mirror - %% of `sub_matches` for the notify query — `#{SubId => #{Name => Pid}}`. Its purpose - %% is the initial snapshot: a process that only starts matching the notify query later - %% (registered after the subscription was created) must still learn who is already - %% present, so when a pid **enters** this set it is sent the full current watch - %% snapshot — symmetric to a watch member entering triggering a `joined` delta. - %% Recomputed on assume and advanced per commit alongside sub_matches. - notify_matches = #{} :: #{sub_id() => #{term() => pid()}}, - %% Peer-member monitors (all members) - members :: #{dgen_registry_elector:member_id() => reference()}, - monitors :: #{reference() => dgen_registry_elector:member_id()}, - %% Registered-process monitors (leader only) - %% The registered-process monitors (leader only): Name => monitor ref. There - %% is deliberately no reverse map — each monitor carries `{tag, {down, Name}}` - %% (erlang:monitor/3), so its DOWN arrives already naming the binding. - name_to_ref :: #{term() => reference()}, - %% Follower-only: calls forwarded to the leader and awaiting their reply, keyed by - %% the Ref the leader echoes back (`register` / `set_meta` / `unregister` kinds — - %% see the pending_forward() type). One map for all three, so there is a single - %% place forwarded-call state lives and a single resolution when leadership moves: - %% register and set_meta are rejected so their callers retry (reject_forwards); - %% unregister is instead re-driven, since its intent is idempotent and must not be - %% lost (redrive_unregs). - pending_forwards = #{} :: #{reference() => pending_forward()}, - %% Follower-only: forwarded registrations whose `yes` reply arrived while this - %% member's replica was **gapped** (the batch's {names_batch} broadcast was - %% refused pending a resync, so applied_version is behind the reply's commit - %% version). Acking `yes` at that point would make the "second holder" - %% invisible to the freshest-wins handoff gather (§5.5/§5.7) — a leader crash - %% before the resync landed could then silently drop an acked binding. The ack - %% is deferred as {Version, From} and released by flush_deferred/1 once - %% applied_version reaches Version (the resync snapshot both delivers the - %% binding and advances the version, making the second holder version-visible). - %% Rejected (`no`) on a leadership change, like the register forwards it came from. - deferred_yes = [] :: [{non_neg_integer(), gen_server:from()}], - %% Unregisters accepted while no leader was reachable (answered `ok` under the - %% CP contract — the commit must wait), kept as {Name, ReleasedPid} until - %% redrive_unregs/1 can hand them to a leader as pid-guarded retracts. (Distinct - %% from an `unregister` entry in `pending_forwards`, which was successfully - %% forwarded and is awaiting a reply; these never reached a leader at all.) - pending_unregs = [] :: [{term(), pid()}], - %% Registrations received while there was **no reachable leader** (none elected - %% yet, or the elected one is unreachable). Rather than answer a false `no` — which - %% OTP's via machinery reads as `already_started` — the register is *blocked*: its - %% caller is stashed here as {EnqueuedAtMs, Name, Pid, Meta, From} and re-driven - %% (redrive_registers/1) the moment a leader is established, so it succeeds if the - %% leader appears within the caller's `register_timeout` and otherwise the caller's - %% own call times out (§3 / §5.2). Pruned by wall-clock age on the periodic pass so - %% the stash cannot grow unbounded while no leader ever appears (a partition). - pending_registers = [] :: [{integer(), term(), pid(), meta(), gen_server:from()}], - %% Leader-only: direct (local-origin) registrations that have committed but are - %% awaiting follower replica confirmation before being acked `yes` (the two-holder - %% invariant, §5.5/§8). BatchRef => {Direct, TimerRef, Needed, Acked}, where - %% Direct is `[{Origin, Name, Pid}]`, Needed is the `register_replicas` target - %% (bounded by follower count), and Acked is the set of distinct followers that - %% confirmed. Answered `yes` once `map_size(Acked) >= Needed`, or on timeout per - %% the `strict_replication` policy (degrade-open `yes`, or fail-closed `no` + - %% retract). - pending_acks = #{} :: - #{ - reference() => - {[{origin(), term(), pid()}], reference(), non_neg_integer(), #{ - dgen_registry_elector:member_id() => true - }} - }, - %% Token used in our own {join, Self, Token} announcement. Refreshed on - %% each nodeup so any stale {member_down, Self, OldToken} in the queue is - %% discarded by the elector. - join_token :: reference(), - %% Tokens received from the elector via snapshots, keyed by peer MemberId. - %% Echoed back in {member_down, PeerId, Token} so the elector can distinguish - %% a stale DOWN (from before a re-join) from a fresh one. - peer_tokens :: #{dgen_registry_elector:member_id() => reference()}, - %% Leader-only group-commit buffer: write ops (adds + removes) awaiting their - %% batched commit, as a FIFO queue (oldest at the front, the commit order). An - %% add's From is the deferred caller (a local direct register or a forwarding - %% follower's call), answered via gen_server:reply/2 once the batch commits. - pending = queue:new() :: queue:queue(batch_op()), - %% Number of ops in `pending`, maintained incrementally so the batch size is - %% known in O(1) (no length/1 scan on the commit path). - num_pending = 0 :: non_neg_integer(), - %% Leader-only: the in-flight commit, or undefined when none is running. Only - %% one commit runs at a time (group commit): ops arriving while it is in flight - %% accumulate in `pending` and are committed in the next batch once it completes. - %% Holds the worker correlation ref, the monitor ref (so a worker that dies - %% without delivering a result does not wedge the commit lock), and the plan to - %% apply on success. - committing = undefined :: - undefined | #{ref := reference(), mref := reference(), plan := map()}, - %% Leader-only: the last committed version, pinned as the next commit's read - %% version to skip a GRV (cached GRV). A stale value is handled by the worker - %% retrying with a fresh GRV. Reset on losing leadership. - last_version = undefined :: undefined | integer(), - %% The highest commit version this member has applied to `names` — advanced by - %% every name broadcast (which carries its batch's commit version) and set from - %% the snapshot on a leadership change. Globally monotonic (FDB commit versions), - %% so on a handoff the new leader picks the member with the highest applied_version - %% as the freshest source for the gather. All members track it. - %% - %% Crucially, a broadcast advances it only when it is **contiguous**: each - %% broadcast carries its batch's predecessor version, and a follower that observes - %% a gap (it missed a batch — a dropped cast during a disconnect, a lost in-flight - %% message) does *not* apply past the gap; it requests a resync snapshot instead. - %% This is what makes "highest applied_version holds the freshest binding for - %% every name" actually true: every replica is a prefix of the leader's stream, - %% so freshest-wins reconstruction can never silently lose a binding a gathered - %% member holds, nor resurrect one it unregistered (§5.7 / Guarantee 4). - applied_version = 0 :: non_neg_integer(), - %% Timer for an outstanding resync request (a gap was observed and a snapshot was - %% requested from the broadcast's leader); `undefined` when none is outstanding. - %% Cleared when any snapshot arrives; on expiry the next gap re-requests. - resync_timer = undefined :: undefined | reference(), - %% Whether this member has ever synced registry state (applied a snapshot or - %% assumed leadership). Joins announce `fresh = not synced`; a fresh member holds - %% no bindings, so the leader's `degraded` computation ignores it when it is - %% unreachable at a gather (see dgen_registry_elector's member_info doc). On the - %% first sync the member re-announces itself with `fresh = false`. - synced = false :: boolean(), - %% Correlates the in-flight asynchronous handoff gather with the continuation that - %% finishes assuming leadership (§5.7). When the elector asks this member to assume - %% (a genuine leadership change), the peer-snapshot gather runs off the loop; this - %% ref tags that gather, and the `{assume_gathered, …}` continuation applies its - %% reconstruction *only* if this ref still matches — so a newer assume, or a snapshot - %% that made us a follower meanwhile, harmlessly supersedes a stale gather. - %% `undefined` when no assume is in flight. - assume_ref = undefined :: undefined | reference(), - %% Monotonically increasing leader term counter set by the elector. - %% Broadcasts from a prior leader carry a smaller epoch and are discarded. - epoch :: non_neg_integer(), - %% Conflict-detection trail (§5.6), kept on *every* member: name/pid pairs - %% explicitly released (an unregister of a live process) → the wall-clock ms it - %% happened. Keyed by `{Name, Pid}` — not bare pid — so releasing a pid under - %% one of its names never masks a genuine conflict on *another* name the same - %% pid also holds (a pid may be registered under several names). Bare-pid keys - %% from a pre-upgrade member's gathered trail are still honoured by the - %% detector during a rolling upgrade (see released_entry/3). - %% The leader records entries at commit; followers record them from the released - %% pid carried on {names_batch} unregister ops; and at a handoff gather every - %% reachable member's trail is merged into the new leader's, so the trail - %% survives leadership changes. A gather suppresses a divergence on an entry - %% still in here (it was legitimately unregistered, a lagging member just has not - %% caught up), so only a reconstruction-drop divergence — which leaves no trail — - %% is killed. TTL-pruned (dgen_config:conflict_release_ttl/1) by a periodic - %% self-message, with pruning suspended while any member is disconnected (a - %% partition must not outlive the trail that protects against its lagging rows). - recently_released = #{} :: #{{term(), pid()} | pid() => integer()}, - %% Leader-only kill budget (§5.6): name → recent kill timestamps (ms). Bounds - %% kills to `conflict_kill_budget` per name per window so a regenerating conflict - %% escalates to an operator instead of looping. - kill_budget = #{} :: #{term() => [integer()]}, - %% Leader-only: `true` when this leader's most recent handoff gather was - %% incomplete (a committed member did not respond — unreachable), so it may be - %% missing that member's bindings. Under `reject_when_degraded` the leader then - %% refuses to register a name it does not already hold (a potential re-issue, - %% §5.6). Recomputed on every handoff; default off otherwise. - degraded = false :: boolean(), - %% Per-registry tuning options (from dgen_registry:start_link/3), resolved through - %% dgen_config with application-env and built-in-default fallbacks. Scopes the - %% replication, termination, and degraded-reject knobs to this registry. - config = #{} :: dgen_config:config(), - %% Wall-clock ms of the most recent member departure this member observed - %% (remove_member/2). Trail pruning (§5.6) is suspended not only while a - %% *current* member is disconnected but also within `conflict_release_ttl` of - %% any departure: a member dropped from the set during a partition is no - %% longer "a disconnected member", yet its stale rows can surface at a rejoin - %% gather — the trail that discriminates lag from divergence must outlive - %% that window too. (A member that stays gone longer than the TTL is outside - %% the trail's protection horizon either way.) - last_departure = 0 :: integer() -}). - -%% --------------------------------------------------------------------------- -%% Public API -%% --------------------------------------------------------------------------- - --if(?DOCATTRS). --doc "Starts the member process registered as `Name`.". --endif. --spec start_link(Name :: atom(), Args :: map()) -> gen_statem:start_ret(). -start_link(Name, Args) -> - gen_statem:start_link({local, Name}, ?MODULE, Args, []). - -%% --------------------------------------------------------------------------- -%% gen_server callbacks -%% --------------------------------------------------------------------------- - -callback_mode() -> - [handle_event_function, state_enter]. - -init(#{name := Name, tenant := Tenant, tuid := Tuid} = Args) -> - Config = maps:get(config, Args, #{}), - MemberId = {node(), Name}, - %% Create the local names replica as a named, `protected` ETS table owned by this - %% process: this member is the sole writer, callers read it lock-free from their - %% own process (`dgen_registry:whereis_name/1`). `read_concurrency` suits the - %% read-mostly access pattern. Recreated empty on every (re)start — no heir — so a - %% crash drops the local replica and it is rebuilt from the leader's snapshot. - NamesTab = ets:new(dgen_registry:names_table(Name), [ - named_table, set, protected, {read_concurrency, true} - ]), - %% Subscribe to node up/down: the member reacts to `{nodeup, _}` by re-announcing - %% its own `{join}` (partition-heal, membership identity). The node-level mesh / - %% reap / probe machinery lives in the sibling `dgen_registry_connector`, which - %% subscribes independently — this member no longer meshes or reaps. - net_kernel:monitor_nodes(true), - %% Resolve the telemetry lookup here rather than lazily on the first event. It - %% is cached per VM either way (see `telemetry_available/0`), so this only moves - %% *when* the one uncached call happens -- and where it happens is the point. - %% Uncached, it is a synchronous call into `code_server`; lazily, it lands inside - %% whichever member emits the first event, which under simulation is a scheduled - %% process blocking on a process the scheduler does not own. Everything after - %% that is ordered by wall clock, and nothing reports it: `code:ensure_loaded/1` - %% on a *missing* module loads nothing, so the "no module loaded mid-run" audit - %% stays clean while the leak is wide open. Here it runs while the tree is - %% starting, which under `eta_run` is before a scheduler exists at all. - _ = telemetry_available(), - %% Periodic maintenance: expire the conflict-detection trail (§5.6) and the - %% per-name kill-budget timestamps. - arm(?PRUNE_INTERVAL, prune_released), - %% Periodic replication heartbeat (leader only, see the define): lets a follower - %% that lost the tail of the broadcast stream discover it and resync, rather than - %% sitting diverged until the next write. - arm(?REPLICA_HEARTBEAT_INTERVAL, replica_heartbeat), - %% Announcing presence needs the elector's pid, which is deferred to - %% `discover_elector` below: the supervisor is still synchronously waiting on *this* - %% init/1 to return when it runs, so `supervisor:which_children/1` would deadlock if - %% called from here directly. Queuing the internal event (gen_statem's replacement - %% for `{continue, …}`) lets the supervisor consider this child started, - %% unblocking it, before the lookup runs. - {ok, searching, - #state{ - member_id = MemberId, - elector = undefined, - tenant = Tenant, - tuid = Tuid, - leader = undefined, - names_tab = NamesTab, - members = #{}, - monitors = #{}, - name_to_ref = #{}, - pending_forwards = #{}, - deferred_yes = [], - pending_unregs = [], - pending_registers = [], - pending_acks = #{}, - join_token = make_ref(), - peer_tokens = #{}, - pending = queue:new(), - num_pending = 0, - committing = undefined, - last_version = undefined, - applied_version = 0, - epoch = 0, - recently_released = #{}, - kill_budget = #{}, - degraded = false, - config = Config, - last_departure = 0 - }, - %% gen_statem has no handle_continue; the same "run after the supervisor - %% unblocks" contract is an internal event queued from init's actions. - [{next_event, internal, discover_elector}]}. - -%% --------------------------------------------------------------------------- -%% handle_event/4 — dispatch, and the derived state -%% --------------------------------------------------------------------------- - -%% Phase 2 of the statem port: the statem state is a DERIVED projection of the -%% role fields in Data — `state_of/1` over leader / member_id / assume_ref — -%% recomputed after every event by `next/2`. Data stays the single source of -%% truth (dozens of body reads want the leader's identity, not just the role), -%% the projection cannot drift from it, and every role change is a genuine -%% gen_statem transition — the seam phase 3's enter calls hang off. -%% -%% The state-matched clauses below are the hoisted dispatch: events whose old -%% gen_server heads matched leadership now match the state itself, with the -%% original Data guards kept where they say something the role alone does not -%% (an epoch, a ref). Everything else falls through to the generic shim at the -%% bottom, whose handlers' own leadership branches remain authoritative — the -%% hoist changes where dispatch is visible, never what it decides. - -%% -- state enter: the leadership lifecycle, in one place --------------------- - -%% Phase 3: the assume/relinquish pairing lives on the role edges themselves, -%% so no transition path — assume continuation, snapshot demotion, epoch nudge, -%% or any added later — can forget half of it. Entering `leader` seeds the -%% subscription matches and arms the deferred monitor sweep (assume_leadership); -%% leaving `leader` demonitors every registered pid (relinquish_leadership). -%% What deliberately does NOT live here: reject_forwards fires on any change of -%% leader IDENTITY, including follower→follower, which is not a state edge; and -%% fail_pending_acks stays with do_leader_changed's lost-leadership branch — -%% a re-assumed leader's stale pending acks are already fenced by their -%% BatchRef, and resolving them early would change behavior, not just shape. -%% -%% One timing shift, deliberate and safe: a leader entering `assuming` now -%% relinquishes at assume-start rather than at the gather continuation, so a -%% subject dying during the gather window is unmonitored until the new term's -%% sweep — which catches it with an immediate `noproc` DOWN, the same -%% self-healing property the deferred sweep already rests on. -handle_event(enter, OldState, leader, Data) when OldState =/= leader -> - {keep_state, assume_leadership(Data)}; -handle_event(enter, leader, NewState, Data) when NewState =/= leader -> - {keep_state, relinquish_leadership(Data)}; -handle_event(enter, _OldState, _NewState, Data) -> - %% Includes init's self-enter (searching -> searching) and every edge not - %% touching `leader`. - {keep_state, Data}; -%% -- the assume lifecycle --------------------------------------------------- - -%% A continuing leader absorbs a join on the fast path; every other -%% configuration runs the genuine leadership change. The old heads' epoch and -%% synced guards stay on Data — the `leader` state says who we are, not which -%% term we are in. -handle_event( - {call, From}, - {elector_assume_and_distribute, MemberId, AllIds, Tokens, FreshIds, Epoch}, - leader, - Data = #state{epoch = Epoch, synced = true} -) when MemberId =/= undefined -> - {reply, Reply, Data1} = - assume_fast_path(MemberId, AllIds, Tokens, FreshIds, Epoch, Data), - next(Data1, [{reply, From, Reply}]); -handle_event( - {call, From}, - {elector_assume_and_distribute, MemberId, AllIds, Tokens, FreshIds, Epoch}, - _AnyState, - Data -) -> - {reply, Reply, Data1} = - assume_genuine(MemberId, AllIds, Tokens, FreshIds, Epoch, Data), - next(Data1, [{reply, From, Reply}]); -%% The handoff gather's continuation lands only in `assuming`, and only for the -%% assume that armed it (the ref) — anywhere else it was superseded by a newer -%% assume or by a snapshot that made us a follower, and is dropped where the -%% mismatch is visible. -handle_event( - info, - {assume_gathered, {_, _, _, _, _, Ref}, _, _, _, _, _} = Msg, - assuming, - Data = #state{assume_ref = Ref} -) -> - {noreply, Data1} = handle_info(Msg, Data), - next(Data1, []); -handle_event( - info, {assume_gathered, _Ctx, _Self, _Freshest, _MaxV, _Peers, _Subs}, _AnyState, Data -) -> - {keep_state, Data}; -%% The joiner-gather continuation is leader work in the current term; the -%% member-still-known guard stays on Data. -handle_event( - info, - {joiner_gathered, MemberId, _AllIds, _Tokens, Epoch, _Result} = Msg, - leader, - Data = #state{epoch = Epoch, members = Members} -) when is_map_key(MemberId, Members) -> - {noreply, Data1} = handle_info(Msg, Data), - next(Data1, []); -handle_event(info, {joiner_gathered, _M, _A, _T, _E, _R}, _AnyState, Data) -> - {keep_state, Data}; -%% -- leader-only periodic work --------------------------------------------- - -%% Only the leader advertises; everyone keeps the timer running so a later -%% election heartbeats immediately. -handle_event(info, replica_heartbeat, leader, Data) -> - broadcast_heartbeat(Data), - arm(?REPLICA_HEARTBEAT_INTERVAL, replica_heartbeat), - {keep_state, Data}; -handle_event(info, replica_heartbeat, _AnyState, Data) -> - arm(?REPLICA_HEARTBEAT_INTERVAL, replica_heartbeat), - {keep_state, Data}; -%% The deferred monitor-establishment sweep is leader work in the term that -%% armed it; a stale step (deposed, or a newer assume) strands here. -handle_event(info, {monitor_sweep, Epoch, _Cont} = Msg, leader, Data = #state{epoch = Epoch}) -> - {noreply, Data1} = handle_info(Msg, Data), - next(Data1, []); -handle_event(info, {monitor_sweep, _Epoch, _Cont}, _AnyState, Data) -> - {keep_state, Data}; -%% -- leader-only protocol serves ------------------------------------------- - -%% A follower's resync request: only a member that believes it leads answers -%% (the requester's own state makes a stale serve harmless — see the handler). -handle_event(cast, {resync_req, _FollowerId} = Msg, leader, Data) -> - {noreply, Data1} = handle_cast(Msg, Data), - next(Data1, []); -handle_event(cast, {resync_req, _FollowerId}, _AnyState, Data) -> - {keep_state, Data}; -%% A forwarded unregister: the leader commits it; a non-leader stays silent (the -%% follower keeps the removal stashed and re-drives it at the next leader). -handle_event(cast, {unregister_req, _Ref, _FollowerId, _Name} = Msg, leader, Data) -> - {noreply, Data1} = handle_cast(Msg, Data), - next(Data1, []); -handle_event(cast, {unregister_req, _Ref, _FollowerId, _Name}, _AnyState, Data) -> - {keep_state, Data}; -%% -- the generic shim ------------------------------------------------------- - -handle_event({call, From}, Msg, _State, Data) -> - case handle_call(Msg, From, Data) of - {reply, Reply, Data1} -> next(Data1, [{reply, From, Reply}]); - {noreply, Data1} -> next(Data1, []) - end; -handle_event(cast, Msg, _State, Data) -> - {noreply, Data1} = handle_cast(Msg, Data), - next(Data1, []); -handle_event(info, Msg, _State, Data) -> - {noreply, Data1} = handle_info(Msg, Data), - next(Data1, []); -handle_event(internal, Continue, _State, Data) -> - {noreply, Data1} = handle_continue(Continue, Data), - next(Data1, []). - -%% The statem state, derived. `assuming` wins over the leader/follower read -%% because a mid-assume member has cleared its leader view on purpose; a member -%% with no leader and no assume in flight is searching. The follower/gapped -%% split arrives in phase 3 with the resync-ownership move. -state_of(#state{assume_ref = Ref}) when Ref =/= undefined -> assuming; -state_of(#state{leader = undefined}) -> searching; -state_of(#state{leader = Self, member_id = Self}) -> leader; -state_of(#state{}) -> follower. - -%% `{next_state, SameState, …}` is not a state change to gen_statem (same-state -%% transitions keep timeouts and never fire enter calls), so deriving on every -%% event is free of behavioral effect until a role actually moves. -next(Data, Actions) -> - {next_state, state_of(Data), Data, Actions}. - -%% --------------------------------------------------------------------------- -%% handle_continue/2 -%% --------------------------------------------------------------------------- - -%% Finds the elector via the shared supervisor (see the moduledoc's "Process -%% identity" note in dgen_registry) — safe here because the supervisor is no -%% longer blocked on this process's own init/1 by the time this runs — then -%% announces presence. The mesh/probe startup lives in the connector. -handle_continue( - discover_elector, State = #state{member_id = {_Node, Name} = MemberId, join_token = Token} -) -> - Elector = dgen_registry:elector_pid(Name), - true = is_pid(Elector), - %% Announce presence; the elector will call {elector_assume_and_distribute} - %% on the new leader, which then sends {apply_names_snapshot} to this member. - %% `fresh = true`: this member has never synced (empty replica), so an - %% unreachable-at-gather window for it cannot be hiding any bindings. - dgen_server:cast(Elector, {join, MemberId, Token, true}), - {noreply, State#state{elector = Elector}}. - -%% --------------------------------------------------------------------------- -%% handle_call/3 -%% --------------------------------------------------------------------------- - -%% Readiness probe (see dgen_registry:await_ready/2). This member can serve -%% registrations once it knows a leader *and* has synced registry state — it has -%% applied a snapshot (follower) or assumed leadership itself. Before that a -%% register would be fast-rejected (`no`, no leader) or read an empty replica. -%% Side-effect free: a plain state read, no name is touched. -handle_call(ready, _From, State = #state{leader = Leader, synced = Synced}) -> - {reply, Leader =/= undefined andalso Synced, State}; -%% Observability probe (see dgen_registry:status/1). Reports this member's own -%% *belief* about leadership — which is distinct from the elector's committed view -%% (`get_leader/1`), and the gap between them is exactly the handoff window a -%% deposed leader has not yet heard about (§5.1). Side-effect free, and returns a -%% map so callers never depend on the #state{} record's field order. -handle_call(status, _From, State) -> - #state{ - member_id = Self, - leader = Leader, - epoch = Epoch, - synced = Synced, - applied_version = AppliedVersion - } = State, - {reply, - #{ - member_id => Self, - leader => Leader, - is_leader => Leader =:= Self, - epoch => Epoch, - synced => Synced, - applied_version => AppliedVersion - }, - State}; -%% ---- Name registration ---------------------------------------------------- - -%% All routing (leader/follower/no-leader) is handled by route_register/5, which -%% either defers the reply (leader parks it in the commit buffer; a follower forwards -%% it; a not-ready member stashes it to await a leader) or answers immediately for an -%% adjudicated verdict. The only immediate reply is `no`, and only when the name is -%% genuinely refused — see route_register/5. See the moduledoc "blocks until a leader -%% exists" note. -handle_call({register, LogicalName, Pid, Meta}, From, State) -> - {noreply, route_register(LogicalName, Pid, Meta, From, State)}; -%% ---- Name unregistration (leader-routed call, mirroring the register path) -- - -handle_call({unregister, LogicalName}, From, State) -> - route_unregister(LogicalName, From, State); -%% ---- Metadata write (set_metadata) ---------------------------------------- - -%% Leader: park the metadata replacement in the group-commit buffer; the origin -%% {meta, From} gets `ok` (or {error, not_registered}) once the batch commits. -handle_call( - {set_metadata, LogicalName, Index, Data}, - From, - State = #state{leader = Leader, member_id = Leader} -) -> - {noreply, enqueue_op({set_meta, LogicalName, Index, Data, {meta, From}}, State)}; -%% Follower: forward to the leader asynchronously, stashing From under a Ref. The -%% leader answers via a {set_meta_reply} cast routed back through this member, so the -%% leader's {names_batch} broadcast (FIFO, ahead of the reply) has updated our row -%% before we answer the caller — read-after-write on this node's snapshot read. -handle_call( - {set_metadata, LogicalName, Index, Data}, - From, - State = #state{leader = Leader, member_id = Self, pending_forwards = PF} -) when Leader =/= undefined, Leader =/= Self -> - %% Same unreachable-leader fast-fail as the register forward above. - case member_reachable(Leader) of - true -> - Ref = make_ref(), - cast_to_member(Leader, {set_meta_req, Ref, Self, LogicalName, Index, Data}), - {noreply, State#state{pending_forwards = PF#{Ref => {set_meta, From}}}}; - false -> - {reply, {error, no_leader}, State} - end; -%% No leader yet. -handle_call({set_metadata, _LogicalName, _Index, _Data}, _From, State) -> - {reply, {error, no_leader}, State}; -%% ---- Consistent metadata read (leader only) -------------------------------- - -handle_call( - {get_metadata, LogicalName}, - From, - State = #state{leader = Leader, member_id = Leader, names_tab = Tab} -) -> - %% Consistent reads are fenced (Guarantee 5): the row is captured here (at the - %% mailbox point) and the reply is sent only after a spawned helper verifies our - %% leadership against the durable leader key — a deposed leader that has not yet - %% heard about the handoff cannot serve a stale answer as authoritative. - reply_fenced(From, lookup_metadata(Tab, LogicalName), undefined, State), - {noreply, State}; -handle_call( - {get_metadata, LogicalName}, - From, - State = #state{leader = Leader} -) when Leader =/= undefined -> - %% Forward to the leader, which answers `From` directly from its authoritative row - %% (same non-blocking forward as {whereis}). Unreachable leader → answer now. - case member_reachable(Leader) of - true -> - cast_to_member(Leader, {get_metadata_req, LogicalName, From}), - {noreply, State}; - false -> - {reply, undefined, State} - end; -handle_call({get_metadata, _LogicalName}, _From, State) -> - {reply, undefined, State}; -%% ---- Indexed query (runs on the member's mailbox for batch-consistency, §6) - - -%% Snapshot query: run against this member's own index + table. Running it in the -%% handler (on the mailbox) means it observes the local replica at a batch boundary, -%% never a half-applied group commit — a batch-consistent snapshot. -handle_call( - {query, Constraints}, _From, State = #state{names_tab = Tab, inv_index = Inv} -) -> - {reply, run_query(Constraints, Tab, Inv), State}; -%% Consistent query: must run on the leader. If we are the leader, run it here -%% (on the mailbox — batch-consistent) and fence the reply like the other -%% consistent reads; otherwise forward (non-blocking) and let the leader answer -%% the caller directly. -handle_call( - {query_consistent, Constraints}, - From, - State = #state{leader = Leader, member_id = Leader, names_tab = Tab, inv_index = Inv} -) -> - reply_fenced(From, run_query(Constraints, Tab, Inv), [], State), - {noreply, State}; -handle_call( - {query_consistent, Constraints}, From, State = #state{leader = Leader} -) when Leader =/= undefined -> - case member_reachable(Leader) of - true -> - cast_to_member(Leader, {query_req, Constraints, From}), - {noreply, State}; - false -> - {reply, [], State} - end; -handle_call({query_consistent, _Constraints}, _From, State) -> - {reply, [], State}; -%% ---- Consistent read (leader only) ---------------------------------------- - -handle_call( - {whereis, LogicalName}, - From, - State = #state{leader = Leader, member_id = Leader} -) -> - %% Fenced consistent read — see the {get_metadata} leader clause. - reply_fenced(From, lookup_name(State#state.names_tab, LogicalName), undefined, State), - {noreply, State}; -handle_call( - {whereis, LogicalName}, - From, - State = #state{leader = Leader} -) when Leader =/= undefined -> - %% Forward the caller's `From` to the leader, which replies to it directly — the - %% follower never blocks its loop on a `gen_server:call` to the leader, so a - %% consistent read can no longer head-of-line block the follower's other messages. - %% If the leader's node is unreachable, answer now rather than hang the caller. - case member_reachable(Leader) of - true -> - cast_to_member(Leader, {whereis_req, LogicalName, From}), - {noreply, State}; - false -> - {reply, undefined, State} - end; -handle_call({whereis, _LogicalName}, _From, State) -> - {reply, undefined, State}; -%% ---- Snapshot read --------------------------------------------------------- -%% Served lock-free in the calling process via `ets:lookup` on the member's -%% `protected` names table (`dgen_registry:whereis_name/1`); no handler here. - -%% ---- Handoff gather (any member) ------------------------------------------- - -%% Return our current records map (`#{Name => {Pid, Index, Data}}`), the version we -%% have applied up to, and our copy of the released-pid trail, so a newly-assuming -%% leader can reconstruct the bindings *and their metadata* from the freshest of -%% every reachable member (the cross-member gather, §5.7 / §4.4) and merge the -%% conflict-detector trails (§5.6) so a legitimate unregister is not mistaken for a -%% divergence just because leadership moved. -handle_call( - get_names_snapshot, - _From, - State = #state{names_tab = Tab, applied_version = Version, recently_released = Released} -) -> - %% Ship the (potentially huge) replica as an off-heap binary, folded straight - %% off the table — see encode_table/1. The gathering leader decodes it. - {reply, {encode_table(Tab), Version, Released}, State}; -handle_call(_Request, _From, State) -> - {reply, {error, unknown_call}, State}. - -%% ---- Elector calls (dispatched from handle_event; during lock period) ------ - -%% Called by the elector to atomically assume leadership and fan out the -%% names snapshot to all followers. Dispatch — fast path in the `leader` state -%% for the same epoch, genuine change everywhere else — lives on the -%% handle_event clauses; these are their bodies. -%% -%% The bindings are reconstructed entirely from the freshest of every reachable -%% member's names map (`gather_maps/3`) — there is no durable name state to read -%% (the DB holds only a fence/version key, §4.4). The old leader, if alive, is one of -%% those members and relinquishes when it later receives `{apply_names_snapshot}`; -%% the handoff window is fenced (its commits conflict on the leader key), so no -%% synchronous relinquish-and-transfer step is needed. -%% -%% `MemberId` is the newly joining member that triggered this transition, or -%% `undefined` for a `member_down` event. -%% -%% `AllIds` is the full current member list. -%% -%% After updating own state the leader sends `{apply_names_snapshot}` casts -%% to every follower from its own process (maintaining FIFO ordering with -%% subsequent `{names_batch}` broadcasts). -%% Fast path — a *continuing* leader (already leader for this exact epoch, holding -%% synced state) handling a member's join. Its own replica is authoritative: every -%% follower is a prefix of its broadcast stream, so no member can hold a fresher -%% binding and there is nothing to gather. So skip the O(members) gather, the -%% wholesale replica rebuild, and the leadership re-assumption (which would demonitor -%% and re-monitor every registered pid) — and, crucially, do NOT re-snapshot the -%% already-synced followers. Just monitor the joiner, ship *it* the current snapshot -%% (inline, so it precedes any later {names_batch} broadcast — FIFO ordering), -%% and tell the existing followers to monitor the joiner too via a small token-only -%% message. The full gather+distribute below is reserved for a genuine leadership -%% change, where a new leader must reconstruct from the freshest surviving member. -assume_fast_path(MemberId, AllIds, Tokens, FreshIds, Epoch, State = #state{member_id = Self}) -> - State1 = merge_peer_tokens( - Tokens, add_member_monitors(extra_member_ids(MemberId, AllIds, Self), State) - ), - %% A joiner that has never synced (fresh — a brand-new member, zero cost to - %% skip), or our own no-op re-join, holds no bindings: onboard immediately. - %% A **rejoining** previously-synced member, however, may hold divergent live - %% bindings from before it dropped out of the member set — the §5.6 - %% partition-heal case, which under a *continuing* leader never passes through - %% the full handoff gather. Gather its replica first (off this loop, bounded - %% by ?GATHER_TIMEOUT) and adjudicate it against our authoritative table in - %% the {joiner_gathered} continuation before onboarding, so a name reissued - %% while the member was away is repaired by termination rather than silently - %% overwritten with the old claimant left running. Detection is best-effort: - %% an unanswerable joiner is onboarded anyway (same posture as the full - %% gather skipping an unreachable member). - case MemberId =:= Self orelse lists:member(MemberId, FreshIds) of - true -> - {reply, ok, onboard_joiner(MemberId, AllIds, Tokens, State1)}; - false -> - spawn_joiner_gather(MemberId, AllIds, Tokens, Epoch, self()), - {reply, ok, State1} - end. - -assume_genuine(MemberId, AllIds, Tokens, FreshIds, Epoch, State = #state{member_id = Self}) -> - %% Genuine leadership change (a fresh leader, or a member_down that moved - %% leadership). The reconstruction gathers the freshest of every reachable member's - %% replica — a **network** fan-out bounded by ?GATHER_TIMEOUT — so it runs **off this - %% loop**: the member keeps draining messages during the wait, and the elector's call - %% is answered `ok` immediately (an inline multi-second gather would time the - %% elector's call out and stall any membership change queued behind it, the churn - %% collapse). A *consistent* snapshot of our own replica and applied_version is - %% captured now (so a broadcast landing during the gather cannot skew freshest-wins), - %% and the `{assume_gathered, …}` continuation finishes assuming when the gather - %% returns — iff `assume_ref` still matches (a newer assume, or a snapshot making us a - %% follower, harmlessly supersedes it). Until then we are not yet leader/synced, so - %% writes block (no reachable leader) and are re-driven by the continuation, never - %% served against a half-built map. - Tab = State#state.names_tab, - SelfRecords = current_records(Tab), - OtherIds = lists:delete(Self, AllIds), - Ref = make_ref(), - spawn_assume_gather( - {MemberId, AllIds, Tokens, FreshIds, Epoch, Ref}, - SelfRecords, - State#state.applied_version, - OtherIds, - State#state.elector, - self(), - State#state.tenant, - State#state.tuid - ), - %% Clear the leader for the window: we are mid-handoff with no stable leader, so - %% writes block (no reachable leader) and are re-driven by the continuation — rather - %% than forwarding to the old, possibly-dead leader. The continuation sets - %% leader=Self. `synced` is left as-is; the continuation's mark_synced handles it. - {reply, ok, State#state{assume_ref = Ref, leader = undefined}}. - -%% --------------------------------------------------------------------------- -%% handle_cast/2 -%% --------------------------------------------------------------------------- - -%% Leader: a registration forwarded by a follower. Park it as an add whose origin -%% is the forwarding member, answered via {register_reply} once committed. -handle_cast( - {register_req, Ref, FollowerId, LogicalName, Pid, Meta}, - State = #state{leader = Leader, member_id = Leader} -) -> - case reject_new_when_degraded(LogicalName, State) of - true -> - cast_to_member(FollowerId, {register_reply, Ref, no}), - {noreply, State}; - false -> - {noreply, enqueue_op({add, LogicalName, Pid, Meta, {forward, FollowerId, Ref}}, State)} - end; -%% Not the leader (race: the follower's leader belief was stale) — reject so the -%% follower's caller retries against the current leader. -handle_cast({register_req, Ref, FollowerId, _LogicalName, _Pid, _Meta}, State) -> - cast_to_member(FollowerId, {register_reply, Ref, no}), - {noreply, State}; -%% Leader: an unregister forwarded by a follower. Same shape as the direct -%% unregister call: capture our (authoritative) bound pid, optimistic delete, and -%% park the removal; the origin is answered via {unregister_reply} once committed. -handle_cast( - {unregister_req, Ref, FollowerId, LogicalName}, - State = #state{leader = Leader, member_id = Leader, names_tab = Tab} -) -> - ReleasedPid = lookup_name(Tab, LogicalName), - {noreply, - enqueue_op( - {remove, LogicalName, ReleasedPid, {forward_unreg, FollowerId, Ref}}, - row_delete(State, LogicalName) - )}; -%% Follower: the leader committed a forwarded unregister — answer the caller `ok`. -%% FIFO puts this behind the batch's {names_batch} broadcast, so this member's -%% row state reflects the removal by the time the caller sees the reply. -handle_cast({unregister_reply, Ref}, State = #state{pending_forwards = PF}) -> - case maps:take(Ref, PF) of - {{unregister, _Name, _ReleasedPid, From}, PF1} -> - gen_server:reply(From, ok), - {noreply, State#state{pending_forwards = PF1}}; - _ -> - %% Unknown Ref — already answered (e.g. re-driven on a leadership change). - {noreply, State} - end; -%% Leader: a set_metadata forwarded by a follower. Park it as a set_meta op whose -%% origin is the forwarding member, answered via {set_meta_reply} once committed. -handle_cast( - {set_meta_req, Ref, FollowerId, LogicalName, Index, Data}, - State = #state{leader = Leader, member_id = Leader} -) -> - {noreply, - enqueue_op({set_meta, LogicalName, Index, Data, {forward_meta, FollowerId, Ref}}, State)}; -%% Not the leader (stale belief) — reject so the follower's caller retries. -handle_cast({set_meta_req, Ref, FollowerId, _LogicalName, _Index, _Data}, State) -> - cast_to_member(FollowerId, {set_meta_reply, Ref, {error, no_leader}}), - {noreply, State}; -%% Follower: the leader's answer to a forwarded set_metadata. By now the matching -%% {names_batch} broadcast (FIFO ahead of this) has updated our row, so answering the -%% caller here preserves read-after-write for a subsequent local get_metadata. -handle_cast({set_meta_reply, Ref, Result}, State = #state{pending_forwards = PF}) -> - case maps:take(Ref, PF) of - {{set_meta, From}, PF1} -> - gen_server:reply(From, Result), - {noreply, State#state{pending_forwards = PF1}}; - _ -> - %% Unknown Ref — already answered (e.g. rejected on a leadership change). - {noreply, State} - end; -%% Leader: a consistent metadata read forwarded by a follower. Reply to the original -%% caller's `From` directly from the authoritative row — fenced, like every -%% consistent read (see reply_fenced). -handle_cast( - {get_metadata_req, LogicalName, From}, - State = #state{leader = Leader, member_id = Leader, names_tab = Tab} -) -> - reply_fenced(From, lookup_metadata(Tab, LogicalName), undefined, State), - {noreply, State}; -handle_cast({get_metadata_req, _LogicalName, From}, State) -> - gen_server:reply(From, undefined), - {noreply, State}; -%% Leader: a consistent query forwarded by a follower. Run it on our mailbox (between -%% batches → batch-consistent) and answer the original caller's `From` directly — -%% fenced, like every consistent read. -handle_cast( - {query_req, Constraints, From}, - State = #state{leader = Leader, member_id = Leader, names_tab = Tab, inv_index = Inv} -) -> - reply_fenced(From, run_query(Constraints, Tab, Inv), [], State), - {noreply, State}; -handle_cast({query_req, _Constraints, From}, State) -> - gen_server:reply(From, []), - {noreply, State}; -%% Leader: a consistent read forwarded by a follower (whereis_name_consistent). Reply -%% to the original caller's `From` directly from the authoritative map — fenced, -%% like every consistent read. -handle_cast( - {whereis_req, LogicalName, From}, - State = #state{leader = Leader, member_id = Leader, names_tab = Tab} -) -> - reply_fenced(From, lookup_name(Tab, LogicalName), undefined, State), - {noreply, State}; -%% Not the leader (stale belief, raced a leadership change) — answer `undefined`; the -%% caller can retry and reach the current leader. -handle_cast({whereis_req, _LogicalName, From}, State) -> - gen_server:reply(From, undefined), - {noreply, State}; -%% Follower: the leader's answer to a forwarded registration. The committed form -%% carries the batch's commit Version so the ack can be **version-guarded** (§5.5): -%% a `yes` is answered only when this member's replica has genuinely applied up to -%% that version — normally already true, because the batch's {names_batch} -%% broadcast precedes the reply (FIFO) — making the forwarding follower a -%% version-visible second holder the handoff gather (§5.7) is guaranteed to see. -%% If the broadcast was gap-refused (this member missed an earlier batch and is -%% awaiting a resync), the ack is deferred until the resync lands rather than -%% answered against a replica whose extra row freshest-wins reconstruction would -%% ignore — the fix for the silent-loss window a single leader crash could open. -handle_cast({register_reply, Ref, Result, Version}, State) -> - {noreply, handle_register_reply(Ref, Result, Version, State)}; -%% Legacy 3-tuple reply (a pre-version leader during a rolling upgrade): no version -%% to guard on — apply the old immediate behaviour. Marked `legacy` rather than -%% version 0, because "0 is at-or-behind everything" put it on the same branch as a -%% guarded reply, and those two must now do different things (see -%% handle_register_reply/4). -handle_cast({register_reply, Ref, Result}, State) -> - {noreply, handle_register_reply(Ref, Result, legacy, State)}; -%% Follower: the leader asks us to confirm we hold a batch's bindings. This cast -%% arrives (FIFO) after the batch's {names_batch} cast, so normally we have -%% applied them and hold the bindings — ack back to the leader. The ack is -%% version-guarded: if we have *not* applied up to the batch's version (we observed -%% a gap and are awaiting a resync, or the broadcasts were stale-epoch-dropped), we -%% do not hold the bindings and must not claim to — the leader falls back to other -%% followers' acks or its timeout policy. -handle_cast( - {replicate_sync, BatchRef, LeaderId, Version}, - State = #state{member_id = Self, applied_version = Applied} -) -> - case Applied >= Version of - true -> cast_to_member(LeaderId, {replicate_ack, BatchRef, Self}); - false -> ok - end, - {noreply, State}; -%% Leader: a follower confirms it holds a batch's bindings. Count *distinct* -%% followers (a duplicate ack from the same one does not count); once -%% `register_replicas` of them have confirmed, the two-holder (or n-holder) invariant -%% is met for that batch's direct registrations — answer them `yes`. -handle_cast({replicate_ack, BatchRef, FollowerId}, State = #state{pending_acks = PA}) -> - case maps:get(BatchRef, PA, undefined) of - {Direct, TimerRef, Needed, Acked} -> - Acked1 = Acked#{FollowerId => true}, - case map_size(Acked1) >= Needed of - true -> - erlang:cancel_timer(TimerRef), - deliver_direct(Direct), - {noreply, State#state{pending_acks = maps:remove(BatchRef, PA)}}; - false -> - {noreply, State#state{ - pending_acks = PA#{BatchRef => {Direct, TimerRef, Needed, Acked1}} - }} - end; - undefined -> - %% Already answered (enough acks or the timeout) — ignore. - {noreply, State} - end; -%% Replication broadcast. One message per committed batch, carrying `{Ops, Epoch, -%% PrevVersion, Version, LeaderId}`: the batch's ops in commit order, the leader's -%% epoch, the commit version of the batch *before* this one (the leader's -%% applied_version when the batch was applied), this batch's commit version, and the -%% sender. `apply_bcast/6` applies the batch only when it is contiguous with our -%% replica (PrevVersion matches ours); a gap means we missed a batch — we stop -%% applying and request a resync snapshot instead, so our replica always remains a -%% *prefix* of the leader's stream (see the applied_version field doc). -%% -%% The batch applies as a unit, which is what keeps "same applied_version implies -%% same content" true — see broadcast_batch/5 for why per-name messages could not. -handle_cast({names_batch, Ops, Epoch, PrevV, Version, LeaderId}, State) -> - Apply = fun(S) -> lists:foldl(fun apply_bcast_op/2, S, Ops) end, - {noreply, apply_bcast(Epoch, PrevV, Version, LeaderId, Apply, State)}; -%% Follower: a member that observed a gap asks the leader for a fresh snapshot. -%% Answer with the same {apply_names_snapshot} shape the handoff fan-out uses — -%% empty extra-member and token payloads (they merge as no-ops) — carrying our -%% current epoch and applied version, which re-baselines the requester. -handle_cast( - {resync_req, FollowerId}, - State = #state{ - leader = Leader, - member_id = Leader, - names_tab = Tab, - epoch = Epoch, - applied_version = Version - } -) -> - RecordsBin = encode_table(Tab), - cast_to_member( - FollowerId, {apply_names_snapshot, RecordsBin, Leader, [], #{}, Epoch, Version} - ), - {noreply, State}; -%% A pid-guarded remote retract (strict_replication fail-closed, §8): a member that -%% failed to durably retract a binding — it lost leadership with the retract still -%% buffered, or its retract batch failed — forwards it here so the binding whose -%% caller was answered `no` cannot silently live on. Guarded by pid, so a name -%% since re-registered to someone else is never clobbered. -handle_cast( - {retract_req, LogicalName, Pid}, State = #state{leader = Leader, member_id = Leader} -) -> - {noreply, enqueue_op({retract, LogicalName, Pid}, State)}; -handle_cast({retract_req, LogicalName, Pid}, State = #state{leader = Leader}) when - Leader =/= undefined --> - cast_to_member(Leader, {retract_req, LogicalName, Pid}), - {noreply, State}; -handle_cast({retract_req, LogicalName, Pid}, State) -> - %% No leader right now — most likely this member is mid-handoff (the assume - %% window, where leader is transiently `undefined`). Stash the pid-guarded clear - %% in `pending_unregs` rather than dropping it, so the assume continuation - %% (redrive_unregs) re-drives it once a leader is established — otherwise a - %% re-driven unregister that lands during a handoff would be silently lost. - {noreply, stash_pending_unreg(LogicalName, Pid, State)}; -%% Leadership transition snapshot sent by the new leader to all followers. -%% Applies the leader transition, the record update (pid + metadata per name), and -%% extra member monitors atomically within a single cast — no other message can -%% interleave. The snapshot carries the leader's applied_version, which re-baselines -%% this member. `RecordsPayload` is the `term_to_binary`-encoded records map (a legacy -%% `[{Name, {Pid, Index, Data}}]` list is also accepted — see decode_records/1). -handle_cast( - {apply_names_snapshot, RecordsPayload, NewLeader, ExtraMembers, Tokens, Epoch, Version}, - State = #state{ - member_id = Self, leader = OldLeader, epoch = CurrentEpoch, applied_version = CurrentVersion - } -) -> - %% The re-baseline must be version-MONOTONIC, not merely epoch-guarded. A snapshot - %% carries the sending leader's applied_version; applying one whose version is BEHIND - %% what we have already applied would wholesale-overwrite our replica *backward*, - %% silently dropping a row we (or a peer) may have already acked. That is exactly the - %% handoff-gather race proven in formal/DgenRegistryReplication.tla: an old assume / - %% resync snapshot delivered late — after we applied a newer broadcast — must be - %% ignored, not obeyed. A legitimate re-baseline never moves us backward: a genuine - %% new leader is caught up to the durable version key before it fans out (see - %% gather_caught_up/6), and a resync only ever carries the leader's current (>=) - %% version. So `Version >= CurrentVersion` rejects only genuinely stale snapshots; - %% a rejected one is harmless — the normal gap/resync machinery re-baselines us from - %% the current leader. - case Epoch >= CurrentEpoch andalso Version >= CurrentVersion of - true -> - %% Overwrite the local replica and inverted index with the leader's snapshot - %% before the leader transition: do_leader_changed -> assume_leadership (if we - %% are becoming leader) reads the names back from the table to set up monitors. - %% The snapshot satisfies any outstanding resync request (cancel it) and makes - %% this member synced (its first snapshot re-announces `fresh = false`). - State0 = records_replace(cancel_resync(State), decode_records(RecordsPayload)), - State1 = do_leader_changed(NewLeader, OldLeader, Self, State0), - %% A leader's snapshot supersedes any assume gather we had in flight (we are - %% a follower of NewLeader now): drop its ref so the stale {assume_gathered} - %% continuation is ignored when it lands. - State2 = State1#state{epoch = Epoch, applied_version = Version, assume_ref = undefined}, - State3 = add_member_monitors(ExtraMembers, State2), - State4 = merge_peer_tokens(Tokens, State3), - %% Handle any buffered ops under the new leadership view: reject them - %% if we are now a follower, or commit them if we remain leader. The - %% snapshot advanced applied_version, so release any forwarded-`yes` - %% acks that were deferred on a gap (flush_deferred; a leadership - %% change already rejected them in do_leader_changed), and re-drive any - %% stashed unregisters and blocked registrations against the now-known - %% leader (redrive_unregs / redrive_registers). - {noreply, - maybe_start_commit( - redrive_registers(redrive_unregs(flush_deferred(mark_synced(State4)))) - )}; - false -> - {noreply, State} - end; -%% Legacy fire-and-forget unregister casts (from a pre-call node during a rolling -%% upgrade). The public API now routes unregisters as tracked calls (see the -%% {unregister} handle_call clauses); these keep the old wire shape working with -%% the old (silent-drop) semantics. -handle_cast( - {unregister, LogicalName}, - State = #state{leader = Leader, member_id = Leader, names_tab = Tab} -) -> - %% Capture the bound pid (if any) *before* the optimistic removal: the plan seeds - %% its working map from the (now mutated) table, so without this plan_op/3 could - %% not tell a genuine unregister from a no-op and would skip the durable clear + - %% broadcast. A bound pid is also recorded in `recently_released` on commit (the - %% conflict-detector trail, §5.6). - ReleasedPid = lookup_name(Tab, LogicalName), - {noreply, - enqueue_op({remove, LogicalName, ReleasedPid, undefined}, row_delete(State, LogicalName))}; -handle_cast({unregister, LogicalName}, State = #state{leader = Leader}) when - Leader =/= undefined --> - cast_to_member(Leader, {unregister, LogicalName}), - {noreply, row_delete(State, LogicalName)}; -%% A new member joined and the (continuing) leader's fast path is onboarding it -%% directly. We are an existing follower whose replica is already up to date, so we -%% must not touch it — we only start monitoring the joiner and record its token, so a -%% future DOWN we observe for it is fenced with the token the elector holds. (On a -%% genuine leadership change the new leader still re-snapshots everyone the full way.) -handle_cast({peer_joined, MemberId, Tokens}, State) -> - {noreply, merge_peer_tokens(Tokens, add_member_monitors([MemberId], State))}; -%% A durable presence subscription changed (§4.9): the elector pushes the delta to the -%% current leader. Apply it only if we are still the leader for a current-or-newer -%% epoch — a stale push (we were deposed) is dropped and the survivor gets the full set -%% reseeded on its assume. A `subscribe` seeds the subscription's watch membership and -%% fires its initial snapshot; an `unsubscribe` drops it. -handle_cast( - {presence_update, Update, Epoch}, - State = #state{member_id = Self, leader = Self, epoch = Cur} -) when Epoch >= Cur -> - {noreply, apply_presence_update(Update, State)}; -handle_cast({presence_update, _Update, _Epoch}, State) -> - {noreply, State}; -handle_cast(_, State) -> - {noreply, State}. - -%% Apply a pushed subscription delta to the leader's live presence state. -apply_presence_update({subscribe, SubId, Watch, Notify}, State = #state{subs = Subs}) -> - seed_subscription(SubId, State#state{subs = Subs#{SubId => {Watch, Notify}}}); -apply_presence_update( - {unsubscribe, SubId}, State = #state{subs = Subs, sub_matches = SM, notify_matches = NM} -) -> - State#state{ - subs = maps:remove(SubId, Subs), - sub_matches = maps:remove(SubId, SM), - notify_matches = maps:remove(SubId, NM) - }; -apply_presence_update(unsubscribe_all, State) -> - State#state{subs = #{}, sub_matches = #{}, notify_matches = #{}}. - -%% --------------------------------------------------------------------------- -%% handle_info/2 -%% --------------------------------------------------------------------------- - -%% Not enough followers confirmed a replica for these direct registrations within the -%% timeout. The policy is `strict_replication` (§8): -%% - false (default): **degrade open** — ack `yes` leader-only (the binding is -%% applied and the async broadcast keeps propagating) and emit telemetry. The -%% residual (a 1-holder reg whose holder dies) is a multi-fault case deferred to -%% §5.6 termination. -%% - true: **fail closed** — reject (`no`) and retract the just-applied binding via -%% an unregister, so the caller never sees a `yes` that was not replicated. -handle_info({replicate_timeout, BatchRef}, State = #state{pending_acks = PA, config = Config}) -> - case maps:take(BatchRef, PA) of - {{Direct, _TimerRef, Needed, Acked}, PA1} -> - State1 = State#state{pending_acks = PA1}, - case dgen_config:strict_replication(Config) of - false -> - notify_degrade_open(Needed, map_size(Acked), length(Direct)), - deliver_direct(Direct), - {noreply, State1}; - true -> - {noreply, reject_and_retract_direct(Direct, State1)} - end; - error -> - %% Already answered by an ack — ignore. - {noreply, State} - end; -%% Result of an in-flight group commit (from the dgen_transaction worker). -handle_info( - {dgen_transaction, Ref, Reply}, - State = #state{ - committing = #{ref := Ref, mref := MRef, plan := Plan}, - member_id = Self, - leader = Leader, - epoch = Epoch - } -) -> - %% Drop the worker monitor (and any DOWN already queued behind this result — - %% signal ordering guarantees the result arrives before the DOWN). - erlang:demonitor(MRef, [flush]), - PlanEpoch = maps:get(epoch, Plan), - State1 = - case Reply of - {committed, Version} when Leader =:= Self, PlanEpoch =:= Epoch -> - apply_committed_plan(Plan, Version, State); - {committed, _Version} -> - %% The commit landed, but this member's leadership view moved while - %% the worker was in flight (deposed, or a new epoch). Applying an - %% old-term plan over new-term state — or broadcasting it stamped - %% with the old epoch — would diverge the replicas, so the plan is - %% discarded: nothing was applied or broadcast (the durable effect - %% of a commit is only the version bump), callers are answered as - %% rejected and retry against the current leader, and the batch's - %% destructive ops are salvaged below so no unregister is lost. - salvage_failed_plan(reject_plan(Plan, State), Plan); - _Fenced_or_error -> - %% {aborted, fenced} (lost leadership) or {error, _} (backend - %% failure / retry exhausted): the batch did not commit. Reject - %% every registration in it (callers retry); re-drive the - %% destructive ops so a lost unregister/retract cannot leave a - %% permanent replica divergence. - salvage_failed_plan(reject_plan(Plan, State), Plan) - end, - %% Start the next batch if more ops have accumulated. - {noreply, maybe_start_commit(State1#state{committing = undefined})}; -handle_info({dgen_transaction, _StaleRef, _Reply}, State) -> - %% Result for a commit we are no longer tracking — ignore. - {noreply, State}; -%% The commit worker died without delivering a result (e.g. it was killed). Fail -%% the batch so its registrations are answered and the commit lock is released. -handle_info( - {'DOWN', MRef, process, _Pid, _Reason}, - State = #state{committing = #{mref := MRef, plan := Plan}} -) -> - State1 = salvage_failed_plan(reject_plan(Plan, State), Plan), - {noreply, maybe_start_commit(State1#state{committing = undefined})}; -%% A registered process died (leader only). The monitor was created with -%% `{tag, {down, Name}}` (erlang:monitor/3), so the death arrives already naming -%% its binding — no reverse ref map to consult (or to build n entries of at every -%% leadership assumption). Park the auto-unregister in the group-commit buffer, -%% carrying Ref so the flush can apply the ref-match guard (a stale DOWN for a -%% name already re-registered must not evict the new binding). If we have since -%% lost leadership the flush drops it and the new leader, which monitors this pid -%% itself, drives the removal. -handle_info({{down, LogicalName}, Ref, process, _Pid, _Reason}, State) -> - {noreply, enqueue_op({down, LogicalName, Ref}, State)}; -handle_info({'DOWN', Ref, process, _Pid, _Reason}, State) -> - #state{ - monitors = Monitors, - elector = Elector, - member_id = Self - } = State, - case maps:get(Ref, Monitors, undefined) of - undefined -> - %% Not a peer-member monitor, and registered-process monitors carry a - %% `{down, Name}` tag (previous clause) — an untagged unknown DOWN is - %% a late straggler from a monitor already demonitored. Ignore. - {noreply, State}; - Self -> - %% Stale self-monitor — should not happen, ignore. - {noreply, State}; - DeadMemberId -> - %% Fence the {member_down} with the token the elector *currently* holds for - %% this peer, read fresh from the durable elector state — not our locally - %% cached peer_tokens entry, which a dropped snapshot cast (e.g. the peer - %% re-announced a fresh token on a nodeup while our node was transiently - %% disconnected) may have left stale. A stale token would make the elector - %% discard this DOWN as if the peer had rejoined, stranding a genuinely dead - %% member in the set and preventing failover (§5.7). The fresh read still - %% correctly discards a true stale DOWN: if the peer really did rejoin, the - %% durable token has already advanced past what we send. - %% - %% The read + cast run in a short-lived helper, never inline: a synchronous - %% call from this loop to the elector could deadlock against the elector's - %% own post-commit `{elector_assume_and_distribute}` call back into this - %% member. The helper falls back to the cached token if the elector read - %% fails (e.g. mid-shutdown). - Cached = maps:get(DeadMemberId, State#state.peer_tokens, undefined), - spawn_member_down(Elector, DeadMemberId, Cached), - {noreply, remove_member(DeadMemberId, State)} - end; -handle_info({nodeup, _Node}, State = #state{elector = Elector, member_id = Self, leader = Leader}) -> - %% Re-announce to the elector — this member may have been removed from the - %% member set while the node was unreachable (partition). A fresh token is - %% generated so any stale {member_down, Self, OldToken} already in the queue - %% is discarded by the elector when it is eventually processed. (The mesh - %% re-connect and probe-gate bookkeeping for this nodeup live in the connector, - %% which subscribes to node events independently.) - NewToken = make_ref(), - dgen_server:cast(Elector, {join, Self, NewToken, not State#state.synced}), - %% A nodeup is a partition heal: if the leader we already know is now reachable - %% again, re-drive any unregisters retained while it was unreachable *immediately*, - %% rather than waiting for the re-onboarding snapshot that also triggers - %% redrive_unregs/1. This covers both ways an unregister survives a partition — a - %% removal *stashed* because the leader was already seen unreachable (`pending_unregs`) - %% and one *forwarded into the dying link* before the disconnect propagated - %% (`pending_forwards`) — so the Non-goal-5 guarantee ("an explicit unregister is - %% never silently lost") does not hinge on the snapshot arriving. Pid-guarded and - %% stash-clearing, so it composes harmlessly with the snapshot path. - %% - %% Guarded on the leader being reachable (or us): redrive_unregs/1 forwards and clears - %% the stash unconditionally (safe on the snapshot path, where the snapshot's sender - %% is by definition reachable), so calling it here while the leader is still - %% unreachable would drop the retract into a dropped cast *and* clear the stash — - %% losing it. A different node's nodeup with our leader still unreachable is left for - %% the snapshot / a later heal. - State1 = State#state{join_token = NewToken}, - LeaderReachable = - Leader =:= Self orelse (Leader =/= undefined andalso member_reachable(Leader)), - {noreply, - case LeaderReachable of - true -> redrive_unregs(State1); - false -> State1 - end}; -%% Continuation of the continuing-leader fast path for a rejoining (non-fresh) -%% member: its replica snapshot arrived from the gather helper. Adjudicate it -%% against our authoritative table (§5.6) — a name a rejoiner still binds to a -%% *different, live, un-trailed* pid than ours is a genuine conflict, resolved by -%% kill-both under the usual budget/alarm/config — then onboard the joiner with -%% our snapshot. The killed authority's monitor DOWN drives the normal -%% unregister/broadcast cleanup, so no bespoke removal path is needed. The -%% joiner's release trail is merged either way (the same merge the full-path -%% gather does), so a legitimately-unregistered-but-lagging row is suppressed, -%% not killed. Guarded on leadership, epoch, and current membership: a handoff -%% or re-join that superseded this gather owns the onboarding instead. -handle_info( - {joiner_gathered, MemberId, AllIds, Tokens, Epoch, Result}, - State = #state{member_id = Self, leader = Self, epoch = Epoch, members = Members} -) when is_map_key(MemberId, Members) -> - State1 = - case Result of - {ok, {Records, _Version, Released}} -> - MergedReleased = maps:merge_with( - fun(_K, A, B) -> max(A, B) end, - State#state.recently_released, - Released - ), - %% One materialization of our own replica (the authority) instead of - %% one plus two pid projections — detect_conflicts compares records - %% maps in place now. The resolved map is discarded either way: the - %% leader's table is authoritative here, and a conflict resolution - %% only kills/alarms (the joiner is re-baselined by the snapshot). - OwnRecords = current_records(State#state.names_tab), - Conflicts = detect_conflicts(OwnRecords, [Records], MergedReleased), - {_Records, S1} = resolve_conflicts( - Conflicts, OwnRecords, State#state{recently_released = MergedReleased} - ), - S1; - error -> - %% The joiner did not answer in time — onboard it regardless; a - %% divergence it still holds will surface at the next gather. - State - end, - {noreply, onboard_joiner(MemberId, AllIds, Tokens, State1)}; -%% The asynchronous handoff gather (spawn_assume_gather) has returned: finish assuming -%% leadership with the freshest reconstruction, off the critical path the elector's call -%% was on. Applied only if `assume_ref` still matches — a newer assume, or a snapshot -%% that made us a follower, has since replaced it and owns reconstruction instead. -%% -%% This is exactly the reconstruction the inline handler used to do, just deferred to -%% here: the freshest map *is* the reconstructed state (its highest-version owner holds -%% the freshest binding for every name — broadcasts are totally ordered + FIFO, and gap -%% detection keeps every replica a prefix of that stream; a binding no surviving member -%% held is absent and may be re-issued, single-fault uniqueness backstopped by §5.6, not -%% a durable taken-set). MaxVersion becomes our applied_version and re-baselines the -%% followers. -handle_info( - {assume_gathered, {MemberId, AllIds, Tokens, FreshIds, Epoch, Ref}, SelfRecords, - FreshestRecords, MaxVersion, PeerResults, Subs}, - State = #state{member_id = Self, assume_ref = Ref} -) -> - OtherIds = lists:delete(Self, AllIds), - PeerRecordMaps = [Recs || {Recs, _V, _Rel} <- maps:values(PeerResults)], - %% Incomplete gather (a committed, non-fresh member did not respond → unreachable): - %% we may be missing its bindings, so the leader is `degraded` (§5.6). A member the - %% elector marked *fresh* (never synced) holds nothing, so its absence is not a - %% degrade — a healthy scale-up does not flag one. - Unreachable = OtherIds -- maps:keys(PeerResults), - Degraded = lists:any(fun(Id) -> not lists:member(Id, FreshIds) end, Unreachable), - %% Merge every reachable member's released-pid trail into our own (§5.6): the trail - %% distinguishes "legitimately unregistered, reported by a lagging member" from a - %% genuine divergence, and must survive leadership changes. - MergedReleased = lists:foldl( - fun({_Recs, _V, Rel}, Acc) -> maps:merge_with(fun(_P, A, B) -> max(A, B) end, Acc, Rel) end, - State#state.recently_released, - maps:values(PeerResults) - ), - %% Resolve genuine uniqueness conflicts the gather exposes (§5.6) — kill-both + - %% alarm + bounded budget; conflicted names are dropped (the fan-out propagates the - %% drop). A no-op in the common conflict-free handoff — and an ALLOCATION-FREE - %% one: detect_conflicts compares the records maps in place, and with no - %% conflicts `Records` below IS `FreshestRecords`, untouched. Metadata rides the - %% surviving bindings automatically (resolve_conflicts only ever *removes* names - %% from the records map). This path used to project three pid maps, build a - %% union map, and re-project the survivors with maps:keys + maps:with — five - %% n-entry allocations discarded on every healthy handoff. - {Records, State0} = resolve_conflicts( - detect_conflicts( - FreshestRecords, - [SelfRecords | PeerRecordMaps], - MergedReleased - ), - FreshestRecords, - State#state{recently_released = MergedReleased} - ), - %% Reconstruct the local replica (pid + metadata) + inverted index wholesale, - %% then become leader. Clearing assume_ref marks the assume complete; the - %% `leader` state edge this produces runs assume_leadership (subscription - %% reseed + monitor sweep) in the enter call — and the old inline relinquish - %% is gone too, because leaving `leader` already ran it when this assume - %% began. `Subs` (pulled from the co-located elector off this loop, §4.9) - %% is seeded into Data here so the enter call's recompute_sub_matches - %% computes each watch set against the reconstructed replica; a silent - %% reseed — subscribers keep their view, and a subscribe racing the handoff - %% pushes a {presence_update} delta applied once established below. - State0a = records_replace(State0, Records), - State2 = State0a#state{ - leader = Self, - epoch = Epoch, - applied_version = MaxVersion, - degraded = Degraded, - assume_ref = undefined, - subs = Subs - }, - State3 = add_member_monitors(extra_member_ids(MemberId, AllIds, Self), State2), - State4 = merge_peer_tokens(Tokens, State3), - %% Encode the snapshot once; every follower's cast shares the one refc binary. - RecordsBin = encode_records(Records), - lists:foreach( - fun(Id) -> - cast_to_member( - Id, - {apply_names_snapshot, RecordsBin, Self, extra_member_ids(MemberId, AllIds, Id), - Tokens, Epoch, MaxVersion} - ) - end, - OtherIds - ), - %% We are the leader now: reject any registrations we forwarded as a follower (the - %% old leader will never answer them), re-drive stashed unregisters and blocked - %% registrations into our own commit buffer, then commit anything buffered while we - %% assumed. The gather made us synced; any outstanding resync is moot. - {noreply, - maybe_start_commit( - redrive_registers( - redrive_unregs(reject_forwards(mark_synced(cancel_resync(State4)))) - ) - )}; -%% Periodic maintenance: expire stale entries from the conflict-detection trail -%% (§5.6) and the per-name kill-budget timestamps, then re-arm. Trail pruning is -%% suspended while any current member is disconnected: a disconnected member misses -%% unregister broadcasts outright, so its stale rows can show up at a rejoin gather -%% long after any broadcast lag — the trail must outlive the partition (see the -%% ?PRUNE_INTERVAL note). -handle_info( - prune_released, - State = #state{recently_released = Rel, kill_budget = KB, members = Members, config = Config} -) -> - arm(?PRUNE_INTERVAL, prune_released), - Now = erlang:system_time(millisecond), - AllConnected = lists:all( - fun({Node, _Name}) -> dgen_utils:node_reachable(Node) end, - maps:keys(Members) - ), - TTL = dgen_config:conflict_release_ttl(Config), - %% Pruning is also suspended within a TTL of the most recent member departure: - %% a member dropped from the set (so no longer "a disconnected member") may - %% rejoin with stale rows, and the trail must still discriminate lag from - %% divergence at that rejoin's gather (§5.6). A member gone longer than the - %% TTL is outside the trail's protection horizon either way. - RecentDeparture = Now - State#state.last_departure < TTL, - Rel1 = - case AllConnected andalso not RecentDeparture of - true -> - Cutoff = Now - TTL, - maps:filter(fun(_Key, Ts) -> Ts >= Cutoff end, Rel); - false -> - Rel - end, - {_Count, Window} = dgen_config:conflict_kill_budget(Config), - KB1 = maps:filter( - fun(_Name, Tss) -> Tss =/= [] end, - maps:map(fun(_Name, Tss) -> [Ts || Ts <- Tss, Ts >= Now - Window] end, KB) - ), - %% Also drop stashed registrations whose callers have timed out (bounds - %% pending_registers while no leader appears — see the field doc). - {noreply, prune_pending_registers(State#state{recently_released = Rel1, kill_budget = KB1})}; -%% The resync request we sent went unanswered (dropped cast, deposed target) — -%% clear the guard so the next gap-observing broadcast (or the leader's replication -%% heartbeat, which is one) requests again. -handle_info(resync_timeout, State) -> - {noreply, State#state{resync_timer = undefined}}; -%% Replication heartbeat (see ?REPLICA_HEARTBEAT_INTERVAL). Only the leader sends, -%% and only an *empty* batch stamped at its current applied version: a follower that -%% is caught up applies nothing, one that is behind takes apply_bcast/6's gap branch -%% and asks for a resync. This is what makes replication converge without new -%% writes; every other gap-detection trigger needs traffic that a quiescent cluster -%% does not have. -%% (replica_heartbeat is dispatched entirely from handle_event — the leader -%% advertises via broadcast_heartbeat/1, defined beside broadcast_batch/5's -%% MUTATION ifdef since it is itself mutation-planted; everyone re-arms.) -%% -%% The deferred monitor-establishment sweep (see assume_leadership/1): one chunk of -%% the names table per message, so live traffic interleaves. Three guards make it -%% safe against everything that can move underneath it: -%% - the head fences on `leader =:= Self` AND the arming epoch, so deposition or -%% a newer assume strands stale steps at the fallthrough clause; -%% - a name already in name_to_ref is skipped — a registration committed since -%% the sweep began was monitored by apply_monitor_ops on its own commit; -%% - each pair is re-checked against the live table, so a name unregistered or -%% re-bound since its chunk was selected is skipped rather than monitored -%% against a stale pid (ETS select continuations are safe under concurrent -%% writes but see each object as of its own chunk). -%% A monitored pid that died before its chunk arrived fires an immediate `noproc` -%% DOWN, which is the property that makes deferral lose nothing. -handle_info( - {monitor_sweep, Epoch, Cont0}, - State = #state{member_id = Self, leader = Self, epoch = Epoch, names_tab = Tab} -) -> - Select = - case Cont0 of - start -> - ets:select( - Tab, [{{'$1', '$2', '_', '_'}, [], [{{'$1', '$2'}}]}], ?MONITOR_SWEEP_CHUNK - ); - _ -> - ets:select(Cont0) - end, - case Select of - '$end_of_table' -> - {noreply, State}; - {Pairs, Cont1} -> - NTR1 = - lists:foldl( - fun({Name, Pid}, Acc) -> - case is_map_key(Name, Acc) orelse lookup_name(Tab, Name) =/= Pid of - true -> Acc; - false -> Acc#{Name => monitor_name(Name, Pid)} - end - end, - State#state.name_to_ref, - Pairs - ), - self() ! {monitor_sweep, Epoch, Cont1}, - {noreply, State#state{name_to_ref = NTR1}} - end; -%% Re-drive the destructive ops of a batch that failed to commit (see -%% salvage_failed_plan): re-enqueue them if we are (still) the leader, or forward -%% the pid-guarded clears to the current leader if we were deposed — so an -%% unregister or strict-mode retract is never silently lost, which would otherwise -%% leave replicas divergent (or a `no`-answered binding alive) until an unrelated -%% handoff. -handle_info({requeue_ops, Ops}, State = #state{leader = Leader, member_id = Self}) when - Leader =:= Self --> - {noreply, lists:foldl(fun enqueue_op/2, State, Ops)}; -handle_info({requeue_ops, Ops}, State = #state{leader = Leader}) when Leader =/= undefined -> - lists:foreach( - fun - ({remove, Name, Pid, _Origin}) when is_pid(Pid) -> forward_retract(Leader, Name, Pid); - ({retract, Name, Pid}) -> forward_retract(Leader, Name, Pid); - %% A remove that never had a bound pid nets to nothing; a down is the - %% new leader's to observe via its own monitor. - (_) -> ok - end, - Ops - ), - {noreply, State}; -handle_info({requeue_ops, Ops}, State) -> - %% No leader right now (likely mid-handoff — the assume window, leader = - %% undefined). Stash the pid-guarded clears in `pending_unregs` so the assume - %% continuation (or the next snapshot) re-drives them, rather than dropping them - %% and leaving a divergence until an unrelated handoff. Downs are dropped — the - %% new leader monitors those pids itself. - Stashed = lists:foldl( - fun - ({remove, Name, Pid, _Origin}, S) when is_pid(Pid) -> stash_pending_unreg(Name, Pid, S); - ({retract, Name, Pid}, S) -> stash_pending_unreg(Name, Pid, S); - (_, S) -> S - end, - State, - Ops - ), - {noreply, Stashed}; -%% Periodic durable-epoch check (piggybacked on the mesh pass): the election record -%% in the database is ahead of what this member has heard. That means a leadership -%% handoff happened that never reached us — the elector's post-commit assume action -%% failed (consumer crash, unreachable leader), or we missed the snapshot fan-out. -%% Re-announcing produces a fresh membership event whose assume/fan-out brings us — -%% and, if we are the durable leader, our leadership — up to date. -handle_info({durable_epoch, DurableEpoch}, State = #state{epoch = Epoch}) when - is_integer(DurableEpoch), DurableEpoch > Epoch --> - #state{elector = Elector, member_id = Self, join_token = Token, synced = Synced} = State, - dgen_server:cast(Elector, {join, Self, Token, not Synced}), - {noreply, State}; -handle_info({durable_epoch, _DurableEpoch}, State) -> - {noreply, State}; -handle_info(_Info, State) -> - {noreply, State}. - -terminate(_Reason, _StatemState, _State) -> - net_kernel:monitor_nodes(false), - ok. - -code_change(_OldVsn, StatemState, State, _Extra) -> - {ok, StatemState, State}. - -%% --------------------------------------------------------------------------- -%% Internal helpers -%% --------------------------------------------------------------------------- - -do_leader_changed(NewLeader, OldLeader, Self, State0) -> - %% On any leadership change, reject registrations forwarded to the old leader — - %% its {register_reply} will never arrive; callers retry against the new leader. - State = - case NewLeader =/= OldLeader of - true -> reject_forwards(State0); - false -> State0 - end, - %% The role-edge halves — relinquish on leaving `leader`, assume on entering - %% it — run in the state enter calls, off this function's returned Data. - if - OldLeader =:= Self, NewLeader =/= Self -> - %% Lost leadership — resolve any direct registrations still awaiting - %% replica acks (their timers must not fire into a follower). The - %% demonitor half runs on the leader-exit enter call. - fail_pending_acks(State#state{leader = NewLeader}); - true -> - State#state{leader = NewLeader} - end. - -%% Become leader WITHOUT establishing the registered-process monitors inline: the -%% sweep that creates them (~0.7us per monitor — 130ms+ of client-visible handoff -%% window at 200k names) is deferred to chunked {monitor_sweep, …} self-sends that -%% interleave with live traffic. Correct, not just faster, because monitor -%% creation is self-healing for past deaths: a monitor on an already-dead pid -%% fires an immediate `noproc` DOWN (and `noconnection` for an unreachable one), -%% so a subject that dies before its chunk arrives is caught at establishment, -%% never lost. The observable cost is a slightly longer window in which a dead -%% pid's name reads as bound — a lag that is already legal and asynchronous -%% (dead-pid cleanup has no promptness bound; see the design's Non-goals and the -%% sim invariants' `dead_ok?`). No safety invariant references monitor timing. -%% -%% The sweep is epoch-fenced: each step re-checks `leader =:= Self` and the epoch -%% it was armed under, so deposition (or a newer assume) strands stale steps -%% harmlessly — the same supersession discipline as `assume_ref`. -assume_leadership(State = #state{epoch = Epoch}) -> - self() ! {monitor_sweep, Epoch, start}, - %% Seed each subscription's watch-set membership from the reconstructed replica, so - %% the first commit under this leadership computes its deltas against the true - %% current state (§4.9). Leader-only derived state — a follower never reads it. - recompute_sub_matches(State). - -%% Recompute every subscription's watch-set and notify-set membership -%% (`#{SubId => #{Name => Pid}}` each) by running its watch/notify queries against the -%% current replica. Called on assuming leadership (the only time the maps must be rebuilt -%% from scratch); thereafter apply_committed_plan keeps them current incrementally. A -%% silent reseed — it does not re-fire snapshots; the new leader just advances deltas -%% correctly from here (subscribers keep the view they already had). -recompute_sub_matches(State = #state{subs = Subs, names_tab = Tab, inv_index = Inv}) -> - SM = maps:map( - fun(_SubId, {Watch, _Notify}) -> query_match_names(Watch, Tab, Inv) end, - Subs - ), - NM = maps:map( - fun(_SubId, {_Watch, Notify}) -> query_match_names(Notify, Tab, Inv) end, - Subs - ), - State#state{sub_matches = SM, notify_matches = NM}. - -%% Demonitor all registered Pids; keep the names map for snapshot reads. The -%% cached commit version is dropped too — it is this member's term and must not pin a -%% later term's read version. -relinquish_leadership(State = #state{name_to_ref = NTR}) -> - maps:foreach( - fun(_LogicalName, Ref) -> - erlang:demonitor(Ref, [flush]) - end, - NTR - ), - State#state{name_to_ref = #{}, last_version = undefined}. - -%% Gather every reachable member's names map across this (assuming) leader's own -%% replica and every other reachable member. Because the leader's broadcasts are -%% totally ordered (commit versions are globally monotonic) and FIFO, and gap -%% detection keeps every replica a *prefix* of that stream (see applied_version), -%% the member with the highest applied_version holds the freshest binding for -%% *every* name, so the single freshest map is the reconstruction source — and a -%% version tie implies identical content, so tie-breaking (self-first) is safe. -%% The other members' snapshots are returned too, for the conflict detector (§5.6) -%% — a partition can leave a member holding a divergent live binding the freshest -%% map never saw — and for the released-trail merge. Returns -%% `{FreshestNames, MaxVersion, PeerResults}` where PeerResults is -%% `#{MemberId => {Records, Version, Released}}` for each *other* member that -%% answered. The per-peer snapshot calls run in parallel (each in a short-lived -%% helper process), so the assuming leader's loop is blocked for at most about one -%% ?GATHER_TIMEOUT overall rather than one per unreachable peer. -gather_maps(SelfNames, SelfVersion, OtherIds) -> - PeerResults = gather_peer_snapshots(OtherIds), - {FreshestNames, MaxVersion} = maps:fold( - fun(_Id, {Names, Version, _Released}, {_BestNames, BestVersion} = Best) -> - case Version > BestVersion of - true -> {Names, Version}; - false -> Best - end - end, - {SelfNames, SelfVersion}, - PeerResults - ), - {FreshestNames, MaxVersion, PeerResults}. - -%% Onboard a joining member on the continuing-leader fast path: ship it our full -%% snapshot (unavoidable O(names) for one member — exactly what the full path would -%% send it), carrying the member list so it monitors every peer, and tell the -%% existing followers to monitor the joiner via a small token-only {peer_joined} -%% (their prefix-consistent replicas are untouched — no records travel there). -onboard_joiner(MemberId, AllIds, Tokens, State = #state{member_id = Self, epoch = Epoch}) -> - RecordsBin = encode_table(State#state.names_tab), - cast_to_member( - MemberId, - {apply_names_snapshot, RecordsBin, Self, AllIds, Tokens, Epoch, State#state.applied_version} - ), - lists:foreach( - fun(Id) -> cast_to_member(Id, {peer_joined, MemberId, Tokens}) end, - [Id || Id <- AllIds, Id =/= Self, Id =/= MemberId] - ), - State. - -%% Fetch a rejoining member's replica snapshot off the leader's loop (bounded by -%% member_names/1's ?GATHER_TIMEOUT; an unreachable joiner fast-fails), reporting -%% back as {joiner_gathered, …} for the fast-path §5.6 adjudication. AllIds/Tokens -%% ride along so the continuation onboards with exactly the member list the elector -%% committed for this join event. -spawn_joiner_gather(MemberId, AllIds, Tokens, Epoch, Owner) -> - _ = spawn(fun() -> - Owner ! {joiner_gathered, MemberId, AllIds, Tokens, Epoch, member_names(MemberId)} - end), - ok. - -%% Run the full handoff gather (the peer-snapshot RPCs, bounded overall by -%% ?GATHER_TIMEOUT) off the member's loop, reporting the freshest reconstruction back as -%% {assume_gathered, …} for the continuation to finish assuming leadership (§5.7). -%% `SelfRecords` / `SelfVersion` are captured by the caller — a consistent snapshot of -%% our own replica — so a broadcast that lands during the gather cannot skew the -%% freshest-wins comparison. `Ctx` carries the elector's assume payload plus the -%% correlation ref the continuation matches against `assume_ref`. -%% -%% The durable presence subscriptions (§4.9) are pulled from the co-located `Elector` -%% here too — **off the member's loop, deliberately**: doing it in the continuation -%% (on the loop) would be a synchronous member→elector call that can dead-lock against -%% the elector's own synchronous assume call back into this member (both block until a -%% timeout fires). Pulling in this helper keeps the member loop responsive, and reading -%% after the gather means any subscribe committed during the (up to ~?GATHER_TIMEOUT) -%% gather is captured; a subscribe racing the final millisecond pushes a -%% {presence_update} delta the now-established leader applies. -spawn_assume_gather(Ctx, SelfRecords, SelfVersion, OtherIds, Elector, Owner, Tenant, Tuid) -> - _ = spawn(fun() -> - %% Fall back to a self-only reconstruction if the gather ever throws, so the - %% continuation still fires and the member is never stranded leaderless (it - %% assumes with its own replica, degraded — a later handoff reconciles). - {FreshestRecords, MaxVersion, PeerResults} = - gather_caught_up(SelfRecords, SelfVersion, OtherIds, Tenant, Tuid), - Subs = fetch_subscriptions(Elector), - Owner ! {assume_gathered, Ctx, SelfRecords, FreshestRecords, MaxVersion, PeerResults, Subs} - end), - ok. - -%% Gather, then ensure the reconstruction is at least as fresh as the durable version -%% key before returning — the handoff-gather race fix proven in -%% formal/DgenRegistryReplication.tla (the SafeAssume fence). The committed frontier is -%% the durable version key (dgen_registry_names:read_committed_frontier/2); a live member -%% can never be ahead of it, so a gathered MaxVersion below it means the gather is -%% incomplete — most -%% often because it raced an in-flight (committed but not-yet-applied) broadcast still -%% queued in a reachable peer's mailbox. Reconstructing from that stale map and fanning -%% it out would overwrite the very follower about to hold the missing binding, silently -%% dropping an already-acked registration (the finding). The lagging peer applies its -%% queued broadcast within milliseconds, so we briefly re-gather until MaxVersion catches -%% up to the frontier or ?GATHER_TIMEOUT elapses. Past the deadline the missing version's -%% holder is genuinely unreachable (a real degrade / multi-fault, not a race), so we -%% proceed with what we have — the continuation flags `degraded` and §5.6 backstops any -%% conflict. Runs entirely in the off-loop gather helper, so the member loop stays -%% responsive and healthy handoffs (where MaxVersion already equals the frontier on the -%% first pass) pay nothing beyond one durable read. -gather_caught_up(SelfRecords, SelfVersion, OtherIds, Tenant, Tuid) -> - Deadline = erlang:monotonic_time(millisecond) + ?GATHER_TIMEOUT, - gather_caught_up(SelfRecords, SelfVersion, OtherIds, Tenant, Tuid, Deadline). - -gather_caught_up(SelfRecords, SelfVersion, OtherIds, Tenant, Tuid, Deadline) -> - Result = - {_FreshestRecords, MaxVersion, _PeerResults} = - try - gather_maps(SelfRecords, SelfVersion, OtherIds) - catch - _:_ -> {SelfRecords, SelfVersion, #{}} - end, - %% Best-effort read of the committed frontier — normalised to the same scale as a - %% member's applied_version (read_committed_frontier strips the versionstamp's batch - %% bytes). A backend hiccup (or a test double without a real tenant) falls back to 0, - %% which disables the fence rather than wedging the handoff — the pre-fix behaviour, - %% never worse. - DurableVersion = - try - dgen_registry_names:read_committed_frontier(Tenant, Tuid) - catch - _:_ -> 0 - end, - case MaxVersion >= DurableVersion orelse erlang:monotonic_time(millisecond) >= Deadline of - true -> - Result; - false -> - timer:sleep(?ASSUME_CATCHUP_INTERVAL), - gather_caught_up(SelfRecords, SelfVersion, OtherIds, Tenant, Tuid, Deadline) - end. - -%% Read the durable subscription set from the (co-located) elector — best-effort, under a -%% short bound so a momentarily-busy elector cannot stall the gather helper: on timeout -%% the leader assumes with the empty set and a later {presence_update} delta / the next -%% handoff reconciles. Called only from off-loop helpers (never the member loop). -fetch_subscriptions(Elector) when is_pid(Elector) -> - try dgen_server:priority_call(Elector, get_subscriptions, ?SUBS_PULL_TIMEOUT) of - Subs when is_map(Subs) -> Subs; - _ -> #{} - catch - _:_ -> #{} - end; -fetch_subscriptions(_Elector) -> - #{}. - -%% Fan out one member_names/1 helper per peer and collect the answers under a -%% single overall deadline. Late answers (a peer that responds after the deadline) -%% arrive as ordinary messages and are dropped by the catch-all handle_info clause. -gather_peer_snapshots([]) -> - #{}; -gather_peer_snapshots(OtherIds) -> - Ref = make_ref(), - Parent = self(), - lists:foreach( - fun(MemberId) -> - spawn(fun() -> Parent ! {Ref, MemberId, member_names(MemberId)} end) - end, - OtherIds - ), - Deadline = erlang:monotonic_time(millisecond) + ?GATHER_TIMEOUT + 500, - collect_gather(Ref, OtherIds, Deadline, #{}). - -collect_gather(_Ref, [], _Deadline, Acc) -> - Acc; -collect_gather(Ref, Waiting, Deadline, Acc) -> - Timeout = max(0, Deadline - erlang:monotonic_time(millisecond)), - _ = ?ETA_LOG({gather_wait, length(Waiting), Timeout}), - receive - {Ref, MemberId, {ok, Snapshot}} -> - collect_gather(Ref, lists:delete(MemberId, Waiting), Deadline, Acc#{ - MemberId => Snapshot - }); - {Ref, MemberId, error} -> - collect_gather(Ref, lists:delete(MemberId, Waiting), Deadline, Acc) - after Timeout -> - _ = ?ETA_LOG({gather_timed_out, length(Waiting)}), - Acc - end. - -%% Detect genuine uniqueness conflicts among the gathered maps (§5.6). A conflict is -%% a name held by *two different live pids* — the authority (the freshest map's pid) -%% and some other member's pid that is (a) different, (b) not in `recently_released` -%% (so a legitimately-unregistered-but-lagging binding is suppressed — that is the -%% discriminator that makes this safe), and (c) actually alive. Single-fault -%% uniqueness (no durable taken-set) lets a partition produce this: a follower holds a -%% binding the leader dropped on reconstruction and re-issued elsewhere. A name the -%% freshest map lacks entirely (authority `undefined`) is *not* a conflict — there is -%% no second live claimant — so it is left to the freshest-wins reconstruction. -%% Aliveness is probed only when a candidate divergence exists, so the common -%% all-agree handoff costs no probes. Returns `[{Name, AuthorityPid, [DivergentPid]}]`. -%% -%% Takes RECORDS maps (`#{Name => {Pid, Index, Data}}`) directly and compares each -%% entry against the authority inline, accumulating only for *divergent* names — so -%% the common all-agree handoff allocates nothing but the fold's loop frame. This -%% used to project every map to pids (`record_pids`) and build a name→pid-set map -%% over the union first: at 200k names that was three full n-entry maps plus the -%% union map (~0.9us/name) allocated and discarded on a path where a divergence is -%% the rare case. An entry from the freshest map itself always agrees with its own -%% authority, so including it in `RecordMaps` is harmless. -detect_conflicts(FreshestRecords, RecordMaps, Released) -> - Candidates = lists:foldl( - fun(Map, Acc0) -> - maps:fold( - fun(Name, {Pid, _Index, _Data}, Acc) -> - case maps:get(Name, FreshestRecords, undefined) of - {Pid, _, _} -> - %% Agrees with the authority — the common case, free. - Acc; - _AbsentOrDifferent -> - case released_entry(Name, Pid, Released) of - true -> - Acc; - false -> - Pids = maps:get(Name, Acc, #{}), - Acc#{Name => Pids#{Pid => true}} - end - end - end, - Acc0, - Map - ) - end, - #{}, - RecordMaps - ), - maps:fold( - fun(Name, PidSet, Acc) -> - Authority = - case maps:get(Name, FreshestRecords, undefined) of - {APid, _Index, _Data} -> APid; - undefined -> undefined - end, - Others = [P || P <- maps:keys(PidSet), P =/= Authority], - case Others =:= [] orelse not (is_pid(Authority) andalso is_pid_alive(Authority)) of - true -> - Acc; - false -> - case [P || P <- Others, is_pid_alive(P)] of - [] -> Acc; - Divergent -> [{Name, Authority, Divergent} | Acc] - end - end - end, - [], - Candidates - ). - -%% Was this pid explicitly released *from this name* recently? Trail keys are -%% {Name, Pid} pairs, so releasing a pid under one of its names never suppresses a -%% genuine conflict on another name the same pid also holds. Bare-pid keys — the -%% legacy trail shape, seen in a pre-upgrade member's gathered trail during a -%% rolling upgrade — are honoured too (the coarser, name-agnostic suppression the -%% old code applied), until the mixed entries age out of the TTL. -released_entry(Name, Pid, Released) -> - maps:is_key({Name, Pid}, Released) orelse maps:is_key(Pid, Released). - -%% Resolve detected conflicts (§5.6). Kill-both (terminate the authority pid and -%% every divergent pid), drop the name from the reconstructed map (the snapshot -%% fan-out propagates the drop, so supervised processes restart and re-register -%% cleanly under the single fenced leader), and alarm — subject to a per-name kill -%% budget and the `terminate_on_conflict` config. Operates on the freshest RECORDS -%% map (values `{Pid, Index, Data}`) and only ever `maps:remove`s from it, so with -%% no conflicts the map passes through by reference. Returns `{CleanRecords, State}`. -resolve_conflicts([], Names, State) -> - {Names, State}; -resolve_conflicts(Conflicts, Names, State = #state{config = Config}) -> - Terminate = dgen_config:terminate_on_conflict(Config), - lists:foldl( - fun({Name, Authority, Divergent}, {NAcc, S}) -> - resolve_one_conflict(Name, Authority, Divergent, Terminate, NAcc, S) - end, - {Names, State}, - Conflicts - ). - -%% Detect-only mode: alarm + keep the authority binding (best-effort), no kill. -resolve_one_conflict(Name, Authority, Divergent, false, Names, State) -> - alarm_conflict(Name, Authority, Divergent, detect_only), - {Names, State}; -resolve_one_conflict( - Name, Authority, Divergent, true, Names, State = #state{kill_budget = KB, config = Config} -) -> - {Count, Window} = dgen_config:conflict_kill_budget(Config), - Now = erlang:system_time(millisecond), - Recent = [Ts || Ts <- maps:get(Name, KB, []), Ts >= Now - Window], - case length(Recent) < Count of - false -> - %% Budget exhausted — escalate to an operator and stop killing this name; - %% keep the authority binding so the name stays served. - alarm_conflict(Name, Authority, Divergent, budget_exhausted), - {Names, State}; - true -> - Kill = [Authority || is_pid_alive(Authority)] ++ Divergent, - lists:foreach(fun(P) -> exit(P, kill) end, Kill), - alarm_conflict(Name, Authority, Divergent, terminated), - {maps:remove(Name, Names), State#state{kill_budget = KB#{Name => [Now | Recent]}}} - end. - -alarm_conflict(Name, Authority, Divergent, Action) -> - logger:error( - "dgen_registry: name conflict — name=~p authority=~p divergent=~p action=~p. " - "Two different live pids held one name (single-fault uniqueness breach, §5.6).", - [Name, Authority, Divergent, Action] - ), - %% Also surface through telemetry (when available), like degrade-open events — - %% `action` distinguishes terminated / detect_only / budget_exhausted, the last - %% being the one an operator must act on. - emit_telemetry( - [dgen_registry, conflict], - #{divergent_count => length(Divergent)}, - #{name => Name, authority => Authority, action => Action} - ). - -%% Is a (possibly remote) pid alive? An unreachable node is treated as not-alive so a -%% conflict is never escalated on a binding we cannot verify (the next gather re-checks -%% once the node is reachable). -is_pid_alive(Pid) when node(Pid) =:= node() -> - is_process_alive(Pid); -is_pid_alive(Pid) -> - Node = node(Pid), - case lists:member(Node, nodes()) of - false -> - false; - true -> - try erpc:call(Node, erlang, is_process_alive, [Pid], ?ALIVE_PROBE_TIMEOUT) of - Alive when is_boolean(Alive) -> Alive; - _ -> false - catch - _:_ -> false - end - end. - -%% Bounded, fault-tolerant snapshot read of a peer member's names map + version + -%% released trail. A member on a disconnected node, or one that does not answer -%% within ?GATHER_TIMEOUT, is skipped — its bindings are unavailable for this -%% gather. With no durable taken-set, a binding only that member held is lost -%% (single-fault uniqueness); if it was replicated (two-holder), the freshest -%% surviving holder still carries it. -member_names({Node, Name}) -> - case lists:member(Node, nodes()) of - false -> - error; - true -> - %% Everything — the call, the decode, and the shape check — runs inside the - %% try: a `try ... of` body is *not* covered by the catch, and decode_records - %% can fault on a corrupt payload, so keep it protected and fall to `error`. - try - {Payload, Version, Released} = - gen_statem:call({Name, Node}, get_names_snapshot, ?GATHER_TIMEOUT), - Names = decode_records(Payload), - true = is_map(Names) andalso is_integer(Version) andalso is_map(Released), - {ok, {Names, Version, Released}} - catch - _:_ -> error - end - end. - -%% §5.6 prevention: while the leader is degraded (its last handoff gather was -%% incomplete, so a committed member is unreachable) and `reject_when_degraded` is -%% enabled, refuse to register a name the leader does not already hold — it could be -%% the unreachable member's binding, and registering it would be a re-issue. A name -%% already in the map flows to the normal dup path. Default off (reactive posture). -reject_new_when_degraded(Name, #state{degraded = true, names_tab = Tab, config = Config}) -> - dgen_config:reject_when_degraded(Config) andalso not name_exists(Tab, Name); -reject_new_when_degraded(_Name, _State) -> - false. - -%% Route a registration to wherever it must go, deferring the caller's reply in every -%% case except an adjudicated refusal. `no` is answered *only* when the name is -%% genuinely refused — held by a different pid is decided downstream by the leader; the -%% one immediate `no` here is the `reject_when_degraded` prevention mode (§5.6), an -%% explicit opt-in. Crucially, **no leader (or an unreachable one) does not answer -%% `no`** — that would be a false `already_started` to OTP. Instead the caller is -%% *blocked*: the registration is stashed to await a leader (redrive_registers/1), so -%% it succeeds if one appears within the caller's `register_timeout` and otherwise the -%% caller's own call times out. Returns the new state; the reply is either already -%% sent (immediate `no`) or deferred. -route_register(Name, Pid, Meta, From, State = #state{leader = Leader, member_id = Self}) -> - if - Leader =:= Self -> - %% Leader: park in the group-commit buffer, answered on commit — unless the - %% degraded-prevention mode refuses it outright. - case reject_new_when_degraded(Name, State) of - true -> - gen_server:reply(From, no), - State; - false -> - enqueue_op({add, Name, Pid, Meta, {local, From}}, State) - end; - Leader =:= undefined -> - %% No leader elected yet (startup, or a brief handoff window) — block. - stash_pending_register(Name, Pid, Meta, From, State); - true -> - %% A different member leads. Forward if reachable (the caller blocks on the - %% eventual {register_reply}); if unreachable (partition), block and stash - %% rather than forward into a dropped cast — the removal-style re-drive picks - %% it up when a leader is reachable again. - case member_reachable(Leader) of - true -> - Ref = make_ref(), - cast_to_member(Leader, {register_req, Ref, Self, Name, Pid, Meta}), - State#state{ - pending_forwards = - (State#state.pending_forwards)#{ - Ref => {register, Name, Pid, Meta, From} - } - }; - false -> - stash_pending_register(Name, Pid, Meta, From, State) - end - end. - -%% Route an unregister (or unsubscribe — the reserved subscription name) to wherever it -%% must go, mirroring route_register. Returns a handle_call reply tuple. The bound pid -%% is captured *before* any optimistic row_delete so the leader can seed the durable -%% clear and the conflict trail (§5.6): -%% - Leader: optimistically remove the local row (a caller-side whereis_name/1 on this -%% node sees the delete at once) and park the durable removal in the group-commit -%% buffer with the caller as origin — answered `ok` on commit. A batch that fails -%% salvages its removes, so `ok` means "accepted and re-driven", never "dropped". -%% - Follower, reachable leader: delete the local row and forward as a *tracked* -%% request stashed under Ref (a dropped forward or deposed leader is recovered by -%% redrive_unregs/1 — a pid-guarded retract to the then-current leader). -%% - Follower, unreachable leader / no leader: answer `ok` now and stash the intent; -%% it is re-driven once a leader is reachable (CP — the commit waits, the intent is -%% never lost). -route_unregister( - LogicalName, From, State = #state{leader = Leader, member_id = Self, names_tab = Tab} -) -> - ReleasedPid = lookup_name(Tab, LogicalName), - if - Leader =:= Self -> - {noreply, - enqueue_op( - {remove, LogicalName, ReleasedPid, {unreg, From}}, - row_delete(State, LogicalName) - )}; - Leader =/= undefined -> - State1 = row_delete(State, LogicalName), - case member_reachable(Leader) of - true -> - Ref = make_ref(), - cast_to_member(Leader, {unregister_req, Ref, Self, LogicalName}), - {noreply, State1#state{ - pending_forwards = (State1#state.pending_forwards)#{ - Ref => {unregister, LogicalName, ReleasedPid, From} - } - }}; - false -> - {reply, ok, stash_pending_unreg(LogicalName, ReleasedPid, State1)} - end; - true -> - {reply, ok, - stash_pending_unreg(LogicalName, ReleasedPid, row_delete(State, LogicalName))} - end. - -%% Stash a registration that could not be routed (no reachable leader), tagged with -%% the wall-clock ms it arrived so the periodic prune can drop it once its caller has -%% (about) timed out. -stash_pending_register(Name, Pid, Meta, From, State = #state{pending_registers = PR}) -> - Now = erlang:system_time(millisecond), - State#state{pending_registers = [{Now, Name, Pid, Meta, From} | PR]}. - -%% Re-drive every stashed registration now that a leader may be known — called on the -%% leadership-establishing events (assuming leadership, applying a leader's snapshot). -%% Each entry is re-routed: it commits (we are leader), forwards (a follower with a -%% reachable leader), or re-stashes (still no reachable leader). An entry whose caller -%% has already timed out (older than `register_timeout`) is dropped without a reply. -redrive_registers(State = #state{pending_registers = []}) -> - State; -redrive_registers(State = #state{pending_registers = PR, config = Config}) -> - TTL = dgen_config:register_timeout(Config), - Now = erlang:system_time(millisecond), - %% Clear first; route_register re-stashes any that still cannot proceed. Oldest - %% first (the list is newest-first) for rough FIFO fairness among waiters. - lists:foldl( - fun({At, Name, Pid, Meta, From}, S) -> - case Now - At >= TTL of - true -> S; - false -> route_register(Name, Pid, Meta, From, S) - end - end, - State#state{pending_registers = []}, - lists:reverse(PR) - ). - -%% Drop stashed registrations whose callers have (about) timed out — the periodic -%% bound on `pending_registers` so a prolonged no-leader window (a partition) cannot -%% grow it without limit. No reply is sent (the caller has moved on). -prune_pending_registers(State = #state{pending_registers = []}) -> - State; -prune_pending_registers(State = #state{pending_registers = PR, config = Config}) -> - TTL = dgen_config:register_timeout(Config), - Now = erlang:system_time(millisecond), - State#state{pending_registers = [E || {At, _, _, _, _} = E <- PR, Now - At < TTL]}. - -%% Append a write op to the pending queue (FIFO), then try to start a commit. If -%% one is already in flight, the op just accumulates and rides the next batch -%% (started when the in-flight commit completes). -enqueue_op(Op, State = #state{pending = Pending, num_pending = NumPending}) -> - maybe_start_commit(State#state{ - pending = queue:in(Op, Pending), num_pending = NumPending + 1 - }). - -%% Start a group commit for the pending buffer if appropriate. At most one commit -%% runs at a time; ops that arrive while it is in flight accumulate and ride the -%% next batch. Plans the batch (in arrival order), spawns a dgen_transaction -%% worker to commit it off this process's loop, and stashes the plan to apply when -%% the worker reports back. -maybe_start_commit(State = #state{committing = C}) when C =/= undefined -> - %% A commit is in flight; this batch waits for it. - State; -maybe_start_commit(State = #state{num_pending = 0}) -> - State; -maybe_start_commit(State = #state{member_id = Self, leader = Leader, pending = Pending}) when - Leader =/= Self --> - %% Not (or no longer) the leader — reject every buffered registration and - %% set_metadata (caller retries against the new leader); forward buffered - %% removes/retracts to the new leader as pid-guarded retracts so an unregister - %% (or a strict-mode retraction whose caller already heard `no`) is not silently - %% lost with the deposition. Downs are dropped: the new leader monitors the - %% pids itself and observes their exits directly. - lists:foreach( - fun - ({add, _N, _P, _Meta, Origin}) -> - deliver_reply(Origin, reject_value(Origin)); - ({set_meta, _N, _I, _D, Origin}) -> - deliver_reply(Origin, reject_value(Origin)); - ({remove, N, P, Origin}) -> - %% Forward the pid-guarded clear, and answer the tracked caller `ok` - %% (the forwarded retract carries the removal onward). - case is_pid(P) of - true -> forward_retract(Leader, N, P); - false -> ok - end, - case Origin of - undefined -> ok; - _ -> deliver_reply(Origin, reject_value(Origin)) - end; - ({retract, N, P}) -> - forward_retract(Leader, N, P); - (_) -> - ok - end, - queue:to_list(Pending) - ), - State#state{pending = queue:new(), num_pending = 0}; -maybe_start_commit(State) -> - #state{ - member_id = Self, - epoch = Epoch, - tenant = Tenant, - tuid = Tuid, - names_tab = Tab, - name_to_ref = NTR, - pending = Pending, - num_pending = NumPending, - last_version = LastVersion, - config = Config - } = State, - %% Take up to `commit_batch_size` oldest ops (front of the queue); the rest ride - %% the following batch. num_pending keeps the size O(1), so no length/1 scan. - %% Bounding the batch keeps a burst (e.g. a departing node's DOWN flood) from - %% coalescing into one enormous commit whose O(batch) plan/apply/broadcast would - %% freeze the loop — see dgen_config:commit_batch_size/1. - BatchSize = min(NumPending, dgen_config:commit_batch_size(Config)), - {ThisBatch, Rest} = take_n(Pending, BatchSize), - State1 = State#state{pending = Rest, num_pending = NumPending - BatchSize}, - %% plan_batch resolves each op's binding lazily (its own batch-local overlay, falling - %% back to a point ets:lookup) rather than needing a pre-seeded map of the whole - %% registry — see plan_batch's doc. Planning cost is O(batch), not O(registry size). - Plan = plan_batch(ThisBatch, Tab, NTR, Epoch), - #{dbop := DBOp, replies := RepliesRev} = Plan, - case map_size(DBOp) =:= 0 of - true -> - %% Nothing durable changed — only rejected registrations. No commit is - %% needed (the batch neither advances the version nor needs fencing). - %% Answer them and continue with whatever did not fit in this batch. - send_replies(lists:reverse(RepliesRev)), - maybe_start_commit(State1); - false -> - Ref = make_ref(), - %% Pin last_version as the read version to skip the GRV (cached GRV). - %% Safe: if it is stale, the worker's leader-key read raises too-old and - %% dgen_transaction retries with a fresh GRV (a body-level retry); a - %% fresh start (last_version = undefined) uses a fresh GRV. The worker is - %% monitored so its death without a result still resolves the commit. The - %% commit's only write is the fenced version bump; the name delta is - %% applied in memory (and broadcast) when the worker reports back. - {ok, {_Pid, MRef}} = dgen_registry_names:start_commit( - Tenant, - Tuid, - Self, - #{owner => self(), ref => Ref, epoch => Epoch, read_version => LastVersion} - ), - State1#state{committing = #{ref => Ref, mref => MRef, plan => Plan}} - end. - -%% Apply a committed batch: bind/clear the touched names in the local replica (a -%% delta over the table — only DBOp names are touched, so a concurrent optimistic -%% unregister of an untouched name is not clobbered), (de)monitor pids, answer the -%% registration callers, replicate, and record the committed version (for the -%% read-version-pinning optimization). -apply_committed_plan(Plan, Version, State) -> - #{ - dbop := DBOp, - names := WNames, - meta := WMeta, - replies := RepliesRev, - bcasts := BcastsRev, - released := Released, - direct_meta := DirectMeta - } = Plan, - #state{ - member_id = Self, - name_to_ref = NTR0, - members = Members, - recently_released = Rel0, - applied_version = PrevVersion - } = State, - %% Apply the durable delta to the local replica (and the inverted index). Only the - %% names in DBOp are touched, so a concurrent optimistic unregister of an untouched - %% name (already removed from the table) is never clobbered. - State0 = apply_dbop(State, DBOp, WNames, WMeta), - NTR1 = apply_monitor_ops(DBOp, WNames, NTR0), - %% Replicate to followers first, so the replicate_sync below (FIFO behind it) - %% is seen by a follower only after it already holds the batch's bindings. - %% - %% The whole batch travels as **one** `{names_batch, …}` message carrying this - %% batch's commit Version, the predecessor version (our applied_version before - %% this batch) and our id, so followers apply only contiguous batches and a - %% member that missed one detects the gap and resyncs (see apply_bcast/6). - %% - %% One message per batch, not one per changed name: a batch has to be atomic on - %% the wire. When it was N messages sharing a version, the first to arrive - %% advanced the receiver's applied_version to that version and the rest matched - %% the "already mid-batch" clause — so a member that received a *strict subset* - %% (a link dropping mid-batch) ended up reporting the full version while holding - %% only part of it. Nothing could detect that afterwards: gap detection compares - %% versions, and the versions matched. It also broke the "same version implies - %% same content" property the freshest-wins handoff gather relies on (§5.7), so a - %% handoff could adopt the deficient replica and fan it out. Delivered whole or - %% not at all, a lost batch always leaves a version discontinuity, which is - %% exactly what apply_bcast/6 already knows how to repair. - broadcast_batch(Members, lists:reverse(BcastsRev), PrevVersion, Version, Self), - Now = erlang:system_time(millisecond), - %% Trail entries are {Name, Pid} pairs (§5.6 — see the recently_released doc). - Rel1 = lists:foldl(fun(NamePid, Acc) -> Acc#{NamePid => Now} end, Rel0, Released), - State1 = State0#state{ - name_to_ref = NTR1, - last_version = Version, - applied_version = Version, - recently_released = Rel1 - }, - %% Answer callers. Forwarded registrations and all rejections are answered - %% immediately (a forwarded `yes` is already two-holder — leader + forwarding - %% follower). A *direct* `yes` (origin on the leader's own node) has only the - %% leader as a holder, so it waits for a follower to confirm a replica before - %% acking — the two-holder invariant. See the moduledoc for the caveats. - {DirectYes, Immediate} = lists:partition(fun is_direct_yes/1, lists:reverse(RepliesRev)), - lists:foreach( - fun({Origin, Result}) -> deliver_committed_reply(Origin, Result, Version) end, Immediate - ), - %% Enrich each direct `yes` with its {Name, Pid} (from the plan) so the replicate - %% path can roll the binding back under strict_replication. - EnrichedDirect = [ - {Origin, Name, Pid} - || {Origin, yes} <- DirectYes, {Name, Pid} <- [maps:get(Origin, DirectMeta)] - ], - %% Fire presence notifications for this batch (§4.9): the replica now reflects the - %% committed batch, so the watch-set deltas are computed against it and delivered to - %% each subscription's notify targets. Advances sub_matches. - State2 = fire_presence(DBOp, State1), - confirm_direct(EnrichedDirect, Version, State2). - -%% Deliver presence notifications for a just-applied committed batch (leader only, §4.9) -%% and advance the leader-only `sub_matches` / `notify_matches`. The batch's durable -%% delta `DBOp` names every registration that changed, and `State` already reflects the -%% post-batch replica (apply_dbop ran first). For each subscription two things happen, -%% both driven by diffing the changed names: -%% - watch side: a name entering/leaving the watch set yields a `joined`/`left` delta, -%% delivered to the subscription's *continuing* notify targets; -%% - notify side: a pid entering the notify set is a **new** viewer and is sent the -%% full current watch snapshot, so a notifier that registered after the subscription -%% still learns who is already present. -fire_presence(_DBOp, State = #state{subs = Subs}) when map_size(Subs) =:= 0 -> - State; -fire_presence(DBOp, State = #state{names_tab = Tab, subs = Subs}) -> - RealNames = maps:keys(DBOp), - maps:fold( - fun(SubId, {Watch, Notify}, S) -> - diff_subscription(SubId, Watch, Notify, RealNames, Tab, S) - end, - State, - Subs - ). - -%% Seed a subscription (on the elector's push): compute its full watch and notify sets -%% from the current replica, record them, and deliver the watch set as an initial -%% `joined` snapshot to the current notify targets. A re-subscribe (upsert with changed -%% queries) reseeds and re-delivers. (Notify targets that appear *later* are handled by -%% diff_subscription, not here.) -seed_subscription( - SubId, - State = #state{ - subs = Subs, names_tab = Tab, inv_index = Inv, sub_matches = SM, notify_matches = NM - } -) -> - case Subs of - #{SubId := {Watch, Notify}} -> - WatchMatches = query_match_names(Watch, Tab, Inv), - NotifyMatches = query_match_names(Notify, Tab, Inv), - send_presence(SubId, snapshot_events(WatchMatches), pidset(NotifyMatches)), - State#state{ - sub_matches = SM#{SubId => WatchMatches}, - notify_matches = NM#{SubId => NotifyMatches} - }; - _ -> - State - end. - -%% Diff one subscription against the batch's row changes and fire its notifications, -%% advancing both tracked sets. The watch diff yields `joined`/`left` deltas; the notify -%% diff yields the current notify-target set. A pid that newly *entered* the notify set -%% (a viewer that just appeared) gets the full current watch snapshot; every *continuing* -%% notify target gets the watch delta. Computed on pid-sets, so a pid already a target -%% under another name is not treated as new (no duplicate snapshot). -diff_subscription(SubId, Watch, Notify, RealNames, Tab, State) -> - #state{sub_matches = SM, notify_matches = NM} = State, - {WatchEvents, WatchAfter} = diff_query_matches(Watch, RealNames, maps:get(SubId, SM, #{}), Tab), - NotifyBefore = maps:get(SubId, NM, #{}), - {_NotifyEvents, NotifyAfter} = diff_query_matches(Notify, RealNames, NotifyBefore, Tab), - AfterPids = pidset(NotifyAfter), - NewTargets = AfterPids -- pidset(NotifyBefore), - ContinuingTargets = AfterPids -- NewTargets, - send_presence(SubId, WatchEvents, ContinuingTargets), - send_presence(SubId, snapshot_events(WatchAfter), NewTargets), - State#state{sub_matches = SM#{SubId => WatchAfter}, notify_matches = NM#{SubId => NotifyAfter}}. - -%% Diff a query's membership over the batch's changed names: fold each changed name's -%% post-batch match (present in the replica *and* satisfying the query) against whether it -%% was a member before, yielding `joined`/`left` events and the updated `#{Name => Pid}` -%% membership. Used for both the watch and notify queries. -diff_query_matches(Query, RealNames, Before, Tab) -> - lists:foldl( - fun(Name, {EvAcc, MatchAcc}) -> - WasPid = maps:get(Name, Before, undefined), - NowMatch = - case ets:lookup(Tab, Name) of - [{_N, RowPid, Index, _Data} | _] -> - case query_satisfies(Query, Index) of - true -> {true, RowPid}; - false -> false - end; - [] -> - false - end, - case {WasPid, NowMatch} of - {undefined, {true, Pid}} -> - {[{joined, Name, Pid} | EvAcc], MatchAcc#{Name => Pid}}; - {undefined, false} -> - {EvAcc, MatchAcc}; - {WasPid, {true, WasPid}} -> - {EvAcc, MatchAcc}; - {OldPid, {true, NewPid}} -> - {[{joined, Name, NewPid}, {left, Name, OldPid} | EvAcc], MatchAcc#{ - Name => NewPid - }}; - {OldPid, false} -> - {[{left, Name, OldPid} | EvAcc], maps:remove(Name, MatchAcc)} - end - end, - {[], Before}, - RealNames - ). - -%% The current watch set (`#{Name => Pid}`) as a list of `joined` events — the initial -%% snapshot payload. -snapshot_events(Matches) -> - [{joined, Name, Pid} || {Name, Pid} <- maps:to_list(Matches)]. - -%% The distinct pids of a `#{Name => Pid}` membership map (a pid registered under several -%% matching names is one recipient). -pidset(Matches) -> - lists:usort(maps:values(Matches)). - -%% Send one `{dgen_presence, SubId, Events}` message to each recipient pid. Keyed by the -%% subscription's application-supplied id, so a durable subscriber re-addresses its feed -%% by the same stable key across restarts. A no-op with no events or no recipients. -send_presence(_SubId, [], _Pids) -> - ok; -send_presence(_SubId, _Events, []) -> - ok; -send_presence(SubId, Events, Pids) -> - Msg = {dgen_presence, SubId, Events}, - lists:foreach(fun(Pid) -> Pid ! Msg end, Pids). - -is_direct_yes({{local, _From}, yes}) -> true; -is_direct_yes(_Other) -> false. - -%% Replicate-before-ack for direct registrations (§5.5, §8). Waits for -%% `register_replicas` *distinct* follower acks (bounded by the follower count) before -%% acking `yes`. With no follower to replicate to (or replicas disabled) there is no -%% second holder to wait for — nothing survives the sole node anyway — so ack now. -%% Otherwise broadcast a replicate_sync and resolve on the acks or, on timeout, the -%% `strict_replication` policy (degrade-open or fail-closed). `Direct` entries are -%% `{Origin, Name, Pid}`. -confirm_direct([], _Version, State) -> - State; -confirm_direct( - Direct, - Version, - State = #state{members = Members, member_id = Self, pending_acks = PA, config = Config} -) -> - Needed = min(dgen_config:register_replicas(Config), map_size(Members)), - case Needed =< 0 of - true -> - deliver_direct(Direct), - State; - false -> - BatchRef = make_ref(), - %% The sync carries the batch's commit version so a follower only acks - %% once it has actually applied up to this batch — a follower that - %% gap-skipped the broadcasts (awaiting resync) does not hold the - %% bindings and must not be counted as a holder. - broadcast_to_peers(Members, {replicate_sync, BatchRef, Self, Version}), - TimerRef = arm( - dgen_config:replicate_timeout(Config), {replicate_timeout, BatchRef} - ), - State#state{ - pending_acks = PA#{BatchRef => {Direct, TimerRef, Needed, #{}}} - } - end. - -deliver_direct(Direct) -> - lists:foreach(fun({Origin, _Name, _Pid}) -> deliver_reply(Origin, yes) end, Direct). - -%% strict_replication fail-closed: reject each unreplicated direct registration and -%% retract its just-applied binding (a pid-guarded clear that rides the commit -%% pipeline, so it is durably fenced and broadcast to followers). The caller never -%% sees a `yes` that was not replicated. -reject_and_retract_direct(Direct, State) -> - lists:foldl( - fun({Origin, Name, Pid}, S) -> - deliver_reply(Origin, no), - enqueue_op({retract, Name, Pid}, S) - end, - State, - Direct - ). - -%% Surface a degrade-open event (§8): a direct registration acked `yes` with fewer -%% than `register_replicas` replica acks. Emitted as telemetry (if the optional -%% `telemetry` application is present) and logged. -notify_degrade_open(Needed, Got, Count) -> - logger:warning( - "dgen_registry: degrade-open — acked ~b direct registration(s) leader-only " - "(needed ~b replica ack(s), got ~b) on replicate timeout (§8).", - [Count, Needed, Got] - ), - emit_telemetry( - [dgen_registry, register, degrade_open], - #{count => Count}, - #{needed => Needed, got => Got} - ). - -%% Optional telemetry: emit only if the `telemetry` library is available, so it stays -%% a soft dependency (the registry does not list it in `deps`). -%% -%% Whether it is available is resolved **once** and cached. `code:ensure_loaded/1` -%% short-circuits on an already-loaded module, but not on a missing one — it falls -%% through to a synchronous call to `code_server`. So the configuration that pays -%% for this check is exactly the common one (no telemetry), on every event, forever. -%% -%% Found by the eta framework rather than by profiling, and the *scheduling* cost is -%% what made it visible: `code_server` is a process the simulation does not control, -%% so a member emitting an event blocked on something outside the schedule and the -%% run stopped being reproducible. Degrade-open only fires when replicas cannot ack, -%% which is why it appeared under injected message loss and nowhere else. -%% -%% Resolved once per VM, in the same spirit as `dgen_config`'s backend lookup: a -%% `telemetry` loaded *after* the first event is not picked up. It is an application -%% dependency, present or absent for the life of the node. --define(TELEMETRY_KEY, {?MODULE, telemetry_available}). - -emit_telemetry(Event, Measurements, Metadata) -> - case telemetry_available() of - true -> apply(telemetry, execute, [Event, Measurements, Metadata]); - false -> ok - end. - -telemetry_available() -> - case persistent_term:get(?TELEMETRY_KEY, undefined) of - undefined -> - Available = code:ensure_loaded(telemetry) =:= {module, telemetry}, - persistent_term:put(?TELEMETRY_KEY, Available), - Available; - Available -> - Available - end. - -%% Reject a batch that did not commit (fenced or errored): answer every op with its -%% rejection verdict (`no` for a registration, `{error, no_leader}` for a set_metadata -%% — the leader changed mid-flight, so the caller should retry). No durable or monitor -%% state changed. -reject_plan(#{replies := RepliesRev}, State) -> - lists:foreach(fun({Origin, _}) -> deliver_reply(Origin, reject_value(Origin)) end, RepliesRev), - State. - -reject_value({meta, _From}) -> {error, no_leader}; -reject_value({forward_meta, _MemberId, _Ref}) -> {error, no_leader}; -%% A rejected batch's removes are salvaged and re-driven (salvage_failed_plan), so -%% `ok` remains the truthful answer for an unregister even when its batch failed. -reject_value({unreg, _From}) -> ok; -reject_value({forward_unreg, _MemberId, _Ref}) -> ok; -reject_value(_Origin) -> no. - -%% Answer a write op's origin. A direct (local) registration, set_metadata, or -%% unregister is answered with gen_server:reply/2; a forwarded one with the matching -%% reply cast ({register_reply} / {set_meta_reply} / {unregister_reply}) to the -%% forwarding member, which then answers its own caller. -deliver_reply({local, From}, Result) -> - gen_server:reply(From, Result); -deliver_reply({meta, From}, Result) -> - gen_server:reply(From, Result); -deliver_reply({unreg, From}, Result) -> - gen_server:reply(From, Result); -deliver_reply({forward, MemberId, Ref}, Result) -> - cast_to_member(MemberId, {register_reply, Ref, Result}); -deliver_reply({forward_meta, MemberId, Ref}, Result) -> - cast_to_member(MemberId, {set_meta_reply, Ref, Result}); -deliver_reply({forward_unreg, MemberId, Ref}, _Result) -> - cast_to_member(MemberId, {unregister_reply, Ref}). - -%% Post-commit reply delivery: a forwarded registration's reply is stamped with the -%% batch's commit version so the forwarding follower can version-guard its `yes` ack -%% (see the {register_reply, _, _, _} handler). Every other origin (and every -%% rejection path, which has no commit version) uses the plain deliver_reply/2. -deliver_committed_reply({forward, MemberId, Ref}, Result, Version) -> - cast_to_member(MemberId, {register_reply, Ref, Result, Version}); -deliver_committed_reply(Origin, Result, _Version) -> - deliver_reply(Origin, Result). - -%% Follower: the leader these were forwarded to is no longer current. Reject the -%% register and set_metadata forwards so their callers retry against the new leader -%% (registrations `no`, including the gap-deferred `yes` acks whose resync will never -%% arrive from the deposed leader; set_metadata `{error, no_leader}`). Unregister -%% forwards are **kept** — their intent is idempotent and must not be lost, so -%% redrive_unregs re-drives them to the new leader rather than failing the caller. -reject_forwards(State = #state{pending_forwards = PF, deferred_yes = Deferred}) -> - Kept = maps:fold( - fun - (_Ref, {register, _Name, _Pid, _Meta, From}, Acc) -> - gen_server:reply(From, no), - Acc; - (_Ref, {set_meta, From}, Acc) -> - gen_server:reply(From, {error, no_leader}), - Acc; - (Ref, {unregister, _, _, _} = Unreg, Acc) -> - Acc#{Ref => Unreg} - end, - #{}, - PF - ), - lists:foreach(fun({_Version, From}) -> gen_server:reply(From, no) end, Deferred), - State#state{pending_forwards = Kept, deferred_yes = []}. - -%% Resolve the leader's answer to a forwarded registration (see the -%% {register_reply, _, _, _} handler doc). `yes` at-or-behind our applied version: -%% the FIFO-ordered broadcast already applied the binding, so just answer — the -%% normal path. `yes` ahead of it: our replica is gapped (the broadcast was refused -%% pending a resync), so defer the ack until applied_version reaches the batch's -%% version — answering now would create a second holder the freshest-wins gather -%% cannot see (§5.5/§5.7). The gap-refused broadcast already requested the resync; -%% re-request here as a belt-and-braces (request_resync no-ops while one is -%% outstanding). -%% -%% **The at-or-behind branch must not write the row.** It used to `row_insert` the -%% binding, on the reasoning that the insert is idempotent over a broadcast that has -%% already applied it. It is not: a group commit may bind *and clear* one name — a -%% register and an unregister of it landing in the same batch — and the broadcast -%% carries both ops in order, so the replica correctly ends up without the binding. -%% Re-inserting it here resurrects exactly what the batch removed, and the version -%% guard makes that fire precisely when the batch has been applied. The member is -%% then holding a binding no other member has, at the same applied_version as all of -%% them, which is the one divergence gap detection cannot see (§4.5) and the one the -%% freshest-wins gather may fan out (§5.7). -%% -%% Passing the guard means the replica has applied this batch (or a snapshot at or -%% past its version), and that content is authoritative — including a later op in the -%% same batch. `flush_deferred/1` already answers on exactly that reasoning, and this -%% branch is now consistent with it. Found by `eta_run` on an unfaulted 3-member -%% cluster; see the regression test in dgen_registry_sim_test.exs. -handle_register_reply(Ref, Result, Version, State = #state{pending_forwards = PF}) -> - case maps:take(Ref, PF) of - {{register, LogicalName, Pid, {Index, Data}, From}, PF1} -> - State1 = State#state{pending_forwards = PF1}, - case Result of - yes when Version =:= legacy -> - %% A pre-version leader (rolling upgrade): there is no version to - %% guard on, and its per-name broadcasts are a wire format this - %% member no longer applies, so this insert is the only thing that - %% would bind the row. Left exactly as it was. - gen_server:reply(From, yes), - row_insert(State1, LogicalName, Pid, Index, Data); - yes when Version =< State1#state.applied_version -> - gen_server:reply(From, yes), - State1; - yes -> - State2 = - case State1#state.leader of - undefined -> State1; - Leader -> request_resync(Leader, State1) - end, - State2#state{ - deferred_yes = [{Version, From} | State2#state.deferred_yes] - }; - no -> - gen_server:reply(From, no), - State1 - end; - _ -> - %% Unknown Ref — already answered (e.g. rejected on a leadership change). - State - end. - -%% Release gap-deferred forwarded-`yes` acks whose batch version the replica has now -%% genuinely applied (the resync snapshot, or a contiguous broadcast run, advanced -%% applied_version past them). The binding itself is guaranteed present: reaching -%% the version contiguously means the batch's own broadcast was applied, and a -%% snapshot at-or-past it carries the committed binding. -flush_deferred(State = #state{deferred_yes = []}) -> - State; -flush_deferred(State = #state{deferred_yes = Deferred, applied_version = Applied}) -> - {Due, Rest} = lists:partition(fun({Version, _From}) -> Version =< Applied end, Deferred), - lists:foreach(fun({_Version, From}) -> gen_server:reply(From, yes) end, Due), - State#state{deferred_yes = Rest}. - -%% Stash an unregister accepted while no leader was reachable (nothing to retract if -%% the name was locally unbound — matching the old nets-to-nothing semantics); it is -%% re-driven by redrive_unregs/1 once a leader is known. -stash_pending_unreg(_Name, undefined, State) -> - State; -stash_pending_unreg(Name, Pid, State = #state{pending_unregs = Pending}) -> - State#state{pending_unregs = [{Name, Pid} | Pending]}. - -%% Re-drive every stashed unregister — the follower-side counterpart of the leader's -%% salvage_failed_plan: forwarded removals whose reply never came (dropped cast, -%% deposed leader) and removals accepted while no leader was reachable are handed to -%% the current leader as **pid-guarded retracts** (or enqueued directly if this -%% member is now the leader itself), and any still-waiting caller is answered `ok`. -%% Pid-guarding makes the re-drive idempotent and reissue-safe: a name since -%% re-registered to a different pid is never clobbered, and a removal the old leader -%% did commit after all nets to nothing. Called on every snapshot apply and on -%% assuming leadership — the events that establish who the leader is. -redrive_unregs(State = #state{pending_forwards = PF, pending_unregs = Pending}) -> - {Unregs, RestPF} = take_unreg_forwards(PF), - case Unregs =:= [] andalso Pending =:= [] of - true -> - State; - false -> - redrive_unregs(State, Unregs, RestPF) - end. - -%% No leader to hand these to yet — leave everything stashed (RestPF/Pending -%% untouched) for the next snapshot/leadership event. -redrive_unregs(State = #state{leader = undefined}, _Unregs, _RestPF) -> - State; -redrive_unregs(State = #state{leader = Leader, member_id = Self}, Unregs, RestPF) -> - Entries = - Unregs ++ [{Name, Pid, undefined} || {Name, Pid} <- State#state.pending_unregs], - State1 = State#state{pending_forwards = RestPF, pending_unregs = []}, - lists:foldl( - fun({Name, Pid, From}, S) -> - From =/= undefined andalso gen_server:reply(From, ok), - case is_pid(Pid) of - false -> - %% Nothing was bound locally at call time — nothing to retract. - S; - true when Leader =:= Self -> - enqueue_op({retract, Name, Pid}, S); - true -> - forward_retract(Leader, Name, Pid), - S - end - end, - State1, - Entries - ). - -%% Split `pending_forwards` into its unregister entries (as {Name, Pid, From} for the -%% re-drive) and the remaining register/set_meta forwards (kept in flight — a -%% snapshot refresh that did not change leadership must not disturb them). -take_unreg_forwards(PF) -> - maps:fold( - fun - (_Ref, {unregister, Name, Pid, From}, {Acc, Rest}) -> - {[{Name, Pid, From} | Acc], Rest}; - (Ref, Other, {Acc, Rest}) -> - {Acc, Rest#{Ref => Other}} - end, - {[], #{}}, - PF - ). - -%% Dequeue exactly N ops from the front of the queue (oldest first), returning -%% {OpsInOrder, RestQueue}. The caller guarantees N =< the queue's length -%% (tracked in num_pending), so the queue never empties early. -take_n(Queue, N) -> - take_n(Queue, N, []). - -take_n(Queue, 0, Acc) -> - {lists:reverse(Acc), Queue}; -take_n(Queue, N, Acc) -> - {{value, Op}, Queue1} = queue:out(Queue), - take_n(Queue1, N - 1, [Op | Acc]). - -%% Fold the ordered ops into a plan: the working name overlay, the net durable delta -%% (`dbop`: Name => {set, PidNode} | {meta, PidNode} | clear, last write wins), the -%% deferred replies, and the follower broadcasts (both newest-first). `wntr` tracks each -%% name's current monitor ref so a `down` honours the ref-match guard; a name (re)bound -%% within this batch carries a `{pending, Pid}` placeholder that never equals a real -%% ref, so a stale DOWN for it is skipped. -%% -%% `names` is seeded **empty**, not with the whole registry: every plan_op clause only -%% ever needs the binding for the one name it touches, resolved via `seed_lookup/3`, -%% which checks this batch's own overlay first and falls back to a point `ets:lookup` -%% for a name nothing earlier in the batch has touched yet. This keeps planning cost -%% proportional to the batch, not the registry. `tab` threads the table through for -%% that fallback. -plan_batch(Ops, Tab, NTR, Epoch) -> - Init = #{ - tab => Tab, - names => #{}, - meta => #{}, - wntr => NTR, - dbop => #{}, - replies => [], - bcasts => [], - released => [], - direct_meta => #{}, - %% The epoch the plan was built under: the commit-result handler refuses to - %% apply a plan whose epoch is no longer current (leadership moved while the - %% worker was in flight), and the broadcasts are stamped with it. - epoch => Epoch, - %% The raw ops, kept so a failed batch's destructive ops (removes/retracts/ - %% downs) can be salvaged (salvage_failed_plan) instead of silently dropped. - ops => Ops - }, - lists:foldl(fun(Op, Acc) -> plan_op(Op, Epoch, Acc) end, Init, Ops). - -%% Resolve a name's pid "as of partway through this batch": this batch's own overlay -%% (`WN`) for a name an earlier op in the batch already touched, falling back to a point -%% `ets:lookup` for the committed pre-batch truth otherwise. The overlay uses the atom -%% `removed` to mark a name an earlier op in *this* batch explicitly cleared — needed -%% because only `unregister`'s optimistic delete updates ETS immediately (see the -%% moduledoc's "Storage" section); a `retract` or `down` processed earlier in the same -%% batch does not touch ETS until the batch commits, so without this marker a later op -%% in the same batch would see the stale, still-bound row. -seed_lookup(WN, Tab, Name) -> - case WN of - #{Name := removed} -> undefined; - #{Name := Pid} -> Pid; - _ -> lookup_name(Tab, Name) - end. - -plan_op({add, Name, Pid, {Index, Data} = Meta, Origin}, Epoch, Acc) -> - #{ - tab := Tab, - names := WN, - meta := WMeta, - wntr := WNTR, - dbop := DB, - replies := Rs, - bcasts := Bs, - direct_meta := DM - } = Acc, - case seed_lookup(WN, Tab, Name) of - Taken when is_pid(Taken), Taken =/= Pid -> - %% Held by a *different* live pid — reject (the singleton contract). - Acc#{replies := [{Origin, no} | Rs]}; - _Free_or_same -> - %% Free (`undefined`) or already this same pid: bind it and answer `yes`. - %% Re-registering the *same* pid under the *same* name is an idempotent - %% success, not a conflict — so a caller whose register call timed out (or - %% raced a failover) but whose registration actually committed can simply - %% redrive and get a decisive `yes` for its own binding, rather than a - %% confusing `no`. It rides the normal bind path (a same-pid re-register - %% re-writes the identical row, refreshes the monitor, and re-applies the - %% call's metadata — use set_metadata/2 to change metadata independently). - %% - %% Record {Name, Pid} for a direct-origin reg so the replicate-before-ack - %% path (confirm_direct) can roll the binding back under strict_replication. - DM1 = - case Origin of - {local, _From} -> DM#{Origin => {Name, Pid}}; - _ -> DM - end, - Acc#{ - names := WN#{Name => Pid}, - meta := WMeta#{Name => Meta}, - wntr := WNTR#{Name => {pending, Pid}}, - dbop := DB#{Name => {set, node(Pid)}}, - replies := [{Origin, yes} | Rs], - bcasts := [{name_registered, Name, Pid, Index, Data, Epoch} | Bs], - direct_meta := DM1 - } - end; -plan_op({set_meta, Name, Index, Data, Origin}, Epoch, Acc) -> - #{tab := Tab, names := WN, meta := WMeta, dbop := DB, replies := Rs, bcasts := Bs} = Acc, - case seed_lookup(WN, Tab, Name) of - undefined -> - %% No binding to attach metadata to. - Acc#{replies := [{Origin, {error, not_registered}} | Rs]}; - Pid -> - %% Rewrite the row's metadata. Keep a {set,…} dbop (a registration earlier - %% in this same batch) so the new pid is still monitored; otherwise use a - %% {meta,…} dbop that writes the row without disturbing the existing monitor. - DBVal = - case maps:get(Name, DB, undefined) of - {set, _} = SetOp -> SetOp; - _ -> {meta, node(Pid)} - end, - Acc#{ - meta := WMeta#{Name => {Index, Data}}, - dbop := DB#{Name => DBVal}, - replies := [{Origin, ok} | Rs], - bcasts := [{metadata_set, Name, Index, Data, Epoch} | Bs] - } - end; -plan_op({remove, Name, ReleasedPid, Origin}, Epoch, Acc) -> - #{ - tab := Tab, - names := WN, - wntr := WNTR, - dbop := DB, - replies := Rs, - bcasts := Bs, - released := Rel - } = Acc, - %% Clear durably if the name was bound at enqueue (ReleasedPid is a pid) or is bound - %% now, as of this point in the batch (seed_lookup — overlay first, ETS fallback). - %% ReleasedPid covers the leader's optimistic removal (already gone from ETS); the - %% seed_lookup covers a name added earlier in this same batch (ETS would not show - %% that yet — only unregister's own optimistic delete updates ETS immediately). - %% Record the released {Name, Pid} in the conflict-detector trail so a lagging - %% member reporting it is not later mistaken for a divergence. The origin (a - %% tracked unregister call, if any) is answered `ok` either way — clearing an - %% already-unbound name is an idempotent no-op, not a failure. - Rs1 = unreg_reply(Origin, Rs), - WasBound = is_pid(ReleasedPid), - Current = seed_lookup(WN, Tab, Name), - %% Guard the clear against the *current* holder. ReleasedPid is captured at - %% arrival, but this plan runs later — and the arrival-time optimistic - %% row_delete frees the name in ETS immediately, so a register that was already - %% enqueued (parked behind an in-flight commit) plans against the freed table, - %% answers `yes`, and binds a new pid before this remove is planned. Clearing - %% unconditionally then deletes that new holder: one unregister frees the name - %% twice — once optimistically for the parked register, once durably against - %% the pid it never targeted — and three `yes` acks for one name with a single - %% unregister between them has no legal serialization (Guarantee 1). Found by - %% the DST harness's end-of-run ack-history fold (`check_final`, seed 5 under - %% loss); invisible to every replica comparison because all members apply the - %% same wrong batch and agree. - %% - %% So: a removal whose captured target is a pid only clears while the name's - %% current holder (batch overlay first, ETS fallback) is that pid or nobody - %% (nobody = the optimistic delete, which is this removal's own footprint). A - %% different current pid means the target binding is already gone — answer the - %% idempotent `ok` and keep the new holder. A capture of `undefined` keeps its - %% existing meaning: the unregister serializes after whatever bound the name - %% (in-batch add included), which is a legal linearization of "unregister by - %% name", and clears it. - Rebound = WasBound andalso is_pid(Current) andalso Current =/= ReleasedPid, - case (WasBound orelse Current =/= undefined) andalso not Rebound of - true -> - Rel1 = - case WasBound of - true -> [{Name, ReleasedPid} | Rel]; - false -> Rel - end, - %% The broadcast carries the released pid so every member records it in - %% its own copy of the conflict-detector trail (§5.6). - ReleasedOut = - case WasBound of - true -> ReleasedPid; - false -> undefined - end, - Acc#{ - names := WN#{Name => removed}, - wntr := maps:remove(Name, WNTR), - dbop := DB#{Name => clear}, - replies := Rs1, - bcasts := [{name_unregistered, Name, ReleasedOut, Epoch} | Bs], - released := Rel1 - }; - false -> - Acc#{replies := Rs1} - end; -plan_op({retract, Name, Pid}, Epoch, Acc) -> - #{tab := Tab, names := WN, wntr := WNTR, dbop := DB, bcasts := Bs, released := Rel} = Acc, - %% Pid-guarded retract (strict_replication fail-closed): clear the binding only if - %% it is *still* this pid's, so a name re-registered to someone else since the - %% failed reg is not clobbered. Trail the retracted {Name, Pid} so a lagging - %% follower that still holds the transient binding is not later mistaken for a - %% conflict (§5.6). - case seed_lookup(WN, Tab, Name) of - Pid -> - Acc#{ - names := WN#{Name => removed}, - wntr := maps:remove(Name, WNTR), - dbop := DB#{Name => clear}, - bcasts := [{name_unregistered, Name, Pid, Epoch} | Bs], - released := [{Name, Pid} | Rel] - }; - _ -> - %% Binding changed since the failed registration — nothing to retract. - Acc - end; -plan_op({down, Name, Ref}, Epoch, Acc) -> - #{names := WN, wntr := WNTR, dbop := DB, bcasts := Bs} = Acc, - case maps:get(Name, WNTR, undefined) of - Ref -> - %% Ref still matches the live binding — honour the auto-unregister. The - %% pid is dead, so no released pid rides the broadcast (the trail is for - %% *live* releases only). - Acc#{ - names := WN#{Name => removed}, - wntr := maps:remove(Name, WNTR), - dbop := DB#{Name => clear}, - bcasts := [{name_unregistered, Name, undefined, Epoch} | Bs] - }; - _ -> - %% Stale DOWN (name re-registered, or already removed) — ignore. - Acc - end. - -%% Append an `ok` reply for a tracked unregister's origin; a legacy/salvaged remove -%% (origin `undefined`) has no caller to answer. -unreg_reply(undefined, Rs) -> Rs; -unreg_reply(Origin, Rs) -> [{Origin, ok} | Rs]. - -%% Apply the durable delta's monitor side effects: for a clear, demonitor the name's -%% prior ref; for a set (new/changed binding), demonitor the old ref and monitor the -%% new pid; for a meta (metadata-only update), leave the existing monitor untouched (the -%% pid is unchanged). Monitors carry `{tag, {down, Name}}`, so the DOWN names its -%% binding and no reverse ref map exists. Returns the updated name_to_ref map. -apply_monitor_ops(DBOp, WNames, NTR) -> - maps:fold( - fun - (_Name, {meta, _PidNode}, Acc) -> - Acc; - (Name, clear, NTRacc) -> - demonitor_name(Name, NTRacc); - (Name, {set, _PidNode}, NTRacc) -> - NTR1 = demonitor_name(Name, NTRacc), - Pid = maps:get(Name, WNames), - NTR1#{Name => monitor_name(Name, Pid)} - end, - NTR, - DBOp - ). - -send_replies(Replies) -> - lists:foreach(fun({Origin, Reply}) -> deliver_reply(Origin, Reply) end, Replies). - -%% Apply one op of a replicated batch to the local replica. Called only from the -%% batch handler, under apply_bcast/6's contiguity guard, so the whole batch lands -%% or none of it does. -apply_bcast_op({name_registered, Name, Pid, Index, Data}, S) -> - row_insert(S, Name, Pid, Index, Data); -%% Update the row's Index and Data, keeping its pid (a no-op if the row is absent — -%% e.g. this member missed the registration; the next handoff reconciles it). -apply_bcast_op({metadata_set, Name, Index, Data}, S) -> - row_update_meta(S, Name, Index, Data); -%% ReleasedPid is the live pid this unregister explicitly released (`undefined` for -%% a death-driven cleanup). Recording it — keyed by {Name, Pid}, so a release under -%% one name never masks a conflict on another name the same pid holds — keeps every -%% member's copy of the conflict-detector trail (§5.6) in step, so the trail -%% survives leadership changes via the handoff gather merge. -apply_bcast_op({name_unregistered, Name, ReleasedPid}, S0) -> - S1 = row_delete(S0, Name), - case is_pid(ReleasedPid) of - true -> - Rel = S1#state.recently_released, - S1#state{ - recently_released = Rel#{ - {Name, ReleasedPid} => erlang:system_time(millisecond) - } - }; - false -> - S1 - end. - -%% Broadcast a committed batch to every peer as a single `{names_batch, Ops, Epoch, -%% PrevV, Version, LeaderId}` message: the batch's ops in commit order, the epoch it -%% was planned under, the predecessor version, this batch's commit version, and the -%% sending leader's id. Followers use the version pair to apply only *contiguous* -%% batches (gap detection — see apply_bcast/6) and the leader id as the resync -%% target when a gap is observed. -%% -%% Every op in a batch is planned under one epoch, so it is carried once on the -%% envelope rather than repeated on each op — which matters for a batch of -%% `commit_batch_size` (5000 by default) ops. An empty batch sends nothing. --ifdef(MUTATION_PARTIAL_BATCH). -%% MUTATION — see the note above apply_bcast/6. One message per op, every one -%% stamped with the same {PrevV, Version}: the pre-fix wire format, reintroduced so -%% the eta framework can be asked to rediscover the divergence it produces. -broadcast_batch(_Members, [], _PrevV, _Version, _Self) -> - ok; -broadcast_batch(Members, Ops, PrevV, Version, Self) -> - lists:foreach( - fun(Op) -> - broadcast_to_peers( - Members, - {names_batch, [strip_bcast_epoch(Op)], bcast_epoch(Op), PrevV, Version, Self} - ) - end, - Ops - ). --else. -broadcast_batch(_Members, [], _PrevV, _Version, _Self) -> - ok; -broadcast_batch(Members, [First | _] = Ops, PrevV, Version, Self) -> - Epoch = bcast_epoch(First), - Stripped = [strip_bcast_epoch(Op) || Op <- Ops], - broadcast_to_peers(Members, {names_batch, Stripped, Epoch, PrevV, Version, Self}). --endif. - --ifdef(MUTATION_QUIET_RESYNC). -%% MUTATION (`DGEN_MUTATION=quiet_resync`, test builds only) — the heartbeat -%% reverted to the pre-fix shape: the leader's timer fires and advertises -%% nothing, so gap detection is traffic-triggered again and a follower that -%% loses the *tail* of the stream stays diverged for as long as the cluster is -%% quiet (sim README, finding 2). Planted so the mutation suite can be asked to -%% rediscover it; see test/dgen_registry_mutation_quiet_test.exs. -broadcast_heartbeat(_State) -> - ok. --else. -%% The leader's periodic empty batch, stamped {Applied, Applied}: a caught-up -%% follower applies nothing, a behind one takes apply_bcast/6's gap branch and -%% requests a resync. The traffic-independent half of gap detection. -broadcast_heartbeat(#state{ - members = Members, epoch = Epoch, applied_version = Applied, member_id = Self -}) -> - broadcast_to_peers(Members, {names_batch, [], Epoch, Applied, Applied, Self}), - ok. --endif. - -bcast_epoch({name_registered, _Name, _Pid, _Index, _Data, Epoch}) -> Epoch; -bcast_epoch({metadata_set, _Name, _Index, _Data, Epoch}) -> Epoch; -bcast_epoch({name_unregistered, _Name, _ReleasedPid, Epoch}) -> Epoch. - -strip_bcast_epoch({name_registered, Name, Pid, Index, Data, _Epoch}) -> - {name_registered, Name, Pid, Index, Data}; -strip_bcast_epoch({metadata_set, Name, Index, Data, _Epoch}) -> - {metadata_set, Name, Index, Data}; -strip_bcast_epoch({name_unregistered, Name, ReleasedPid, _Epoch}) -> - {name_unregistered, Name, ReleasedPid}. - -%% Apply a replication broadcast if — and only if — it is contiguous with this -%% member's replica, so the replica always remains a prefix of the leader's totally -%% ordered stream (what makes freshest-wins reconstruction sound, §5.7): -%% -%% - An older epoch's broadcast (a deposed leader's) is dropped. -%% - `PrevV =:= applied_version`: the next batch in sequence — apply it whole. -%% - `V =< applied_version`: at or behind our baseline — a duplicate batch, or one -%% already superseded by a snapshot; drop. -%% - Otherwise there is a **gap**: we missed at least one batch (a cast dropped -%% while we were briefly disconnected, a message lost with a dying connection). -%% Do not apply and do not advance; request a full snapshot from the sender -%% instead. Guarded by resync_timer so a burst of gapped batches asks once. -%% -%% A batch arrives as one message (broadcast_batch/5), so `V =:= applied_version` is -%% now purely the duplicate case and folds into `V =< applied_version`. It used to -%% also mean "another message of the batch we are already applying", which is what -%% made a partially-delivered batch undetectable — the receiver had already advanced -%% to V on the batch's first message, so the loss of any later one left no version -%% discontinuity to notice. -%% -%% ## The mutation -%% -%% `-DMUTATION_PARTIAL_BATCH` puts both halves of that back: `broadcast_batch/5` -%% sends one message per op sharing a version, and the clause below re-admits -%% "another message of the batch we are at". It exists so the eta framework can be -%% required to *rediscover* a divergence we already understand, from a cold start and -%% with no test written for it — acceptance criterion 1 in -%% the `eta` library's docs/design.md. Test builds only, off unless the -%% `DGEN_MUTATION` environment variable asks for it, and never in a release: see -%% `erlc_options/1` in mix.exs. --ifdef(MUTATION_PARTIAL_BATCH). -apply_bcast(Epoch, PrevV, V, LeaderId, ApplyFun, State) -> - #state{epoch = CurrentEpoch, applied_version = Applied} = State, - if - Epoch < CurrentEpoch -> State; - PrevV =:= Applied -> bump_applied(V, ApplyFun(cancel_resync(State))); - %% MUTATION: "another message of the batch we are already applying" — apply - %% it without advancing, because we are already at V. - V =:= Applied -> ApplyFun(State); - V =< Applied -> State; - true -> request_resync(LeaderId, State) - end. --else. -apply_bcast(Epoch, PrevV, V, LeaderId, ApplyFun, State) -> - #state{epoch = CurrentEpoch, applied_version = Applied} = State, - if - Epoch < CurrentEpoch -> State; - PrevV =:= Applied -> bump_applied(V, ApplyFun(cancel_resync(State))); - V =< Applied -> State; - true -> request_resync(LeaderId, State) - end. --endif. - -%% Ask `LeaderId` for a full snapshot (we observed a gap in its stream), at most -%% once per ?RESYNC_RETRY window. The reply is a regular {apply_names_snapshot}. -request_resync(LeaderId, State = #state{resync_timer = undefined, member_id = Self}) -> - cast_to_member(LeaderId, {resync_req, Self}), - Ref = arm(?RESYNC_RETRY, resync_timeout), - State#state{resync_timer = Ref}; -request_resync(_LeaderId, State) -> - State. - -cancel_resync(State = #state{resync_timer = undefined}) -> - State; -cancel_resync(State = #state{resync_timer = Ref}) -> - erlang:cancel_timer(Ref), - State#state{resync_timer = undefined}. - -%% Advance applied_version to the highest version applied (within an epoch a leader's -%% broadcasts carry non-decreasing versions, and apply_bcast only lets contiguous -%% ones through, but max/2 keeps this robust regardless). Advancing may release -%% gap-deferred forwarded-`yes` acks (flush_deferred) — a no-op when none are held. -bump_applied(Version, State = #state{applied_version = Applied}) -> - flush_deferred(State#state{applied_version = max(Applied, Version)}). - -%% Record that this member now holds synced registry state, announcing the -%% transition to the elector once (`fresh = false`) so the membership record's -%% freshness converges — see the `synced` field doc and the elector's member_info. -mark_synced(State = #state{synced = true}) -> - State; -mark_synced(State = #state{elector = Elector, member_id = Self, join_token = Token}) -> - dgen_server:cast(Elector, {join, Self, Token, false}), - State#state{synced = true}. - -%% Is a peer member's node currently connected (so a cast to it will not be -%% dropped and a forward will actually be answered)? -member_reachable({Node, _Name}) -> - dgen_utils:node_reachable(Node). - -%% Report a peer's death to the elector, fencing the `{member_down}` with the token the -%% elector currently holds for that member (read fresh from durable state via a priority -%% read that bypasses the queue), falling back to `Cached` if that read fails. Runs in a -%% short-lived helper so the member loop never blocks on a call to the elector — see the -%% DOWN handler for the deadlock this avoids and why the fresh read matters. -spawn_member_down(Elector, MemberId, Cached) -> - _ = spawn(fun() -> - Token = dgen_registry_elector:member_token(Elector, MemberId, Cached), - dgen_server:cast(Elector, {member_down, MemberId, Token}) - end), - ok. - -%% Answer a consistent read only after verifying, against the durable leader key, -%% that this member really is the fenced leader for its current epoch (Guarantee 5: -%% consistent reads are authoritative, never stale). Without this, a deposed -%% leader that has not yet heard about the handoff — the minority side of a -%% partition believes `leader = self` until a snapshot it may not receive — would -%% serve its frozen replica as authoritative. Fencing writes alone cannot prevent -%% that; reads need their own fence. -%% -%% `Value` is captured in the handler, at the member's mailbox point; the verify -%% runs in a short-lived helper so the DB round-trip never blocks the member's -%% loop. This is linearizable because leadership is continuous within an epoch: -%% if `{Self, Epoch}` still holds at verify time, no other leader existed between -%% the capture and the verify, so the captured value was (and still is at the -%% verification point) the authoritative answer. Any verification failure — a -%% mismatch, or an unreachable backend — answers `Denied` (the CP refusal), the -%% same value an unreachable-leader forward answers. -reply_fenced(From, Value, Denied, #state{ - tenant = Tenant, tuid = Tuid, member_id = Self, epoch = Epoch -}) -> - _ = spawn(fun() -> - Reply = - case dgen_registry_names:verify_leader(Tenant, Tuid, Self, Epoch) of - true -> Value; - false -> Denied - end, - gen_server:reply(From, Reply) - end), - ok. - -%% Forward a pid-guarded retract to the (believed) current leader — used when a -%% durable clear this member owed (an unregister, or a strict-mode retraction) -%% cannot ride its own commit pipeline anymore. Best-effort by necessity: with no -%% leader there is nowhere to send it, and a later handoff gather reconciles. -forward_retract(undefined, _Name, _Pid) -> - ok; -forward_retract(Leader, Name, Pid) -> - cast_to_member(Leader, {retract_req, Name, Pid}). - -%% A batch failed to commit (fenced / backend error / worker death) or committed -%% but its plan is no longer applicable (leadership moved in flight). Its -%% registrations were rejected — callers retry — but its destructive ops (removes, -%% retracts, downs) must be re-driven: dropping an unregister would leave this -%% member's optimistic delete diverged from every other replica until an unrelated -%% handoff, and dropping a retract would leave a `no`-answered binding alive. -%% Scheduled after a short delay so a persistently failing backend does not spin. -salvage_failed_plan(State, #{ops := Ops}) -> - Destructive = [ - strip_remove_origin(Op) - || Op <- Ops, - element(1, Op) =:= remove orelse element(1, Op) =:= retract orelse - element(1, Op) =:= down - ], - case Destructive of - [] -> ok; - _ -> arm(?REQUEUE_DELAY, {requeue_ops, Destructive}) - end, - State. - -%% Arm a timer, and say so. -%% -%% Under simulation `eta_transform` points the `erlang:send_after/3` below at -%% `eta_time`, so every deadline here lands in the virtual clock's wheel and the -%% driver records reaching it as a `{clock, Ms}` trace entry. Those entries are -%% otherwise anonymous: a run that advances to 6500 tells you a timer was due and -%% nothing about whose it was. Logging the arm is what makes one attributable. -%% -%% `?ETA_LOG` is a no-op in a production build, where this is a plain -%% `send_after/3` with an extra function call around it. -arm(Delay, Msg) -> - _ = ?ETA_LOG({arm, Msg, Delay}), - erlang:send_after(Delay, self(), Msg). - -%% A salvaged remove's caller was already answered by reject_plan (`ok` — see -%% reject_value); strip the origin so the re-driven op does not answer it twice. -strip_remove_origin({remove, Name, Pid, _Origin}) -> {remove, Name, Pid, undefined}; -strip_remove_origin(Op) -> Op. - -%% Losing leadership with direct registrations still awaiting replica acks: resolve -%% them now, by the same policy as a replicate timeout (§8). Degrade-open answers -%% `yes` (the binding committed under our term and we hold it, so the handoff gather -%% carries it); fail-closed answers `no` and forwards the pid-guarded retract to the -%% new leader, so the caller's `no` cannot leave the binding alive. -fail_pending_acks(State = #state{pending_acks = PA}) when map_size(PA) =:= 0 -> - State; -fail_pending_acks(State = #state{pending_acks = PA, leader = NewLeader, config = Config}) -> - Strict = dgen_config:strict_replication(Config), - maps:foreach( - fun(_BatchRef, {Direct, TimerRef, Needed, Acked}) -> - erlang:cancel_timer(TimerRef), - case Strict of - false -> - notify_degrade_open(Needed, map_size(Acked), length(Direct)), - deliver_direct(Direct); - true -> - lists:foreach( - fun({Origin, Name, Pid}) -> - deliver_reply(Origin, no), - forward_retract(NewLeader, Name, Pid) - end, - Direct - ) - end - end, - PA - ), - State#state{pending_acks = #{}}. - -%% --------------------------------------------------------------------------- -%% Local names replica (ETS) — the member is this table's sole writer. -%% --------------------------------------------------------------------------- - -%% Rows are `{Name, Pid, Index, Data}`: Pid is the bound process, Index the queryable -%% metadata map, Data the opaque payload. A plain registration uses `#{}` / `undefined`. - -%% Per-name pid read (member-side; callers read pid via ets:lookup directly). -lookup_name(Tab, Name) -> - case ets:lookup(Tab, Name) of - [{_Name, Pid, _Index, _Data} | _] -> Pid; - [] -> undefined - end. - -%% Per-name metadata read for the consistent get_metadata path → the public shape -%% `{ok, #{pid, index, data}}` or `undefined` (mirrors dgen_registry:get_metadata/1). -lookup_metadata(Tab, Name) -> - case ets:lookup(Tab, Name) of - [{_Name, Pid, Index, Data} | _] -> {ok, #{pid => Pid, index => Index, data => Data}}; - [] -> undefined - end. - -name_exists(Tab, Name) -> - ets:member(Tab, Name). - -insert_record(Tab, Name, Pid, Index, Data) -> - ets:insert(Tab, {Name, Pid, Index, Data}), - ok. - -%% Replace a row's metadata (Index + Data) in place, keeping its pid. A no-op if the -%% row is absent (update_element returns false), which is what a metadata broadcast for -%% a name this member has not registered should be. -update_metadata(Tab, Name, Index, Data) -> - ets:update_element(Tab, Name, [{3, Index}, {4, Data}]), - ok. - -delete_name(Tab, Name) -> - ets:delete(Tab, Name), - ok. - -%% Materialise the whole replica as a transient records map `#{Name => {Pid, Index, -%% Data}}` — used by the handoff gather/fan-out, which must carry metadata. -current_records(Tab) -> - ets:foldl( - fun({Name, Pid, Index, Data}, Acc) -> Acc#{Name => {Pid, Index, Data}} end, #{}, Tab - ). - -%% Snapshot payload codec (see the "distribute" note above). A snapshot ships as -%% a single `term_to_binary` blob rather than an on-heap term. A large binary is -%% a refc binary — stored off-heap and never scanned by GC — so a 100k+-name -%% snapshot no longer bloats the sender's or receiver's process heap, nor thrashes -%% their collectors while it sits in a mailbox. Encoding once and casting the -%% *same* binary to every follower makes the fan-out share one blob instead of -%% copying the whole term per follower. -%% -%% Two encoders, by what the sender holds: -%% -%% - `encode_table/1` — the "serve my replica" sites (the gather reply, the -%% resync serve, the joiner snapshot). There is no native ETS→binary dump -%% (`ets:tab2file` is a slower, disk-only walk of the same terms), and the -%% records MAP those sites used to build existed only to be encoded — at 200k -%% names ~96ms of pure map materialization ahead of the encode. One `ets:foldl` -%% straight into a `[{Name, Record}]` pair list is ~3x cheaper, and the list -%% inside the blob decodes through `decode_records/1`'s existing list -%% normalization. -%% - `encode_records/1` — the assume fan-out, where the reconstructed records map -%% is already in hand (list vs map inside the blob is a wash there). -%% -%% Compression is zlib level 1, deliberately: measured at 200k names, the default -%% `[compressed]` (level 6) spent 60ms for a 0.76MB blob where level 1 spends -%% 37ms for 1.07MB — decode time unchanged. Snapshot encodes sit on the -%% client-visible handoff/resync paths, so encode time buys more than the ~40% -%% wire bytes it costs; going uncompressed (12ms) is off the table because the -%% blob is ~10MB plain and crosses distribution once per follower. -%% -%% `decode_records/1` normalises every shape to the map all callers want, -%% re-dispatching on what the blob contains: a map (this build's fan-out, and any -%% pre-list-era sender), a `[{Name, Record}]` list (`encode_table/1`, and the -%% pre-binary-era apply_names_snapshot wire), or a bare map term -%% (pre-binary-era get_names_snapshot). As with the map→binary switch before it, -%% the tolerant decoder ships in the same release as the new sender, so -%% mixed-version tolerance for list-in-binary begins one release back from here. --define(SNAPSHOT_ENCODE_OPTS, [{compressed, 1}]). - -encode_table(Tab) -> - Pairs = ets:foldl( - fun({Name, Pid, Index, Data}, Acc) -> [{Name, {Pid, Index, Data}} | Acc] end, [], Tab - ), - term_to_binary(Pairs, ?SNAPSHOT_ENCODE_OPTS). - -encode_records(Records) when is_map(Records) -> - term_to_binary(Records, ?SNAPSHOT_ENCODE_OPTS). - -decode_records(Bin) when is_binary(Bin) -> decode_records(binary_to_term(Bin)); -decode_records(List) when is_list(List) -> maps:from_list(List); -decode_records(Map) when is_map(Map) -> Map. - -%% Wholesale replace the replica with a records map `#{Name => {Pid, Index, Data}}` (a -%% leadership-snapshot apply / handoff reconstruction): clear the table, insert the new -%% rows, and rebuild the inverted index from them. The member is the sole writer, so no -%% concurrent write can interleave between the clear and the insert. Returns the state -%% with the rebuilt index. -records_replace(State = #state{names_tab = Tab}, Records) -> - ets:delete_all_objects(Tab), - ets:insert( - Tab, [{Name, Pid, Index, Data} || {Name, {Pid, Index, Data}} <- maps:to_list(Records)] - ), - State#state{inv_index = index_rebuild(Records)}. - -%% Apply a committed batch's durable delta to the replica, keeping the inverted index in -%% step. Only names in `DBOp` are touched, so a concurrent optimistic unregister of an -%% untouched name is never clobbered (the ETS analogue of the old delta-over-the-map fold): -%% - clear: delete the row. -%% - set: write the full row (a new/changed binding — pid from the plan's working name -%% map, metadata from its working meta map). -%% - meta: update only the metadata of the *existing* row (an `update_element`, a no-op -%% if the row was optimistically unregistered between plan and apply — so a -%% metadata-only op can never resurrect a removed binding). -apply_dbop(State, DBOp, WNames, WMeta) -> - maps:fold( - fun - (Name, clear, S) -> - row_delete(S, Name); - (Name, {set, _PidNode}, S) -> - {Index, Data} = maps:get(Name, WMeta), - row_insert(S, Name, maps:get(Name, WNames), Index, Data); - (Name, {meta, _PidNode}, S) -> - {Index, Data} = maps:get(Name, WMeta), - row_update_meta(S, Name, Index, Data) - end, - State, - DBOp - ). - -%% --------------------------------------------------------------------------- -%% Inverted index over the rows' queryable `Index` metadata (on every member; §6). -%% --------------------------------------------------------------------------- - -%% State-threading row writers — each keeps the ETS row and the inverted index in step, -%% retracting the name's old index postings (read from the table before the write) -%% before applying its new ones. Use these (not the bare insert/delete/update helpers) -%% at every site that mutates a row, so the index never drifts. -row_insert(S = #state{names_tab = Tab, inv_index = Inv}, Name, Pid, Index, Data) -> - Old = row_index(Tab, Name), - insert_record(Tab, Name, Pid, Index, Data), - S#state{inv_index = index_add(index_remove(Inv, Name, Old), Name, Index)}. - -row_update_meta(S = #state{names_tab = Tab, inv_index = Inv}, Name, Index, Data) -> - Old = row_index(Tab, Name), - update_metadata(Tab, Name, Index, Data), - %% update_metadata is a no-op when the row is absent; do not invent index postings - %% for a name this member does not hold. - case name_exists(Tab, Name) of - true -> S#state{inv_index = index_add(index_remove(Inv, Name, Old), Name, Index)}; - false -> S - end. - -row_delete(S = #state{names_tab = Tab, inv_index = Inv}, Name) -> - Old = row_index(Tab, Name), - delete_name(Tab, Name), - S#state{inv_index = index_remove(Inv, Name, Old)}. - -%% A name's current indexed-attribute map (`#{}` if it has no row), so a mutation can -%% retract its old postings before applying new ones. -row_index(Tab, Name) -> - case ets:lookup(Tab, Name) of - [{_N, _Pid, Index, _Data} | _] -> Index; - [] -> #{} - end. - -%% Add/remove a name's postings for each {attr, value} pair in `Index`. Removal prunes -%% empty value- and attribute-maps so the index does not accumulate empty shells. -index_add(Inv, Name, Index) -> - maps:fold( - fun(Attr, Value, Acc) -> - Vs = maps:get(Attr, Acc, #{}), - Names = maps:get(Value, Vs, #{}), - Acc#{Attr => Vs#{Value => Names#{Name => []}}} - end, - Inv, - Index - ). - -index_remove(Inv, Name, Index) -> - maps:fold( - fun(Attr, Value, Acc) -> - case Acc of - #{Attr := Vs} -> - case Vs of - #{Value := Names} -> - Names1 = maps:remove(Name, Names), - Vs1 = - case map_size(Names1) of - 0 -> maps:remove(Value, Vs); - _ -> Vs#{Value => Names1} - end, - case map_size(Vs1) of - 0 -> maps:remove(Attr, Acc); - _ -> Acc#{Attr => Vs1} - end; - _ -> - Acc - end; - _ -> - Acc - end - end, - Inv, - Index - ). - -%% Rebuild the whole inverted index from a records map (handoff / snapshot apply). -index_rebuild(Records) -> - maps:fold( - fun(Name, {_Pid, Index, _Data}, Acc) -> index_add(Acc, Name, Index) end, - #{}, - Records - ). - -%% Resolve an AND-equal query against the inverted index + table (§6). `Constraints` is -%% `#{attr() => value()}`; the result is `[#{name, pid, index, data}]`, every registration -%% whose `Index` satisfies *all* clauses. Empty constraints yield `[]` (the public API -%% rejects them before reaching here). Steps: take each clause's posting set, short- -%% circuit if any is empty, intersect smallest-first, then materialise + re-check each -%% surviving name against its current row (so a stale index posting can never produce a -%% false positive). -run_query(Constraints, _Tab, _Inv) when map_size(Constraints) =:= 0 -> - []; -run_query(Constraints, Tab, Inv) -> - Clauses = maps:to_list(Constraints), - Postings = [posting(Inv, Attr, Value) || {Attr, Value} <- Clauses], - case lists:any(fun(P) -> map_size(P) =:= 0 end, Postings) of - true -> - %% Some clause matches nothing — the conjunction is empty. - []; - false -> - [Smallest | Rest] = lists:sort( - fun(A, B) -> map_size(A) =< map_size(B) end, Postings - ), - Names = [N || N <- maps:keys(Smallest), in_all(N, Rest)], - materialize(Names, Constraints, Tab) - end. - -%% Posting set (`#{Name => []}`) for one clause, or `#{}` if the attr/value is absent. -posting(Inv, Attr, Value) -> - case Inv of - #{Attr := #{Value := Names}} -> Names; - _ -> #{} - end. - -in_all(Name, Postings) -> - lists:all(fun(P) -> maps:is_key(Name, P) end, Postings). - -%% Read each candidate's row and keep it only if the row still satisfies every clause -%% (defends against any index drift), shaping it into the public `#{name, pid, index, -%% data}` match. -materialize(Names, Constraints, Tab) -> - lists:foldl( - fun(Name, Acc) -> - case ets:lookup(Tab, Name) of - [{_N, Pid, Index, Data} | _] -> - case satisfies(Constraints, Index) of - true -> - [#{name => Name, pid => Pid, index => Index, data => Data} | Acc]; - false -> - Acc - end; - [] -> - Acc - end - end, - [], - Names - ). - -%% --------------------------------------------------------------------------- -%% Tagged queries (§4.9) — the extensible query() form used by presence -%% subscriptions. Only `{all, Constraints}` (AND-equal, the run_query semantics) -%% exists today; new kinds slot in here without touching call sites. -%% --------------------------------------------------------------------------- - -%% Run a query against the replica, returning the full `#{name, pid, index, data}` -%% matches (the run_query shape). -query_run({all, Constraints}, Tab, Inv) -> - run_query(Constraints, Tab, Inv). - -%% Does `Index` (a row's indexed-attribute map) satisfy the query? -query_satisfies({all, Constraints}, Index) -> - map_size(Constraints) > 0 andalso satisfies(Constraints, Index). - -%% The watch-set membership `#{Name => Pid}` a query currently resolves to. -query_match_names(Query, Tab, Inv) -> - maps:from_list([{N, P} || #{name := N, pid := P} <- query_run(Query, Tab, Inv)]). - -%% Does `Index` (a row's indexed-attribute map) satisfy every clause exactly? -satisfies(Constraints, Index) -> - maps:fold( - fun(Attr, Value, Ok) -> Ok andalso maps:get(Attr, Index, '$absent') =:= Value end, - true, - Constraints - ). - -add_member_monitors(MemberIds, State) -> - #state{member_id = Self, members = Members, monitors = Monitors} = State, - {NewMembers, NewMonitors} = lists:foldl( - fun(MemberId, {MA, MonA}) -> - case MemberId =:= Self orelse maps:is_key(MemberId, MA) of - true -> - {MA, MonA}; - false -> - {Node, Name} = MemberId, - Ref = erlang:monitor(process, {Name, Node}), - {MA#{MemberId => Ref}, MonA#{Ref => MemberId}} - end - end, - {Members, Monitors}, - MemberIds - ), - State#state{members = NewMembers, monitors = NewMonitors}. - -remove_member(MemberId, State = #state{members = Members, monitors = Monitors}) -> - case maps:get(MemberId, Members, undefined) of - undefined -> - State; - Ref -> - erlang:demonitor(Ref, [flush]), - %% Drop the peer's token too, so `peer_tokens` does not accumulate stale - %% entries across node churn (it is repopulated from the snapshot on - %% rejoin). The caller reads the token for its {member_down} cast before - %% calling this, so pruning here is safe. The departure timestamp keeps - %% trail pruning suspended for a TTL window (§5.6): the departed member - %% is no longer "a disconnected member", but its stale rows can still - %% surface at a rejoin gather within that window. - State#state{ - members = maps:remove(MemberId, Members), - monitors = maps:remove(Ref, Monitors), - peer_tokens = maps:remove(MemberId, State#state.peer_tokens), - last_departure = erlang:system_time(millisecond) - } - end. - -demonitor_name(LogicalName, NTR) -> - case maps:get(LogicalName, NTR, undefined) of - undefined -> - NTR; - OldRef -> - erlang:demonitor(OldRef, [flush]), - maps:remove(LogicalName, NTR) - end. - -%% The one place a registered-process monitor is created. The tag makes the DOWN -%% self-describing ({{down, Name}, Ref, …}); the flush-on-demonitor elsewhere -%% covers tagged messages exactly as it covers 'DOWN' ones. -monitor_name(Name, Pid) -> - erlang:monitor(process, Pid, [{tag, {down, Name}}]). - -%% Returns the list of member IDs to add as monitors for a given member during -%% a join/member_down leadership transition. -%% - undefined MemberId (member_down): no extra monitors for anyone. -%% - Id = MemberId (new joining member): add all current peers. -%% - Id ≠ MemberId (existing member): add only the new member. -extra_member_ids(undefined, _AllIds, _Id) -> []; -extra_member_ids(MemberId, AllIds, MemberId) -> AllIds; -extra_member_ids(MemberId, _AllIds, _Id) -> [MemberId]. - -%% Merge incoming token map into local peer_tokens, taking the newer token when -%% both sides know about the same member (higher value = more recent make_ref -%% within a single BEAM session, but refs are opaque so we always overwrite — -%% the elector is the authoritative source and always sends current tokens). -merge_peer_tokens(Tokens, State = #state{peer_tokens = PeerTokens}) -> - State#state{peer_tokens = maps:merge(PeerTokens, Tokens)}. - -broadcast_to_peers(Members, Msg) -> - maps:foreach(fun(MemberId, _) -> cast_to_member(MemberId, Msg) end, Members). - -%% Cast to a peer member. A fire-and-forget cast to `{Name, Node}` on a -%% currently-disconnected node would trigger an automatic distribution reconnect -%% (the same hazard `dgen_registry_elector:call_to_member/2` guards against): if -%% it fires during a partition, both sides see `{nodeup, _}`, re-join with fresh -%% tokens, and the partition is healed before either side ever removed the other -%% — so the departure is never observed. Drop the cast when the target node is -%% not already connected. -%% -%% Every inter-member protocol message goes through here: broadcasts, register and -%% set_metadata replies, replicate_sync/ack, resync requests, snapshots, retracts. -%% -%% That used to matter for a second reason. A `persistent_term` hook sat on this -%% function so a simulation could own delivery, and this was the one place it -%% could — a seam this module maintained on the test harness's behalf. It is -%% gone. `eta_transform` rewrites the `gen_server:cast/2` below to -%% `eta_net:cast/2` under `-ifdef(DST)`, so a simulation interposes at the send -%% itself, and `eta_net` is inert unless a run installs a network — so a release -%% build is a plain cast with nothing in front of it, rather than a -%% persistent_term read of a key nobody ever sets. See `test/support/sim/`. -cast_to_member(To, Msg) -> - do_cast_to_member(To, Msg). - -do_cast_to_member({Node, Name}, Msg) when Node =:= node() -> - gen_statem:cast({Name, Node}, Msg); -do_cast_to_member({Node, Name}, Msg) -> - case lists:member(Node, nodes()) of - true -> gen_statem:cast({Name, Node}, Msg); - false -> ok - end. diff --git a/test/dgen_registry_cluster_test.exs b/test/dgen_registry_cluster_test.exs index 1ff3229..4268258 100644 --- a/test/dgen_registry_cluster_test.exs +++ b/test/dgen_registry_cluster_test.exs @@ -1543,14 +1543,21 @@ defmodule DGen.RegistryClusterTest do # The name is genuinely free again. Leadership may still be settling: each # side's backstop reap can re-elect during the partition, so the primary can - # come out of the heal as a stale leader whose fenced batch, or whose forward - # to the new leader, answers `no` ("leader changed, retry"). Give it the same - # budget as the convergence checks above, not the 2s default. + # come out of the heal as a stale leader. Its registration is then retried + # until it learns who leads, and re-sent there — never answered `no` — so a + # single call can block past its timeout and exit (§3). Retry it with the + # same budget as the convergence checks above, not the 2s default. pid2 = spawn(fn -> Process.sleep(:infinity) end) on_exit(fn -> Process.exit(pid2, :kill) end) assert eventually( - fn -> :dgen_registry.register_name({reg, :redrive_me}, pid2) == :yes end, + fn -> + try do + :dgen_registry.register_name({reg, :redrive_me}, pid2) == :yes + catch + :exit, {:timeout, _} -> false + end + end, 20_000 ), "name was not re-registrable after the re-driven unregister" diff --git a/test/dgen_registry_conflicts_test.exs b/test/dgen_registry_conflicts_test.exs new file mode 100644 index 0000000..faff91b --- /dev/null +++ b/test/dgen_registry_conflicts_test.exs @@ -0,0 +1,296 @@ +defmodule DGen.RegistryConflictsTest do + @moduledoc """ + Conflict repair (§5.6), in isolation — no member process, no backend. + + `dgen_registry_conflicts` is pure: liveness is passed in, and `resolve/5` + returns the kills and alarms for the member to carry out. These tests pin + which gathered states are a conflict, what the remedy does with one (and when + it holds back), when the release trail may expire, and the version-defined + `degraded/5`. + """ + use ExUnit.Case, async: true + + alias :dgen_registry_conflicts, as: Conflicts + + defp spawn_live do + pid = spawn(fn -> Process.sleep(:infinity) end) + on_exit(fn -> Process.exit(pid, :kill) end) + pid + end + + defp eventually(fun, timeout \\ 2_000) do + deadline = System.monotonic_time(:millisecond) + timeout + do_eventually(fun, deadline) + end + + defp do_eventually(fun, deadline) do + cond do + fun.() -> true + System.monotonic_time(:millisecond) > deadline -> false + true -> Process.sleep(10) && do_eventually(fun, deadline) + end + end + + # The detector with real local liveness, as the old 3-arity form probed it. + defp detect(authority, witnesses, trail), + do: Conflicts.detect(authority, witnesses, trail, &Process.alive?/1) + + defp detect(authority, witnesses, trail, alive), + do: Conflicts.detect(authority, witnesses, trail, alive) + + defp rec(pid), do: {pid, %{}, nil} + + # --------------------------------------------------------------------------- + # §5.6 conflict-detection predicate (pure, unit-tested in isolation) + # + # The full end-to-end scenario that produces a genuine conflict needs a + # multi-node, leadership-orchestrated setup (a member holds a divergent live + # binding that a leadership-change gather exposes); that lives in the cluster + # suite ("conflict resolution by termination" in dgen_registry_cluster_test.exs). + # Here we pin the safety-critical predicate itself: which gathered states count + # as a conflict, and — crucially — which do not (so we never false-kill a live + # process). + # --------------------------------------------------------------------------- + + describe "detect/4 — which gathered states are a conflict" do + # The detector takes RECORDS maps (`%{name => {pid, index, data}}`) and + # compares entries against the authority in place — the shape change that made + # the all-agree handoff allocation-free. `rec/1` builds the record wrapper; + # the semantics pinned below are unchanged from the pid-map era. + + test "two different live pids for one name is a conflict" do + p1 = spawn_live() + p2 = spawn_live() + + assert [{:n, ^p1, [^p2]}] = + detect( + %{n: rec(p1)}, + [%{n: rec(p1)}, %{n: rec(p2)}], + %{} + ) + end + + test "a recently-released name/pid pair is suppressed (lag, not conflict)" do + p1 = spawn_live() + p2 = spawn_live() + # Trail entries are keyed {name, pid} (§5.6). + released = %{{:n, p2} => System.system_time(:millisecond)} + + assert [] = + detect( + %{n: rec(p1)}, + [%{n: rec(p1)}, %{n: rec(p2)}], + released + ) + end + + test "a release under one name does not mask a conflict on another name" do + p1 = spawn_live() + p2 = spawn_live() + # p2 was explicitly released from :other — that must not suppress the + # genuine divergence on :n, which p2 also (still) claims. + released = %{{:other, p2} => System.system_time(:millisecond)} + + assert [{:n, ^p1, [^p2]}] = + detect( + %{n: rec(p1)}, + [%{n: rec(p1)}, %{n: rec(p2)}], + released + ) + end + + test "a legacy bare-pid trail entry (rolling upgrade) still suppresses" do + p1 = spawn_live() + p2 = spawn_live() + # A pre-upgrade member's gathered trail keys by pid alone; the detector + # honours that coarser shape until the mixed entries age out. + released = %{p2 => System.system_time(:millisecond)} + + assert [] = + detect( + %{n: rec(p1)}, + [%{n: rec(p1)}, %{n: rec(p2)}], + released + ) + end + + test "a dead divergent pid is not a conflict" do + p1 = spawn_live() + p2 = spawn(fn -> :ok end) + Process.exit(p2, :kill) + eventually(fn -> not Process.alive?(p2) end) + + assert [] = + detect( + %{n: rec(p1)}, + [%{n: rec(p1)}, %{n: rec(p2)}], + %{} + ) + end + + test "a dead authority is not a conflict (nothing to defend)" do + p1 = spawn(fn -> :ok end) + Process.exit(p1, :kill) + eventually(fn -> not Process.alive?(p1) end) + p2 = spawn_live() + + assert [] = + detect( + %{n: rec(p1)}, + [%{n: rec(p1)}, %{n: rec(p2)}], + %{} + ) + end + + test "agreement across all maps is not a conflict" do + p1 = spawn_live() + + assert [] = + detect( + %{n: rec(p1)}, + [%{n: rec(p1)}, %{n: rec(p1)}], + %{} + ) + end + + test "agreement with different metadata is still agreement (pids decide)" do + p1 = spawn_live() + + assert [] = + detect( + %{n: {p1, %{v: 1}, nil}}, + [%{n: {p1, %{v: 2}, :other}}], + %{} + ) + end + + test "a name with no authority (absent from the freshest map) is not a conflict" do + p1 = spawn_live() + + assert [] = detect(%{}, [%{n: rec(p1)}], %{}) + end + + # The leader adjudicates on its own loop, so liveness is probed ahead of time in + # the gather helpers (conflict_pids/3 says which pids) and handed in as data + # (detect/4). No probe — no RPC — runs during adjudication. + test "conflict_pids/3 names exactly the pids adjudication would probe" do + [p1, p2, p3, p4] = for _ <- 1..4, do: spawn_live() + + fresh = %{n: rec(p1), agreed: rec(p3)} + witnesses = [%{n: rec(p2), agreed: rec(p3), trailed: rec(p4)}] + released = %{{:trailed, p4} => System.system_time(:millisecond)} + + assert Enum.sort([p1, p2]) == + :dgen_registry_conflicts.conflict_pids(fresh, witnesses, released) + end + + test "detect/4 decides on the liveness it is given, not by probing" do + p1 = spawn_live() + p2 = spawn_live() + fresh = %{n: rec(p1)} + witnesses = [%{n: rec(p2)}] + + assert [{:n, ^p1, [^p2]}] = + detect(fresh, witnesses, %{}, fn _ -> true end) + + # Both are alive, but the evidence says the divergent one is not: no conflict. + assert [] = + detect(fresh, witnesses, %{}, &(&1 == p1)) + end + end + + describe "resolve/5 — the remedy" do + @config %{terminate_on_conflict: true, conflict_kill_budget: {2, 60_000}} + + test "kills the authority and every divergent pid, drops the name, alarms" do + [a, d1, d2] = for _ <- 1..3, do: spawn_live() + records = %{n: rec(a), kept: rec(a)} + conflict = {:n, a, [d1, d2]} + + assert {%{kept: _} = left, _c, effects} = + Conflicts.resolve([conflict], records, @config, 1_000, Conflicts.new()) + + refute Map.has_key?(left, :n) + assert effects == [{:kill, a}, {:kill, d1}, {:kill, d2}, {:alarm, conflict, :terminated}] + end + + test "an exhausted budget alarms and keeps the name served" do + [a, d] = for _ <- 1..2, do: spawn_live() + records = %{n: rec(a)} + conflict = {:n, a, [d]} + + {_, c, _} = Conflicts.resolve([conflict], records, @config, 1_000, Conflicts.new()) + {_, c, _} = Conflicts.resolve([conflict], records, @config, 2_000, c) + + assert {^records, _c, [{:alarm, ^conflict, :budget_exhausted}]} = + Conflicts.resolve([conflict], records, @config, 3_000, c) + + # Outside the window the budget is available again. + assert {left, _c, [_ | _]} = Conflicts.resolve([conflict], records, @config, 70_000, c) + refute Map.has_key?(left, :n) + end + + test "detect-only mode alarms and kills nothing" do + [a, d] = for _ <- 1..2, do: spawn_live() + records = %{n: rec(a)} + conflict = {:n, a, [d]} + config = %{@config | terminate_on_conflict: false} + + assert {^records, _c, [{:alarm, ^conflict, :detect_only}]} = + Conflicts.resolve([conflict], records, config, 1_000, Conflicts.new()) + end + + test "no conflicts pass the reconstruction through untouched" do + records = %{n: rec(spawn_live())} + assert {^records, _c, []} = Conflicts.resolve([], records, @config, 0, Conflicts.new()) + end + end + + describe "the release trail" do + @ttl_config %{conflict_release_ttl: 1_000, conflict_kill_budget: {3, 60_000}} + + test "merging keeps each entry's latest release time" do + p = spawn_live() + c = Conflicts.released([{:n, p}], 10, Conflicts.new()) + c = Conflicts.merge([%{{:n, p} => 5}, %{{:m, p} => 7}], c) + + assert Conflicts.trail(c) == %{{:n, p} => 10, {:m, p} => 7} + end + + test "entries expire after the TTL only while every member is connected" do + p = spawn_live() + c = Conflicts.released([{:n, p}], 0, Conflicts.new()) + + assert Conflicts.trail(Conflicts.prune(5_000, false, @ttl_config, c)) != %{} + assert Conflicts.trail(Conflicts.prune(5_000, true, @ttl_config, c)) == %{} + end + + test "entries outlive a member departure by a TTL" do + p = spawn_live() + + c = + Conflicts.released([{:n, p}], 0, Conflicts.new()) |> then(&Conflicts.departed(4_500, &1)) + + assert Conflicts.trail(Conflicts.prune(5_000, true, @ttl_config, c)) != %{} + assert Conflicts.trail(Conflicts.prune(6_000, true, @ttl_config, c)) == %{} + end + end + + describe "degraded/5" do + test "is exactly a reconstruction below the durable version key" do + assert Conflicts.degraded(9, 10, [:a], [], %{}) + refute Conflicts.degraded(10, 10, [:a], [], %{}) + end + + test "ignores who answered the gather when the key is known" do + # :b never answered and was never synced — still not degraded. + refute Conflicts.degraded(10, 10, [:a, :b], [], %{a: :snapshot}) + end + + test "falls back to reachability when the key could not be read" do + assert Conflicts.degraded(10, :unknown, [:a, :b], [], %{a: :snapshot}) + # A member marked fresh (never synced) holds nothing to miss. + refute Conflicts.degraded(10, :unknown, [:a, :b], [:b], %{a: :snapshot}) + end + end +end diff --git a/test/dgen_registry_intents_test.exs b/test/dgen_registry_intents_test.exs new file mode 100644 index 0000000..4359ca5 --- /dev/null +++ b/test/dgen_registry_intents_test.exs @@ -0,0 +1,221 @@ +defmodule DGen.RegistryIntentsTest do + @moduledoc """ + A member's unfinished client requests, in isolation — no member process. + + `dgen_registry_intents` is pure: every call returns the new intents and the + effects for the member to carry out. These tests drive it through its three + events (`on_reply/4`, `on_version/2`, `redrive/5`) and pin the re-drive rule: + re-drive what is idempotent (register, unregister-as-retract), and answer + `{:error, :no_leader}` for what is not (set_meta). + """ + use ExUnit.Case, async: true + + alias :dgen_registry_intents, as: Intents + + @l1 {:n1@host, :reg} + @l2 {:n2@host, :reg} + @ttl 5_000 + + setup do + {:ok, pid: spawn(fn -> Process.sleep(:infinity) end), from: {self(), make_ref()}} + end + + defp register(pid), do: {:register, :name, pid, {%{}, :undefined}} + + # The reference of the single :forward effect in `effects`. + defp forwarded_ref(effects) do + [ref] = for {:forward, _leader, ref, _kind} <- effects, do: ref + ref + end + + describe "submit/5" do + test "a register with a reachable leader is forwarded, tracked", %{pid: pid, from: from} do + {_t, effects} = Intents.submit(register(pid), from, 0, {:remote, @l1}, Intents.new()) + + assert [{:forward, @l1, ref, {:register, :name, ^pid, _}}] = effects + assert is_reference(ref) + end + + test "a register with no reachable leader is held, unanswered", %{pid: pid, from: from} do + assert {_t, []} = Intents.submit(register(pid), from, 0, :none, Intents.new()) + end + + test "an unregister with no reachable leader is answered ok now and held", %{ + pid: pid, + from: from + } do + {t, effects} = Intents.submit({:unregister, :name, pid}, from, 0, :none, Intents.new()) + assert effects == [{:reply, from, :ok}] + + # ...and its effect is still owed: the next leader gets a pid-guarded retract. + assert {_t, [{:retract, @l1, :name, ^pid}]} = + Intents.redrive({:remote, @l1}, :moved, 0, @ttl, t) + end + + test "an unregister of a locally unbound name has nothing to retract", %{from: from} do + assert {_t, [{:reply, ^from, :ok}]} = + Intents.submit({:unregister, :name, :undefined}, from, 0, :none, Intents.new()) + end + + test "a set_meta with no reachable leader fails fast", %{from: from} do + assert {_t, [{:reply, ^from, {:error, :no_leader}}]} = + Intents.submit({:set_meta, :name, %{}, nil}, from, 0, :none, Intents.new()) + end + end + + describe "on_reply/4 — a register's verdict" do + setup %{pid: pid, from: from} do + {t, effects} = Intents.submit(register(pid), from, 0, {:remote, @l1}, Intents.new()) + {:ok, t: t, ref: forwarded_ref(effects)} + end + + test "yes at or behind our version is answered", %{t: t, ref: ref, from: from} do + assert {_t, [{:reply, ^from, :yes}]} = Intents.on_reply(ref, {:register, :yes, 10}, 10, t) + end + + test "yes ahead of our version waits for the replica and asks for a resync", %{ + t: t, + ref: ref, + from: from + } do + {t, effects} = Intents.on_reply(ref, {:register, :yes, 20}, 10, t) + assert effects == [{:resync, @l1}] + + assert {_t, []} = Intents.on_version(15, t) + assert {_t, [{:reply, ^from, :yes}]} = Intents.on_version(20, t) + end + + test "no is answered no", %{t: t, ref: ref, from: from} do + assert {_t, [{:reply, ^from, :no}]} = Intents.on_reply(ref, {:register, :no, 10}, 10, t) + end + + test "a legacy yes binds the row itself", %{t: t, ref: ref, from: from, pid: pid} do + assert {_t, [{:reply, ^from, :yes}, {:insert_row, :name, ^pid, _}]} = + Intents.on_reply(ref, {:register, :yes, :legacy}, 0, t) + end + + test "a reply for an unknown reference is ignored", %{t: t} do + assert {^t, []} = Intents.on_reply(make_ref(), {:register, :yes, 10}, 10, t) + end + end + + describe "redrive/5 — registers are re-sent, never answered no" do + test "a forward to a deposed leader goes to the new one, under a fresh reference", %{ + pid: pid, + from: from + } do + {t, effects} = Intents.submit(register(pid), from, 0, {:remote, @l1}, Intents.new()) + old_ref = forwarded_ref(effects) + + {t, effects} = Intents.redrive({:remote, @l2}, :moved, 0, @ttl, t) + assert [{:forward, @l2, new_ref, {:register, :name, ^pid, _}}] = effects + assert new_ref != old_ref + + # The deposed leader's late answer no longer finds anyone to answer. + assert {_t, []} = Intents.on_reply(old_ref, {:register, :no, 10}, 10, t) + end + + test "a forward to the leader that is still current is kept on a new view", %{ + pid: pid, + from: from + } do + {t, effects} = Intents.submit(register(pid), from, 0, {:remote, @l1}, Intents.new()) + ref = forwarded_ref(effects) + + # A snapshot from the same leader: it received the forward (or postponed it + # through its assume window), so re-sending would only commit it twice. + assert {t, []} = Intents.redrive({:remote, @l1}, :moved, 0, @ttl, t) + assert {_t, [{:reply, ^from, :yes}]} = Intents.on_reply(ref, {:register, :yes, 1}, 1, t) + end + + test "a restored link re-sends a forward to the same leader (the cast may be lost)", %{ + pid: pid, + from: from + } do + {t, _} = Intents.submit(register(pid), from, 0, {:remote, @l1}, Intents.new()) + + assert {_t, [{:forward, @l1, _ref, _kind}]} = + Intents.redrive({:remote, @l1}, :all, 0, @ttl, t) + end + + test "a gap-deferred yes is re-driven if its leader no longer leads", %{ + pid: pid, + from: from + } do + {t, effects} = Intents.submit(register(pid), from, 0, {:remote, @l1}, Intents.new()) + {t, _} = Intents.on_reply(forwarded_ref(effects), {:register, :yes, 20}, 10, t) + + # Same leader: still waiting on our replica, nothing to send. + assert {_t, []} = Intents.redrive({:remote, @l1}, :moved, 0, @ttl, t) + # New leader: re-sent, where it is decided idempotently. + assert {_t, [{:forward, @l2, _ref, _kind}]} = + Intents.redrive({:remote, @l2}, :moved, 0, @ttl, t) + end + + test "when we lead, a register goes to our own commit pipeline", %{pid: pid, from: from} do + {t, _} = Intents.submit(register(pid), from, 0, :none, Intents.new()) + + assert {_t, [{:enqueue, {:register, :name, ^pid, _}, ^from}]} = + Intents.redrive(:self, :moved, 0, @ttl, t) + end + + test "a register whose caller has timed out is dropped without a reply", %{ + pid: pid, + from: from + } do + {t, _} = Intents.submit(register(pid), from, 0, :none, Intents.new()) + + assert {_t, []} = Intents.redrive({:remote, @l1}, :moved, @ttl, @ttl, t) + end + end + + describe "redrive/5 — unregisters and set_meta" do + test "a forwarded unregister becomes a pid-guarded retract, caller answered ok", %{ + pid: pid, + from: from + } do + {t, effects} = + Intents.submit({:unregister, :name, pid}, from, 0, {:remote, @l1}, Intents.new()) + + assert [{:forward, @l1, _ref, {:unregister, :name, ^pid}}] = effects + + assert {_t, [{:reply, ^from, :ok}, {:retract, @l2, :name, ^pid}]} = + Intents.redrive({:remote, @l2}, :moved, 0, @ttl, t) + end + + test "an unregister is never dropped for age", %{pid: pid, from: from} do + {t, _} = Intents.submit({:unregister, :name, pid}, from, 0, :none, Intents.new()) + t = Intents.prune(10 * @ttl, @ttl, t) + + assert {_t, [{:enqueue, {:unregister, :name, ^pid}, :answered}]} = + Intents.redrive(:self, :moved, 10 * @ttl, @ttl, t) + end + + test "a set_meta keeps waiting on the same leader, fails on a new one", %{from: from} do + {t, _} = + Intents.submit({:set_meta, :name, %{}, nil}, from, 0, {:remote, @l1}, Intents.new()) + + assert {_t, []} = Intents.redrive({:remote, @l1}, :moved, 0, @ttl, t) + + assert {_t, [{:reply, ^from, {:error, :no_leader}}]} = + Intents.redrive({:remote, @l2}, :moved, 0, @ttl, t) + end + end + + describe "prune/3" do + test "drops timed-out registers and set_metas only", %{pid: pid, from: from} do + {t, _} = Intents.submit(register(pid), from, 0, :none, Intents.new()) + + {t, _} = + Intents.submit({:set_meta, :n2, %{}, nil}, from, 0, {:remote, @l1}, t) + + {t, _} = Intents.submit({:unregister, :n3, pid}, from, 0, :none, t) + + t = Intents.prune(@ttl, @ttl, t) + + # Only the unregister is left to re-drive. + assert {_t, [{:retract, @l1, :n3, ^pid}]} = + Intents.redrive({:remote, @l1}, :moved, @ttl, @ttl, t) + end + end +end diff --git a/test/dgen_registry_observability_test.exs b/test/dgen_registry_observability_test.exs index 9de728a..0432532 100644 --- a/test/dgen_registry_observability_test.exs +++ b/test/dgen_registry_observability_test.exs @@ -36,7 +36,7 @@ defmodule DGen.RegistryObservabilityTest do # The published view and the member's own state record must agree. defp assert_agrees!(name, context) do - published = :eta_observe.read(name) + published = Cluster.status(name) actual = :dgen_registry.status(name) assert is_map(published), "#{context}: #{name} published nothing" @@ -65,7 +65,7 @@ defmodule DGen.RegistryObservabilityTest do leaders = for name <- Cluster.alive(c), - s = :eta_observe.read(name), + s = Cluster.status(name), s.leader == s.member_id, do: name @@ -89,7 +89,7 @@ defmodule DGen.RegistryObservabilityTest do # applied_version moves with the writes, which is the point of republishing # on every callback rather than only on leadership transitions. - assert :eta_observe.read(leader).applied_version > 0 + assert Cluster.status(leader).applied_version > 0 end test "across a leadership change", %{tenant: tenant} do @@ -97,7 +97,7 @@ defmodule DGen.RegistryObservabilityTest do on_exit(fn -> Cluster.stop(c) end) {_node, old_leader} = Cluster.leader(c) - old_epoch = :eta_observe.read(old_leader).epoch + old_epoch = Cluster.status(old_leader).epoch Cluster.crash(c, old_leader) @@ -109,10 +109,21 @@ defmodule DGen.RegistryObservabilityTest do end), "no new leader was elected" + # The elector names the new leader before that member has finished assuming + # (the handoff gather runs off its loop), and the published view and + # status/1 are two reads at two instants. Wait for the handoff to settle, + # then require agreement. + assert eventually(fn -> + Enum.all?(Cluster.alive(c), fn name -> + match?(%{leader: leader} when leader != :undefined, Cluster.status(name)) + end) + end), + "the handoff never settled" + assert_all_agree!(c, "after a leadership change") {_node, new_leader} = Cluster.leader(c) - new = :eta_observe.read(new_leader) + new = Cluster.status(new_leader) assert new.leader == new.member_id assert new.epoch > old_epoch, "the epoch did not advance" @@ -171,14 +182,14 @@ defmodule DGen.RegistryObservabilityTest do [name] = Cluster.alive(c) member = Process.whereis(name) - expected = :eta_observe.read(name) + expected = Cluster.status(name) assert expected.leader == expected.member_id :erlang.suspend_process(member) try do # The whole point: a live read of a process that cannot answer anything. - assert :eta_observe.read(name) == expected + assert Cluster.status(name) == expected # And the failure mode it replaces. Not an error — a plausible answer, # which is what makes an invariant built on it quietly vacuous. diff --git a/test/dgen_registry_pipeline_test.exs b/test/dgen_registry_pipeline_test.exs new file mode 100644 index 0000000..2186ee5 --- /dev/null +++ b/test/dgen_registry_pipeline_test.exs @@ -0,0 +1,165 @@ +defmodule DGen.RegistryPipelineTest do + @moduledoc """ + The leader's write pipeline, in isolation — no member process, no backend. + + Pins the phases (queued → committing → awaiting holders) and the one rule for an + op the leader cannot finish, `surrender/2`: by kind, the same idempotence rule as + the follower's intents. A registration is never answered `no` because the leader + could not finish it. + """ + use ExUnit.Case, async: true + + alias :dgen_registry_pipeline, as: Pipeline + + @meta {%{}, :undefined} + @follower {:n2@host, :reg} + + setup do + {:ok, pid: spawn(fn -> Process.sleep(:infinity) end), from: {self(), make_ref()}} + end + + describe "queued and committing" do + test "batches are taken oldest first, bounded, and one commit runs at a time", %{pid: pid} do + p = Enum.reduce(1..3, Pipeline.new(), &Pipeline.enqueue({:retract, &1, pid}, &2)) + + {batch, p} = Pipeline.take_batch(2, p) + assert batch == [{:retract, 1, pid}, {:retract, 2, pid}] + assert Pipeline.num_pending(p) == 1 + + ref = make_ref() + mref = make_ref() + p = Pipeline.start(ref, mref, %{ops: batch}, p) + assert Pipeline.in_flight(p) + + assert :stale = Pipeline.finish(make_ref(), p) + assert {:ok, %{ops: ^batch}, ^mref, p} = Pipeline.finish(ref, p) + refute Pipeline.in_flight(p) + end + + test "a dead worker releases the slot too", %{pid: pid} do + mref = make_ref() + p = Pipeline.start(make_ref(), mref, %{ops: [{:retract, :n, pid}]}, Pipeline.new()) + + assert :stale = Pipeline.worker_down(make_ref(), p) + assert {:ok, %{ops: [_]}, p} = Pipeline.worker_down(mref, p) + refute Pipeline.in_flight(p) + end + + test "the cached read version is dropped at the end of a term" do + p = Pipeline.committed(42, Pipeline.new()) + assert Pipeline.read_version(p) == 42 + assert Pipeline.read_version(Pipeline.new_term(p)) == :undefined + end + end + + describe "awaiting holders" do + test "distinct followers count once each, up to the target", %{pid: pid, from: from} do + direct = [{{:local, from}, :n, pid}] + batch = make_ref() + timer = make_ref() + p = Pipeline.await_acks(batch, direct, timer, 2, Pipeline.new()) + + assert {:waiting, p} = Pipeline.ack(batch, @follower, p) + assert {:waiting, p} = Pipeline.ack(batch, @follower, p) + assert {:done, ^direct, ^timer, p} = Pipeline.ack(batch, {:n3@host, :reg}, p) + assert :none = Pipeline.ack_timeout(batch, p) + end + + test "a timeout reports how many confirmed", %{pid: pid, from: from} do + batch = make_ref() + p = Pipeline.await_acks(batch, [{{:local, from}, :n, pid}], make_ref(), 2, Pipeline.new()) + {:waiting, p} = Pipeline.ack(batch, @follower, p) + + assert {:expired, [_], 2, 1, _p} = Pipeline.ack_timeout(batch, p) + end + + test "losing leadership takes every batch, in arrival order", %{pid: pid, from: from} do + p = + Enum.reduce(1..3, Pipeline.new(), fn n, acc -> + Pipeline.await_acks(make_ref(), [{{:local, from}, n, pid}], make_ref(), 1, acc) + end) + + {batches, p} = Pipeline.take_acks(p) + assert [1, 2, 3] == for({[{_, n, _}], _, _, _} <- batches, do: n) + assert {[], _} = Pipeline.take_acks(p) + end + end + + describe "surrender(:deposed, ops)" do + test "a local registration becomes a follower intent, keeping its caller", %{ + pid: pid, + from: from + } do + assert [{:intent, {:register, :n, ^pid, @meta}, ^from}] = + Pipeline.surrender(:deposed, [{:add, :n, pid, @meta, {:local, from}}]) + end + + test "a forwarded registration is left to its follower — no `no`", %{pid: pid} do + assert [] = + Pipeline.surrender(:deposed, [ + {:add, :n, pid, @meta, {:forward, @follower, make_ref()}} + ]) + end + + test "a set_meta fails: a replace is not safe to re-send", %{from: from} do + assert [{:reply, {:meta, ^from}, {:error, :no_leader}}] = + Pipeline.surrender(:deposed, [{:set_meta, :n, %{}, nil, {:meta, from}}]) + end + + test "an unregister is answered ok and held as a pid-guarded retract", %{ + pid: pid, + from: from + } do + assert [ + {:reply, {:unreg, ^from}, :ok}, + {:intent, {:unregister, :n, ^pid}, :answered} + ] = Pipeline.surrender(:deposed, [{:remove, :n, pid, {:unreg, from}}]) + + assert [{:intent, {:unregister, :m, ^pid}, :answered}] = + Pipeline.surrender(:deposed, [{:retract, :m, pid}]) + + # Nothing was bound when it was asked: nothing to retract. + assert [{:reply, {:unreg, ^from}, :ok}] = + Pipeline.surrender(:deposed, [{:remove, :n, :undefined, {:unreg, from}}]) + end + + test "a down is dropped: the next leader monitors the pid itself" do + assert [] = Pipeline.surrender(:deposed, [{:down, :n, make_ref()}]) + end + end + + describe "surrender(:failed, ops)" do + test "everything idempotent is retried, in order; only set_meta fails", %{ + pid: pid, + from: from + } do + add = {:add, :a, pid, @meta, {:local, from}} + fwd = {:add, :b, pid, @meta, {:forward, @follower, make_ref()}} + down = {:down, :c, make_ref()} + retract = {:retract, :d, pid} + + effects = + Pipeline.surrender(:failed, [ + add, + {:set_meta, :x, %{}, nil, {:meta, from}}, + {:remove, :e, pid, {:unreg, from}}, + fwd, + down, + retract + ]) + + assert effects == [ + {:reply, {:meta, from}, {:error, :no_leader}}, + # The unregister's caller is answered now; its removal is retried + # without an origin so it is not answered twice. + {:reply, {:unreg, from}, :ok}, + {:retry, [add, {:remove, :e, pid, :undefined}, fwd, down, retract]} + ] + end + + test "nothing to retry, no retry effect", %{from: from} do + assert [{:reply, _, {:error, :no_leader}}] = + Pipeline.surrender(:failed, [{:set_meta, :x, %{}, nil, {:meta, from}}]) + end + end +end diff --git a/test/dgen_registry_replica_test.exs b/test/dgen_registry_replica_test.exs new file mode 100644 index 0000000..4249e2f --- /dev/null +++ b/test/dgen_registry_replica_test.exs @@ -0,0 +1,168 @@ +defmodule DGen.RegistryReplicaTest do + @moduledoc """ + The replica's admission rules, in isolation — no member process, no backend. + + Each `describe` is one of the TLA+ model's receive actions + (formal/DgenRegistryReplication.tla) that `dgen_registry_replica` implements: + `offer_batch/5` is RecvBcastApply / RecvBcastDrop / RecvBcastGap, and + `offer_snapshot/4` is RecvSnap with the SafeAssumeSnap guard. The invariant they + protect is that two replicas at the same version hold the same content. + """ + use ExUnit.Case, async: true + + alias :dgen_registry_replica, as: Replica + + # A fresh replica at {epoch, version}, holding `records`. Each test gets its own + # uniquely named table (the replica's table is a named ETS table). + defp replica(epoch, version, records \\ %{}) do + name = :"replica_test_#{System.unique_integer([:positive])}" + Replica.adopt(epoch, version, records, Replica.new(name)) + end + + defp reg(name, pid, index \\ %{}), do: {:name_registered, name, pid, index, nil} + defp unreg(name, pid), do: {:name_unregistered, name, pid} + + defp pids(r), do: Map.new(Replica.records(r), fn {n, {p, _, _}} -> {n, p} end) + + setup do + {:ok, + p1: spawn(fn -> Process.sleep(:infinity) end), p2: spawn(fn -> Process.sleep(:infinity) end)} + end + + describe "offer_batch/5 — RecvBcastApply" do + test "a contiguous batch applies whole and advances the version", %{p1: p1, p2: p2} do + r = replica(1, 10) + + assert {:applied, r} = Replica.offer_batch(1, 10, 20, [reg(:a, p1), reg(:b, p2)], r) + assert Replica.version(r) == 20 + assert pids(r) == %{a: p1, b: p2} + end + + test "a batch that binds and clears one name ends without it", %{p1: p1} do + r = replica(1, 10) + + assert {:applied, r} = Replica.offer_batch(1, 10, 20, [reg(:a, p1), unreg(:a, p1)], r) + assert pids(r) == %{} + end + + test "a newer epoch's contiguous batch applies but does not raise the epoch", %{p1: p1} do + r = replica(1, 10) + + assert {:applied, r} = Replica.offer_batch(2, 10, 20, [reg(:a, p1)], r) + assert Replica.epoch(r) == 1 + assert Replica.version(r) == 20 + end + + test "the heartbeat (an empty batch at {V, V}) is a no-op on a caught-up replica" do + r = replica(1, 10) + + assert {:applied, r} = Replica.offer_batch(1, 10, 10, [], r) + assert Replica.version(r) == 10 + end + + test "metadata changes keep the inverted index in step", %{p1: p1} do + r = replica(1, 10) + {:applied, r} = Replica.offer_batch(1, 10, 20, [reg(:a, p1, %{role: :w})], r) + assert [%{name: :a}] = Replica.query(r, {:all, %{role: :w}}) + + {:applied, r} = Replica.offer_batch(1, 20, 30, [{:metadata_set, :a, %{role: :x}, nil}], r) + assert [] = Replica.query(r, {:all, %{role: :w}}) + assert [%{name: :a}] = Replica.query(r, {:all, %{role: :x}}) + end + end + + describe "offer_batch/5 — RecvBcastDrop" do + test "a deposed leader's batch is refused", %{p1: p1} do + r = replica(2, 10) + + assert :stale_epoch = Replica.offer_batch(1, 10, 20, [reg(:a, p1)], r) + end + + test "a batch at or behind our version is a duplicate", %{p1: p1} do + r = replica(1, 20) + + assert :duplicate = Replica.offer_batch(1, 10, 20, [reg(:a, p1)], r) + assert :duplicate = Replica.offer_batch(1, 5, 15, [reg(:a, p1)], r) + end + end + + describe "offer_batch/5 — RecvBcastGap" do + test "a batch past a hole applies nothing", %{p1: p1} do + r = replica(1, 10, %{}) + + assert :gap = Replica.offer_batch(1, 15, 20, [reg(:a, p1)], r) + assert pids(r) == %{} + assert Replica.version(r) == 10 + end + + test "a heartbeat ahead of us reveals a lost tail" do + r = replica(1, 10) + + assert :gap = Replica.offer_batch(1, 20, 20, [], r) + end + end + + describe "offer_snapshot/4 — RecvSnap" do + test "a current-epoch snapshot at or past our version re-baselines", %{p1: p1, p2: p2} do + r = replica(1, 10, %{a: {p1, %{}, nil}}) + + assert {:accepted, r} = Replica.offer_snapshot(1, 20, %{b: {p2, %{}, nil}}, r) + assert pids(r) == %{b: p2} + assert Replica.version(r) == 20 + end + + test "a newer epoch's snapshot sets the epoch", %{p1: p1} do + r = replica(1, 10) + + assert {:accepted, r} = Replica.offer_snapshot(3, 10, %{a: {p1, %{}, nil}}, r) + assert Replica.epoch(r) == 3 + end + + test "a snapshot behind our version is refused — no rollback", %{p1: p1, p2: p2} do + r = replica(1, 20, %{a: {p1, %{}, nil}}) + + assert :stale = Replica.offer_snapshot(2, 10, %{b: {p2, %{}, nil}}, r) + assert pids(r) == %{a: p1} + end + + test "an older epoch's snapshot is refused", %{p1: p1} do + r = replica(2, 10) + + assert :stale = Replica.offer_snapshot(1, 30, %{a: {p1, %{}, nil}}, r) + end + end + + describe "adopt/4 — a wholesale re-baseline" do + test "replaces the content in place: kept names updated, absent names removed", %{ + p1: p1, + p2: p2 + } do + r = replica(1, 10, %{a: {p1, %{k: 1}, nil}, b: {p1, %{}, nil}}) + + r = Replica.adopt(2, 20, %{a: {p2, %{k: 2}, nil}, c: {p2, %{}, nil}}, r) + assert pids(r) == %{a: p2, c: p2} + assert [] = Replica.query(r, {:all, %{k: 1}}) + assert [%{name: :a}] = Replica.query(r, {:all, %{k: 2}}) + assert {Replica.epoch(r), Replica.version(r)} == {2, 20} + end + end + + describe "commit/5 — the leader's own batch" do + test "applies only the delta's names and advances the version", %{p1: p1, p2: p2} do + r = replica(1, 10, %{a: {p1, %{}, nil}, b: {p2, %{}, nil}}) + + r = Replica.commit(20, %{a: :clear}, %{}, %{}, r) + assert pids(r) == %{b: p2} + assert Replica.version(r) == 20 + end + + test "a metadata-only op never resurrects an optimistically deleted row", %{p1: p1} do + r = replica(1, 10, %{a: {p1, %{}, nil}}) + r = Replica.delete(r, :a) + + r = Replica.commit(20, %{a: {:meta, node()}}, %{}, %{a: {%{k: 1}, nil}}, r) + assert pids(r) == %{} + assert [] = Replica.query(r, {:all, %{k: 1}}) + end + end +end diff --git a/test/dgen_registry_test.exs b/test/dgen_registry_test.exs index 60b4e2d..7750b50 100644 --- a/test/dgen_registry_test.exs +++ b/test/dgen_registry_test.exs @@ -691,142 +691,10 @@ defmodule DGen.RegistryTest do end end - # --------------------------------------------------------------------------- - # §5.6 conflict-detection predicate (pure, unit-tested in isolation) - # - # The full end-to-end scenario that produces a genuine conflict needs a - # multi-node, leadership-orchestrated setup (a member holds a divergent live - # binding that a leadership-change gather exposes); that lives in the cluster - # suite ("conflict resolution by termination" in dgen_registry_cluster_test.exs). - # Here we pin the safety-critical predicate itself: which gathered states count - # as a conflict, and — crucially — which do not (so we never false-kill a live - # process). - # --------------------------------------------------------------------------- - - describe "detect_conflicts/3" do - # The detector takes RECORDS maps (`%{name => {pid, index, data}}`) and - # compares entries against the authority in place — the shape change that made - # the all-agree handoff allocation-free. `rec/1` builds the record wrapper; - # the semantics pinned below are unchanged from the pid-map era. - defp rec(pid), do: {pid, %{}, nil} - - test "two different live pids for one name is a conflict" do - p1 = spawn_live() - p2 = spawn_live() - - assert [{:n, ^p1, [^p2]}] = - :dgen_registry_member.detect_conflicts( - %{n: rec(p1)}, - [%{n: rec(p1)}, %{n: rec(p2)}], - %{} - ) - end - - test "a recently-released name/pid pair is suppressed (lag, not conflict)" do - p1 = spawn_live() - p2 = spawn_live() - # Trail entries are keyed {name, pid} (§5.6). - released = %{{:n, p2} => System.system_time(:millisecond)} - - assert [] = - :dgen_registry_member.detect_conflicts( - %{n: rec(p1)}, - [%{n: rec(p1)}, %{n: rec(p2)}], - released - ) - end - - test "a release under one name does not mask a conflict on another name" do - p1 = spawn_live() - p2 = spawn_live() - # p2 was explicitly released from :other — that must not suppress the - # genuine divergence on :n, which p2 also (still) claims. - released = %{{:other, p2} => System.system_time(:millisecond)} - - assert [{:n, ^p1, [^p2]}] = - :dgen_registry_member.detect_conflicts( - %{n: rec(p1)}, - [%{n: rec(p1)}, %{n: rec(p2)}], - released - ) - end - - test "a legacy bare-pid trail entry (rolling upgrade) still suppresses" do - p1 = spawn_live() - p2 = spawn_live() - # A pre-upgrade member's gathered trail keys by pid alone; the detector - # honours that coarser shape until the mixed entries age out. - released = %{p2 => System.system_time(:millisecond)} - - assert [] = - :dgen_registry_member.detect_conflicts( - %{n: rec(p1)}, - [%{n: rec(p1)}, %{n: rec(p2)}], - released - ) - end - - test "a dead divergent pid is not a conflict" do - p1 = spawn_live() - p2 = spawn(fn -> :ok end) - Process.exit(p2, :kill) - eventually(fn -> not Process.alive?(p2) end) - - assert [] = - :dgen_registry_member.detect_conflicts( - %{n: rec(p1)}, - [%{n: rec(p1)}, %{n: rec(p2)}], - %{} - ) - end - - test "a dead authority is not a conflict (nothing to defend)" do - p1 = spawn(fn -> :ok end) - Process.exit(p1, :kill) - eventually(fn -> not Process.alive?(p1) end) - p2 = spawn_live() - - assert [] = - :dgen_registry_member.detect_conflicts( - %{n: rec(p1)}, - [%{n: rec(p1)}, %{n: rec(p2)}], - %{} - ) - end - - test "agreement across all maps is not a conflict" do - p1 = spawn_live() - - assert [] = - :dgen_registry_member.detect_conflicts( - %{n: rec(p1)}, - [%{n: rec(p1)}, %{n: rec(p1)}], - %{} - ) - end - - test "agreement with different metadata is still agreement (pids decide)" do - p1 = spawn_live() - - assert [] = - :dgen_registry_member.detect_conflicts( - %{n: {p1, %{v: 1}, nil}}, - [%{n: {p1, %{v: 2}, :other}}], - %{} - ) - end - - test "a name with no authority (absent from the freshest map) is not a conflict" do - p1 = spawn_live() - - assert [] = :dgen_registry_member.detect_conflicts(%{}, [%{n: rec(p1)}], %{}) - end - end - # --------------------------------------------------------------------------- # Snapshot codec (pure, unit-tested in isolation) # - # encode_table/1 folds the ETS table straight into a pair list inside the blob + # encode/1 folds the replica's ETS table straight into a pair list inside the blob # (no records-map materialization); decode_records/1 must normalize every wire # shape — this build's list-in-binary and map-in-binary, plus both pre-binary # legacy forms — to the one map callers use. A shape the decoder silently @@ -834,22 +702,25 @@ defmodule DGen.RegistryTest do # the tolerance is pinned here rather than trusted. # --------------------------------------------------------------------------- - describe "snapshot codec (encode_table/1, decode_records/1)" do - test "a table round-trips through encode_table to the same records map" do - tab = :ets.new(:codec_bench, [:set, :public]) + describe "snapshot codec (dgen_registry_replica encode/1, decode_records/1)" do + test "a replica round-trips through encode to the same records map" do p1 = spawn_live() p2 = spawn_live() - :ets.insert(tab, [{:a, p1, %{v: 1}, nil}, {:b, p2, %{}, "data"}]) + + replica = + :dgen_registry_replica.new(:codec_bench) + |> :dgen_registry_replica.insert(:a, p1, %{v: 1}, nil) + |> :dgen_registry_replica.insert(:b, p2, %{}, "data") assert %{a: {^p1, %{v: 1}, nil}, b: {^p2, %{}, "data"}} = - :dgen_registry_member.decode_records(:dgen_registry_member.encode_table(tab)) + :dgen_registry_replica.decode_records(:dgen_registry_replica.encode(replica)) end - test "an empty table encodes to an empty map" do - tab = :ets.new(:codec_empty, [:set, :public]) + test "an empty replica encodes to an empty map" do + replica = :dgen_registry_replica.new(:codec_empty) assert %{} == - :dgen_registry_member.decode_records(:dgen_registry_member.encode_table(tab)) + :dgen_registry_replica.decode_records(:dgen_registry_replica.encode(replica)) end test "a map-in-binary blob (the fan-out encoder) decodes to the same map" do @@ -857,7 +728,9 @@ defmodule DGen.RegistryTest do records = %{a: {p1, %{}, nil}} assert ^records = - :dgen_registry_member.decode_records(:dgen_registry_member.encode_records(records)) + :dgen_registry_replica.decode_records( + :dgen_registry_replica.encode_records(records) + ) end test "legacy wire shapes still normalize (rolling upgrade)" do @@ -865,15 +738,15 @@ defmodule DGen.RegistryTest do # Pre-binary-era apply_names_snapshot: an on-heap [{name, record}] list. assert %{a: {^p1, %{}, nil}} = - :dgen_registry_member.decode_records([{:a, {p1, %{}, nil}}]) + :dgen_registry_replica.decode_records([{:a, {p1, %{}, nil}}]) # Pre-binary-era get_names_snapshot: a bare records map. assert %{a: {^p1, %{}, nil}} = - :dgen_registry_member.decode_records(%{a: {p1, %{}, nil}}) + :dgen_registry_replica.decode_records(%{a: {p1, %{}, nil}}) # Pre-list-era binary: a map inside the blob (what an old node still sends). assert %{a: {^p1, %{}, nil}} = - :dgen_registry_member.decode_records( + :dgen_registry_replica.decode_records( :erlang.term_to_binary(%{a: {p1, %{}, nil}}, [:compressed]) ) end @@ -1311,7 +1184,7 @@ defmodule DGen.RegistryTest do # old assume/resync snapshot delivered late, after the member applied a newer # broadcast) would overwrite the replica *backward* and silently drop an already # -acked binding. The fix requires the snapshot's version to be >= the member's - # applied version. See formal/README.md "Discovered finding and the fix". + # applied version. See formal/README.md (the HandoffRace configs) and design §5.7. # --------------------------------------------------------------------------- describe "apply_names_snapshot version-monotonicity (handoff-gather race)" do @@ -1403,4 +1276,302 @@ defmodule DGen.RegistryTest do assert :undefined = :dgen_registry.elector_pid(:no_such_registry_at_all) end end + + # --------------------------------------------------------------------------- + # Intents across a leadership change (dgen_registry_intents) + # + # A registration forwarded to a leader that is deposed before answering is + # re-driven to the new leader, never answered `no`: `no` is reserved for an + # adjudicated refusal (§3). The leaders here are test processes registered + # under fake member names on this node, so the test sees exactly what the + # member forwards; snapshots naming them demote the real member to a follower. + # --------------------------------------------------------------------------- + + describe "a forwarded registration across a leadership change" do + defp fake_leader do + name = :"fake_leader_#{:erlang.unique_integer([:positive])}" + Process.register(self(), name) + {node(), name} + end + + defp follow(member, leader, epoch, version) do + GenServer.cast(member, {:apply_names_snapshot, %{}, leader, [], %{}, epoch, version}) + :sys.get_state(member) + end + + defp await_forward(timeout \\ 1_000) do + receive do + {:"$gen_cast", {:register_req, ref, _from, name, pid, _meta}} -> {ref, name, pid} + after + timeout -> flunk("no register_req was forwarded") + end + end + + test "is re-sent to the new leader, and its caller gets that leader's verdict", %{ + reg: reg + } do + member = :dgen_registry.member_name(reg) + pid = spawn(fn -> Process.sleep(:infinity) end) + on_exit(fn -> Process.exit(pid, :kill) end) + + epoch = :dgen_registry.get_epoch(reg) + v = applied_version(member) + + # Demote the member to a follower of leader 1 (this test process). + l1 = fake_leader() + follow(member, l1, epoch + 1, v) + + caller = Task.async(fn -> :dgen_registry.register_name({reg, :moved}, pid) end) + {ref1, :moved, ^pid} = await_forward() + + # Leader 1 is deposed before it answers. The member re-drives the forward to + # leader 2 (same test process, new name), under a fresh reference. + Process.unregister(elem(l1, 1)) + l2 = fake_leader() + follow(member, l2, epoch + 2, v) + {ref2, :moved, ^pid} = await_forward() + assert ref2 != ref1 + + # The caller is still waiting: nothing answered it `no`. + assert Task.yield(caller, 100) == nil + + # Leader 1's late `no` finds no one to answer; leader 2's `yes` decides it. + GenServer.cast(member, {:register_reply, ref1, :no, v}) + GenServer.cast(member, {:register_reply, ref2, :yes, v}) + assert :yes == Task.await(caller) + end + + test "is not re-sent on a snapshot from the leader it already reached", %{reg: reg} do + member = :dgen_registry.member_name(reg) + pid = spawn(fn -> Process.sleep(:infinity) end) + on_exit(fn -> Process.exit(pid, :kill) end) + + epoch = :dgen_registry.get_epoch(reg) + v = applied_version(member) + l1 = fake_leader() + follow(member, l1, epoch + 1, v) + + caller = Task.async(fn -> :dgen_registry.register_name({reg, :reached}, pid) end) + {ref1, :reached, ^pid} = await_forward() + + # A resync snapshot from the same leader: the forward was delivered, so a second + # copy would only be a second commit — racing, e.g., a concurrent unregister. + follow(member, l1, epoch + 1, v) + refute_receive {:"$gen_cast", {:register_req, _, _, _, _, _}}, 100 + + GenServer.cast(member, {:register_reply, ref1, :yes, v}) + assert :yes == Task.await(caller) + end + + test "a member that does not lead stays silent rather than answer `no`", %{reg: reg} do + member = :dgen_registry.member_name(reg) + pid = spawn(fn -> Process.sleep(:infinity) end) + on_exit(fn -> Process.exit(pid, :kill) end) + + # Demote the member to a follower of some other (idle) leader. + other = :"other_leader_#{:erlang.unique_integer([:positive])}" + Process.register(pid, other) + follow(member, {node(), other}, :dgen_registry.get_epoch(reg) + 1, applied_version(member)) + + # A follower forwards a registration to it, believing it leads. + follower = fake_leader() + ref = make_ref() + GenServer.cast(member, {:register_req, ref, follower, :name, pid, {%{}, :undefined}}) + :sys.get_state(member) + + refute_receive {:"$gen_cast", {:register_reply, ^ref, _}}, 200 + refute_receive {:"$gen_cast", {:register_reply, ^ref, _, _}} + end + end + + # --------------------------------------------------------------------------- + # Leader exits: an op the leader buffered but can no longer commit + # + # A leader that is re-elected without ever hearing it was deposed (the + # minority side of a partition) takes the genuine assume path, and its + # `leader` is `undefined` for the assume window. If an in-flight commit + # reports back inside that window, the buffered ops behind it are flushed with + # no leader to send them to. A buffered unregister must not be dropped there: + # its caller was answered `ok`, and its durable clear is owed (Non-goal 5). + # Nor may the registration whose commit was in flight be answered `no`: it is + # retried, and re-sent to whoever leads (dgen_registry_pipeline:surrender/2). + # + # The window is driven by hand: the member is suspended while a register (to + # put a commit in flight), an unregister (to sit buffered behind it) and an + # assume for a newer epoch are queued, then resumed. The elector is suspended + # too, so the assume's off-loop gather waits out its subscription pull + # (?SUBS_PULL_TIMEOUT), which holds the window open. A test process then plays + # the leader that actually won, by sending a snapshot naming it. + # --------------------------------------------------------------------------- + + describe "a buffered op the leader can no longer commit" do + test "ops caught in the assume window reach the next leader, not `no` or nowhere", %{ + reg: reg + } do + member = :dgen_registry.member_name(reg) + self_id = {node(), member} + epoch = :dgen_registry.get_epoch(reg) + + held = spawn(fn -> Process.sleep(:infinity) end) + in_flight = spawn(fn -> Process.sleep(:infinity) end) + on_exit(fn -> Enum.each([held, in_flight], &Process.exit(&1, :kill)) end) + + :yes = :dgen_registry.register_name({reg, :held}, held) + v = applied_version(member) + + # A suspension is counted per suspender, so it is undone here, not in on_exit. + elector = :dgen_registry.elector_pid(reg) + :erlang.suspend_process(elector) + + try do + drive_assume_window(reg, member, self_id, epoch, held, in_flight, v) + after + :erlang.resume_process(elector) + end + end + + defp drive_assume_window(reg, member, self_id, epoch, held, in_flight, v) do + :sys.suspend(member) + register = Task.async(fn -> :dgen_registry.register_name({reg, :in_flight}, in_flight) end) + Process.sleep(50) + unregister = Task.async(fn -> :dgen_registry.unregister_name({reg, :held}) end) + Process.sleep(50) + + assume = + Task.async(fn -> + :gen_statem.call( + member, + {:elector_assume_and_distribute, self_id, [self_id], %{}, [], epoch + 1} + ) + end) + + Process.sleep(50) + :sys.resume(member) + + assert :ok == Task.await(assume) + # The flush answers the buffered unregister's caller: proof it ran in the window. + assert :ok == Task.await(unregister) + + # Non-vacuity: the member is still mid-assume. + {statem_state, _data} = :sys.get_state(member) + assert statem_state == :assuming + + # Another member won the election; its snapshot still holds the binding. + winner = :"winner_#{:erlang.unique_integer([:positive])}" + Process.register(self(), winner) + + GenServer.cast( + member, + {:apply_names_snapshot, %{held: {held, %{}, :undefined}}, {node(), winner}, [], %{}, + epoch + 2, v} + ) + + assert_receive {:"$gen_cast", {:retract_req, :held, ^held}}, + 1_000, + "the buffered unregister's pid-guarded clear never reached the new " <> + "leader — it was dropped when the buffer was flushed with no leader" + + # The register whose commit was in flight (it landed under the old epoch, so it + # failed) is retried, finds this member deposed, and becomes an intent re-sent + # to the winner — its caller gets the winner's verdict, not a `no`. + assert_receive {:"$gen_cast", {:register_req, ref, _from, :in_flight, ^in_flight, _meta}}, + 3_000, + "the in-flight registration was not re-sent to the new leader" + + assert Task.yield(register, 0) == nil, "the in-flight registration was answered early" + GenServer.cast(member, {:register_reply, ref, :yes, v}) + assert :yes == Task.await(register) + end + end + + # --------------------------------------------------------------------------- + # reject_when_degraded keys on versions (§5.6, §8) + # + # A leader is degraded exactly when its handoff reconstruction stopped short of + # the durable version key: every replica is a prefix of the committed stream, so + # one at the key holds every committed binding (the TLA+ model's + # DegradedFlagSound). Which members answered the gather does not matter. + # + # Each test demotes the member to a follower of a fake leader (a snapshot at the + # current epoch, so the durable fence still names this member), then hands it + # leadership back through a genuine assume whose gather it controls. + # --------------------------------------------------------------------------- + + describe "reject_when_degraded" do + setup %{tenant: tenant} do + reg = :"degr_#{:erlang.unique_integer([:positive])}" + {:ok, sup} = :dgen_registry.start_link(reg, tenant, %{reject_when_degraded: true}) + assert :ok == :dgen_registry.await_ready(reg, 5_000) + on_exit(fn -> stop_registry(sup) end) + + member = :dgen_registry.member_name(reg) + self_id = {node(), member} + epoch = :dgen_registry.get_epoch(reg) + v = applied_version(member) + + fake = spawn(fn -> Process.sleep(:infinity) end) + on_exit(fn -> Process.exit(fake, :kill) end) + fake_name = :"degr_fake_#{:erlang.unique_integer([:positive])}" + Process.register(fake, fake_name) + GenServer.cast(member, {:apply_names_snapshot, %{}, {node(), fake_name}, [], %{}, epoch, v}) + {:follower, _} = :sys.get_state(member) + + %{reg: reg, member: member, self_id: self_id, epoch: epoch} + end + + defp reassume(member, self_id, all_ids, epoch) do + :ok = + :gen_statem.call( + member, + {:elector_assume_and_distribute, self_id, all_ids, %{}, [], epoch} + ) + + assert eventually(fn -> match?({:leader, _}, :sys.get_state(member)) end, 5_000) + end + + test "a member that did not answer but held nothing extra does not degrade it", %{ + reg: reg, + member: member, + self_id: self_id, + epoch: epoch + } do + # A previously-synced member that never answers the gather — a scale-down, or + # one that was simply behind. The reachability rule this replaced counted it. + ghost = {:ghost@nowhere, member} + reassume(member, self_id, [self_id, ghost], epoch) + + pid = spawn(fn -> Process.sleep(:infinity) end) + on_exit(fn -> Process.exit(pid, :kill) end) + assert :yes == :dgen_registry.register_name({reg, :new_name}, pid) + end + + test "a reconstruction behind the durable version key degrades it", %{ + reg: reg, + member: member, + self_id: self_id, + epoch: epoch, + tenant: tenant + } do + # A commit this member never applied — the only holders of whatever it + # carried are gone. Every member that could answer, answers. + tuid = {"dgen_registry.", Atom.to_string(reg)} + ref = make_ref() + + {:ok, _} = + :dgen_registry_names.start_commit(tenant, tuid, self_id, %{ + owner: self(), + ref: ref, + epoch: epoch + }) + + assert_receive {:dgen_transaction, ^ref, {:committed, _}}, 5_000 + + # The gather retries toward the key until ?GATHER_TIMEOUT, then assumes degraded. + reassume(member, self_id, [self_id], epoch) + + pid = spawn(fn -> Process.sleep(:infinity) end) + on_exit(fn -> Process.exit(pid, :kill) end) + assert :no == :dgen_registry.register_name({reg, :new_name}, pid) + end + end end diff --git a/test/support/sim/README.md b/test/support/sim/README.md index 03610e1..5183a66 100644 --- a/test/support/sim/README.md +++ b/test/support/sim/README.md @@ -1,105 +1,119 @@ # dgen_registry Simulation Harness Fault-injecting simulation of the replication protocol (§4.5, §5 of -[the design doc](../../../docs/design/dgen_registry_design.md)) against the **real** -`dgen_registry` code, in a single BEAM. +[the design doc](../../../docs/design/dgen_registry_design.md)), run against the +real `dgen_registry` code in a single BEAM. -## Why this exists alongside the TLA+ model - -[`formal/`](../../../formal/README.md) proves the *protocol* correct by exploring -every interleaving of an abstraction of it. That is a strong result and a narrow -one: it says nothing about whether `dgen_registry_member.erl` actually implements -that protocol. The two failure modes are disjoint — - -| | catches a wrong protocol | catches a wrong implementation | -|---|---|---| -| TLA+ / TLC | yes, exhaustively | no | -| this harness | only if sampled | yes | - -So the properties in `DGen.Sim.Invariants` are deliberately named after the +[`formal/`](../../../formal/README.md) proves an abstraction of the *protocol* +correct by checking every interleaving. This harness checks that the +*implementation* matches the protocol, by sampling interleavings of the real +code. The properties in `DGen.Sim.Invariants` have the same names as the invariants in `DgenRegistryReplication.tla`. A violation found here should be -reproducible there, and vice versa; if it is not, one of the two is wrong about -the system. - -## How it works - -Three things make it possible: +reproducible in the model, and the other way round. If it isn't, one of the two +is wrong about the system. -1. **`keyspace`** (`dgen_registry:start_link/3`). A member's id is `{node(), - Name}`, and `Name` also fixes the durable keyspace — so one VM normally hosts - at most one member of a registry. The `keyspace` option splits identity from - keyspace, letting N differently-named members share one registry's elector - queue, leader key, and version key. They are real, complete supervision trees - committing against the real backend. +## Running -2. **`eta_net`.** Every inter-member protocol message goes through - `dgen_registry_member:cast_to_member/2` — broadcasts, register and - set_metadata replies, `replicate_sync`/`replicate_ack`, resync requests, - snapshots, retracts — and `eta_transform` rewrites the `gen_server:cast/2` - underneath it to `eta_net:cast/2`, which owns delivery. +```sh +mix dst # every :simulation-tagged sweep, on the dgen_mem backend +mix dst --seed 0 # extra arguments pass through to mix test +``` -3. **`dgen_registry:status/1`.** Each member's own belief about leadership, - epoch, sync state, and applied version — which is what the invariants compare - across members. +There are two suites: -A 3-member cluster starts in ~100ms, against ~5s for a `:peer` node, which is what -makes running hundreds of faulted operations per test practical. +- **`test/dgen_registry_eta_test.exs`** is fully deterministic. It runs the + cluster under `eta_run` on `dgen_mem`, and one seed controls the scheduler, + the clock and the network. Use it when a failure has to replay exactly. It + checks that it is deterministic: each seed runs three times and must produce + one trace, and every `eta_run:audit/1` check is fatal. The sweeps are the + lossy replication stream (`drop_p`) and node faults (`fault_p`). +- **`test/dgen_registry_sim_test.exs`** runs in real time on the real scheduler. + The fault schedule is seeded, but process scheduling and timers are not, so a + failure can be reproduced at the fault level but not bit-for-bit. -## The fault model is narrow on purpose +`mix dst` runs only the `:simulation` sweeps. The other tests in both suites run +under an ordinary `mix test`. The deterministic ones need +`DGEN_BACKEND=dgen_mem`, and `test_helper.exs` lists what each backend excludes. -Only faults real Erlang distribution can produce are injected. Erlang guarantees -that messages delivered between one ordered pair of processes arrive in send -order; it does not guarantee delivery. So, per ordered `{from, to}` pair, the -network may **deliver**, **drop**, **delay**, or **cut** (partition until healed). +A failure message includes its seed. Re-run that seed to reproduce it. A +recorded `eta` trace replays with `DGen.Sim.RegistryHarness.replay_fixture/2`. +`test/fixtures/partial_batch.eta` is the pinned example. -It never reorders two messages on the same ordered pair, because distribution -never does — this is the same model as the spec's `DropMsg` action. Injecting -reordering there would manufacture counterexamples the real system cannot produce, -and every "bug" found would be a false positive. +**Mutation suites.** Two defects can be compiled in on purpose to show the +suite catches them. Both are opt-in, and the build is test-only: -**Loss comes with a signal, and the harness must supply it.** Independent -per-message drop is still a mild over-approximation: in reality messages are lost -because a *link* failed, and a failed link delivers `nodedown`/`nodeup` to both -ends. The registry deliberately hangs recovery off those signals — -`handle_info({nodeup, _})` re-announces the member's join and re-drives unregisters -that were stashed or forwarded into the dying link (Non-goal 5). So -`DGen.Sim.Cluster.converge/2` heals through `eta_net:heal_partition/3` with a -`nodeup` signal rather than merely removing the cuts. +```sh +DGEN_MUTATION=partial_batch DGEN_BACKEND=dgen_mem mix test --only mutation +DGEN_MUTATION=quiet_resync mix test --only mutation_quiet +``` -This was learned the hard way: without it the harness reported a follower that had -optimistically deleted a row whose `unregister_req` was dropped, diverging from its -peers at the same applied_version and never recovering. That looked exactly like a -product bug and was not one — it was the harness modelling lost messages without -the disconnect that must accompany them. Worth internalising: the fault model has -to be faithful in what it *implies*, not only in what it injects. +## Invariants -Only inter-member traffic is faulted. The elector's durable membership/election -queue is untouched, so elections still make progress and a run cannot wedge on a -fault the design never claims to survive. +From `DGen.Sim.Invariants`. Each one returns `:ok` or `{:violation, details}` +and must not raise on a cluster that is partly torn down. -That scoping is stated as a **topology**, not as a predicate. -`Cluster.place_members/1` puts each member's tree on a simulated node of its own, -and `eta_net` faults a send only when both ends are faultable and their nodes -differ — so member-to-member traffic is faultable and nothing else is. Anything a -member spawns inherits its node *and its faultability*, so a transaction worker's -traffic to a peer is faulted too. +| When | Property | +|---|---| +| always | `leader_epoch_unique/1` | +| at quiescence | `same_version_same_replica/1` (`check_quiescent/1`) | +| after convergence | `unique_binding/2`, `acked_bindings_present/2` | +| end of an `eta_run` | `RegistryHarness.check_final/2`: `acked_bindings_present/2` plus a UniqueBinding check over the history of `yes` acks (the spec's `acked` set) | -The elector and the connector are `attach`ed rather than `place`d: on the node, -never on the wire. Their messages are not network traffic — they stand in for -operations against the durable store — and dropping them injects a failure the -real system cannot have. Faulting them produces `acked_bindings_present` -violations that look exactly like the replication defect this suite hunts and are -artefacts of the harness; the tell is that widening the fault model, rather than -any change to the system, produced the failure. But leaving them *unplaced* is -wrong too, and for a reason that only shows up once node faults exist: an -unplaced process is on no node at all, so it learns nothing when one fails and -survives a node kill that took its own member. `attach/2` is the distinction. +`same_version_same_replica` is checked only once the cluster is converged. +Members write their replica optimistically: `route_unregister/3` deletes a row +straight away, and a follower inserts a row when the leader's `yes` arrives. In +both cases `applied_version` doesn't advance. So two members at the same version +can legitimately differ until the batch arrives. A difference that is still +there after healing and quiescence is a bug. -## Node faults +## How it works -A cut channel is not a lost node. `partition/4`, `heal_partition/4`, `isolate/3` -and `kill_node/3` on `DGen.Sim.Cluster` inject the link-level failure instead of -the message-level one, and each carries the events the failure produces: +- **`keyspace`** (`dgen_registry:start_link/3`) separates a member's identity + from its durable keyspace. That lets N members with different names share one + registry's elector queue, leader key and version key in one VM. Each member is + a complete, real supervision tree. +- **`eta_net`**: every message between members goes through + `dgen_registry_member:cast_to_member/2`. `eta_transform` rewrites the + `gen_server:cast/2` under it to `eta_net:cast/2`, so `eta_net` controls + delivery. A module needs `-include("dgen_eta.hrl")` if it *runs inside* a + scheduled process, even if it doesn't own one. The API modules `dgen` and + `dgen_registry` are examples. +- **`dgen_registry:status/1`** reports each member's own view of leader, epoch, + sync state and applied version. The invariants compare these across members. +- **`DGen.Sim.Cluster`** starts, faults and heals the cluster. + **`DGen.Sim.RegistryHarness`** adapts it to `eta_harness`: it starts the + cluster, picks which processes to schedule, runs the workload and checks the + result. Its moduledoc lists the real-time dependencies that had to be removed + before runs became deterministic. + +## Fault model + +Only faults that Erlang distribution can actually produce are injected. For each +ordered `{from, to}` pair, the network can **deliver**, **drop**, **delay** or +**cut** a message. It never reorders messages within a pair, which matches the +spec's `DropMsg`. Injecting reordering would only produce false positives. + +**A lost message comes with a link signal.** In real distribution, messages are +lost because a link failed, and the failure sends `nodedown`/`nodeup` to both +ends. The registry relies on those signals: `handle_info({nodeup, _})` re-joins +and re-drives stashed unregisters. So `Cluster.converge/2` heals through +`eta_net:heal_partition/3` with a `nodeup` signal. A fault model that leaves out +these signals produces failures that look like product bugs but aren't. + +**Topology decides what is faultable.** `Cluster.place_members/1` puts each +member's tree on its own simulated node. `eta_net` faults a send only when both +ends are faultable and on different nodes. Anything a member spawns inherits its +node and whether it can be faulted. The elector and connector are `attach`ed: +they are on the node but never on the wire, because their messages stand in for +operations on the durable store. If they were faultable, the run would produce +failures the real system can't have. If they weren't on any node, they would +survive a node kill that took down their own member. + +### Node faults + +`partition/4`, `heal_partition/4`, `isolate/3` and `kill_node/3` on +`DGen.Sim.Cluster` inject link-level failures, along with the events each one +causes: | | messages | `{nodedown, Peer}` | `noconnection` DOWNs | processes die | |---|---|---|---|---| @@ -109,351 +123,58 @@ the message-level one, and each carries the events the failure produces: | `kill_node/3` | in flight cancelled | survivors | across the node | the whole tree | | `crash/2` | — | — | — | the whole tree | -The signal is **derived per side**: A is told `{nodedown, member_2}` and B is told -`{nodedown, member_1}`, because that is what a partition actually says. One -undifferentiated term would tell both sides the same thing, which is never what -happened. `%{learns: :a}` gives the one-sided form, which is what two -independently timing-out ends produce. - -`kill_node/3` and `crash/2` are both "the tree dies", and the difference is the -whole reason node faults exist. Under `crash/2` the peers find out because the -processes are gone, and their monitors report `killed`. Under `kill_node/3` the -peers' monitors are retired *before* anything dies and report `noconnection` — -the asymmetry real distribution has, where a remote watcher and a local one see -different reasons for the same death — and the survivors are then told the node -went. The supervisor is frozen before the kill and killed after it, so -`one_for_all` cannot restart the node that just died — under `eta_run` it is -already frozen, being one of the processes the scheduler owns. - -### What is still not simulated: `net_kernel` - -`eta_net` delivers the *events* a node failure produces and nothing else. A -simulated node is a name in a table: `nodes()` does not list it, and every member -id's node component is the one real `node()`, so `dgen_utils:node_reachable/1` -answers `true` for all of them. - -The consequence is worth stating precisely, because it decides which reactions -these tests cover: - -- **Covered.** `dgen_registry_member`'s reactions, which do not read the node - name: `handle_info({nodeup, _})`'s rejoin and unregister re-drive, and the - peer-monitor `DOWN` that drives `{member_down}` into the elector. -- **Not covered.** `dgen_registry_connector`'s reachability-keyed backstops — the - `{nodedown, Node}` reap and the leader-liveness probe. Both filter member ids - by node, and no member id names a simulated node, so both are no-ops here. - They belong in `dgen_registry_cluster_test.exs` with real peers. - -### Where the `noconnection` DOWNs come from, and the race that decides it - -`dgen_registry_member`'s failure detection is an `erlang:monitor` on each peer, -which `eta_transform` points at `eta_net:monitor/2`. That monitor is *simulated* -— the only kind a partition can sever — when both ends are already on different -simulated nodes at the moment it is created. Peers are monitored while the cluster -forms, so placement has to happen before that, which is `Cluster.start/3`'s -`:simulate_peer_monitors`. - -Placing each tree as it starts is necessary and **not sufficient**, and the gap is -instructive: a member learns of its peers when the elector distributes the member -set back, which is an ordinary message arriving some unbounded time later. So which -peer monitors ended up simulated came down to real-time luck, and a run stopped -being a function of its seed — about a fifth of seeds. `Cluster.start/3` therefore -*holds* each member (`sys:suspend/1`) from the moment its tree starts until every -tree is placed, so no monitor anywhere can be created before the topology is -complete. Ordering the two, rather than hoping one wins, is what made it -deterministic. - -It stays off for the real-clock suite in this directory, deliberately. A simulated -monitor learns of an ordinary exit from `eta_sched`'s exit trace: under `eta_run` it -fires on a crash exactly as a real one would, and with no scheduler it never fires -at all. Turning it on here would silently disable the `crash/2` detection the tests -above rely on. - -`eta_net:stats/0` counts `signalled` and `noconnection` for exactly this reason. -They are the non-vacuity guard for a node fault the way `dropped` is for a lossy -policy: a run that partitioned a node nothing was on, or severed no monitor, -exercised no recovery and reports the same `ok`. Both sweeps in -`dgen_registry_eta_test.exs` assert on them. - -## The member is a gen_statem - -`dgen_registry_member` runs as a `gen_statem` (states: `searching`, `assuming`, -`leader`, `follower` — see its moduledoc), which under simulation engages eta's -statem support: starts gate through `eta_sched:statem_start_link/3`, -`gen_statem:call/cast/reply` route through `eta_net`, `-eta_observe` publishes -on statem callback returns, and any `state_timeout` would be virtualized by -`eta_statem` (none is currently armed; event and generic timeouts raise under a -running clock by design, so the member's periodic timers stay `arm/2` -self-sends). The port was gated on this suite at every phase — the behaviour -swap, the derived states, and the enter-call lifecycle each landed with the -full battery green and the pinned mutation fixture replaying unchanged. - -## Determinism: what is and is not guaranteed - -The **fault schedule** is seeded and reproducible — for a given seed, the same -decision is made about the same message on the same channel. Process scheduling, -commit timing, and timer expiry are *not* controlled, so a failing run is -reproducible at the fault level rather than bit-for-bit. In practice re-running a -seed usually reproduces a failure, and `eta_net:stats/0` says what the network -did; it is not a guarantee. - -**Full determinism is available, and lives elsewhere.** -`test/dgen_registry_eta_test.exs` drives the same cluster through `eta_run`, with -the scheduler, the clock and the network all seeded from one value, against the -in-memory `dgen_mem` backend. That is the suite to reach for when a failure has to -replay exactly — under every fault model it offers, message and node alike. It -asserts it, too: `"every seed reproduces its own schedule"` runs each seed three -times and requires one trace, and `eta_run:audit/1` is fatal on every check. - -### What node faults cost to get there - -A cut is absolute where `drop_p` is scoped, so a partition drops every cross-node -message and drives recovery paths — `peer_joined`, `replicate_sync`, -`apply_names_snapshot`, the §5.7 handoff gather — that a scoped policy never -reaches. Switching node faults on therefore found seven real-time dependencies -that message loss alone never touched. About a fifth of seeds produced more than -one schedule; all five are fixed and it is now nil. - -None showed up in `audit/1`, and the reason is worth internalising: the audit -catches a process **running** outside the schedule, and every one of these was a -process **blocking** on something outside it — `code_server`, an unscheduled -supervisor, OTP's own `gen:do_call/4`. What found them was diffing two runs of one -seed at the step where their runnable sets first disagreed, and reading the stack -of the process that differed. `RegistryHarness`'s moduledoc lists all seven; -findings 3, 4 and 5 below are the ones that are also bugs off the simulator. - -And measure under load. Six of the seven were visible on an idle machine; the last -— an OTP 28 supervisor's `hibernate_after` timer, which is real — only appeared -with several suites running at once, because what it turned on was how much real -time a run took. `mix dst` looked clean serially and failed one run in three when -six ran together. - -Two things are *not* leaks and are allowed. A node fault leaves stray timers — a -killed node's periodic timers outlive it, and a call whose callee is behind a cut -waits out an hour of virtual `register_timeout` the run never reaches. Both are -the fault behaving correctly, and `stray_timers` is allowed on that sweep alone. - -This suite keeps its own value alongside it: it runs the same invariants against -the same code without suspending anything, so it exercises the real BEAM scheduler -rather than a serialization of it. It was also enough to find the three issues -below. - -## Running - -```sh -mix dst -``` - -`mix dst` runs only the `:simulation`-tagged sweeps (the multi-seed soaks in -both suites). The rest of the two suites — the targeted regressions here and -the untagged determinism/plumbing tests in `dgen_registry_eta_test.exs` — ride -an ordinary `mix test` (the deterministic ones need `DGEN_BACKEND=dgen_mem`; -see `test_helper.exs` for what each backend excludes). The `:mutation` suite is -opt-in and needs the defect compiled in — see `test_helper.exs`. +Each side gets its own signal: A is told `{nodedown, member_2}` and B is told +`{nodedown, member_1}`. `%{learns: :a}` makes the signal one-sided. With +`kill_node/3`, peer monitors report `noconnection`. With `crash/2`, they report +`killed`. The supervisor is frozen before the kill so `one_for_all` can't +restart the node. + +Peer monitors are simulated (and so can be severed) only if both ends are already +on different simulated nodes when the monitor is created. `Cluster.start/3`'s +`:simulate_peer_monitors` holds each member with `sys:suspend/1` until every +tree has been placed. Without that, which monitors were simulated would depend +on real-time luck. It stays off in the real-time suite, because a simulated +monitor never fires on a crash when no scheduler is running. + +`eta_net:stats/0` counts `dropped`, `signalled` and `noconnection`. The sweeps +assert on these counts so that a run that injected nothing can't count as a +pass. `stray_timers` is allowed on the node-fault sweep only: a killed node's +timers outlive it, and a call blocked behind a cut waits on a virtual timeout +that the run never reaches. + +**Not simulated: `net_kernel`.** A simulated node is only a name. `nodes()` +doesn't list it, and `dgen_utils:node_reachable/1` returns `true` for every +member. So `dgen_registry_member`'s signal-driven reactions are covered, but +`dgen_registry_connector`'s node-keyed backstops (the `{nodedown, Node}` reap +and the leader-liveness probe) are not. Those belong in +`dgen_registry_cluster_test.exs`, which uses real peers. ## Findings -All were found by this harness and are **fixed**. Each is kept here because -the reasoning is the useful part — the mechanism, why it was invisible to the -existing tests, and what now pins it down. - -### 1. A partially-delivered group commit diverged replicas permanently — FIXED - -A group commit used to broadcast **one message per changed name**, every one stamped -with the same `{PrevVersion, Version}`. `apply_bcast/6` applied the batch's first -message via `PrevV =:= Applied` and bumped `applied_version` to `V`; each later -message then matched a `V =:= Applied` clause and applied at `V`. - -So `applied_version` advanced on a batch's **first** message, not its last. A member -receiving a strict subset of a batch reported version `V` while holding only part of -it. Two consequences, the second being the serious one: - -1. It broke the "version tie ⇒ identical content" property that §5.7's freshest-wins - reconstruction rests on. `gather_maps` picks by version alone, so a handoff could - pick the member that was *missing* rows — dropping committed, acked bindings and - fanning the deficient snapshot out to everyone. - -2. **It was unrecoverable.** Gap detection compares only `{PrevV, V}` and there was no - per-batch completeness marker, so a member that lost part of a batch it had already - advanced onto presented no discontinuity for any later broadcast to catch. Every - subsequent batch applied cleanly over the hole. The soak saw all three members - agreeing on a leader, all at the same `applied_version`, one holding fewer names, - staying that way indefinitely under continued traffic. - -Reachable in production via a mid-batch disconnect: the link drops after *k* of *N* -messages. `commit_batch_size` defaults to 5000 and the design explicitly expects large -batches ("a departing node's flood of `DOWN`s"), so multi-message batches were the -normal case. - -**Fix:** a batch ships as one `{names_batch, Ops, Epoch, PrevV, Version, LeaderId}` -message (`broadcast_batch/5`). Delivered whole or not at all, a lost batch always -leaves a version discontinuity, which the existing resync already repairs. The -`V =:= Applied` clause collapses into the duplicate guard, so `apply_bcast/6` now -matches the spec's `RecvBcastApply` exactly. - -**This also closed the model's abstraction gap.** `RegisterForward` commits exactly one -name per version, so in the spec a version was always a single broadcast and that -clause was only ever reachable for a duplicate — the model could not express the bug. -It now describes the shipped protocol rather than an idealisation of it. - -Regression: `"dropping a batch leaves a detectable gap that resyncs, not a silent -hole"`. - -### 2. Gap detection was traffic-triggered, so a quiescent cluster stayed diverged — FIXED - -`request_resync/2` had exactly two callers — `apply_bcast/6` (a later batch reveals the -gap) and `handle_register_reply/4` — and `resync_timeout` only cleared the -once-per-window guard: - -```erlang -%% The resync request we sent went unanswered (dropped cast, deposed target) — -%% clear the guard so the next gap-observing broadcast requests again. -handle_info(resync_timeout, State) -> - {noreply, State#state{resync_timer = undefined}}; -``` - -So a follower that lost the **tail** of the stream — the last batch before writes -stopped — had nothing left to reveal the gap and held a stale replica for as long as -the cluster stayed quiescent. A lost `resync_req` (or the snapshot answering it) had -the same shape. Safety was unaffected (the replica was still a valid prefix), but the -divergence was unbounded in time rather than "a short replication interval" (Non-goal -4), and for a missed *unregister* it sat badly with Non-goal 5's "eventual propagation -is actively driven at every hop". - -**Fix:** the leader broadcasts an empty batch stamped at its current applied version -every `?REPLICA_HEARTBEAT_INTERVAL`. It needs no new handling — a caught-up follower -matches `PrevV =:= applied_version` and applies nothing, a behind follower matches -neither that nor `V =< Applied` and takes the existing gap branch. Cost is one small -cast per follower per interval, independent of the name count. - -Regression: `"a follower that loses the tail of the stream still converges when -quiescent"`. - -### 3. A `try ... of` left the mesh nudge's send unprotected — FIXED - -Found by node-fault injection, and only by it. `spawn_mesh_fetch/3`'s helper calls -`nudge_durable_epoch/2`, which read the committed epoch and sent it to the member: - -```erlang -try dgen_server:priority_call(Elector, get_epoch) of - DurableEpoch when is_integer(DurableEpoch) -> - MemberName ! {durable_epoch, DurableEpoch}; %% not protected - _ -> ok -catch - _:_ -> ok -end, -``` - -An `of` body is **outside** the protected expression, so the `catch _:_ -> ok` -that every other failure on this best-effort path routes through did not cover the -send. The member is addressed by *registered name*, and sending to a name with no -process raises `badarg` — so a nudge arriving in the window where the member is -gone crashed the helper it ran in. - -Reachable in production wherever that window opens: the tree restarting under -`one_for_all`, or the node lost. It was invisible to every existing test because -nothing closed a member while a mesh fetch was in flight; a `kill_node/3` does it -on roughly one sweep in ten. - -**Fix:** `try ... catch` around a `case`, so the send is inside the protected -expression. Behaviour is otherwise unchanged — the epoch is re-read on the next -`?MESH_INTERVAL` either way. - -### 4. Two client-facing modules were not built with the transform — FIXED - -`dgen` and `dgen_registry` had no `-include("dgen_eta.hrl")`. They read as API and -supervisor code, so it looked like they had nothing to instrument — but *where they -run* is the caller's process, which under simulation is a scheduled one. Every -`gen_server:call/2,3` in them therefore went into OTP's `gen:do_call/4`, which no -transform reaches, taking a wall-clock `receive ... after` and a real -`erlang:monitor` with it. Both put a scheduled process's progress outside the -schedule. - -Not a production bug — it costs nothing at run time — but a real hole in the -simulation, and the rule it teaches generalises: **a module needs the transform if -it runs inside a scheduled process, not if it owns one.** - -**Fix:** the include, in both. Neither uses anything on `eta_transform`'s -`?NET_UNSUPPORTED` list, so the rewrite is total. - -### 5. `telemetry_available/0` called `code_server` from inside a member — FIXED - -Finding 4 of the previous round cached the telemetry lookup in `persistent_term` -to stop `code:ensure_loaded/1` making a `code_server` round trip per event. The -cache is per VM, so exactly one call still had to happen — and it happened lazily, -inside whichever member emitted the first event. That is a scheduled process making -a synchronous call into one the scheduler does not own. - -It is invisible to `eta_run:audit/1`, which is the part worth remembering: -`code:ensure_loaded/1` on a *missing* module loads nothing, so the "no module -loaded mid-run" check stays clean while the leak is wide open. It was found by -reading the stack of a member that was parked in `code_server:call/1` at the step -where two runs of one seed first disagreed. - -**Fix:** resolve it in `dgen_registry_member:init/1`, while the tree is starting — -which under `eta_run` is before a scheduler exists at all. - -### 6. `set_metadata/2` did not honour a configurable timeout — FIXED - -`register_name/2,3` bounded its wait with `register_timeout` (§8), but `set_metadata/2` -and `unregister_name/1` called `gen_server:call/2` with its hidden 5s default, so -tuning the knob silently governed only one of the three writes on the pipeline. Under -injected loss this was the dominant cost of a soak run. - -**Fix:** all three leader-routed writes resolve `register_timeout`. The default is -unchanged, so only callers who had already set it see a difference. What a timeout -*means* still differs per call, deliberately: registration exits, unregister answers -`ok` (the removal is stashed and re-driven), `set_metadata` answers -`{error, no_leader}`. - -### 7. One unregister freed a name twice, double-issuing it — FIXED - -Found by the end-of-run **ack-history fold** (`check_final/2` on the harness), on -its very first sweep — and by construction invisible to every replica-based check, -because all members apply the same wrong batch and agree perfectly. - -The mechanism, three steps on the leader: - -1. A register for `B` is enqueued while a commit is in flight (holder `A` in ETS), - so it parks in the pending queue unplanned. -2. An unregister arrives. The leader captures `ReleasedPid = A` and does its - **optimistic `row_delete`** — the name is free in ETS *now* — then enqueues - `{remove, Name, A, …}` behind the parked register. -3. The batch plans in FIFO order. `B`'s add consults the (optimistically emptied) - table, answers **`yes` while `A`'s acked registration still stands**; the - remove then cleared the name **unconditionally** — deleting `B`, whom it never - targeted. A later register `C` gets `yes` too. - -Three `yes` acks for one name with a single unregister between them has no legal -serialization (Guarantee 1), and the destroyed binding was an acked one -(Guarantee 4, no fault anywhere). Reachable in production with no faults at all: -a register and an unregister for one name arriving while a commit is in flight is -ordinary concurrency. Every existing check missed it structurally — the sequential -real-clock workload never parks a register behind an unregister, and replica -comparisons see a cluster in perfect agreement about the wrong history. - -**Fix:** `plan_op/3`'s remove clause guards the clear against the *current* -holder (batch overlay first, ETS fallback): a removal whose captured target is a -pid clears only while the holder is that pid or nobody (nobody being the -optimistic delete's own footprint). A different pid means the target binding is -already gone — the unregister answers its idempotent `ok` and the new holder -survives. A capture of `undefined` keeps its meaning: the unregister serializes -after whatever bound the name, a legal linearization of unregister-by-name. - -Regression: the lossy eta sweep at seed 5 reproduces the exact interleaving -deterministically, and `check_final/2` now folds the ack history on every -fault-free-ending run of every sweep. - -## A note on invariants that are not invariants - -`same_version_same_replica` was first written as a continuously-checked property and -had to be moved to converged-only — not because of a bug, but because members write -their replica **optimistically, outside the replicated stream**, without advancing -`applied_version`: `route_unregister/3` deletes the row the moment the caller asks, and -a follower's `handle_register_reply/4` inserts it when the leader's `yes` arrives. -Both leave a member differing from its peers at the same version until the batch comes -back around. - -That mid-flight difference is ordinary speculation. What must not survive is a -difference still present after the network is healed and the cluster has quiesced, -which is exactly the shape finding 1 had. An invariant that is only *usually* true -manufactures false findings and trains you to ignore it. +Defects this harness found. All are fixed. Tests and the spec refer to them by +number. + +1. **Partially delivered group commit.** A batch used to go out as one message + per name, all with the same version. A member that received only some of them + reported the full version, and gap detection couldn't see the hole. The fix + ships a batch as a single `{names_batch, …}` message, which also matches the + spec's one broadcast per version. Regression: `"dropping a batch leaves a + detectable gap that resyncs, not a silent hole"`. +2. **Gap detection needed traffic.** A follower that lost the tail of the stream + stayed behind for as long as the cluster was idle. The fix is the leader's + periodic empty-batch heartbeat. Regression: `"a follower that loses the tail + of the stream still converges when quiescent"`. +3. **Unprotected send in the mesh nudge.** A send inside `try ... of` wasn't + covered by the `catch`, so sending to a member that had exited crashed the + helper. +4. **`dgen` and `dgen_registry` weren't built with the eta transform.** Their + `gen_server:call`s escaped the scheduler. This was a simulation-only gap. +5. **`telemetry_available/0` called `code_server` from inside a member.** It is + now resolved in `init/1`. +6. **`set_metadata/2` and `unregister_name/1` ignored `register_timeout`.** +7. **One unregister freed a name twice.** An optimistic delete plus an + unconditional remove let a parked register get `yes` while an acked binding + still stood, and then deleted it. `plan_op/3` now clears a name only if the + current holder is the pid the unregister targeted. Found by the ack-history + check in `check_final/2`. Replica comparisons can't catch it, because every + member agrees on the same wrong history. diff --git a/test/support/sim/cluster.ex b/test/support/sim/cluster.ex index 3ff3c80..d9c207a 100644 --- a/test/support/sim/cluster.ex +++ b/test/support/sim/cluster.ex @@ -362,8 +362,26 @@ defmodule DGen.Sim.Cluster do `eta_observe` reads the fields the member publishes on every callback return, so this works on a frozen system. That is what lets `Invariants` be used unchanged from under `eta_sched`. See `eta_observe`. + + The member publishes its stream position as one opaque `replica` field (see + `dgen_registry_replica`); it is flattened here into the `epoch` and + `applied_version` keys `dgen_registry:status/1` answers with, so the two views + stay comparable field for field. """ - def status(name), do: :eta_observe.read(name) + def status(name) do + case :eta_observe.read(name) do + %{replica: replica} = published -> + published + |> Map.delete(:replica) + |> Map.merge(%{ + epoch: :dgen_registry_replica.epoch(replica), + applied_version: :dgen_registry_replica.version(replica) + }) + + other -> + other + end + end @doc "The member id every live member's elector currently agrees leads, or nil if they disagree." def leader(%__MODULE__{} = c) do