Skip to content

feat!: add cleanIncoming and change incoming packets key - #127

Merged
robertsLando merged 2 commits into
masterfrom
feat/clean-incoming
Sep 16, 2026
Merged

robertsLando merged 2 commits into
masterfrom
feat/clean-incoming

Conversation

@robertsLando

Copy link
Copy Markdown
Member

What this does

Adds cleanIncoming(client) to the redis persistence. aedes-persistence v11 makes it a required interface method: aedes calls it on a CONNECT with cleanSession set, alongside cleanSubscriptions, so the QoS 2 dedup table does not survive a discarded session.

Without it (GHSA-p8r9-qf8w-p73r), the first colliding messageId published by a reconnected client is matched against the stale entry, acknowledged, and never delivered — a silent message loss, and a violation of [MQTT-3.1.2-6].

persistence.js wraps AsyncRedisPersistence in CallBackPersistence, which already delegates both the callback form cleanIncoming(client, cb) → cb(err, client) and the promise form. So the only thing needed here is the async method.

Redis commands, and why this is safe at scale

Incoming packets used to be one string key per message, incoming:<clientId>:<messageId>. There was no per-client index, so deleting a client's packets in one shot was not possible — only KEYS/SCAN over the keyspace, which blocks the server or walks every key on every clean-session CONNECT.

They now live in one hash per client, incoming:<clientId>, with messageId as the field:

operation before after
incomingStorePacket SET incoming:<c>:<mid> HSET incoming:<c> <mid>
incomingGetPacket GET incoming:<c>:<mid> HGET incoming:<c> <mid>
incomingDelPacket DEL incoming:<c>:<mid> HDEL incoming:<c> <mid>
cleanIncoming — DEL incoming:<c>
  • One command per call, same as before on the hot QoS 2 path — no write amplification, no secondary index to keep in sync.
  • cleanIncoming is a single DEL of a single key. No KEYS, no SCAN, no loop over incomingDelPacket. Cost is proportional to that one client's in-flight packets, not to the keyspace.
  • Cluster-safe. Every incoming operation is now single-key, so it always resolves to one slot. The old layout would have needed a CROSSSLOT-invalid variadic DEL (or one DEL per messageId) to clean by client.
  • Fewer keys overall — one per client with in-flight QoS 2 traffic instead of one per in-flight message — and small hashes are listpack-encoded, so it is also cheaper on memory.

The alternatives considered and rejected: SCAN MATCH incoming:<c>:* (walks the whole keyspace per clean-session CONNECT, and fans out per node on a cluster), and keeping string keys plus a per-client SET index of messageIds (doubles the writes on the hot path, adds an index that can drift, and still needs a cross-slot delete).

Schema change

This is a breaking key-schema change, so it lands as v12. Documented in UPGRADE.md alongside the existing v9/v10 notes, and noted in the README next to the "Changes in v4.x" entry.

Incoming packets are in-flight state only, so nothing durable is lost across the upgrade. The old string keys are unreachable afterwards: start from a clean database, or run the new migrations.js from11to12, which SCANs for incoming:*, deletes only the string-typed (legacy) ones, and leaves the new hashes alone. It follows the same (db, cb) shape as the existing from9to10.

Requires aedes-persistence 11.0.0

Depends on moscajs/aedes-persistence#111 (fix/clean-incoming-on-session-end), which is not published yet. package.json is set to the correct final state, "aedes-persistence": "^11.0.0".

CI on this PR will be red until aedes-persistence 11.0.0 is on npm — npm install cannot resolve ^11.0.0. That is expected; no action needed here beyond re-running CI after the release.

How this was verified locally

Installed the unreleased branch without saving it:

npm i --no-save github:moscajs/aedes-persistence#fix/clean-incoming-on-session-end

and ran the full suite against Redis 6.2.5 and the cluster compose service, on Node 22:

suite result
test/abs.js (abstract conformance, standalone) 51/51 pass, 0 fail
test/cluster.js (abstract conformance, cluster) 51/51 pass, 0 fail
test/own.js 8/8 pass, 0 fail
npm run lint clean

The three cleanIncoming-relevant conformance cases all pass in both the standalone and cluster runs:

ok 44 - add incoming packet, get it, and clear with messageId
ok 45 - clean incoming packets
ok 46 - clean incoming packets with no stored packets

clean incoming packets covers the "other clients must not be touched" assertion, and clean incoming packets with no stored packets covers the no-op case (DEL on a missing key returns 0, it does not error).

The from11to12 migration was exercised against a live Redis: legacy string keys removed, new per-client hashes and unrelated keys (retained:*) untouched, idempotent on a second run and a no-op on an empty database.

aedes-persistence v11 makes cleanIncoming a required method: aedes calls it
on a clean-session CONNECT so the QoS 2 dedup table does not survive the
session ([MQTT-3.1.2-6]).

Incoming packets moved from one string key per message
(incoming:<clientId>:<messageId>) to one hash per client
(incoming:<clientId>), keyed by messageId. cleanIncoming is then a single
DEL of that key: no KEYS, no SCAN, and single-slot on a cluster.
incomingStorePacket/GetPacket/DelPacket issue one command each, as before.

Adds migrations.js from11to12 to drop the now unreachable legacy keys.
Review follow-ups on the v12 incoming key schema change.

migrations.js:
- Scan every master node when handed a Redis.Cluster. SCAN is a keyless
  command, so ioredis routed it to one arbitrary node: a cluster-wide scan
  saw 102 of 300 legacy keys, then reported success. The migration now
  fans out over `db.nodes('master')`, matching clusterMatchRetained.
- Honour the connection's keyPrefix. ioredis does not prefix a SCAN MATCH
  pattern but does prefix key arguments, so a prefixed connection matched
  nothing and would have re-prefixed the keys it got back.
- Match `incoming:*:*` instead of `incoming:*`. clientIds are URI-encoded,
  so only the old keys carry a second colon; live v12 hashes are no longer
  read at all. TYPE stays as a guard on what does match, and TYPE and DEL
  are now pipelined per scan page instead of two round trips per key.
- Call back with the number of keys removed, so an operator can tell a
  real run from one pointed at the wrong database.
- Call back outside the try, so a throwing callback cannot fire it twice.
- Use DEL, not UNLINK: UNLINK needs Redis 4.0 and the cluster fixture in
  docker/ is 3.2.8, where the migration failed outright.

test/migrations.js: new. Covers legacy keys removed, v12 hashes and
unrelated keys left alone, idempotency, an empty database, keyPrefix, and
full cluster coverage. All four cases fail against the previous version.

UPGRADE.md: state the preconditions. Running the migration while a v11
broker still serves destroys live dedup state, which causes the duplicate
delivery the schema change exists to prevent. Also documents the rolling
upgrade, how to invoke it, and the downgrade leak.
@robertsLando robertsLando changed the title feat: add cleanIncoming (GHSA-p8r9-qf8w-p73r) feat: add cleanIncoming Sep 16, 2026
@robertsLando robertsLando changed the title feat: add cleanIncoming feat!: add cleanIncoming and change incoming packets key Sep 16, 2026
@robertsLando
robertsLando merged commit 3e902a8 into master Sep 16, 2026
7 of 9 checks passed
@robertsLando
robertsLando deleted the feat/clean-incoming branch September 16, 2026 08:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant