Skip to content

fix: bound stream trimming and recover persistence workers - #33

Open
stanley2058 wants to merge 1 commit into
fix/dev-3194-transport-persistence-correctnessfrom
fix/dev-3195-safe-trimming-worker-recovery
Open

stanley2058 wants to merge 1 commit into
fix/dev-3194-transport-persistence-correctnessfrom
fix/dev-3195-safe-trimming-worker-recovery

Conversation

@stanley2058

@stanley2058 stanley2058 commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Note

🤖 GPT-6-Astra responding on behalf of Stanley

Builds on #32.

This PR fixes stream trimming and persistence worker recovery:

  • Server-side saves reload an independent document snapshot with its Redis stream position. Trimming removes only entries covered by that save and older than the minimum message lifetime. Queue workers use the same bound instead of the task ID or latest stream head. Calls without a known persisted position retain the stream.
  • Worker requests and success/error replies carry unique request IDs. Delayed or mismatched replies cannot complete another save. Worker errors, exits, and health-check replacement release pending requests, and failed saves retry without waiting for another client update. Namespace cleanup waits for queued saves and retries.
  • Queue workers also persist pending Yjs structs and delete sets before trimming their entries. Temporary documents are released after failed saves.

Server-side persistence flow:

flowchart TD
    A[Acquire persistence lease] --> B{Lease acquired?}
    B -->|No| C[Leave updates for the current owner]
    B -->|Yes| D[Load snapshot and stream position]
    D --> E{Worker ready?}
    E -->|No| F[Save snapshot directly]
    E -->|Yes| G[Send snapshot with requestId]
    G --> H[Wait for a matching worker reply]
    H -.->|Ignore stale or mismatched replies| H
    F -->|Success| I[Trim only saved entries older than the retention cutoff]
    H -->|Success| I
    F -->|Failure| J[Keep stream entries and schedule retry]
    H -->|Failure, exit, or replacement| J
    J --> A
Loading

Adds 11 regression tests covering delayed saves, updates in the same millisecond, retention, reclaimed tasks, out-of-order updates, failed saves, stale replies, cleanup, worker replacement, and a real worker thread. CI runs for stacked PRs as well as PRs targeting main.

Validated with all 26 regression tests on Node 18, npm run lint (Standard and TypeScript), and npm run dist.

The storage adapter interface is unchanged. Server-side saves add a Redis and storage read per save. Custom persistence workers must echo requestId with the room; server and worker should use matching package versions. Storage must reject failed saves. Storage-enforced commit ordering and Redis expiration/eviction remain separate concerns.

@stanley2058
stanley2058 added this pull request to stack #34 October 5, 2026 07:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant