Skip to content

fix(edgebound): restore edge RPC forwarding after reconnect - #109

Merged
singchia merged 9 commits into
mainfrom
fix/edge-reconnect-stale-session
Sep 24, 2026
Merged

singchia merged 9 commits into
mainfrom
fix/edge-reconnect-stale-session

Conversation

@youzi-1122

@youzi-1122 youzi-1122 commented Sep 23, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

The incident and local edge 72 logs show repeated edge online, old end exists / edge online same time, old end exists together with no such rpc: register_edge. In v1.2.4, overlapping reconnects share a SyncHub key: an earlier waiter can be rejected with please connect later, while the latest waiter remains blocked on the old session's offline callback. RPC forwarding is installed only after this wait succeeds.

The logs establish that reconnect loop; they do not prove the report's inference that each RPC was delivered to the dead connection.

Changes

  • Replace stale sessions immediately, without waiting for old callbacks or connection closure. Close the previous end asynchronously with panic containment.
  • Bind offline, stream and RPC-registration callbacks to connection instances. Late callbacks cannot delete the replacement, even when the remote address is reused. Preserve early streams/registrations, reject sessions already closed before installation, and clear old RPC inventory.
  • Order active-session/repository updates and edge-count publication under the existing lock, upsert SQLite edge rows, and remove closed sessions from routing even if repository cleanup fails. Keep network notifications outside the cache lock.
  • Install forwarding before notifying the online informer; release the manager lock before administrative connection closure.
  • Match the image runtime to the builder's glibc with Debian bookworm. Set VERSION to 1.2.6. This PR does not publish a release tag or registry image.

Verification

  • Address-reuse and stale RPC-inventory regressions failed before their fixes.
  • Focused edgebound and SQLite regression tests passed 20 runs with -race, covering blocked close, successive replacements, late callbacks, address reuse, early offline, streams, RPC inventory and concurrent edge-count updates. The count regression first reproduced an incorrect count of 1 with 2 active edges, then passed after restoring update order.
  • Real TCP reconnect, bidirectional RPC and stream E2E tests passed 3 runs. The reconnect test replaces the same edge ID five times per run and verifies both call directions reach the current session.
  • go vet on the changed packages and E2E package, plus make frontier, passed.
  • Controlled A/B fault injection: v1.2.4 failed the single-reconnect case 3/3 times and reproduced both incident warnings on overlapping reconnects; the earlier fixed revision 4651964 passed 20/20 race-enabled runs. Permanent regressions cover these cases on the current head. The old image was not deployed to Kubernetes.
  • Local ARM64 image v1.2.6-rc.6 (source b7dce8a, before the count-order follow-up) was built and deployed. After Frontier restarted, the Kubernetes controller automatically re-registered and edges 71/72/73 were online with fresh timestamps. An earlier image revision also built and started on AMD64 under emulation.

Existing test limitations

  • Full TCP E2E race checking exposes an unsynchronized Hijack handler read/write in dependency geminio v1.3.0-rc.2. The same race reproduces with the original TestEdgeCallService on unchanged main 01f158c; this PR does not fix it.
  • The full edgebound suite contains existing tests passing a nil exchange to a constructor that dereferences it.
  • GitHub's Go workflow checks the build; it does not run the tests above.

@vercel

vercel Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
frontier Ready Ready Preview Sep 23, 2026 6:14am UTC

@singchia singchia left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

按维护者确认合并。此次会话隔离和非阻塞重连修复方向合理。仍需后续跟踪上下线通知乱序,以及新增回归测试接入 CI;当前 CI 仅验证构建,本次批准不表示已完成生产验收。

@singchia
singchia merged commit dda09a1 into main Sep 24, 2026
3 checks passed

This branch was successfully deployed

1 active deployment
Preview — d9705e3c Deployed Sep 23, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants