Skip to content

34 35 fix minows rpc lifecycle - #41

Merged
JonKuqi merged 3 commits into
mainfrom
34-35-fix-minows-rpc-lifecycle
Aug 27, 2026
Merged

JonKuqi merged 3 commits into
mainfrom
34-35-fix-minows-rpc-lifecycle

Conversation

@JonKuqi

@JonKuqi JonKuqi commented Aug 25, 2026 •

Copy link
Copy Markdown
Collaborator
  • Fixed MinoWS RPC stream lifecycle leaks by closing completed unary and streaming RPC streams.
  • Reworked cancellation handling so blocked unary calls are still aborted while completed calls no longer leave cleanup goroutines behind.
  • Made normal streaming shutdown graceful and ensured partially opened resources are cleaned up on setup failures.

closes #34
closes #35

Problem 1

The problem was revealed after a load test on dvoting creating forms, causing the streams to leak and blocked.

Every unary Call opened a new libp2p stream in unicast. Neither the client nor server closed that stream after the response completed:

client unicast -> open stream -> send -> receive reply -> return
server handleCall -> receive -> process -> reply -> return

openStream instead created a goroutine that waited for the caller's context:

dela/mino/minows/rpc.go

Lines 252 to 258 in 8881acd

go func() {
<-ctx.Done()
err := stream.Reset()
if err != nil {
r.logger.Error().Err(err).Msg("could not reset stream")
}
}()

D-Voting reused long-lived contexts. Completed streams and cancellation goroutines therefore accumulated until libp2p rejected new streams.

After the load test on create forms:

stream reset ... code: 0x1002 ... error code: 4098

Libp2p defines 0x1002 as StreamResourceLimitExceeded.

The fix gives the stream to the RPC operation that actually owns it:

unicast
  -> opens client stream
  -> closes it when the call finishes

handleCall
  -> receives server stream
  -> closes it when the handler finishes

Cancellation still resets an in-flight unary call so a blocked read is interrupted, but the cancellation callback is removed when the call has already completed.

Cancellation still resets an unfinished unary call to interrupt blocked I/O. The cancellation callback is stopped when the call completes normally.

Problem 2 — Streaming RPC resources were not closed correctly

Streaming RPCs had several related lifecycle problems:

  • Participant streams remained open after handler.Stream returned.
  • Normal context cancellation used Reset, which could discard a recently written message before the remote side consumed it.
  • Failed setup could leave already-opened streams and the temporary initiator host allocated.

The fix gives each streaming resource an explicit lifetime:

successful setup:
open streams
    |
    v
run streaming RPC
    |
    v
close participant streams
    |
    v
close temporary host

failed setup:
open some streams
    |
    v
connection or setup failure
    |
    v
close opened streams
    |
    v
close temporary host

handleStream now closes the participant-side stream when its handler returns. Normal orchestrator shutdown uses graceful Close, while Reset remains limited to aborting unfinished unary calls.

If any requested participant cannot be reached, Stream still returns an error. Allowing threshold protocols to proceed with reachable participants is handled separately.

Close vs Reset

These streams are ending as part of the normal RPC lifecycle, not because of an error. Close releases them gracefully after the RPC is done, while Reset is reserved for aborting an unfinished call and can discard in-flight data.

Reset + immediate cancellation -> message could be lost
Reset + short delay             -> message arrived
Close + immediate cancellation -> message arrived

Verification

The added regressions cover:

  • 300 completed unary calls using the same long-lived context;
  • releasing a participant stream after its handler returns;
  • delivering a successful send before immediate context cancellation;
  • returning context.Canceled for sends after session termination.
go test ./mino/minows -count=1

@JonKuqi JonKuqi self-assigned this Aug 25, 2026
@JonKuqi
JonKuqi requested a review from PascalinDe August 25, 2026 13:47
@JonKuqi
JonKuqi marked this pull request as ready for review August 25, 2026 13:47
@JonKuqi
JonKuqi force-pushed the 34-35-fix-minows-rpc-lifecycle branch from da4ba14 to cab68f8 Compare August 25, 2026 16:48
Comment thread mino/minows/rpc.go Resolved
Comment thread mino/minows/rpc.go Resolved
Comment thread mino/minows/rpc.go Resolved
Comment thread mino/minows/rpc.go Resolved
@JonKuqi
JonKuqi requested a review from PascalinDe August 26, 2026 12:21
@JonKuqi
JonKuqi force-pushed the 34-35-fix-minows-rpc-lifecycle branch from d2d2cfd to cb96354 Compare August 26, 2026 15:23
Base automatically changed from 33-fix-minows-session-lifetime to main August 26, 2026 15:26
@JonKuqi
JonKuqi force-pushed the 34-35-fix-minows-rpc-lifecycle branch from cb96354 to efaf310 Compare August 26, 2026 15:26

@PascalinDe PascalinDe left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JonKuqi
JonKuqi merged commit 3e6af53 into main Aug 27, 2026
@JonKuqi
JonKuqi deleted the 34-35-fix-minows-rpc-lifecycle branch August 27, 2026 08:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MinoWS aborts streaming RPCs with Reset on normal context completion MinoWS leaks completed RPC streams until libp2p rejects new streams

2 participants