Fix recordHeartbeat: swallowed cancel/reset/pause exceptions and missing retry - #2990
Open
brainlimitexceeded wants to merge 1 commit into
Conversation
…ons and missing retry Two related bugs in the same method: Fixes temporalio#2983: recordHeartbeat wrapped the RPC call and the response-flag checks in a single try block, so its own ActivityCanceledException / ActivityResetException / ActivityPausedException were caught by the generic catch (Exception e) and turned into ActivityCompletionFailureException by processException. Callers could not tell a cancelled/reset/paused activity apart from a failed RPC without unwrapping getCause(). The interface also declared `throws CanceledFailure`, a type the method has never actually thrown (CanceledFailure and ActivityCompletionException are unrelated siblings under TemporalException) -- changed to the type it genuinely throws, ActivityCompletionException. Fixes temporalio#2984: recordHeartbeat was the only one of this class's four RPC methods (complete/fail/reportCancellation/recordHeartbeat) that didn't go through grpcRetryer.retryWithResult(...). A single transient error (RESOURCE_EXHAUSTED from namespace rate limiting, DEADLINE_EXCEEDED, UNAVAILABLE) could fail the heartbeat outright, which for async completion can cost a long-running activity via heartbeat timeout. Now wrapped in the same retry helper the sibling methods already use. Both fixes land in the same restructure: the RPC call is now isolated in its own try/catch (wrapped in retryWithResult), and the cancel/reset/paused flag checks happen outside that catch so the correct exception always reaches the caller.
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two related bugs in
ManualActivityCompletionClientImpl.recordHeartbeat, both fixed by the same restructure.Fixes #2983 —
recordHeartbeatwrapped the RPC call and the response-flag checks in a singletryblock, so its ownActivityCanceledException/ActivityResetException/ActivityPausedExceptionwere caught by the genericcatch (Exception e)and turned intoActivityCompletionFailureExceptionviaprocessException. Callers couldn't distinguish "activity was cancelled/reset/paused" from "the heartbeat RPC itself failed" without unwrappinggetCause().The interface also declared
throws CanceledFailure— a type this method has never actually thrown.CanceledFailureandActivityCompletionExceptionare unrelated sibling types underTemporalException; the method genuinely throwsActivityCanceledException(which extendsActivityCompletionException). Changed the declared type to match reality.Fixes #2984 —
recordHeartbeatwas the only one of this class's four RPC methods (complete/fail/reportCancellation/recordHeartbeat) that didn't go throughgrpcRetryer.retryWithResult(...). A single transient error (RESOURCE_EXHAUSTEDfrom namespace rate limiting,DEADLINE_EXCEEDED,UNAVAILABLE) could fail the heartbeat outright — for async completion, where the heartbeat is the only thing keeping a long-running activity alive, that can cost the activity via heartbeat timeout.What changed
ManualActivityCompletionClientImpl.recordHeartbeat: isolate the RPC call in its owntry/catch(now viagrpcRetryer.retryWithResult(...), matching the other three methods in this class), and move the cancel/reset/paused flag checks outside that catch so the correct exception always reaches the caller.ManualActivityCompletionClient.recordHeartbeat:throws CanceledFailure→throws ActivityCompletionException, with Javadoc listing the concrete exception types.Test plan
New
ManualActivityCompletionClientImplTest(this class had no direct unit test before), covering:cancelRequested/activityReset/activityPausedresponse flags throw the specific exception, not swallowed intoActivityCompletionFailureException(both taskToken and byId code paths)StatusRuntimeException(RESOURCE_EXHAUSTED) is retried and the call succeeds on the next attemptINTERNAL) still correctly surfaces asActivityCompletionFailureException(no regression)ActivityCompletionClientImplTest,HeartbeatContextImplTest,ActivityResetTest,ActivityPauseTest,ActivityHeartbeatThrottlingTest) — all passing, no regressionsspotlessCheckclean