Skip to content

Fix recordHeartbeat: swallowed cancel/reset/pause exceptions and missing retry - #2990

Open
brainlimitexceeded wants to merge 1 commit into
temporalio:mainfrom
brainlimitexceeded:fix/heartbeat-swallowed-exceptions-and-retry
Open

Fix recordHeartbeat: swallowed cancel/reset/pause exceptions and missing retry#2990
brainlimitexceeded wants to merge 1 commit into
temporalio:mainfrom
brainlimitexceeded:fix/heartbeat-swallowed-exceptions-and-retry

Conversation

@brainlimitexceeded

Copy link
Copy Markdown

Summary

Two related bugs in ManualActivityCompletionClientImpl.recordHeartbeat, both fixed by the same restructure.

Fixes #2983recordHeartbeat wrapped the RPC call and the response-flag checks in a single try block, so its own ActivityCanceledException/ActivityResetException/ActivityPausedException were caught by the generic catch (Exception e) and turned into ActivityCompletionFailureException via processException. Callers couldn't distinguish "activity was cancelled/reset/paused" from "the heartbeat RPC itself failed" without unwrapping getCause().

The interface also declared throws CanceledFailure — a type this method has never actually thrown. CanceledFailure and ActivityCompletionException are unrelated sibling types under TemporalException; the method genuinely throws ActivityCanceledException (which extends ActivityCompletionException). Changed the declared type to match reality.

Fixes #2984recordHeartbeat was the only one of this class's four RPC methods (complete/fail/reportCancellation/recordHeartbeat) that didn't go through grpcRetryer.retryWithResult(...). A single transient error (RESOURCE_EXHAUSTED from namespace rate limiting, DEADLINE_EXCEEDED, UNAVAILABLE) could fail the heartbeat outright — for async completion, where the heartbeat is the only thing keeping a long-running activity alive, that can cost the activity via heartbeat timeout.

What changed

  • ManualActivityCompletionClientImpl.recordHeartbeat: isolate the RPC call in its own try/catch (now via grpcRetryer.retryWithResult(...), matching the other three methods in this class), and move the cancel/reset/paused flag checks outside that catch so the correct exception always reaches the caller.
  • ManualActivityCompletionClient.recordHeartbeat: throws CanceledFailurethrows ActivityCompletionException, with Javadoc listing the concrete exception types.

Test plan

New ManualActivityCompletionClientImplTest (this class had no direct unit test before), covering:

  • cancelRequested/activityReset/activityPaused response flags throw the specific exception, not swallowed into ActivityCompletionFailureException (both taskToken and byId code paths)
  • A transient StatusRuntimeException (RESOURCE_EXHAUSTED) is retried and the call succeeds on the next attempt
  • A non-transient error (INTERNAL) still correctly surfaces as ActivityCompletionFailureException (no regression)
  • Ran the broader related suite (ActivityCompletionClientImplTest, HeartbeatContextImplTest, ActivityResetTest, ActivityPauseTest, ActivityHeartbeatThrottlingTest) — all passing, no regressions
  • spotlessCheck clean

…ons and missing retry

Two related bugs in the same method:

Fixes temporalio#2983: recordHeartbeat wrapped the RPC call and the response-flag
checks in a single try block, so its own ActivityCanceledException /
ActivityResetException / ActivityPausedException were caught by the
generic catch (Exception e) and turned into
ActivityCompletionFailureException by processException. Callers could
not tell a cancelled/reset/paused activity apart from a failed RPC
without unwrapping getCause(). The interface also declared
`throws CanceledFailure`, a type the method has never actually thrown
(CanceledFailure and ActivityCompletionException are unrelated
siblings under TemporalException) -- changed to the type it genuinely
throws, ActivityCompletionException.

Fixes temporalio#2984: recordHeartbeat was the only one of this class's four
RPC methods (complete/fail/reportCancellation/recordHeartbeat) that
didn't go through grpcRetryer.retryWithResult(...). A single transient
error (RESOURCE_EXHAUSTED from namespace rate limiting, DEADLINE_EXCEEDED,
UNAVAILABLE) could fail the heartbeat outright, which for async
completion can cost a long-running activity via heartbeat timeout. Now
wrapped in the same retry helper the sibling methods already use.

Both fixes land in the same restructure: the RPC call is now isolated
in its own try/catch (wrapped in retryWithResult), and the
cancel/reset/paused flag checks happen outside that catch so the
correct exception always reaches the caller.
@brainlimitexceeded
brainlimitexceeded requested a review from a team as a code owner August 6, 2026 21:46
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants