Gate activity polling on app lease ownership

[wangbill (YunchuWang)]

Scope: activity receive-start gating only

Changes the Azure Storage activity scheduling behavior discussed in azure-functions-durable-extension#1540.

Ownership is required before starting a new activity receive, not rechecked before dispatch. This PR reuses UseAppLease (default true) as an intentional next-major behavior change.

Implementation

Existing app lease acquisition/startup
  -> open local activity gate

LockNextTaskActivityWorkItem
  -> await WaitForActivityOwnershipAsync(caller/shutdown token)
  -> original GetMessageAsync
  -> original session/tracing/registration/execution path

AppLeaseManager holds a small lock-protected Boolean and asynchronous wake-up task. Existing startup/acquisition opens the gate; observed loss or shutdown closes it. UseAppLease=false opens it immediately. Same-AppName workers remain independent queue competitors.

The receive method adds only the cancellation-aware ownership wait before its original queue call. After that call, its code matches the base implementation.

Explicit boundary

An already-started GetMessageAsync call is a polling loop. Ownership loss does not cancel it or reject the message it later returns. It may therefore return and execute an activity after the app becomes passive, potentially well after the lease changed if the queue was empty.

The next receive waits while the gate is closed. Caller/service-shutdown cancellation and existing in-flight renewal/completion remain unchanged.

This is deliberately weaker than requiring ownership at activity admission. It reduces normal passive activity consumption; it is not immediate quiescence, atomic lease transfer, exactly-once execution, or a guarantee that incompatible app versions can safely share a hub.

Removed and excluded

• No post-dequeue ownership recheck or ownership-based abandonment path.
• No OnActivityMessageDequeued production test callback.
• No ownership epochs, reference counts, disposable ownership handles or ownership-loss receive tokens.
• No partition lifecycle, stale-reader, drain, restart, takeover protocol or missing-owner acquisition repairs.
• TablePartitionManager.cs and TestTablePartitionManager.cs match base b385165a exactly.
• Orchestration/entity processing and existing takeover behavior otherwise remain at the base implementation.

Earlier broader commits remain in history; the current net diff is activity-only. Known baseline lifecycle/failback problems, Functions scaler awareness, dependency rollout and public docs need separate work.

Validation of current revision

Exact runtime revision: bdcec1d36c392e0dcbd4d241a308098155413019; comparison base: b385165ac10ecebbf183fdfdb07db33307756792. No production code changed during validation.

Real Azure Storage

The six focused AppLeaseActivityTests passed on both net8.0 and net48 against a dedicated real Azure Storage account. This includes the chosen receive-start-only policy: a controlled ownership-state change does not reject a pending receive, while the next receive waits. Controlled state changes are not described as a live failover test.

Independent worker-process workloads used actual orchestration -> activity -> persisted completion on real Storage:

Scenario	Current revision	Exact base
Single app	20/20 activities completed	20/20 completed
Same AppName, two worker processes	40/40 completed across both workers	40/40 completed across both workers
Initially passive, different app	30/30 ran on the primary; passive ran none	Passive processed 55/60, reproducing the original competition
UseAppLease=false	40/40 completed across both apps	40/40 completed across both apps

No duplicate starts were observed in those checked workloads.

Deployed Azure Functions E2E

Two real Windows Y1 Consumption Function Apps ran a private .NET 8 in-process Durable Functions build using the exact current runtime packages. Loaded Core, AzureStorage, and extension assembly hashes/informational versions matched the fresh package/publish/deployment bundle chain before scenarios were counted.

Scenario	Result
Single-app chain, fan-out and durable timer	Passed, with persisted completion and expected activity results
Initial passive-app exclusion	Passed: two 24-activity batches entered through responsive passive-app HTTP; all 48 activities executed on the primary
Passive-app management	Passed: waiting for ownership did not prevent the exercised host-management operations
Same-app multiworker execution	Passed: 96 activities executed across two actual worker identities (52/44), with overlapping execution
UseAppLease=false	Passed: two 32-activity batches each executed across both apps
One-way takeover and in-flight completion	Passed: four original-app activities completed and 24 subsequent activities executed on the new app

App-lease timings used the defaults: 60-second lease, 25-second renewal, 300-second acquisition interval. Four partitions and four concurrent activities per worker were configured. Worker count was capped at one except the two-worker scenario. Existing receives/in-flight activities after loss were allowed by the stated contract, not treated as a zero-overlap failure.

CI and readiness

All 48 current-commit CI checks passed at the readiness decision, and no review threads were unresolved. The earlier distributed CI failure is not carried forward as a failure of this current successful run.

This evidence supports moving the receive-start-only change out of draft for maintainer review: no blocking regression was observed in the exercised scope. It is not proof of universal regression freedom or an approval to merge.

Limits and cleanup

Only isolated synthetic test resources were used; all resource groups created for this validation were deleted and their absence independently confirmed.

This change still does not provide atomic/exactly-once ownership transfer, immediate cancellation of a receive on lease loss, or fixes for baseline same-process failback/partition lifecycle defects. Scale-to-zero, exhaustive failure injection and performance benchmarking were not covered. Older broader implementations' results are not being substituted for the current revision.

[Copilot]

🟡 Changes recommended

New tests are added under Test/ (not included in the solution/test projects) and there is a confirmed trace/telemetry lifecycle bug on the ownership-loss abandon path in activity dequeue.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR corrects UseAppLease=true behavior in the Azure Storage backend so activity polling/execution is gated by app-lease ownership, aligning it with existing orchestration/entity gating and enabling safe failover and same-AppName scale-out.

Changes:

• Introduces an ownership epoch/signal (AppLeaseOwnershipSignal) and exposes it via AppLeaseManager.WaitForOwnershipAsync to coordinate lease ownership and cancellation.
• Updates AppLeaseManager to reset ownership on lease loss/stop, add a forced-handoff transition fence, and improve shutdown behavior on renewer-triggered lease loss.
• Gates LockNextTaskActivityWorkItem behind lease ownership and abandons dequeued messages that arrive after an ownership loss.

File summaries

File	Description
Test/DurableTask.AzureStorage.Tests/AppLeaseActivityTests.cs	Adds new tests for app-lease activity gating and failover behavior.
src/DurableTask.AzureStorage/Partitioning/AppLeaseOwnershipSignal.cs	Adds an ownership epoch + lost-token mechanism used to gate dispatch/polling.
src/DurableTask.AzureStorage/Partitioning/AppLeaseManager.cs	Wires ownership signaling into app-lease acquisition/renewal/stop, adds transition fencing, and handles lease-loss shutdown more defensively.
src/DurableTask.AzureStorage/AzureStorageOrchestrationServiceSettings.cs	Updates UseAppLease doc comment to reflect orchestration/entity/activity gating and same-AppName parallelism.
src/DurableTask.AzureStorage/AzureStorageOrchestrationService.cs	Gates activity dequeue/dispatch on ownership; cancels outstanding receives on lease loss and abandons post-loss dequeues.

Review details

• Files reviewed: 5/5 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

TryAcquireAppLeaseAsync can attempt a lease change with a missing current owner ID, which can throw and break forced-takeover flows.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 7/7 changed files
• Comments generated: 2
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

It changes core lease/dispatch and partition-manager lifecycle behavior with concurrency implications that warrant final human validation despite added tests.

Review details

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

src/DurableTask.AzureStorage/Partitioning/AppLeaseOwnershipSignal.cs:173

• AppLeaseOwnership can throw NullReferenceException if LostToken or TryBeginDispatch() are accessed after Dispose() (since Dispose() nulls the epoch). Since this type is public + IDisposable, it should fail deterministically with ObjectDisposedException instead of an NRE.
  test/DurableTask.AzureStorage.Tests/AppLeaseActivityTests.cs:702
• This polling helper sets up a timeout/cancellation token, but the storage call ignores it. Passing the token into DownloadTextAsync prevents storage retries/network stalls from running past the test timeout.

• Files reviewed: 7/7 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

The current ownership-signal wiring appears to block activity polling for additional workers sharing the same AppName, which conflicts with the intended same-AppName scale-out behavior.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 5/5 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

Distributed lease transitions and live failover remain insufficiently validated for automated approval.

Review details

• Files reviewed: 5/5 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

The ownership signal lacks the promised per-period cancellation, allowing stale receives across ownership transitions.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 1
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

Pending receives bypass admission checks after ownership loss, and same-app secondary workers never open their gates.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 2
• Review effort level: Balanced