[kube-gopher]

What type of PR is this?
/kind enhancement

What this PR does / why we need it:

• The unit is the ServingGroup (ServingGroupRecreate), the Role instance (RoleRecreate), or the Pod (None).
• Failure 1 is recovered as before. After that, the controller waits for max(restartGracePeriodSeconds, backoff), where the backoff is 10s, 20s, 40s, and so on, capped at 5m.
• The count resets after the unit has stayed Running for 10m, or when the failed Pod belongs to a new revision.
• Evicted or preempted Pods (DisruptionTarget=True) and Pods deleted by others don't count.
• During the wait, the failed Pod stays in place for debugging. If it becomes Ready again, it is kept.
• Units in backoff are reported by a RecoveryBackOff condition and a RecoveryBackOff Warning event. The condition only appears after a unit has actually backed off.
• New controller-manager flags --recovery-backoff-base, --recovery-backoff-max, and --recovery-backoff-reset, with Helm values under workload.controllerManager.recoveryBackoff. --recovery-backoff-base=0s restores the old behavior.

Which issue(s) this PR fixes:
Fixes #1951

Bug evidence (required for bug-related PRs):

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

[volcano-sh-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign lizhencheng9527 for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• charts/OWNERS
• client-go/OWNERS
• cmd/OWNERS
• docs/OWNERS
• pkg/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[kube-gopher]

/remove-kind feature
/kind enhancement