[chenhuiluo]

Rolling updates delete old-revision Pods immediately, SIGKILLing in-flight LLM requests (client-visible 502/connection resets). This adds a Pod-boundary drain so an old Pod finishes its in-flight requests before termination:

• controller: unchanged delete path. A ModelServing annotation (modelserving.volcano.sh/drain-timeout, opt-in, >= 60s) enables the drain; at Pod creation the controller injects a preStop hook and the hook env vars (nothing is copied into user templates).
• router: a Pod with a deletionTimestamp stays tracked in the store (in-flight metrics keep being scraped) and is excluded from all scheduling-candidate queries; the store is cleared on the object-delete event. The router never writes to Pods.
• preStop hook: polls the engine metrics and exits as soon as running/waiting read zero; the effective wait is min(DRAIN_TIMEOUT, 70% * GRACE_SECONDS). A role-template annotation (drain-kv-buffer, <= 30s) optionally holds a KV buffer for PD prefill Pods so paired decodes finish pulling the last KV.
• examples: aggregated and PD disaggregated ModelServing samples.
• proposal: the design is described in docs/proposal/modelserving-lossless-upgrade.md, tracked on PR: docs(proposal): add lossless upgrade proposal for ModelServing  #1884; Multi-replica routers work by construction since the drain judgment uses engine metrics inside the Pod.

What type of PR is this?
/kind feature

What this PR does / why we need it:
Rolling updates of a ModelServing delete old-revision Pods via DeleteCollection with empty DeleteOptions. LLM inference requests — especially streaming — often run for tens of seconds to minutes, so the old Pod dies with its requests still generating and clients see broken connections.

This PR drains an old Pod at the Pod boundary before it terminates:

• The controller's delete path is unchanged; the drain is opt-in via one annotation (modelserving.volcano.sh/drain-timeout, e.g. "300s"). Invalid or too-small values are rejected with a warning event.
• The router reactively drains Pods that carry a deletionTimestamp: they stay tracked in the store (metrics keep being scraped) but are excluded from all scheduling-candidate queries, so no new request is dispatched to them while their in-flight requests finish.
• A preStop hook injected at Pod creation polls the engine metrics (vLLM/SGLang counters) and exits as soon as running/waiting read zero, so SIGTERM reaches an idle engine. The wait is bounded by min(DRAIN_TIMEOUT, 70% * GRACE_SECONDS); a role-template annotation (drain-kv-buffer, <= 30s) lets PD prefill Pods hold a KV buffer so
  paired decodes finish pulling the last KV.
• Best-effort by design: engines without parseable metrics and residual races fall back to the grace period — never worse than the current immediate-delete behavior.

Which issue(s) this PR fixes:
Fixes #1778

Bug evidence (required for bug-related PRs):
N/A

Special notes for your reviewer:

• The controller's delete path is intentionally untouched; the drain is opt-in and defaults to off (no annotation = immediate delete, unchanged behavior).
• The router never writes to Pods in this design, so no pods/patch permission is added.
• The drain judgment runs inside the Pod from engine metrics, so multi-replica router deployments work without Redis-based counter aggregation.
• Under RoleRollingUpdate, a prefill Pod draining while its decodes keep serving traffic may wait out the in-hook timeout (the decode counters stay above zero); this is
  covered by the best-effort framing in the proposal's PD section — the default ServingGroupRollingUpdate keeps the judgment exact.
• Examples: examples/model-serving/lossless-upgrade.yaml (aggregated) and examples/model-serving/lossless-upgrade-pd-disaggregation.yaml (PD).

Does this PR introduce a user-facing change?:

ModelServing: added opt-in lossless upgrade. Setting the annotation `modelserving.volcano.sh/drain-timeout` on a ModelServing makes rolling updates drain terminating Pods (the router stops new dispatch and a preStop hook waits for in-flight requests to finish) instead of deleting them immediately.

[volcano-sh-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign yaozengzeng for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• examples/OWNERS
• pkg/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[LiZhenCheng9527]

Regular expressions require that metric names be immediately followed by a space, but the names output by vllm/sglang are followed by “{”.

[chenhuiluo]

Regular expressions require that metric names be immediately followed by a space, but the names output by vllm/sglang are followed by “{”.

Good catch. Fixed: the metric name now accepts an optional label block ((?:{[^}]*})?\s+(\S+)), so vllm:num_requests_running{engine="0",...} 0.0 parses; an empty match now keeps polling instead of exiting immediately. Verified against real vLLM output and covered by tests.