[chenhuiluo]

Decodes in the same PD group share the identical prefill candidate set, but the PD scheduling loop re-ran RunFilterPlugins/RunScorePlugins and TopN for every decode candidate, recomputing the same winner up to topN times (5 by default). With PD groups holding multiple decode replicas (e.g. 1P5D) this repeats the work on every
request, including the Redis pipeline issued by the kvcache-aware score plugin.

Memoize the selected prefill per PD group within one Schedule call, so the group's prefill set is filtered/scored once and reused by the remaining decode candidates.Pairing results are unchanged.

Benchmark (1P5D, all decodes in one group, topN=5):
allocs/op 85 -> 39 (-54%)
B/op 4960 -> 2224 (-55%)
ns/op ~25.2us -> ~10.4us (median, ~-59%)

Deterministic test: filter/score plugin invocation count drops from 6 to 2 (one decode pass + one shared prefill pass); existing PD scheduling tests pass.

What type of PR is this?

/kind enhancement

What this PR does / why we need it:

In the optimized PD scheduling path (SchedulerImpl.Schedule, pkg/kthena-router/scheduler/scheduler_impl.go), the loop over the top-N decode candidates re-ran filter/score plugins and TopN on the prefill set for every candidate. Decodes that belong to the same PD group resolve to the same prefill candidate set (GetPrefillPodsForDecodeGroup returns the group's prefills), so for a group with N decode replicas the identical prefill set was filtered and scored N times, with an identical result every time.

This is on the per-request data path. With the kvcache-aware score plugin enabled, each redundant score pass also issues a Redis pipeline, so the repeated work includes network I/O.

The fix memoizes the selected best prefill per PD group for the duration of one Schedule call. The first decode of a group triggers filter/score/TopN; the remaining decodes of that group reuse the cached winner. Pairing results are identical — the memo only skips recomputing the same answer with the same inputs (same candidate set, same filter/score context within one schedule call).

Benchmark (BenchmarkSchedulePDSameGroup, 1P5D all in one group, topN=5):

metric	before	after	delta
allocs/op	85	39	-54%
B/op	4,960	2,224	-55%
ns/op (median of 10)	~25.2µs	~10.4µs	~-59%

The benchmark uses a lightweight counting plugin, so it does not include the repeated Redis round-trips of kvcache-aware; the production-path saving is larger than the table shows.

Which issue(s) this PR fixes:
Fixes #

Bug evidence (required for bug-related PRs):
N/A

Special notes for your reviewer:

• No behavior change: the memo is a local map inside one Schedule call; it is not shared across calls or goroutines, so no locking is involved. Pairing output is unchanged.
• New deterministic test TestPDScheduleRepeatedGroupCallsFilterScoreOnce asserts the plugin invocation count drops from 6 to 2 for the 1P5D case (1 decode pass + 1 shared prefill pass), instead of relying on timing numbers. Existing PD scheduling tests (TestSchedulePDGroup, TestSchedulePDGroupStickyPair) still pass.
• The behavior is correct also when the top-N decodes span multiple groups: each distinct group key is scored once, i.e. exactly the same work as before, with no extra
  cost beyond one map lookup.

Does this PR introduce a user-facing change?:

NONE

[volcano-sh-bot]

@chenhuiluo: The label(s) kind/performance cannot be applied, because the repository doesn't have them.

Details

In response to this:

Decodes in the same PD group share the identical prefill candidate set, but the PD scheduling loop re-ran RunFilterPlugins/RunScorePlugins and TopN for every decode candidate, recomputing the same winner up to topN times (5 by default). With PD groups holding multiple decode replicas (e.g. 1P5D) this repeats the work on every
request, including the Redis pipeline issued by the kvcache-aware score plugin.

Memoize the selected prefill per PD group within one Schedule call, so the group's prefill set is filtered/scored once and reused by the remaining decode candidates.Pairing results are unchanged.

Benchmark (1P5D, all decodes in one group, topN=5):
allocs/op 85 -> 39 (-54%)
B/op 4960 -> 2224 (-55%)
ns/op ~25.2us -> ~10.4us (median, ~-59%)

Deterministic test: filter/score plugin invocation count drops from 6 to 2 (one decode pass + one shared prefill pass); existing PD scheduling tests pass.

What type of PR is this?

/kind performance

What this PR does / why we need it:

In the optimized PD scheduling path (SchedulerImpl.Schedule, pkg/kthena-router/scheduler/scheduler_impl.go), the loop over the top-N decode candidates re-ran filter/score plugins and TopN on the prefill set for every candidate. Decodes that belong to the same PD group resolve to the same prefill candidate set (GetPrefillPodsForDecodeGroup returns the group's prefills), so for a group with N decode replicas the identical prefill set was filtered and scored N times, with an identical result every time.

This is on the per-request data path. With the kvcache-aware score plugin enabled, each redundant score pass also issues a Redis pipeline, so the repeated work includes network I/O.

The fix memoizes the selected best prefill per PD group for the duration of one Schedule call. The first decode of a group triggers filter/score/TopN; the remaining decodes of that group reuse the cached winner. Pairing results are identical — the memo only skips recomputing the same answer with the same inputs (same candidate set, same filter/score context within one schedule call).

Benchmark (BenchmarkSchedulePDSameGroup, 1P5D all in one group, topN=5):

metric	before	after	delta
allocs/op	85	39	-54%
B/op	4,960	2,224	-55%
ns/op (median of 10)	~25.2µs	~10.4µs	~-59%

The benchmark uses a lightweight counting plugin, so it does not include the repeated Redis round-trips of kvcache-aware; the production-path saving is larger than the table shows.

Which issue(s) this PR fixes:
Fixes #

Bug evidence (required for bug-related PRs):
N/A

Special notes for your reviewer:

• No behavior change: the memo is a local map inside one Schedule call; it is not shared across calls or goroutines, so no locking is involved. Pairing output is unchanged.
• New deterministic test TestPDScheduleRepeatedGroupCallsFilterScoreOnce asserts the plugin invocation count drops from 6 to 2 for the 1P5D case (1 decode pass + 1 shared prefill pass), instead of relying on timing numbers. Existing PD scheduling tests (TestSchedulePDGroup, TestSchedulePDGroupStickyPair) still pass.
• The behavior is correct also when the top-N decodes span multiple groups: each distinct group key is scored once, i.e. exactly the same work as before, with no extra
  cost beyond one map lookup.

Does this PR introduce a user-facing change?:

NONE

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[volcano-sh-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign yaozengzeng for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• pkg/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment