Repository navigation
Conversation
…ronwood (TPU v7x)
There was a problem hiding this comment.
Code Review
This pull request introduces a comprehensive guide and Kubernetes manifests for serving the Kimi-K3 model using vLLM with TorchTPU Pallas megakernels on Ironwood TPUs in a disaggregated 1P1D setup. The review feedback identifies a configuration mismatch in the deployment manifest: the prefill engine configuration redundantly includes the speculative decoding draft model parameter, while the decode engine configuration (which actually runs speculative decoding) is missing this parameter. Correcting these configurations will ensure proper model loading and execution.
| --max-num-seqs 4 \ | ||
| --max-num-batched-tokens 8192 \ | ||
| --async-scheduling \ | ||
| --model-loader-extra-config "{\"presharded\":\"/models/presharded/kimi-k3-tp32\",\"draft\":\"${DSPARK_DIR}\"}" \ |
There was a problem hiding this comment.
The prefill engine does not run speculative decoding or load the draft model. Therefore, passing the "draft":"${DSPARK_DIR}" parameter in --model-loader-extra-config is redundant and unused here. It should be removed to keep the configuration clean and avoid potential confusion.
--model-loader-extra-config '{"presharded":"/models/presharded/kimi-k3-tp32"}' \| --max-num-seqs 4 \ | ||
| --max-num-batched-tokens 8192 \ | ||
| --async-scheduling \ | ||
| --model-loader-extra-config "{\"presharded\":\"/models/presharded/kimi-k3-tp32\",\"draft\":\"${DSPARK_DIR}\"}" \ |
There was a problem hiding this comment.
The prefill engine does not run speculative decoding or load the draft model. Therefore, passing the "draft":"${DSPARK_DIR}" parameter in --model-loader-extra-config is redundant and unused here. It should be removed to keep the configuration clean and avoid potential confusion.
--model-loader-extra-config '{"presharded":"/models/presharded/kimi-k3-tp32"}' \| --max-num-seqs 16 \ | ||
| --max-num-batched-tokens 512 \ | ||
| --async-scheduling \ | ||
| --model-loader-extra-config '{"presharded":"/models/presharded/kimi-k3-tp32"}' \ |
There was a problem hiding this comment.
The decode engine is the one running speculative decoding with DSpark and loading the draft model. The megakernel loader needs the "draft":"${DSPARK_DIR}" parameter in --model-loader-extra-config to correctly locate and load the presharded draft model weights. Additionally, we must use double quotes and escape them so that the shell expands the ${DSPARK_DIR} variable correctly.
--model-loader-extra-config "{\"presharded\":\"/models/presharded/kimi-k3-tp32\",\"draft\":\"${DSPARK_DIR}\"}" \| --max-num-seqs 16 \ | ||
| --max-num-batched-tokens 512 \ | ||
| --async-scheduling \ | ||
| --model-loader-extra-config '{"presharded":"/models/presharded/kimi-k3-tp32"}' \ |
There was a problem hiding this comment.
The decode engine is the one running speculative decoding with DSpark and loading the draft model. The megakernel loader needs the "draft":"${DSPARK_DIR}" parameter in --model-loader-extra-config to correctly locate and load the presharded draft model weights. Additionally, we must use double quotes and escape them so that the shell expands the ${DSPARK_DIR} variable correctly.
--model-loader-extra-config "{\"presharded\":\"/models/presharded/kimi-k3-tp32\",\"draft\":\"${DSPARK_DIR}\"}" \
Adds an Ironwood (TPU v7x) GKE recipe for serving
moonshotai/Kimi-K3in a disaggregated 1P1D configuration (2 × 2x2x4= 32 TPU v7x chips,TP=32per role) using the TorchTPU Pallas megakernel backend, Raiden KV transfer,vllm-router, andInferact/Kimi-K3-DSparkspeculative decoding.Important
Do not merge until the Kimi-K3 megakernel branch (
inferact/k3-megakernel, commit30c9ba966bdf193c1786d29606608c62c8806808) is merged intovllm-project/vllm-torchtpumain.Step 3 of
README.mdalready targetsmain(./docker/build_image.sh -t ${IMAGE_URI} --target dev --vllm-rs) so no follow-up edit is needed once the upstream merge lands.Building the Image Before
mainMerge (Reviewer Reproduction)To test this recipe before
inferact/k3-megakernellands onmain, build from commit30c9ba966bdf193c1786d29606608c62c8806808(requiresgcloudread access tous-python.pkg.dev/ml-oss-artifacts-transient/torch-tpu-virtual-registry):dockerfile.patch(only needed on30c9ba966; already present onmain)Verified Benchmark Results (
2 × 2x2x4TPU v7x = 32 chips,8k/1k,temperature=0)Verified across two independent end-to-end GKE runs (
vllm bench serve, random8192input /1024output,0failed requests):