Repository navigation
#recipebot [Ironwood][SGLang-TorchTPU] Kimi-K3.1-Pro multi-host recipe - #385
Conversation
|
@joeywan-google @anthonsu PTAL — SGLang-TorchTPU multi-host Kimi-K3.1-Pro recipe (tpu7x 2x2x4), following the Cluster Toolkit + inline benchmark Pod layout. Will update the PR description with the BigQuery uBench run_ids in ~15 min once the live runs finish. |
There was a problem hiding this comment.
Code Review
This pull request adds a deployment guide and Kubernetes manifests to serve the Kimi-K3.1-Pro model with SGLang (TorchTPU) on multi-host Ironwood TPUs. A critical issue was identified in the server manifest where the non-existent environment variable LWS_GROUP_SIZE is used for the --nnodes argument, which would cause the server startup to fail. It is recommended to use LWS_SIZE instead.
| --dtype bfloat16 \ | ||
| --tp-size 32 \ | ||
| --ep-size 8 \ | ||
| --nnodes "${LWS_GROUP_SIZE}" \ |
There was a problem hiding this comment.
The LeaderWorkerSet (LWS) controller injects the environment variable LWS_SIZE to represent the total number of pods in the group. The variable LWS_GROUP_SIZE does not exist and will resolve to an empty string, causing the SGLang server startup to fail because --nnodes expects a valid integer. Please use LWS_SIZE instead.
--nnodes "${LWS_SIZE}" \
Summary
Adds a multi-host SGLang-TorchTPU inference recipe for
moonshotai/Kimi-K3.1-Proon Ironwood (TPU7x,2x2x4, 4 hosts / 16 chips / 32 TensorCores,--tp-size 32 --ep-size 8).lmsysorg/sglang:dev-kimi-k31-pro-tpu.README.md+kimi-k3-1-pro-server.yaml(StorageClass+ 4-hostLeaderWorkerSet+Service), with cluster creation via Cluster Toolkit (examples/gke-tpu-7x/gke-tpu-7x.yaml,tpu_topology: 2x2x4,num_slices: 1) and benchmark client Pods (1k/1kand8k/1k) inlined inREADME.md(apply → logs → delete).sglang-torchtpubenchmark/tpu/recipe/configs/kimi_k3_1/server.pyrecipe (pallas_tpu_mla_sparseattention,fp8_e4m3KV cache,page_size: 128, 480K context length,torch.compile, and optional host-DRAM hierarchical KV cache note for multi-turn/agentic workloads).xxxplaceholders per recipe policy.Files
inference/ironwood/SGLang/SGLang-TorchTPU/Kimi-K3.1-Pro/multi-host/README.mdinference/ironwood/SGLang/SGLang-TorchTPU/Kimi-K3.1-Pro/multi-host/kimi-k3-1-pro-server.yamlValidation (uBench run_ids)
sglang_inference-torchtpu_sglang/kimi_k3_1_pro_v7x_2x2x4_1k_1k-2026-11-05_034516-afbb715c-af44-4b67-bcbe-5db03126d280sglang_inference-torchtpu_sglang/kimi_k3_1_pro_v7x_2x2x4_8k_1k-2026-11-05_024609-96b67c89-5b9b-4917-901e-6f55c07ba5besglang_inference-torchtpu_sglang/kimi_k3_1_pro_v7x_2x2x4_agentx-2026-11-05_024559-033cab69-8872-434a-b180-27f7eac66056