Skip to content

#recipebot [Ironwood][SGLang-TorchTPU] Kimi-K3.1-Pro multi-host recipe - #385

Merged
joeywan-google merged 1 commit into
AI-Hypercomputer:mainfrom
syeda-anjum:sglang-kimi-k3-1-pro-v7x-multi-host
Nov 5, 2026
Merged

joeywan-google merged 1 commit into
AI-Hypercomputer:mainfrom
syeda-anjum:sglang-kimi-k3-1-pro-v7x-multi-host

Conversation

@syeda-anjum

@syeda-anjum syeda-anjum commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds a multi-host SGLang-TorchTPU inference recipe for moonshotai/Kimi-K3.1-Pro on Ironwood (TPU7x, 2x2x4, 4 hosts / 16 chips / 32 TensorCores, --tp-size 32 --ep-size 8).

  • Public image: lmsysorg/sglang:dev-kimi-k31-pro-tpu.
  • Layout follows the existing recipes: README.md + kimi-k3-1-pro-server.yaml (StorageClass + 4-host LeaderWorkerSet + Service), with cluster creation via Cluster Toolkit (examples/gke-tpu-7x/gke-tpu-7x.yaml, tpu_topology: 2x2x4, num_slices: 1) and benchmark client Pods (1k/1k and 8k/1k) inlined in README.md (apply → logs → delete).
  • Server flags follow the upstream sglang-torchtpu benchmark/tpu/recipe/configs/kimi_k3_1/server.py recipe (pallas_tpu_mla_sparse attention, fp8_e4m3 KV cache, page_size: 128, 480K context length, torch.compile, and optional host-DRAM hierarchical KV cache note for multi-turn/agentic workloads).
  • Benchmark numbers are left as xxx placeholders per recipe policy.

Files

  • inference/ironwood/SGLang/SGLang-TorchTPU/Kimi-K3.1-Pro/multi-host/README.md
  • inference/ironwood/SGLang/SGLang-TorchTPU/Kimi-K3.1-Pro/multi-host/kimi-k3-1-pro-server.yaml

Validation (uBench run_ids)

  • 1k/1k: sglang_inference-torchtpu_sglang/kimi_k3_1_pro_v7x_2x2x4_1k_1k-2026-11-05_034516-afbb715c-af44-4b67-bcbe-5db03126d280
  • 8k/1k: sglang_inference-torchtpu_sglang/kimi_k3_1_pro_v7x_2x2x4_8k_1k-2026-11-05_024609-96b67c89-5b9b-4917-901e-6f55c07ba5be
  • AgentX: sglang_inference-torchtpu_sglang/kimi_k3_1_pro_v7x_2x2x4_agentx-2026-11-05_024559-033cab69-8872-434a-b180-27f7eac66056

@syeda-anjum

Copy link
Copy Markdown
Contributor Author

@joeywan-google @anthonsu PTAL — SGLang-TorchTPU multi-host Kimi-K3.1-Pro recipe (tpu7x 2x2x4), following the Cluster Toolkit + inline benchmark Pod layout. Will update the PR description with the BigQuery uBench run_ids in ~15 min once the live runs finish.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds a deployment guide and Kubernetes manifests to serve the Kimi-K3.1-Pro model with SGLang (TorchTPU) on multi-host Ironwood TPUs. A critical issue was identified in the server manifest where the non-existent environment variable LWS_GROUP_SIZE is used for the --nnodes argument, which would cause the server startup to fail. It is recommended to use LWS_SIZE instead.

--dtype bfloat16 \
--tp-size 32 \
--ep-size 8 \
--nnodes "${LWS_GROUP_SIZE}" \

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The LeaderWorkerSet (LWS) controller injects the environment variable LWS_SIZE to represent the total number of pods in the group. The variable LWS_GROUP_SIZE does not exist and will resolve to an empty string, causing the SGLang server startup to fail because --nnodes expects a valid integer. Please use LWS_SIZE instead.

                  --nnodes "${LWS_SIZE}" \

@joeywan-google
joeywan-google merged commit d9f7d91 into AI-Hypercomputer:main Nov 5, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants