Skip to content

[vLLM-TorchTPU] Add DeepSeek-V4-Flash and DeepSeek-V4-Pro Ironwood recipes - #384

Merged
anthonsu merged 1 commit into
AI-Hypercomputer:mainfrom
inucool12:feat/add-deepseek-v4-torchtpu-recipes
Nov 4, 2026
Merged

anthonsu merged 1 commit into
AI-Hypercomputer:mainfrom
inucool12:feat/add-deepseek-v4-torchtpu-recipes

Conversation

@inucool12

Copy link
Copy Markdown
Contributor

Summary

Add GKE serving and benchmark recipes for DeepSeek-V4-Flash and DeepSeek-V4-Pro using the vLLM TorchTPU backend (vllm/vllm-torchtpu:v0.29.0) on Ironwood (TPU v7x):

  • inference/ironwood/vLLM/vLLM-TorchTPU/DeepSeek-V4-Flash:
    • Single-host Deployment on tpu7x-standard-4t (2x2x1 topology, 4 chips / 8 TensorCores).
    • TP=1, DP=8, EP=8, deepseek_v4_fp8 quantization with load-time FP4 MoE weight requantization (W4A8), and fp8_e4m3 KV cache.
    • 8K/1K (8192 input / 1024 output, 2048 prompts) serving benchmark configuration and sample results.
  • inference/ironwood/vLLM/vLLM-TorchTPU/DeepSeek-V4-Pro:
    • Multi-host LeaderWorkerSet + Ray deployment across 2× tpu7x-standard-4t (2x2x2 topology, 8 chips / 16 TensorCores).
    • TP=1, DP=16, EP=16 (--enable-ep-weight-filter, --safetensors-load-strategy=prefetch), deepseek_v4_fp8 quantization with fused EP MoE v2 W4A8 (USE_MOE_FUSED_EP_KERNEL=1, MOE_FUSED_EP_ENABLE_W4A8=1), and fp8_e4m3 KV cache.
    • 8K/1K (8192 input / 1024 output, 2048 prompts, max-concurrency=1024) serving benchmark configuration.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces deployment guides and Kubernetes manifests for serving DeepSeek-V4-Flash and DeepSeek-V4-Pro models using vLLM with the TorchTPU backend on GKE with Ironwood TPUs. The feedback highlights critical startup issues due to invalid CLI arguments like --no-enable-prefix-caching and --generation-config in the server manifests. Additionally, it is recommended to replace the tcpSocket readiness probes with httpGet probes on the /health endpoint to ensure traffic is not routed to the pods before the models are fully loaded and compiled.

@inucool12
inucool12 marked this pull request as ready for review November 3, 2026 22:21
@anthonsu
anthonsu merged commit 2f5c3b1 into AI-Hypercomputer:main Nov 4, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants