Repository navigation
[vLLM-TorchTPU] Add DeepSeek-V4-Flash and DeepSeek-V4-Pro Ironwood recipes #384
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
anthonsu
merged 1 commit into
AI-Hypercomputer:main
from
inucool12:feat/add-deepseek-v4-torchtpu-recipes
Nov 4, 2026
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
252 changes: 252 additions & 0 deletions
252
inference/ironwood/vLLM/vLLM-TorchTPU/DeepSeek-V4-Flash/README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,252 @@ | ||
| # Serve DeepSeek-V4-Flash with vLLM (TorchTPU) on Ironwood TPU | ||
|
|
||
| In this guide, we show how to serve DeepSeek-V4-Flash (`deepseek-ai/DeepSeek-V4-Flash`, Mixture-of-Experts model with Multi-head Latent Attention) with vLLM using the **TorchTPU** backend on Ironwood (TPU v7x) using GKE. | ||
|
|
||
| We deploy it on a single **tpu7x-standard-4t** node pool (`2x2x1` topology, 4 TPU v7x chips, 8 TensorCores) using **Data Parallelism (DP) size of 8**, **Expert Parallelism (EP) size of 8**, and **Tensor Parallelism (TP) size of 1**, with `deepseek_v4_fp8` quantization, load-time FP4 MoE weight requantization (`W4A8`), and `fp8_e4m3` KV cache. | ||
|
|
||
| ## Verified Models | ||
|
|
||
| The following DeepSeek-V4-Flash model is verified for deployment on TPU with the vLLM TorchTPU backend. | ||
|
|
||
| | Model | Quantization | Min TPUs (Chips) | Topology | HuggingFace | | ||
| | :---- | :---- | :---- | :---- | :---- | | ||
| | DeepSeek-V4-Flash | `deepseek_v4_fp8` (W4A8 MoE), `fp8_e4m3` KV | 4× (`tpu7x-standard-4t`) | `2x2x1` (1 host) | [deepseek-ai/DeepSeek-V4-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) | | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| 1. A GKE cluster with TPU v7x support (GKE version `1.34.0-gke.2201000` or later). | ||
| 2. GCloud CLI and Kubectl installed and configured. | ||
| 3. A Hugging Face account and access token with access to `deepseek-ai/DeepSeek-V4-Flash`. | ||
|
|
||
| --- | ||
|
|
||
| ## Step 1: Define Parameters | ||
|
|
||
| Export the necessary environment variables for your Google Cloud project and GKE cluster: | ||
|
|
||
| ```shell | ||
| export PROJECT_ID="<YOUR_PROJECT_ID>" | ||
| export CLUSTER_NAME="<YOUR_CLUSTER_NAME>" | ||
| export REGION="<YOUR_REGION>" | ||
| export ZONE="<YOUR_ZONE>" | ||
| export NODEPOOL_NAME="dsv4-flash-tpu-pool" | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Step 2: Create the TPU Node Pool | ||
|
|
||
| Create a TPU v7x (Ironwood) node pool with the `tpu7x-standard-4t` machine type. This machine type provides 4 chips (8 logical TensorCore devices): | ||
|
|
||
| ```shell | ||
| gcloud container node-pools create ${NODEPOOL_NAME} \ | ||
| --project=${PROJECT_ID} \ | ||
| --location=${REGION} \ | ||
| --node-locations=${ZONE} \ | ||
| --num-nodes=1 \ | ||
| --machine-type=tpu7x-standard-4t \ | ||
| --cluster=${CLUSTER_NAME} | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Step 3: Setup Kubernetes Configurations | ||
|
|
||
| 1. Configure `kubectl` to communicate with your cluster: | ||
|
|
||
| ```shell | ||
| gcloud container clusters get-credentials ${CLUSTER_NAME} --location=${REGION} --project=${PROJECT_ID} | ||
| ``` | ||
|
|
||
| 2. Create the Namespace `vllm-deepseek-v4-flash` if you want to run it in an isolated namespace (recommended): | ||
|
|
||
| ```shell | ||
| kubectl create namespace vllm-deepseek-v4-flash | ||
| ``` | ||
|
|
||
| 3. Create a Kubernetes secret for your Hugging Face token in the namespace: | ||
|
|
||
| ```shell | ||
| export HF_TOKEN="YOUR_HUGGING_FACE_API_TOKEN" | ||
| kubectl create secret generic hf-secret \ | ||
| --namespace=vllm-deepseek-v4-flash \ | ||
| --from-literal=hf_api_token=${HF_TOKEN} | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Step 4: Apply the vLLM Server Manifest | ||
|
|
||
| Apply the vLLM server manifest using the provided [deepseek_v4_flash-server.yaml](./deepseek_v4_flash-server.yaml) file in this directory: | ||
|
|
||
| ```shell | ||
| cd inference/ironwood/vLLM/vLLM-TorchTPU/DeepSeek-V4-Flash | ||
| kubectl apply -f deepseek_v4_flash-server.yaml | ||
| ``` | ||
|
|
||
| Monitor the progress of the server deployment: | ||
|
|
||
| ```shell | ||
| kubectl get pods -n vllm-deepseek-v4-flash -w | ||
| ``` | ||
|
|
||
| You can check the server logs using: | ||
|
|
||
| ```shell | ||
| kubectl logs -n vllm-deepseek-v4-flash deployment/vllm-deepseek-v4-flash -c vllm-torchtpu -f | ||
| ``` | ||
|
|
||
| Once the server has successfully loaded and compiled the model, you should see logs like: | ||
|
|
||
| ``` | ||
| (APIServer pid=1) INFO: Started server process [1] | ||
| (APIServer pid=1) INFO: Waiting for application startup. | ||
| (APIServer pid=1) INFO: Application startup complete. | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Step 5: Interact with the Model (Optional) | ||
|
|
||
| You can test the server using a port forward and `curl`: | ||
|
|
||
| 1. Port forward the vLLM service to port `8000` on your local machine: | ||
|
|
||
| ```shell | ||
| kubectl port-forward -n vllm-deepseek-v4-flash service/vllm-deepseek-v4-flash-service 8000:8000 | ||
| ``` | ||
|
|
||
| 2. Run a `curl` request: | ||
|
|
||
| ```shell | ||
| curl http://localhost:8000/v1/chat/completions \ | ||
| -H "Content-Type: application/json" \ | ||
| -d '{ | ||
| "model": "deepseek-ai/DeepSeek-V4-Flash", | ||
| "messages": [ | ||
| {"role": "user", "content": "Hello! Tell me a joke."} | ||
| ], | ||
| "max_tokens": 100, | ||
| "temperature": 0.0 | ||
| }' | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Step 6: Run Serving Benchmarks | ||
|
|
||
| We use `vllm bench serve` to evaluate serving throughput and latency against our running vLLM server for an **8K Input / 1K Output (8k/1k)** workload. | ||
|
|
||
| ### Workload: 8K Input / 1K Output (8k/1k) | ||
|
|
||
| Save the following manifest as `deepseek_v4_flash-benchmark-8k1k.yaml`: | ||
|
|
||
| ```yaml | ||
| apiVersion: v1 | ||
| kind: Pod | ||
| metadata: | ||
| name: dsv4-flash-bench-8k1k | ||
| namespace: vllm-deepseek-v4-flash | ||
| spec: | ||
| restartPolicy: Never | ||
| terminationGracePeriodSeconds: 60 | ||
| containers: | ||
| - name: vllm-bench | ||
| image: vllm/vllm-torchtpu:v0.29.0 | ||
| command: ["/bin/bash", "-c"] | ||
| args: | ||
| - | | ||
| while ! curl -sf http://vllm-deepseek-v4-flash-service:8000/health; do sleep 30 && echo 'Waiting for server...'; done | ||
| vllm bench serve \ | ||
| --backend=vllm \ | ||
| --base-url=http://vllm-deepseek-v4-flash-service:8000 \ | ||
| --model=deepseek-ai/DeepSeek-V4-Flash \ | ||
| --tokenizer=deepseek-ai/DeepSeek-V4-Flash \ | ||
| --trust-remote-code \ | ||
| --dataset-name=random \ | ||
| --random-input-len=8192 \ | ||
| --random-output-len=1024 \ | ||
| --num-prompts=2048 \ | ||
| --ignore-eos \ | ||
| --skip-chat-template | ||
| env: | ||
| - name: PYTHONUNBUFFERED | ||
| value: "1" | ||
| - name: HUGGING_FACE_HUB_TOKEN | ||
| valueFrom: | ||
| secretKeyRef: | ||
| key: hf_api_token | ||
| name: hf-secret | ||
| ``` | ||
|
|
||
| Apply it using `kubectl`: | ||
|
|
||
| ```shell | ||
| kubectl apply -f deepseek_v4_flash-benchmark-8k1k.yaml | ||
| ``` | ||
|
|
||
| Check the logs of the benchmark run: | ||
|
|
||
| ```shell | ||
| kubectl logs -n vllm-deepseek-v4-flash dsv4-flash-bench-8k1k -f | ||
| ``` | ||
|
|
||
| When complete, clean up the benchmark pod: | ||
|
|
||
| ```shell | ||
| kubectl delete -f deepseek_v4_flash-benchmark-8k1k.yaml | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ### Example Benchmark Output | ||
|
|
||
| The benchmark client will print a table resembling the following: | ||
|
|
||
| ``` | ||
| ============ Serving Benchmark Result ============ | ||
| Successful requests: 2048 | ||
| Failed requests: 0 | ||
| Benchmark duration (s): 389.73 | ||
| Total input tokens: 16777216 | ||
| Total generated tokens: 2097152 | ||
| Request throughput (req/s): 5.25 | ||
| Output token throughput (tok/s): 5380.99 | ||
| Peak output token throughput (tok/s): 13816.00 | ||
| Peak concurrent requests: 2048.00 | ||
| Total Token throughput (tok/s): 48428.95 | ||
| ---------------Time to First Token---------------- | ||
| Mean TTFT (ms): 173417.51 | ||
| Median TTFT (ms): 174208.15 | ||
| P99 TTFT (ms): 341178.90 | ||
| -----Time per Output Token (excl. 1st token)------ | ||
| Mean TPOT (ms): 157.86 | ||
| Median TPOT (ms): 155.73 | ||
| P99 TPOT (ms): 201.95 | ||
| ---------------Inter-token Latency---------------- | ||
| Mean ITL (ms): 157.87 | ||
| Median ITL (ms): 147.82 | ||
| P99 ITL (ms): 664.94 | ||
| ================================================== | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Step 7: Cleanup | ||
|
|
||
| When you are finished with the benchmark and no longer need the server, you can tear down the resources: | ||
|
|
||
| 1. Delete the vLLM server: | ||
|
|
||
| ```shell | ||
| kubectl delete -f deepseek_v4_flash-server.yaml | ||
| ``` | ||
|
|
||
| 2. (Optional) Delete the GKE TPU node pool: | ||
|
|
||
| ```shell | ||
| gcloud container node-pools delete ${NODEPOOL_NAME} \ | ||
| --cluster=${CLUSTER_NAME} \ | ||
| --location=${REGION} \ | ||
| --project=${PROJECT_ID} | ||
| ``` |
146 changes: 146 additions & 0 deletions
146
inference/ironwood/vLLM/vLLM-TorchTPU/DeepSeek-V4-Flash/deepseek_v4_flash-server.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,146 @@ | ||
| apiVersion: storage.k8s.io/v1 | ||
| kind: StorageClass | ||
| metadata: | ||
| name: hyperdisk-balanced-tpu | ||
| provisioner: pd.csi.storage.gke.io | ||
| parameters: | ||
| type: hyperdisk-balanced | ||
| reclaimPolicy: Delete | ||
| volumeBindingMode: WaitForFirstConsumer | ||
| allowVolumeExpansion: true | ||
| --- | ||
| apiVersion: v1 | ||
| kind: PersistentVolumeClaim | ||
| metadata: | ||
| name: hf-cache-pvc | ||
| namespace: vllm-deepseek-v4-flash | ||
| spec: | ||
| accessModes: | ||
| - ReadWriteOnce | ||
| resources: | ||
| requests: | ||
| storage: 500Gi | ||
| storageClassName: hyperdisk-balanced-tpu | ||
| --- | ||
| apiVersion: apps/v1 | ||
| kind: Deployment | ||
| metadata: | ||
| name: vllm-deepseek-v4-flash | ||
| namespace: vllm-deepseek-v4-flash | ||
| spec: | ||
| replicas: 1 | ||
| selector: | ||
| matchLabels: | ||
| app: vllm-deepseek-v4-flash | ||
| template: | ||
| metadata: | ||
| labels: | ||
| app: vllm-deepseek-v4-flash | ||
| spec: | ||
| nodeSelector: | ||
| cloud.google.com/gke-tpu-accelerator: tpu7x | ||
| cloud.google.com/gke-tpu-topology: 2x2x1 | ||
| tolerations: | ||
| - key: "google.com/tpu" | ||
| operator: "Exists" | ||
| effect: "NoSchedule" | ||
| containers: | ||
| - name: vllm-torchtpu | ||
| image: vllm/vllm-torchtpu:v0.29.0 | ||
| imagePullPolicy: Always | ||
| command: ["/bin/bash", "-c"] | ||
| args: | ||
| - | | ||
| set -e | ||
| mkdir -p "${HF_HOME}" "${TMPDIR}" "${VLLM_CACHE_ROOT}" "${TORCHINDUCTOR_CACHE_DIR}" | ||
|
|
||
| vllm serve deepseek-ai/DeepSeek-V4-Flash \ | ||
| --served-model-name=deepseek-ai/DeepSeek-V4-Flash \ | ||
| --host=0.0.0.0 \ | ||
| --port=8000 \ | ||
| --download-dir=/data \ | ||
| --tensor-parallel-size=1 \ | ||
| --data-parallel-size=8 \ | ||
| --enable-expert-parallel \ | ||
| --quantization=deepseek_v4_fp8 \ | ||
| --kv-cache-dtype=fp8_e4m3 \ | ||
| --trust-remote-code \ | ||
| --max-model-len=9216 \ | ||
| --max-num-batched-tokens=1024 \ | ||
| --max-num-seqs=256 \ | ||
| --no-enable-prefix-caching \ | ||
| --gpu-memory-utilization=0.9 \ | ||
| --async-scheduling \ | ||
| --generation-config="vllm" \ | ||
| --override-generation-config='{"do_sample": false, "temperature": 0.0, "model": "deepseek-ai/DeepSeek-V4-Flash"}' | ||
| env: | ||
| - name: HF_HOME | ||
| value: "/data/huggingface_cache" | ||
| - name: TMPDIR | ||
| value: "/data/tmp" | ||
| - name: VLLM_CACHE_ROOT | ||
| value: "/data/vllm_cache" | ||
| - name: TORCHINDUCTOR_CACHE_DIR | ||
| value: "/data/torchinductor_cache" | ||
| - name: HUGGING_FACE_HUB_TOKEN | ||
| valueFrom: | ||
| secretKeyRef: | ||
| name: hf-secret | ||
| key: hf_api_token | ||
| - name: MOE_REQUANTIZE_WEIGHT_DTYPE | ||
| value: "fp4" | ||
| - name: MOE_REQUANTIZE_BLOCK_SIZE | ||
| value: "512" | ||
| - name: TPU_ROPE_CACHE_ROW_MAJOR | ||
| value: "1" | ||
| - name: TPU_MOE_HASH_TABLE_ROW_MAJOR | ||
| value: "1" | ||
| - name: TORCH_DIST_TIMEOUT | ||
| value: "1200" | ||
| - name: VLLM_RPC_TIMEOUT | ||
| value: "1200000" | ||
| - name: VLLM_SHM_BROADCAST_TIMEOUT_S | ||
| value: "1200" | ||
| - name: VLLM_ENGINE_ITERATION_TIMEOUT_S | ||
| value: "1200" | ||
| - name: VLLM_ENGINE_READY_TIMEOUT_S | ||
| value: "3600" | ||
| ports: | ||
| - containerPort: 8000 | ||
| resources: | ||
| limits: | ||
| google.com/tpu: "4" | ||
| requests: | ||
| google.com/tpu: "4" | ||
| readinessProbe: | ||
| tcpSocket: | ||
| port: 8000 | ||
| initialDelaySeconds: 15 | ||
| periodSeconds: 10 | ||
|
inucool12 marked this conversation as resolved.
Resolved
|
||
| volumeMounts: | ||
| - name: hf-cache | ||
| mountPath: /data | ||
| - name: dshm | ||
| mountPath: /dev/shm | ||
| volumes: | ||
| - name: dshm | ||
| emptyDir: | ||
| medium: Memory | ||
| - name: hf-cache | ||
| persistentVolumeClaim: | ||
| claimName: hf-cache-pvc | ||
| --- | ||
| apiVersion: v1 | ||
| kind: Service | ||
| metadata: | ||
| name: vllm-deepseek-v4-flash-service | ||
| namespace: vllm-deepseek-v4-flash | ||
| spec: | ||
| selector: | ||
| app: vllm-deepseek-v4-flash | ||
| type: ClusterIP | ||
| ports: | ||
| - name: http | ||
| protocol: TCP | ||
| port: 8000 | ||
| targetPort: 8000 | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.