diff --git a/inference/ironwood/vLLM/vLLM-TorchTPU/DeepSeek-V4-Flash/README.md b/inference/ironwood/vLLM/vLLM-TorchTPU/DeepSeek-V4-Flash/README.md new file mode 100644 index 00000000..53657874 --- /dev/null +++ b/inference/ironwood/vLLM/vLLM-TorchTPU/DeepSeek-V4-Flash/README.md @@ -0,0 +1,252 @@ +# Serve DeepSeek-V4-Flash with vLLM (TorchTPU) on Ironwood TPU + +In this guide, we show how to serve DeepSeek-V4-Flash (`deepseek-ai/DeepSeek-V4-Flash`, Mixture-of-Experts model with Multi-head Latent Attention) with vLLM using the **TorchTPU** backend on Ironwood (TPU v7x) using GKE. + +We deploy it on a single **tpu7x-standard-4t** node pool (`2x2x1` topology, 4 TPU v7x chips, 8 TensorCores) using **Data Parallelism (DP) size of 8**, **Expert Parallelism (EP) size of 8**, and **Tensor Parallelism (TP) size of 1**, with `deepseek_v4_fp8` quantization, load-time FP4 MoE weight requantization (`W4A8`), and `fp8_e4m3` KV cache. + +## Verified Models + +The following DeepSeek-V4-Flash model is verified for deployment on TPU with the vLLM TorchTPU backend. + +| Model | Quantization | Min TPUs (Chips) | Topology | HuggingFace | +| :---- | :---- | :---- | :---- | :---- | +| DeepSeek-V4-Flash | `deepseek_v4_fp8` (W4A8 MoE), `fp8_e4m3` KV | 4× (`tpu7x-standard-4t`) | `2x2x1` (1 host) | [deepseek-ai/DeepSeek-V4-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) | + +## Prerequisites + +1. A GKE cluster with TPU v7x support (GKE version `1.34.0-gke.2201000` or later). +2. GCloud CLI and Kubectl installed and configured. +3. A Hugging Face account and access token with access to `deepseek-ai/DeepSeek-V4-Flash`. + +--- + +## Step 1: Define Parameters + +Export the necessary environment variables for your Google Cloud project and GKE cluster: + +```shell +export PROJECT_ID="" +export CLUSTER_NAME="" +export REGION="" +export ZONE="" +export NODEPOOL_NAME="dsv4-flash-tpu-pool" +``` + +--- + +## Step 2: Create the TPU Node Pool + +Create a TPU v7x (Ironwood) node pool with the `tpu7x-standard-4t` machine type. This machine type provides 4 chips (8 logical TensorCore devices): + +```shell +gcloud container node-pools create ${NODEPOOL_NAME} \ + --project=${PROJECT_ID} \ + --location=${REGION} \ + --node-locations=${ZONE} \ + --num-nodes=1 \ + --machine-type=tpu7x-standard-4t \ + --cluster=${CLUSTER_NAME} +``` + +--- + +## Step 3: Setup Kubernetes Configurations + +1. Configure `kubectl` to communicate with your cluster: + +```shell +gcloud container clusters get-credentials ${CLUSTER_NAME} --location=${REGION} --project=${PROJECT_ID} +``` + +2. Create the Namespace `vllm-deepseek-v4-flash` if you want to run it in an isolated namespace (recommended): + +```shell +kubectl create namespace vllm-deepseek-v4-flash +``` + +3. Create a Kubernetes secret for your Hugging Face token in the namespace: + +```shell +export HF_TOKEN="YOUR_HUGGING_FACE_API_TOKEN" +kubectl create secret generic hf-secret \ + --namespace=vllm-deepseek-v4-flash \ + --from-literal=hf_api_token=${HF_TOKEN} +``` + +--- + +## Step 4: Apply the vLLM Server Manifest + +Apply the vLLM server manifest using the provided [deepseek_v4_flash-server.yaml](./deepseek_v4_flash-server.yaml) file in this directory: + +```shell +cd inference/ironwood/vLLM/vLLM-TorchTPU/DeepSeek-V4-Flash +kubectl apply -f deepseek_v4_flash-server.yaml +``` + +Monitor the progress of the server deployment: + +```shell +kubectl get pods -n vllm-deepseek-v4-flash -w +``` + +You can check the server logs using: + +```shell +kubectl logs -n vllm-deepseek-v4-flash deployment/vllm-deepseek-v4-flash -c vllm-torchtpu -f +``` + +Once the server has successfully loaded and compiled the model, you should see logs like: + +``` +(APIServer pid=1) INFO: Started server process [1] +(APIServer pid=1) INFO: Waiting for application startup. +(APIServer pid=1) INFO: Application startup complete. +``` + +--- + +## Step 5: Interact with the Model (Optional) + +You can test the server using a port forward and `curl`: + +1. Port forward the vLLM service to port `8000` on your local machine: + +```shell +kubectl port-forward -n vllm-deepseek-v4-flash service/vllm-deepseek-v4-flash-service 8000:8000 +``` + +2. Run a `curl` request: + +```shell +curl http://localhost:8000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "deepseek-ai/DeepSeek-V4-Flash", + "messages": [ + {"role": "user", "content": "Hello! Tell me a joke."} + ], + "max_tokens": 100, + "temperature": 0.0 + }' +``` + +--- + +## Step 6: Run Serving Benchmarks + +We use `vllm bench serve` to evaluate serving throughput and latency against our running vLLM server for an **8K Input / 1K Output (8k/1k)** workload. + +### Workload: 8K Input / 1K Output (8k/1k) + +Save the following manifest as `deepseek_v4_flash-benchmark-8k1k.yaml`: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: dsv4-flash-bench-8k1k + namespace: vllm-deepseek-v4-flash +spec: + restartPolicy: Never + terminationGracePeriodSeconds: 60 + containers: + - name: vllm-bench + image: vllm/vllm-torchtpu:v0.29.0 + command: ["/bin/bash", "-c"] + args: + - | + while ! curl -sf http://vllm-deepseek-v4-flash-service:8000/health; do sleep 30 && echo 'Waiting for server...'; done + vllm bench serve \ + --backend=vllm \ + --base-url=http://vllm-deepseek-v4-flash-service:8000 \ + --model=deepseek-ai/DeepSeek-V4-Flash \ + --tokenizer=deepseek-ai/DeepSeek-V4-Flash \ + --trust-remote-code \ + --dataset-name=random \ + --random-input-len=8192 \ + --random-output-len=1024 \ + --num-prompts=2048 \ + --ignore-eos \ + --skip-chat-template + env: + - name: PYTHONUNBUFFERED + value: "1" + - name: HUGGING_FACE_HUB_TOKEN + valueFrom: + secretKeyRef: + key: hf_api_token + name: hf-secret +``` + +Apply it using `kubectl`: + +```shell +kubectl apply -f deepseek_v4_flash-benchmark-8k1k.yaml +``` + +Check the logs of the benchmark run: + +```shell +kubectl logs -n vllm-deepseek-v4-flash dsv4-flash-bench-8k1k -f +``` + +When complete, clean up the benchmark pod: + +```shell +kubectl delete -f deepseek_v4_flash-benchmark-8k1k.yaml +``` + +--- + +### Example Benchmark Output + +The benchmark client will print a table resembling the following: + +``` +============ Serving Benchmark Result ============ +Successful requests: 2048 +Failed requests: 0 +Benchmark duration (s): 389.73 +Total input tokens: 16777216 +Total generated tokens: 2097152 +Request throughput (req/s): 5.25 +Output token throughput (tok/s): 5380.99 +Peak output token throughput (tok/s): 13816.00 +Peak concurrent requests: 2048.00 +Total Token throughput (tok/s): 48428.95 +---------------Time to First Token---------------- +Mean TTFT (ms): 173417.51 +Median TTFT (ms): 174208.15 +P99 TTFT (ms): 341178.90 +-----Time per Output Token (excl. 1st token)------ +Mean TPOT (ms): 157.86 +Median TPOT (ms): 155.73 +P99 TPOT (ms): 201.95 +---------------Inter-token Latency---------------- +Mean ITL (ms): 157.87 +Median ITL (ms): 147.82 +P99 ITL (ms): 664.94 +================================================== +``` + +--- + +## Step 7: Cleanup + +When you are finished with the benchmark and no longer need the server, you can tear down the resources: + +1. Delete the vLLM server: + +```shell +kubectl delete -f deepseek_v4_flash-server.yaml +``` + +2. (Optional) Delete the GKE TPU node pool: + +```shell +gcloud container node-pools delete ${NODEPOOL_NAME} \ + --cluster=${CLUSTER_NAME} \ + --location=${REGION} \ + --project=${PROJECT_ID} +``` diff --git a/inference/ironwood/vLLM/vLLM-TorchTPU/DeepSeek-V4-Flash/deepseek_v4_flash-server.yaml b/inference/ironwood/vLLM/vLLM-TorchTPU/DeepSeek-V4-Flash/deepseek_v4_flash-server.yaml new file mode 100644 index 00000000..eb53ec81 --- /dev/null +++ b/inference/ironwood/vLLM/vLLM-TorchTPU/DeepSeek-V4-Flash/deepseek_v4_flash-server.yaml @@ -0,0 +1,146 @@ +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: hyperdisk-balanced-tpu +provisioner: pd.csi.storage.gke.io +parameters: + type: hyperdisk-balanced +reclaimPolicy: Delete +volumeBindingMode: WaitForFirstConsumer +allowVolumeExpansion: true +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: hf-cache-pvc + namespace: vllm-deepseek-v4-flash +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 500Gi + storageClassName: hyperdisk-balanced-tpu +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: vllm-deepseek-v4-flash + namespace: vllm-deepseek-v4-flash +spec: + replicas: 1 + selector: + matchLabels: + app: vllm-deepseek-v4-flash + template: + metadata: + labels: + app: vllm-deepseek-v4-flash + spec: + nodeSelector: + cloud.google.com/gke-tpu-accelerator: tpu7x + cloud.google.com/gke-tpu-topology: 2x2x1 + tolerations: + - key: "google.com/tpu" + operator: "Exists" + effect: "NoSchedule" + containers: + - name: vllm-torchtpu + image: vllm/vllm-torchtpu:v0.29.0 + imagePullPolicy: Always + command: ["/bin/bash", "-c"] + args: + - | + set -e + mkdir -p "${HF_HOME}" "${TMPDIR}" "${VLLM_CACHE_ROOT}" "${TORCHINDUCTOR_CACHE_DIR}" + + vllm serve deepseek-ai/DeepSeek-V4-Flash \ + --served-model-name=deepseek-ai/DeepSeek-V4-Flash \ + --host=0.0.0.0 \ + --port=8000 \ + --download-dir=/data \ + --tensor-parallel-size=1 \ + --data-parallel-size=8 \ + --enable-expert-parallel \ + --quantization=deepseek_v4_fp8 \ + --kv-cache-dtype=fp8_e4m3 \ + --trust-remote-code \ + --max-model-len=9216 \ + --max-num-batched-tokens=1024 \ + --max-num-seqs=256 \ + --no-enable-prefix-caching \ + --gpu-memory-utilization=0.9 \ + --async-scheduling \ + --generation-config="vllm" \ + --override-generation-config='{"do_sample": false, "temperature": 0.0, "model": "deepseek-ai/DeepSeek-V4-Flash"}' + env: + - name: HF_HOME + value: "/data/huggingface_cache" + - name: TMPDIR + value: "/data/tmp" + - name: VLLM_CACHE_ROOT + value: "/data/vllm_cache" + - name: TORCHINDUCTOR_CACHE_DIR + value: "/data/torchinductor_cache" + - name: HUGGING_FACE_HUB_TOKEN + valueFrom: + secretKeyRef: + name: hf-secret + key: hf_api_token + - name: MOE_REQUANTIZE_WEIGHT_DTYPE + value: "fp4" + - name: MOE_REQUANTIZE_BLOCK_SIZE + value: "512" + - name: TPU_ROPE_CACHE_ROW_MAJOR + value: "1" + - name: TPU_MOE_HASH_TABLE_ROW_MAJOR + value: "1" + - name: TORCH_DIST_TIMEOUT + value: "1200" + - name: VLLM_RPC_TIMEOUT + value: "1200000" + - name: VLLM_SHM_BROADCAST_TIMEOUT_S + value: "1200" + - name: VLLM_ENGINE_ITERATION_TIMEOUT_S + value: "1200" + - name: VLLM_ENGINE_READY_TIMEOUT_S + value: "3600" + ports: + - containerPort: 8000 + resources: + limits: + google.com/tpu: "4" + requests: + google.com/tpu: "4" + readinessProbe: + tcpSocket: + port: 8000 + initialDelaySeconds: 15 + periodSeconds: 10 + volumeMounts: + - name: hf-cache + mountPath: /data + - name: dshm + mountPath: /dev/shm + volumes: + - name: dshm + emptyDir: + medium: Memory + - name: hf-cache + persistentVolumeClaim: + claimName: hf-cache-pvc +--- +apiVersion: v1 +kind: Service +metadata: + name: vllm-deepseek-v4-flash-service + namespace: vllm-deepseek-v4-flash +spec: + selector: + app: vllm-deepseek-v4-flash + type: ClusterIP + ports: + - name: http + protocol: TCP + port: 8000 + targetPort: 8000 diff --git a/inference/ironwood/vLLM/vLLM-TorchTPU/DeepSeek-V4-Pro/README.md b/inference/ironwood/vLLM/vLLM-TorchTPU/DeepSeek-V4-Pro/README.md new file mode 100644 index 00000000..81e17302 --- /dev/null +++ b/inference/ironwood/vLLM/vLLM-TorchTPU/DeepSeek-V4-Pro/README.md @@ -0,0 +1,257 @@ +# Serve DeepSeek-V4-Pro with vLLM (TorchTPU) on Ironwood TPU (Multi-Host) + +In this guide, we show how to serve DeepSeek-V4-Pro (`deepseek-ai/DeepSeek-V4-Pro-0813`, Mixture-of-Experts model with Multi-head Latent Attention) with vLLM using the **TorchTPU** backend on Ironwood (TPU v7x) using GKE. + +We deploy it across a multi-host **tpu7x-standard-4t** node pool (`2x2x2` topology, 2 hosts, 8 TPU v7x chips, 16 TensorCores) using a Kubernetes **LeaderWorkerSet** and **Ray**, with **Data Parallelism (DP) size of 16**, **Expert Parallelism (EP) size of 16**, and **Tensor Parallelism (TP) size of 1**, using `deepseek_v4_fp8` quantization, fused EP MoE v2 with load-time FP4 MoE weight requantization (`W4A8`), and `fp8_e4m3` KV cache. + +## Verified Models + +The following DeepSeek-V4-Pro model is verified for deployment on TPU with the vLLM TorchTPU backend. + +| Model | Quantization | Min TPUs (Chips) | Topology | HuggingFace | +| :--- | :--- | :--- | :--- | :--- | +| DeepSeek-V4-Pro-0813 | `deepseek_v4_fp8` (W4A8 MoE), `fp8_e4m3` KV | 8× (`tpu7x-standard-4t`) | `2x2x2` (2 hosts) | [deepseek-ai/DeepSeek-V4-Pro-0813](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813) | + +## Prerequisites + +1. A GKE cluster with TPU v7x support (GKE version `1.34.0-gke.2201000` or later) and the LeaderWorkerSet (LWS) API enabled. +2. GCloud CLI and Kubectl installed and configured. +3. A Hugging Face account and access token with access to `deepseek-ai/DeepSeek-V4-Pro-0813`. + +--- + +## Step 1: Define Parameters + +Export the necessary environment variables for your Google Cloud project and GKE cluster: + +```shell +export PROJECT_ID="" +export CLUSTER_NAME="" +export REGION="" +export ZONE="" +export NODEPOOL_NAME="dsv4-pro-tpu-pool" +``` + +--- + +## Step 2: Create the Multi-Host TPU Node Pool + +Create a multi-host TPU v7x (Ironwood) node pool with the `tpu7x-standard-4t` machine type and a `2x2x2` topology. This creates 2 nodes (hosts) with 4 chips each, for a total of 8 chips (16 logical TensorCore devices): + +```shell +gcloud container node-pools create ${NODEPOOL_NAME} \ + --project=${PROJECT_ID} \ + --location=${REGION} \ + --node-locations=${ZONE} \ + --num-nodes=2 \ + --machine-type=tpu7x-standard-4t \ + --tpu-topology=2x2x2 \ + --cluster=${CLUSTER_NAME} +``` + +--- + +## Step 3: Setup Kubernetes Configurations + +1. Configure `kubectl` to communicate with your cluster: + +```shell +gcloud container clusters get-credentials ${CLUSTER_NAME} --location=${REGION} --project=${PROJECT_ID} +``` + +2. Create the Namespace `vllm-deepseek-v4-pro` if you want to run it in an isolated namespace (recommended): + +```shell +kubectl create namespace vllm-deepseek-v4-pro +``` + +3. Create a Kubernetes secret for your Hugging Face token in the namespace: + +```shell +export HF_TOKEN="YOUR_HUGGING_FACE_API_TOKEN" +kubectl create secret generic hf-secret \ + --namespace=vllm-deepseek-v4-pro \ + --from-literal=hf_api_token=${HF_TOKEN} +``` + +--- + +## Step 4: Apply the vLLM Server Manifest + +Apply the vLLM server manifest using the provided [deepseek_v4_pro-server.yaml](./deepseek_v4_pro-server.yaml) file in this directory: + +```shell +cd inference/ironwood/vLLM/vLLM-TorchTPU/DeepSeek-V4-Pro +kubectl apply -f deepseek_v4_pro-server.yaml +``` + +The manifest creates a `LeaderWorkerSet` with one leader pod and one worker pod, one on each TPU host. The leader starts a Ray head node, waits for the worker to join the Ray cluster (16 TPU devices total), and then launches `vllm serve` across both hosts. + +Monitor the progress of the server deployment: + +```shell +kubectl get pods -n vllm-deepseek-v4-pro -w +``` + +You can check the server logs on the leader pod using: + +```shell +kubectl logs -n vllm-deepseek-v4-pro vllm-deepseek-v4-pro-0 -c vllm-torchtpu -f +``` + +Once the server has successfully loaded and compiled the model, you should see logs like: + +``` +(APIServer pid=1) INFO: Started server process [1] +(APIServer pid=1) INFO: Waiting for application startup. +(APIServer pid=1) INFO: Application startup complete. +``` + +--- + +## Step 5: Interact with the Model (Optional) + +You can test the server using a port forward and `curl`: + +1. Port forward the vLLM service to port `8000` on your local machine: + +```shell +kubectl port-forward -n vllm-deepseek-v4-pro service/vllm-deepseek-v4-pro-service 8000:8000 +``` + +2. Run a `curl` request: + +```shell +curl http://localhost:8000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "deepseek-ai/DeepSeek-V4-Pro-0813", + "messages": [ + {"role": "user", "content": "Write a short poem that includes fruits like apples, bananas and pineapples."} + ], + "max_tokens": 200, + "temperature": 0.0 + }' +``` + +--- + +## Step 6: Run Serving Benchmarks + +We use `vllm bench serve` to evaluate serving throughput and latency against our running vLLM server for an **8K Input / 1K Output (8k/1k)** workload. + +### Workload: 8K Input / 1K Output (8k/1k) + +Save the following manifest as `deepseek_v4_pro-benchmark-8k1k.yaml`: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: dsv4-pro-bench-8k1k + namespace: vllm-deepseek-v4-pro +spec: + restartPolicy: Never + terminationGracePeriodSeconds: 60 + containers: + - name: vllm-bench + image: vllm/vllm-torchtpu:v0.29.0 + command: ["/bin/bash", "-c"] + args: + - | + while ! curl -sf http://vllm-deepseek-v4-pro-service:8000/health; do sleep 30 && echo 'Waiting for server...'; done + vllm bench serve \ + --backend=vllm \ + --base-url=http://vllm-deepseek-v4-pro-service:8000 \ + --model=deepseek-ai/DeepSeek-V4-Pro-0813 \ + --tokenizer=deepseek-ai/DeepSeek-V4-Pro-0813 \ + --trust-remote-code \ + --dataset-name=random \ + --random-input-len=8192 \ + --random-output-len=1024 \ + --num-prompts=2048 \ + --random-range-ratio=0 \ + --ignore-eos \ + --max-concurrency=1024 + env: + - name: PYTHONUNBUFFERED + value: "1" + - name: HUGGING_FACE_HUB_TOKEN + valueFrom: + secretKeyRef: + key: hf_api_token + name: hf-secret +``` + +Apply it using `kubectl`: + +```shell +kubectl apply -f deepseek_v4_pro-benchmark-8k1k.yaml +``` + +Check the logs of the benchmark run: + +```shell +kubectl logs -n vllm-deepseek-v4-pro dsv4-pro-bench-8k1k -f +``` + +When complete, clean up the benchmark pod: + +```shell +kubectl delete -f deepseek_v4_pro-benchmark-8k1k.yaml +``` + +--- + +### Example Benchmark Output + +The benchmark client will print a table resembling the following: + +``` +============ Serving Benchmark Result ============ +Successful requests: 2048 +Failed requests: 0 +Maximum request concurrency: 1024 +Benchmark duration (s): xxx +Total input tokens: 16777216 +Total generated tokens: 2097152 +Request throughput (req/s): xx +Output token throughput (tok/s): xxx +Peak output token throughput (tok/s): xxx +Peak concurrent requests: 1024.00 +Total Token throughput (tok/s): xxx +---------------Time to First Token---------------- +Mean TTFT (ms): xxx +Median TTFT (ms): xxx +P99 TTFT (ms): xxx +-----Time per Output Token (excl. 1st token)------ +Mean TPOT (ms): xxx +Median TPOT (ms): xxx +P99 TPOT (ms): xxx +---------------Inter-token Latency---------------- +Mean ITL (ms): xxx +Median ITL (ms): xxx +P99 ITL (ms): xxx +================================================== +``` + +--- + +## Step 7: Cleanup + +When you are finished with the benchmark and no longer need the server, you can tear down the resources: + +1. Delete the vLLM server: + +```shell +kubectl delete -f deepseek_v4_pro-server.yaml +``` + +2. (Optional) Delete the GKE TPU node pool: + +```shell +gcloud container node-pools delete ${NODEPOOL_NAME} \ + --cluster=${CLUSTER_NAME} \ + --location=${REGION} \ + --project=${PROJECT_ID} +``` diff --git a/inference/ironwood/vLLM/vLLM-TorchTPU/DeepSeek-V4-Pro/deepseek_v4_pro-server.yaml b/inference/ironwood/vLLM/vLLM-TorchTPU/DeepSeek-V4-Pro/deepseek_v4_pro-server.yaml new file mode 100644 index 00000000..00afbbe1 --- /dev/null +++ b/inference/ironwood/vLLM/vLLM-TorchTPU/DeepSeek-V4-Pro/deepseek_v4_pro-server.yaml @@ -0,0 +1,297 @@ +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: hyperdisk-balanced-tpu +provisioner: pd.csi.storage.gke.io +parameters: + type: hyperdisk-balanced +reclaimPolicy: Delete +volumeBindingMode: WaitForFirstConsumer +allowVolumeExpansion: true +--- +apiVersion: leaderworkerset.x-k8s.io/v1 +kind: LeaderWorkerSet +metadata: + name: vllm-deepseek-v4-pro + namespace: vllm-deepseek-v4-pro + annotations: + leaderworkerset.sigs.k8s.io/exclusive-topology: cloud.google.com/gke-nodepool +spec: + replicas: 1 + leaderWorkerTemplate: + size: 2 + restartPolicy: RecreateGroupOnPodRestart + leaderTemplate: + metadata: + labels: + role: leader + app: vllm-deepseek-v4-pro + spec: + nodeSelector: + cloud.google.com/gke-tpu-accelerator: tpu7x + cloud.google.com/gke-tpu-topology: 2x2x2 + tolerations: + - key: "google.com/tpu" + operator: "Exists" + effect: "NoSchedule" + containers: + - name: vllm-torchtpu + image: vllm/vllm-torchtpu:v0.29.0 + imagePullPolicy: Always + command: ["/bin/bash", "-c"] + args: + - | + set -e + mkdir -p "${HF_HOME}" "${TMPDIR}" "${VLLM_CACHE_ROOT}" "${TORCHINDUCTOR_CACHE_DIR}" + + # Start the Ray head node on the leader (LWS_WORKER_INDEX=0). + ray start --head --port=6379 + + # Wait until both hosts (leader + worker, 16 TPU devices total) have joined the Ray cluster. + until ray status 2>/dev/null | grep -q "16.0 TPU"; do + echo "Waiting for all TPU hosts to join the Ray cluster..." + sleep 5 + done + ray status + + vllm serve deepseek-ai/DeepSeek-V4-Pro-0813 \ + --served-model-name=deepseek-ai/DeepSeek-V4-Pro-0813 \ + --host=0.0.0.0 \ + --port=8000 \ + --download-dir=/data \ + --tensor-parallel-size=1 \ + --data-parallel-size=16 \ + --enable-expert-parallel \ + --enable-ep-weight-filter \ + --safetensors-load-strategy=prefetch \ + --quantization=deepseek_v4_fp8 \ + --kv-cache-dtype=fp8_e4m3 \ + --trust-remote-code \ + --max-model-len=9216 \ + --max-num-batched-tokens=1024 \ + --max-num-seqs=256 \ + --no-enable-prefix-caching \ + --gpu-memory-utilization=0.90 \ + --async-scheduling + env: + - name: HF_HOME + value: "/data/huggingface_cache" + - name: TMPDIR + value: "/data/tmp" + - name: VLLM_CACHE_ROOT + value: "/data/vllm_cache" + - name: TORCHINDUCTOR_CACHE_DIR + value: "/data/torchinductor_cache" + - name: HUGGING_FACE_HUB_TOKEN + valueFrom: + secretKeyRef: + name: hf-secret + key: hf_api_token + - name: TPU_MULTIHOST_BACKEND + value: "ray" + - name: RAY_ADDRESS + value: "127.0.0.1:6379" + - name: TPU_PROCESS_ADDRESSES + value: "" + - name: VLLM_DISABLE_COMPILE_CACHE + value: "1" + - name: VLLM_WORKER_MULTIPROC_METHOD + value: "spawn" + - name: TORCHINDUCTOR_AUTOGRAD_CACHE + value: "0" + - name: VLLM_ENGINE_READY_TIMEOUT_S + value: "10800" + - name: VLLM_TARGET_DEVICE + value: "tpu" + - name: PJRT_DEVICE + value: "TPU" + - name: MOE_REQUANTIZE_WEIGHT_DTYPE + value: "fp4" + - name: MOE_REQUANTIZE_BLOCK_SIZE + value: "512" + - name: TORCH_DIST_TIMEOUT + value: "1200" + - name: VLLM_RPC_TIMEOUT + value: "1200000" + - name: VLLM_SHM_BROADCAST_TIMEOUT_S + value: "1200" + - name: VLLM_ENGINE_ITERATION_TIMEOUT_S + value: "1200" + - name: TPU_ROPE_CACHE_ROW_MAJOR + value: "1" + - name: TPU_MOE_HASH_TABLE_ROW_MAJOR + value: "1" + - name: USE_MOE_FUSED_EP_KERNEL + value: "1" + - name: MOE_FUSED_EP_ENABLE_W4A8 + value: "1" + - name: MOE_FUSED_EP_V2_HEIGHT_RUNGS_OVERRIDE + value: "2" + - name: TPU_MOE_SKIP_PADDED_TOKENS + value: "1" + - name: RAGGED_GATHER_REDUCE_VERSION + value: "v3" + - name: LIBTPU_INIT_ARGS + value: "--xla_tpu_use_minor_sharding_for_major_trivial_input=true --xla_tpu_enable_sparse_core_collective_offload_reduce_scatter=false --xla_tpu_ars_combiner_threshold_in_bytes=0 --xla_tpu_enable_async_collective_merger=false --xla_enable_async_all_gather=true --xla_tpu_enable_sparse_core_collective_offload_all_gather=true --xla_tpu_sparse_core_all_gather_offload_min_size_in_bytes=262144" + - name: VLLM_RAY_EXTRA_ENV_VARS_TO_COPY + value: "MOE_REQUANTIZE_WEIGHT_DTYPE,MOE_REQUANTIZE_BLOCK_SIZE,TPU_ROPE_CACHE_ROW_MAJOR,TPU_MOE_HASH_TABLE_ROW_MAJOR,TORCH_DIST_TIMEOUT,USE_MOE_FUSED_EP_KERNEL,MOE_FUSED_EP_ENABLE_W4A8,MOE_FUSED_EP_V2_HEIGHT_RUNGS_OVERRIDE,TPU_MOE_SKIP_PADDED_TOKENS,RAGGED_GATHER_REDUCE_VERSION,LIBTPU_INIT_ARGS" + ports: + - containerPort: 8000 + resources: + limits: + google.com/tpu: "4" + requests: + google.com/tpu: "4" + readinessProbe: + tcpSocket: + port: 8000 + initialDelaySeconds: 15 + periodSeconds: 10 + volumeMounts: + - name: data-volume + mountPath: /data + - name: dshm + mountPath: /dev/shm + volumes: + - name: dshm + emptyDir: + medium: Memory + - name: data-volume + ephemeral: + volumeClaimTemplate: + spec: + accessModes: + - ReadWriteOnce + storageClassName: hyperdisk-balanced-tpu + resources: + requests: + storage: 1500Gi + workerTemplate: + metadata: + labels: + role: worker + app: vllm-deepseek-v4-pro + spec: + nodeSelector: + cloud.google.com/gke-tpu-accelerator: tpu7x + cloud.google.com/gke-tpu-topology: 2x2x2 + tolerations: + - key: "google.com/tpu" + operator: "Exists" + effect: "NoSchedule" + containers: + - name: vllm-torchtpu + image: vllm/vllm-torchtpu:v0.29.0 + imagePullPolicy: Always + command: ["/bin/bash", "-c"] + args: + - | + set -e + mkdir -p "${HF_HOME}" "${TMPDIR}" "${VLLM_CACHE_ROOT}" "${TORCHINDUCTOR_CACHE_DIR}" + + # Wait for the Ray head on the leader to be reachable. + until (echo > "/dev/tcp/${LWS_LEADER_ADDRESS}/6379") 2>/dev/null; do + echo "Waiting for Ray head at ${LWS_LEADER_ADDRESS}:6379..." + sleep 5 + done + + # Join the Ray cluster as a worker node (LWS_WORKER_INDEX!=0). + ray start --address="${LWS_LEADER_ADDRESS}:6379" --block + env: + - name: HF_HOME + value: "/data/huggingface_cache" + - name: TMPDIR + value: "/data/tmp" + - name: VLLM_CACHE_ROOT + value: "/data/vllm_cache" + - name: TORCHINDUCTOR_CACHE_DIR + value: "/data/torchinductor_cache" + - name: HUGGING_FACE_HUB_TOKEN + valueFrom: + secretKeyRef: + name: hf-secret + key: hf_api_token + - name: TPU_MULTIHOST_BACKEND + value: "ray" + - name: TPU_PROCESS_ADDRESSES + value: "" + - name: VLLM_DISABLE_COMPILE_CACHE + value: "1" + - name: VLLM_WORKER_MULTIPROC_METHOD + value: "spawn" + - name: TORCHINDUCTOR_AUTOGRAD_CACHE + value: "0" + - name: VLLM_ENGINE_READY_TIMEOUT_S + value: "10800" + - name: VLLM_TARGET_DEVICE + value: "tpu" + - name: PJRT_DEVICE + value: "TPU" + - name: MOE_REQUANTIZE_WEIGHT_DTYPE + value: "fp4" + - name: MOE_REQUANTIZE_BLOCK_SIZE + value: "512" + - name: TORCH_DIST_TIMEOUT + value: "1200" + - name: VLLM_RPC_TIMEOUT + value: "1200000" + - name: VLLM_SHM_BROADCAST_TIMEOUT_S + value: "1200" + - name: VLLM_ENGINE_ITERATION_TIMEOUT_S + value: "1200" + - name: TPU_ROPE_CACHE_ROW_MAJOR + value: "1" + - name: TPU_MOE_HASH_TABLE_ROW_MAJOR + value: "1" + - name: USE_MOE_FUSED_EP_KERNEL + value: "1" + - name: MOE_FUSED_EP_ENABLE_W4A8 + value: "1" + - name: MOE_FUSED_EP_V2_HEIGHT_RUNGS_OVERRIDE + value: "2" + - name: TPU_MOE_SKIP_PADDED_TOKENS + value: "1" + - name: RAGGED_GATHER_REDUCE_VERSION + value: "v3" + - name: LIBTPU_INIT_ARGS + value: "--xla_tpu_use_minor_sharding_for_major_trivial_input=true --xla_tpu_enable_sparse_core_collective_offload_reduce_scatter=false --xla_tpu_ars_combiner_threshold_in_bytes=0 --xla_tpu_enable_async_collective_merger=false --xla_enable_async_all_gather=true --xla_tpu_enable_sparse_core_collective_offload_all_gather=true --xla_tpu_sparse_core_all_gather_offload_min_size_in_bytes=262144" + resources: + limits: + google.com/tpu: "4" + requests: + google.com/tpu: "4" + volumeMounts: + - name: data-volume + mountPath: /data + - name: dshm + mountPath: /dev/shm + volumes: + - name: dshm + emptyDir: + medium: Memory + - name: data-volume + ephemeral: + volumeClaimTemplate: + spec: + accessModes: + - ReadWriteOnce + storageClassName: hyperdisk-balanced-tpu + resources: + requests: + storage: 1500Gi +--- +apiVersion: v1 +kind: Service +metadata: + name: vllm-deepseek-v4-pro-service + namespace: vllm-deepseek-v4-pro +spec: + selector: + app: vllm-deepseek-v4-pro + role: leader + type: ClusterIP + ports: + - name: http + protocol: TCP + port: 8000 + targetPort: 8000