Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
apiVersion: v2
name: tpu7x-benchmark
description: tpu7x-benchmark
type: application
version: 0.1.0
appVersion: "1.16.0"
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@

# Run vLLM (TorchTPU) torchtpu_vllm/ts_repro_deepseek_r1_0528_tp16_c64_v029 on tpu7x on GKE

This recipe covers running a vLLM inference workload on tpu7x on
GKE.

This recipe uses the **TorchTPU** vLLM backend. The server container is named `vllm-torchtpu` on single-host recipes and `vllm-server` on multi-host (LeaderWorkerSet) recipes; use the label selectors below to view its logs regardless of topology.

## Create the GKE Cluster
Create your tpu7x cluster using [XPK](https://github.com/AI-Hypercomputer/xpk).
The next sections assume you have created a cluster with tpu7x
nodes.


## Deploy vLLM Workload on GKE

### Configure kubectl to communicate with your cluster

```
gcloud container clusters get-credentials ${CLUSTER_NAME} --location=${LOCATION}
```

### Generate a new Hugging Face token if you don't already have one

On the huggingface website, create an account if necessary, and go to Your
Profile > Settings > Access Tokens.
Select Create new token.
Specify a name of your choice and a role with at least Read permissions.
Select Generate a token, follow the prompts and save your token.

(NOTE: Also ensure that your account has access to the model on Hugging Face.
For example, for llama3, you will need to explicitly get permission):

### Run the benchmark
Note: Please search and replace `<YOUR_BUCKET_NAME>` in `values.yaml` with your actual Google Cloud Storage bucket name to store logs and cache.

In this directory, run:

```
helm install ${RUN_NAME} . --set hf_token=${HF_TOKEN}
```

The benchmark will launch a client and a server pod. Every object in the recipe
is labelled with `app.kubernetes.io/instance=${RUN_NAME}`, and the pods are
additionally labelled `app.kubernetes.io/component=server` or
`app.kubernetes.io/component=client`. To see what was created:

```
$ kubectl get pods -l app.kubernetes.io/instance=${RUN_NAME}
```

On the server pod, at the end of the server startup you'll see logs such as:

```
$ kubectl logs -f --prefix --all-containers --max-log-requests=100 -l app.kubernetes.io/instance=${RUN_NAME},app.kubernetes.io/component=server

(APIServer pid=1) INFO: Started server process [1]
(APIServer pid=1) INFO: Waiting for application startup.
(APIServer pid=1) INFO: Application startup complete.
```

The client pod will wait until the server is up and then start the benchmark.
On the client pod, you'll see logs such as:

```
$ kubectl logs -f --prefix --all-containers -l app.kubernetes.io/instance=${RUN_NAME},app.kubernetes.io/component=client

============ Serving Benchmark Result ============
Successful requests: 10
Failed requests: 0
Benchmark duration (s): xx
Total input tokens: xxx
Total generated tokens: xxx
Request throughput (req/s): xx
Output token throughput (tok/s): xxx
Peak output token throughput (tok/s): xxx
Peak concurrent requests: 10.00
Total Token throughput (tok/s): xxx
---------------Time to First Token----------------
Mean TTFT (ms): xxx
Median TTFT (ms): xxx
P99 TTFT (ms): xxx
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): xxx
Median TPOT (ms): xxx
P99 TPOT (ms): xxx
---------------Inter-token Latency----------------
Mean ITL (ms): xxx
Median ITL (ms): xxx
P99 ITL (ms): xxx
==================================================
```


### Customizing the benchmark

In order to change the parameters of the benchmark, such as the model, number of
prompts, etc, you can modify the settings in values.yaml.

### Cleanup
When you are done running the benchmark, the server will still be running. You
can cleanup by running:

```
helm uninstall ${RUN_NAME}
```

Helm does not delete volumes provisioned through `volumeClaimTemplates`, so on
multi-host recipes some PersistentVolumeClaims outlive the release and keep
consuming disk quota. Reap any leftovers with:

```
kubectl delete pvc -l app.kubernetes.io/instance=${RUN_NAME}
```
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
{{/*
Helpers for the serving pods' /data Hyperdisk volume.

The claim MUST be declared at the LeaderWorkerSet controller level
(`leaderWorkerTemplate.volumeClaimTemplates`) and NEVER as an inline pod
Generic Ephemeral Volume (`ephemeral.volumeClaimTemplate`) inside
`leaderTemplate.spec.volumes` / `workerTemplate.spec.volumes`.

Why (b/565473329): Kueue copies the pod template into the
ProvisioningRequest PodTemplate, and GKE Cluster Autoscaler evaluates the
VolumeBinding predicate against *virtual* pods that never exist in the API
server. ephemeral-volume-controller therefore never creates the
`<virtual-pod>-data-volume` PVC those pods reference, VolumeBinding fails on
every scale-up cycle, and the LeaderWorkerSet stays SchedulingGated until the
run times out.

Declaring the claim on the controller instead lets the pod templates carry a
plain `emptyDir` placeholder, which Cluster Autoscaler can schedule, while
the LWS/StatefulSet controller substitutes the real PVC on the pods it
actually creates ("a claim in this list takes precedence over any volumes in
the template, with the same name").
*/}}

{{/*
Controller-level claim template for /data.
Args (dict): releaseName (string), size (quantity, e.g. "100Gi").
*/}}
{{- define "tpu7x.dataVolumeClaimTemplates" -}}
volumeClaimTemplates:
- metadata:
name: data-volume
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "{{ .releaseName }}-hyperdisk-balanced-tpu"
resources:
requests:
storage: {{ .size }}
# This is what actually reclaims the Hyperdisks (b/539580630).
#
# helm uninstall does not delete volumeClaimTemplate PVCs, and
# helm_recipe_runner._delete_release_pvcs cannot reap these either: the LWS
# controller strips metadata.labels when it copies volumeClaimTemplates into
# the underlying StatefulSets, so the app.kubernetes.io/instance selector the
# runner uses never matches. Verified on bodaborg-tpu7x-nap: the LWS requested
# {labels: {app.kubernetes.io/instance: ...}} and the StatefulSet received
# {name: data-volume, namespace: ...} only.
#
# The retention policy does not rely on labels. It makes the StatefulSet the
# PVC's owner (blockOwnerDeletion=true), so Kubernetes garbage-collects the
# PVC, and the Hyperdisk behind it, when the StatefulSet goes away.
persistentVolumeClaimRetentionPolicy:
whenDeleted: Delete
whenScaled: Delete
{{- end -}}

{{/*
Pod-level placeholder for /data. Substituted by the real PVC whenever
"tpu7x.dataVolumeClaimTemplates" is emitted; used as-is (tmpfs) otherwise.
Takes no arguments.
*/}}
{{- define "tpu7x.dataVolumePlaceholder" -}}
# Placeholder only. Do NOT replace this with an inline
# `ephemeral.volumeClaimTemplate`: see "tpu7x.dataVolumeClaimTemplates" above
# and b/565473329.
- emptyDir:
medium: Memory
name: data-volume
{{- end -}}
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
{{/*
Pod-template annotations for multi-host (LeaderWorkerSet) serving pods.

Args (dict): ctx (the chart root context, i.e. `$` / `.`).

Emits nothing when no annotation applies, so callers must guard the
`annotations:` key on a non-empty result.

kueue.x-k8s.io/podset-required-topology:
The LWS `exclusive-topology` annotation pins every worker to its leader's
node pool, but Kueue Topology-Aware Scheduling (TAS) independently stamps a
`kubernetes.io/hostname` nodeSelector on each pod of the group. With an
unconstrained TAS request, Kueue may pick hosts from two different TPU
slices (node pools); the worker then ends up selecting a hostname outside
its own pool and stays Pending forever (seen on bodaborg-tpu7x-nap whenever
two 2x2x2 pools are provisioned together, e.g. P/D disaggregation).
Requiring the node-pool topology level makes Kueue place the whole group
inside one pool, consistent with `exclusive-topology`.

Opt-in via `kueue_required_topology` (generator_specific_configs) because
it is only valid when the ClusterQueue's ResourceFlavor is TAS-enabled and
its Topology defines that level; on a non-TAS flavor Kueue would never
admit the workload.
*/}}
{{- define "tpu7x.lws.podAnnotations" -}}
{{- $ctx := .ctx -}}
{{- if $ctx.Values.gcsfuse.enabled }}
gke-gcsfuse/volumes: "true"
{{- end }}
{{- if and $ctx.Values.kueue_local_queue $ctx.Values.kueue_required_topology }}
kueue.x-k8s.io/podset-required-topology: {{ $ctx.Values.kueue_required_topology | quote }}
{{- end }}
{{- end -}}
Loading
Loading