Skip to content

[vLLM-TorchTPU] Add Kimi-K3 megakernel recipe on Ironwood (TPU v7x) - #388

Open
Ardiegon wants to merge 2 commits into
AI-Hypercomputer:mainfrom
Ardiegon:oskarb/kimi-k3-v7x-megakernel-recipe
Open

Ardiegon wants to merge 2 commits into
AI-Hypercomputer:mainfrom
Ardiegon:oskarb/kimi-k3-v7x-megakernel-recipe

Conversation

@Ardiegon

Copy link
Copy Markdown

Adds an Ironwood (TPU v7x) GKE recipe for serving moonshotai/Kimi-K3 in a disaggregated 1P1D configuration (2 × 2x2x4 = 32 TPU v7x chips, TP=32 per role) using the TorchTPU Pallas megakernel backend, Raiden KV transfer, vllm-router, and Inferact/Kimi-K3-DSpark speculative decoding.

Important

Do not merge until the Kimi-K3 megakernel branch (inferact/k3-megakernel, commit 30c9ba966bdf193c1786d29606608c62c8806808) is merged into vllm-project/vllm-torchtpu main.
Step 3 of README.md already targets main (./docker/build_image.sh -t ${IMAGE_URI} --target dev --vllm-rs) so no follow-up edit is needed once the upstream merge lands.

Building the Image Before main Merge (Reviewer Reproduction)

To test this recipe before inferact/k3-megakernel lands on main, build from commit 30c9ba966bdf193c1786d29606608c62c8806808 (requires gcloud read access to us-python.pkg.dev/ml-oss-artifacts-transient/torch-tpu-virtual-registry):

git clone https://github.com/vllm-project/vllm-torchtpu.git /tmp/vllm-torchtpu
cd /tmp/vllm-torchtpu
git checkout 30c9ba966bdf193c1786d29606608c62c8806808
git apply dockerfile.patch   # see collapsible patch below (backports vllm-rs, vllm-router, numactl from main)
mkdir -p vllm

TOKEN_FILE=$(mktemp)
gcloud auth print-access-token > "${TOKEN_FILE}"
DOCKER_BUILDKIT=1 docker build \
    --target dev \
    --build-arg TORCH_TPU_SOURCE=registry \
    --secret id=gcloud_token,src="${TOKEN_FILE}" \
    -t "${IMAGE_URI}" \
    -f docker/Dockerfile .
rm -f "${TOKEN_FILE}"
docker push "${IMAGE_URI}"
dockerfile.patch (only needed on 30c9ba966; already present on main)
--- a/docker/Dockerfile
+++ b/docker/Dockerfile
@@ -2,6 +2,62 @@
 # This Dockerfile builds images for running vLLM on Google Cloud TPUs.
 # It pre-installs all dependencies from pyproject.toml (including torch-tpu) in the base variant.
 
+FROM python:3.12-slim-bookworm AS rust-build
+RUN apt-get update && apt-get install -y curl git build-essential perl && rm -rf /var/lib/apt/lists/*
+ENV RUSTUP_HOME=/opt/rustup CARGO_HOME=/opt/cargo PATH=/opt/cargo/bin:$PATH
+RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --no-modify-path --profile minimal --default-toolchain none
+ARG VLLM_COMMIT_HASH=v0.29.0
+WORKDIR /build/vllm
+RUN test -n "${VLLM_COMMIT_HASH}" || { echo "VLLM_COMMIT_HASH is required to build vllm-rs" >&2; exit 1; } && \
+    git clone --quiet --filter=blob:none --no-checkout https://github.com/vllm-project/vllm.git . && \
+    git sparse-checkout set rust && \
+    git checkout --quiet "${VLLM_COMMIT_HASH}" && \
+    rustup toolchain install --profile minimal "$(sed -n 's/^channel *= *"\(.*\)"/\1/p' rust-toolchain.toml)"
+RUN --mount=type=cache,target=/opt/cargo/registry \
+    --mount=type=cache,target=/opt/cargo/git \
+    ulimit -n "$(ulimit -Hn)" && \
+    version="$(git describe --tags --exact-match 2>/dev/null || git rev-parse --short HEAD)" && \
+    VLLM_RS_BUILD_VERSION="${version#v}" CARGO_TARGET_DIR=/build/target \
+    cargo build --release --locked --manifest-path rust/src/cmd/Cargo.toml --bin vllm-rs --features native-tls-vendored && \
+    install -m 0755 /build/target/release/vllm-rs /usr/local/bin/vllm-rs && \
+    vllm-rs --version
+
+FROM python:3.12-slim-bookworm AS router-build
+RUN apt-get update && apt-get install -y curl git build-essential pkg-config libssl-dev && rm -rf /var/lib/apt/lists/*
+ENV RUSTUP_HOME=/opt/rustup CARGO_HOME=/opt/cargo PATH=/opt/cargo/bin:$PATH
+RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --no-modify-path --profile minimal --default-toolchain 1.95
+ARG VLLM_ROUTER_COMMIT=5d3c3af9a41dec2d5ecff412024b7a743763472d
+WORKDIR /build/router
+RUN git clone --quiet --filter=blob:none https://github.com/vllm-project/router.git . && \
+    git checkout --quiet "${VLLM_ROUTER_COMMIT}"
+RUN --mount=type=cache,target=/opt/cargo/registry \
+    --mount=type=cache,target=/opt/cargo/git \
+    ulimit -n "$(ulimit -Hn)" && \
+    cargo build --release --locked --bin vllm-router --no-default-features && \
+    install -m 0755 target/release/vllm-router /usr/local/bin/vllm-router && \
+    vllm-router --version
+
 # Variant 1: The Runtime (Base Image with pre-installed dependencies)
 # This variant provides the core runtime environment with all external dependencies
 # (including torch-tpu) pre-installed.
@@ -15,7 +71,7 @@ FROM python:3.12-slim-bookworm AS base
 # toolchain for packages that build from source, and libgoogle-perftools4 for
 # the tcmalloc allocator enabled below. build-essential already pulls in gcc,
 # g++, make and libc6-dev.
-RUN apt-get update && apt-get install -y curl git build-essential cmake ninja-build libgoogle-perftools4 && rm -rf /var/lib/apt/lists/*
+RUN apt-get update && apt-get install -y curl git build-essential cmake ninja-build libgoogle-perftools4 numactl && rm -rf /var/lib/apt/lists/*
 
 # tcmalloc for every derived image. Set after the install above, so that
 # intermediate steps do not fail preloading a library that is not there yet.
@@ -59,6 +115,11 @@ RUN --mount=type=cache,target=/root/.cache/uv \
       --index-strategy unsafe-best-match \
       -r pyproject.toml
 
+COPY --from=rust-build /usr/local/bin/vllm-rs /usr/local/bin/vllm-rs
+ENV VLLM_RUST_FRONTEND_PATH=/usr/local/bin/vllm-rs
+COPY --from=router-build /usr/local/bin/vllm-router /usr/local/bin/vllm-router
+
+
 # Variant 2: The CI Environment (CI/CD Variant)
 # This variant builds on top of the runtime base, installing additional packages
 # required for running test suites (pytest, linting, etc.).
@@ -99,8 +160,18 @@ FROM base AS dev
 # so that changes to folders like src/ or tests/ do not invalidate the
 # dependency installation layer cache.
 RUN --mount=type=cache,target=/root/.cache/uv \
+    --mount=type=secret,id=gcloud_token \
+    set -eu; \
+    TOKEN=""; \
+    if [ -f /run/secrets/gcloud_token ]; then TOKEN="$(cat /run/secrets/gcloud_token)"; fi; \
+    if [ -z "${TOKEN}" ]; then echo "Missing BuildKit secret gcloud_token for registry mode" >&2; exit 1; fi; \
+    export UV_INDEX_TORCH_TPU_REGISTRY_USERNAME=oauth2accesstoken; \
+    export UV_INDEX_TORCH_TPU_REGISTRY_PASSWORD="${TOKEN}"; \
     uv pip install --system \
+      --index https://us-python.pkg.dev/ml-oss-artifacts-transient/torch-tpu-virtual-registry/simple/ \
+      --extra-index-url https://pypi.org/simple \
+      --index-strategy unsafe-best-match \
       -r pyproject.toml \
       --extra test \
       --extra benchmarking

Verified Benchmark Results (2 × 2x2x4 TPU v7x = 32 chips, 8k/1k, temperature=0)

Verified across two independent end-to-end GKE runs (vllm bench serve, random 8192 input / 1024 output, 0 failed requests):

Run Concurrency Prompts Output Throughput (tok/s) Total Throughput (tok/s) Median TTFT (ms) Median TPOT (ms)
Run 1 1 8 346.83 3,121.50 455.92 2.24
Run 2 (blind repro) 1 8 348.00 3,131.96 440.68 2.24
Run 1 16 16 1,225.33 11,027.97 1,851.05 8.04
Run 2 (blind repro) 16 16 1,281.29 11,531.57 2,123.92 8.12

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a comprehensive guide and Kubernetes manifests for serving the Kimi-K3 model using vLLM with TorchTPU Pallas megakernels on Ironwood TPUs in a disaggregated 1P1D setup. The review feedback identifies a configuration mismatch in the deployment manifest: the prefill engine configuration redundantly includes the speculative decoding draft model parameter, while the decode engine configuration (which actually runs speculative decoding) is missing this parameter. Correcting these configurations will ensure proper model loading and execution.

--max-num-seqs 4 \
--max-num-batched-tokens 8192 \
--async-scheduling \
--model-loader-extra-config "{\"presharded\":\"/models/presharded/kimi-k3-tp32\",\"draft\":\"${DSPARK_DIR}\"}" \

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The prefill engine does not run speculative decoding or load the draft model. Therefore, passing the "draft":"${DSPARK_DIR}" parameter in --model-loader-extra-config is redundant and unused here. It should be removed to keep the configuration clean and avoid potential confusion.

              --model-loader-extra-config '{"presharded":"/models/presharded/kimi-k3-tp32"}' \

--max-num-seqs 4 \
--max-num-batched-tokens 8192 \
--async-scheduling \
--model-loader-extra-config "{\"presharded\":\"/models/presharded/kimi-k3-tp32\",\"draft\":\"${DSPARK_DIR}\"}" \

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The prefill engine does not run speculative decoding or load the draft model. Therefore, passing the "draft":"${DSPARK_DIR}" parameter in --model-loader-extra-config is redundant and unused here. It should be removed to keep the configuration clean and avoid potential confusion.

              --model-loader-extra-config '{"presharded":"/models/presharded/kimi-k3-tp32"}' \

--max-num-seqs 16 \
--max-num-batched-tokens 512 \
--async-scheduling \
--model-loader-extra-config '{"presharded":"/models/presharded/kimi-k3-tp32"}' \

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The decode engine is the one running speculative decoding with DSpark and loading the draft model. The megakernel loader needs the "draft":"${DSPARK_DIR}" parameter in --model-loader-extra-config to correctly locate and load the presharded draft model weights. Additionally, we must use double quotes and escape them so that the shell expands the ${DSPARK_DIR} variable correctly.

              --model-loader-extra-config "{\"presharded\":\"/models/presharded/kimi-k3-tp32\",\"draft\":\"${DSPARK_DIR}\"}" \

--max-num-seqs 16 \
--max-num-batched-tokens 512 \
--async-scheduling \
--model-loader-extra-config '{"presharded":"/models/presharded/kimi-k3-tp32"}' \

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The decode engine is the one running speculative decoding with DSpark and loading the draft model. The megakernel loader needs the "draft":"${DSPARK_DIR}" parameter in --model-loader-extra-config to correctly locate and load the presharded draft model weights. Additionally, we must use double quotes and escape them so that the shell expands the ${DSPARK_DIR} variable correctly.

              --model-loader-extra-config "{\"presharded\":\"/models/presharded/kimi-k3-tp32\",\"draft\":\"${DSPARK_DIR}\"}" \

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant