Kubernetes-native LLM inference. I work on the layer between the request and the model: scheduling, routing, gateways, and serving. When it breaks, I reproduce it, measure it, fix it, and send the fix upstream.
Building inference infrastructure in public, one merged fix at a time -> @say_lee_x
Open to inference infrastructure roles and contracts. Remote, India.
Now: working across the Kubernetes inference stack: llm-d-router for prefix-cache-aware routing and KV events, AIBrix for the OpenAI-compatible gateway and batch/file APIs, kueue for GPU job scheduling, KServe for model serving, and kgateway for Gateway API behavior. Also sending fixes to litellm, Knative, Volcano, and Crossplane.
16 merged upstream across AIBrix, llm-d-router, kueue and kgateway.
| PR | What it fixed |
|---|---|
| llm-d-router#3803 | keep an explicit zero cache read in Anthropic usage |
| llm-d-router#3688 | select the token producer backend explicitly |
| llm-d-router#3672 | cover the default utilization filter with prefix affinity |
| llm-d-router#3674 | cancel an abandoned enqueue when the client can no longer address it |
| llm-d-router#3484 | release per-request producer state on early return |
| llm-d-router#3378 | separate the model name and cache salt in the prefix-cache block hash, so distinct models no longer collide |
| llm-d-router#3375 | read and rewrite the model name in the SGLang /generate parser |
| aibrix#3332 | divide the TPOT histogram sum by its count so avg_tpot_pod_5m is an average |
| aibrix#3024 | route OpenAI-compatible /v1/audio/speech requests through the gateway |
| aibrix#3000 | de-flake a timezone-dependent job expiry test |
| aibrix#2979 | stop non-ASCII filenames from breaking Files API downloads |
| aibrix#2977 | enforce the read size limit on the bytes actually read |
| aibrix#2959 | de-flake job restore after a metadata crash |
| kueue#17782 | spell out previous in Workload update log keys |
| kueue#17641 | show the ProvisioningRequest retry loop in the admission cycle docs |
| kgateway#14828 | make leader election durations configurable |
A public conformance suite for OpenAI-compatible gateways: openai-gateway-conformance. It checks the contract, not just the JSON: finish_reason after a max_tokens cut, usage that adds up, request ids that survive the hop, one finish event per stream, usage after finish, non-cumulative deltas, [DONE] at the end, and buffered streams that fake a fast first token.
The first version runs 19 tests and 10 fault modes, with a pass/fail matrix. It is CPU-first, so it runs in CI without a GPU. The mock passes 19/19 and catches 10/10 faults. LiteLLM 1.106.0 passes 15/19, with four recorded gaps in error shape and request tracing. Next targets: AIBrix and llm-d-router.
- llm-d-router#3702: report live-only pods and snapshot load duration from KV events
- kueue#18118: drop every unhealthy node the new assignment leaves
- kserve#6517: report Ready from the deployment's Available condition
- knative/serving#16723: fix Revision readiness when image cache reconciliation fails
- litellm#45967: stop json_mode from reporting truncated output as a clean stop
- litellm#45423: keep
delta.refusalchunks instead of dropping them - kgateway#14815: populate GatewayExtension status with an Accepted condition
More on GitHub.
- Reproduce first: a failing test or a minimal script before the fix.
- Measure the before and after, not just the patch.
- Fix the root cause and send it upstream, with the regression test.
Go · Python · Kubernetes · vLLM and SGLang · OpenAI-compatible gateways · SSE · Redis · Postgres
Open to inference infrastructure and platform roles, full-time or contract. The fastest way to reach me is X.