A curated guide to running models efficiently and reliably: inference fundamentals, engines, research papers, benchmarks and production practices.
The focus is LLM inference, with dedicated coverage of embeddings, speech, vision and diffusion.
Choose a path, then use the topic index for deeper reading. The order within each path is intentional.
| Your goal | Read in this order | Practical outcome |
|---|---|---|
| Learn the fundamentals | How inference works -> KV cache -> prefill and decode -> batching -> latency metrics -> bottleneck analysis | Explain where time and memory go during generation. |
| Deploy a first model | Model selection and evaluation -> engine selection -> sizing -> containers -> benchmarking methodology | Serve one model and measure its quality and latency against a representative workload. |
| Improve production serving | Benchmarking and profiling -> optimization choices -> quantization -> structured output -> reliability -> cost | Identify a bottleneck, change one variable and verify latency, quality and cost. |
Tags: Paper, YYYY marks research papers; Paid marks commercial books; Historical marks archived projects kept for background; Reference code marks research implementations whose methods now ship in production engines. Vendor benchmark results describe their tested configurations.
- Start here
- 1. Inference fundamentals
- 1.1 What inference is
- 1.2 Transformers and attention
- 1.3 The KV cache
- 1.4 Prefill, decode and serving modes
- 1.5 Batching and request scheduling
- 1.6 Latency and throughput metrics
- 1.7 Bottleneck analysis
- 1.8 Inference optimization overview
- 1.9 Model selection, evaluation and fine-tuning
- 1.10 Mixture of Experts and state-space models
- 2. Hardware
- 3. Software
- 4. Inference techniques
- 5. Multimodal inference
- 6. Production inference
- 6.1 Containers and GPU enablement
- 6.2 Kubernetes serving platforms
- 6.3 Gateways, routing and autoscaling
- 6.4 Cloud and GPU hosting platforms
- 6.5 Observability, security and debugging
- 6.6 Cost and efficiency
- 6.7 Offline batch inference, RL rollouts and reasoning budgets
- 6.8 Serving reliability and deployment safety
- 7. Learning resources
- Contributing
- License
- The Difference Between AI Training and Inference - How serving a trained model differs from training it.
- How LLM Inference Works - Prompt-to-response walkthrough of tokenization, embedding and autoregressive generation.
- The Illustrated Transformer - Diagram-led walkthrough of self-attention and the encoder-decoder transformer.
- FlashAttention - Paper, 2022. IO-aware exact attention that tiles computation to avoid materializing the attention matrix.
- FlashAttention-2 - Paper, 2023. Better work partitioning and parallelism across thread blocks and warps.
- FlashAttention-3 - Paper, 2024. Hopper-specific attention using asynchronous tensor cores, TMA and FP8.
- Flash-Decoding for Long-Context Inference - Splitting the KV sequence across thread blocks to keep the GPU busy during decode.
- Fast Transformer Decoding: One Write-Head is All You Need - Paper, 2019. Multi-query attention (MQA), which shares keys and values across heads to shrink the KV cache.
- GQA: Training Generalized Multi-Query Transformer Models - Paper, 2023. Grouped-query attention, the middle ground between multi-head and multi-query attention used by most open models.
- DeepSeek-V2 - Paper, 2024. Introduces multi-head latent attention (MLA), which compresses the KV cache into a low-rank latent.
- KV Caching Explained - How caching keys and values speeds up transformer decoding.
- What Is a KV Cache, and Why Does It Make LLM Inference Faster? - What the KV cache stores and why LLM inference depends on it.
- Paged Attention from First Principles - PagedAttention explained from virtual-memory basics.
- Prefill vs Decode - Why prefill often hits compute limits and small-batch decode often hits memory-bandwidth limits; the balance changes with batching and workload.
- Using Asynchronous Inference in Production - Queue-based inference for long-running or bursty workloads.
- vLLM OpenAI-Compatible Server - The chat, completions and embeddings endpoints most engines now expose, including streaming responses.
- OpenAI-Compatible API - LLM Inference Handbook chapter on why the OpenAI API shape became the common interface for self-hosted models.
- Chat Templates - How chat messages are rendered into the token sequence a model expects; a mismatched template silently degrades output.
- Continuous Batching from First Principles - Hugging Face walkthrough from static batching to iteration-level scheduling.
- Continuous Batching: Optimizing LLM Inference Throughput - Interactive explanation of how requests join and leave a running batch.
- Efficient Request Queueing - How queueing in front of an LLM server affects latency and throughput under load.
- Orca: A Distributed Serving System for Transformer-Based Generative Models - Paper, 2022. OSDI paper that introduced iteration-level scheduling, the basis of continuous batching.
- Efficient Memory Management for Large Language Model Serving with PagedAttention - Paper, 2023. SOSP 2023 vLLM paper on paging the KV cache to raise batch sizes.
- Sarathi-Serve - Paper, 2024. Chunked prefill and stall-free batching to balance throughput against decode latency.
- LLM Inference Benchmarking: Fundamental Concepts - Definitions of TTFT, inter-token latency, tokens per second and end-to-end latency.
- NIM LLM Benchmarking Metrics - How NVIDIA defines and measures TTFT, inter-token latency, throughput and request latency.
- Key Metrics for LLM Inference - LLM Inference Handbook chapter on latency, throughput, goodput and tail latency, with an interactive TPOT calculator.
- Reproducible Performance Metrics for LLM Inference - Anyscale's case for measuring TTFT, inter-token latency and cost the same way across providers.
- vLLM Production Metrics - The Prometheus metrics vLLM exposes for queueing, latency, KV-cache usage and throughput.
- Transformer Inference Arithmetic - Back-of-the-envelope compute and memory math for transformer inference.
- How Fast Can We Perform a Forward Pass? - Theoretical and practical limits on forward-pass latency.
- Making Deep Learning Go Brrrr From First Principles - Separates compute, memory-bandwidth and overhead bottlenecks.
- A Guide to LLM Inference and Performance - Ops:byte ratio, arithmetic intensity and memory bandwidth applied to choosing a GPU.
- What Is the Roofline Model? - Short definition of the roofline model with a worked diagram.
- GPU Performance Background User's Guide - NVIDIA's background guide to math-bound and memory-bound operations.
- All About Rooflines - Roofline analysis and arithmetic intensity applied to model workloads.
- Mastering LLM Techniques: Inference Optimization - Prefill and decode, compute and memory limits, batching and KV-cache trade-offs.
- Top 5 AI Model Optimization Techniques for Faster, Smarter Inference - Post-training quantization, quantization-aware training, quantization-aware distillation, speculative decoding and pruning with distillation.
- Inference Performance from First Principles - Aleph Alpha's analytical model of prefill and decode bottlenecks across compute, memory movement and communication, for dense and MoE models.
- Choosing the Right Model - Criteria for picking an open model for a given task, size and budget.
- LLM Benchmarking Framework with LLM-as-a-JURY - AWS 360-Eval, which benchmarks candidate models on your own data using a jury of LLM judges.
- LLM Evaluation: A Beginner's Guide - Evaluation methods for LLM systems, from benchmarks to LLM-as-a-judge.
- Langfuse - Open-source platform for tracing, evaluating and monitoring LLM applications.
- Arize Phoenix - Open-source LLM tracing and evaluation built on OpenTelemetry.
- lm-evaluation-harness - EleutherAI's framework for running standard academic benchmarks against a model.
- Lighteval - Hugging Face toolkit for evaluating LLMs across inference backends such as vLLM and SGLang.
- DeepEval - Pytest-style framework for unit-testing LLM outputs with metrics and LLM-as-a-judge.
- promptfoo - CLI for testing prompts, models and agents, including red-teaming.
- Inspect - UK AI Security Institute's framework for LLM and agent evaluations.
- PEFT - Parameter-efficient fine-tuning methods such as LoRA, which produce small adapters that engines can serve together on one base model.
- TRL - Hugging Face library for supervised fine-tuning, DPO and RL-based post-training.
- Unsloth - Memory-efficient fine-tuning and RL for open models, with export to GGUF and vLLM.
- Axolotl - Config-driven fine-tuning for open models, from LoRA to multi-GPU full fine-tuning.
- LlamaFactory - Unified fine-tuning for 100+ LLMs and VLMs with a CLI and web UI.
- LLM Model Pruning and Knowledge Distillation with NVIDIA NeMo Framework - Producing a smaller student model through pruning and distillation.
- SparseGPT - Paper, 2023. One-shot pruning of large GPT-family models to 50% or more sparsity without retraining.
- Wanda - Paper, 2023. Pruning by weight magnitude times input activation, with no retraining or weight updates.
- Compact Language Models via Pruning and Knowledge Distillation - Paper, 2024. NVIDIA Minitron recipe for deriving smaller models from a larger one.
- Mixture of Experts Explained - How sparse expert layers work, and what they change for training and inference.
- Mixtral of Experts - Paper, 2024. Sparse MoE model that activates two of eight experts per token.
- DeepSeek-V3 Technical Report - Paper, 2024. Large MoE model with MLA, auxiliary-loss-free load balancing and multi-token prediction.
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces - Paper, 2023. State-space model with constant-size recurrent state instead of a growing KV cache.
- Jamba - Paper, 2024. Hybrid Transformer-Mamba MoE model that interleaves attention and state-space layers.
- Gated Delta Networks - Paper, 2024. Linear-attention layer combining gating with the delta rule, used in recent hybrid models.
- Nemotron-H - Paper, 2025. NVIDIA's hybrid Mamba-Transformer models that replace most attention layers to cut inference cost.
- Hybrid Models as First-Class Citizens in vLLM - How vLLM manages state-space and attention caches together for hybrid models.
- Intro to GPUs for the Researcher - Practical GPU mental model aimed at raising model FLOPS utilization.
- How to Think About GPUs - How GPU chips and their networks work, and what that means for LLMs.
- What Is GPU Memory and Why It Matters for LLM Inference - How weights, KV cache and activations compete for VRAM.
- NVIDIA Hopper Architecture In-Depth - Architecture guide to the NVIDIA H100 Tensor Core GPU.
- Inside NVIDIA Blackwell Ultra - Blackwell Ultra architecture, NVFP4 and attention-layer acceleration.
- Inside the NVIDIA Vera Rubin Platform - Vendor overview of the Vera Rubin rack-scale platform and its chips, including the Vera CPU.
- AMD CDNA 3 Architecture White Paper - Architecture of the MI300 series, including chiplets and memory.
- AMD CDNA 4 Architecture White Paper - Architecture of the MI350 series and its low-precision data types.
- ROCm Documentation - Documentation for AMD's ROCm software platform for Instinct accelerators.
- AMD Instinct MI300 and MI350 Workload Optimization - AMD's guide to tuning vLLM and other inference workloads on Instinct GPUs.
- Ironwood: The first Google TPU for the age of inference - Google's announcement of its seventh-generation, inference-focused TPU.
- TPU Architecture - Official reference for TensorCores, MXUs, slices, pods and multislice topologies.
- How to Think About TPUs - TPU compute, memory and interconnect explained for model workloads.
- vLLM JAX - vLLM's TPU backend with JAX and PyTorch model support (formerly tpu-inference).
- JetStream - Throughput- and memory-optimized LLM inference engine for TPUs and other XLA devices.
- AWS Neuron Documentation - SDK, compiler and vLLM integration for AWS Trainium and Inferentia accelerators.
- NxD Inference - AWS NeuronX Distributed Inference library and model hub for serving LLMs on Trainium and Inferentia.
- Intel Gaudi Documentation - Software stack, PyTorch integration and vLLM support for Intel Gaudi accelerators.
- vLLM Gaudi Plugin - Community-maintained hardware plugin for running vLLM on Intel Gaudi.
- Cerebras Inference Documentation - API and model documentation for inference on Cerebras wafer-scale systems.
- Groq Documentation - API documentation for inference on Groq LPUs.
- TT-Metalium - Tenstorrent's low-level programming model and model library for its accelerators.
- Qualcomm Efficient Transformers - Library for porting and running Hugging Face models on Qualcomm Cloud AI 100.
- NVLink and NVLink Switch - Vendor overview of NVIDIA's GPU-to-GPU interconnect and switch fabric.
- RoCE networks for distributed AI training at scale - How Meta designs and runs RDMA over Converged Ethernet networks for its GPU clusters.
- NVIDIA InfiniBand - Vendor overview of the InfiniBand fabric used in most multi-node GPU clusters.
- GPUDirect RDMA - How NICs read and write GPU memory directly, which multi-node KV transfer and expert parallelism rely on.
- Amazon EC2 Instance Types - AWS accelerated-computing families for GPU, Trainium and Inferentia.
- Google Cloud GPU Machine Types - GPU models, machine families and regional availability on Compute Engine.
- Azure VM Sizes - Azure VM families, including the GPU-accelerated N-series.
- Artificial Analysis Hardware Benchmarks - Independent throughput, speed and cost benchmarks of inference hardware, from datacenter systems to laptops and phones.
- Calculating GPU Memory for Serving LLMs - LLM Inference Handbook chapter on estimating weight and KV-cache memory before choosing a GPU.
- Accelerate Model Memory Estimator - CLI that estimates a Hub model's memory footprint across precisions without downloading weights.
- vLLM Optimization and Tuning - Runtime controls for memory utilization, preemption, chunked prefill and batch sizes.
- PyTorch - Core framework, including torch.compile and the PyTorch profiler.
- JAX - Composable transformations and XLA compilation, widely used on TPUs.
- Transformers - Hugging Face model definitions for text, vision, audio and multimodal models, used for both inference and training.
- Tokenizers - Fast tokenizer implementations used by most open language models.
- Common AI Model Formats - Comparison of GGUF, safetensors, PyTorch and ONNX formats.
- safetensors - Safe, zero-copy tensor serialization format used by most open checkpoints.
- GGUF - Specification of the single-file quantized model format used by llama.cpp.
- ONNX Runtime - Cross-platform inference runtime for ONNX models.
- TensorRT - NVIDIA SDK that compiles and optimizes models for inference on NVIDIA GPUs.
- OpenVINO - Intel toolkit for optimizing and running models on Intel CPUs, GPUs and NPUs.
- vLLM - High-throughput, memory-efficient LLM serving engine with continuous batching and PagedAttention.
- Inside vLLM - Walkthrough of vLLM internals, from paged attention, continuous batching, prefix caching and speculative decoding to multi-GPU and multi-node serving.
- SGLang - High-performance serving framework for language and multimodal models.
- TensorRT-LLM - NVIDIA library and runtime for optimized LLM inference on NVIDIA GPUs.
- Triton Inference Server - NVIDIA's multi-framework model server, with TensorRT-LLM, vLLM, PyTorch and ONNX backends.
- LMDeploy - Toolkit for compressing, deploying and serving LLMs, built on its own TurboMind and PyTorch inference engines.
- mistral.rs - Fast LLM inference engine written in Rust.
- Modular MAX - Modular's inference framework and serving stack, with kernels written in Mojo.
- KTransformers - CPU/GPU hybrid inference that keeps MoE experts in CPU memory to run large MoE models on limited VRAM.
- DeepSpeed-MII - Microsoft DeepSpeed's inference library, built on DeepSpeed-FastGen.
- Text Generation Inference - Hugging Face's LLM server. Historical: in maintenance mode and archived.
- llama.cpp - LLM inference in C/C++ for CPUs and GPUs, using the GGUF model format.
- Ollama - Downloads and runs open models locally behind a simple API.
- llamafile - Packages a model and llama.cpp into a single executable that runs on most operating systems.
- LocalAI - Self-hosted, OpenAI-compatible server for running LLM, image and audio models locally.
- PowerInfer - Consumer-GPU engine exploiting activation sparsity across GPU and CPU.
- AirLLM - Layer-by-layer loading that runs large models on small GPUs at the cost of latency.
- ExLlamaV3 - Inference library for quantized models on consumer GPUs, using the EXL3 format.
- CTranslate2 - Optimized C++ and Python runtime for transformer models on CPU and GPU.
- MLX - Array framework for machine learning on Apple Silicon.
- vllm-metal - Community plugin for running vLLM on Apple Silicon GPUs.
- On-Device Llama 3.1 with Core ML - Apple's walkthrough of optimizing an LLM for on-device Core ML inference.
- ExecuTorch - PyTorch runtime for on-device inference on mobile and embedded hardware.
- MLC LLM - Compiler-based deployment of LLMs to phones, browsers and GPUs.
- LiteRT-LM - Google AI Edge's open-source inference framework for running LLMs on edge devices.
- MediaPipe LLM Inference - Google AI Edge API for running LLMs on Android, iOS and the web.
- WebLLM - In-browser LLM inference with WebGPU acceleration.
- Transformers.js - Runs Hugging Face models in the browser and Node.js on ONNX Runtime, with WebGPU support.
- Text Embeddings Inference - Hugging Face server for embedding, reranker and sequence-classification models.
- Infinity - OpenAI-compatible server for embedding, reranking and CLIP models.
- vLLM Pooling Models - Serving embedding, classification and reward models in vLLM.
- Sentence Transformers: Speeding up Inference - ONNX, OpenVINO and quantized backends for embedding models.
- GuideLLM - vLLM project's tool for benchmarking deployments against realistic traffic.
- genai-bench - SGLang's token-level benchmark for LLM serving endpoints.
- AIPerf - Generative AI load testing, trace replay and SLO analysis.
- MLPerf Inference - Reference implementations of the MLPerf Inference benchmarks.
- Vidur - Simulator that estimates LLM serving performance across configurations without running them.
- InferenceX - SemiAnalysis's open benchmark that continuously runs inference engines across NVIDIA and AMD hardware (formerly InferenceMAX).
- PyTorch Profiler - Recording operator- and kernel-level timings and memory from PyTorch code.
- Holistic Trace Analysis - Meta's library for analyzing PyTorch profiler traces across ranks, including idle time and communication overlap.
- Perfetto - Trace viewer used to inspect PyTorch and other Chrome-format traces.
- Nsight Systems User Guide - NVIDIA's system-wide timeline profiler for CPU, GPU and communication activity.
- ROCm Systems Profiler - AMD's system-wide timeline profiler for CPU and GPU activity.
- Nsight Compute Profiling Guide - Profiling CUDA kernels and interpreting hardware metrics.
- ROCm Compute Profiler - Kernel-level performance analysis for AMD Instinct GPUs.
- CUTLASS - CUDA C++ templates and Python DSLs for high-performance GEMM and related kernels.
- cuDNN Frontend - C++ and Python API for cuDNN graph operations, including fused attention.
- FlashAttention - Official FlashAttention kernels for NVIDIA and AMD GPUs.
- FlashInfer - Attention, sampling and MoE kernels for LLM serving, used by vLLM and SGLang.
- FlashMLA - DeepSeek's multi-head latent attention decoding kernels.
- DeepGEMM - DeepSeek's FP8 GEMM kernels with fine-grained scaling, including grouped GEMMs for MoE.
- Marlin - FP16xINT4 mixed-precision matmul kernel that underpins INT4 serving in vLLM and other engines. Reference code: production kernels ship in vLLM.
- ThunderKittens - Small embedded DSL of tile primitives for writing fast GPU kernels.
- AITER - AI Tensor Engine for ROCm: optimized inference operators for AMD GPUs.
- Composable Kernel - AMD's templated library for fused GEMM, attention and other operators.
- Triton - Language and compiler for writing GPU kernels in Python.
- TileLang - Tile-level DSL for writing GEMM, attention and other high-performance kernels.
- Mojo Manual - Modular's systems language for CPU and GPU kernels.
- Introduction to torch.compile - PyTorch's JIT compiler, used by vLLM and SGLang to fuse operators.
- Getting Started with CUDA Graphs - Capturing and replaying GPU work to cut kernel launch overhead, which matters most in small-batch decode.
- FlexAttention - PyTorch API for writing custom attention variants that compile to fused kernels.
- CUDA C++ Programming Guide - NVIDIA's reference for the CUDA programming model, memory hierarchy and execution.
- HIP Documentation - AMD's C++ GPU programming model and runtime, with a porting path from CUDA.
- Gluon Tutorials - Triton's lower-level language for explicit control over layouts and memory.
- Learn by Doing: TorchInductor Reduction Kernels - How TorchInductor generates reduction kernels.
- Apache TVM - Open machine learning compiler framework.
- XLA - Machine learning compiler for GPUs, CPUs and TPUs, used by JAX.
- IREE - MLIR-based compiler and runtime for deploying models from datacenter to edge.
- How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance - Step-by-step path from a naive kernel to near-cuBLAS performance.
- Outperforming cuBLAS on H100 - Worklog of Hopper-specific matmul optimizations.
- Inside NVIDIA GPUs: Anatomy of High-Performance Matmul Kernels - From PTX/SASS to warp tiling and asynchronous tensor-core pipelines.
- Categorical Foundations for CuTe Layouts - The mathematics behind CuTe's layout algebra.
- Demystifying Numeric Conversions in CuTeDSL - Type-conversion primitives in CuTeDSL.
- Blackwell Pipelining with CuTeDSL - Overlapping work with CuTeDSL asynchronous pipelines on Blackwell.
- A Visual Guide to Quantization - Illustrated introduction to LLM quantization.
- Post Training Quantization - Walkthrough of post-training weight quantization.
- Quantization-Aware Training for Large Language Models with PyTorch - Recovering accuracy lost to quantization by simulating it during training.
- Microscaling Data Formats for Deep Learning - Paper, 2023. Block-scaled MX formats (MXFP8, MXFP6, MXFP4, MXINT8) with a shared scale per block of elements.
- OCP Microscaling Formats (MX) Specification - The Open Compute Project specification for MX data formats.
- Introducing NVFP4 for Efficient and Accurate Low-Precision Inference - NVIDIA's 4-bit floating-point format for Blackwell and how its two-level scaling differs from MXFP4.
- gpt-oss - OpenAI's open-weight models, released with MXFP4-quantized MoE weights and reference inference code.
- llama.cpp Quantization Tool - The GGUF quantization types (k-quants, i-quants) and how to produce them.
- FP8 Formats for Deep Learning - Paper, 2022. The E4M3 and E5M2 FP8 formats now supported by Hopper, Blackwell and MI300 hardware.
- GPTQ - Paper, 2022. One-shot weight quantization to 3-4 bits using approximate second-order information.
- AWQ - Paper, 2023. Activation-aware weight quantization that protects the most salient weight channels.
- SmoothQuant - Paper, 2022. Moving activation outliers into weights to enable W8A8 quantization.
- QuaRot - Paper, 2024. Rotating hidden states to remove outliers for end-to-end 4-bit weights, activations and KV cache.
- SpinQuant - Paper, 2024. Learned rotations for low-bit weight, activation and KV-cache quantization.
- KIVI - Paper, 2024. Tuning-free 2-bit KV-cache quantization, per-channel for keys and per-token for values.
- LLM Compressor - vLLM project's library for producing GPTQ, AWQ, FP8 and other quantized checkpoints.
- NVIDIA Model Optimizer - Quantization, pruning, distillation and sparsity for deployment with TensorRT-LLM, vLLM and SGLang.
- torchao - PyTorch-native quantization and sparsity for training and inference.
- bitsandbytes - 8-bit and 4-bit quantization for PyTorch, widely used for QLoRA.
- GPTQModel - GPTQ quantization toolkit with CUDA, ROCm, XPU and CPU backends.
- AutoRound - Intel's sign-gradient post-training quantization for low-bit LLMs and VLMs.
- Fast Inference from Transformers via Speculative Decoding - Paper, 2022. Foundational draft-target speculative decoding algorithm with parallel verification.
- EAGLE - Official implementation of the EAGLE-1, EAGLE-2 and EAGLE-3 speculative decoding methods.
- Medusa - Multiple decoding heads for candidate token generation. Reference code: production implementations are available in TensorRT-LLM.
- SpecInfer - Paper, 2023. Verifying a tree of speculated tokens in one pass instead of a single sequence.
- Better and Faster Large Language Models via Multi-token Prediction - Paper, 2024. Training models to predict several tokens at once, used for self-speculative decoding.
- Lookahead Decoding - Draft-free parallel decoding using Jacobi iteration and an n-gram pool. Reference code: production implementations are available in TensorRT-LLM.
- Prompt Lookup Decoding - Using n-gram matches from the prompt as drafts, effective for summarization and code editing. Reference code: production equivalent is n-gram speculation in vLLM.
- MagicDec - Paper, 2024. When speculative decoding still helps at large batch sizes for long-context requests.
- vLLM Speculative Decoding - Configuring draft-model, n-gram, MTP and EAGLE speculation in vLLM.
- Speculators: Production-Ready Speculative Decoding - Why the speculators library standardizes draft-model formats.
- Prompt Caching - How cached prompt tokens skip recomputation and why that makes them cheaper.
- vLLM Automatic Prefix Caching - Design of vLLM's hash-based prefix cache.
- Fast and Expressive LLM Inference with RadixAttention and SGLang - Radix-tree KV-cache reuse across requests in SGLang.
- Ray Serve: Reduce LLM Inference Latency by 60% with Custom Request Routing - Prefix-aware routing to cut TTFT in multi-turn chat.
- How to Reduce KV Cache Bottlenecks with NVIDIA Dynamo - KV-cache offload and reuse in Dynamo.
- Inside vLLM's New KV Offloading Connector - Offloading KV-cache to CPU DRAM and tuning host-to-device throughput.
- LMCache - KV-cache layer that stores and reuses KV cache across GPU, CPU, disk and remote tiers.
- PrisKV - Colocated, tiered KV-cache store from the AIBrix project.
- kvcached - Elastic, virtualized KV-cache that lets multiple models share a GPU.
- Mooncake - KV-cache-centric serving platform behind Moonshot AI's Kimi, whose Mooncake Store pools, reuses and offloads KV cache across serving instances.
- Rotary Embeddings: A Relative Revolution - Explains RoPE, the position encoding most context-extension methods modify.
- RoFormer - Paper, 2021. The paper that introduced rotary position embeddings (RoPE).
- YaRN - Paper, 2023. Efficient RoPE scaling for extending a model's context window.
- LongRoPE - Paper, 2024. Extending RoPE-based models beyond 2 million tokens.
- DroPE - Paper, 2025. Extends context by dropping positional embeddings instead of long-context fine-tuning.
- Ring Attention with Blockwise Transformers for Near-Infinite Context - Paper, 2023. Splitting long sequences across devices with blockwise attention passed around a ring.
- Efficient Streaming Language Models with Attention Sinks - Paper, 2023. StreamingLLM: keeping initial sink tokens plus a sliding window for unbounded streaming generation.
- H2O: Heavy-Hitter Oracle - Paper, 2023. Evicting KV-cache entries that receive little attention.
- SnapKV - Compressing the KV cache of long prompts by selecting important positions before generation. Reference code: also implemented in kvpress.
- kvpress - NVIDIA library implementing and benchmarking many KV-cache compression methods.
- Native Sparse Attention - Paper, 2025. DeepSeek's hardware-aligned, trainable sparse attention for long-context models.
- Evaluating Long Context (Reasoning) Ability - Why models degrade well before their advertised context length, measured on LongCodeEdit.
- vLLM Parallelism and Scaling - Configuring tensor, pipeline, data and expert parallelism for single-node and multi-node vLLM deployments.
- How Multi-Node Inference Works for Massive LLMs like DeepSeek-R1 - A DeepSeek-R1 deployment using 16 H100s across two nodes, explaining the memory and communication choices in that configuration.
- Megatron-LM - Paper, 2019. The paper that defined intra-layer tensor parallelism for transformers.
- Shift Parallelism - Paper, 2025. Switching between tensor and data parallelism to balance latency and throughput.
- vLLM Large Scale Serving: DeepSeek @ 2.2k tok/s/H200 with Wide-EP - Wide expert parallelism, dual-batch overlap and disaggregated serving for DeepSeek on H200 clusters.
- NCCL - NVIDIA collective communication library for multi-GPU and multi-node transfers.
- RCCL - AMD's collective communication library for ROCm GPUs.
- NVSHMEM - NVIDIA's GPU-initiated communication library, used by DeepEP and other MoE kernels.
- DeepEP - Expert-parallel communication library for MoE dispatch and combine.
- UCCL - GPU communication library for collectives, P2P KV transfer and expert parallelism.
- llm-d Wide Expert Parallelism - How llm-d deploys large MoE models with data and expert parallelism across nodes.
- Disaggregated Inference at Scale with PyTorch and vLLM - Running separate prefill and decode pools with vLLM.
- DistServe - Paper, 2024. OSDI 2024 paper on placing prefill and decode on separate GPUs to meet TTFT and TPOT targets.
- Splitwise - Paper, 2023. Splitting prompt and token phases onto different machines, including different GPU types.
- Disaggregated Serving in TensorRT-LLM - TensorRT-LLM's disaggregated serving design.
- SGLang PD Disaggregation - Configuring prefill/decode disaggregation in SGLang.
- NVIDIA Dynamo - Datacenter-scale distributed inference serving framework with disaggregated prefill and decode.
- NIXL - NVIDIA Inference Xfer Library for moving KV cache between GPUs, nodes and storage, used by Dynamo, vLLM and llm-d.
- llm-d Architecture - Component design and topology of llm-d.
- llm-d - Kubernetes-native distributed LLM inference with KV-cache-aware scheduling.
- llm-d P/D Disaggregation - When to split prefill and decode in llm-d and how KV-cache is transferred between them.
- How to Generate Text - Greedy search, beam search, top-k and top-p sampling compared.
- Inference Parameters - LLM Inference Handbook chapter on temperature, top-p, penalties and other generation controls.
- Min-p Sampling - Paper, 2024. Truncating the token distribution relative to the top token's probability.
- XGrammar - Fast grammar-constrained decoding engine, integrated into serving engines including vLLM and SGLang; backend defaults depend on engine version.
- llguidance - Low-latency constrained decoding for JSON schemas, regular expressions and context-free grammars.
- Outlines - Library for generating structured output from LLMs using regex, JSON schemas and grammars.
- LM Format Enforcer - Enforcing JSON schema or regex output by filtering allowed tokens.
- vLLM Structured Outputs - Guided JSON, regex, choice and grammar decoding in vLLM.
- vLLM Tool Calling - How vLLM parses model-specific tool-call formats into OpenAI-compatible responses.
- vLLM Reasoning Outputs - Separating reasoning content from the final answer for reasoning models.
- What Is an Omni-Model? - NVIDIA definition of models that take and produce several modalities.
- Qwen2.5-Omni Technical Report - Paper, 2025. Thinker-talker architecture that reads text, images, audio and video and streams speech back.
- Qwen3-Omni - End-to-end omni-modal model that understands text, audio, images and video and generates speech in real time.
- vLLM-Omni - vLLM framework for serving omni-modal models.
- vLLM-Omni: Fully Disaggregated Serving for Any-to-Any Multimodal Models - Paper, 2026. Paper on serving multimodal pipelines as independently scheduled stages.
- Vision Language Models Explained - Hugging Face introduction to VLM architectures, training and how to choose one.
- Vision Language Models (Better, Faster, Stronger) - Hugging Face update on newer VLM trends, including any-to-any models, reasoning, small models and MoE decoders.
- Vision Language Models - Inference Engineering book chapter on serving models that take images and video, and the cost of the vision encoder.
- vLLM Multimodal Inputs - Passing images, video and audio to multimodal models in vLLM.
- Vision Language Model Prompt Engineering Guide for Image and Video Understanding - NVIDIA guide to feeding single images, multiple images and video frames to VLMs.
- Robust Speech Recognition via Large-Scale Weak Supervision - Paper, 2022. The Whisper paper.
- Silero VAD - Lightweight voice activity detector for finding speech before transcription.
- faster-whisper - Whisper transcription reimplemented on CTranslate2 for faster inference.
- whisper.cpp - Port of Whisper to C/C++ for CPU, GPU and on-device transcription.
- WhisperX - Batched Whisper transcription with word-level timestamps and speaker diarization.
- NeMo Speech - NVIDIA's speech toolkit, home of the Parakeet and Canary ASR models.
- sherpa-onnx - Offline streaming ASR, TTS, VAD and diarization on ONNX Runtime for mobile, embedded and server targets.
- Piper - Fast, local neural text-to-speech engine.
- Kokoro - Small 82M-parameter open-weight text-to-speech model.
- Pipecat - Open-source framework for real-time voice and multimodal agents that chains streaming ASR, LLM and TTS services.
- LiveKit Agents - Framework for real-time voice AI agents over WebRTC, with turn detection and interruption handling.
- What Is Speech-to-Speech? - Cascaded versus end-to-end speech-to-speech pipelines and where each fits.
- Moshi - Speech-text foundation model and full-duplex dialogue framework with a streaming audio codec.
- Ultralytics YOLO - Training, export and inference for YOLO detection, segmentation, pose and tracking models.
- Ultralytics Export Formats - Exporting YOLO models to ONNX, TensorRT, OpenVINO, Core ML and other deployment formats.
- Triton Inference Server with Ultralytics YOLO - Exporting a YOLO model to ONNX and serving it from a Triton model repository.
- NVIDIA DeepStream SDK - GStreamer-based toolkit for multi-stream video analytics pipelines on NVIDIA GPUs and Jetson.
- DeepStream-Yolo - Community implementation for running YOLO models inside DeepStream pipelines.
- Savant - Python framework for video analytics pipelines, built on DeepStream for NVIDIA GPUs and Jetson.
- DL Streamer - Intel's GStreamer-based pipeline framework for video analytics on Intel CPUs, GPUs and NPUs.
- Roboflow Inference - Self-hosted server for running vision models and video-stream workflows on cloud or edge devices.
- Diffusion Models - Introductory explanation of how diffusion models are trained and sampled.
- Diffusers - Hugging Face library of diffusion models and pipelines for image, video and audio generation.
- ComfyUI - Node-graph interface, API and backend for running diffusion pipelines.
- SGLang Diffusion - SGLang's serving stack applied to image and video generation.
- xDiT - Parallel inference engine for diffusion transformers, splitting image and video generation across GPUs.
- FastVideo - Inference and post-training framework for accelerated video generation.
- vLLM-Omni Diffusion Guide - Attention backends and acceleration options for diffusion models in vLLM-Omni.
- stable-diffusion.cpp - Diffusion model inference in C/C++ on ggml, for CPUs and consumer GPUs.
- Latent Consistency Models - Paper, 2023. Distilling diffusion models to generate images in a few sampling steps.
- DeepCache - Training-free acceleration by reusing high-level U-Net features across denoising steps.
- TeaCache - Training-free caching that skips redundant timesteps in video and image diffusion models.
- Nunchaku - 4-bit inference engine for diffusion models based on SVDQuant.
- NVIDIA Container Toolkit - Runtime hooks that expose NVIDIA GPUs to containers.
- AMD Container Toolkit - Runtime and tools that expose AMD GPUs to containers.
- NVIDIA NIM - Prebuilt, optimized inference microservice containers for NVIDIA GPUs.
- BentoML - Framework for packaging models as inference APIs, job queues and multi-model pipelines.
- NVIDIA Device Plugin - Exposes NVIDIA GPUs as schedulable Kubernetes resources.
- NVIDIA GPU Operator - Installs and manages NVIDIA drivers, device plugin and monitoring on Kubernetes.
- AMD GPU Device Plugin - Exposes AMD GPUs as schedulable Kubernetes resources.
- AMD GPU Operator - Installs and manages AMD GPU drivers, device plugin and metrics on Kubernetes.
- Model Streamer - Concurrent streaming of safetensors weights from object storage or disk into GPU memory (formerly Run:ai Model Streamer).
- fastsafetensors - Faster safetensors loading using GPUDirect Storage and parallel reads.
- Tensorizer - CoreWeave's serialization format and loader for streaming model weights from HTTP, S3 or disk.
- KServe OCI Storage (Modelcars) - Packaging models as OCI images so nodes cache and share them like container layers.
- Kubernetes Image Volumes - Mounting an OCI image as a read-only volume, which lets model weights ship separately from the serving image.
- ModelPack Model Spec - Open specification for packaging and distributing models as OCI artifacts.
- KitOps - CNCF tool for packaging and versioning models, datasets and configuration as OCI artifacts.
- KServe - Kubernetes platform for deploying generative and predictive models across frameworks.
- Ray - Distributed compute engine whose Ray Serve library serves models at scale.
- vLLM Production Stack - vLLM's reference system for Kubernetes-native, cluster-wide deployment.
- AIBrix - vLLM project's Kubernetes control plane for LLM serving, with LoRA management, routing and autoscaling.
- KAITO - CNCF Kubernetes AI Toolchain Operator that provisions GPU nodes and deploys open models.
- HAMi - CNCF project for sharing and isolating heterogeneous GPUs on Kubernetes.
- LeaderWorkerSet - Kubernetes API for deploying a group of pods as one unit, used for multi-node inference.
- Kueue - Kubernetes job queueing and quota management for sharing GPU capacity across teams.
- NVIDIA Multi-Instance GPU User Guide - Partitioning a GPU into isolated instances with dedicated memory and compute.
- GPU Operator: GPU Sharing - Configuring time-slicing so several pods share one GPU on Kubernetes.
- Kubernetes Dynamic Resource Allocation - The Kubernetes API for requesting and sharing devices such as GPUs with structured parameters.
- DRA Driver for NVIDIA GPUs - Kubernetes DRA driver that allocates NVIDIA GPUs, MIG devices and shared GPUs.
- The Baseten Inference Stack - Runtime and infrastructure layers of a production stack, including geo-aware load balancing.
- LiteLLM - AI gateway and SDK that calls 100+ LLM APIs in OpenAI format, with load balancing and cost tracking.
- Agent Router (formerly Envoy AI Gateway) - Unified access to generative AI services, built on Envoy Gateway.
- Portkey Gateway - Open-source AI gateway with routing, retries, fallbacks and guardrails across LLM providers.
- Gateway API Inference Extension - Kubernetes SIG project that adds model-aware routing to Gateway API through InferencePool.
- llm-d Router - llm-d's scheduler that routes requests by prefix-cache hits, load and predicted latency.
- KEDA - Event-driven Kubernetes autoscaler, often used to scale inference pods on queue depth.
- Karpenter - Kubernetes node autoscaler that provisions GPU nodes to fit pending pods.
- vLLM LoRA Adapters - Serving many LoRA adapters on one base model and routing requests to them.
- S-LoRA - Paper, 2023. Serving thousands of concurrent LoRA adapters with unified paging of adapter weights and KV cache.
- Punica - Paper, 2023. Batching requests for different LoRA adapters with a custom segmented-gather kernel.
- vLLM Semantic Router - Programmable layer that classifies requests and routes them to the right model.
- Autoscaling Endpoints for LLM Inference - Choosing scaling metrics, tuning scaling windows and budgeting for cold starts.
- vLLM: Why Requests Take Hours Under Load - Diagnosing KV block exhaustion and queue starvation.
- Serverless vs. Self-hosted LLM Inference - Trade-offs between managed APIs and self-hosted deployments in control, cost, scaling and compliance.
- Bring Your Own Cloud - The middle ground of managed serving software running in your own cloud account.
- On-Prem LLM Deployments - When on-premises inference makes sense and what it requires to operate.
- Load and Serve Models on Amazon EKS - Deploying vLLM on GPUs and streaming weights from S3 with the Run:ai Model Streamer to cut cold starts.
- EKS Best Practices: Application Scaling and Performance - Model artifact management, serving frameworks and container startup optimization.
- GPU Inference on EKS Auto Mode - GPU node pools with AWS-managed node lifecycle and patching.
- Introducing Disaggregated Inference on AWS Powered by llm-d - Running llm-d prefill/decode disaggregation and wide expert parallelism on AWS.
- Deploy GKE Inference Gateway Powered by llm-d - Managed gateway using llm-d scheduling.
- Best Practices for Autoscaling LLM Inference Workloads on GKE - Choosing HPA signals such as queue size and batch size over GPU utilization.
- Tuning the GKE HPA to Run Inference on GPUs - Experiments comparing autoscaling metrics for LLM servers.
- Serve an LLM Using TPUs on GKE with KubeRay - Deploying a RayCluster on a TPU node pool and serving an LLM with Ray Serve.
- Deploy an AI Model on AKS with the AI Toolchain Operator - Managed KAITO add-on that deploys open models with vLLM and provisions GPU nodes.
- Use GPUs on AKS - Creating GPU node pools and installing the NVIDIA device plugin.
- GPU Node Partitioning Strategies in AKS - Comparison of MIG, time-slicing and MPS for sharing GPUs.
- Create a Multi-Instance GPU Node Pool in AKS - Creating MIG-partitioned node pools on AKS.
- Serving Llama 3.1 with vLLM on Lambda - Lambda's guide to running vLLM in Docker on on-demand A100 and H100 instances.
- About CoreWeave Inference - CoreWeave's options for serving models: serverless, dedicated, or self-managed on its Kubernetes service.
- vLLM on Vast.ai - Vast.ai's template for serving LLMs with vLLM on marketplace GPUs.
- Serve LLM Inference on E2E GPU Nodes - E2E Networks' guide to serving an LLM on its GPU virtual machines.
- RunPod Serverless vLLM - Deploying an OpenAI-compatible vLLM endpoint on RunPod Serverless that scales with traffic.
- Replicate: Deploy a Custom Model - Packaging a model with Cog and running it as a dedicated, autoscaling deployment on Replicate.
- Cog - Replicate's open-source tool for packaging models as production-ready containers.
- E2E Networks TIR Model Endpoints - Creating managed endpoints with vLLM, SGLang or Triton runtimes on E2E's TIR platform.
- Modal: LLM Inference with vLLM - Modal's example of serving an OpenAI-compatible vLLM server on serverless GPUs.
- Baseten Model Deployment - Deploying models on Baseten with Truss, from a config file to an autoscaling API.
- Truss - Baseten's open-source framework for packaging models into deployable containers.
- Hugging Face Inference Endpoints - Deploying Hub models on managed, autoscaling infrastructure across cloud providers.
- Together AI Dedicated Inference - Running models on dedicated, single-tenant GPU endpoints on Together AI.
- DCGM Exporter - Prometheus exporter for NVIDIA GPU metrics, built on DCGM.
- AMD Device Metrics Exporter - Exports AMD GPU metrics to Prometheus and other collectors.
- OpenTelemetry Semantic Conventions for Generative AI - Evolving conventions for spans and metrics covering model calls, tokens and latency; pin the convention version when instrumenting.
- OpenLLMetry - OpenTelemetry instrumentation for LLM providers, frameworks and vector databases.
- Helicone - Open-source LLM observability platform and gateway for request logging, cost and latency.
- Langfuse and Arize Phoenix, listed under Evaluation, also trace production LLM traffic.
- Confidential Compute for AI Inference - How Chutes uses trusted execution environments for verifiable private inference on third-party GPUs.
- OWASP Top 10 for LLM Applications - Common security risks for LLM applications, from prompt injection to unbounded consumption.
- ModelScan - Scans model files for unsafe serialized code, such as malicious pickle payloads.
- Pickle Scanning - Why pickle-based checkpoints can execute code on load, and how the Hugging Face Hub scans for it.
- Model Transparency - Sigstore and OpenSSF tooling for signing models and verifying their provenance.
- Llama Guard - Paper, 2023. LLM-based safeguard model for classifying prompts and responses.
- NeMo Guardrails - Programmable input, output and dialog guardrails for LLM applications.
- Token-Level Truth - HaluGate: token-level hallucination detection in vLLM Semantic Router, checked against tool outputs and context.
- One Token to Corrupt Them All - How AI21 traced a state-corruption bug in vLLM's scheduler and memory management for Mamba models.
- LLM Inference Benchmarking: How Much Does Your LLM Inference Cost? - NVIDIA method for turning benchmark results into cost per million tokens and total cost of ownership.
- The Hidden Economics of LLM Inference - Queueing math and GPU utilization, and why pooled multi-tenant serving costs less than single-tenant serving.
- Energy Use of AI Inference, Efficiency Pathways, and Test-Time Scaling - Paper, 2025. Per-query energy estimates and efficiency pathways at scale.
- GPU Inefficiency in AI Workloads - Why production GPUs sit underutilized.
- Zeus - Library for measuring and optimizing GPU energy consumption of AI workloads.
- ML.ENERGY Leaderboard - Energy-per-request measurements for open models across serving configurations.
- vLLM Offline Inference - Running vLLM as a library for batch generation without an HTTP server.
- Ray Data LLM - Batch LLM inference over large datasets with vLLM on Ray Data.
- verl - RL post-training framework that uses vLLM and SGLang as rollout engines.
- slime - RL scaling framework that connects Megatron training with SGLang rollouts.
- Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models - Paper, 2025. Survey of methods for shortening reasoning traces and lowering their inference cost.
- s1: Simple Test-Time Scaling - Paper, 2025. Budget forcing: capping or extending a model's reasoning length at decode time.
- Separating reasoning tokens from the final answer is covered in vLLM Reasoning Outputs.
- Ray Serve Production Best Practices - Request handling, timeouts and operational configuration for Ray Serve deployments.
- Ray Serve Fault Tolerance - How Ray Serve recovers from replica, node and head-node failures.
- Kubernetes Pod Lifecycle - Probes, termination grace periods and shutdown semantics for serving pods.
- Fairness in Serving Large Language Models - Paper, 2023. Virtual Token Counter (VTC), a fair scheduler that accounts for both input and output tokens per client.
- Llumnix - Paper, 2024. Rescheduling requests across serving instances with live KV-cache migration for load balancing and priorities.
- KV-cache exhaustion under load is covered in vLLM: Why Requests Take Hours Under Load, and engine-side signals for SLOs in vLLM Production Metrics.
- Inference Engineering - Philip Kiely's book on building faster, cheaper and more reliable generative AI inference systems.
- How To Scale Your Model - TPU/GPU architecture, communication and parallelism for training and inference.
- The Ultra-Scale Playbook - Hugging Face's interactive guide to parallelism and communication for training LLMs on GPU clusters.
- Hands-On LLM Serving and Optimization - Paid book. Chi Wang and Peiheng Hu's O'Reilly book on hosting and optimizing LLMs at scale.
- MIT 6.5940: TinyML and Efficient Deep Learning Computing - Song Han's course on pruning, quantization, distillation and efficient LLM inference.
- CMU 15-442/15-642: Machine Learning Systems - Course on the systems behind ML, from GPU programming to LLM serving.
- Stanford CS336: Language Modeling from Scratch - Course that builds a language model end to end, including kernels, parallelism and inference.
- Programming Massively Parallel Processors - Paid book. Hwu, Kirk and El Hajj's textbook on CUDA and parallel programming patterns.
- 100 Days of LLM Inference - Day-by-day runnable experiments that follow the Inference Engineering book, from KV cache and kernels to quantization, speculative decoding and disaggregated serving.
- LLM Inference Engineering - Step-by-step blog series from autoregressive generation, prefill and decode to KV cache, PagedAttention, continuous batching, vLLM, SGLang and GPUs.
- AI Inference Resources - AER Labs' tiered reading guide covering LLM serving, GPU kernels, attention, quantization, distributed inference and production deployment.
- Inference Engineering Library - Community-shared books, courses, code and research for inference engineering, filterable by format and level.
- GPU MODE Lectures - Slides and code from the GPU MODE lecture series on CUDA, Triton and inference kernels.
- A Survey on Efficient Inference for Large Language Models - Paper, 2024. Taxonomy of data-, model- and system-level inference optimizations.
- Awesome LLM Inference - Companion list of LLM and VLM inference papers with code.
- The Rise of vLLM - How vLLM grew from a research project into a widely used engine.
- SGLang: An Efficient Open-Source Framework for Large-Scale LLM Serving - Ray Summit 2025 talk on SGLang's architecture and production use.
- How to Become an Inference Engineer - Baseten talk on the skills and path to becoming an inference engineer.
- Inference OSS Ecosystem featuring vLLM - Disaggregation and wide-EP for MoE on GB200 racks.
- Inside the MAX Framework - Modular meetup walkthrough of the MAX inference stack.
- Introduction to LLM Serving with SGLang - AI Engineer workshop by Philip Kiely and Yineng Zhang on serving LLMs with SGLang.
- GPU MODE - Recorded lectures on GPU programming, kernels and inference systems.
Read the contribution guide before submitting. Use the issue templates to propose a topic.
This repository's original content is licensed under the MIT License. Linked projects, documentation, papers, images and code retain their respective licenses.