Building the C++ inference library for Blackwell (sm_120) — working configuration and notes

[mahmoudshehata74]

The prebuilt Python wheels cover Blackwell through the cu129 / cu130 package indexes, but the C++ inference library download table currently tops out at CUDA 12.6:

Backend	CUDA / cuDNN / TensorRT	gcc
GPU	CUDA 11.8 / cuDNN 8.9 / TRT 8.6	11.2
GPU	CUDA 12.6 / cuDNN 9.5 / TRT 10.5	11.2

Since Blackwell support arrived in CUDA 12.8, both are compiled for architectures ending at sm_90, and an RTX 50-series card fails at runtime:

The installed PaddlePaddle package was compiled for: 61 70 75 80 86 89 90
But your GPU is: 120

For projects linking libpaddle_inference.so from C++, building from source is currently the only option. I did this successfully and wanted to share the configuration in case it saves someone else the trial and error.

Working configuration

Paddle v3.3.0, CUDA 13.0, cuDNN 9.16, TensorRT 10.14.1, gcc 13.3, Ubuntu 24.04:

cmake -S $PADDLE_SRC -B $PADDLE_SRC/build \
  -DWITH_GPU=ON \
  -DON_INFER=ON \
  -DWITH_PYTHON=OFF \
  -DWITH_TESTING=OFF \
  -DWITH_DISTRIBUTE=OFF \
  -DWITH_NCCL=OFF \
  -DWITH_RCCL=OFF \
  -DWITH_MKL=ON \
  -DWITH_AVX=ON \
  -DWITH_TENSORRT=ON \
  -DTENSORRT_ROOT=/usr \
  -DTENSORRT_INCLUDE_DIR=/usr/include/x86_64-linux-gnu \
  -DTENSORRT_LIBRARY_DIR=/usr/lib/x86_64-linux-gnu \
  -DCUDA_ARCH_NAME=Manual \
  -DCUDA_ARCH_BIN="89;120" \
  -DCMAKE_BUILD_TYPE=Release
make inference_lib_dist -j4

Result — one library covering both Ada and Blackwell:

$ cuobjdump -lelf libpaddle_inference.so | grep -oE 'sm_[0-9]+' | sort -u
sm_120
sm_89

version.txt reports CUDA version: 13.0, CUDNN version: v9.16, WITH_TENSORRT: ON, TensorRT version: v10.14.1.48. A predictor created with EnableUseGpu(500, 0) runs and logs GPU Compute Capability: 12.0.

Notes that might be useful

WITH_NCCL=OFF is required for a single-GPU build. WITH_DISTRIBUTE=OFF alone does not disable the NCCL header check, and configure fails with:

CMake Error at cmake/nccl.cmake:20 (file):
  file failed to open for reading: NCCL_INCLUDE_DIR-NOTFOUND/nccl.h

CUDA_ARCH_NAME=Manual must precede CUDA_ARCH_BIN, otherwise the arch list is ignored. Worth confirming in the configure output:

-- NVCC_FLAGS_EXTRA: -gencode arch=compute_89,code=sm_89 -gencode arch=compute_120,code=sm_120

gcc 13.3 works despite cmake/flags.cmake warning that it is "too high, recommended to use GCC 8.2". No compiler errors on this version.

-DWITH_FLASHATTN=OFF does not exist in v3.3.0 — CMake accepts and silently ignores it. FlashAttention is a hard dependency of inference_lib_dist and adds significant build time compiling sm_80 / sm_90 kernels. If there is a supported way to skip it for inference-only builds, I would be glad to know.

For apt-installed TensorRT, TENSORRT_ROOT=/usr — headers in /usr/include/x86_64-linux-gnu, libraries in /usr/lib/x86_64-linux-gnu. There is no TensorRT-10.x/ directory to point at.

Runtime linking: libiomp5.so lives in third_party/install/mklml/lib, not paddle/lib. Applications that only add paddle/lib to their rpath will fail to start.

Build took roughly 4 hours with -j4 on 16 cores and 32 GB of RAM.

An environment issue worth flagging

Not a Paddle problem, but it cost me a full build and may affect others on Ubuntu 22.04+: systemd-oomd killed cicc mid-compile while 28 GB of RAM and 46 GB of swap were free.

Out of memory: Killed process (cicc) total-vm:20726956kB, anon-rss:2103472kB, oom_score_adj:200

cicc reserves a large virtual address space it never touches (20.7 GB virtual, 2.1 GB resident). systemd-oomd kills on PSI memory pressure rather than actual exhaustion, and the default heuristic overcommit mode compounds it. Masking systemd-oomd.service and systemd-oomd.socket and setting vm.overcommit_memory=1 resolved it. Disabling without masking is not enough — the socket unit restarts the service.

Full write-up

I documented the whole process, every obstacle, and performance measurements here:

https://github.com/mahmoudshehata74/paddle-inference-cpp-sm120

Includes build scripts, a preflight environment checker, and benchmark data for PP-OCRv5 on an RTX 5060 (twelve optimisation attempts, of which only one — reducing the number of text lines — made any difference).

Questions

1. Are official C++ inference library builds for CUDA 12.8+ / Blackwell planned? If so I would happily test them.
2. Is there a supported way to exclude FlashAttention from an inference-only build in 3.3.x?
3. Is inference_lib_dist the right target for a minimal C++ inference library, or is there something narrower?

Happy to supply full configure and build logs if useful.