[build] #1171 part 1: default multi-arch CUDA build does emit sm_80 SASS on master — mechanism needs re-diagnosis, and CI never covers the default path

[yingjerkao]

Split out from part 1 of #1171 (part 2 — unchecked kernel launches — is fixed by
#1175). Filing this as its own issue because the mechanism #1171 proposed does
not reproduce on current master, so "part 1" needs re-diagnosis before
anyone changes the architecture default.

#1171 was explicit that section 1 was a hypothesis rather than a diagnosis, and
that two variables changed between the failing and passing builds. This is that
verification.

What #1171 part 1 claimed

Neither CUDA_SEPARABLE_COMPILATION nor CUDA_RESOLVE_DEVICE_SYMBOLS is set
anywhere in the repo, so the lto_XX IR is never linked into SASS. The result
is a binary containing, for sm_80, neither SASS nor PTX — nothing the driver
can run.

What actually happens on master

1. Device symbol resolution has been enabled since 2023.
CytnxBKNDCMakeLists.cmake:252-255 sets both properties on the cytnx target:

set_target_properties(cytnx PROPERTIES CUDA_SEPARABLE_COMPILATION ON)
set_target_properties(cytnx PROPERTIES CUDA_RESOLVE_DEVICE_SYMBOLS ON)

Added in ec9af22f (2023-09-10), so this predates #1171 rather than being a
recent fix.

2. The device-link step is generated. A stock default-architecture configure
(-DUSE_CUDA=ON -DBUILD_PYTHON=OFF -DCMAKE_BUILD_TYPE=Debug, no arch override)
produces CMakeFiles/cytnx.dir/dlink.txt containing:

-dlink -dlto
--generate-code=arch=compute_75,code=[sm_75]
--generate-code=arch=compute_80,code=[sm_80]
--generate-code=arch=compute_86,code=[sm_86]
--generate-code=arch=compute_89,code=[sm_89]
--generate-code=arch=compute_90,code=[compute_90,sm_90]

Note code=[sm_XX] — real SASS, not lto_XX.

3. That step really does produce per-architecture SASS. Compiling a kernel
with the exact lto_XX flags CMake emits, then device-linking with the exact
dlink.txt flags:

$ nvcc -dc --generate-code=arch=compute_75,code=[lto_75] ... -c t.cu -o t.o
$ nvcc -dlink -dlto --generate-code=arch=compute_75,code=[sm_75] ... t.o -o t_dlink.o
$ cuobjdump -lelf t_dlink.o
ELF file    1: lto.sm_75.cubin
ELF file    2: lto.sm_80.cubin
ELF file    3: lto.sm_86.cubin
ELF file    4: lto.sm_89.cubin
ELF file    5: lto.sm_90.cubin

sm_80 SASS is present. The "neither SASS nor PTX for sm_80" mechanism does
not hold here.

Environment: CUDA 13.0.88 (same major as #1171), CMake 4.0.2, RTX 4070 Ti
SUPER. #1171 used CMake 3.28.3 on A100 — see the open question below.

The cache observation is real but benign

CMAKE_CUDA_ARCHITECTURES:STRING=75 in the cache is reproducible, but it is
intended and harmless:

• enable_language(CUDA) (CMakeLists.txt:270) writes CMake's own toolchain-aware
  default into the cache. The comment at :265-269 explains why that call has to
  come first.
• CMakeLists.txt:280 then sets a normal variable that shadows the cache for
  the directory scope, which is what reaches the compile line.
• Nothing in the repo consumes the cache entry — CytnxConfig.cmake.in does not
  reference architectures, and the only other reader is ci-gpu_tests.yml, which
  passes its own -D.

So it misleads someone inspecting CMakeCache.txt by hand, but it does not affect
what is built. Worth at most a clarifying comment, not a fix.

What is confirmed actionable

Section 3 of #1171 stands, and is the part worth doing:
ci-gpu_tests.yml:49,82 pins GPU_CUDA_ARCH: ${{ vars.GPU_CUDA_ARCH || '89' }}
and builds with -DCMAKE_CUDA_ARCHITECTURES="$GPU_CUDA_ARCH", so CI never
exercises the default multi-arch path that ordinary users get. Adding a job (or
a periodic one, since the full fat-binary build is slow) that configures with no
architecture override and at minimum checks the emitted codegen would catch a
regression here automatically — and would have settled this question without
anyone reproducing by hand.

Open question

The failure in #1171 was real: every GPU kernel returned zeros on sm_80, and
-DCMAKE_CUDA_ARCHITECTURES=80 fixed it. That still wants an explanation, since
the proposed mechanism does not survive the check above. Remaining candidate
variables:

• CMake version. 3.28.3 ([build] Default CUDA architectures: GPU kernels silently return zeros on sm_80, and a wrong-arch build never reports an error #1171) vs 4.0.2 (here). If 3.28 does not emit the
  dlink step under CUDA_RESOLVE_DEVICE_SYMBOLS + IPO, that alone explains the
  difference and makes this a minimum-CMake-version issue rather than an
  architecture-default one. This is the first thing to test.
• The --coverage stripping, which [build] Default CUDA architectures: GPU kernels silently return zeros on sm_80, and a wrong-arch build never reports an error #1171 explicitly names as the second thing
  that changed between its failing and passing builds.
• Something environment-specific to that A100 host.

Please do not "fix" this by changing the architecture default

Changing CMAKE_CUDA_ARCHITECTURES at CMakeLists.txt:280 alters which SASS
ships to every user, and the current list is deliberate (the CUDA 13 sm_75 floor
at :275-278 is load-bearing). Until the mechanism is confirmed, a change there
would be a guess with wide blast radius.

Suggested order of work

1. Reproduce on CMake 3.28.3 — check whether dlink.txt is generated at all.
2. If it is not, raise cmake_minimum_required and close this out.
3. Independently, add the CI job that builds the default multi-arch path.

Reproducing the check above

cmake -S . -B /tmp/archtest -DUSE_CUDA=ON -DUSE_CUTENSOR=OFF -DUSE_CUQUANTUM=OFF \
      -DBUILD_PYTHON=OFF -DUSE_HPTT=OFF -DCMAKE_BUILD_TYPE=Debug
cat /tmp/archtest/CMakeFiles/cytnx.dir/dlink.txt   # expect code=[sm_XX]

Relates to #1171 (part 1); part 2 fixed by #1175.

[yingjerkao]

Tested on CMake 3.28.3 — hypothesis eliminated

The issue named the CMake version as the prime suspect and the first thing to
test. Done: 3.28.3 behaves identically to 4.0.2. The device link happens, and
sm_80 SASS survives all the way into a linked executable.

Environment: CMake 3.28.3 (pip, isolated venv), CUDA 13.0.88, GCC, Ubuntu —
matching #1171's reported toolchain except the GPU.

1. Configure output is byte-identical between CMake versions

Same configure as before, run with 3.28.3:

$ cat CMakeFiles/cytnx.dir/dlink.txt   # 3.28.3
-dlink -dlto
--generate-code=arch=compute_75,code=[sm_75]
--generate-code=arch=compute_80,code=[sm_80]
--generate-code=arch=compute_86,code=[sm_86]
--generate-code=arch=compute_89,code=[sm_89]
--generate-code=arch=compute_90,code=[compute_90,sm_90]

Diffed against the 4.0.2 run: identical. The compile line is lto_XX and the
cache is 75 in both, exactly as reported.

2. The real CMake build path embeds sm_80 SASS

The earlier check used raw nvcc, which left open whether CMake's own build path
behaves the same. A minimal static-library project mirroring Cytnx's CUDA settings
— 6-arch fat-binary default, CMAKE_INTERPROCEDURAL_OPTIMIZATION TRUE,
CUDA_SEPARABLE_COMPILATION ON, CUDA_RESOLVE_DEVICE_SYMBOLS ON — built with
3.28.3:

$ ar t libmini.a
k.cu.o
cmake_device_link.o          <-- device link ran

$ cuobjdump -lelf libmini.a
ELF file 1: lto.sm_75.cubin   ELF file 2: lto.sm_80.cubin
ELF file 3: lto.sm_86.cubin   ELF file 4: lto.sm_89.cubin
ELF file 5: lto.sm_90.cubin
$ cuobjdump -lptx libmini.a
PTX file 1: libmini.1.sm_90.ptx

3. It survives into the linked executable

A static archive only contributes members whose symbols are referenced, so
cmake_device_link.o carrying the SASS could in principle have been dropped at
link time — a plausible refinement of the original mechanism. It is not dropped:

$ cuobjdump -lelf app        # executable linked against libmini.a
ELF file 1: lto.sm_75.cubin   ELF file 2: lto.sm_80.cubin
ELF file 3: lto.sm_86.cubin   ELF file 4: lto.sm_89.cubin
ELF file 5: lto.sm_90.cubin

So the full chain — LTO compile → device link → archive → executable — retains
sm_80 SASS on the reporter's CMake version.

The CMakeLists code has not changed since #1171 was filed

#1171 was opened 2026-08-02. The last commit touching the architecture block
(CMakeLists.txt:256-285) is f78217f5, 2026-07-09. So this is the same code
that produced the original failure — not a version that was fixed in between.

Where that leaves part 1

The "no runnable code for sm_80" mechanism is now falsified across both CMake
versions and through the complete build path. The symptom in #1171 was real and
still has no confirmed cause. Remaining candidates, in the order I'd rank them:

1. The --coverage variable. [build] Default CUDA architectures: GPU kernels silently return zeros on sm_80, and a wrong-arch build never reports an error #1171 names it as the second thing that changed
   between its failing and passing builds, and that session also hit the
   __gcov_* link failure. A build dir carrying a mix of instrumented and
   stripped CUDA objects is a plausible source of a binary that does not behave
   like either configuration — but that is a hypothesis, not a diagnosis, and it
   needs the same kind of check this comment applied to the CMake theory.
2. Something specific to that A100 host (driver 580.126.20, 3x A100-SXM4-80GB).
3. Reproducing on the original machine with cuobjdump -lelf on the actual
   gpu_test_main would settle it immediately — that single command distinguishes
   "no image for sm_80" from every other explanation.

The recommendation not to change the architecture default stands, now more
firmly: two proposed mechanisms for a defect in it have failed to reproduce.

Section 3 of #1171 is unaffected and remains the actionable item — CI pins
GPU_CUDA_ARCH and never builds the default multi-arch path.

Reproducing this

python3 -m venv /tmp/cmakeenv && /tmp/cmakeenv/bin/pip install cmake==3.28.3
/tmp/cmakeenv/bin/cmake -S . -B /tmp/archtest328 -DUSE_CUDA=ON -DUSE_CUTENSOR=OFF \
  -DUSE_CUQUANTUM=OFF -DBUILD_PYTHON=OFF -DUSE_HPTT=OFF -DCMAKE_BUILD_TYPE=Debug
cat /tmp/archtest328/CMakeFiles/cytnx.dir/dlink.txt   # expect code=[sm_XX]

Caveat unchanged from the issue body: this uses a small kernel, not the full
library. It establishes that the build system emits and preserves per-arch SASS;
it does not prove every Cytnx translation unit does.

[yingjerkao]

--coverage tested — also does not reproduce

I ranked --coverage as the leading remaining candidate. Tested now, and it is
dead too. All three proposed mechanisms for part 1 have failed to reproduce.

Setup

Two otherwise-identical static-library projects with Cytnx's CUDA settings
(6-arch default, forced IPO, separable compilation, resolve-device-symbols),
differing only by the coverage block copied verbatim from CMakeLists.txt:437-441:

target_compile_options(minicov PRIVATE
  $<$<COMPILE_LANGUAGE:CXX>:--coverage>
  $<$<COMPILE_LANGUAGE:CUDA>:-Xcompiler=--coverage>)
target_link_options(minicov INTERFACE --coverage)

Built with CMake 3.28.3 / CUDA 13.0.88, then run on an sm_89 GPU.

Result

Build	SASS in executable	Runtime
with --coverage (RUN_TESTS=ON)	sm_75 sm_80 sm_86 sm_89 sm_90	value=42 CORRECT
without --coverage	sm_75 sm_80 sm_86 sm_89 sm_90	value=42 CORRECT

The coverage build genuinely carries instrumentation (43 __gcov* symbols), so
the flag really was applied. It changes neither the emitted SASS nor the result.

Negative control — the harness does detect this failure

A passing test is worthless if the harness cannot fail. Rebuilding the same
project for 75-real only and running it on the sm_89 box:

SASS: sm_75
launch=named symbol not found   value=0   [ZEROS/WRONG]

That is #1171's exact symptom — zeros, and with no launch check nothing reported
— and it matches the named symbol not found demonstration in #1175. So the
harness is sensitive to precisely the failure being investigated, and the
default-architecture builds above genuinely pass.

Incidental finding that matters here

My first control used 75-real 80-real and passed on sm_89. That is correct
CUDA behavior, not a harness bug: cubins are binary-compatible across minor
versions within a major generation, so sm_80 SASS runs on sm_86 and sm_89.

This makes the default list more robust than #1171 assumed — the single 80-real
entry covers all of Ampere/Ada 8.x. It also sharpens the puzzle: the reporting
host was an A100, i.e. sm_80 exactly, which the default list targets natively.

Status of part 1

Hypothesis	Verdict
Device link never happens; no SASS for sm_80	Falsified — dlink runs, SASS reaches the executable
CMake version (3.28.3 vs 4.0.2)	Falsified — byte-identical codegen
--coverage on CUDA objects	Falsified — same SASS, correct results

The symptom in #1171 remains real and unexplained. What has been ruled out is
everything reproducible from the repository's own configuration. That points at
something specific to the original machine or to the state of that particular
build tree, neither of which can be settled from here.

One command on the A100 host still settles it, and is now the only cheap step
left:

cuobjdump -lelf gpu_test_main | grep -o 'sm_[0-9]*' | sort -u

If sm_80 is absent, the original mechanism is right about that machine and
everything above is beside the point. If present, the cause is elsewhere entirely
and part 1 should probably be closed in favour of section 3.

Section 3 (CI never builds the default multi-arch path) is untouched by any of
this and remains the actionable item.

Caveat

Same as before: small kernel, not the full library. These tests establish what the
build system does with Cytnx's CUDA settings; they do not prove every Cytnx
translation unit behaves identically.