[bug] Can't compile varlen mamba with base image 25.11

[jlamypoirier]

🐞 Describe the Bug

We cannot compile the varlen implementation of Mamba from https://github.com/jxiw/varlen_mamba in the new base image 25.11. setup.py hard-codes some cuda architectures which are no longer supported, which is easily fixable, but then I'm getting cpp compile errors, ex.

      [6/10] /usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output /workspace/varlen_mamba/build/temp.linux-aarch64-cpython-312/csrc/selective_scan/selective_scan_bwd_fp16_complex.o.d -I/workspace/varlen_mamba/csrc/selective_scan -I/usr/local/lib/python3.12/dist-packages/torch/include -I/usr/local/lib/python3.12/dist-packages/torch/include/torch/csrc/api/include -I/usr/local/cuda/include -I/usr/include/python3.12 -c -c /workspace/varlen_mamba/csrc/selective_scan/selective_scan_bwd_fp16_complex.cu -o /workspace/varlen_mamba/build/temp.linux-aarch64-cpython-312/csrc/selective_scan/selective_scan_bwd_fp16_complex.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -O3 -std=c++17 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_BFLOAT16_OPERATORS__ -U__CUDA_NO_BFLOAT16_CONVERSIONS__ -U__CUDA_NO_BFLOAT162_OPERATORS__ -U__CUDA_NO_BFLOAT162_CONVERSIONS__ --expt-relaxed-constexpr --expt-extended-lambda --use_fast_math --ptxas-options=-v -lineinfo -gencode arch=compute_80,code=sm_80 -gencode arch=compute_87,code=sm_87 -gencode arch=compute_90,code=sm_90 --threads 4 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=selective_scan_cuda
      FAILED: [code=255] /workspace/varlen_mamba/build/temp.linux-aarch64-cpython-312/csrc/selective_scan/selective_scan_bwd_fp16_complex.o
      /usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output /workspace/varlen_mamba/build/temp.linux-aarch64-cpython-312/csrc/selective_scan/selective_scan_bwd_fp16_complex.o.d -I/workspace/varlen_mamba/csrc/selective_scan -I/usr/local/lib/python3.12/dist-packages/torch/include -I/usr/local/lib/python3.12/dist-packages/torch/include/torch/csrc/api/include -I/usr/local/cuda/include -I/usr/include/python3.12 -c -c /workspace/varlen_mamba/csrc/selective_scan/selective_scan_bwd_fp16_complex.cu -o /workspace/varlen_mamba/build/temp.linux-aarch64-cpython-312/csrc/selective_scan/selective_scan_bwd_fp16_complex.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -O3 -std=c++17 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_BFLOAT16_OPERATORS__ -U__CUDA_NO_BFLOAT16_CONVERSIONS__ -U__CUDA_NO_BFLOAT162_OPERATORS__ -U__CUDA_NO_BFLOAT162_CONVERSIONS__ --expt-relaxed-constexpr --expt-extended-lambda --use_fast_math --ptxas-options=-v -lineinfo -gencode arch=compute_80,code=sm_80 -gencode arch=compute_87,code=sm_87 -gencode arch=compute_90,code=sm_90 --threads 4 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=selective_scan_cuda
      /workspace/varlen_mamba/csrc/selective_scan/reverse_scan.cuh(132): error: namespace "cub" has no member "LaneId"
                : lane_id(cub::LaneId())
                               ^
      
      /workspace/varlen_mamba/csrc/selective_scan/reverse_scan.cuh(335): error: namespace "cub" has no member "CTA_SYNC"
                    cub::CTA_SYNC();
                         ^
      
      /workspace/varlen_mamba/csrc/selective_scan/reverse_scan.cuh(353): error: namespace "cub" has no member "CTA_SYNC"
                    cub::CTA_SYNC();
                         ^
      
      3 errors detected in the compilation of "/workspace/varlen_mamba/csrc/selective_scan/selective_scan_bwd_fp16_complex.cu".

[oleksost]

@jlamypoirier I think varlen mamba is very low priority at this point, new mixers (GDA, KDA) are varlen by default (CMIIW @tscholak )

[tscholak]

https://github.com/jxiw/varlen_mamba is a fork of an outdated version of upstream mamba-ssm.
I don't think we should take on the maintenance burden of keeping that 3rd party fork in sync with upstream. Maybe @jxiw will eventually have his varlen code merged into upstream mamba-ssm.
Besides, as @oleksost points out, mamba is low priority right now, so we can afford to wait this out.