Skip to content

✨ Python bindings for CudaCogReader - #58

Merged
weiji14 merged 27 commits into
mainfrom
dlpack_to_cupy
Jan 17, 2026
Merged

weiji14 merged 27 commits into
mainfrom
dlpack_to_cupy

Conversation

@weiji14

@weiji14 weiji14 commented Oct 28, 2025 •

Copy link
Copy Markdown
Owner

Allow Python users to use the nvTIFF backend, reading a GeoTIFF into a DLPack that can be consumed by CuPy, Pytorch (2.9+), or any other Python CUDA GPU library that implements the Python Specification for DLPack.

Preview at https://cog3pio--58.org.readthedocs.build/en/58/api/#dlpack

Usage:

import cupy as cp
from cog3pio import CudaCogReader

cog = CudaCogReader(
   path="https://github.com/rasterio/rasterio/raw/1.4.3/tests/data/RGBA.byte.tif"
)
array: cp.ndarray = cp.from_dlpack(cog)
array.shape
# (2271752,)
array.dtype
# dtype('uint8')

TODO:

  • Initial boilerplate implementation
  • Handle optional 'cuda' extras dependency properly on CI
    • GitHub Actions (linux-x86_64 and linux-aarch64 only)
    • readthedocs
  • Pass raw pointer safely? (somewhat done in 6ed6ceb, but need to follow up on this)
  • Handle __dlpack__ kwargs for cupy.from_dlpack()
    • stream - only legacy default stream (None or 1) and per-thread default stream (2), stream >=3 is not supported yet by cudarc I think
    • max_version
      - [ ] dl_device (TODO next time)
      - [ ] copy (TODO next time)
  • Remove unwraps, proper error messaging
  • Resolve memory leak, or use of invalid stream somehow (Stream handling fixed in ♻️ Refactor to pass in per-thread default stream to dlpack method #67 and 964ef03, but there might be a memory leak somewhere still 🙈)
  • Add benchmark test, but use pytest.importorskip to have it run when 'cupy' is installed 🚩
  • More documentation improvements
  • Xarray integration (TODO in separate PR?)

External things to do to improve implementation here:

  • Implement Error trait on nvtiff_sys::result::NvTiffError
  • Make a cupy-xarray 0.1.5 release without cupy as dependency so that cupy-cuda13x can be used instead

⚠️ Limitations/Notes:

  • Initial implementation has correct 1D shape in cupy 🎉 But will need to manually reshape to 2D or 3D tensor yourself for now 🙈
  • Tensor values are nonsense, possibly due to raw pointer not being passed around safely 🙈 Workaround (discovered accidentally) is to instantiate the CudaCogReader class instance twice?! 😅
    • Edit: turns out to be a tokio runtime issue? Fixed in commit 34ccbc3 by using tokio::runtime::Builder::new_multi_thread()
  • Need to figure out the wheel build situation. Do I create two packages for the Linux builds, one without nvTIFF, and one with nvTIFF bindings bundled? Yes, will need to, otherwise users will still hit into ImportError: libnvtiff.so.0: cannot open shared object file: No such file or directory when running import cog3pio without nvTIFF installed (e.g. if they don't have a CUDA GPU). Might need to look into https://wheelnext.dev/proposals/pep817_wheel_variant_support/
    • One idea was to have the nvTIFF support added to the free-threaded builds only. But blocked by CuPy not having free-threaded builds yet (as of v14.0.0rc1). Torch 2.9.0 does have 314t wheels though 👀

References:

Follow-up to #57, part of #26

@weiji14 weiji14 added this to the 0.1.0 milestone Oct 28, 2025
@weiji14 weiji14 self-assigned this Oct 28, 2025
@weiji14 weiji14 added the feature New feature or request label Oct 28, 2025
Setting up boilerplate code for using CudaCogReader in Python to read GeoTIFFs into a DLPack and then transfer to CuPy! Temporarily using unsafe impl Send and Sync to workaround raw pointer in CudaCogReader not being able to be shared between threads safely. Still need to properly handle some kwargs in __dlpack__ for cupy.from_dlpack(). A cupy array is returned with the correct shape, but numbers are wrong, possibly because pointer isn't managed properly.
Using bundled bindings to get compilation to work on CI. Tests with cuda should still be skipped since GPU CI is not available. Also bumped from cudarc 0.17.3 to 0.17.4.
Since nvTIFF isn't on osx, and can't be bothered with Windows yet. Do `maturin build --features cuda` on Linux CI tests (Python) only.
CudaCogReader might not be available on some platforms, so hide it behind some gates.
For some reason, calling CudaCogReader twice makes things work, i.e. the returned cupy.ndarray has the correct numbers. Thinking it might be some CUDA stream issue (https://docs.cupy.dev/en/v13.6.0/user_guide/basic.html#current-stream), but cupy should already be using the default null stream by default.

Putting some print() and dbg!() statements here and there. Bumped cupy-cuda12x to cupy-cuda13x.
@weiji14
weiji14 force-pushed the dlpack_to_cupy branch 3 times, most recently from 5e0408c to 0056bbf Compare October 30, 2025 00:17
Fix `Unable to find libclang: "couldn't find any valid shared libraries matching: ['libclang.so', 'libclang-*.so', 'libclang.so.*', 'libclang-*.so.*'], set the `LIBCLANG_PATH` environment variable to a path where one of these files can be found (invalid: [])"`. Need to install this inside the manylinux_2_28 docker container.
Depending on which manylinux_2_28 docker image is pulled for each target arch, the underlying distribution could either be AlmaLinux or Ubuntu based, so need to handle either way of installing nvTIFF and clang-dev.
Fix nvtiff-sys compilation errors by installing missing CUDA runtime dependencies (cuda-crt and cuda-cudart-devel) and patching the nvtiff.h file following https://docs.rs/nvtiff-sys/0.1.2/nvtiff_sys/#instructions
Default `pyo3` flag set in pyproject.toml is overidden when passing `--features` flag to maturin build, so need to set `cuda,pyo3` instead. Also copy code from e75d171 to free-threaded build section.
Too tricky to get nvTIFF working on armv7, s390x and ppc64le due to some linker error like `/usr/armv7-unknown-linux-gnueabihf/lib/gcc/armv7-unknown-linux-gnueabihf/7.5.0/../../../../armv7-unknown-linux-gnueabihf/bin/ld: cannot find -lnvtiff`, so disabling them on those platforms.
@weiji14
weiji14 force-pushed the dlpack_to_cupy branch 4 times, most recently from 715ca46 to 1c5965f Compare November 4, 2025 01:47
@weiji14
weiji14 force-pushed the dlpack_to_cupy branch 4 times, most recently from f0d0e16 to 1c6108a Compare November 4, 2025 03:53
Need to include 'cuda' feature flag to maturin on ReadtheDocs, and get libnvtiff-dev from conda-forge. Added a warning to the docstring indicating that CudaCogReader is experimental, and only available on linux-x86_64 and linux-aarch64 builds.
@weiji14
weiji14 force-pushed the dlpack_to_cupy branch 2 times, most recently from 4c8a4b2 to ae17b83 Compare November 4, 2025 05:57
Point to where the header files are located. nvtiff.h is in $CONDA_PREFIX/include. cuda_runtime.h and crt/host_config.h are in $CONDA_PREFIX/targets/x86_64-linux/include.
Not sure if raw pointer in CudaCogReader is thread-safe enough to do `unsafe impl Send/Sync`, so using unsendable instead for now. Xref https://pyo3.rs/v0.27.1/migration.html#pyclass-structs-must-now-be-send-or-unsendable
Check that stream and max_version arguments are valid. Currently only supporting stream=1 or None, and DLPack version 1.x (dlpark is using DLPack 1.1). Have added some docstrings for these parameters. Not implementing copy kwarg yet though.
Bump nvtiff-sys from 0.1.2 to 0.1.3 to get Error trait on NvtiffStatusError, and then we can cast to string and pass error message to PyValueError.
Brute force symlinking to get nvtiff-sys to compile with conda-forge's libnvtiff that is under $CONDA_PREFIX/include/ instead of $CONDA_PREFIX/targets/x86_64-linux/include/ where most other header files are. One key part is to use RUSTFLAGS instead of LD_LIBRARY_PATH to actually get rustc to search the correct lib/ folder for the .so files.
Update readthedocs and GitHub Actions CI to install nvtiff 0.6.0. Remove patch for nvtiff 0.5.0 in regard to removing default arguments.
@weiji14

weiji14 commented Jan 7, 2026

Copy link
Copy Markdown
Owner Author

Couldn't use miri, so trying out cargo-careful to detect where memory leak might be happening:

$ cargo +nightly careful test -Zcareful-sanitizer=leak --features cuda
Using sanitizier `leak`.
Preparing a careful sysroot (target: x86_64-unknown-linux-gnu, sanitizer: leak)... done
   Compiling cog3pio v0.0.1 (/home/user/projects/cog3pio)
    Finished `test` profile [optimized + debuginfo] target(s) in 2.79s
     Running unittests src/lib.rs (target/x86_64-unknown-linux-gnu/debug/deps/cog3pio-b49d40a5c9410edb)

running 8 tests
test io::geotiff::tests::test_read_geotiff ... ok
test io::geotiff::tests::test_cogreader_dlpack ... ok
test io::geotiff::tests::test_read_geotiff_uint16_dtype ... ok
test io::geotiff::tests::test_read_geotiff_float16_dtype ... ok
test io::geotiff::tests::test_cogreader_transform ... ok
test io::nvtiff::tests::cudacogreader_dlpack ... FAILED
test io::geotiff::tests::test_cogreader_num_samples ... ok
test io::geotiff::tests::test_read_geotiff_multi_band ... ok

failures:

---- io::nvtiff::tests::cudacogreader_dlpack stdout ----
[src/io/nvtiff.rs:95:9] status_cpustream = 0
[src/io/nvtiff.rs:101:9] status_parse = 0
[src/io/nvtiff.rs:108:9] status_numimages = 0
[src/io/nvtiff.rs:120:9] status_imageinfo = 0
[src/io/nvtiff.rs:148:9] num_bytes = 24
[src/io/nvtiff.rs:200:9] status_decoder = 0
[src/io/nvtiff.rs:214:9] status_check = 4

thread 'io::nvtiff::tests::cudacogreader_dlpack' (45078) panicked at src/io/nvtiff.rs:313:63:
called `Result::unwrap()` on an `Err` value: StatusError(TiffNotSupported)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    io::nvtiff::tests::cudacogreader_dlpack

test result: FAILED. 7 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.82s


=================================================================
==45070==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 8 byte(s) in 1 object(s) allocated from:
    #0 0x55f433c8286f  (/home/user/projects/cog3pio/target/x86_64-unknown-linux-gnu/debug/deps/cog3pio-b49d40a5c9410edb+0x2fe86f) (BuildId: 18d388fb478e70c86ff1e3d919d75fcfd65da7a6)
    #1 0x7f2ecf937beb  (/usr/lib/x86_64-linux-gnu/libnvtiff.so.0+0x137beb) (BuildId: 716a5c18299b658fdbea4e6b3bb728ce8a48aa6a)

Indirect leak of 1664 byte(s) in 19 object(s) allocated from:
    #0 0x55f433c8286f  (/home/user/projects/cog3pio/target/x86_64-unknown-linux-gnu/debug/deps/cog3pio-b49d40a5c9410edb+0x2fe86f) (BuildId: 18d388fb478e70c86ff1e3d919d75fcfd65da7a6)
    #1 0x7f2ecf937beb  (/usr/lib/x86_64-linux-gnu/libnvtiff.so.0+0x137beb) (BuildId: 716a5c18299b658fdbea4e6b3bb728ce8a48aa6a)

Indirect leak of 160 byte(s) in 1 object(s) allocated from:
    #0 0x55f433c82ec3  (/home/user/projects/cog3pio/target/x86_64-unknown-linux-gnu/debug/deps/cog3pio-b49d40a5c9410edb+0x2feec3) (BuildId: 18d388fb478e70c86ff1e3d919d75fcfd65da7a6)
    #1 0x7f2ecf858cef  (/usr/lib/x86_64-linux-gnu/libnvtiff.so.0+0x58cef) (BuildId: 716a5c18299b658fdbea4e6b3bb728ce8a48aa6a)

Indirect leak of 64 byte(s) in 1 object(s) allocated from:
    #0 0x55f433c8286f  (/home/user/projects/cog3pio/target/x86_64-unknown-linux-gnu/debug/deps/cog3pio-b49d40a5c9410edb+0x2fe86f) (BuildId: 18d388fb478e70c86ff1e3d919d75fcfd65da7a6)
    #1 0x7f2ecf937beb  (/usr/lib/x86_64-linux-gnu/libnvtiff.so.0+0x137beb) (BuildId: 716a5c18299b658fdbea4e6b3bb728ce8a48aa6a)
    #2 0x0001aaaa0002  (<unknown module>)

SUMMARY: LeakSanitizer: 1896 byte(s) leaked in 22 allocation(s).
error: test failed, to rerun pass `--lib`

Caused by:
  process didn't exit successfully: `/home/user/projects/cog3pio/target/x86_64-unknown-linux-gnu/debug/deps/cog3pio-b49d40a5c9410edb` (exit status: 23)
note: test exited abnormally; to see the full output pass --no-capture to the harness.

Been having trouble nvTIFF v0.6.0 on my Debian 13 system lately from nvidia repos (not in https://developer.download.nvidia.com/compute/cuda/repos/debian13/x86_64/, so manually downloading from debian12 instead). That, and CUDA 13 getting thrown into the mix makes things trickier, so this might take a while to work.

Refactor PyCudaCogReader to have CUDA stream be instantiated in `__dlpack__` instead of `new` method, enabling TIFF decoding on per-thread default stream instead of just legacy default stream. The __dlpack_device__ still returns a hardcoded GPU:0 device int, but this is now linked to the PyCudaCogReader.device attribute.
By switching tokio runtime from `new_current_thread` to `new_multi_thread`. Reason isn't entirely clear, but found out that something in the `path_to_stream` function must have been the issue because local files didn't need the double CudaCogReader call hack, so something was wrong with remote HTTP files. ObjectStore wasn't the issue, but tokio was for some reason!
@codspeed

codspeed Bot commented Jan 12, 2026 •

Copy link
Copy Markdown

CodSpeed Performance Report

Merging this PR will degrade performance by 22.96%

Comparing dlpack_to_cupy (7ea89ac) with main (f8e9664)

Summary

❌ 1 (👁 1) regressed benchmark
✅ 6 untouched benchmarks

Performance Changes

Benchmark BASE HEAD Efficiency
👁 test_read_geotiff_local 239.2 µs 310.5 µs -22.96%

@weiji14

weiji14 commented Jan 12, 2026

Copy link
Copy Markdown
Owner Author

Quick update to say that while I'm not 100% confident that all memory leaks are plugged, some refactors to the CUDA stream handling (from #67 + commit 964ef03) and a crucial patch fixing the strange need for a double CudaCogReader call (in commit 34ccbc3) means that I'm a lot happier about the implementation here. At such, I'm ready to call this PR near satisfactory as an initial proof of concept.

Remaining polish will be to add more unit tests and documentation. There are a few more nice-to-haves to satisfy the producer-side of https://dmlc.github.io/dlpack/latest/python_spec.html#implementation, but those can be chipped away in follow-up PRs. That, and the xarray integration should come separately in the future.

Hopefully should be able to merge before end of the week (barring any new hiccups)!

Compare reading into a cupy.ndarray between cog3pio and rasterio. First one is using cog3pio.CudaCogReader followed by a cupy.from_dlpack; second one is using rasterio.open+read followed by cupy.asarray. Using cupyx.profiler.benchmark to calculate both GPU and CPU times, reported to stdout.
Finalizing some docstring wording and comment out some dbg! statements showing up in stderr. Consistently recommending cuda 13 versions of libraries, and added extra backticks to some variables/methods in the docstrings. Couldn't quite get intersphinx to work yet. Also checking the GPU decoding box in the main README.md file.
@weiji14
weiji14 marked this pull request as ready for review January 17, 2026 07:17

@weiji14 weiji14 left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, wish this was more perfect, but let's not slow down progress. LGTM 🚀

Comment thread .github/workflows/ci.yml
Comment on lines 96 to +100
- name: Build free-threaded wheels
uses: PyO3/maturin-action@aef21716ff3dcae8a1c301d23ec3e4446972a6e3 # v1.49.1
with:
target: ${{ matrix.platform.target }}
args: --release --out dist --interpreter python3.13t
args: --features ${{ matrix.feat-flags || 'pyo3' }} --release --out dist --interpreter python3.13t

@weiji14 weiji14 Jan 17, 2026 •

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • One idea was to have the nvTIFF support added to the free-threaded builds only. But blocked by CuPy not having free-threaded builds yet (as of v14.0.0rc1). Torch 2.9.0 does have 314t wheels though 👀

Potentially, could just go ahead and add CudaCogReader/nvTIFF support on free-threaded builds only, and expect a future version of CuPy with free-threading support to be able to consume the DLPack tensor? Edit: cupy=14.1.0 has free-threaded wheels for Linux, so opened #96 to build 3.14t wheels.

Will just have CudaCogReader/nvTIFF support on both non-free-threaded and free-threaded in CI for now, and think about this before the v0.1.0 release.

Comment thread src/python/cudacog.rs

/// Get device type and device ID in DLPack format.
///
/// Meant for use by ``from_dlpack()``.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note to add intersphinx link to https://data-apis.org/array-api/2024.12/API_specification/generated/array_api.from_dlpack.html at some point once MyST/Jupyter-Book 2 supports it.

Comment thread src/python/cudacog.rs
Comment on lines +115 to +116
max_version: Option<(u32, u32)>,
kwargs: Option<&Bound<'_, PyAny>>,

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: implement dl_device and copy parameters to conform to __dlpack__ standard.

Suggested change
max_version: Option<(u32, u32)>,
kwargs: Option<&Bound<'_, PyAny>>,
max_version: Option<(u32, u32)>,
// dl_device: Option<(dlpark::ffi::DeviceType, u8)>,
// copy: Option<bool>,
kwargs: Option<&Bound<'_, PyAny>>,

Comment thread src/python/cudacog.rs
Comment on lines +128 to +130
3.. => Err(PyNotImplementedError::new_err(
"only legacy default stream (1) or per-thread default stream (2) is
supported for now, got {s_uint}",

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May need to wait for cudarc to support arbitrary stream integers >2? Or could we just decode on legacy or per-thread default stream and then do device-to-device copy to another stream?

Comment thread src/python/cudacog.rs

Ok(Self {
inner: cog,
device: 0,

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will consider not having the CUDA device hardcoded to 0 here. But then I'm thinking, could we have just a single CogReader struct, and let the device (CPU/CUDA/...) be set, which then points to different device decoder implementations?

@weiji14
weiji14 merged commit bdb1e0b into main Jan 17, 2026
20 checks passed
@weiji14
weiji14 deleted the dlpack_to_cupy branch January 17, 2026 07:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant