Repository navigation
✨ Python bindings for CudaCogReader - #58
Conversation
682fae0 to
58cdee2
Compare
Setting up boilerplate code for using CudaCogReader in Python to read GeoTIFFs into a DLPack and then transfer to CuPy! Temporarily using unsafe impl Send and Sync to workaround raw pointer in CudaCogReader not being able to be shared between threads safely. Still need to properly handle some kwargs in __dlpack__ for cupy.from_dlpack(). A cupy array is returned with the correct shape, but numbers are wrong, possibly because pointer isn't managed properly.
58cdee2 to
6499ec9
Compare
Using bundled bindings to get compilation to work on CI. Tests with cuda should still be skipped since GPU CI is not available. Also bumped from cudarc 0.17.3 to 0.17.4.
Since nvTIFF isn't on osx, and can't be bothered with Windows yet. Do `maturin build --features cuda` on Linux CI tests (Python) only.
CudaCogReader might not be available on some platforms, so hide it behind some gates.
For some reason, calling CudaCogReader twice makes things work, i.e. the returned cupy.ndarray has the correct numbers. Thinking it might be some CUDA stream issue (https://docs.cupy.dev/en/v13.6.0/user_guide/basic.html#current-stream), but cupy should already be using the default null stream by default. Putting some print() and dbg!() statements here and there. Bumped cupy-cuda12x to cupy-cuda13x.
76e29f1 to
aed13e5
Compare
Install nvTIFF binaries from nvidia repos following instructions on https://developer.nvidia.com/nvtiff-0-5-0-download-archive?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=24.04&target_type=deb_network. Could have tried to get it from PyPI following https://docs.nvidia.com/cuda/nvtiff/installation.html#pypi, but then would need to figure out the lib paths and stuff.
aed13e5 to
cb05982
Compare
5e0408c to
0056bbf
Compare
Fix `Unable to find libclang: "couldn't find any valid shared libraries matching: ['libclang.so', 'libclang-*.so', 'libclang.so.*', 'libclang-*.so.*'], set the `LIBCLANG_PATH` environment variable to a path where one of these files can be found (invalid: [])"`. Need to install this inside the manylinux_2_28 docker container.
0056bbf to
a409689
Compare
Depending on which manylinux_2_28 docker image is pulled for each target arch, the underlying distribution could either be AlmaLinux or Ubuntu based, so need to handle either way of installing nvTIFF and clang-dev.
Fix nvtiff-sys compilation errors by installing missing CUDA runtime dependencies (cuda-crt and cuda-cudart-devel) and patching the nvtiff.h file following https://docs.rs/nvtiff-sys/0.1.2/nvtiff_sys/#instructions
Default `pyo3` flag set in pyproject.toml is overidden when passing `--features` flag to maturin build, so need to set `cuda,pyo3` instead. Also copy code from e75d171 to free-threaded build section.
Too tricky to get nvTIFF working on armv7, s390x and ppc64le due to some linker error like `/usr/armv7-unknown-linux-gnueabihf/lib/gcc/armv7-unknown-linux-gnueabihf/7.5.0/../../../../armv7-unknown-linux-gnueabihf/bin/ld: cannot find -lnvtiff`, so disabling them on those platforms.
715ca46 to
1c5965f
Compare
f0d0e16 to
1c6108a
Compare
Need to include 'cuda' feature flag to maturin on ReadtheDocs, and get libnvtiff-dev from conda-forge. Added a warning to the docstring indicating that CudaCogReader is experimental, and only available on linux-x86_64 and linux-aarch64 builds.
4c8a4b2 to
ae17b83
Compare
Point to where the header files are located. nvtiff.h is in $CONDA_PREFIX/include. cuda_runtime.h and crt/host_config.h are in $CONDA_PREFIX/targets/x86_64-linux/include.
ae17b83 to
6cc193b
Compare
Not sure if raw pointer in CudaCogReader is thread-safe enough to do `unsafe impl Send/Sync`, so using unsendable instead for now. Xref https://pyo3.rs/v0.27.1/migration.html#pyclass-structs-must-now-be-send-or-unsendable
Check that stream and max_version arguments are valid. Currently only supporting stream=1 or None, and DLPack version 1.x (dlpark is using DLPack 1.1). Have added some docstrings for these parameters. Not implementing copy kwarg yet though.
Bump nvtiff-sys from 0.1.2 to 0.1.3 to get Error trait on NvtiffStatusError, and then we can cast to string and pass error message to PyValueError.
Brute force symlinking to get nvtiff-sys to compile with conda-forge's libnvtiff that is under $CONDA_PREFIX/include/ instead of $CONDA_PREFIX/targets/x86_64-linux/include/ where most other header files are. One key part is to use RUSTFLAGS instead of LD_LIBRARY_PATH to actually get rustc to search the correct lib/ folder for the .so files.
Update readthedocs and GitHub Actions CI to install nvtiff 0.6.0. Remove patch for nvtiff 0.5.0 in regard to removing default arguments.
|
Couldn't use miri, so trying out cargo-careful to detect where memory leak might be happening: Been having trouble nvTIFF v0.6.0 on my Debian 13 system lately from nvidia repos (not in https://developer.download.nvidia.com/compute/cuda/repos/debian13/x86_64/, so manually downloading from debian12 instead). That, and CUDA 13 getting thrown into the mix makes things trickier, so this might take a while to work. |
Refactor PyCudaCogReader to have CUDA stream be instantiated in `__dlpack__` instead of `new` method, enabling TIFF decoding on per-thread default stream instead of just legacy default stream. The __dlpack_device__ still returns a hardcoded GPU:0 device int, but this is now linked to the PyCudaCogReader.device attribute.
3478868 to
964ef03
Compare
By switching tokio runtime from `new_current_thread` to `new_multi_thread`. Reason isn't entirely clear, but found out that something in the `path_to_stream` function must have been the issue because local files didn't need the double CudaCogReader call hack, so something was wrong with remote HTTP files. ObjectStore wasn't the issue, but tokio was for some reason!
CodSpeed Performance ReportMerging this PR will degrade performance by 22.96%Comparing Summary
Performance Changes
|
|
Quick update to say that while I'm not 100% confident that all memory leaks are plugged, some refactors to the CUDA stream handling (from #67 + commit 964ef03) and a crucial patch fixing the strange need for a double Remaining polish will be to add more unit tests and documentation. There are a few more nice-to-haves to satisfy the producer-side of https://dmlc.github.io/dlpack/latest/python_spec.html#implementation, but those can be chipped away in follow-up PRs. That, and the xarray integration should come separately in the future. Hopefully should be able to merge before end of the week (barring any new hiccups)! |
Compare reading into a cupy.ndarray between cog3pio and rasterio. First one is using cog3pio.CudaCogReader followed by a cupy.from_dlpack; second one is using rasterio.open+read followed by cupy.asarray. Using cupyx.profiler.benchmark to calculate both GPU and CPU times, reported to stdout.
Finalizing some docstring wording and comment out some dbg! statements showing up in stderr. Consistently recommending cuda 13 versions of libraries, and added extra backticks to some variables/methods in the docstrings. Couldn't quite get intersphinx to work yet. Also checking the GPU decoding box in the main README.md file.
weiji14
left a comment
There was a problem hiding this comment.
Ok, wish this was more perfect, but let's not slow down progress. LGTM 🚀
| - name: Build free-threaded wheels | ||
| uses: PyO3/maturin-action@aef21716ff3dcae8a1c301d23ec3e4446972a6e3 # v1.49.1 | ||
| with: | ||
| target: ${{ matrix.platform.target }} | ||
| args: --release --out dist --interpreter python3.13t | ||
| args: --features ${{ matrix.feat-flags || 'pyo3' }} --release --out dist --interpreter python3.13t |
There was a problem hiding this comment.
- One idea was to have the nvTIFF support added to the free-threaded builds only. But blocked by CuPy not having free-threaded builds yet (as of v14.0.0rc1). Torch 2.9.0 does have 314t wheels though 👀
Potentially, could just go ahead and add CudaCogReader/nvTIFF support on free-threaded builds only, and expect a future version of CuPy with free-threading support to be able to consume the DLPack tensor? Edit: cupy=14.1.0 has free-threaded wheels for Linux, so opened #96 to build 3.14t wheels.
Will just have CudaCogReader/nvTIFF support on both non-free-threaded and free-threaded in CI for now, and think about this before the v0.1.0 release.
|
|
||
| /// Get device type and device ID in DLPack format. | ||
| /// | ||
| /// Meant for use by ``from_dlpack()``. |
There was a problem hiding this comment.
Note to add intersphinx link to https://data-apis.org/array-api/2024.12/API_specification/generated/array_api.from_dlpack.html at some point once MyST/Jupyter-Book 2 supports it.
| max_version: Option<(u32, u32)>, | ||
| kwargs: Option<&Bound<'_, PyAny>>, |
There was a problem hiding this comment.
TODO: implement dl_device and copy parameters to conform to __dlpack__ standard.
| max_version: Option<(u32, u32)>, | |
| kwargs: Option<&Bound<'_, PyAny>>, | |
| max_version: Option<(u32, u32)>, | |
| // dl_device: Option<(dlpark::ffi::DeviceType, u8)>, | |
| // copy: Option<bool>, | |
| kwargs: Option<&Bound<'_, PyAny>>, |
| 3.. => Err(PyNotImplementedError::new_err( | ||
| "only legacy default stream (1) or per-thread default stream (2) is | ||
| supported for now, got {s_uint}", |
There was a problem hiding this comment.
May need to wait for cudarc to support arbitrary stream integers >2? Or could we just decode on legacy or per-thread default stream and then do device-to-device copy to another stream?
|
|
||
| Ok(Self { | ||
| inner: cog, | ||
| device: 0, |
There was a problem hiding this comment.
Will consider not having the CUDA device hardcoded to 0 here. But then I'm thinking, could we have just a single CogReader struct, and let the device (CPU/CUDA/...) be set, which then points to different device decoder implementations?
Allow Python users to use the nvTIFF backend, reading a GeoTIFF into a DLPack that can be consumed by CuPy, Pytorch (2.9+), or any other Python CUDA GPU library that implements the Python Specification for DLPack.
Preview at https://cog3pio--58.org.readthedocs.build/en/58/api/#dlpack
Usage:
TODO:
__dlpack__kwargs forcupy.from_dlpack()- [ ] dl_device(TODO next time)- [ ] copy(TODO next time)Resolve memory leak, or use of invalid stream somehow (Stream handling fixed in ♻️ Refactor to pass in per-thread default stream to dlpack method #67 and 964ef03, but there might be a memory leak somewhere still 🙈)pytest.importorskipto have it run when 'cupy' is installed 🚩External things to do to improve implementation here:
nvtiff_sys::result::NvTiffErrorcupyas dependency so thatcupy-cuda13xcan be used insteadTensor values are nonsense, possibly due to raw pointer not being passed around safely 🙈 Workaround (discovered accidentally) is to instantiate the CudaCogReader class instance twice?! 😅tokio::runtime::Builder::new_multi_thread()ImportError: libnvtiff.so.0: cannot open shared object file: No such file or directorywhen runningimport cog3piowithout nvTIFF installed (e.g. if they don't have a CUDA GPU). Might need to look into https://wheelnext.dev/proposals/pep817_wheel_variant_support/References:
Follow-up to #57, part of #26