Repository navigation
Migrate DeepJIT CUDA integration to the PyTorch stable ABI #11
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 7 commits
e90669c
a40ccf5
e090381
8662604
c2e8a2e
e4f3e5b
81ca29e
bad7a43
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -27,7 +27,7 @@ DeepJIT handles the JIT infrastructure so that kernel libraries can focus on the | |
| | **CUDA** | NVCC compiles CUDA source to CUBIN; the CUDA Driver API loads and launches kernels. | CUDA headers 12.4+, NVCC 12.9+, and PyTorch with CUDA support. | | ||
| | **Ascend** | Bisheng and ld.lld compile and link Ascend kernel source; ACL loads and launches kernels. | CANN with `bin/bisheng`, `bin/ld.lld`, and the Ascend `adv_api` headers; ACL and `torch_npu` headers and runtime. | | ||
|
|
||
| The host environment must provide Linux, a C++20 compiler and standard library with `std::format` support, Python, pybind11, and the dependencies for the selected backend. DeepJIT is intended to be embedded into your extension as a header-only dependency. | ||
| The host environment must provide Linux, a C++20 compiler and standard library with `std::format` support, and the dependencies for the selected backend. The default GIL support also requires Python and pybind11; consumers that manage the GIL themselves can disable it as described under [Integration](#integration). DeepJIT is intended to be embedded into your extension as a header-only dependency. CUDA consumers should compile with `TORCH_TARGET_VERSION=0x020d000000000000` and `USE_CUDA`; this targets PyTorch's stable ABI with PyTorch 2.13 as the minimum runtime version. | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @cleonard530 if we use the old cuda stream shim, what's the min runtime version we can target?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We could target 2.10 then ( |
||
|
|
||
| See [Integration](#integration) for setup, [CUDA](#cuda) for GPU usage, and [Ascend](#ascend) for NPU usage. | ||
|
|
||
|
|
@@ -66,6 +66,13 @@ The root `CMakeLists.txt` is for debugging and IDE indexing. Integrate the heade | |
|
|
||
| ## Integration | ||
|
|
||
| GIL management is enabled by default and includes pybind11. Consumers that manage | ||
| the GIL themselves, such as kernels called through `torch.ops`, can compile with | ||
| `-DDJ_DISABLE_GIL=1` to make `deep_jit::GilScopedRelease` a no-op without including | ||
| pybind11 or the Python C API from this helper. In CMake, use | ||
| `target_compile_definitions(my_target PRIVATE DJ_DISABLE_GIL=1)`. Set the macro | ||
| consistently for every source file compiled into the consumer target. | ||
|
|
||
| Add `DeepJIT/include` to the include path of the host target, then include: | ||
|
|
||
| ```cmake | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,7 +1,11 @@ | ||
| #pragma once | ||
|
|
||
| #include <optional> | ||
| #ifndef DJ_DISABLE_GIL | ||
| #define DJ_DISABLE_GIL 0 | ||
| #endif | ||
|
|
||
| #if !DJ_DISABLE_GIL | ||
| #include <optional> | ||
| #include <pybind11/pybind11.h> | ||
|
|
||
| namespace deep_jit { | ||
|
|
@@ -22,3 +26,21 @@ class GilScopedRelease { | |
| }; | ||
|
|
||
| } // namespace deep_jit | ||
|
|
||
| #else | ||
|
|
||
| namespace deep_jit { | ||
|
|
||
| class GilScopedRelease { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm not very familiar with the nogil support of pytorch upstream, would you mind elaborate a little bit? Does this mean that DeepJIT itself supports nogil? Or downstream libraries (e.g. deepgemm / deepep) needs extra changes.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. When kernels are registered with Currently, DeepGEMM dispatches through pybind, so the explicit GIL release is still required there. We are working on moving DeepGEMM's registration from pybind to I'm not as familiar with DeepEP, so if it still relies on pybind or otherwise requires the GIL to be released at this point, that flag would need to remain for it as well, or any other downstream task that aren't registered through Once the downstream libraries have all moved from pybind to Let me know if you have any other questions or need anything clarified. Thanks! |
||
| public: | ||
| GilScopedRelease() = default; | ||
|
|
||
| GilScopedRelease(const GilScopedRelease&) = delete; | ||
| GilScopedRelease& operator=(const GilScopedRelease&) = delete; | ||
| GilScopedRelease(GilScopedRelease&&) = delete; | ||
| GilScopedRelease& operator=(GilScopedRelease&&) = delete; | ||
| }; | ||
|
|
||
| } // namespace deep_jit | ||
|
|
||
| #endif | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -22,8 +22,9 @@ | |
| #include <vector> | ||
|
|
||
| #include <cuda_runtime.h> | ||
| #include <c10/cuda/CUDAGuard.h> | ||
| #include <pybind11/pybind11.h> | ||
| #include <torch/csrc/stable/c/shim.h> | ||
| #include <torch/headeronly/util/shim_utils.h> | ||
| #include <unistd.h> | ||
|
|
||
| #include <deep_jit/backend/cuda/backend.hpp> | ||
|
|
@@ -52,6 +53,40 @@ int python_api_num_initializations = 0; | |
| std::shared_ptr<Runtime> process_test_runtime; | ||
| std::shared_ptr<Runtime> gil_test_runtime; | ||
|
|
||
| cudaStream_t get_stream_from_pool(const int32_t device_index) { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If i'm not mistaken, upstream torch is planning to add stable api for these two, so maybe we could move them to a separate util file for future removal?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, we can definitely move this and similar torch functions to a util file!
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done! |
||
| void* stream = nullptr; | ||
| TORCH_ERROR_CODE_CHECK( | ||
| torch_get_cuda_stream_from_pool(false, device_index, &stream)); | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Would be good to upstream a high level C++ wrapper API for this one |
||
| return static_cast<cudaStream_t>(stream); | ||
| } | ||
|
|
||
| class TorchCUDAStreamGuard { | ||
| public: | ||
| TorchCUDAStreamGuard() = delete; | ||
|
|
||
| explicit TorchCUDAStreamGuard(cudaStream_t stream, int32_t device_index) { | ||
| TORCH_ERROR_CODE_CHECK( | ||
| aoti_torch_create_cuda_stream_guard( | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This one too |
||
| static_cast<void*>(stream), device_index, &guard_)); | ||
| } | ||
|
|
||
| ~TorchCUDAStreamGuard() { | ||
| if (guard_) | ||
| (void)aoti_torch_delete_cuda_stream_guard(guard_); | ||
| } | ||
|
|
||
| // Match c10::cuda::CUDAStreamGuard: copying is disallowed because the | ||
| // guard has unique ownership, and moving is disallowed because an RAII | ||
| // stream guard has no uninitialized state to leave behind. | ||
| TorchCUDAStreamGuard(const TorchCUDAStreamGuard&) = delete; | ||
| TorchCUDAStreamGuard& operator=(const TorchCUDAStreamGuard&) = delete; | ||
| TorchCUDAStreamGuard(TorchCUDAStreamGuard&&) = delete; | ||
| TorchCUDAStreamGuard& operator=(TorchCUDAStreamGuard&&) = delete; | ||
|
|
||
| private: | ||
| CUDAStreamGuardHandle guard_ = nullptr; | ||
| }; | ||
|
|
||
| const fs::path& get_test_cuda_project_dir() { | ||
| static const fs::path path = [] { | ||
| const auto value = std::getenv("DEEP_JIT_CUDA_TEST_SOURCE_DIR"); | ||
|
|
@@ -1819,34 +1854,34 @@ void test_runtime_launch_features(Runtime& runtime) { | |
|
|
||
| runtime.default_launch_options.stream = std::nullopt; | ||
| runtime.default_launch_options.enable_pdl = false; | ||
| const auto current_stream = c10::cuda::getStreamFromPool(); | ||
| const auto current_stream = get_stream_from_pool(device_index); | ||
|
|
||
| cudaGraph_t graph = nullptr; | ||
| cudaGraphExec_t graph_exec = nullptr; | ||
| { | ||
| const c10::cuda::CUDAStreamGuard stream_guard(current_stream); | ||
| DJ_CUDA_RUNTIME_CHECK(cudaStreamBeginCapture(current_stream.stream(), cudaStreamCaptureModeGlobal)); | ||
| const TorchCUDAStreamGuard stream_guard(current_stream, device_index); | ||
| DJ_CUDA_RUNTIME_CHECK(cudaStreamBeginCapture(current_stream, cudaStreamCaptureModeGlobal)); | ||
| runtime.launch( | ||
| kernel, { | ||
| .num_smem_bytes = sizeof(int), | ||
| .grid_dim = dim3(2, 1, 1), | ||
| .block_dim = dim3(32, 1, 1), | ||
| }, | ||
| output, argument_storage, 10); | ||
| DJ_CUDA_RUNTIME_CHECK(cudaStreamEndCapture(current_stream.stream(), &graph)); | ||
| DJ_CUDA_RUNTIME_CHECK(cudaStreamEndCapture(current_stream, &graph)); | ||
| } | ||
| check_graph(graph); | ||
| DJ_CUDA_RUNTIME_CHECK(cudaGraphInstantiate(&graph_exec, graph, nullptr, nullptr, 0)); | ||
| DJ_CUDA_RUNTIME_CHECK(cudaGraphLaunch(graph_exec, current_stream.stream())); | ||
| DJ_CUDA_RUNTIME_CHECK(cudaStreamSynchronize(current_stream.stream())); | ||
| DJ_CUDA_RUNTIME_CHECK(cudaGraphLaunch(graph_exec, current_stream)); | ||
| DJ_CUDA_RUNTIME_CHECK(cudaStreamSynchronize(current_stream)); | ||
| check_output(43, 44); | ||
| DJ_CUDA_RUNTIME_CHECK(cudaGraphExecDestroy(graph_exec)); | ||
| DJ_CUDA_RUNTIME_CHECK(cudaGraphDestroy(graph)); | ||
|
|
||
| graph = nullptr; | ||
| graph_exec = nullptr; | ||
| { | ||
| const c10::cuda::CUDAStreamGuard stream_guard(current_stream); | ||
| const TorchCUDAStreamGuard stream_guard(current_stream, device_index); | ||
| DJ_CUDA_RUNTIME_CHECK(cudaStreamBeginCapture(stream, cudaStreamCaptureModeGlobal)); | ||
| runtime.launch( | ||
| kernel, { | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It might be better to target 2.10 as minimal requirement as it's feasible. For ascend it's ok to target 2.13 if it's necessary.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done!
To move to torch 2.10, we needed to replace
with
which requires the macro
USE_CUDAto be defined since that method is guarded in PyTorch, so I added this guard in the new include/deep_jit/backend/cuda/stable_torch_utils.h file for it to workThe final
#ifdefblock is so the rest of the build is not polluted with theUSE_CUDAmacro.