Repository navigation
Migrate DeepJIT CUDA integration to the PyTorch stable ABI #11
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
LyricZhao
merged 8 commits into
deepseek-ai:main
from
cleonard530:migrate_cuda_to_stable_abi
Sep 30, 2026
Merged
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
e90669c
migrated cuda build to torch stable abi
cleonard530 a40ccf5
updated kernel.h and main.cpp to match ehat was there before more acc…
cleonard530 e090381
updated TORCH_TARGET_VERSION to 2.13 and used nativeHandle() to get c…
cleonard530 8662604
Update tests/test_cuda.py
cleonard530 c2e8a2e
Update CMakeLists.txt
cleonard530 e4f3e5b
added a macro to disable pybind and the gil release. A no-op GilScope…
cleonard530 81ca29e
mark no-op GIL guards as maybe_unused. Keep DJ_DISABLE_GIL=1 builds c…
cleonard530 bad7a43
Move Torch stable shim helpers into shared header
cleonard530 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,70 @@ | ||
| #pragma once | ||
|
|
||
| #include <cstdint> | ||
|
|
||
| #include <cuda.h> | ||
| #include <cuda_runtime_api.h> | ||
|
|
||
| // Expose the CUDA declarations in Torch's shims without adding USE_CUDA to | ||
| // the rest of the including translation unit. | ||
| #ifndef USE_CUDA | ||
| #define DEEP_JIT_UNDEFINE_USE_CUDA | ||
| #define USE_CUDA | ||
| #endif | ||
| #include <torch/csrc/stable/accelerator.h> | ||
| #include <torch/csrc/stable/c/shim.h> | ||
| #include <torch/headeronly/util/shim_utils.h> | ||
| #ifdef DEEP_JIT_UNDEFINE_USE_CUDA | ||
| #undef USE_CUDA | ||
| #undef DEEP_JIT_UNDEFINE_USE_CUDA | ||
| #endif | ||
|
|
||
| namespace deep_jit::cuda { | ||
|
|
||
| // Returns the current CUDA stream for use with the CUDA Driver API. | ||
| inline CUstream get_current_cuda_stream(const int32_t device_index) { | ||
| void* stream_ptr = nullptr; | ||
| TORCH_ERROR_CODE_CHECK( | ||
| aoti_torch_get_current_cuda_stream(device_index, &stream_ptr)); | ||
| return static_cast<CUstream>(stream_ptr); | ||
| } | ||
|
|
||
| // Upstream torch plans to add stable API's for get_stream_from_pool and TorchCUDAStreamGuard. | ||
| // Once they are added and TORCH_TARGET_VERSION catches up, this file can be deleted | ||
| // and the APIs can be called directly. | ||
|
|
||
| inline cudaStream_t get_stream_from_pool(const int32_t device_index) { | ||
| void* stream = nullptr; | ||
| TORCH_ERROR_CODE_CHECK( | ||
| torch_get_cuda_stream_from_pool(false, device_index, &stream)); | ||
| return static_cast<cudaStream_t>(stream); | ||
| } | ||
|
|
||
| class TorchCUDAStreamGuard { | ||
| public: | ||
| TorchCUDAStreamGuard() = delete; | ||
|
|
||
| explicit TorchCUDAStreamGuard(cudaStream_t stream, int32_t device_index) { | ||
| TORCH_ERROR_CODE_CHECK( | ||
| aoti_torch_create_cuda_stream_guard( | ||
| static_cast<void*>(stream), device_index, &guard_)); | ||
| } | ||
|
|
||
| ~TorchCUDAStreamGuard() { | ||
| if (guard_) | ||
| (void)aoti_torch_delete_cuda_stream_guard(guard_); | ||
| } | ||
|
|
||
| // Match c10::cuda::CUDAStreamGuard: copying is disallowed because the | ||
| // guard has unique ownership, and moving is disallowed because an RAII | ||
| // stream guard has no uninitialized state to leave behind. | ||
| TorchCUDAStreamGuard(const TorchCUDAStreamGuard&) = delete; | ||
| TorchCUDAStreamGuard& operator=(const TorchCUDAStreamGuard&) = delete; | ||
| TorchCUDAStreamGuard(TorchCUDAStreamGuard&&) = delete; | ||
| TorchCUDAStreamGuard& operator=(TorchCUDAStreamGuard&&) = delete; | ||
|
|
||
| private: | ||
| CUDAStreamGuardHandle guard_ = nullptr; | ||
| }; | ||
|
|
||
| } // namespace deep_jit::cuda |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not very familiar with the nogil support of pytorch upstream, would you mind elaborate a little bit? Does this mean that DeepJIT itself supports nogil? Or downstream libraries (e.g. deepgemm / deepep) needs extra changes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When kernels are registered with
TORCH_LIBRARY, the operator is registered through PyTorch's dispatcher mechanism (basically becomes just like any other PyTorch op) and can be invoked from Python throughtorch.ops.<op_name>. When a PyTorch operator is invoked from Python, PyTorch releases the GIL before executing the operator through the dispatcher, so by the time it reachesGilScopedRelease gil_releasehere, the GIL has already been released and that guard is effectively a no-op. This means that DeepJIT doesn't need to explicitly release the GIL for downstream operators that are registered throughTORCH_LIBRARY/STABLE_TORCH_LIBRARY.Currently, DeepGEMM dispatches through pybind, so the explicit GIL release is still required there. We are working on moving DeepGEMM's registration from pybind to
TORCH_LIBRARY(see vllm-project/DeepGEMM#15 and deepseek-ai/DeepGEMM#393). We need that migration to happen before we can remove the explicit GIL release from this path, which is why we added a macro flagDJ_DISABLE_GILrather than removing the release outright. This allows existing downstream users that still dispatch through pybind to continue working correctly while we migrate.I'm not as familiar with DeepEP, so if it still relies on pybind or otherwise requires the GIL to be released at this point, that flag would need to remain for it as well, or any other downstream task that aren't registered through
TORCH_LIBRARY.Once the downstream libraries have all moved from pybind to
TORCH_LIBRARY/STABLE_TORCH_LIBRARY, and assuming they don't otherwise interact with Python while executing their kernels, we should be able to remove the explicitGilScopedReleaseentirely.Let me know if you have any other questions or need anything clarified. Thanks!