You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In my experiments, I cannot get torch CUDA graphs to work with HF generate. CUDA graphs work fine when calling the forward pass of a model, but either due to static input/output sizes or something else, stream capture fails when calling .generate(). Can support for torch CUDA graphs be added?
Motivation
LLMs have a lot of kernel launches and CUDA graphs can remove most of the launch time. In my experiments with just forward call, CUDA graphs can be twice as fast as non-CUDA graph versions of the same model.
This is kind of planned as we want to support static caching to compile the models and have faster inference 😉 cc @gante might have already been asked in other issues as well
Hi Arthur,
I saw the PR got merged in - what is the recommended way to use cuda graphs during generation? I am wrapping the entire model with a torch cuda graph wrapper right now and am getting the same graph breaking errors as before.
Thanks,
Albert
Get Outlook for Android<https://aka.ms/AAb9ysg>
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
Hey @tsengalb99, sorry for the very slow follow-up here, we're catching up on the backlog.
This is now supported: as Arthur mentioned, the blocker was the dynamic KV cache, and once StaticCache landed (#27931, #29374) generate can be compiled into CUDA graphs. The recommended way is to pass cache_implementation="static" to generate, which pre-allocates a fixed-size cache and automatically compiles the decoding stage for greedy/sample decoding. This is what gives you the CUDA-graph speedups without the stream-capture breaks you hit when wrapping the whole model yourself. See the docs here https://huggingface.co/docs/transformers/kv_cache section fixed size cache
Activity
ArthurZucker commentedon Dec 5, 2023
This is kind of planned as we want to support static caching to compile the models and have faster inference 😉 cc @gante might have already been asked in other issues as well
gante commentedon Dec 7, 2023
@tsengalb99 as Arthur wrote, we are working on it :D Expect to see updates soon
tsengalb99 commentedon Dec 28, 2023
Are there any updates on this? And what is the main reason why cuda graphs don't work right now?
ArthurZucker commentedon Jan 2, 2024
Follow this PR #27931 for update, the dynamic KV cache is an issue
ArthurZucker commentedon Jan 30, 2024
PR is still very much active and now supports cuda graphs
tsengalb99 commentedon Feb 2, 2024
ArthurZucker commentedon Feb 5, 2024
Only needs a final review so this week 😉
tsengalb99 commentedon Feb 9, 2024
ArthurZucker commentedon Feb 12, 2024
Hey! Here is how I used it: https://gist.github.com/ArthurZucker/af34221def212259b43d55a2811d2dbb.
I used compiled, so not 100 sure how the explicit call will work! Feel free to reach out if it does not work!
github-actions commentedon Mar 7, 2024
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
ArthurZucker commentedon Mar 7, 2024
A PR is coming for this! #29374
qgallouedec commentedon Jul 17, 2026
Hey @tsengalb99, sorry for the very slow follow-up here, we're catching up on the backlog.
This is now supported: as Arthur mentioned, the blocker was the dynamic KV cache, and once StaticCache landed (#27931, #29374) generate can be compiled into CUDA graphs. The recommended way is to pass
cache_implementation="static"to generate, which pre-allocates a fixed-size cache and automatically compiles the decoding stage for greedy/sample decoding. This is what gives you the CUDA-graph speedups without the stream-capture breaks you hit when wrapping the whole model yourself. See the docs here https://huggingface.co/docs/transformers/kv_cache section fixed size cache