Skip to content

torch CUDA graphs with HF generate #27837

Closed

Description

@tsengalb99

Feature request

In my experiments, I cannot get torch CUDA graphs to work with HF generate. CUDA graphs work fine when calling the forward pass of a model, but either due to static input/output sizes or something else, stream capture fails when calling .generate(). Can support for torch CUDA graphs be added?

Motivation

LLMs have a lot of kernel launches and CUDA graphs can remove most of the launch time. In my experiments with just forward call, CUDA graphs can be twice as fast as non-CUDA graph versions of the same model.

Your contribution

n/a

Activity

  1. ArthurZucker commented on Dec 5, 2023

    @ArthurZucker
    Collaborator

    This is kind of planned as we want to support static caching to compile the models and have faster inference 😉 cc @gante might have already been asked in other issues as well

  2. gante commented on Dec 7, 2023

    @gante
    Contributor

    @tsengalb99 as Arthur wrote, we are working on it :D Expect to see updates soon

  3. tsengalb99 commented on Dec 28, 2023

    @tsengalb99
    Author

    Are there any updates on this? And what is the main reason why cuda graphs don't work right now?

  4. ArthurZucker commented on Jan 2, 2024

    @ArthurZucker
    Collaborator

    Follow this PR #27931 for update, the dynamic KV cache is an issue

  5. ArthurZucker commented on Jan 30, 2024

    @ArthurZucker
    Collaborator

    PR is still very much active and now supports cuda graphs

  6. tsengalb99 commented on Feb 2, 2024

    @tsengalb99
    Author
  7. ArthurZucker commented on Feb 5, 2024

    @ArthurZucker
    Collaborator

    Only needs a final review so this week 😉

  8. tsengalb99 commented on Feb 9, 2024

    @tsengalb99
    Author
  9. ArthurZucker commented on Feb 12, 2024

    @ArthurZucker
    Collaborator

    Hey! Here is how I used it: https://gist.github.com/ArthurZucker/af34221def212259b43d55a2811d2dbb.
    I used compiled, so not 100 sure how the explicit call will work! Feel free to reach out if it does not work!

  10. github-actions commented on Mar 7, 2024

    @github-actions
    Contributor

    This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

    Please note that issues that do not follow the contributing guidelines are likely to be ignored.

  11. added
    WIPLabel your PR/Issue with WIP for some long outstanding Issues/PRs that are work in progress
    on Mar 7, 2024
  12. ArthurZucker commented on Mar 7, 2024

    @ArthurZucker
    Collaborator

    A PR is coming for this! #29374

  13. qgallouedec commented on Jul 17, 2026

    @qgallouedec
    Member

    Hey @tsengalb99, sorry for the very slow follow-up here, we're catching up on the backlog.

    This is now supported: as Arthur mentioned, the blocker was the dynamic KV cache, and once StaticCache landed (#27931, #29374) generate can be compiled into CUDA graphs. The recommended way is to pass cache_implementation="static" to generate, which pre-allocates a fixed-size cache and automatically compiles the decoding stage for greedy/sample decoding. This is what gives you the CUDA-graph speedups without the stream-capture breaks you hit when wrapping the whole model yourself. See the docs here https://huggingface.co/docs/transformers/kv_cache section fixed size cache

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    WIPLabel your PR/Issue with WIP for some long outstanding Issues/PRs that are work in progress

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions