Skip to content

tracker: generate compatibility with torch.compile #28981

Closed

Description

@gante

generate 🤜 🤛 torch.compile

Part of the PyTorch 2024 H2 roadmap.

This issue is a tracker of the compatibility between .generate and torch.compile (intro docs by pytorch). The goal is to enable fullgraph=True compilation on the main generate use cases.

⚠️ Is your generate use case not covered by this tracker? Check if it was requested below and upvote it if it was. Otherwise, add a comment. We will consider expanding the selection below on widely requested use cases 🤗

Decoding Strategies (end-to-end compilation)

  • greedy_search / sample are compatible (Generate: end-to-end compilation #30788)
  • beam_search / beam_sample are compatible, depends on the step above
  • assisted_decoding (aka speculative decoding) is compatible, depends on the steps above

Generate Flags and Options

  • all LogitsProcessor classes were checked for compatibility (and the appropriate exceptions are raised when not compatible)
  • all StoppingCriteria classes were checked for compatibility (and the appropriate exceptions are raised when not compatible)

Models

Notes:

  1. models tagged as "important models" in our CI + popular models
  2. language models released starting from v4.42 should ALL support compile

Decoder-only:

Encoder-decoder:

Quantization

  • BNB support
  • GPTQ support
  • AWQ support

Others

  • We have a benchmark script to quickly compare the impact of PRs
  • Add section to existing docs on the topic
  • Confirm that pipelines work after compiling generate

Activity

  1. changed the title [-]`generate` compatibility with `torch.compile`[/-] [+]tracker: `generate` compatibility with `torch.compile`[/+] on Feb 12, 2024
  2. self-assigned this
    on Feb 12, 2024
  3. github-actions commented on Apr 19, 2024

    @github-actions
    Contributor

    This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

    Please note that issues that do not follow the contributing guidelines are likely to be ignored.

  4. reopened this on May 2, 2024
  5. added
    WIPLabel your PR/Issue with WIP for some long outstanding Issues/PRs that are work in progress
    on May 9, 2024
  6. kadirnar commented on May 21, 2024

    @kadirnar
    Contributor

    👀

  7. guangy10 commented on Jul 26, 2024

    @guangy10
    Contributor

    👀

  8. jiqing-feng commented on Jul 30, 2024

    @jiqing-feng
    Contributor

    👀

  9. mlazos commented on Aug 5, 2024

    @mlazos

    Hi @gante, I'm planning on looking at these issues from the torch.compile side - two questions: (1) is there someone from HF that's committed to working on this in the event I identify some model changes that would help run torch.compile more smoothly? and (2) are these the right models we should be focusing on? are there others that are also in need of torch.compile support?

    Let me know when you can, also happy to chat on slack (I'm on the HF slack already)

  10. fzyzcjy commented on Sep 27, 2024

    @fzyzcjy
    Contributor

    Hi, is there any updates? I am especially interested in the things related to #30647. Thanks!

  11. vbonnivardprobayes commented on Oct 15, 2024

    @vbonnivardprobayes

    Hi! Do you plan on adding DONUT too? It would be highly appreciated :) As its decoder is a BART one, do you plan on doing the two alltogether for instance?
    Thanks!

  12. gante commented on Oct 17, 2024

    @gante
    ContributorAuthor

    @vbonnivardprobayes T5 compatibility is close to being done (#34089), BART and related models will get the changes next :) (cc @zucchini-nlp )

  13. gante commented on Oct 17, 2024

    @gante
    ContributorAuthor

    @fzyzcjy Not on beam search. I'll be separating prefil into a separate function as my next task, then I'll probably work on vectorized beam search :)

    (see other tracker: #30810)

  14. 3 remaining items

  15. joanvelja commented on Nov 6, 2024

    @joanvelja

    @SunMarc so this applies to models scattered across multiple GPUs with DeepSpeed via Accelerate too?

  16. anijain2305 commented on Jan 3, 2025

    @anijain2305
    Contributor

    @gante Can you update the description to reflect any new model compile support?

  17. zucchini-nlp commented on Jan 9, 2025

    @zucchini-nlp
    Member

    I updated the models which have compile compatibilty already and added a few more links for open PRs. This list is not comprehensive and shows commonly used models. At this moment all decoder-only models should be compile friendly, and the encoder-decoder ones will come in the next batch

  18. gante commented on Jan 30, 2025

    @gante
    ContributorAuthor

    This tracker is mostly complete, and we can better track support by looking for cache_position in the signature of model.forward -- if the model accepts past_key_values but DOESN'T accept cache_position, then it doesn't have support for compilation 🤗

    End-to-end compilation will not be pursued, at least not until torch has solid support for data-dependent conditional branching. Instead, we will be working towards enabling the compilation of each decoding iteration, which contains the bulk of the computations.

  19. jiqing-feng commented on Feb 8, 2025

    @jiqing-feng
    Contributor

    Hi @gante . Is there any plan to support static cache for opt model?

  20. zucchini-nlp commented on Feb 10, 2025

    @zucchini-nlp
    Member

    @jiqing-feng it will be added very soon with #35724 :)

  21. textbc1 commented on Apr 30, 2025

    @textbc1

    @jiqing-feng it will be added very soon with #35724 :)

    Could you please help by sharing inputs on a possible fix for this specific error : AttributeError: 'StaticCache' object has no attribute 'max_batch_size'. Did you mean: 'batch_size'?

    Thanks in advance

  22. gante commented on Apr 30, 2025

    @gante
    ContributorAuthor

    max_batch_size was renamed into batch_size -- updating the name of the attribute on your end should fix it :)

  23. vadimkantorov commented on Dec 23, 2025

    @vadimkantorov

    This tracker is mostly complete, and we can better track support by looking for cache_position in the signature of model.forward -- if the model accepts past_key_values but DOESN'T accept cache_position, then it doesn't have support for compilation 🤗

    I've tried to fullgraph-compile a recent PleIAs/Modal (for training time, not inference), and it succeeds with SDPA backend, but probably not impossible to make it work with FA/varlen too:

    @gante Do you have any ideas of feasibility of fixing un/pad for fullgraph-ability? IMO in the worst case it would entail adding an optional model arg for user to pass pre-computed un/pad metainfo, but otherwise should be feasible


    Also, are there CI-automated tests for checking popular models under various torch.compile options?

  24. zucchini-nlp commented on Dec 23, 2025

    @zucchini-nlp
    Member

    Hey @vadimkantorov , FA2 is known to be hard to compile in fullgraph mode. We don't support it yet and compile in non-fullgraph mode for FA2 models. If you have any proposal to make it fullgraph-able, feel free to raise a PR

    Also, are there CI-automated tests for checking popular models under various torch.compile options?

    Yes, we have tests for checking compile and checking that there are no graph breaks with all model architectures. Generation tests are in tests/utils/test_utils.py, e.g. test_generate_with_static_cache or test_generate_compile_model_forward_fullgraph

  25. vadimkantorov commented on Dec 23, 2025

    @vadimkantorov

    Yes, we have tests for checking compile and checking that there are no graph breaks with all model architectures.

    Oh, very nice! Is the resulting nightly/release results table published anywhere?

    If you have any proposal to make it fullgraph-able

    Are there any other specific issues with it besides pad/unpad utils in HF? (as this meta-info could be lifter to model args in the worst case similar to cache_positions you mentioned in #43011 (comment) - or at least these methods could be maybe monkey-patched to return pre-calculated user-provided values?)


    Also, is there a non-inplace version of Cache - i.e. so that the new cache is returned as output as torch.cat'ing of the old and new cached values? thus supporting torch.func.grad (it currently doesn't support any inplace):

    Or does DynamicCache do this?

  26. vadimkantorov commented on Dec 26, 2025

    @vadimkantorov

    FA2 is known to be hard to compile in fullgraph mode.

    There is also some very recent merged PR:

    But not sure if it solves fullgraph (and if it also concerns FAv2)

  27. zucchini-nlp commented on Jan 5, 2026

    @zucchini-nlp
    Member

    Are there any other specific issues with it besides pad/unpad utils in HF?

    I don't really remember. Looking at the above linked PR, could be that unpad is the only issue left now, though it needs validation

    Also, is there a non-inplace version of Cache?

    DynamicCache will do the cat'ting, yes. Though the cache class is always updated in-place, i.e. cache.update(keys, values) does not return a new instance of Cache

  28. guilhermeleobas commented on Jan 5, 2026

    @guilhermeleobas

    Hey @vadimkantorov , FA2 is known to be hard to compile in fullgraph mode. We don't support it yet and compile in non-fullgraph mode for FA2 models. If you have any proposal to make it fullgraph-able, feel free to raise a PR

    Do you have a reproducer for this case? I've added support for torch.compile in FA3 and I'm happy to take a look at this. This would also ensure that we can use fullgraph=True in recent versions of flash attention.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

WIPLabel your PR/Issue with WIP for some long outstanding Issues/PRs that are work in progress

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions