Repository navigation
Closed
Labels
WIPLabel your PR/Issue with WIP for some long outstanding Issues/PRs that are work in progressLabel your PR/Issue with WIP for some long outstanding Issues/PRs that are work in progress
Description
generate 🤜 🤛 torch.compile
Part of the PyTorch 2024 H2 roadmap.
This issue is a tracker of the compatibility between .generate and torch.compile (intro docs by pytorch). The goal is to enable fullgraph=True compilation on the main generate use cases.
generate use case not covered by this tracker? Check if it was requested below and upvote it if it was. Otherwise, add a comment. We will consider expanding the selection below on widely requested use cases 🤗
Decoding Strategies (end-to-end compilation)
-
greedy_search/sampleare compatible (Generate: end-to-end compilation #30788) -
beam_search/beam_sampleare compatible, depends on the step above -
assisted_decoding(aka speculative decoding) is compatible, depends on the steps above
Generate Flags and Options
- all
LogitsProcessorclasses were checked for compatibility (and the appropriate exceptions are raised when not compatible) - all
StoppingCriteriaclasses were checked for compatibility (and the appropriate exceptions are raised when not compatible)
Models
Notes:
- models tagged as "important models" in our CI + popular models
- language models released starting from v4.42 should ALL support compile
Decoder-only:
- GPT-J is compatible (Cache: new Cache format in decoder-only models #31421)
- GPT2 is compatible
- Llama is compatible ([
Core generation] Adds support for static KV cache #27931) - Gemma is compatible ([
gemma] Adds support for Gemma 💎 #29167) - Llava is compatible (VLMs: major clean up 🧼 #34502 removed dynamic control flow)
- Llava-Next is compatible (LLaVA
torch.compileimplementation #29891) - Mistral is compatible (Add torch.compile for Mistral #30642)
- Mixtral is compatible (Add torch compile for mixtral #30793)
- Phi is compatible (Compile compatibilty for decoder-only models #32617)
- Phi3 is compatible (Phi: static cache & compile compatibility #30688)
- BLOOM is compatible (note: this one might be tricky due to cache format) Compile compatibilty for decoder-only models #32617
- Mamba is compatible (requested here) (PR: Add torch.compile Support For Mamba #31247)
- Persimmon is compatible (Compile compatibilty for decoder-only models #32617)
- Qwen2 Compile compatibilty for decoder-only models #32617
- Qwen2-VL Compile compatibilty for decoder-only models #32617
- Falcon Compile compatibilty for decoder-only models #32617
- GPTNeoX Compile compatibilty for decoder-only models #32617
- Starcoder2 Compile compatibilty for decoder-only models #32617
- StableLM Compile compatibilty for decoder-only models #32617
Encoder-decoder:
- BART is compatible (Bart: new cache format #35314)
- T5 is compatible (T5 compile compatibilty #34089)
- Whisper is compatible ([whisper] static kv cache #31166)
Quantization
- BNB support
- GPTQ support
- AWQ support
Others
- We have a benchmark script to quickly compare the impact of PRs
- Add section to existing docs on the topic
- Confirm that pipelines work after compiling generate
Reacted by Raushan Turganbay, Marc Sun, vb, Omer Hacohen, Yixiao, Kadir Nar, fzyzcjy, Sumesh Punakkal Kariyil, Jilt Sebastian, OS-leonardopratesi and 3 more
Metadata
Metadata
Assignees
Labels
WIPLabel your PR/Issue with WIP for some long outstanding Issues/PRs that are work in progressLabel your PR/Issue with WIP for some long outstanding Issues/PRs that are work in progress
Activity
[-]`generate` compatibility with `torch.compile`[/-][+]tracker: `generate` compatibility with `torch.compile`[/+]github-actions commentedon Apr 19, 2024
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
kadirnar commentedon May 21, 2024
👀
guangy10 commentedon Jul 26, 2024
👀
jiqing-feng commentedon Jul 30, 2024
👀
mlazos commentedon Aug 5, 2024
Hi @gante, I'm planning on looking at these issues from the torch.compile side - two questions: (1) is there someone from HF that's committed to working on this in the event I identify some model changes that would help run
torch.compilemore smoothly? and (2) are these the right models we should be focusing on? are there others that are also in need of torch.compile support?Let me know when you can, also happy to chat on slack (I'm on the HF slack already)
fzyzcjy commentedon Sep 27, 2024
Hi, is there any updates? I am especially interested in the things related to #30647. Thanks!
vbonnivardprobayes commentedon Oct 15, 2024
Hi! Do you plan on adding DONUT too? It would be highly appreciated :) As its decoder is a BART one, do you plan on doing the two alltogether for instance?
Thanks!
gante commentedon Oct 17, 2024
@vbonnivardprobayes T5 compatibility is close to being done (#34089), BART and related models will get the changes next :) (cc @zucchini-nlp )
gante commentedon Oct 17, 2024
@fzyzcjy Not on beam search. I'll be separating prefil into a separate function as my next task, then I'll probably work on vectorized beam search :)
(see other tracker: #30810)
3 remaining items
joanvelja commentedon Nov 6, 2024
@SunMarc so this applies to models scattered across multiple GPUs with DeepSpeed via Accelerate too?
anijain2305 commentedon Jan 3, 2025
@gante Can you update the description to reflect any new model compile support?
zucchini-nlp commentedon Jan 9, 2025
I updated the models which have compile compatibilty already and added a few more links for open PRs. This list is not comprehensive and shows commonly used models. At this moment all decoder-only models should be compile friendly, and the encoder-decoder ones will come in the next batch
gante commentedon Jan 30, 2025
This tracker is mostly complete, and we can better track support by looking for
cache_positionin the signature ofmodel.forward-- if the model acceptspast_key_valuesbut DOESN'T acceptcache_position, then it doesn't have support for compilation 🤗End-to-end compilation will not be pursued, at least not until
torchhas solid support for data-dependent conditional branching. Instead, we will be working towards enabling the compilation of each decoding iteration, which contains the bulk of the computations.jiqing-feng commentedon Feb 8, 2025
Hi @gante . Is there any plan to support static cache for opt model?
zucchini-nlp commentedon Feb 10, 2025
@jiqing-feng it will be added very soon with #35724 :)
textbc1 commentedon Apr 30, 2025
Could you please help by sharing inputs on a possible fix for this specific error : AttributeError: 'StaticCache' object has no attribute 'max_batch_size'. Did you mean: 'batch_size'?
Thanks in advance
gante commentedon Apr 30, 2025
max_batch_sizewas renamed intobatch_size-- updating the name of the attribute on your end should fix it :)vadimkantorov commentedon Dec 23, 2025
I've tried to fullgraph-compile a recent
PleIAs/Modal(for training time, not inference), and it succeeds with SDPA backend, but probably not impossible to make it work with FA/varlen too:masking_utils.create_causal_maskfails in@torch.compile(fullgraph=True)#42950@gante Do you have any ideas of feasibility of fixing un/pad for fullgraph-ability? IMO in the worst case it would entail adding an optional model arg for user to pass pre-computed un/pad metainfo, but otherwise should be feasible
Also, are there CI-automated tests for checking popular models under various torch.compile options?
zucchini-nlp commentedon Dec 23, 2025
Hey @vadimkantorov , FA2 is known to be hard to compile in fullgraph mode. We don't support it yet and compile in non-fullgraph mode for FA2 models. If you have any proposal to make it fullgraph-able, feel free to raise a PR
Yes, we have tests for checking compile and checking that there are no graph breaks with all model architectures. Generation tests are in
tests/utils/test_utils.py, e.g.test_generate_with_static_cacheortest_generate_compile_model_forward_fullgraphvadimkantorov commentedon Dec 23, 2025
Oh, very nice! Is the resulting nightly/release results table published anywhere?
Are there any other specific issues with it besides pad/unpad utils in HF? (as this meta-info could be lifter to model args in the worst case similar to
cache_positionsyou mentioned in #43011 (comment) - or at least these methods could be maybe monkey-patched to return pre-calculated user-provided values?)Also, is there a non-inplace version of Cache - i.e. so that the new cache is returned as output as torch.cat'ing of the old and new cached values? thus supporting
torch.func.grad(it currently doesn't support any inplace):torch.func.gradto allow some inplace ops pytorch/pytorch#171158Or does DynamicCache do this?
vadimkantorov commentedon Dec 26, 2025
There is also some very recent merged PR:
But not sure if it solves fullgraph (and if it also concerns FAv2)
zucchini-nlp commentedon Jan 5, 2026
I don't really remember. Looking at the above linked PR, could be that
unpadis the only issue left now, though it needs validationDynamicCache will do the cat'ting, yes. Though the cache class is always updated in-place, i.e.
cache.update(keys, values)does not return a new instance ofCacheguilhermeleobas commentedon Jan 5, 2026
Do you have a reproducer for this case? I've added support for torch.compile in FA3 and I'm happy to take a look at this. This would also ensure that we can use
fullgraph=Truein recent versions of flash attention.