Repository navigation
[core] 🚨 Completely remove cache positions #44181
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 43 commits
Commits
Show all changes
51 commits
Select commit
Hold shift + click to select a range
57b5870
fully remove from the mask api
Cyrilvallez b9672ff
fully remove from cache api
Cyrilvallez 88a0b2e
remove it from llama + modulars
Cyrilvallez f5dc06c
remove them from all modulars of llama descendants
Cyrilvallez b792266
reapply all those modualrs
Cyrilvallez c2fde0a
a few more
Cyrilvallez 0f12a85
style
Cyrilvallez 3a33d19
merge main
Cyrilvallez b2d91fc
style
Cyrilvallez 49b47fe
better get_seq_length
Cyrilvallez 2a21bfa
merge main again
Cyrilvallez 94a44d0
fix signatures
Cyrilvallez 417da7a
fix device
Cyrilvallez beffc18
win against dynamo
Cyrilvallez 924fb63
merge main AGAIN
Cyrilvallez f15531f
compatible with cuda graphs
Cyrilvallez 3102c0b
add args to absorb
Cyrilvallez 5ce0c8a
fix lfm2
Cyrilvallez 318f621
fix executorch
Cyrilvallez 0cf77d6
fix zamba2
Cyrilvallez e86122e
fix encoder-decoder
Cyrilvallez 4b21a57
fix
Cyrilvallez 367f44d
fix t5gemma
Cyrilvallez 8d9a5d4
oupsi
Cyrilvallez 82bb20c
oupsi again
Cyrilvallez c066e27
paddleocr
Cyrilvallez 32fd1ee
fix lfms
Cyrilvallez 49cf111
fix export
Cyrilvallez 3605140
more export fix
Cyrilvallez ae59554
lfm2
Cyrilvallez 418a02d
fix cuda graph super weird issue
Cyrilvallez ece2c78
oupsi forgot to add it elsewhere
Cyrilvallez ca733ab
fix
Cyrilvallez 4712b29
fix all finally
Cyrilvallez 29421b0
style
Cyrilvallez 0e2b882
fix break
Cyrilvallez 883fd05
Merge branch 'main' into remove-cache-pos
Cyrilvallez 8e79feb
latest model
Cyrilvallez 0d6f071
style
Cyrilvallez c2d8180
add doc
Cyrilvallez 5385b2f
Merge branch 'main' into remove-cache-pos
Cyrilvallez 428cce2
Merge branch 'main' into remove-cache-pos
Cyrilvallez bec6545
fix audioflamingo
Cyrilvallez 3b3f519
doc
Cyrilvallez 4056eb4
bc on internal mask functions
Cyrilvallez 8f6ca3d
oupsi version
Cyrilvallez b3c0b33
Merge branch 'main' into remove-cache-pos
Cyrilvallez 5eb5459
Merge branch 'main' into remove-cache-pos
Cyrilvallez 7ac53a9
nemotron
Cyrilvallez 598a83c
Merge branch 'main' into remove-cache-pos
Cyrilvallez 9030f13
eurobert
Cyrilvallez File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we add something in the docs to clarify, this seems like something unseemingly hard to catch by outsiders - and I don't think we will remove this limitation any time soon 😓
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We could, but note that this is ALREADY the case on main, and it looks like nobody noticed/raised issue... It would generate garbage beyond the sliding window, it's just explicit in the code now!
To solve it, we could either have a mimimal version of sliding cache for export, which would work ONLY for decoding (i.e. 1 new tokens all the time, without being able to feed more than 1 token after prefill), or we could juste use full cache with proper sliding masking, but we loose some compute
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yea, still think it would be nice to clarify somewhere. True, it's also broken in main, maybe out of scope for this PR