Generate: New `Cache` abstraction and Attention Sinks support#26681

Merged

ydshieh merged 35 commits intohuggingface:mainfrom

tomaarsen:feat/kv_cache_class

Dec 8, 2023

Commits on Dec 7, 2023

Draft version of new KV Caching

tomaarsen
authored and
gante
committed
Address numerous PR suggestions

tomaarsen
authored and
gante
committed
Implement the SinkCache through backward+forward rotations

tomaarsen
authored and
gante
committed
Integrate (Sink)Cache with Llama FA2

tomaarsen
authored and
gante
committed
Set use_legacy_cache=True as default, allows for test passes

tomaarsen
authored and
gante
committed
Move from/to_legacy_cache to ...Model class

tomaarsen
authored and
gante
committed
Undo unnecessary newline change

tomaarsen
authored and
gante
committed
Remove copy utility from deprecated OpenLlama

tomaarsen
authored and
gante
committed
Match import style

tomaarsen
authored and
gante
committed
manual rebase with main
gante
committed
Cache class working with generate (#1)

gante
and
tomaarsen
committed
move import
gante
committed
add default to model_kwargs.get('use_legacy_cache')
gante
committed
correct failing test
gante
committed
Apply suggestions from code review

gante
and
patrickvonplaten
committed
apply PR suggestions
gante
committed
fix failing test
gante
committed
Apply suggestions from code review

committed
PR comments
gante
committed
tmp commit
gante
committed
add docstrings
gante
committed
more tests, more docstrings, add to docs
gante
committed
derp
gante
committed
tmp commit
gante
committed
tmp dbg
gante
committed
more dbg
gante
committed
fix beam search bug
gante
committed
cache can be a list of tuples in some models
gante
committed
fix group beam search
gante
committed
all but sinkcache integration tests
gante
committed
fix sink cache and add hard integration test
gante
committed
now also compatible with input_embeds input
gante
committed
PR comments
gante
committed
add Cache support to Phi+FA2
gante
committed
make fixup
gante
committed