You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I tested with yesterday's commit 56b64bf and the problem still persists.
When debugging the issue I found this:
A custom 4D mask gets pasted to the top corner of causal_mask in _update_causal_mask()here
first decoding iteration (with empty cache) runs smoothly, so the tests pass
at the next decoding iteration, cache_position is set to torch.arange() starting from non-zero past_seen_tokenshere
later, in the decoding layer, the cache_position tensor is used to extract attention mask from causal mask. But since past_seen_tokens is not zero, the extracted mask is DIFFERENT FROM THE ORIGINAL MASK.
proposed solutions:
a) ignore cache_position when handling custom 4D attention mask
b) document instructions how to cook cache_position when passing custom 4D attention mask, so that it gets delivered intact
c) somehow else ensure delivery of the 4D mask intact. Maybe with extra option.
Also the test should be updated to perform more than one forward iteration, to allow testing with cache. I am not attempting a PR here because don't know the greater context of the changes in this part of transformers. Will be glad to test though.
Please fix this - I need it to work for my fancy speculative decoding trees (will show you soon).
The one that uses kv cache fails with 4.39.dev but works OK with 4.37.2. I also wrapped it as a new test case, ready to be pasted into transformers/tests/test_modeling_utils.
Activity
gante commentedon Mar 12, 2024
@poedator thank you for opening this issue! The PR linked above should fix it 🙏
poedator commentedon Mar 15, 2024
Hi, @gante ,
Sorry, but it is not fixed yet.
I tested with yesterday's commit 56b64bf and the problem still persists.
When debugging the issue I found this:
causal_maskin_update_causal_mask()heretorch.arange()starting from non-zeropast_seen_tokensherecache_positiontensor is used to extract attention mask from causal mask. But sincepast_seen_tokensis not zero, the extracted mask is DIFFERENT FROM THE ORIGINAL MASK.proposed solutions:
a) ignore cache_position when handling custom 4D attention mask
b) document instructions how to cook
cache_positionwhen passing custom 4D attention mask, so that it gets delivered intactc) somehow else ensure delivery of the 4D mask intact. Maybe with extra option.
Also the test should be updated to perform more than one forward iteration, to allow testing with cache. I am not attempting a PR here because don't know the greater context of the changes in this part of transformers. Will be glad to test though.
Please fix this - I need it to work for my fancy speculative decoding trees (will show you soon).
poedator commentedon Mar 16, 2024
I put together a notebook with tests.
https://gist.github.com/poedator/5c2faaff175aa8a4f12671a8e0ce835c
The one that uses kv cache fails with 4.39.dev but works OK with 4.37.2. I also wrapped it as a new test case, ready to be pasted into transformers/tests/test_modeling_utils.