Add update_quantized_cache op #5527

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Closed

kimishpatel wants to merge 12 commits into gh/kimishpatel/68/base from gh/kimishpatel/68/head

Contributor

kimishpatel commented Sep 20, 2024 •

edited

Loading

Stack from ghstack (oldest at bottom):

Why?

ton of copies due to functionalization
mutable buffer support without such custom inplace ops will results in giant
copies at the end
Making inplace ops work will likely take longer and not clear safe path

Differential Revision: D62301838


          Add update_quantized_cache op

3de8619

Why?
- ton of copies due to functionalization
- mutable buffer support without such custom inplace ops will results in giant
  copies at the end
- Making inplace ops work will likely take longer and not clear safe path

Differential Revision: [D62301838](https://our.internmc.facebook.com/intern/diff/D62301838/)

[ghstack-poisoned]

pytorch-bot bot commented Sep 20, 2024 •

edited

Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/5527

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 0645095 with merge base ee32848 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

facebook-github-bot added the CLA Signed label

Contributor

facebook-github-bot commented Sep 20, 2024

This pull request was exported from Phabricator. Differential Revision: D62301838

facebook-github-bot added the fb-exported label

This was referenced Sep 20, 2024

[ExecuTorch] Some updated to kv cache #5523

Closed

Fix dequantize per channel to handle double scale type #5524

Closed

[ExecuTorch] Add quantized kv cache to llama #5525

Closed

Refactor custom SDPA op to separate kv cache update from the custom sdpa op #5526

Closed

[Executorch][llama] Update SDPA op to use quantized kv cache #5528

Closed

[Executorch][llama] Refactoring sdpa #5529

Closed

[Executorch] Update EXECUTORCH_LIBRARY macro #5530

Closed

[Executorch][llama] Add custom_sdpa and use that instead of sdpa_with_kv_cache #5531

Closed


          Update on "Add update_quantized_cache op"

e3ff6b1

Why?
- ton of copies due to functionalization
- mutable buffer support without such custom inplace ops will results in giant
  copies at the end
- Making inplace ops work will likely take longer and not clear safe path

Differential Revision: [D62301838](https://our.internmc.facebook.com/intern/diff/D62301838/)

[ghstack-poisoned]

Contributor

facebook-github-bot commented Sep 24, 2024

This pull request was exported from Phabricator. Differential Revision: D62301838

kimishpatel mentioned this pull request

[Executorch][quant] Optimize per channel dequantize #5596

Closed


          Update on "Add update_quantized_cache op"

ede4406

Why?
- ton of copies due to functionalization
- mutable buffer support without such custom inplace ops will results in giant
  copies at the end
- Making inplace ops work will likely take longer and not clear safe path

Differential Revision: [D62301838](https://our.internmc.facebook.com/intern/diff/D62301838/)

[ghstack-poisoned]

Contributor

facebook-github-bot commented Sep 24, 2024

This pull request was exported from Phabricator. Differential Revision: D62301838

This was referenced Sep 24, 2024

[ExecuTorch] Some updated to kv cache #5597

Closed

[ExecuTorch] Add quantized kv cache to llama #5598

Closed

Refactor custom SDPA op to separate kv cache update from the custom sdpa op #5599

Closed

[Executorch][llama] Update SDPA op to use quantized kv cache #5600

Closed

[Executorch][llama] Refactoring sdpa #5601

Closed

[Executorch] Update EXECUTORCH_LIBRARY macro #5602

Closed

[Executorch][llama] Add custom_sdpa and use that instead of sdpa_with_kv_cache #5603

Closed

[Executorch][quant] Optimize per channel dequantize #5604

Closed


          Update on "Add update_quantized_cache op"

6fec56f

Why?
- ton of copies due to functionalization
- mutable buffer support without such custom inplace ops will results in giant
  copies at the end
- Making inplace ops work will likely take longer and not clear safe path

Differential Revision: [D62301838](https://our.internmc.facebook.com/intern/diff/D62301838/)

[ghstack-poisoned]

Contributor

facebook-github-bot commented Sep 25, 2024

This pull request was exported from Phabricator. Differential Revision: D62301838

This was referenced Sep 25, 2024

[ExecuTorch] Some updated to kv cache #5615

Closed

[ExecuTorch] Add quantized kv cache to llama #5616

Closed

This was referenced Sep 25, 2024

[Executorch][llama] Refactoring sdpa #5653

Closed

[Executorch] Update EXECUTORCH_LIBRARY macro #5654

Closed

[Executorch][llama] Add custom_sdpa and use that instead of sdpa_with_kv_cache #5655

Closed

[Executorch][quant] Optimize per channel dequantize #5656

Closed


          Update on "Add update_quantized_cache op"

7a94bd4

Why?
- ton of copies due to functionalization
- mutable buffer support without such custom inplace ops will results in giant
  copies at the end
- Making inplace ops work will likely take longer and not clear safe path

Differential Revision: [D62301838](https://our.internmc.facebook.com/intern/diff/D62301838/)

[ghstack-poisoned]

Contributor

facebook-github-bot commented Sep 25, 2024

This pull request was exported from Phabricator. Differential Revision: D62301838


          Update on "Add update_quantized_cache op"

f38581d

Why?
- ton of copies due to functionalization
- mutable buffer support without such custom inplace ops will results in giant
  copies at the end
- Making inplace ops work will likely take longer and not clear safe path

Differential Revision: [D62301838](https://our.internmc.facebook.com/intern/diff/D62301838/)

[ghstack-poisoned]

Contributor

facebook-github-bot commented Sep 25, 2024

This pull request was exported from Phabricator. Differential Revision: D62301838

This was referenced Sep 25, 2024

[ExecuTorch] Some updated to kv cache #5663

Closed

[ExecuTorch] Add quantized kv cache to llama #5664

Closed

Refactor custom SDPA op to separate kv cache update from the custom sdpa op #5665

Closed

[Executorch][llama] Update SDPA op to use quantized kv cache #5666

Closed

[Executorch][llama] Refactoring sdpa #5667

Closed

[Executorch] Update EXECUTORCH_LIBRARY macro #5668

Closed

[Executorch][llama] Add custom_sdpa and use that instead of sdpa_with_kv_cache #5669

Closed

[Executorch][quant] Optimize per channel dequantize #5670

Merged


          Update on "Add update_quantized_cache op"

faf154f

Why?
- ton of copies due to functionalization
- mutable buffer support without such custom inplace ops will results in giant
  copies at the end
- Making inplace ops work will likely take longer and not clear safe path

Differential Revision: [D62301838](https://our.internmc.facebook.com/intern/diff/D62301838/)

[ghstack-poisoned]

Contributor

facebook-github-bot commented Sep 26, 2024

This pull request was exported from Phabricator. Differential Revision: D62301838

kimishpatel mentioned this pull request

Dont quantize the current token for attention #5715

Merged


          Update on "Add update_quantized_cache op"

e7f6cff

Why?
- ton of copies due to functionalization
- mutable buffer support without such custom inplace ops will results in giant
  copies at the end
- Making inplace ops work will likely take longer and not clear safe path

Differential Revision: [D62301838](https://our.internmc.facebook.com/intern/diff/D62301838/)

[ghstack-poisoned]

Contributor

facebook-github-bot commented Sep 27, 2024

This pull request was exported from Phabricator. Differential Revision: D62301838


          Update on "Add update_quantized_cache op"

Why?
- ton of copies due to functionalization
- mutable buffer support without such custom inplace ops will results in giant
  copies at the end
- Making inplace ops work will likely take longer and not clear safe path

Differential Revision: [D62301838](https://our.internmc.facebook.com/intern/diff/D62301838/)

[ghstack-poisoned]

Contributor

facebook-github-bot commented Sep 30, 2024

This pull request was exported from Phabricator. Differential Revision: D62301838

digantdesai approved these changes

View reviewed changes


          Update on "Add update_quantized_cache op"

Why?
- ton of copies due to functionalization
- mutable buffer support without such custom inplace ops will results in giant
  copies at the end
- Making inplace ops work will likely take longer and not clear safe path

Differential Revision: [D62301838](https://our.internmc.facebook.com/intern/diff/D62301838/)

[ghstack-poisoned]

Contributor

facebook-github-bot commented Oct 1, 2024

This pull request was exported from Phabricator. Differential Revision: D62301838

facebook-github-bot closed this in

5f324ce

facebook-github-bot added the Merged label

Contributor

facebook-github-bot commented Oct 2, 2024

This pull request has been merged in 5f324ce.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed fb-exported Merged