Repository navigation
[FA] Native torch integration
#45153
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 31 commits
d8e4c4f
5c057ff
a763e02
ca2ae0c
4470fe8
2aac274
dc866c8
ef5d6a6
93c239c
18af76f
6c09baf
a28a9bf
cabd09b
987c269
5fb4582
7a0ce68
f43f855
25bc50e
0e4f5ee
aaff872
6ea0a0e
91cde1c
7ac6072
ffb0f2a
05aae16
54e2bd0
151e7f3
10e34c7
202e873
3a5b41c
5d06085
6aa8493
0e65665
c6df671
8b77f46
8469405
60e6316
186558f
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,13 +1,11 @@ | ||
| import torch | ||
|
|
||
| from ..modeling_flash_attention_utils import _flash_attention_forward, flash_attn_supports_top_left_mask | ||
| from ..modeling_flash_attention_utils import _flash_attention_forward | ||
| from ..utils import logging | ||
|
|
||
|
|
||
| logger = logging.get_logger(__name__) | ||
|
|
||
| _use_top_left_mask = flash_attn_supports_top_left_mask() | ||
|
|
||
|
|
||
| def get_target_dtype(query: torch.Tensor, module: torch.nn.Module) -> torch.dtype: | ||
| """If the query is in float32, return a target dtype compatible with flash attention. Return None otherwise.""" | ||
|
|
@@ -84,7 +82,6 @@ def flash_attention_forward( | |
| softmax_scale=scaling, | ||
| sliding_window=sliding_window, | ||
| softcap=softcap, | ||
| use_top_left_mask=_use_top_left_mask, | ||
| target_dtype=target_dtype, | ||
| attn_implementation=module.config._attn_implementation, | ||
| layer_idx=module.layer_idx if hasattr(module, "layer_idx") else None, | ||
|
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Was not used for quite a while now and just hidden behind kwargs |
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -12,6 +12,7 @@ | |
|
|
||
| import math | ||
| import os | ||
| import warnings | ||
|
|
||
| import torch | ||
|
|
||
|
|
@@ -23,15 +24,14 @@ | |
|
|
||
|
|
||
| # FlashAttention2 is supported on Ascend NPU with down-right aligned causal mask by default. | ||
| # Set environment variable `NPU_FA2_SPARSE_MODE` to 2 when using top-left aligned causal mask. | ||
| TOP_LEFT_ALIGNED_CAUSAL_MASK_MODE = 2 | ||
| # It can set an environment variable `NPU_FA2_SPARSE_MODE` to control this behavior. | ||
| TOP_LEFT_ALIGNED_CAUSAL_MASK_MODE = 2 # Deprecated | ||
|
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Deprecating top left mask: The min FA version of 2.3.3 no longer uses that and we should just align
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Bump ^ |
||
| DOWN_RIGHT_ALIGNED_CAUSAL_MASK_MODE = 3 | ||
|
|
||
| SPARSE_MODE = int(os.getenv("NPU_FA2_SPARSE_MODE", default=DOWN_RIGHT_ALIGNED_CAUSAL_MASK_MODE)) | ||
| if SPARSE_MODE not in [TOP_LEFT_ALIGNED_CAUSAL_MASK_MODE, DOWN_RIGHT_ALIGNED_CAUSAL_MASK_MODE]: | ||
| if SPARSE_MODE != DOWN_RIGHT_ALIGNED_CAUSAL_MASK_MODE: | ||
| raise ValueError( | ||
| "Environment variable `NPU_FA2_SPARSE_MODE` can only be set as 2 (top-left aligned causal mask) " | ||
| "or 3 (down-right aligned causal mask)." | ||
| "Environment variable `NPU_FA2_SPARSE_MODE` can only be set as 3 (down-right aligned causal mask)." | ||
| ) | ||
|
|
||
| ATTN_MASK_NPU_CACHE = {} | ||
|
|
@@ -45,7 +45,12 @@ def get_attn_mask_npu(device): | |
|
|
||
|
|
||
| def is_npu_fa2_top_left_aligned_causal_mask(): | ||
| return SPARSE_MODE == TOP_LEFT_ALIGNED_CAUSAL_MASK_MODE if is_torch_npu_available() else False | ||
| warnings.warn( | ||
| "`is_npu_fa2_top_left_aligned_causal_mask` is deprecated and will be removed in v5.16. " | ||
| "This is no longer needed as the minimum required FA version is no longer affected by this.", | ||
| FutureWarning, | ||
| ) | ||
| return False | ||
|
|
||
|
|
||
| def npu_flash_attn_func( | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it not needed anymore? We went beyond the versions that required this hack?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is already silently not used anymore except for NPU users that set an environment variable. Imo, we should deprecate this (maybe a different PR but I need some cleanups here either way)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Bump ^