Skip to content

Allow sdpa fusion to accept custom scale factor #2210

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Apr 24, 2025

Conversation

shubhambhokare1
Copy link
Contributor

No description provided.

Copy link

codecov bot commented Apr 17, 2025

❌ 12 Tests Failed:

Tests completed Failed Passed Skipped
15180 12 15168 2522
View the top 3 failed test(s) by shortest run time
onnxscript.backend.onnx_export_test.TestOnnxBackEnd::test_export2python_produces_correct_onnx_script_model_0114_test_bitshift_right_uint32
Stack Traces | 0.004s run time
onnxscript\backend\onnx_export_test.py:137: in extract_functions
    mod = importlib.import_module(import_name)
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\importlib\__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
E   ModuleNotFoundError: No module named 'tests.onnx_backend_test_code.test_bitshift_right_uint32'

The above exception was the direct cause of the following exception:
.nox\test\lib\site-packages\parameterized\parameterized.py:620: in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
onnxscript\backend\onnx_export_test.py:271: in test_export2python_produces_correct_onnx_script_model
    functions = extract_functions(backend_test.name, code, self.test_folder)
onnxscript\backend\onnx_export_test.py:139: in extract_functions
    raise AssertionError(
E   AssertionError: Unable to import 'tests.onnx_backend_test_code.test_bitshift_right_uint32' (e=No module named 'tests.onnx_backend_test_code.test_bitshift_right_uint32') (file: 'D:\\a\\onnxscript\\onnxscript\\tests\\onnx_backend_test_code\\test_bitshift_right_uint32.py', absolute path: 'D:\\a\\onnxscript\\onnxscript\\tests\\onnx_backend_test_code\\test_bitshift_right_uint32.py', current folder: D:\a\onnxscript\onnxscript
E   ---- CONTENT --
E   import numpy
E   from onnx import TensorProto
E   from onnx.helper import make_tensor
E   from onnxscript import script, external_tensor
E   from onnxscript.values import Opset
E   from onnxscript.onnx_types import UINT32
E   from onnxscript.onnx_opset import opset11
E   
E   @script()
E   def bck_test_bitshift_right_uint32(x: UINT32[3], y: UINT32[3]) -> (UINT32[3]):
E       z = opset11.BitShift(x, y, direction='RIGHT')
E       return z
onnxscript.backend.onnx_export_test.TestOnnxBackEnd::test_export2python_produces_correct_onnx_script_model_0173_test_cast_DOUBLE_to_FLOAT
Stack Traces | 0.004s run time
onnxscript\backend\onnx_export_test.py:137: in extract_functions
    mod = importlib.import_module(import_name)
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\importlib\__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
E   ModuleNotFoundError: No module named 'tests.onnx_backend_test_code.test_cast_DOUBLE_to_FLOAT'

The above exception was the direct cause of the following exception:
.nox\test\lib\site-packages\parameterized\parameterized.py:620: in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
onnxscript\backend\onnx_export_test.py:271: in test_export2python_produces_correct_onnx_script_model
    functions = extract_functions(backend_test.name, code, self.test_folder)
onnxscript\backend\onnx_export_test.py:139: in extract_functions
    raise AssertionError(
E   AssertionError: Unable to import 'tests.onnx_backend_test_code.test_cast_DOUBLE_to_FLOAT' (e=No module named 'tests.onnx_backend_test_code.test_cast_DOUBLE_to_FLOAT') (file: 'D:\\a\\onnxscript\\onnxscript\\tests\\onnx_backend_test_code\\test_cast_DOUBLE_to_FLOAT.py', absolute path: 'D:\\a\\onnxscript\\onnxscript\\tests\\onnx_backend_test_code\\test_cast_DOUBLE_to_FLOAT.py', current folder: D:\a\onnxscript\onnxscript
E   ---- CONTENT --
E   import numpy
E   from onnx import TensorProto
E   from onnx.helper import make_tensor
E   from onnxscript import script, external_tensor
E   from onnxscript.values import Opset
E   from onnxscript.onnx_types import DOUBLE, FLOAT
E   from onnxscript.onnx_opset import opset21
E   
E   @script()
E   def bck_test_cast_DOUBLE_to_FLOAT(input: DOUBLE[3,4]) -> (FLOAT[3,4]):
E       output = opset21.Cast(input, to=1)
E       return output
onnxscript.backend.onnx_export_test.TestOnnxBackEnd::test_export2python_produces_correct_onnx_script_model_1182_test_sqrt_example
Stack Traces | 0.004s run time
onnxscript\backend\onnx_export_test.py:137: in extract_functions
    mod = importlib.import_module(import_name)
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\importlib\__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
E   ModuleNotFoundError: No module named 'tests.onnx_backend_test_code.test_sqrt_example'

The above exception was the direct cause of the following exception:
.nox\test\lib\site-packages\parameterized\parameterized.py:620: in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
onnxscript\backend\onnx_export_test.py:271: in test_export2python_produces_correct_onnx_script_model
    functions = extract_functions(backend_test.name, code, self.test_folder)
onnxscript\backend\onnx_export_test.py:139: in extract_functions
    raise AssertionError(
E   AssertionError: Unable to import 'tests.onnx_backend_test_code.test_sqrt_example' (e=No module named 'tests.onnx_backend_test_code.test_sqrt_example') (file: 'D:\\a\\onnxscript\\onnxscript\\tests\\onnx_backend_test_code\\test_sqrt_example.py', absolute path: 'D:\\a\\onnxscript\\onnxscript\\tests\\onnx_backend_test_code\\test_sqrt_example.py', current folder: D:\a\onnxscript\onnxscript
E   ---- CONTENT --
E   import numpy
E   from onnx import TensorProto
E   from onnx.helper import make_tensor
E   from onnxscript import script, external_tensor
E   from onnxscript.values import Opset
E   from onnxscript.onnx_types import FLOAT
E   from onnxscript.onnx_opset import opset13
E   
E   @script()
E   def bck_test_sqrt_example(x: FLOAT[3]) -> (FLOAT[3]):
E       y = opset13.Sqrt(x)
E       return y

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@shubhambhokare1 shubhambhokare1 merged commit fc2e5da into main Apr 24, 2025
22 of 27 checks passed
@shubhambhokare1 shubhambhokare1 deleted the sbhokare/scale-attr branch April 24, 2025 22:15
shubhambhokare1 added a commit that referenced this pull request May 7, 2025
Add fusion rules to support the optimization of Whisper models.

Fusions added:

- Basic Fusions:
    * additional pattern for erfgelu [moved to #2222]
- SkipLayerNorm:
    * #2259 
    * Fusion patterns where skip_sum is also an output
    * Bias + SkipLayerNorm -> SkipLayerNorm (with bias) [moved to #2222]
- BiasGelu Fusion [moved to #2222]
- SDPA:
    * Support for pattern where only q is pre-scaled
- MHA:
    * Patterns with/without past/present keys/values
    * Patterns with non-rotary embeddings
    * Patterns with/without mask
    * Patterns with cross-attention (only for past key/value patterns)
 - MHA Bias Fusion:
* Bias was offloaded to Attention fusion previously, this fusion fixes
that
 - Attention:
    * Patterns where Q, K and V do not come from slicing

TODO:
- [x]  Fix SDPA singular prescale case, due to lost shape information
- [x] - Enable check conditions when #2210 is merged
- [x] - Improve/Rewrite whisper model test case to be similar to that of
smollm (for eg)
- [x] - Fix failing test cases to account for new patterns
- [x] - Add isolated test cases for new fusions like BiasGelu,
SkipLayerNorm etc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Development

Successfully merging this pull request may close these issues.

2 participants