Skip to content

Extend sdpa tests #2118

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Mar 21, 2025
Merged

Extend sdpa tests #2118

merged 4 commits into from
Mar 21, 2025

Conversation

shubhambhokare1
Copy link
Contributor

Add tests for:

  • Pre-mul
  • Post-div
  • Post-mul

Copy link

codecov bot commented Mar 21, 2025

❌ 4 Tests Failed:

Tests completed Failed Passed Skipped
13861 4 13857 1899
View the top 3 failed test(s) by shortest run time
onnxscript.backend.onnx_export_test.TestOnnxBackEnd::test_export2python_produces_correct_onnx_script_model_1137_test_softmax_axis_2
Stack Traces | 0.005s run time
onnxscript\backend\onnx_export_test.py:137: in extract_functions
    mod = importlib.import_module(import_name)
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\importlib\__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
E   ModuleNotFoundError: No module named 'tests.onnx_backend_test_code.test_softmax_axis_2'

The above exception was the direct cause of the following exception:
.nox\test\lib\site-packages\parameterized\parameterized.py:620: in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
onnxscript\backend\onnx_export_test.py:271: in test_export2python_produces_correct_onnx_script_model
    functions = extract_functions(backend_test.name, code, self.test_folder)
onnxscript\backend\onnx_export_test.py:139: in extract_functions
    raise AssertionError(
E   AssertionError: Unable to import 'tests.onnx_backend_test_code.test_softmax_axis_2' (e=No module named 'tests.onnx_backend_test_code.test_softmax_axis_2') (file: 'D:\\a\\onnxscript\\onnxscript\\tests\\onnx_backend_test_code\\test_softmax_axis_2.py', absolute path: 'D:\\a\\onnxscript\\onnxscript\\tests\\onnx_backend_test_code\\test_softmax_axis_2.py', current folder: D:\a\onnxscript\onnxscript
E   ---- CONTENT --
E   import numpy
E   from onnx import TensorProto
E   from onnx.helper import make_tensor
E   from onnxscript import script, external_tensor
E   from onnxscript.values import Opset
E   from onnxscript.onnx_types import FLOAT
E   from onnxscript.onnx_opset import opset13
E   
E   @script()
E   def bck_test_softmax_axis_2(x: FLOAT[3,4,5]) -> (FLOAT[3,4,5]):
E       y = opset13.Softmax(x, axis=2)
E       return y
onnxscript.backend.onnx_export_test.TestOnnxBackEnd::test_export2python_produces_correct_onnx_script_model_0125_test_ai_onnx_ml_tree_ensemble_set_membership
Stack Traces | 0.006s run time
onnxscript/converter.py:467: in _eval_constant_expr
    return eval(cpl, self.globals, locals)  # pylint: disable=eval-used
E   NameError: name 'nan' is not defined

The above exception was the direct cause of the following exception:
..../test_ort_nightly/lib/python3.11.../site-packages/parameterized/parameterized.py:620: in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
onnxscript/backend/onnx_export_test.py:271: in test_export2python_produces_correct_onnx_script_model
    functions = extract_functions(backend_test.name, code, self.test_folder)
onnxscript/backend/onnx_export_test.py:137: in extract_functions
    mod = importlib.import_module(import_name)
.../Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1204: in _gcd_import
    ???
<frozen importlib._bootstrap>:1176: in _find_and_load
    ???
<frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:690: in _load_unlocked
    ???
..../test_ort_nightly/lib/python3.11.../_pytest/assertion/rewrite.py:185: in exec_module
    exec(co, module.__dict__)
tests/onnx_backend_test_code/test_ai_onnx_ml_tree_ensemble_set_membership.py:9: in <module>
    @script()
onnxscript/main.py:91: in transform
    result = script_check(f_ast, opset, env, src, default_opset=default_opset)
onnxscript/main.py:35: in script_check
    return convert.translate_function_def(f)
onnxscript/converter.py:1460: in translate_function_def
    fn_ir = self._translate_function_def_common(stmt)
onnxscript/converter.py:1447: in _translate_function_def_common
    self._translate_stmt(s, index_of_stmt=i)
onnxscript/converter.py:969: in _translate_stmt
    return self._translate_assign_stmt(node)
onnxscript/converter.py:1056: in _translate_assign_stmt
    assign(lhs, rhs)
onnxscript/converter.py:1000: in assign
    t = self._translate_expr(rhs, lhs).name
onnxscript/converter.py:553: in _translate_expr
    r = self._translate_call_expr(node)
onnxscript/converter.py:832: in _translate_call_expr
    attrs = [
onnxscript/converter.py:833: in <listcomp>
    self._translate_attr(x, y, callee.op_schema.attributes[x])
onnxscript/converter.py:517: in _translate_attr
    val = self._eval_constant_expr(expr)
onnxscript/converter.py:469: in _eval_constant_expr
    raise NameError(
E   NameError: ERROR: Missing names, globals contains ['__name__', '__doc__', '__package__', '__loader__', '__spec__', '__file__', '__cached__', '__builtins__', '@py_builtins', '@pytest_ar', 'numpy', 'TensorProto', 'make_tensor', 'script', 'external_tensor', 'Opset', 'FLOAT', 'ai_onnx_ml5'], locals [].
E   at: Function 'bck_test_ai_onnx_ml_tree_ensemble_set_membership', line 3
E       Y = ai_onnx_ml5.TreeEnsemble(X, aggregate_function=1, leaf_targetids=[0, 1, 2, 3], leaf_weights=make_tensor("value", 1, dims=[4], vals=[1.0, 10.0, 1000.0, 100.0]), membership_values=make_tensor("value", 1, dims=[8], vals=[1.2000000476837158, 3.700000047683716, 8.0, 9.0, nan, 12.0, 7.0, nan]), n_targets=4, nodes_falseleafs=[1, 0, 1], nodes_falsenodeids=[2, 2, 3], nodes_featureids=[0, 0, 0], nodes_modes=make_tensor("value", 2, dims=[3], vals=[0, 6, 6]), nodes_splits=make_tensor("value", 1, dims=[3], vals=[11.0, 232344.0, nan]), nodes_trueleafs=[0, 1, 1], nodes_truenodeids=[1, 0, 1], post_transform=0, tree_roots=[0])
E                                                                                                                                                                                             ^
onnxscript.backend.onnx_export_test.TestOnnxBackEnd::test_export2python_produces_correct_onnx_script_model_1030_test_sce_mean_weight_expanded
Stack Traces | 0.007s run time
onnxscript\backend\onnx_export_test.py:137: in extract_functions
    mod = importlib.import_module(import_name)
C:\hostedtoolcache\windows\Python\3.12.9\x64\Lib\importlib\__init__.py:90: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
E   ModuleNotFoundError: No module named 'tests.onnx_backend_test_code.test_sce_mean_weight_expanded'

The above exception was the direct cause of the following exception:
.nox\test\Lib\site-packages\parameterized\parameterized.py:620: in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
onnxscript\backend\onnx_export_test.py:271: in test_export2python_produces_correct_onnx_script_model
    functions = extract_functions(backend_test.name, code, self.test_folder)
onnxscript\backend\onnx_export_test.py:139: in extract_functions
    raise AssertionError(
E   AssertionError: Unable to import 'tests.onnx_backend_test_code.test_sce_mean_weight_expanded' (e=No module named 'tests.onnx_backend_test_code.test_sce_mean_weight_expanded') (file: 'D:\\a\\onnxscript\\onnxscript\\tests\\onnx_backend_test_code\\test_sce_mean_weight_expanded.py', absolute path: 'D:\\a\\onnxscript\\onnxscript\\tests\\onnx_backend_test_code\\test_sce_mean_weight_expanded.py', current folder: D:\a\onnxscript\onnxscript
E   ---- CONTENT --
E   import numpy
E   from onnx import TensorProto
E   from onnx.helper import make_tensor
E   from onnxscript import script, external_tensor
E   from onnxscript.values import Opset
E   from onnxscript.onnx_types import FLOAT, INT64
E   from onnxscript.onnx_opset import opset13
E   
E   @script()
E   def bck_test_sce_mean_weight_expanded(x: FLOAT[3,5], y: INT64[3], w: FLOAT[5]) -> (FLOAT):
E       SoftmaxCrossEntropyLoss_test_sce_mean_weight_expanded_function_Shape3D = opset13.Constant(value=make_tensor("value", 7, dims=[3], vals=[0, 0, -1]))
E       SoftmaxCrossEntropyLoss_test_sce_mean_weight_expanded_function_X_NCD = opset13.Reshape(x, SoftmaxCrossEntropyLoss_test_sce_mean_weight_expanded_function_Shape3D)
E       SoftmaxCrossEntropyLoss_test_sce_mean_weight_expanded_function_X_NDC = opset13.Transpose(SoftmaxCrossEntropyLoss_test_sce_mean_weight_expanded_function_X_NCD, perm=[0, 2, 1])
E       SoftmaxCrossEntropyLoss_test_sce_mean_weight_expanded_function_X_LogSM = opset13.LogSoftmax(SoftmaxCrossEntropyLoss_test_sce_mean_weight_expanded_function_X_NDC, axis=2)
E       SoftmaxCrossEntropyLoss_test_sce_mean_weight_expanded_function_X_LogSM_NCD = opset13.Transpose(SoftmaxCrossEntropyLoss_test_sce_mean_weight_expanded_function_X_LogSM, perm=[0, 2, 1])
E       SoftmaxCrossEntropyLoss_test_sce_mean_weight_expanded_function_X_shape = opset13.Shape(x)
E       SoftmaxCrossEntropyLoss_test_sce_mean_weight_expanded_function_X_Log = opset13.Reshape(SoftmaxCrossEntropyLoss_test_sce_mean_weight_expanded_function_X_LogSM_NCD, SoftmaxCrossEntropyLoss_test_sce_mean_weight_expanded_function_X_shape)
E       z = opset13.NegativeLogLikelihoodLoss(SoftmaxCrossEntropyLoss_test_sce_mean_weight_expanded_function_X_Log, y, w, reduction='mean')
E       return z

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@shubhambhokare1 shubhambhokare1 enabled auto-merge (squash) March 21, 2025 22:26
@shubhambhokare1 shubhambhokare1 merged commit fcc9802 into main Mar 21, 2025
24 of 29 checks passed
@shubhambhokare1 shubhambhokare1 deleted the sbhokare/sdpa-tests branch March 21, 2025 22:41
bmehta001 pushed a commit to bmehta001/onnxscript that referenced this pull request Apr 11, 2025
Add tests for:
- Pre-mul
- Post-div
- Post-mul
bmehta001 pushed a commit to bmehta001/onnxscript that referenced this pull request Apr 11, 2025
Add tests for:
- Pre-mul
- Post-div
- Post-mul
bmehta001 pushed a commit to bmehta001/onnxscript that referenced this pull request Apr 11, 2025
Add tests for:
- Pre-mul
- Post-div
- Post-mul
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

Successfully merging this pull request may close these issues.

3 participants