[regression][atenlib] #459 broke pytorch gpt2 export

[BowenBao]

Discovered by pytorch/pytorch#95676, https://github.com/pytorch/pytorch/actions/runs/4306983689/jobs/7511796215. Introduced by #459

TL;DR

It seems axes is considered onnxAttributes, but got type TorchScriptTensor.

Is it because the type annotation for axes is wrong?
https://github.com/microsoft/onnx-script/blob/e66ca88c9b0beb6e45545d4c60d96c65e70c9db8/onnxscript/function_libs/torch_aten/ops/core.py#L3995

Why is this not caught in unittest?

Using sarif to pinpoint the error

Checking the generated diagnostic test_report_test_gpt2_tiny.sarif, with WARNING + ERROR filter

Viewing _add_torchscript_op_call as part of the analysis step, we could observe (in json unfortunately since sarif extension does not visualize step detail info)

           "state":{
            "name":"onnxscript.atenlib::_aten_native_layer_norm_onnx: <class 'str'>",
            "onnxInputs":"[TorchScriptTensor(None), TorchScriptTensor(None), TorchScriptTensor(None)]: <class 'list'>",
            "onnxAttributes":"OrderedDict([('axes', TorchScriptTensor(None)), ('eps', 1e-05)]): <class 'collections.OrderedDict'>",
            "nOutputs":"3: <class 'int'>"
           }

It shows axes is considered onnxAttributes, but got type TorchScriptTensor.

[abock]

@xiaowuhu would you please take a look? Thanks!

[titaiwangms]

I think the root cause is this: 690ed5d#r102668770. Wondering how it's not caught by op_correctness_test.py though. I am thinking maybe it's the job of experimental-branch to protect this regression happening, as it's about setting default value on attributes (related PR: onnx/onnx#4911). However, the experimental-branch is broken for a while. It's the issue of axes become input since opset 18.

[wschin]

Op level test is not enough to protect end-to-end exporting. Is there a plan to test against all HF models in somewhere?

[BowenBao]

Fixed by #490.

[justinchuby]

Op level test is not enough to protect end-to-end exporting. Is there a plan to test against all HF models in somewhere?

That's a good idea. I am thinking if we should run the related pytorch fx tests in this repo from the torch nightly build

[BowenBao]

@justinchuby I'm curious if it is possible to cover TorchScriptGraph with ops_correctness_test?

[justinchuby]

We can construct but not execute. I think it's still useful?

[BowenBao]

We can construct but not execute. I think it's still useful?

Yea should be enough to catch errors like this, which is raised from graph building. But why can't we execute though? I thought we can to_model_proto then run through onnx backend?

[justinchuby]

actually you are right. We can run it the same way we run the O(1) tests

[justinchuby]

Done