PyTorch ONNX exporter produces invalid Gemm node

[SebastianJanampa]

Hello everyone,

I am not familiar with onnx. I have exported a model from torch to onnx, the torch model works fine, but the when I try to do inference with the onnx file, I got the following error message.

[ShapeInferenceError] All inputs to Concat must have same rank. Input 1 has rank 0 != 1

How could I solve this error?

Thank you so much

[justinchuby]

This is saying the input is a scalar and should have been a vector instead. Rank==len(shape of tensor). How did you export your model? Please test with torch.onnx.export(..., dynamo=True, report=True) using the latest torch-nightly. Attach the generated report if there is an error. Thanks!

[SebastianJanampa]

Hi,

Thank you so much for your help. Here is the report. Why does Onnx generate an Onnx file even though it says there's an error?

onnx_export_2025-05-25_04-44-35-030550_success.md

[justinchuby]

Can you set verify=True in the export call to verify the model, and share the report again? I don't seem to find issue with the generated model.

[SebastianJanampa]

@justinchuby sure.

I got this error message.

Applied 434 of general pattern rewrite rules.
Check export onnx model done...
Traceback (most recent call last):
  File "/home/ubuntu/rtgrouppose/export_onnx.py", line 112, in <module>
    main(args)
  File "/home/ubuntu/rtgrouppose/export_onnx.py", line 98, in main
    onnx_model_simplify, check = onnxsim.simplify(output_file, test_input_shapes=input_shapes)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/onnxsim/onnx_simplifier.py", line 199, in simplify
    model_opt_bytes = C.simplify(
RuntimeError: /project/third_party/onnx-optimizer/third_party/onnx/onnx/common/ir.h:274: find: Assertion `!required || it != values_.end()` failed: /project/third_party/onnx-optimizer/third_party/onnx/onnx/common/ir.h:278: find: required undefined attribute 'value'

Here is the report I got.

onnx_export_2025-05-25_05-42-45-931230_runtime.md

and this is the command I use.

torch.onnx.export(
        model,
        (data, size),
        output_file,
        input_names=['images', 'orig_target_sizes'],
        output_names=['scores', 'label', 'keypoints'],
        dynamic_axes=dynamic_axes,
        opset_version=16,
        verbose=False,
        dynamo=True,
        verify=True,
        report=True,
        do_constant_folding=True,
    )

    if args.check:
        import onnx
        onnx_model = onnx.load(output_file)
        onnx.checker.check_model(onnx_model)
        print('Check export onnx model done...')

    if args.simplify:
        import onnx
        import onnxsim
        dynamic = True
        # input_shapes = {'images': [1, 3, 640, 640], 'orig_target_sizes': [1, 2]} if dynamic else None
        input_shapes = {'images': data.shape, 'orig_target_sizes': size.shape} if dynamic else None
        onnx_model_simplify, check = onnxsim.simplify(output_file, test_input_shapes=input_shapes)
        onnx.save(onnx_model_simplify, output_file)
        print(f'Simplify onnx model {check}...')

args.simplify and args.check are True

[justinchuby]

Thanks. Will push a fix to onnxscript next week.

[justinchuby]

You can track with microsoft/onnxscript#2339

[SebastianJanampa]

Thanks @justinchuby

Could you provide me with some information about the error? What exactly is going on? How did you find it?

Right now, I am developing a transformer model for 2d pose estimation. I used D-FINE repo as a reference and I don't remember using 1d tensor at any moment. Here is my repo, it is still under development and I need to fix some files and add some references, but you could check my files and code and see that I do not use any 1d tensor

[justinchuby]

The error is this: In # File: /home/ubuntu/rtgrouppose/src/models/rtgrouppose_/fdr_transformer.py:139 in forward, code: x = F.linear(x, project.to(x.device)).reshape(-1, 4), you called linear on x and project. Since project has shape [33], and ONNX requires both tensors to be 2d (https://onnx.ai/onnx/operators/onnx__Gemm.html#inputs), an error is raised.

The fix would be either:

1. update your code in fdr_transformer.py:139 to use matmul instead of linear.
2. update your code in fdr_transformer.py:139 to reshape the project tensor to 2d before calling linear
3. update onnxscript aten_linear implementation to accept a 1d input. According to documentation weight is always 2d: https://docs.pytorch.org/docs/stable/generated/torch.nn.Linear.html, nevertheless [torchlib] Update linear implementation to support 1d weights microsoft/onnxscript#2340

Here is how you can locate the error

(1) See error message onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Node (node_Gemm_3722) Op (Gemm) [ShapeInferenceError] Second input does not have rank 2

(2) In the report, locate node_Gemm_3722 in the ONNX model.

    3721 |  # node_Cast_3721
            %"_to_copy_1"<FLOAT,[33]> ⬅️ ::Cast(%"cat_14") {to=FLOAT}
    3722 |  # node_Gemm_3722
            %"linear_32"<FLOAT,[2040*s0]> ⬅️ ::Gemm(%"softmax_4", %"_to_copy_1") {beta=1.0, transB=True, alpha=1.0, transA=0}
    3723 |  # node_Constant_3723
            %"val_2071"<?,?> ⬅️ ::Constant() {value=Tensor<INT64,[2]>(array([-1,  4]), name=None)}

(3) Observe that the Gemm node's input 1 is %"_to_copy_1", search that in the pytorch ExportedProgram above:

             # File: /home/ubuntu/rtgrouppose/src/models/rtgrouppose_/fdr_transformer.py:139 in forward, code: x = F.linear(x, project.to(x.device)).reshape(-1, 4)
            _to_copy_1: "f32[33]" = torch.ops.aten._to_copy.default(cat_14, dtype = torch.float32, layout = torch.strided, device = device(type='cpu'));  cat_14 = None
            linear_32: "f32[2040*s0]" = torch.ops.aten.linear.default(softmax_4, _to_copy_1);  softmax_4 = None
            view_378: "f32[510*s0, 4]" = torch.ops.aten.view.default(linear_32, [-1, 4]);  linear_32 = None

[SebastianJanampa]

Hi @justinchuby

Thanks for such a clear explanation.
I followed your recommendation, and I fixed that error, but I got a new one.

Could you help me again, please?

This is the new report

onnx_export_2025-05-26_02-55-49-327066_runtime.md

This is the error message.

onnxruntime.capi.onnxruntime_pybind11_state.InvalidGraph: [ONNXRuntimeError] : 10 : INVALID_GRAPH : This is an invalid model. Type Error: Type 'tensor(int64)' of input parameter (val_6553) of operator (Floor) in node (node_Floor_9392) is invalid.

# node_Floor_9392
            %"floor_divide"<INT64,[s0,60]> ⬅️ ::Floor(%"val_6553")

what does it mean?

[justinchuby]

    9391 |  # node_Div_9391
            %"val_6553"<?,?> ⬅️ ::Div(%"getitem_149", %"val_840")
    9392 |  # node_Floor_9392
            %"floor_divide"<INT64,[s0,60]> ⬅️ ::Floor(%"val_6553")
    9393 |  # node_Mod_9393
            %"label"<INT64,[s0,60]> ⬅️ ::Mod(%"getitem_149", %"val_840") {fmod=0}

ONNX doesn't define the floor operation on int64 inputs. This op was generated from floor_divide:

             # File: <eval_with_key>.14:7 in forward, code: topk_1 = torch.ops.aten.topk.default(view_102, 60, 1);  view_102 = None
            topk_1 = torch.ops.aten.topk.default(view_601, 60, 1);  view_601 = None
            getitem_148: "f32[s0, 60]" = topk_1[0]
            getitem_149: "i64[s0, 60]" = topk_1[1];  topk_1 = None
            
             # File: <eval_with_key>.14:10 in forward, code: floor_divide = torch.ops.aten.floor_divide.default(getitem_70, 2)
            floor_divide: "i64[s0, 60]" = torch.ops.aten.floor_divide.default(getitem_149, 2)
            
             # File: <eval_with_key>.14:11 in forward, code: remainder = torch.ops.aten.remainder.Scalar(getitem_70, 2);  getitem_70 = None
            remainder: "i64[s0, 60]" = torch.ops.aten.remainder.Scalar(getitem_149, 2);  getitem_149 = None
            

If you have access to this part of the source code, you can cast the second output of topk to float32 first. I will separately see how I can fix this bug. Thanks for helping improve the onnx exporter. (microsoft/onnxscript#2343)

[SebastianJanampa]

Thank you @justinchuby for all your help.
Where could I find this error? The report doesn't provide any file or line this time hehe