feat: add onnxslim

[inisis]

What does this PR do?

This pr introduces onnxslim, a new high performance onnx optimization tool.

for gpt2 model, we can reduce number of operators and model size without losing accuracy, and applies to many other models.

optimum-cli export onnx --model gpt2 gpt2_onnx/ --slim

+------------------------------+------------------------------------------+------------------------------------------+
|             Add              |                   114                    |                   114                    |
|             Cast             |                    34                    |                    3                     |
|            Concat            |                   177                    |                   113                    |
|           Constant           |                   1054                   |                    0                     |
|       ConstantOfShape        |                    4                     |                    2                     |
|             Div              |                    37                    |                    25                    |
|            Equal             |                    2                     |                    2                     |
|            Expand            |                    2                     |                    2                     |
|            Gather            |                   186                    |                   185                    |
|             Gemm             |                    48                    |                    48                    |
|             Less             |                    1                     |                    1                     |
|            MatMul            |                    25                    |                    25                    |
|             Mul              |                    99                    |                    97                    |
|             Pow              |                    37                    |                    37                    |
|            Range             |                    1                     |                    1                     |
|          ReduceMean          |                    50                    |                    50                    |
|           Reshape            |                   150                    |                   145                    |
|            Shape             |                   250                    |                   100                    |
|            Slice             |                    76                    |                    14                    |
|           Softmax            |                    12                    |                    12                    |
|            Split             |                    12                    |                    12                    |
|             Sqrt             |                    61                    |                    25                    |
|           Squeeze            |                    52                    |                    2                     |
|             Sub              |                    27                    |                    27                    |
|             Tanh             |                    12                    |                    12                    |
|          Transpose           |                    61                    |                    61                    |
|          Unsqueeze           |                   271                    |                   187                    |
|            Where             |                    5                     |                    5                     |
+------------------------------+------------------------------------------+------------------------------------------+
|          Model Size          |                475.12 MB                 |                475.02 MB                 |
+------------------------------+------------------------------------------+------------------------------------------+
|         Elapsed Time         |                                        9.57 s                                       |
+------------------------------+------------------------------------------+------------------------------------------+

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

Who can review?

@IlyasMoutawwakil @xenova

[IlyasMoutawwakil]

looks clean ! we will need some basic testing for this feature, maybe in test_export_cli.py

[IlyasMoutawwakil]

@inisis the table is very cool ! is it possible to get the "diff" i.e. how many operators were removed per operator type and overall reduction in number of ops ?

[inisis]

@inisis the table is very cool ! is it possible to get the "diff" i.e. how many operators were removed per operator type and overall reduction in number of ops ?

@IlyasMoutawwakil Currently, we don't support that, but this feature can be added, and we provides api for doing so, and there will be different colors in terminal, green means less.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[inisis]

@IlyasMoutawwakil It runs very slow locally, can I use your CI server to do it. I simply enable onnxslim in test_export_cli.py, and if all passed I will remove it.

[IlyasMoutawwakil]

can I use your CI server to do it

The CI runs on basic Github runners.

[inisis]

@IlyasMoutawwakil Currently, CI test will raise No module named 'onnxslim', so where to put the dependency, a possible solution is that when user set slim to True, optimum automatically installs onnxslim if an import error happened.

[IlyasMoutawwakil]

@inisis please add it to "tests" extras, and add a check when slim=True, something like if not is_onnx_slim_available(): raise ...

[xenova]

Just want to add my voice in support of onnxslim: it's an amazing library I use for all my Transformers.js models 🤗. So, having this built-in to Optimum could be really helpful too! 👍

Happy to provide a review when PR is ready too.

[IlyasMoutawwakil]

LGTM ! thanks for iterating on this !