Add ORT fused adam optimizer

[JingyaHuang]

What does this PR do?

• Integrate ort fused adam optimizer
• Subclass TrainingArguments to include ort optimizers
• Update training examples
• Add to the nightly test

---

Who can review

@zhijxu-MS @carzh

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[JingyaHuang]

@JingyaHuang let's discuss here for ort-fused-adam.

with your new changes, i can run it locally, such as "question-answering/run_qa.py".

the issue you encountered looks like you are using an old ort, could you install a newer one. the install command is pip uninstall -y onnxruntime-training && pip install --pre --no-deps --no-index onnxruntime-training -f https://download.onnxruntime.ai/onnxruntime_nightly_cu113.html && TORCH_CUDA_ARCH_LIST='7.0;8.0+PTX' python -m torch_ort.configure

Hi @zhijxu-MS, thanks for the advice. I tested with onnxruntime nightly and it works well for me. 👍

As the previous branch was a little messy, I closed the previous and opened another PR here. Also I updated the examples as well as the test of ORTTrainer. Can you take a look if everything is ok for you? Thx!

[regisss]

LGTM @JingyaHuang!!

[zhijxu-MS]

@JingyaHuang let's discuss here for ort-fused-adam.
with your new changes, i can run it locally, such as "question-answering/run_qa.py".
the issue you encountered looks like you are using an old ort, could you install a newer one. the install command is pip uninstall -y onnxruntime-training && pip install --pre --no-deps --no-index onnxruntime-training -f https://download.onnxruntime.ai/onnxruntime_nightly_cu113.html && TORCH_CUDA_ARCH_LIST='7.0;8.0+PTX' python -m torch_ort.configure

Hi @zhijxu-MS, thanks for the advice. I tested with onnxruntime nightly and it works well for me. 👍

As the previous branch was a little messy, I closed the previous and opened another PR here. Also I updated the examples as well as the test of ORTTrainer. Can you take a look if everything is ok for you? Thx!

i tested 4 optimum examples locally, they can be run w/ or w/o ort-fused-adam. and make sure ort-fused-adam is called by adding log.

[mfuntowicz]

LGTM 😃

[JingyaHuang]

Thanks for the review @mfuntowicz @regisss @zhijxu-MS , will merge it.