Repository navigation
Added Kernel Launch Op names for AMD GPUs - #171
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
|
Thank you for sending this PR @spandoesai ! But may I ask you to have a test plan with data or test cases? |
|
Hey @JoongunPark, thanks for the comment. I have updated the PR description with the test plan and the cases I have tried the code on. Let me know if you need any more information. Thanks! |
Thank you so much, @spandoesai ! |
|
Hey @JoongunPark, you can find the traces along with the scripts that I used in this zip file: Let me know what you find. Thanks! |
Hi @spandoesai , I got below error while testing. InstallationCommandErrorThe error is from HTA I think. But if we remove --rank option, will raise an error on Chakra size. |
|
Hello Joongun, In the initial PR message, I forgot to include the changes required to Holistic Trace Analysis (HTA) to get this Chakra PR working on AMD Devices. Since Chakra is not yet compatible with the latest branch of HTA, I could not push these changes to the HTA repo. However, I am including a patch file that can be applied to the HTA source code (specifically the /hta/common directory) that will enable the use of AMD Kineto traces in Chakra. Please let me know if there are any issues. Thanks! |
Thank you for your update @spandoesai ! Clone HTA and patch with given fileBelow is the result. Produced Output file |
tushar-krishna
left a comment
There was a problem hiding this comment.
Looks good to merge!
Summary
Added Kernel Op names from HIP such as hipLaunchKernel, hipMemCpy, etc so that Chakra can work with AMD Traces.
In order to run the chakra trace linker on AMD devices, a similar PR must be made to HTA too, since the critical path analyzer from HTA is used in the chakra_trace_link function. A PR is posted to HTA (facebookresearch/HolisticTraceAnalysis#225) and updated here soon. In the meantime, this patch can be used in the hta source directory as shown in the PR discussion to enable AMD GPUs.
Detailed Description
In the Chakra Trace Linker, the is_kernel_launch_op() checks if a given op is a kernel launch op by comparing the op name against a set of cuda specific launch kernel names. In order for Chakra to work with AMD Kineto Traces, this has been updated to include the corresponding HIP launch event names. This has been done in such a way that it does not disrupt the original execution for Nvidia devices.
Test Plan:
The commit passes all the GitHub automation tests.
For verifying correctness, three models were profiled for one inference run on an AMD Instinct MI250 GPU and the chakra linker was used on the generated ET and Kineto traces:
Command:
Final Output: (without any warnings)