Skip to content

Added Kernel Launch Op names for AMD GPUs - #171

Merged
tushar-krishna merged 1 commit into
mlcommons:mainfrom
spandoesai:spanmore/pytorch_amd
Feb 26, 2025
Merged

tushar-krishna merged 1 commit into
mlcommons:mainfrom
spandoesai:spanmore/pytorch_amd

Conversation

@spandoesai

@spandoesai spandoesai commented Dec 3, 2024 •

Copy link
Copy Markdown
Contributor

Summary

Added Kernel Op names from HIP such as hipLaunchKernel, hipMemCpy, etc so that Chakra can work with AMD Traces.

In order to run the chakra trace linker on AMD devices, a similar PR must be made to HTA too, since the critical path analyzer from HTA is used in the chakra_trace_link function. A PR is posted to HTA (facebookresearch/HolisticTraceAnalysis#225) and updated here soon. In the meantime, this patch can be used in the hta source directory as shown in the PR discussion to enable AMD GPUs.

Detailed Description

In the Chakra Trace Linker, the is_kernel_launch_op() checks if a given op is a kernel launch op by comparing the op name against a set of cuda specific launch kernel names. In order for Chakra to work with AMD Kineto Traces, this has been updated to include the corresponding HIP launch event names. This has been done in such a way that it does not disrupt the original execution for Nvidia devices.

Test Plan:

The commit passes all the GitHub automation tests.
For verifying correctness, three models were profiled for one inference run on an AMD Instinct MI250 GPU and the chakra linker was used on the generated ET and Kineto traces:

  1. Matrix Multiplication example code from the Chakra Wiki found here
  2. Toy example
class ToyModel(nn.Module):
  def __init__(self):
    super(ToyModel, self).__init__()
    self.layers = nn.ModuleList([nn.Linear(128, 128)])
    self.layers.append(nn.ReLU())
  
  def forward(self, x):
    for layer in self.layers:
      x = layer(x)
      return x
  1. Longformer from Huggingface Transformers

Command:

chakra_trace_link --rank 0 --chakra-host-trace ./pytorch_et.json --chakra-device-trace ./kineto_trace.json --output-file ./linked.json

Final Output: (without any warnings)

...
trace_link.py:49 [INFO]: Linking process successful. Output file is available at ./linked.json.
trace_link.py:50 [INFO]: Please run the chakra_converter for further postprocessing. 

@spandoesai
spandoesai requested a review from a team as a code owner December 3, 2024 02:01
@github-actions

github-actions Bot commented Dec 3, 2024

Copy link
Copy Markdown

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

@JoongunPark

JoongunPark commented Dec 4, 2024 •

Copy link
Copy Markdown
Contributor

Thank you for sending this PR @spandoesai !
I also believe this PR does not harm existing support for NVIDIA GPUs.

But may I ask you to have a test plan with data or test cases?
Thank you!

@spandoesai

Copy link
Copy Markdown
Contributor Author

Hey @JoongunPark, thanks for the comment.

I have updated the PR description with the test plan and the cases I have tried the code on. Let me know if you need any more information.

Thanks!

@JoongunPark

JoongunPark commented Dec 5, 2024 •

Copy link
Copy Markdown
Contributor

Hey @JoongunPark, thanks for the comment.

I have updated the PR description with the test plan and the cases I have tried the code on. Let me know if you need any more information.

Thanks!

Thank you so much, @spandoesai !
It is possible to share the traces you got? If you can, I will test those in my local machine and share the result.
Thank you!

@spandoesai

Copy link
Copy Markdown
Contributor Author

Hey @JoongunPark, you can find the traces along with the scripts that I used in this zip file:
models.zip

Let me know what you find.

Thanks!

@JoongunPark

JoongunPark commented Dec 16, 2024 •

Copy link
Copy Markdown
Contributor

Hey @JoongunPark, you can find the traces along with the scripts that I used in this zip file: models.zip

Let me know what you find.

Thanks!

Hi @spandoesai , I got below error while testing.

Installation

git clone https://github.com/spandoesai/chakra/tree/spanmore/pytorch_amd
conda create -n "chakra-pr" python=3.10.0
conda activate chakra-pr
cd chakra
pip install .

cd ..
git clone git@github.com:facebookresearch/param.git
cd param/et_replay
git checkout 7b19f586dd8b267333114992833a0d7e0d601630
pip install .

Command

$ chakra_trace_link --chakra-host-trace mi250_longformer_et.json --chakra-device-trace mi250_longformer_kineto.json --rank 0 --output-file linked_test.json

Error

[2024-12-16 14:22:35,771] trace.py:328 [INFO]: /home/un/Project/PRs/models/models/longformer/logs
[2024-12-16 14:22:35,772] trace.py:474 [INFO]: ranks=[0]
[2024-12-16 14:22:36,201] trace_parser.py:107 [WARNING]: Parsed /home/un/Project/PRs/models/models/longformer/logs/mi250_longformer_kineto.json time = 0.43 seconds
[2024-12-16 14:22:36,237] trace_parser.py:317 [WARNING]: Rounding down ns resolution events due to issue with events overlapping. ts dtype = float64, dur dtype = float64.Please see https://github.com/pytorch/pytorch/pull/122425
[2024-12-16 14:22:36,317] trace_parser.py:430 [WARNING]: Parsed /home/un/Project/PRs/models/models/longformer/logs/mi250_longformer_kineto.json backend=json in 0.55 seconds; current PID:3295
[2024-12-16 14:22:36,344] trace.py:236 [WARNING]: Overall parsing of /home/un/Project/PRs/models/models/longformer/logs/mi250_longformer_kineto.json in 0.57 seconds; current PID:3295
[2024-12-16 14:22:36,349] trace.py:449 [WARNING]: leaving parse_multiple_ranks duration=0.58 seconds
[2024-12-16 14:22:36,349] trace.py:483 [WARNING]: leaving parse_traces duration=0.58 seconds
[2024-12-16 14:22:36,356] critical_path_analysis.py:1467 [WARNING]: Trace does not contain CUDA Synchronization events so the results of analysis could be inaccurate.
[2024-12-16 14:22:36,356] critical_path_analysis.py:1471 [WARNING]: Please see this PR to learn how to enable CUDA sync events https://github.com/pytorch/pytorch/pull/105187
[2024-12-16 14:22:36,356] critical_path_analysis.py:1493 [INFO]: Looking up events under [0, 0) instance(s) of 'ProfilerStep' annotation.
[2024-12-16 14:22:36,357] critical_path_analysis.py:1509 [INFO]: Looking up events within the window (0, 3336410.0)
[2024-12-16 14:22:36,367] critical_path_analysis.py:1533 [INFO]: Clipped dataframe has 13867 events
[2024-12-16 14:22:36,367] critical_path_analysis.py:1540 [INFO]: Preprocessing took 0.01 seconds
Traceback (most recent call last):
  File "/home/un/miniconda3/envs/chakra-pr/bin/chakra_trace_link", line 8, in <module>
    sys.exit(main())
  File "/home/un/miniconda3/envs/chakra-pr/lib/python3.10/site-packages/chakra/src/trace_link/trace_link.py", line 47, in main
    linker.link(args.rank, args.chakra_host_trace, args.chakra_device_trace, args.output_file)
  File "/home/un/miniconda3/envs/chakra-pr/lib/python3.10/site-packages/chakra/src/trace_link/trace_linker.py", line 70, in link
    sync_deps = self.load_sync_dependencies(rank, chakra_device_trace)
  File "/home/un/miniconda3/envs/chakra-pr/lib/python3.10/site-packages/chakra/src/trace_link/trace_linker.py", line 121, in load_sync_dependencies
    cp_graph, success = trace_analysis.critical_path_analysis(
  File "/home/un/miniconda3/envs/chakra-pr/lib/python3.10/site-packages/hta/trace_analysis.py", line 555, in critical_path_analysis
    return CriticalPathAnalysis.critical_path_analysis(
  File "/home/un/miniconda3/envs/chakra-pr/lib/python3.10/site-packages/hta/analyzers/critical_path_analysis.py", line 1542, in critical_path_analysis
    cp_graph = CPGraph(t_copy, t, rank)
  File "/home/un/miniconda3/envs/chakra-pr/lib/python3.10/site-packages/hta/analyzers/critical_path_analysis.py", line 183, in __init__
    self._construct_graph(cg)
  File "/home/un/miniconda3/envs/chakra-pr/lib/python3.10/site-packages/hta/analyzers/critical_path_analysis.py", line 384, in _construct_graph
    self._construct_graph_from_kernels()
  File "/home/un/miniconda3/envs/chakra-pr/lib/python3.10/site-packages/hta/analyzers/critical_path_analysis.py", line 843, in _construct_graph_from_kernels
    .join(q[["queue_length"]], on="index_correlation")
TypeError: 'NoneType' object is not subscriptable

The error is from HTA I think. But if we remove --rank option, will raise an error on Chakra size.

@spandoescode

Copy link
Copy Markdown

Hello Joongun,

In the initial PR message, I forgot to include the changes required to Holistic Trace Analysis (HTA) to get this Chakra PR working on AMD Devices. Since Chakra is not yet compatible with the latest branch of HTA, I could not push these changes to the HTA repo. However, I am including a patch file that can be applied to the HTA source code (specifically the /hta/common directory) that will enable the use of AMD Kineto traces in Chakra.

Please let me know if there are any issues.

Thanks!
amd_hta_patch.txt

@JoongunPark

JoongunPark commented Feb 5, 2025 •

Copy link
Copy Markdown
Contributor

Hello Joongun,

In the initial PR message, I forgot to include the changes required to Holistic Trace Analysis (HTA) to get this Chakra PR working on AMD Devices. Since Chakra is not yet compatible with the latest branch of HTA, I could not push these changes to the HTA repo. However, I am including a patch file that can be applied to the HTA source code (specifically the /hta/common directory) that will enable the use of AMD Kineto traces in Chakra.

Please let me know if there are any issues.

Thanks! amd_hta_patch.txt

Thank you for your update @spandoesai !
I successfully ran Chakra after applying your patch file to HTA.
Here are the commands.

Clone HTA and patch with given file

$ git clone https://github.com/facebookresearch/HolisticTraceAnalysis.git
$ cd HolisticTraceAnalysis
$ git checkout d731cc2e2249976c97129d409a83bd53d93051f6
$ git submodule update --init
$ mv amd_hta_patch.txt amd_hta_patch.patch
$ git apply amd_hta_patch.patch
$ pip install -r requirements.txt
$ pip install -e .

Below is the result.
One of my concern is we got lots of warnings from trace_linker, which I believe we solve in other PRs.

[2025-02-05 14:24:30,898] trace_linker.py:619 [WARNING]: Missing parent CPU operator for GPU op 'void at::native::elementwise_kernel<128, 2, at::native::gpu_kernel_impl_nocast<at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#7}::operator()() const::{lambda(float)#1}>(at::TensorIteratorBase&, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#7}::operator()() const::{lambda(float)#1} const&)::{lambda(int)#1}>(int, at::native::gpu_kernel_impl_nocast<at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#7}::operator()() const::{lambda(float)#1}>(at::TensorIteratorBase&, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#7}::operator()() const::{lambda(float)#1} const&)::{lambda(int)#1})'. Orphaned GPU operator.
[2025-02-05 14:24:30,898] trace_linker.py:673 [WARNING]: No CUDA runtime operator found for correlation ID 4863. This is not a common case, and there should be a corresponding CUDA runtime operator for a given GPU kernel operator. It can be a case where CUDA runtime operators are not properly identified and added to the map, kineto_correlation_cuda_runtime_map. Please manually check if the corresponding CUDA runtime operator with the correlation is dropped by mistake. It is likely that it is because of incomplete map, cuda_launch_operations, in is_kernel_launch_op. Please update the map properly to cover all CUDA runtime launch operators.
[2025-02-05 14:24:30,898] trace_linker.py:619 [WARNING]: Missing parent CPU operator for GPU op 'void at::native::elementwise_kernel<128, 2, at::native::gpu_kernel_impl_nocast<at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#7}::operator()() const::{lambda(float)#1}>(at::TensorIteratorBase&, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#7}::operator()() const::{lambda(float)#1} const&)::{lambda(int)#1}>(int, at::native::gpu_kernel_impl_nocast<at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#7}::operator()() const::{lambda(float)#1}>(at::TensorIteratorBase&, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#7}::operator()() const::{lambda(float)#1} const&)::{lambda(int)#1})'. Orphaned GPU operator.
[2025-02-05 14:24:33,031] trace_link.py:49 [INFO]: Linking process successful. Output file is available at linked_test.json.
[2025-02-05 14:24:33,031] trace_link.py:50 [INFO]: Please run the chakra_converter for further postprocessing.

Produced Output file

linked_test.zip

@tushar-krishna tushar-krishna left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to merge!

@tushar-krishna
tushar-krishna merged commit 52d7c30 into mlcommons:main Feb 26, 2025
@github-actions github-actions Bot locked and limited conversation to collaborators Feb 26, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants