Skip to content

[failed][ProfilerStep not found in the trace. The analysis result may not be accurate] Using HTA but failed #238

Open

Description

@zejun-chen

🐛 Describe the bug

Hi, all

We run the following test cases on CUDA and successfully get the log file 'ptgpu-a4000_3895004.1743587813287013007.pt.trace.json.gz'.

device = 'cuda'
N = 32
trace_handler = tensorboard_trace_handler(dir_name='/home/test/test_client_profiler/HTA/trace_log', use_gzip=True)
def train(input: torch.Tensor):
    input = torch.matmul(input, input)
    return input

with profile(
  activities = [ProfilerActivity.CPU, ProfilerActivity.CUDA],
  on_trace_ready = trace_handler,
  profile_memory = True,
  record_shapes = True,
  with_stack = True
) as prof:
    input = torch.randn(4096, 4096, device=device)
    for idx in range(N):
        print('idx = ', idx)
        output = train(input)
        output = output.cpu()
    prof.step()

Then i launch the jupyter and run the example code to try to parse the trace logs:

from hta.trace_analysis import TraceAnalysis
analyzer = TraceAnalysis(trace_dir = "/home/test/test_client_profiler/HTA/trace_log")

It failed by following error logs:
Image

Steps to reproduce

See above message and the mentioned test cases

Expected behavior

I could successfully run the following code to get the detailed profiling info.

# Temporal breakdown
temporal_breakdown_df = analyzer.get_temporal_breakdown()

# Idle time breakdown
idle_time_df = analyzer.get_idle_time_breakdown()

# Kernel breakdown
kernel_breakdown_df = analyzer.get_gpu_kernel_breakdown()

Environment

A4000
NVIDIA-SMI 550.120
Driver Version: 550.120
CUDA Version: 12.4

Activity

  1. briancoutinho commented on Apr 23, 2025

    @briancoutinho
    Contributor

    Hi @zejun-chen sorry about the delay.
    So are you asking about "ProfilerStep not found in the trace" message? Looks the analysis did run fine otherwise.
    The "ProfilerStep" is used to identify training iterations and some analysis might be less accurate if we do not have them.
    The annotation is added when "prof.step()" executes.

    So, I think you would not see "ProfilerStepXXX" annotations in your trace if you opened them up using chrome://tracing. The reason is you need to put the "prof.step()" within the for loop in your program.

    with profile(
      activities = [ProfilerActivity.CPU, ProfilerActivity.CUDA],
      on_trace_ready = trace_handler,
      profile_memory = True,
      record_shapes = True,
      with_stack = True
    ) as prof:
        input = torch.randn(4096, 4096, device=device)
        for idx in range(N):
            print('idx = ', idx)
            output = train(input)
            output = output.cpu()
        prof.step()                              # <<<<< move this to the for loop
            prof.step()                          # now it will execute on every train iteration
    

    You can also use a schedule to capture certain number of iterations.
    Also, I would not put profile_memory=True because that is handled using export_memory_timeline, and it is for memory analysis and not execution time analysis.
    Please profiler docs for schedule and profile_memory
    https://pytorch.org/docs/stable/profiler.html

    Hope this help

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions