Repository navigation
Description
🚀 Motivation and context
The goal is to make it possible to analyse memory events from the Trace files to allow users of HTA to do fine-grained analysis of memory allocations and to relate them to other events.
At the moment the memory timelines can be processed from the Pytorch profiler's export_memory_timeline. This output classifies the memory events in one of 7 categories during the export, however the categories are poorly documented and can be incorrect for more complex applications that use Pytorch distributed and torch.compile.
Memory allocations and deallocations are also available in traces and can be processed with the standard HTA pipelines to produce detailed memory timelines. These timelines have the benefit of not being tied to the original classification and can be classified by the user during the analysis. This delayed classification makes it very easy to filter and inspect specific events and tie them back to the specific op that triggered them.
Note: I have a working implementation which works with Llama3-8B profiles from Torchtitan - I've opened #223 as a Draft to facilitate discussion
Description
The suggestion is to implement a get_memory_timeline and get_memory_timeline_per_category methods on the TraceAnalysis object.
from hta.trace_analysis import TraceAnalysis
analyzer = TraceAnalysis(trace_dir=f"{trace_prefix}/tests/data/torchtitan_h100/")
memory_events = analyzer.get_memory_timeline()User specified categories:
def classify_big_allocs(row):
if row["bytes_delta"] > 5e9:
return "very_big_alloc (>5gB)"
elif row["bytes_delta"] > 1e9:
return "big_alloc (>1gB)"
elif row["bytes_delta"] <1e6:
return "small_alloc (<1mb)"
return "normal_alloc"
alloc_sizes, _ = analyzer.get_memory_timeline_per_category(classification_func=classify_big_allocs)This new analysis is based on the memory events present in the trace:
{
"ph": "i", "cat": "cpu_instant_event", "s": "t", "name": "[memory]",
"pid": 1401543, "tid": 1405042,
"ts": 1336304245461.149,
"args": {
"Total Reserved": 82514542592, "Total Allocated": 61060382720, "Bytes": 8405385216, "Addr": 128803317743616, "Device Id": 0, "Device Type": 1, "Ev Idx": 6629
}
},These can then be correlated with CPU events which trigger the allocation to build a precise view of where the allocation is coming from.
Alternatives
No response
Additional context
Sample outputs


Activity
briancoutinho commentedon Apr 23, 2025
@payoto this is a great features, let's merge it in :)
sorry missed the feature proposal
payoto commentedon Apr 23, 2025
Great! I think the PR is mostly ready, I'm away without a laptop so can't check for a couple of weeks. So it's up to you if you want to take the PR out of draft and review, or wait until I'm back to give it a once over before you look at it.
briancoutinho commentedon Apr 24, 2025
@payoto that sounds good, I will leave some initial comments and you can look over and publish the draft after you are back :)
If possible please email me at bcoutinho@meta.com when you want me to review