Skip to content

TraceCounters: count ends before starts at one timestamp in the memory bandwidth and queue length series - #374

Open
anubhavchaturvedi wants to merge 1 commit into
facebookresearch:mainfrom
anubhavchaturvedi:export-D131411321
Open

anubhavchaturvedi wants to merge 1 commit into
facebookresearch:mainfrom
anubhavchaturvedi:export-D131411321

Conversation

@anubhavchaturvedi

Copy link
Copy Markdown
Contributor

Summary:

Change

  • _get_memory_bw_time_series_for_rank and _get_queue_length_time_series_for_rank sort the points of one timestamp by one rule: first the ends of intervals of positive length, then the starts of such intervals, then each zero-length interval, its start just before its end.
  • Points of one class sort by structural keys, not by trace row order. Memory bandwidth: step (a zero-length copy uses its bandwidth for its start and its end), pid, copy, phase. Queue length: pid, tid, queue entry (the GPU event's row index), phase.
  • A launch still counts before its own GPU event. A launch and the start of its GPU event in one microsecond are a zero-length queue entry.
  • _get_memory_bw_time_series_for_rank: removed the workaround that set dur = 1 for zero-length copies (GitHub issue Memory bandwidth shows negative values #20). The new rule makes it unnecessary: a zero-length copy's start sorts just before its own end, so the copy keeps its own point and no level goes negative.
  • New test target tests:test_trace_counters (10 tests, ported from the internal OpenHTA copy).
  • tests/test_trace_analysis.py: updated the stream 7 queue length mean and std (see Impact).

Why

Root cause 1: sort_values(by="ts") uses quicksort, which is not stable. Rows with one timestamp come out in an order that depends on the row order of the whole rank. A kernel can count before its own launch (queue length -1), and the result changes when the trace rows change order.

Root cause 2: both series write one row per interval edge (+value at the start, -value at the end). A stable sort by ts alone keeps the concat order, which puts every start before every end at one timestamp. Copy A [0, 5) at 10 GB/s and copy B [5, 9) at 20 GB/s give the levels 10, 30, 20: the level 30 never existed, and describe() counts it. Two zero-length copies C and D in one microsecond give C, C+D, D. The queue length has the same fault: a kernel that starts at t (its entry leaves the queue) and a launch at t (a new entry) give L+1.

Why the fix is general: the rule follows from the interval semantics [start, end). Each level is then a sum over intervals that hold its timestamp. No level adds an interval that ends at a timestamp to one that starts there. A zero-length interval keeps a point of its own and adds to no other zero-length interval. Ties inside one class sort by structural fields (timestamps, durations, pid, tid, row identity). The rule has no constant and no trace-specific term. It is the same rule and keys as the internal OpenHTA copy (D130996907 and its follow-up).

Impact

  • Metric 1: number of different results over 20 random row orders of one rank. Measured with a temporary test (not in this diff) that shuffles t.traces[0] and hashes the output of both functions, on the test traces vision_transformer, negative_queue_length_values_check, h100 and inference_single_rank (rank 0).
    • Queue length: before 20 of 20 different on all four traces; after 1.
    • Memory bandwidth: 1 before and after (these traces have no adjacent copies).
  • Metric 2: levels that never existed, in synthetic series (new tests).
    • Before: adjacent copies give 10, 30, 20 (count 3, max 30 for 2 copies); adjacent queue entries and two kernels launched and started in one microsecond give 1, 2, 1, 0; queue length -1 values.
    • After: 10, 0, 20, 0 (count 2, max 20); 1, 0, 1, 0; no negative level.
  • Queue length statistics on the test traces (base -> this diff):
    • vision_transformer stream 7: mean 61.0431 -> 61.0345, std 92.3354 -> 92.3387. Count, min, quartiles and max do not change.
    • inference_single_rank stream 7: mean 0.7458 -> 0.7257. h100 stream 16: mean 1.6125 -> 1.5958.
    • Max does not change on any stream.
  • get_time_spent_blocked_on_full_queue at the max queue length: no change on the four traces.
  • Memory bandwidth: no level changes on the four traces. On h100, the end points of 940 zero-length memory kernels move from t + 1 to t (the 1 us workaround is gone). get_memory_bw_summary does not change.
  • Cost: one sort over 4 to 6 keys in place of 1. Not measured in OSS; the internal copy measured no change within single-run noise.

Assumptions and risks

  • Rounded timestamps are exact. A tie in the trace is a tie of instants; an end and a start in one microsecond do not overlap.
  • Rows at one timestamp remain, one per event. Their intermediate levels now under-state (each counts only intervals that hold the timestamp) and never over-state.
  • generate_trace_with_counters shows a zero-length copy as two counter points at one timestamp (zero width), not as a 1 us step. The parser rounds sub-microsecond copies to 0 us; HTA_DISABLE_NS_ROUNDING=1 keeps the real duration. This diff does not change the parser.
  • The rule must stay the same as in the internal OpenHTA copy and HTA-RS. If one changes the rule, change the others.
  • Users that pinned exact queue length statistics see small changes at tie points, as in the updated assertion.

Differential Revision: D131411321

…y bandwidth and queue length series

Summary:
## Change

- `_get_memory_bw_time_series_for_rank` and `_get_queue_length_time_series_for_rank` sort the points of one timestamp by one rule: first the ends of intervals of positive length, then the starts of such intervals, then each zero-length interval, its start just before its end.
- Points of one class sort by structural keys, not by trace row order. Memory bandwidth: step (a zero-length copy uses its bandwidth for its start and its end), pid, copy, phase. Queue length: pid, tid, queue entry (the GPU event's row index), phase.
- A launch still counts before its own GPU event. A launch and the start of its GPU event in one microsecond are a zero-length queue entry.
- `_get_memory_bw_time_series_for_rank`: removed the workaround that set `dur = 1` for zero-length copies (GitHub issue facebookresearch#20). The new rule makes it unnecessary: a zero-length copy's start sorts just before its own end, so the copy keeps its own point and no level goes negative.
- New test target `tests:test_trace_counters` (10 tests, ported from the internal OpenHTA copy).
- `tests/test_trace_analysis.py`: updated the stream 7 queue length mean and std (see Impact).

## Why

Root cause 1: `sort_values(by="ts")` uses quicksort, which is not stable. Rows with one timestamp come out in an order that depends on the row order of the whole rank. A kernel can count before its own launch (queue length -1), and the result changes when the trace rows change order.

Root cause 2: both series write one row per interval edge (+value at the start, -value at the end). A stable sort by `ts` alone keeps the concat order, which puts every start before every end at one timestamp. Copy A [0, 5) at 10 GB/s and copy B [5, 9) at 20 GB/s give the levels 10, 30, 20: the level 30 never existed, and `describe()` counts it. Two zero-length copies C and D in one microsecond give C, C+D, D. The queue length has the same fault: a kernel that starts at t (its entry leaves the queue) and a launch at t (a new entry) give L+1.

Why the fix is general: the rule follows from the interval semantics `[start, end)`. Each level is then a sum over intervals that hold its timestamp. No level adds an interval that ends at a timestamp to one that starts there. A zero-length interval keeps a point of its own and adds to no other zero-length interval. Ties inside one class sort by structural fields (timestamps, durations, pid, tid, row identity). The rule has no constant and no trace-specific term. It is the same rule and keys as the internal OpenHTA copy (D130996907 and its follow-up).

## Impact

- Metric 1: number of different results over 20 random row orders of one rank. Measured with a temporary test (not in this diff) that shuffles `t.traces[0]` and hashes the output of both functions, on the test traces `vision_transformer`, `negative_queue_length_values_check`, `h100` and `inference_single_rank` (rank 0).
  - Queue length: before 20 of 20 different on all four traces; after 1.
  - Memory bandwidth: 1 before and after (these traces have no adjacent copies).
- Metric 2: levels that never existed, in synthetic series (new tests).
  - Before: adjacent copies give 10, 30, 20 (count 3, max 30 for 2 copies); adjacent queue entries and two kernels launched and started in one microsecond give 1, 2, 1, 0; queue length -1 values.
  - After: 10, 0, 20, 0 (count 2, max 20); 1, 0, 1, 0; no negative level.
- Queue length statistics on the test traces (base -> this diff):
  - `vision_transformer` stream 7: mean 61.0431 -> 61.0345, std 92.3354 -> 92.3387. Count, min, quartiles and max do not change.
  - `inference_single_rank` stream 7: mean 0.7458 -> 0.7257. `h100` stream 16: mean 1.6125 -> 1.5958.
  - Max does not change on any stream.
- `get_time_spent_blocked_on_full_queue` at the max queue length: no change on the four traces.
- Memory bandwidth: no level changes on the four traces. On `h100`, the end points of 940 zero-length memory kernels move from `t + 1` to `t` (the 1 us workaround is gone). `get_memory_bw_summary` does not change.
- Cost: one sort over 4 to 6 keys in place of 1. Not measured in OSS; the internal copy measured no change within single-run noise.

## Assumptions and risks

- Rounded timestamps are exact. A tie in the trace is a tie of instants; an end and a start in one microsecond do not overlap.
- Rows at one timestamp remain, one per event. Their intermediate levels now under-state (each counts only intervals that hold the timestamp) and never over-state.
- `generate_trace_with_counters` shows a zero-length copy as two counter points at one timestamp (zero width), not as a 1 us step. The parser rounds sub-microsecond copies to 0 us; `HTA_DISABLE_NS_ROUNDING=1` keeps the real duration. This diff does not change the parser.
- The rule must stay the same as in the internal OpenHTA copy and HTA-RS. If one changes the rule, change the others.
- Users that pinned exact queue length statistics see small changes at tie points, as in the updated assertion.

Differential Revision: D131411321
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 12, 2026
@meta-codesync

meta-codesync Bot commented Nov 12, 2026

Copy link
Copy Markdown

@anubhavchaturvedi has exported this pull request. If you are a Meta employee, you can view the originating Diff in D131411321.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. meta-exported

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant