Repository navigation
TraceCounters: count ends before starts at one timestamp in the memory bandwidth and queue length series - #374
Open
anubhavchaturvedi wants to merge 1 commit into
Conversation
…y bandwidth and queue length series Summary: ## Change - `_get_memory_bw_time_series_for_rank` and `_get_queue_length_time_series_for_rank` sort the points of one timestamp by one rule: first the ends of intervals of positive length, then the starts of such intervals, then each zero-length interval, its start just before its end. - Points of one class sort by structural keys, not by trace row order. Memory bandwidth: step (a zero-length copy uses its bandwidth for its start and its end), pid, copy, phase. Queue length: pid, tid, queue entry (the GPU event's row index), phase. - A launch still counts before its own GPU event. A launch and the start of its GPU event in one microsecond are a zero-length queue entry. - `_get_memory_bw_time_series_for_rank`: removed the workaround that set `dur = 1` for zero-length copies (GitHub issue facebookresearch#20). The new rule makes it unnecessary: a zero-length copy's start sorts just before its own end, so the copy keeps its own point and no level goes negative. - New test target `tests:test_trace_counters` (10 tests, ported from the internal OpenHTA copy). - `tests/test_trace_analysis.py`: updated the stream 7 queue length mean and std (see Impact). ## Why Root cause 1: `sort_values(by="ts")` uses quicksort, which is not stable. Rows with one timestamp come out in an order that depends on the row order of the whole rank. A kernel can count before its own launch (queue length -1), and the result changes when the trace rows change order. Root cause 2: both series write one row per interval edge (+value at the start, -value at the end). A stable sort by `ts` alone keeps the concat order, which puts every start before every end at one timestamp. Copy A [0, 5) at 10 GB/s and copy B [5, 9) at 20 GB/s give the levels 10, 30, 20: the level 30 never existed, and `describe()` counts it. Two zero-length copies C and D in one microsecond give C, C+D, D. The queue length has the same fault: a kernel that starts at t (its entry leaves the queue) and a launch at t (a new entry) give L+1. Why the fix is general: the rule follows from the interval semantics `[start, end)`. Each level is then a sum over intervals that hold its timestamp. No level adds an interval that ends at a timestamp to one that starts there. A zero-length interval keeps a point of its own and adds to no other zero-length interval. Ties inside one class sort by structural fields (timestamps, durations, pid, tid, row identity). The rule has no constant and no trace-specific term. It is the same rule and keys as the internal OpenHTA copy (D130996907 and its follow-up). ## Impact - Metric 1: number of different results over 20 random row orders of one rank. Measured with a temporary test (not in this diff) that shuffles `t.traces[0]` and hashes the output of both functions, on the test traces `vision_transformer`, `negative_queue_length_values_check`, `h100` and `inference_single_rank` (rank 0). - Queue length: before 20 of 20 different on all four traces; after 1. - Memory bandwidth: 1 before and after (these traces have no adjacent copies). - Metric 2: levels that never existed, in synthetic series (new tests). - Before: adjacent copies give 10, 30, 20 (count 3, max 30 for 2 copies); adjacent queue entries and two kernels launched and started in one microsecond give 1, 2, 1, 0; queue length -1 values. - After: 10, 0, 20, 0 (count 2, max 20); 1, 0, 1, 0; no negative level. - Queue length statistics on the test traces (base -> this diff): - `vision_transformer` stream 7: mean 61.0431 -> 61.0345, std 92.3354 -> 92.3387. Count, min, quartiles and max do not change. - `inference_single_rank` stream 7: mean 0.7458 -> 0.7257. `h100` stream 16: mean 1.6125 -> 1.5958. - Max does not change on any stream. - `get_time_spent_blocked_on_full_queue` at the max queue length: no change on the four traces. - Memory bandwidth: no level changes on the four traces. On `h100`, the end points of 940 zero-length memory kernels move from `t + 1` to `t` (the 1 us workaround is gone). `get_memory_bw_summary` does not change. - Cost: one sort over 4 to 6 keys in place of 1. Not measured in OSS; the internal copy measured no change within single-run noise. ## Assumptions and risks - Rounded timestamps are exact. A tie in the trace is a tie of instants; an end and a start in one microsecond do not overlap. - Rows at one timestamp remain, one per event. Their intermediate levels now under-state (each counts only intervals that hold the timestamp) and never over-state. - `generate_trace_with_counters` shows a zero-length copy as two counter points at one timestamp (zero width), not as a 1 us step. The parser rounds sub-microsecond copies to 0 us; `HTA_DISABLE_NS_ROUNDING=1` keeps the real duration. This diff does not change the parser. - The rule must stay the same as in the internal OpenHTA copy and HTA-RS. If one changes the rule, change the others. - Users that pinned exact queue length statistics see small changes at tie points, as in the updated assertion. Differential Revision: D131411321
|
@anubhavchaturvedi has exported this pull request. If you are a Meta employee, you can view the originating Diff in D131411321. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Change
_get_memory_bw_time_series_for_rankand_get_queue_length_time_series_for_ranksort the points of one timestamp by one rule: first the ends of intervals of positive length, then the starts of such intervals, then each zero-length interval, its start just before its end._get_memory_bw_time_series_for_rank: removed the workaround that setdur = 1for zero-length copies (GitHub issue Memory bandwidth shows negative values #20). The new rule makes it unnecessary: a zero-length copy's start sorts just before its own end, so the copy keeps its own point and no level goes negative.tests:test_trace_counters(10 tests, ported from the internal OpenHTA copy).tests/test_trace_analysis.py: updated the stream 7 queue length mean and std (see Impact).Why
Root cause 1:
sort_values(by="ts")uses quicksort, which is not stable. Rows with one timestamp come out in an order that depends on the row order of the whole rank. A kernel can count before its own launch (queue length -1), and the result changes when the trace rows change order.Root cause 2: both series write one row per interval edge (+value at the start, -value at the end). A stable sort by
tsalone keeps the concat order, which puts every start before every end at one timestamp. Copy A [0, 5) at 10 GB/s and copy B [5, 9) at 20 GB/s give the levels 10, 30, 20: the level 30 never existed, anddescribe()counts it. Two zero-length copies C and D in one microsecond give C, C+D, D. The queue length has the same fault: a kernel that starts at t (its entry leaves the queue) and a launch at t (a new entry) give L+1.Why the fix is general: the rule follows from the interval semantics
[start, end). Each level is then a sum over intervals that hold its timestamp. No level adds an interval that ends at a timestamp to one that starts there. A zero-length interval keeps a point of its own and adds to no other zero-length interval. Ties inside one class sort by structural fields (timestamps, durations, pid, tid, row identity). The rule has no constant and no trace-specific term. It is the same rule and keys as the internal OpenHTA copy (D130996907 and its follow-up).Impact
t.traces[0]and hashes the output of both functions, on the test tracesvision_transformer,negative_queue_length_values_check,h100andinference_single_rank(rank 0).vision_transformerstream 7: mean 61.0431 -> 61.0345, std 92.3354 -> 92.3387. Count, min, quartiles and max do not change.inference_single_rankstream 7: mean 0.7458 -> 0.7257.h100stream 16: mean 1.6125 -> 1.5958.get_time_spent_blocked_on_full_queueat the max queue length: no change on the four traces.h100, the end points of 940 zero-length memory kernels move fromt + 1tot(the 1 us workaround is gone).get_memory_bw_summarydoes not change.Assumptions and risks
generate_trace_with_countersshows a zero-length copy as two counter points at one timestamp (zero width), not as a 1 us step. The parser rounds sub-microsecond copies to 0 us;HTA_DISABLE_NS_ROUNDING=1keeps the real duration. This diff does not change the parser.Differential Revision: D131411321