Repository navigation
Ignore CUDA event waits on a later or same-stream record in critical path graphs (#373) - #373
Open
triandicAnt wants to merge 1 commit into
Open
triandicAnt wants to merge 1 commit into
triandicAnt wants to merge 1 commit into
Conversation
|
@triandicAnt has exported this pull request. If you are a Meta employee, you can view the originating Diff in D129450013. |
triandicAnt
added a commit
to triandicAnt/HolisticTraceAnalysis
that referenced
this pull request
Nov 4, 2026
…path graphs (facebookresearch#373) Summary: Proposed by the Zoomer fix agent from the bug hunt finding on T296602757. Unreviewed: a person must review, publish and land it, and landing does not deploy it. Code owners: - HTA (open source): hta This changes another team's code: that oncall must review and accept it, and its owners decide how it ships. **Symptom** - `profiling_critical_path_analysis_fail`, SYSTEM, `RuntimeError: Critical path analysis failed` (Scuba `ai_zoomer_service_events`): 357 events on 357 runs (336 jobs) in the 7 days to 2026-11-03 12:40 PT, 129 in the last 48 hours. - 354 of the 357 are `fire-*` MVAI training jobs. Example runs: 1680121590472551, 1601415481178392, 961019183324577. - Zoomer raises at `ai_infra/efficiency/atc2/training/training_critical_path_analysis.py:143` when HTA's `critical_path()` returns False. **Root cause** (HTA `hta/analyzers/critical_path_analysis.py`) - `_validate_graph` (:1549-1571) rejects the graph. `handle_cuda_sync` (:1255-1300) added GPU→GPU sync edges between kernels on one stream, from two kinds of event wait in these traces: 1. **A wait naming a cudaEventRecord issued after it.** In run 961019183324577, the `Stream Wait Event` with correlation 1334325 at 503326 us names record 1357527, issued at 612418 us. HTA then makes the kernel at 503370 us wait for one at 650316 us on the same stream. Kineto stopped emitting such IDs in D99131994 (2026-04-16), but these traces still carry them. I did not check which Kineto build each job runs. 2. **A wait on an event recorded on the wait's own stream** (`wait_on_stream == stream`). CUDA treats it as a no-op, but HTA turns it into a same-stream sync edge that its own validator rejects. **Fix** - `_check_event_sync_helper` ignores a `Stream Wait Event` or `Event Sync` whose `wait_on_cuda_event_record_corr_id` is larger than the wait's own correlation. This is Kineto's rule: a wait depends only on a record issued before it. - `handle_cuda_sync` skips a `Stream Wait Event` whose `wait_on_stream` equals its stream. **Replay** - 24 failing runs from 10-27 to 11-02. I ran the rank-1 trace Zoomer analyzes through Zoomer's `_critical_path_analysis` with the fixed HTA. - 22 now produce a critical path. All 24 failed in production with this signature, and the 3 I ran on master HTA fail validation locally. - 22 of the 24 carry waits naming a later record (7 to 53,946 per trace). - The 2 left, 1334519609752750 and 1584263437075640, have no such wait and fail on graph cycles. That is a separate cause, left open. **Risk** - HTA's CPA serves users outside Zoomer. Results change only for traces with such waits. - The dropped edges were either impossible (a later record) or already implied by stream order (same stream). The existing event-sync tests still pass. - For Zoomer, SYSTEM failures of this signature drop by up to about 330 a week. No label changes. **Shipping** - HTA is open source: ShipIt exports it to GitHub (facebookresearch/HolisticTraceAnalysis) on land. - Zoomer links HTA from fbcode, so the fix reaches it with the next Zoomer package build and deploy. Landing does not deploy it. Agent trajectory: https://www.internalfb.com/trajectory/12b1d160-5500-459d-b7ea-30c39b8c81de Zoomer fix binding: T296602757 zoomer-hunt-2064398830925675 Differential Revision: D129450013
triandicAnt
force-pushed
the
export-D129450013
branch
from
November 4, 2026 20:17
2fc1483 to
a41844a
Compare
triandicAnt
added a commit
to triandicAnt/HolisticTraceAnalysis
that referenced
this pull request
Nov 4, 2026
…path graphs (facebookresearch#373) Summary: Proposed by the Zoomer fix agent from the bug hunt finding on T296602757. Unreviewed: a person must review, publish and land it, and landing does not deploy it. Code owners: - HTA (open source): hta This changes another team's code: that oncall must review and accept it, and its owners decide how it ships. **Symptom** - `profiling_critical_path_analysis_fail`, SYSTEM, `RuntimeError: Critical path analysis failed` (Scuba `ai_zoomer_service_events`): 357 events on 357 runs (336 jobs) in the 7 days to 2026-11-03 12:40 PT, 129 in the last 48 hours. - 354 of the 357 are `fire-*` MVAI training jobs. Example runs: 1680121590472551, 1601415481178392, 961019183324577. - Zoomer raises at `ai_infra/efficiency/atc2/training/training_critical_path_analysis.py:143` when HTA's `critical_path()` returns False. **Root cause** (HTA `hta/analyzers/critical_path_analysis.py`) - `_validate_graph` (:1549-1571) rejects the graph. `handle_cuda_sync` (:1255-1300) added GPU→GPU sync edges between kernels on one stream, from two kinds of event wait in these traces: 1. **A wait naming a cudaEventRecord issued after it.** In run 961019183324577, the `Stream Wait Event` with correlation 1334325 at 503326 us names record 1357527, issued at 612418 us. HTA then makes the kernel at 503370 us wait for one at 650316 us on the same stream. Kineto stopped emitting such IDs in D99131994 (2026-04-16), but these traces still carry them. I did not check which Kineto build each job runs. 2. **A wait on an event recorded on the wait's own stream** (`wait_on_stream == stream`). CUDA treats it as a no-op, but HTA turns it into a same-stream sync edge that its own validator rejects. **Fix** - `_check_event_sync_helper` ignores a `Stream Wait Event` or `Event Sync` whose `wait_on_cuda_event_record_corr_id` is larger than the wait's own correlation. This is Kineto's rule: a wait depends only on a record issued before it. - `handle_cuda_sync` skips a `Stream Wait Event` whose `wait_on_stream` equals its stream. **Replay** - 24 failing runs from 10-27 to 11-02. I ran the rank-1 trace Zoomer analyzes through Zoomer's `_critical_path_analysis` with the fixed HTA. - 22 now produce a critical path. All 24 failed in production with this signature, and the 3 I ran on master HTA fail validation locally. - 22 of the 24 carry waits naming a later record (7 to 53,946 per trace). - The 2 left, 1334519609752750 and 1584263437075640, have no such wait and fail on graph cycles. That is a separate cause, left open. **Risk** - HTA's CPA serves users outside Zoomer. Results change only for traces with such waits. - The dropped edges were either impossible (a later record) or already implied by stream order (same stream). The existing event-sync tests still pass. - For Zoomer, SYSTEM failures of this signature drop by up to about 330 a week. No label changes. **Shipping** - HTA is open source: ShipIt exports it to GitHub (facebookresearch/HolisticTraceAnalysis) on land. - Zoomer links HTA from fbcode, so the fix reaches it with the next Zoomer package build and deploy. Landing does not deploy it. Agent trajectory: https://www.internalfb.com/trajectory/12b1d160-5500-459d-b7ea-30c39b8c81de Zoomer fix binding: T296602757 zoomer-hunt-2064398830925675 Differential Revision: D129450013
triandicAnt
force-pushed
the
export-D129450013
branch
from
November 4, 2026 23:43
a41844a to
c18ead8
Compare
triandicAnt
added a commit
to triandicAnt/HolisticTraceAnalysis
that referenced
this pull request
Nov 9, 2026
…path graphs (facebookresearch#373) Summary: Proposed by the Zoomer fix agent from the bug hunt finding on T296602757. Unreviewed: a person must review, publish and land it, and landing does not deploy it. Code owners: - HTA (open source): hta This changes another team's code: that oncall must review and accept it, and its owners decide how it ships. **Symptom** - `profiling_critical_path_analysis_fail`, SYSTEM, `RuntimeError: Critical path analysis failed` (Scuba `ai_zoomer_service_events`): 357 events on 357 runs (336 jobs) in the 7 days to 2026-11-03 12:40 PT, 129 in the last 48 hours. - 354 of the 357 are `fire-*` MVAI training jobs. Example runs: 1680121590472551, 1601415481178392, 961019183324577. - Zoomer raises at `ai_infra/efficiency/atc2/training/training_critical_path_analysis.py:143` when HTA's `critical_path()` returns False. **Root cause** (HTA `hta/analyzers/critical_path_analysis.py`) - `_validate_graph` (:1549-1571) rejects the graph. `handle_cuda_sync` (:1255-1300) added GPU→GPU sync edges between kernels on one stream, from two kinds of event wait in these traces: 1. **A wait naming a cudaEventRecord issued after it.** In run 961019183324577, the `Stream Wait Event` with correlation 1334325 at 503326 us names record 1357527, issued at 612418 us. HTA then makes the kernel at 503370 us wait for one at 650316 us on the same stream. Kineto stopped emitting such IDs in D99131994 (2026-04-16), but these traces still carry them. I did not check which Kineto build each job runs. 2. **A wait on an event recorded on the wait's own stream** (`wait_on_stream == stream`). CUDA treats it as a no-op, but HTA turns it into a same-stream sync edge that its own validator rejects. **Fix** - `_check_event_sync_helper` ignores a `Stream Wait Event` or `Event Sync` whose `wait_on_cuda_event_record_corr_id` is larger than the wait's own correlation. This is Kineto's rule: a wait depends only on a record issued before it. - `handle_cuda_sync` skips a `Stream Wait Event` whose `wait_on_stream` equals its stream. **Replay** - 24 failing runs from 10-27 to 11-02. I ran the rank-1 trace Zoomer analyzes through Zoomer's `_critical_path_analysis` with the fixed HTA. - 22 now produce a critical path. All 24 failed in production with this signature, and the 3 I ran on master HTA fail validation locally. - 22 of the 24 carry waits naming a later record (7 to 53,946 per trace). - The 2 left, 1334519609752750 and 1584263437075640, have no such wait and fail on graph cycles. That is a separate cause, left open. **Risk** - HTA's CPA serves users outside Zoomer. Results change only for traces with such waits. - The dropped edges were either impossible (a later record) or already implied by stream order (same stream). The existing event-sync tests still pass. - For Zoomer, SYSTEM failures of this signature drop by up to about 330 a week. No label changes. **Shipping** - HTA is open source: ShipIt exports it to GitHub (facebookresearch/HolisticTraceAnalysis) on land. - Zoomer links HTA from fbcode, so the fix reaches it with the next Zoomer package build and deploy. Landing does not deploy it. Agent trajectory: https://www.internalfb.com/trajectory/12b1d160-5500-459d-b7ea-30c39b8c81de Zoomer fix binding: T296602757 zoomer-hunt-2064398830925675 Differential Revision: D129450013
triandicAnt
force-pushed
the
export-D129450013
branch
from
November 9, 2026 19:24
c18ead8 to
290d3b8
Compare
…path graphs (facebookresearch#373) Summary: Proposed by the Zoomer fix agent from the bug hunt finding on T296602757. Unreviewed: a person must review, publish and land it, and landing does not deploy it. Code owners: - HTA (open source): hta This changes another team's code: that oncall must review and accept it, and its owners decide how it ships. **Symptom** - `profiling_critical_path_analysis_fail`, SYSTEM, `RuntimeError: Critical path analysis failed` (Scuba `ai_zoomer_service_events`): 357 events on 357 runs (336 jobs) in the 7 days to 2026-11-03 12:40 PT, 129 in the last 48 hours. - 354 of the 357 are `fire-*` MVAI training jobs. Example runs: 1680121590472551, 1601415481178392, 961019183324577. - Zoomer raises at `ai_infra/efficiency/atc2/training/training_critical_path_analysis.py:143` when HTA's `critical_path()` returns False. **Root cause** (HTA `hta/analyzers/critical_path_analysis.py`) - `_validate_graph` (:1549-1571) rejects the graph. `handle_cuda_sync` (:1255-1300) added GPU→GPU sync edges between kernels on one stream, from two kinds of event wait in these traces: 1. **A wait naming a cudaEventRecord issued after it.** In run 961019183324577, the `Stream Wait Event` with correlation 1334325 at 503326 us names record 1357527, issued at 612418 us. HTA then makes the kernel at 503370 us wait for one at 650316 us on the same stream. Kineto stopped emitting such IDs in D99131994 (2026-04-16), but these traces still carry them. I did not check which Kineto build each job runs. 2. **A wait on an event recorded on the wait's own stream** (`wait_on_stream == stream`). CUDA treats it as a no-op, but HTA turns it into a same-stream sync edge that its own validator rejects. **Fix** - `_check_event_sync_helper` ignores a `Stream Wait Event` or `Event Sync` whose `wait_on_cuda_event_record_corr_id` is larger than the wait's own correlation. This is Kineto's rule: a wait depends only on a record issued before it. - `handle_cuda_sync` skips a `Stream Wait Event` whose `wait_on_stream` equals its stream. **Replay** - 24 failing runs from 10-27 to 11-02. I ran the rank-1 trace Zoomer analyzes through Zoomer's `_critical_path_analysis` with the fixed HTA. - 22 now produce a critical path. All 24 failed in production with this signature, and the 3 I ran on master HTA fail validation locally. - 22 of the 24 carry waits naming a later record (7 to 53,946 per trace). - The 2 left, 1334519609752750 and 1584263437075640, have no such wait and fail on graph cycles. That is a separate cause, left open. **Risk** - HTA's CPA serves users outside Zoomer. Results change only for traces with such waits. - The dropped edges were either impossible (a later record) or already implied by stream order (same stream). The existing event-sync tests still pass. - For Zoomer, SYSTEM failures of this signature drop by up to about 330 a week. No label changes. **Shipping** - HTA is open source: ShipIt exports it to GitHub (facebookresearch/HolisticTraceAnalysis) on land. - Zoomer links HTA from fbcode, so the fix reaches it with the next Zoomer package build and deploy. Landing does not deploy it. Agent trajectory: https://www.internalfb.com/trajectory/12b1d160-5500-459d-b7ea-30c39b8c81de Zoomer fix binding: T296602757 zoomer-hunt-2064398830925675 Differential Revision: D129450013
triandicAnt
force-pushed
the
export-D129450013
branch
from
November 12, 2026 00:58
290d3b8 to
5b878bd
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Proposed by the Zoomer fix agent from the bug hunt finding on T296602757. Unreviewed: a person must review, publish and land it, and landing does not deploy it.
Code owners:
This changes another team's code: that oncall must review and accept it, and its owners decide how it ships.
Symptom
profiling_critical_path_analysis_fail, SYSTEM,RuntimeError: Critical path analysis failed(Scubaai_zoomer_service_events): 357 events on 357 runs (336 jobs) in the 7 days to 2026-11-03 12:40 PT, 129 in the last 48 hours.fire-*MVAI training jobs. Example runs: 1680121590472551, 1601415481178392, 961019183324577.ai_infra/efficiency/atc2/training/training_critical_path_analysis.py:143when HTA'scritical_path()returns False.Root cause (HTA
hta/analyzers/critical_path_analysis.py)_validate_graph(:1549-1571) rejects the graph.handle_cuda_sync(:1255-1300) added GPU→GPU sync edges between kernels on one stream, from two kinds of event wait in these traces:Stream Wait Eventwith correlation 1334325 at 503326 us names record 1357527, issued at 612418 us. HTA then makes the kernel at 503370 us wait for one at 650316 us on the same stream. Kineto stopped emitting such IDs in D99131994 (2026-04-16), but these traces still carry them. I did not check which Kineto build each job runs.wait_on_stream == stream). CUDA treats it as a no-op, but HTA turns it into a same-stream sync edge that its own validator rejects.Fix
_check_event_sync_helperignores aStream Wait EventorEvent Syncwhosewait_on_cuda_event_record_corr_idis larger than the wait's own correlation. This is Kineto's rule: a wait depends only on a record issued before it.handle_cuda_syncskips aStream Wait Eventwhosewait_on_streamequals its stream.Replay
_critical_path_analysiswith the fixed HTA.Risk
Shipping
Agent trajectory: https://www.internalfb.com/trajectory/12b1d160-5500-459d-b7ea-30c39b8c81de
Zoomer fix binding: T296602757 zoomer-hunt-2064398830925675
Differential Revision: D129450013