Skip to content

Ignore CUDA event waits on a later or same-stream record in critical path graphs (#373) - #373

Open
triandicAnt wants to merge 1 commit into
facebookresearch:mainfrom
triandicAnt:export-D129450013
Open

triandicAnt wants to merge 1 commit into
facebookresearch:mainfrom
triandicAnt:export-D129450013

Conversation

@triandicAnt

@triandicAnt triandicAnt commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Summary:

Proposed by the Zoomer fix agent from the bug hunt finding on T296602757. Unreviewed: a person must review, publish and land it, and landing does not deploy it.

Code owners:

  • HTA (open source): hta
    This changes another team's code: that oncall must review and accept it, and its owners decide how it ships.

Symptom

  • profiling_critical_path_analysis_fail, SYSTEM, RuntimeError: Critical path analysis failed (Scuba ai_zoomer_service_events): 357 events on 357 runs (336 jobs) in the 7 days to 2026-11-03 12:40 PT, 129 in the last 48 hours.
  • 354 of the 357 are fire-* MVAI training jobs. Example runs: 1680121590472551, 1601415481178392, 961019183324577.
  • Zoomer raises at ai_infra/efficiency/atc2/training/training_critical_path_analysis.py:143 when HTA's critical_path() returns False.

Root cause (HTA hta/analyzers/critical_path_analysis.py)

  • _validate_graph (:1549-1571) rejects the graph. handle_cuda_sync (:1255-1300) added GPU→GPU sync edges between kernels on one stream, from two kinds of event wait in these traces:
    1. A wait naming a cudaEventRecord issued after it. In run 961019183324577, the Stream Wait Event with correlation 1334325 at 503326 us names record 1357527, issued at 612418 us. HTA then makes the kernel at 503370 us wait for one at 650316 us on the same stream. Kineto stopped emitting such IDs in D99131994 (2026-04-16), but these traces still carry them. I did not check which Kineto build each job runs.
    2. A wait on an event recorded on the wait's own stream (wait_on_stream == stream). CUDA treats it as a no-op, but HTA turns it into a same-stream sync edge that its own validator rejects.

Fix

  • _check_event_sync_helper ignores a Stream Wait Event or Event Sync whose wait_on_cuda_event_record_corr_id is larger than the wait's own correlation. This is Kineto's rule: a wait depends only on a record issued before it.
  • handle_cuda_sync skips a Stream Wait Event whose wait_on_stream equals its stream.

Replay

  • 24 failing runs from 10-27 to 11-02. I ran the rank-1 trace Zoomer analyzes through Zoomer's _critical_path_analysis with the fixed HTA.
  • 22 now produce a critical path. All 24 failed in production with this signature, and the 3 I ran on master HTA fail validation locally.
  • 22 of the 24 carry waits naming a later record (7 to 53,946 per trace).
  • The 2 left, 1334519609752750 and 1584263437075640, have no such wait and fail on graph cycles. That is a separate cause, left open.

Risk

  • HTA's CPA serves users outside Zoomer. Results change only for traces with such waits.
  • The dropped edges were either impossible (a later record) or already implied by stream order (same stream). The existing event-sync tests still pass.
  • For Zoomer, SYSTEM failures of this signature drop by up to about 330 a week. No label changes.

Shipping

  • HTA is open source: ShipIt exports it to GitHub (facebookresearch/HolisticTraceAnalysis) on land.
  • Zoomer links HTA from fbcode, so the fix reaches it with the next Zoomer package build and deploy. Landing does not deploy it.

Agent trajectory: https://www.internalfb.com/trajectory/12b1d160-5500-459d-b7ea-30c39b8c81de

Zoomer fix binding: T296602757 zoomer-hunt-2064398830925675

Differential Revision: D129450013

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 4, 2026
@meta-codesync

meta-codesync Bot commented Nov 4, 2026

Copy link
Copy Markdown

@triandicAnt has exported this pull request. If you are a Meta employee, you can view the originating Diff in D129450013.

@meta-codesync meta-codesync Bot changed the title Ignore CUDA event waits on a later or same-stream record in critical path graphs Ignore CUDA event waits on a later or same-stream record in critical path graphs (#373) Nov 4, 2026
triandicAnt added a commit to triandicAnt/HolisticTraceAnalysis that referenced this pull request Nov 4, 2026
…path graphs (facebookresearch#373)

Summary:

Proposed by the Zoomer fix agent from the bug hunt finding on T296602757. Unreviewed: a person must review, publish and land it, and landing does not deploy it.

Code owners:
- HTA (open source): hta
This changes another team's code: that oncall must review and accept it, and its owners decide how it ships.

**Symptom**
- `profiling_critical_path_analysis_fail`, SYSTEM, `RuntimeError: Critical path analysis failed` (Scuba `ai_zoomer_service_events`): 357 events on 357 runs (336 jobs) in the 7 days to 2026-11-03 12:40 PT, 129 in the last 48 hours.
- 354 of the 357 are `fire-*` MVAI training jobs. Example runs: 1680121590472551, 1601415481178392, 961019183324577.
- Zoomer raises at `ai_infra/efficiency/atc2/training/training_critical_path_analysis.py:143` when HTA's `critical_path()` returns False.

**Root cause** (HTA `hta/analyzers/critical_path_analysis.py`)
- `_validate_graph` (:1549-1571) rejects the graph. `handle_cuda_sync` (:1255-1300) added GPU→GPU sync edges between kernels on one stream, from two kinds of event wait in these traces:
  1. **A wait naming a cudaEventRecord issued after it.** In run 961019183324577, the `Stream Wait Event` with correlation 1334325 at 503326 us names record 1357527, issued at 612418 us. HTA then makes the kernel at 503370 us wait for one at 650316 us on the same stream. Kineto stopped emitting such IDs in D99131994 (2026-04-16), but these traces still carry them. I did not check which Kineto build each job runs.
  2. **A wait on an event recorded on the wait's own stream** (`wait_on_stream == stream`). CUDA treats it as a no-op, but HTA turns it into a same-stream sync edge that its own validator rejects.

**Fix**
- `_check_event_sync_helper` ignores a `Stream Wait Event` or `Event Sync` whose `wait_on_cuda_event_record_corr_id` is larger than the wait's own correlation. This is Kineto's rule: a wait depends only on a record issued before it.
- `handle_cuda_sync` skips a `Stream Wait Event` whose `wait_on_stream` equals its stream.

**Replay**
- 24 failing runs from 10-27 to 11-02. I ran the rank-1 trace Zoomer analyzes through Zoomer's `_critical_path_analysis` with the fixed HTA.
- 22 now produce a critical path. All 24 failed in production with this signature, and the 3 I ran on master HTA fail validation locally.
- 22 of the 24 carry waits naming a later record (7 to 53,946 per trace).
- The 2 left, 1334519609752750 and 1584263437075640, have no such wait and fail on graph cycles. That is a separate cause, left open.

**Risk**
- HTA's CPA serves users outside Zoomer. Results change only for traces with such waits.
- The dropped edges were either impossible (a later record) or already implied by stream order (same stream). The existing event-sync tests still pass.
- For Zoomer, SYSTEM failures of this signature drop by up to about 330 a week. No label changes.

**Shipping**
- HTA is open source: ShipIt exports it to GitHub (facebookresearch/HolisticTraceAnalysis) on land.
- Zoomer links HTA from fbcode, so the fix reaches it with the next Zoomer package build and deploy. Landing does not deploy it.

Agent trajectory: https://www.internalfb.com/trajectory/12b1d160-5500-459d-b7ea-30c39b8c81de

Zoomer fix binding: T296602757 zoomer-hunt-2064398830925675

Differential Revision: D129450013
triandicAnt added a commit to triandicAnt/HolisticTraceAnalysis that referenced this pull request Nov 4, 2026
…path graphs (facebookresearch#373)

Summary:

Proposed by the Zoomer fix agent from the bug hunt finding on T296602757. Unreviewed: a person must review, publish and land it, and landing does not deploy it.

Code owners:
- HTA (open source): hta
This changes another team's code: that oncall must review and accept it, and its owners decide how it ships.

**Symptom**
- `profiling_critical_path_analysis_fail`, SYSTEM, `RuntimeError: Critical path analysis failed` (Scuba `ai_zoomer_service_events`): 357 events on 357 runs (336 jobs) in the 7 days to 2026-11-03 12:40 PT, 129 in the last 48 hours.
- 354 of the 357 are `fire-*` MVAI training jobs. Example runs: 1680121590472551, 1601415481178392, 961019183324577.
- Zoomer raises at `ai_infra/efficiency/atc2/training/training_critical_path_analysis.py:143` when HTA's `critical_path()` returns False.

**Root cause** (HTA `hta/analyzers/critical_path_analysis.py`)
- `_validate_graph` (:1549-1571) rejects the graph. `handle_cuda_sync` (:1255-1300) added GPU→GPU sync edges between kernels on one stream, from two kinds of event wait in these traces:
  1. **A wait naming a cudaEventRecord issued after it.** In run 961019183324577, the `Stream Wait Event` with correlation 1334325 at 503326 us names record 1357527, issued at 612418 us. HTA then makes the kernel at 503370 us wait for one at 650316 us on the same stream. Kineto stopped emitting such IDs in D99131994 (2026-04-16), but these traces still carry them. I did not check which Kineto build each job runs.
  2. **A wait on an event recorded on the wait's own stream** (`wait_on_stream == stream`). CUDA treats it as a no-op, but HTA turns it into a same-stream sync edge that its own validator rejects.

**Fix**
- `_check_event_sync_helper` ignores a `Stream Wait Event` or `Event Sync` whose `wait_on_cuda_event_record_corr_id` is larger than the wait's own correlation. This is Kineto's rule: a wait depends only on a record issued before it.
- `handle_cuda_sync` skips a `Stream Wait Event` whose `wait_on_stream` equals its stream.

**Replay**
- 24 failing runs from 10-27 to 11-02. I ran the rank-1 trace Zoomer analyzes through Zoomer's `_critical_path_analysis` with the fixed HTA.
- 22 now produce a critical path. All 24 failed in production with this signature, and the 3 I ran on master HTA fail validation locally.
- 22 of the 24 carry waits naming a later record (7 to 53,946 per trace).
- The 2 left, 1334519609752750 and 1584263437075640, have no such wait and fail on graph cycles. That is a separate cause, left open.

**Risk**
- HTA's CPA serves users outside Zoomer. Results change only for traces with such waits.
- The dropped edges were either impossible (a later record) or already implied by stream order (same stream). The existing event-sync tests still pass.
- For Zoomer, SYSTEM failures of this signature drop by up to about 330 a week. No label changes.

**Shipping**
- HTA is open source: ShipIt exports it to GitHub (facebookresearch/HolisticTraceAnalysis) on land.
- Zoomer links HTA from fbcode, so the fix reaches it with the next Zoomer package build and deploy. Landing does not deploy it.

Agent trajectory: https://www.internalfb.com/trajectory/12b1d160-5500-459d-b7ea-30c39b8c81de

Zoomer fix binding: T296602757 zoomer-hunt-2064398830925675

Differential Revision: D129450013
triandicAnt added a commit to triandicAnt/HolisticTraceAnalysis that referenced this pull request Nov 9, 2026
…path graphs (facebookresearch#373)

Summary:

Proposed by the Zoomer fix agent from the bug hunt finding on T296602757. Unreviewed: a person must review, publish and land it, and landing does not deploy it.

Code owners:
- HTA (open source): hta
This changes another team's code: that oncall must review and accept it, and its owners decide how it ships.

**Symptom**
- `profiling_critical_path_analysis_fail`, SYSTEM, `RuntimeError: Critical path analysis failed` (Scuba `ai_zoomer_service_events`): 357 events on 357 runs (336 jobs) in the 7 days to 2026-11-03 12:40 PT, 129 in the last 48 hours.
- 354 of the 357 are `fire-*` MVAI training jobs. Example runs: 1680121590472551, 1601415481178392, 961019183324577.
- Zoomer raises at `ai_infra/efficiency/atc2/training/training_critical_path_analysis.py:143` when HTA's `critical_path()` returns False.

**Root cause** (HTA `hta/analyzers/critical_path_analysis.py`)
- `_validate_graph` (:1549-1571) rejects the graph. `handle_cuda_sync` (:1255-1300) added GPU→GPU sync edges between kernels on one stream, from two kinds of event wait in these traces:
  1. **A wait naming a cudaEventRecord issued after it.** In run 961019183324577, the `Stream Wait Event` with correlation 1334325 at 503326 us names record 1357527, issued at 612418 us. HTA then makes the kernel at 503370 us wait for one at 650316 us on the same stream. Kineto stopped emitting such IDs in D99131994 (2026-04-16), but these traces still carry them. I did not check which Kineto build each job runs.
  2. **A wait on an event recorded on the wait's own stream** (`wait_on_stream == stream`). CUDA treats it as a no-op, but HTA turns it into a same-stream sync edge that its own validator rejects.

**Fix**
- `_check_event_sync_helper` ignores a `Stream Wait Event` or `Event Sync` whose `wait_on_cuda_event_record_corr_id` is larger than the wait's own correlation. This is Kineto's rule: a wait depends only on a record issued before it.
- `handle_cuda_sync` skips a `Stream Wait Event` whose `wait_on_stream` equals its stream.

**Replay**
- 24 failing runs from 10-27 to 11-02. I ran the rank-1 trace Zoomer analyzes through Zoomer's `_critical_path_analysis` with the fixed HTA.
- 22 now produce a critical path. All 24 failed in production with this signature, and the 3 I ran on master HTA fail validation locally.
- 22 of the 24 carry waits naming a later record (7 to 53,946 per trace).
- The 2 left, 1334519609752750 and 1584263437075640, have no such wait and fail on graph cycles. That is a separate cause, left open.

**Risk**
- HTA's CPA serves users outside Zoomer. Results change only for traces with such waits.
- The dropped edges were either impossible (a later record) or already implied by stream order (same stream). The existing event-sync tests still pass.
- For Zoomer, SYSTEM failures of this signature drop by up to about 330 a week. No label changes.

**Shipping**
- HTA is open source: ShipIt exports it to GitHub (facebookresearch/HolisticTraceAnalysis) on land.
- Zoomer links HTA from fbcode, so the fix reaches it with the next Zoomer package build and deploy. Landing does not deploy it.

Agent trajectory: https://www.internalfb.com/trajectory/12b1d160-5500-459d-b7ea-30c39b8c81de

Zoomer fix binding: T296602757 zoomer-hunt-2064398830925675

Differential Revision: D129450013
…path graphs (facebookresearch#373)

Summary:

Proposed by the Zoomer fix agent from the bug hunt finding on T296602757. Unreviewed: a person must review, publish and land it, and landing does not deploy it.

Code owners:
- HTA (open source): hta
This changes another team's code: that oncall must review and accept it, and its owners decide how it ships.

**Symptom**
- `profiling_critical_path_analysis_fail`, SYSTEM, `RuntimeError: Critical path analysis failed` (Scuba `ai_zoomer_service_events`): 357 events on 357 runs (336 jobs) in the 7 days to 2026-11-03 12:40 PT, 129 in the last 48 hours.
- 354 of the 357 are `fire-*` MVAI training jobs. Example runs: 1680121590472551, 1601415481178392, 961019183324577.
- Zoomer raises at `ai_infra/efficiency/atc2/training/training_critical_path_analysis.py:143` when HTA's `critical_path()` returns False.

**Root cause** (HTA `hta/analyzers/critical_path_analysis.py`)
- `_validate_graph` (:1549-1571) rejects the graph. `handle_cuda_sync` (:1255-1300) added GPU→GPU sync edges between kernels on one stream, from two kinds of event wait in these traces:
  1. **A wait naming a cudaEventRecord issued after it.** In run 961019183324577, the `Stream Wait Event` with correlation 1334325 at 503326 us names record 1357527, issued at 612418 us. HTA then makes the kernel at 503370 us wait for one at 650316 us on the same stream. Kineto stopped emitting such IDs in D99131994 (2026-04-16), but these traces still carry them. I did not check which Kineto build each job runs.
  2. **A wait on an event recorded on the wait's own stream** (`wait_on_stream == stream`). CUDA treats it as a no-op, but HTA turns it into a same-stream sync edge that its own validator rejects.

**Fix**
- `_check_event_sync_helper` ignores a `Stream Wait Event` or `Event Sync` whose `wait_on_cuda_event_record_corr_id` is larger than the wait's own correlation. This is Kineto's rule: a wait depends only on a record issued before it.
- `handle_cuda_sync` skips a `Stream Wait Event` whose `wait_on_stream` equals its stream.

**Replay**
- 24 failing runs from 10-27 to 11-02. I ran the rank-1 trace Zoomer analyzes through Zoomer's `_critical_path_analysis` with the fixed HTA.
- 22 now produce a critical path. All 24 failed in production with this signature, and the 3 I ran on master HTA fail validation locally.
- 22 of the 24 carry waits naming a later record (7 to 53,946 per trace).
- The 2 left, 1334519609752750 and 1584263437075640, have no such wait and fail on graph cycles. That is a separate cause, left open.

**Risk**
- HTA's CPA serves users outside Zoomer. Results change only for traces with such waits.
- The dropped edges were either impossible (a later record) or already implied by stream order (same stream). The existing event-sync tests still pass.
- For Zoomer, SYSTEM failures of this signature drop by up to about 330 a week. No label changes.

**Shipping**
- HTA is open source: ShipIt exports it to GitHub (facebookresearch/HolisticTraceAnalysis) on land.
- Zoomer links HTA from fbcode, so the fix reaches it with the next Zoomer package build and deploy. Landing does not deploy it.

Agent trajectory: https://www.internalfb.com/trajectory/12b1d160-5500-459d-b7ea-30c39b8c81de

Zoomer fix binding: T296602757 zoomer-hunt-2064398830925675

Differential Revision: D129450013

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. meta-exported

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant