Skip to content

Fix nan value generated after custom all reduce - #8663

Merged
zhyncs merged 6 commits into
sgl-project:mainfrom
HaiShaw:fix_car_tensor-nan
Aug 15, 2025
Merged

zhyncs merged 6 commits into
sgl-project:mainfrom
HaiShaw:fix_car_tensor-nan

Conversation

@kkHuang-amd

@kkHuang-amd kkHuang-amd commented Aug 1, 2025 •

Copy link
Copy Markdown
Collaborator

Motivation

Refer the previous PR #8532.

We have reviewed the change that only happened on graph capture stage, it is not critical path for performance run.

Modifications

  • Change the empty_like and empty to zeros_like and zeros to ensure the tensor value is initialized to zero
  • Move custom allreduce test to a different job in AMD CI

Accuracy Test

python3 benchmark/gsm8k/bench_sglang.py --num-questions 2000 --parallel 2000 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1319/1319 [00:51<00:00, 25.58it/s] Accuracy: 0.948 Invalid: 0.000 Latency: 52.566 s Output throughput: 2409.507 token/s

Checklist

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary of Changes

Hello @kkHuang-amd, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

I've implemented a fix to address the generation of NaN values that were observed after a custom all-reduce operation. The core of this change involves ensuring that tensors are properly initialized to zeros rather than being left uninitialized, specifically during the warm-up phase of the all-reduce process. This modification aims to improve numerical stability without impacting performance-critical sections of the code.

Highlights

  • Fix NaN Generation: The primary goal of this change is to prevent the generation of NaN (Not-a-Number) values that were occurring after a custom all-reduce operation.
  • Tensor Initialization Change: I've modified the tensor initialization from torch.empty_like to torch.zeros_like to ensure that allocated tensors are explicitly initialized with zeros, thereby avoiding uninitialized memory that could lead to NaNs.
  • Targeted Warm-up Phase Fix: This specific modification is applied to the warm-up phase of the custom_all_reduce function, where an out-of-place allocation pattern is mimicked. The PR description notes that this change is not on a critical performance path.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in issue comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request addresses an issue where NaN values could be generated during the custom all-reduce process. The fix involves replacing torch.empty_like with torch.zeros_like in the CUDA graph warmup path to ensure tensors are initialized to zero. The change is correct and targets a non-performance-critical path. As the PR description suggests a broader change, I've also suggested reviewing other instances of torch.empty_like and torch.empty in the file to ensure consistency and prevent similar issues in other code paths.

@HaiShaw HaiShaw left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@HaiShaw

HaiShaw commented Aug 1, 2025

Copy link
Copy Markdown
Collaborator

@yizhang2077 review please

@hubertlu-tw

Copy link
Copy Markdown
Collaborator

The failing test may be fixed by this PR: #8681

@hubertlu-tw
hubertlu-tw requested a review from merrymercy as a code owner August 5, 2025 15:11
@hubertlu-tw
hubertlu-tw requested a review from zhyncs as a code owner August 5, 2025 20:12

@yizhang2077 yizhang2077 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@amdosoldea

Copy link
Copy Markdown

lgtm.

@zhyncs
zhyncs merged commit 983aa49 into sgl-project:main Aug 15, 2025
narutolhy pushed a commit to narutolhy/sglang that referenced this pull request Aug 17, 2025
MahmoudAshraf97 pushed a commit to MahmoudAshraf97/sglang that referenced this pull request Sep 8, 2025
0826joyce pushed a commit to 0826joyce/sglang-perf-opt that referenced this pull request May 19, 2026
Co-authored-by: wunhuang <wunhuang@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants