Repository navigation
Fix nan value generated after custom all reduce - #8663
Conversation
There was a problem hiding this comment.
Summary of Changes
Hello @kkHuang-amd, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
I've implemented a fix to address the generation of NaN values that were observed after a custom all-reduce operation. The core of this change involves ensuring that tensors are properly initialized to zeros rather than being left uninitialized, specifically during the warm-up phase of the all-reduce process. This modification aims to improve numerical stability without impacting performance-critical sections of the code.
Highlights
- Fix NaN Generation: The primary goal of this change is to prevent the generation of NaN (Not-a-Number) values that were occurring after a custom all-reduce operation.
- Tensor Initialization Change: I've modified the tensor initialization from
torch.empty_liketotorch.zeros_liketo ensure that allocated tensors are explicitly initialized with zeros, thereby avoiding uninitialized memory that could lead to NaNs. - Targeted Warm-up Phase Fix: This specific modification is applied to the warm-up phase of the
custom_all_reducefunction, where an out-of-place allocation pattern is mimicked. The PR description notes that this change is not on a critical performance path.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Code Review
This pull request addresses an issue where NaN values could be generated during the custom all-reduce process. The fix involves replacing torch.empty_like with torch.zeros_like in the CUDA graph warmup path to ensure tensors are initialized to zero. The change is correct and targets a non-performance-critical path. As the PR description suggests a broader change, I've also suggested reviewing other instances of torch.empty_like and torch.empty in the file to ensure consistency and prevent similar issues in other code paths.
|
@yizhang2077 review please |
|
The failing test may be fixed by this PR: #8681 |
|
lgtm. |
Co-authored-by: wunhuang <wunhuang@amd.com>
Co-authored-by: wunhuang <wunhuang@amd.com>
Co-authored-by: wunhuang <wunhuang@amd.com>
Motivation
Refer the previous PR #8532.
We have reviewed the change that only happened on graph capture stage, it is not critical path for performance run.
Modifications
Accuracy Test
python3 benchmark/gsm8k/bench_sglang.py --num-questions 2000 --parallel 2000 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1319/1319 [00:51<00:00, 25.58it/s] Accuracy: 0.948 Invalid: 0.000 Latency: 52.566 s Output throughput: 2409.507 token/sChecklist