Skip to content

fix gathered_buffer is None after tbo filter - #7638

Closed
lbh2001 wants to merge 2 commits into
sgl-project:mainfrom
lbh2001:fix_tbo_filter
Closed

lbh2001 wants to merge 2 commits into
sgl-project:mainfrom
lbh2001:fix_tbo_filter

Conversation

@lbh2001

@lbh2001 lbh2001 commented Jun 29, 2025

Copy link
Copy Markdown
Contributor

Motivation

fix #7511, which was fixed by #7531 but not worked for me

Modifications

Checklist

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary of Changes

Hello @lbh2001, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request aims to resolve a reported bug where the gathered_buffer could unexpectedly become None following a two-batch overlap filter. The change modifies a boolean condition to ensure the correct code path is executed, thereby preventing the None state and addressing the underlying issue that caused the problem for the author.

Highlights

  • Bug Fix: Corrected a conditional logic error within the filter_batch function in two_batch_overlap.py to properly handle scenarios where batch.gathered_buffer might be None after a filter operation, especially when global_server_args_dict["moe_dense_tp_size"] is 1.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in issue comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

The pull request modifies a conditional from and to or to address an issue with gathered_buffer becoming None. However, this change introduces a critical bug that can cause a crash due to a potential AttributeError. The provided code suggestion includes a null check to avoid this error.

Comment on lines 511 to +512
global_server_args_dict["moe_dense_tp_size"] == 1
and batch.gathered_buffer is not None
or batch.gathered_buffer is not None

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

Changing and to or can lead to an AttributeError if batch.gathered_buffer is None. Accessing batch.gathered_buffer.shape[1] in this case will cause a crash.

if global_server_args_dict["moe_dense_tp_size"] == 1 or batch.gathered_buffer is not None:
    sum_len = end_token_index - start_token_index
    if batch.gathered_buffer is not None:
        hidden_size = batch.gathered_buffer.shape[1]
        dtype = batch.gathered_buffer.dtype
        device = batch.gathered_buffer.device
    else:
        hidden_size = batch.token_to_kv_pool.hidden_size
        dtype = batch.token_to_kv_pool.dtype
        device = batch.input_ids.device

    gathered_buffer = torch.zeros(
        (sum_len, hidden_size),
        dtype=dtype,
        device=device,
    )
else:
    gathered_buffer = None

@Qiaolin-Yu

Copy link
Copy Markdown
Collaborator

Hi @lbh2001 , could you share the launching command you use?

@Qiaolin-Yu Qiaolin-Yu self-assigned this Jun 29, 2025
@lbh2001

lbh2001 commented Jun 30, 2025

Copy link
Copy Markdown
Contributor Author

Hi @lbh2001 , could you share the launching command you use?

Hi @Qiaolin-Yu My environment and launch script was record in #7602. Before I change the and to or, I would still face the same bug to the #7511, and I printed the forward_batch after filter_batch function, the gathered_buffer in child_a and child_b were both None . Maybe it was because my moe_dense_tp_size was None.

@Qiaolin-Yu

Qiaolin-Yu commented Jun 30, 2025 •

Copy link
Copy Markdown
Collaborator

moe_dense_tp_size

Hi @lbh2001 , I think currently TBO assumes you have the config of moe_dense_tp_size=1, enable-dp-lm-head, and have opened dp-attention. btw, could you double check if your error is exactly same as AttributeError: 'NoneType' object has no attribute 'shape' in #7511 ? Because imo, if the batch.gathered_buffer is None, it will not go enter the code crashed in #7511.

@lbh2001

lbh2001 commented Jun 30, 2025

Copy link
Copy Markdown
Contributor Author

Hi @lbh2001 , I think currently TBO assumes you have the config of moe_dense_tp_size=1, enable-dp-lm-head, and have opened dp-attention. btw, could you double check if your error is exactly same as AttributeError: 'NoneType' object has no attribute 'shape' in #7511 ? Because imo, if the batch.gathered_buffer is None, it will not go enter the code crashed in #7511.

@Qiaolin-Yu Thanks for your reply.
Sorry, it's not the same bug. My bug occured here, the forward_batch.gathered_buffer was None so it crashed.

def _scattered_to_tp_attn_full(
      hidden_states: torch.Tensor,
      forward_batch: ForwardBatch,
      context: CommunicateContext,
  ) -> torch.Tensor:
      hidden_states, local_hidden_states = (
          forward_batch.gathered_buffer[: forward_batch.input_ids.shape[0]],
          hidden_states,
      )
      attn_tp_all_gather(
          list(hidden_states.tensor_split(context.attn_tp_size)),
          local_hidden_states,
      )
      return hidden_states

In fact, the reason was moe_dense_tp_size was None? I have opened dp-attention, should I explicitly set it to 1 in my config?

@Qiaolin-Yu

Qiaolin-Yu commented Jun 30, 2025 •

Copy link
Copy Markdown
Collaborator

Hi @lbh2001 , I think currently TBO assumes you have the config of moe_dense_tp_size=1, enable-dp-lm-head, and have opened dp-attention. btw, could you double check if your error is exactly same as AttributeError: 'NoneType' object has no attribute 'shape' in #7511 ? Because imo, if the batch.gathered_buffer is None, it will not go enter the code crashed in #7511.

@Qiaolin-Yu Thanks for your reply. Sorry, it's not the same bug. My bug occured here, the forward_batch.gathered_buffer was None so it crashed.

def _scattered_to_tp_attn_full(
      hidden_states: torch.Tensor,
      forward_batch: ForwardBatch,
      context: CommunicateContext,
  ) -> torch.Tensor:
      hidden_states, local_hidden_states = (
          forward_batch.gathered_buffer[: forward_batch.input_ids.shape[0]],
          hidden_states,
      )
      attn_tp_all_gather(
          list(hidden_states.tensor_split(context.attn_tp_size)),
          local_hidden_states,
      )
      return hidden_states

In fact, the reason was moe_dense_tp_size was None? I have opened dp-attention, should I explicitly set it to 1 in my config?

@lbh2001 So your issue seems not related to #7511? And yes, try to pass --moe-dense-tp-size=1 and --enable-dp-lm-head.

@lbh2001

lbh2001 commented Jun 30, 2025

Copy link
Copy Markdown
Contributor Author

@lbh2001 So your issue seems not related to #7511? And yes, try to pass --moe-dense-tp-size=1 and --enable-dp-lm-head.
@Qiaolin-Yu Yes I tried set moe-dense-tp-size=1 just now and it worked, thanks! BTW if I have opened dp-attention but not set moe-dense-tp-size to 1 which is None default, the bug above occurred, is this phenomenon abnormal?

@Qiaolin-Yu

Qiaolin-Yu commented Jun 30, 2025 •

Copy link
Copy Markdown
Collaborator

@lbh2001 So your issue seems not related to #7511? And yes, try to pass --moe-dense-tp-size=1 and --enable-dp-lm-head.
@Qiaolin-Yu Yes I tried set moe-dense-tp-size=1 just now and it worked, thanks! BTW if I have opened dp-attention but not set moe-dense-tp-size to 1 which is None default, the bug above occurred, is this phenomenon abnormal?

Hi @lbh2001 , I think it's normal, since TBO assumes you have set moe-dense-tp-size=1.

@Qiaolin-Yu Qiaolin-Yu closed this Jun 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] TBO crash when turn on cuda-graph in PD disaggregation

3 participants