Skip to content

CUDA error: device-side assert triggered in self.forward_extend_multi_modal(batch) #528

Closed

Description

@LetheRiver0
  • model: llava-v1.6-34b
  • device: 2 H100

I have successfully started up backend for llava-v1.6-34b using similar code shown in https://github.com/sgl-project/sglang/blob/f6dbd24043b8c18d87a14b3c6fe5c4f567f6c1ba/examples/quick_start/srt_example_llava.py
Everything is ok when I run single or batch in one process. However, if I use parallel, like simultaneous run 2 batch function, There is a certain probability get error in backend, it doesn't appear every time, the full error message is shown below
image

I found some similar error in other issues, but it seems is different from my issue, dose anyone meet the same issue and know how to fix it? Thanks~

Activity

  1. LetheRiver0 commented on Jun 13, 2024

    @LetheRiver0
    Author

    Does anyone know how to handle this?

  2. github-actions commented on Aug 13, 2024

    @github-actions
    Contributor

    This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.

  3. added a commit that references this issue on Jan 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions