Skip to content

[Fix] multimodal_gen: Fix dynamic-batching bugs - non-output-rank split, OOM memory climb - #43750

Open
ckvermaAI wants to merge 4 commits into
sgl-project:mainfrom
ckvermaAI:infer-batching
Open

ckvermaAI wants to merge 4 commits into
sgl-project:mainfrom
ckvermaAI:infer-batching

Conversation

@ckvermaAI

@ckvermaAI ckvermaAI commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Two bugs in diffusion dynamic batching (multimodal_gen), reproduced on CUDA (2x A100) and XPU (Arc Pro B60):

  1. False split failure on non-output ranks. _split_batched_output() expects output or output_file_paths, but only the output rank sets them. Every successful dynamic batch on a multi-GPU replica therefore logs Failed to split dynamic batched output cleanly on each non-output rank, even though all requests succeed.
  2. Device cache not released after a failed forward. empty_cache() runs inside the except clause, while the traceback still holds the failed forward's tensors. Their memory stays reserved (2 GiB in the repro below) after the error.

Modifications

  • scheduler.py: run the "no outputs" check in _split_batched_output() only on the output rank.
  • gpu_worker.py: move gc.collect() + empty_cache() after the try/except/finally, once the exception is out of scope.
  • Tests: add a worker stub to the existing Scheduler test stubs, and add test_dynamic_batch_split_succeeds_on_non_output_rank.

Accuracy Tests

Not applicable: no change to model math or outputs.

Speed Tests and Profiling

2x A100-40GB

Bug Setup main this PR
1 Wan2.1-T2V-1.3B, --num-gpus 2 --ulysses-degree 2 --batching-max-size 2, 2 concurrent requests 1x Failed to split... error none
2 Failed forward (4 x 512 MB, then OOM), reserved memory after return 2048 MB 0 MB

The XPU (Arc Pro B60) result for Bug 2 is the same: 2048 MB on main, 0 MB with this PR. Unit tests: test_qwen_image21.py, test_sensenova_u1.py (scheduler subset) pass.

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ✅ Run #39717419152
Latest PR Test (Extra): ❌ Run #39717419087
Latest PR Test (AMD ROCm 10): ❌ Run #39717419280

@siju-samuel siju-samuel left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The output-rank check looks right, but it breaks a few existing scheduler tests, and I'm not sure the gc.collect() frees much while we're still inside the except block. Mind adding your before/after memory numbers to the description too?

@ckvermaAI
ckvermaAI force-pushed the infer-batching branch 2 times, most recently from 9996c72 to df0519f Compare October 12, 2026 15:55
@ckvermaAI ckvermaAI changed the title [diffusion] Fix dynamic-batching bugs: non-output-rank split, OOM memory climb [Fix] multimodal_gen: Fix dynamic-batching bugs - non-output-rank split, OOM memory climb Oct 16, 2026

@siju-samuel siju-samuel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/tag-run-ci-label

@siju-samuel

Copy link
Copy Markdown
Contributor

/tag-run-ci-label

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Oct 16, 2026

@mingfeima mingfeima left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix the regression.

@mingfeima

Copy link
Copy Markdown
Collaborator

@ckvermaAI this is also a bug on CUDA? Do you have prove?

… cache release

Bug 1: _split_batched_output() required output or output_file_paths to
be non-None, but only the replica's output rank (worker.is_output_rank)
materializes either field; every other TP/SP rank legitimately has both
as None after a successful forward. Every successful TP>1 dynamic batch
therefore logged "Failed to split dynamic batched output cleanly" once
per non-output rank, although all requests completed and the reply
(sent only from the output rank) was correct. Gate the None/None check
on worker.is_output_rank. Scheduler test stubs built with
object.__new__(Scheduler) now set a worker stub, and a new case covers
the is_output_rank=False split.

Bug 2: after a failed forward, empty_cache() ran inside the except
clause, where the bound exception's traceback still holds the failed
forward's frames and their tensors. Those blocks were therefore never
returned, and the device kept them reserved after every failed request.
Run gc.collect() + empty_cache() after the try statement instead, once
the except clause has released the frames.

Measured on Intel Arc Pro B60 (XPU) by driving
GPUWorker._execute_forward_common with a forward that allocates
4 x 512 MB in nested frames and then raises OutOfMemoryError, 3 runs:

  | Build                          | allocated | reserved above baseline |
  |--------------------------------|-----------|-------------------------|
  | base                           | 0 MB      | 2048 MB                 |
  | gc.collect() inside except     | 0 MB      | 2048 MB                 |
  | gc + empty_cache after try     | 0 MB      | 0 MB                    |

Both fixes are device-agnostic.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@ckvermaAI

ckvermaAI commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor Author

@ckvermaAI this is also a bug on CUDA? Do you have prove?

@mingfeima Yes, Issue reproduces on CUDA (updated the description with relevant details)

@ckvermaAI

Copy link
Copy Markdown
Contributor Author

@mickqian Please review this PR

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

diffusion SGLang Diffusion run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants