Repository navigation
Conversation
ckvermaAI
requested review from
AgainstEntropy,
BBuf,
HaiShaw,
kevin-mii,
mickqian,
niehen6174,
ping1jing2 and
yichiche
as code owners
October 12, 2026 02:52
siju-samuel
suggested changes
Oct 12, 2026
ckvermaAI
force-pushed
the
infer-batching
branch
2 times, most recently
from
October 12, 2026 15:55
9996c72 to
df0519f
Compare
ckvermaAI
force-pushed
the
infer-batching
branch
from
October 16, 2026 03:34
df0519f to
4630daf
Compare
Contributor
|
/tag-run-ci-label |
Collaborator
|
@ckvermaAI this is also a bug on CUDA? Do you have prove? |
… cache release Bug 1: _split_batched_output() required output or output_file_paths to be non-None, but only the replica's output rank (worker.is_output_rank) materializes either field; every other TP/SP rank legitimately has both as None after a successful forward. Every successful TP>1 dynamic batch therefore logged "Failed to split dynamic batched output cleanly" once per non-output rank, although all requests completed and the reply (sent only from the output rank) was correct. Gate the None/None check on worker.is_output_rank. Scheduler test stubs built with object.__new__(Scheduler) now set a worker stub, and a new case covers the is_output_rank=False split. Bug 2: after a failed forward, empty_cache() ran inside the except clause, where the bound exception's traceback still holds the failed forward's frames and their tensors. Those blocks were therefore never returned, and the device kept them reserved after every failed request. Run gc.collect() + empty_cache() after the try statement instead, once the except clause has released the frames. Measured on Intel Arc Pro B60 (XPU) by driving GPUWorker._execute_forward_common with a forward that allocates 4 x 512 MB in nested frames and then raises OutOfMemoryError, 3 runs: | Build | allocated | reserved above baseline | |--------------------------------|-----------|-------------------------| | base | 0 MB | 2048 MB | | gc.collect() inside except | 0 MB | 2048 MB | | gc + empty_cache after try | 0 MB | 0 MB | Both fixes are device-agnostic. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ckvermaAI
force-pushed
the
infer-batching
branch
from
October 20, 2026 13:48
e4660ec to
df72b9a
Compare
Contributor
Author
@mingfeima Yes, Issue reproduces on CUDA (updated the description with relevant details) |
Contributor
Author
|
@mickqian Please review this PR |
This was referenced Nov 9, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two bugs in diffusion dynamic batching (
multimodal_gen), reproduced on CUDA (2x A100) and XPU (Arc Pro B60):_split_batched_output()expectsoutputoroutput_file_paths, but only the output rank sets them. Every successful dynamic batch on a multi-GPU replica therefore logsFailed to split dynamic batched output cleanlyon each non-output rank, even though all requests succeed.empty_cache()runs inside theexceptclause, while the traceback still holds the failed forward's tensors. Their memory stays reserved (2 GiB in the repro below) after the error.Modifications
scheduler.py: run the "no outputs" check in_split_batched_output()only on the output rank.gpu_worker.py: movegc.collect()+empty_cache()after thetry/except/finally, once the exception is out of scope.workerstub to the existingSchedulertest stubs, and addtest_dynamic_batch_split_succeeds_on_non_output_rank.Accuracy Tests
Not applicable: no change to model math or outputs.
Speed Tests and Profiling
2x A100-40GB
--num-gpus 2 --ulysses-degree 2 --batching-max-size 2, 2 concurrent requestsFailed to split...errorThe XPU (Arc Pro B60) result for Bug 2 is the same: 2048 MB on
main, 0 MB with this PR. Unit tests:test_qwen_image21.py,test_sensenova_u1.py(scheduler subset) pass.Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ✅ Run #39717419152
Latest PR Test (Extra): ❌ Run #39717419087
Latest PR Test (AMD ROCm 10): ❌ Run #39717419280