[czhu-cohere]

Purpose

From profiling trace it seemed like DeepEP HT backend wasn't fully overlapping the compute and communication.

The only thing being overlapped is one of the ubatch combine with the other ubatch non-moe compute.

Going through the DBO code it seems that the core issue is the previous_event that is passed to DeepEP dispatch/combine. The way it's implemented, DeepEP will wait for the entire compute stream to finish before launching dispatch. Since we have yielded via dbo_yield_and_switch_from_compute_to_comm() right before this point, the compute stream will contain compute enqueued by the other ubatch that the current ubatch doesn't need to depend on. The fix is to move the DeepEP event capture right before yield so it correctly captures the compute tail of the current ubatch.

After, the same model trace:

so now there is more overlap but the core dependency is still preserved.

Test Plan

test Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 on 4xB200 DEP4

• prefill throughput measurement
• lm-eval result

Test Result

prefill throughput test

server command

VLLM_MOE_ROUTING_SIMULATION_STRATEGY=uniform_random VLLM_RANDOMIZE_DP_DUMMY_INPUTS=1 vllm serve Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 --tensor-parallel-size 1 --data-parallel-size 4 --enable-expert-parallel --max-model-len 32768 --max-num-seqs 256 --gpu-memory-utilization 0.9 --load-format dummy --no-enable-prefix-caching --all2all-backend deepep_high_throughput --api-server-count 1

bench command

vllm bench serve \
  --backend openai \
  --base-url http://localhost:8000 \
  --endpoint /v1/completions \
  --model Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 \
  --tokenizer Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 \
  --dataset-name random \
  --input-len 8192 \
  --output-len 1 \
  --max-concurrency 64 \
  --num-prompts 640 \
  --num-warmups 128 \
  --ignore-eos 2>&1

before result

============ Serving Benchmark Result ============
Successful requests:                     640       
Failed requests:                         0         
Maximum request concurrency:             64        
Benchmark duration (s):                  53.22     
Total input tokens:                      5242880   
Total generated tokens:                  640       
Request throughput (req/s):              12.03     
Output token throughput (tok/s):         12.03     
Peak output token throughput (tok/s):    16.00     
Peak concurrent requests:                80.00     
Total token throughput (tok/s):          98524.82  
---------------Time to First Token----------------
Mean TTFT (ms):                          5076.72   
Median TTFT (ms):                        5197.57   
P99 TTFT (ms):                           6087.35   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          0.00      
Median TPOT (ms):                        0.00      
P99 TPOT (ms):                           0.00      
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00      
Median ITL (ms):                         0.00      
P99 ITL (ms):                            0.00      
==================================================

after result

============ Serving Benchmark Result ============
Successful requests:                     640       
Failed requests:                         0         
Maximum request concurrency:             64        
Benchmark duration (s):                  40.33     
Total input tokens:                      5242880   
Total generated tokens:                  640       
Request throughput (req/s):              15.87     
Output token throughput (tok/s):         15.87     
Peak output token throughput (tok/s):    20.00     
Peak concurrent requests:                84.00     
Total token throughput (tok/s):          130017.79 
---------------Time to First Token----------------
Mean TTFT (ms):                          3851.45   
Median TTFT (ms):                        3841.10   
P99 TTFT (ms):                           5501.03   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          0.00      
Median TPOT (ms):                        0.00      
P99 TPOT (ms):                           0.00      
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00      
Median ITL (ms):                         0.00      
P99 ITL (ms):                            0.00      
==================================================

about 30% throughput increase

lm-eval correctness test

server command

vllm serve Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 --tensor-parallel-size 1 --data-parallel-size 4 --enable-expert-parallel --max-model-len 32768 --max-num-seqs 256 --gpu-memory-utilization 0.9 --all2all-backend deepep_high_throughput --api-server-count 1

lm-eval test command

python -m lm_eval run --model local-completions --model_args "model=Qwen/Qwen3-30B-A3B-Instruct-2507-FP8,base_url=http://localhost:8000/v1/completions,tokenizer_backend=huggingface,num_concurrent=64" --tasks gsm8k

before

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.8870|±  |0.0087|
|     |       |strict-match    |     5|exact_match|↑  |0.8772|±  |0.0090|

after

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.8878|±  |0.0087|
|     |       |strict-match    |     5|exact_match|↑  |0.8779|±  |0.0090|

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request modifies vllm/model_executor/layers/fused_moe/prepare_finalize/deepep_ht.py to ensure that DeepEP events are captured on the compute stream before yielding and switching to communication. This change prevents over-serialization by ensuring the event only covers the current micro-batch's compute work. I have no feedback to provide as there were no review comments to evaluate.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[robertgshaw2-redhat]

nice job, thanks for your contribution!

[SageMoore]

Nice find! This looks correct to me. Thanks for the contribution! CC @LucasWilkinson

[LucasWilkinson]

Thank for the fix! Overall LGTM, would be good to have @yewentao256 take a look (since dbo_get_previous_event was introduced in #27666 ; the dependency preservation seems to be correct to me)

[yewentao256]

Nice catch! Thanks for the work!

[czhu-cohere]

Thanks @yewentao256 ! I think the CI failure is unrelated, is this OK to merge?

[yewentao256]

Thanks @yewentao256 ! I think the CI failure is unrelated, is this OK to merge?

Let's try again before doing a force merge