Repository navigation
[Fix] trtllm_mha: Use a separate workspace for prefill fmha_v2 backend - #42746
elvischenv wants to merge 6 commits into
Conversation
|
/tag-and-rerun-ci |
138ab7c to
cc23631
Compare
nvpohanh
left a comment
There was a problem hiding this comment.
[by Claude Code] Severity: style | Confidence: High
Issue
The Accuracy Tests section does not give enough detail to rerun the GSM8K numbers. It says "Nemotron-H hybrid model" but does not name the exact checkpoint. It also does not include the server launch command or the lm_eval command.
Fix
Add the model path (for example, the Hugging Face repo id), the full python -m sglang.launch_server ... command used for the --attention-backend trtllm_mha runs, and the lm_eval command with its task and few-shot settings.
|
/rerun-failed-ci |
|
[by Claude Code] The relevant CI failure (https://github.com/sgl-project/sglang/actions/runs/37565716734/job/112751338170) is not caused by this PR. It is a known breakage on |
|
/rerun-failed-ci |
|
/rerun-failed-ci |
1 similar comment
|
/rerun-failed-ci |
|
[by Claude Code] All NV pipelines have passed. |
|
@Fridge003 Could you help to review this? Thanks! |
ischencheng
left a comment
There was a problem hiding this comment.
Q: does this also close #40921? same setup (trtllm_mha for both prefill and decode on SM90, gpt-oss on H200), maybe worth linking it.
Jiminator
left a comment
There was a problem hiding this comment.
TY for the contribution! suggested one small fix
ebe252d to
4873f9e
Compare
On SM90/SM120, fmha_v2 leaves its tile counter at the head of the workspace it shared with XQA decode, where XQA keeps semaphores that must start at zero, corrupting later multi-block (long-KV) decodes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Checks the leftover directly so the test fails deterministically without the fix instead of depending on XQA CTA timing or a dirty retry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4873f9e to
5dfa863
Compare
|
[by Claude Code] I've started babysitting the NVIDIA CI of this PR. I'll check it every 30 minutes until all NV pipelines pass. If a relevant job fails, I may rerun it, merge the latest Anyone can stop me at any time: just post a comment asking me to stop (for example, |
Picks up MIN_BASE_SHA c053fc7 required by the CI maintenance gate (sgl-project#21065), which failed https://github.com/sgl-project/sglang/actions/runs/38375425126/job/115387274540 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Picks up the updated MIN_BASE_SHA ff8d964 required by the CI maintenance gate (sgl-project#21065), which failed https://github.com/sgl-project/sglang/actions/runs/38511334600/job/115690940038 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Picks up sgl-project#44235, which fixes the unrelated CI failure https://github.com/sgl-project/sglang/actions/runs/38519532853/job/115717557650 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
/rerun-failed-ci |
Picks up MIN_BASE_SHA b677851 (sgl-project#21065 rebase-required gate), which blocked the unrelated CI failure https://github.com/sgl-project/sglang/actions/runs/38556647883/job/116305681373 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
/rerun-failed-ci |
1 similar comment
|
/rerun-failed-ci |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
aa70d55 to
5456503
Compare
This comment has been minimized.
This comment has been minimized.
|
Superseded in #42930 |
Motivation
On SM90/SM120,
trtllm_mharuns prefill withtrtllm_fmha_v2_prefilland decode with XQA, and both used the sametrtllm_mha_zero_workspace. fmha_v2 leaves its 4-byte tile counter at offset 0 of its workspace; XQA keeps its multi-block semaphores in the first 8 MB of the same buffer and needs them to start at zero. After any prefill, long-KV (multi-block) XQA decodes merge partial results at the wrong time and return wrong output for a batch slot. XQA does not clear a dirty semaphore, so the corruption persists.This silently drops accuracy with
--attention-backend trtllm_mhaon H100 (MTP off). The split config (--prefill-attention-backend flashinfer --decode-attention-backend trtllm_mha) is not affected.Modifications
trtllm_mha_fmha_v2_workspace, 16 MB, SM90/SM120 only); XQA decode keeps the existing buffer.test_fmha_v2_extend_then_xqa_long_kv_decode(SM90/SM120): a long-KV XQA decode must return bit-identical output before and after an fmha_v2 extend. It fails without the fix on H100.Accuracy Tests
GSM8K (lm_eval, 5-shot, flexible-extract), Nemotron-H hybrid model, BF16, TP4, H100, FP8 KV,
--attention-backend trtllm_mha, MTP off:The no-fix runs used an earlier nightly with an older memory config; for reference, FlashInfer prefill + XQA decode on the with-fix setup scores 0.9196.
Speed Tests and Profiling
N/A. Prefill and decode kernels are unchanged.
Checklist
🤖 Generated with Claude Code
CI States
Latest PR Test (Base): ❌ Run #38701364229
Latest PR Test (Extra): ❌ Run #38701364163
Latest PR Test (AMD ROCm 10): ❌ Run #38701364328