Skip to content

[Fix] trtllm_mha: Use a separate workspace for prefill fmha_v2 backend - #42746

Closed
elvischenv wants to merge 6 commits into
sgl-project:mainfrom
elvischenv:fix-flashinfer-trtllm-xqa-workspace
Closed

elvischenv wants to merge 6 commits into
sgl-project:mainfrom
elvischenv:fix-flashinfer-trtllm-xqa-workspace

Conversation

@elvischenv

@elvischenv elvischenv commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

On SM90/SM120, trtllm_mha runs prefill with trtllm_fmha_v2_prefill and decode with XQA, and both used the same trtllm_mha_zero_workspace. fmha_v2 leaves its 4-byte tile counter at offset 0 of its workspace; XQA keeps its multi-block semaphores in the first 8 MB of the same buffer and needs them to start at zero. After any prefill, long-KV (multi-block) XQA decodes merge partial results at the wrong time and return wrong output for a batch slot. XQA does not clear a dirty semaphore, so the corruption persists.

This silently drops accuracy with --attention-backend trtllm_mha on H100 (MTP off). The split config (--prefill-attention-backend flashinfer --decode-attention-backend trtllm_mha) is not affected.

Modifications

  • Give fmha_v2 prefill its own small workspace (trtllm_mha_fmha_v2_workspace, 16 MB, SM90/SM120 only); XQA decode keeps the existing buffer.
  • Add test_fmha_v2_extend_then_xqa_long_kv_decode (SM90/SM120): a long-KV XQA decode must return bit-identical output before and after an fmha_v2 extend. It fails without the fix on H100.

Accuracy Tests

GSM8K (lm_eval, 5-shot, flexible-extract), Nemotron-H hybrid model, BF16, TP4, H100, FP8 KV, --attention-backend trtllm_mha, MTP off:

GSM8K
MTP0, no fix (3 runs) 0.8825 / 0.8863 / 0.8779
MTP0, with fix 0.9166

The no-fix runs used an earlier nightly with an older memory config; for reference, FlashInfer prefill + XQA decode on the with-fix setup scores 0.9196.

Speed Tests and Profiling

N/A. Prefill and decode kernels are unchanged.

Checklist

🤖 Generated with Claude Code


CI States

Latest PR Test (Base): ❌ Run #38701364229
Latest PR Test (Extra): ❌ Run #38701364163
Latest PR Test (AMD ROCm 10): ❌ Run #38701364328

@github-actions github-actions Bot added the blackwell SM100/SM120 label Oct 6, 2026
@elvischenv
elvischenv marked this pull request as ready for review October 6, 2026 09:56
@elvischenv

Copy link
Copy Markdown
Contributor Author

/tag-and-rerun-ci

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Oct 6, 2026
@elvischenv
elvischenv force-pushed the fix-flashinfer-trtllm-xqa-workspace branch from 138ab7c to cc23631 Compare October 7, 2026 03:13

@nvpohanh nvpohanh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[by Claude Code] Severity: style | Confidence: High

Issue

The Accuracy Tests section does not give enough detail to rerun the GSM8K numbers. It says "Nemotron-H hybrid model" but does not name the exact checkpoint. It also does not include the server launch command or the lm_eval command.

Fix

Add the model path (for example, the Hugging Face repo id), the full python -m sglang.launch_server ... command used for the --attention-backend trtllm_mha runs, and the lm_eval command with its task and few-shot settings.

@nvpohanh

nvpohanh commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator

/rerun-failed-ci

@nvpohanh

nvpohanh commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator

[by Claude Code] The relevant CI failure (https://github.com/sgl-project/sglang/actions/runs/37565716734/job/112751338170) is not caused by this PR. It is a known breakage on main, tracked in #42961. Proposed fix: #42962. CI for this PR is blocked until that is fixed.

@nvpohanh

nvpohanh commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator

/rerun-failed-ci

@nvpohanh nvpohanh added the bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) label Oct 7, 2026
@nvpohanh

nvpohanh commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator

/rerun-failed-ci

1 similar comment
@nvpohanh

nvpohanh commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator

/rerun-failed-ci

@nvpohanh

nvpohanh commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator

[by Claude Code] All NV pipelines have passed.

@nvpohanh

nvpohanh commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

@Fridge003 Could you help to review this? Thanks!

@ischencheng ischencheng left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Q: does this also close #40921? same setup (trtllm_mha for both prefill and decode on SM90, gpt-oss on H200), maybe worth linking it.

@Jiminator Jiminator left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TY for the contribution! suggested one small fix

@elvischenv
elvischenv force-pushed the fix-flashinfer-trtllm-xqa-workspace branch 2 times, most recently from ebe252d to 4873f9e Compare October 13, 2026 06:21
elvischenv and others added 2 commits October 13, 2026 17:18
On SM90/SM120, fmha_v2 leaves its tile counter at the head of the
workspace it shared with XQA decode, where XQA keeps semaphores that must
start at zero, corrupting later multi-block (long-KV) decodes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Checks the leftover directly so the test fails deterministically without
the fix instead of depending on XQA CTA timing or a dirty retry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@elvischenv
elvischenv force-pushed the fix-flashinfer-trtllm-xqa-workspace branch from 4873f9e to 5dfa863 Compare October 13, 2026 09:18
@nvpohanh

Copy link
Copy Markdown
Collaborator

[by Claude Code] I've started babysitting the NVIDIA CI of this PR. I'll check it every 30 minutes until all NV pipelines pass. If a relevant job fails, I may rerun it, merge the latest main into this branch, push a small test-only fix, or post a note here.

Anyone can stop me at any time: just post a comment asking me to stop (for example, stop babysitting). I'll stop at my next check, within 30 minutes, and take no further action on this PR.

nvpohanh and others added 3 commits October 13, 2026 21:21
Picks up MIN_BASE_SHA c053fc7 required by the CI maintenance gate (sgl-project#21065), which failed https://github.com/sgl-project/sglang/actions/runs/38375425126/job/115387274540

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Picks up the updated MIN_BASE_SHA ff8d964 required by the CI maintenance gate (sgl-project#21065), which failed https://github.com/sgl-project/sglang/actions/runs/38511334600/job/115690940038

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nvpohanh

Copy link
Copy Markdown
Collaborator

/rerun-failed-ci

Picks up MIN_BASE_SHA b677851 (sgl-project#21065 rebase-required gate), which blocked the unrelated CI failure https://github.com/sgl-project/sglang/actions/runs/38556647883/job/116305681373

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nvpohanh

Copy link
Copy Markdown
Collaborator

/rerun-failed-ci

1 similar comment
@nvpohanh

Copy link
Copy Markdown
Collaborator

/rerun-failed-ci

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@Jiminator
Jiminator force-pushed the fix-flashinfer-trtllm-xqa-workspace branch from aa70d55 to 5456503 Compare October 17, 2026 08:16
@github-actions

This comment has been minimized.

@elvischenv elvischenv changed the title Use a separate workspace for fmha_v2 prefill in trtllm_mha [Fix] trtllm_mha: Use a separate workspace for prefill fmha_v2 backend Oct 17, 2026
@Jiminator

Jiminator commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

Superseded in #42930

@Jiminator Jiminator closed this Oct 17, 2026
@elvischenv
elvischenv deleted the fix-flashinfer-trtllm-xqa-workspace branch October 17, 2026 09:30
@elvischenv
elvischenv restored the fix-flashinfer-trtllm-xqa-workspace branch October 17, 2026 09:55
Jiminator pushed a commit that referenced this pull request Oct 18, 2026
main keeps fmha_v2 prefill out of XQA's workspace semaphores through #42930,
which superseded #42746. Restore the test file and its CI estimate to main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

blackwell SM100/SM120 bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants