Skip to content

Fix FP8 MLA KV scale propagation - #34765

Closed
deepankerk wants to merge 2 commits into
sgl-project:mainfrom
deepankerk:agent/fix-mla-kv-scale-contract
Closed

deepankerk wants to merge 2 commits into
sgl-project:mainfrom
deepankerk:agent/fix-mla-kv-scale-contract

Conversation

@deepankerk

@deepankerk deepankerk commented Aug 13, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Calibrated FP8 MLA KV cache values must use the same scale convention when
they are written and read. Some MLA and DSA paths wrote the cache with unit
scale, restored only the K scale in BMM1, and omitted the V scale in BMM2.
That makes non-unit calibration inconsistent across the attention operation.

For the existing SGLang scale convention, k_scale and v_scale are
descales. Cache writes therefore multiply by the reciprocal descale, BMM1
applies the K descale with the attention softmax scale, and BMM2 applies the V
descale.

MLA stores one compressed latent row that is consumed as K in BMM1 and V in
BMM2. The K and V descales for that shared row must therefore agree.

Modifications

  • Propagate a non-unit KV quantization multiplier through the fused MLA cache
    writer.
  • Apply reciprocal descales in the fused-RoPE and no-RoPE cache-write paths.
  • Pass K descales to BMM1 and V descales to BMM2 for TRT-LLM MLA and DSA.
  • Cover regular prefill, decode, target verification, and draft extension.
  • Reject missing, non-finite, non-positive, or inconsistent shared MLA
    descales instead of silently using an incoherent scale contract.
  • Reject calibrated non-unit scales on fallback paths that do not expose the
    required read scales.

This uses the existing RadixAttention.k_scale, v_scale,
k_scale_float, and v_scale_float attributes. It does not introduce a new
model-specific scale format.

Accuracy Tests

The included synthetic tests verify:

  • invalid or inconsistent shared MLA scales fail closed;
  • the no-RoPE cache path multiplies K and rotary K by the reciprocal descale;
  • the fused RoPE path receives the reciprocal descale;
  • TRT-LLM MLA decode receives the expected non-unit BMM1 and BMM2 scales; and
  • the fused GPU writer scales cached K while retaining unit-scaled query
    quantization.

Speed Tests and Profiling

This is a scale-correctness change. No performance claim is made in this PR.

Validation

  • Python syntax compilation for the changed Python files
  • registered-test registry validation
  • package registered-test placement validation
  • whitespace validation
  • repository Clang formatting validation for the changed CUDA header

The new synthetic tests are registered for the relevant upstream CI suites.

Checklist

  • Add focused unit tests.
  • No user-facing documentation change is required.
  • Follow the existing SGLang KV-scale attributes and backend structure.

CI States

Latest PR Test (Base): ❌ Run #31745650167
Latest PR Test (Extra): ❌ Run #31745649743

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant