Repository navigation
Conversation
|
The PR is ready for review, but the CI gate is currently blocked because the |
…e grammar mask, issue sgl-project#43514) Byte-exact gh pr diff of sgl-project#43518 (Hahaa909, +161/-5, dflash_utils.py + its unit test). Applied in-image by docker/kubetee-mimo-overlay/Dockerfile with pre/post sha256 tripwires; the PR's own unit test is the build smoke check. Preimage dflash_utils.py @ ed8c313 = 5f8f266964e4dd15b426d197e944c13d7a75c4d1749187a9f09506bc8cfb61a6 Postimage = 81b58d09fb65c51d3bd3b73bc17757b8851269ff529f8a4cc2bf8b0818f437e4 test file = 5664b9282b4d95137169487ef25f476d156210659606ca1125b074d00a197fa3 Retire this patch when sgl-project#43518 merges upstream.
…ject#43518) Layers upstream PR sgl-project#43518 (Hahaa909 — dflash verify stale grammar mask, issue sgl-project#43514) onto the digest-pinned nightly-dev-cu13 base WITHOUT disturbing the base source otherwise (unlike docker/kubetee-overlay's full tree swap). Safety: pre/post sha256 drift tripwires on dflash_utils.py (preimage 5f8f2669…, postimage 81b58d09…) + the PR's own unit test runs as the build smoke check against the patched tree. Target: XiaomiMiMo/MiMo-V2.6-Pro-RL on B200 CC. Retire when sgl-project#43518 merges upstream — see the SOURCE POLICY CARVE-OUT in the header.
Additive inputs (defaults unchanged — existing dsv41 runs unaffected): variant=dsv41|mimo (default dsv41) source_sha [mimo] upstream SHA the base image was built from tag_prefix exact primary tag override variant=mimo builds docker/kubetee-mimo-overlay/Dockerfile — the base image's own source tree with upstream PR sgl-project#43518's patch applied in place (dflash verify stale grammar mask, issue sgl-project#43514; target XiaomiMiMo/MiMo-V2.6-Pro-RL on B200 CC). Tags mimo-cu13-<shortsha>-<date> + mimo-cu13-latest. See the mimo Dockerfile header for the SOURCE POLICY CARVE-OUT and the retire-on-merge rule.
…tments apply_dflash_verify_logits_adjustments runs on the [bs * draft_token_num, vocab] verify logits, and it read sampling_info.grammar_mask -- which at that point belongs to the previous extend/decode step and has one row per REQUEST. It fed the dense fallback, which applies the mask to a zero buffer and broadcasts the result over the whole verify block. Two ways that goes wrong, both now pinned by a regression: * the transition out of grammar (a request's FSM finished, or the batch was filtered down) leaves no new mask and a stale row behind. An all-masked row turned every verify logit -inf; a mask with the pre-filter row count raised "size of tensor a (2) must match the size of tensor b (3)". * even a same-sized live mask is the wrong state. Each verify-tree position has its own grammar state, and the verify callers already build and apply that per-position mask (build_grammar_vocab_mask in dflash_worker_v2 and dspark_worker_v2, applied after these adjustments). Broadcasting a per-request mask here masks a position by its predecessor's state. Clear sampling_info.grammar_mask up front and drop it as a fallback trigger, so per-position verification stays owned by the callers. build_grammar_vocab_mask already cleared the leftover when it produced a fresh mask; this closes the remaining path where it produces none. Everything the fallback existed for is preserved: the custom logit processor still runs first on the unadjusted logits, a live penalizer still takes the dense path with the same ordering (penalizer, then logit_bias -- which is why the expectations are not just elementwise sums), and precomputed acc_additive_penalties and logit_bias still broadcast over the block. DSpark inherits this through apply_logits_adjustments_strided and its own callers; its verify_logits_adjustments_are_noop still counts a lingering mask as work, which is now merely conservative (the call clears it) rather than required. Adapted to this tree's acc_additive_penalties spelling of the overlap-penalty buffer (upstream renamed it acc_linear_penalties). CPU-only: red on the exact base (6 cases -- poisoned logits, the filtered-row RuntimeError, and a per-position argmax of all -inf), green after. The mixed grammar/non-grammar long-prefill overlap workload is unverified here, no GPU. Source: sgl-project/sglang#43518 (cherry picked from commit 74684d99c58fd04911089e37c87f900d6843302d) Co-authored-by: jianghh <3232826884@qq.com>
|
@hnyls2002 (CODEOWNER for Reproducing the symptom in production (SGLang main, 8×B200,
Happy to run the DFLASH + tools battery on our hardware and post numbers if that helps review. |
0984717 to
1cbbf72
Compare
Motivation
Related to #43514. A
SamplingBatchInfocan still carry the previous extend/decode-stagegrammar_maskwhen entering speculative verification. The shared DFlash/DSpark logits-adjustment helper currently applies that per-request mask to a zero-valued penalty buffer and broadcasts the result across the entire verify block. An all-masked stale row makes every verify logit-inf; a stale mask with a different batch size can also cause a shape mismatch.Verify callers already apply a separate grammar mask for each current tree position. Reusing the previous per-request mask here is incorrect even if the batch sizes happen to match.
This is a draft pending normal Linux-runtime execution of the new tests and reproduction of the issue's mixed grammar/non-grammar, long-prefill overlap workload on an accelerator. The function-level regression does not establish that every possible cause of the reported production failure is fixed.
Modifications
sampling_info.grammar_maskinapply_dflash_verify_logits_adjustmentsafter validating the verify-logit shape, before applying any sampling adjustments.CustomTestCase, realSamplingBatchInfo/GrammarMaskimplementations, and CPU tensors. Only the external grammar backend and live penalizer are lightweight test doubles.Relationship to existing work
sampling_info.grammar_mask. This patch does not attempt to fix the separate accumulated-penalty field-name or scaling-penalty issues. The stale-mask clear should be preserved if those changes are integrated first.Accuracy Tests
Added
test/registered/unit/spec/test_dflash_verify_grammar_mask.pywith five test methods covering:acc_linear_penaltiespath survives the stale mask, including float32-to-bfloat16 conversion.Local validation on Windows used an isolated, non-shipped runner that compiles the unchanged production function/class definitions from the checkout and executes the new tests with CPU PyTorch. It bypasses package initialization and replaces only the CI test-case scaffolding with
unittest.TestCase; it is not a normal SGLang suite run. All five test methods expose failures with the original helper and pass with this patch.Normal test collection was attempted but stops during SGLang package initialization because Windows lacks the Unix
resourcemodule. No GPU model/server accuracy test has been run.Please run in the normal Linux CI environment:
The issue's mixed structured-output and long-prefill speculative-decoding workload still needs accelerator validation.
Speed Tests and Profiling
Not benchmarked. The new reset is a host-side attribute assignment, and a stale mask no longer causes allocation/application of the dense penalty buffer. No throughput or latency claim is made.
Checklist
Review and Merge Process
Draft: normal-runtime regression execution and accelerator workload validation are pending before requesting merge.
CI States
Latest PR Test (Base): ❌ Run #40834947733
Latest PR Test (Extra): ❌ Run #40834947836
Latest PR Test (AMD ROCm 10): ❌ Run #40834947791