Skip to content

[Fix] dspark: Draw the simulated-acceptance bonus at the simulated row - #46404

Merged
kpham-sgl merged 1 commit into
sgl-project:mainfrom
weireweire:fix/dspark-sim-bonus-forced-row
Oct 27, 2026
Merged

kpham-sgl merged 1 commit into
sgl-project:mainfrom
weireweire:fix/dspark-sim-bonus-forced-row

Conversation

@weireweire

@weireweire weireweire commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

With SGLANG_SIMULATE_ACC_LEN set, DSpark replaces the accepted length with the simulated value c. The bonus token, however, stays the one the real accept computed for its own length: the target argmax at the real first-mismatch row, or the rejection-sampled token at the real reject row. That token is committed after drafts[:c], at a position the target never predicted it for. It is also the anchor of the next draft block. The committed text degenerates as a result, and verify batches route to fewer experts than at the same acceptance length under real verification. Simulated-acceptance throughput is therefore overstated.

The only configuration that was correct is SGLANG_SIMULATE_ACC_GREEDY=1 with temperature-only requests, which has been opt-in since #45433. EAGLE/MTP and DFlash already take the bonus from the target at the forced row.

Modifications

Under simulation, the bonus is the target's token at verify row c, drawn with the request's sampling law.

  • dspark_worker_v2.py: every sampled batch now passes its sampling info as simulate_bonus_sampling_info. Before, only SGLANG_SIMULATE_ACC_GREEDY=1 with temperature-only batches did. The flag now only selects the greedy draft/accept, under the same eligibility checks as before.
  • Sampling-accept region graph (SGLANG_OPT_DSV41_SAMPLING_ACCEPT_GRAPH): it stays eligible under simulation for the same batches as in real serving (all-sampled, temperature-only, no logit adjustments or grammar), so a simulated run keeps the graph that real serving uses.
    • Before the region runs, the worker stages the step's temperatures and seed with begin_sim_bonus.
    • Inside the region, the bonus is drawn at row c with the epilogue's graph-safe sampler (sim_bonus, renamed from _sim_bonus), the same kernel SGLANG_DSPARK_SIMULATE_FOLD uses.
    • The region's rejection accept still runs, as it does in real serving; under simulation only its length and bonus are replaced.
  • build_accept_outs: when no sampled bonus is supplied, the bonus is the argmax of row c. This covers greedy batches on the eager accept and on the folded greedy accept (DsparkVerifyEpilogue._accept without SGLANG_DSPARK_SIMULATE_FOLD).
  • sample_simulated_bonus: top-k/top-p batches sample row c from build_speculative_verify_target_probs, the target distribution the sampling accept uses. Greedy requests in a mixed batch have top_k=1, so they stay argmax. Temperature-only batches keep the existing Gumbel kernel.
  • static_accept_tail (Triton kernel and torch reference): the argmax-id bonus is read at the committed length rather than the matched length.
  • A sampled simulated bonus is broadcast on the DSPARK_ACCEPT_SAMPLE TP sync site, so the rng preset keeps ranks consistent.
  • test_dspark_simulated_bonus.py: the sampling-info stand-in gains need_top_k_sampling / need_top_p_sampling.

Scope of the sampling law under simulation:

  • Temperature and top-k/top-p are applied by the bonus sampler.
  • Logit adjustments (penalties, logit bias) and grammar masks are honoured only because they are already in the verify logits when the bonus is drawn: run_non_compact / run_compact apply the adjustments and the worker applies the grammar mask. The sampler itself does neither.
  • Batches with adjustments or grammar never use the region graph.
  • min-p is not applied. The non-simulated sampling accept does not apply it either.

Simulated acceptance is a benchmark-only mode.

TP: a sampled bonus is broadcast from rank 0 when the DSPARK_ACCEPT_SAMPLE site is enabled. Without it, ranks draw from identical seeds, the same assumption the simulated-length draw and the non-simulated rejection sampler already make.

Non-simulated paths are unchanged. Every new branch is behind simulated_correct_len is not None, and the tail kernel's non-simulated length is still num_correct.

Accuracy / correctness

Unit-level checks on GB300 run the real kernels, with drafts rejected at row 0 and simulated c = 3:

path before after
eager accept, greedy batch bonus = row-0 argmax row-3 argmax
folded greedy accept, eager call and inside a CUDA graph row-0 argmax row-3 argmax
fused static accept tail (Triton) row-0 argmax row-3 argmax
top-p bonus distribution (T=0.8, p=0.7), max abs diff vs. the renormalized target, 200k draws 0.21 0.0016
non-simulated fused tail unchanged unchanged
sampling-accept region under simulation, bs 1/1, 3/3, 3/4 (padded); eager warm-up, capture, replays at fixed seeds rejection bonus at the real row equals the eager row-c draw; drafts at [:c], bonus at [c]
region bonus distribution, 6,000 replays (T=0.8), max abs diff vs. softmax(row c / T) 0.65 0.0099

Existing tests, before and after: test_dspark_accept_tail.py, test_dspark_sampling_accept_graph.py and test_dspark_sim_draw_ahead.py pass. test_dspark_simulated_bonus.py::test_bf16_full_vocab_low_temperature fails identically with and without this change on NVIDIA. It is registered for AMD CI only.

End to end: DeepSeek-V4.1-Flash FP4, DSpark block 5, TP2 EP2, GB300, AgentX-style agentic traces at 32 concurrent, temperature 1, SGLANG_SIMULATE_ACC_LEN=3.51 (match-expected). The committed text is re-tokenized; about 1,500 turns are shared across runs.

real acceptance simulated, before simulated, after
token equal to one of the previous 6 0.070 0.375 0.145
4-gram already seen in the response 0.282 0.387 0.197
share of the most frequent token 0.042 0.170 0.083
distinct experts per MoE layer, B=4 / 5 / 8 / 10 75.0 / 84.9 / 116.2 / 123.9 53.9 / 62.7 / 85.0 / 94.9 71.8 / 85.3 / 108.3 / 121.9

After the fix, the remaining difference from real acceptance is in the drafted tokens. Simulation accepts them unconditionally, so repeats 2-3 tokens apart rise from 0.027 to 0.096 of tokens. Immediate repeats (0.0008 vs 0.0024) and repeats 4-6 tokens apart match real acceptance. An independent simulated-acceptance implementation that commits the target's token at the forced position measures the same statistics on this workload (0.148 / 0.182 / 0.087).

Benchmarking and Profiling

The size of the change depends on the point: it is largest where MoE is a big share of the step and the degenerate text cut the distinct experts the most.

TP2 EP2, 32 concurrent (the end-to-end setup above), 3600 s:

total tok/s/GPU output tok/s/GPU ITL p50 / p90 (ms) interactivity p50 / p90 (tok/s, 1000/ITL)
simulated, before 122.2K 697 3.43 / 4.40 291.5 / 227.3
simulated, after 118.9K 683 3.85 / 4.92 259.7 / 203.3

Both runs measure AL 3.51. The "after" run is 2.7% lower in total throughput and 11% lower in p90 interactivity (11% higher p90 ITL). This is expected: the "before" run's degenerate text made MoE about 1 ms/step cheaper at B=4-10. Each number is a single run, on different nodes. These two runs applied the same change to an earlier nightly and use the eager accept (no sampling-accept graph).

TP4 EP1, 1 concurrent, NVIDIA megakernel, SGLANG_OPT_DSV41_SAMPLING_ACCEPT_GRAPH=1, SGLANG_SIMULATE_ACC_LEN=3.51, same traces, 1800 s, AL 3.51 in every run. At batch 1 with the experts sharded across four GPUs, MoE is a small share of the step, so the change costs about 1%. This table also shows why the sampling-accept region change is needed:

region graph replays interactivity p50 / p90 (tok/s, 1000/ITL)
this PR ~110k per rank 976 / 949
this PR without the region change (sampled simulated steps fall back to the eager accept) 0 939 / 863
before this PR (region graph with the misplaced bonus), 3600 s captured 987 / 954

Keeping the region in the graph is worth 4% / 10% at p50 / p90. The remaining ~1% to "before" is within what the text change and the shorter run can explain.

Checklist

  • Format code with pre-commit.
  • Existing unit tests updated; no new test file (verified on GPU as above).

CI States

Latest PR Test (Base): ❌ Run #40179008917
Latest PR Test (Extra): ❌ Run #40179008912
Latest PR Test (AMD ROCm 10): ❌ Run #40179008872

Root cause: SGLANG_SIMULATE_ACC_LEN replaces only correct_len after the
accept. The bonus stays the one the accept computed for its own length:
the argmax (or the rejection-sampled token) at the real first-mismatch
row. It is committed after drafts[:c], at a position the target never
predicted it for, and anchors the next draft block, so the committed text
degenerates. Affected paths: the eager accept (every batch unless
SGLANG_SIMULATE_ACC_GREEDY=1 with a temperature-only batch), the folded
greedy accept and the fused static accept tail without
SGLANG_DSPARK_SIMULATE_FOLD, and the sampling-accept region graph.

Fix:
- Worker: under simulation every sampled batch passes its sampling info
  for the bonus; SGLANG_SIMULATE_ACC_GREEDY only switches draft/accept to
  greedy.
- build_accept_outs: without a sampled bonus, the bonus is the argmax of
  the simulated row.
- sample_simulated_bonus: top-k/top-p batches sample the simulated row
  from build_speculative_verify_target_probs, the sampling accept's target
  distribution; temperature-only batches keep the Gumbel kernel.
- Folded greedy accept and the fused tail read the simulated row.
- Sampling-accept region graph: stays eligible under simulation for the
  same batches as real serving; the worker stages temperatures and seed
  with begin_sim_bonus and the region draws the bonus at the simulated row
  with the epilogue's graph-safe sampler (sim_bonus, was _sim_bonus).
- A sampled simulated bonus uses the DSPARK_ACCEPT_SAMPLE TP sync site.

Validation: see PR body.
@akhilg-nv

Copy link
Copy Markdown
Contributor

[by Claude Code]

Thanks for this. It matches what we saw: under simulated acceptance our DSpark sanity replies were degenerate, while real-acceptance replies were coherent. Two small suggestions:

  1. A regression test for the bonus row. The test changes here update stubs and the graph-eligibility check, but nothing asserts which row the bonus comes from, so the original bug would still pass. For example: a toy verify batch whose rows have distinct argmaxes, with simulated correct_len = c set apart from the real first mismatch. Then check that the committed bonus is row c's argmax and not the mismatch row's. It should fail on main.
  2. One benchmark-only shortcut remains. When acceptance is simulated, accept_and_finalize skips the sampling-based accept step (since [Perf] [NVIDIA] DSpark: Enable the simulated-acceptance fast paths on CUDA #45421), because the simulated length overwrites its result anyway. Outputs don't change, but a real server does that step on every verify whenever requests sample (AgentX sends temperature: 1): softmax over draft and target logits plus the rejection test. So simulated runs skip work real serving pays for. Could that step's cost be measured, or kept running under simulation (skipped only with SGLANG_SIMULATE_ACC_GREEDY, like the draft/accept shortcut), so simulated numbers stay comparable to real serving?

@kpham-sgl kpham-sgl self-assigned this Oct 27, 2026

Copy link
Copy Markdown
Collaborator

/rerun-test test_dspark_accept_tail.py test_dspark_sampling_accept_graph.py test_dspark_sim_draw_ahead.py test_dspark_kernel_parity.py

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

🚀 1-gpu-5090 (3 tests): ✅ View workflow run

cd test/ && python3 registered/unit/spec/test_dspark_accept_tail.py
cd test/ && python3 registered/unit/speculative/test_dspark_sampling_accept_graph.py
cd test/ && python3 registered/spec/dspark/test_dspark_kernel_parity.py

🚀 ubuntu-latest (1 test): ✅ View workflow run

cd test/ && python3 registered/unit/spec/test_dspark_sim_draw_ahead.py

@kpham-sgl
kpham-sgl merged commit 24d5562 into sgl-project:main Oct 27, 2026
111 of 127 checks passed
weireweire added a commit to weireweire/sglang that referenced this pull request Oct 27, 2026
… pass

Root cause: sgl-project#41593 keeps waiting requests' device prefixes warm by
touching them in the radix LRU, but on a PD prefill worker it walks only
`waiting_queue`. Requests still in `disagg_prefill_bootstrap_queue`,
waiting for decode to pre-allocate their slots, are never touched. When
decode is full, the bootstrap queue is the deep one, and those prefixes
age out while the requests wait. With DEP2 prefill (9 workers) + DEP16
decode serving an agentic multi-turn trace at 7480 concurrent sessions,
the bootstrap queue held ~300 requests per rank for ~100 s. Device-radix
hits fell from ~65 % to ~22 % of prompt tokens, and the extra external
(Mooncake L3) reloads slowed prefill further, so prefill collapsed after
the client's turn burst and did not recover.

Fix: `calc_priority` takes the PD prefill bootstrap queue, and on the
same refresh pass `_touch_waiting_prefixes` touches up to
WAITING_PREFIX_REFRESH_BOOTSTRAP_MAX (512, a module constant next to the
existing waiting-queue bounds) bootstrap entries before the waiting
queue, in reverse order. SGLANG_ENABLE_WAITING_PREFIX_REFRESH=0 still
turns the whole refresh off. The
existing gate is unchanged: cache-agnostic policy, LRU eviction, not
decode, no fast match, prefix sharing supported.

Bootstrap entries are refreshed only on passes that reach
`calc_priority`. The caller returns earlier when the waiting queue is
empty or no request rows are free, so the bootstrap queue is not
refreshed on its own. With more than 128 waiting entries, the entries
past the first 128 are still not touched and can end up older than the
refreshed bootstrap entries.

Validation: GB300, DSv4.1-Flash, 9 DEP2 prefill + DEP16 decode, 7480
concurrent sessions, 1200 s window, SGLang main at 6563312 plus open
PRs sgl-project#46497/sgl-project#46593/sgl-project#46592/sgl-project#46404 on both arms:
  without: 361,815 tok/s/GPU @ p90 interactivity 145.0, TTFT p90 135.9 s;
           L3 73 % / device 22 % of prompt tokens in minutes 8-20.
  with:    560,412 tok/s/GPU @ p90 interactivity 161.0, TTFT p90 58.9 s;
           L3 21 % / device 74 %.
test_scheduler_held_row_admission's Scheduler stub gains
`disagg_prefill_bootstrap_queue = None` for the new argument.
weireweire added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Oct 28, 2026
…ned low-latency settings

Use nightly-dev-cu13-20261028-586355a8, which includes sgl-project/sglang#46404 and #46891. TP4 c1 runs as the batch-1 NVIDIA megakernel point of the Dynamo + SGLang aggregated row, without SIM_DRAW_AHEAD so the simulated step does the same work as a real one. TP4 c4/c16 and TP2/EP2 c16/c32 pin the flashmla attention backend instead of auto (MegaAttention pads per-rank heads to 64).
weireweire added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Oct 28, 2026
…ightly and drop TP2/EP2 c16

nightly-dev-cu13-20261028-586355a8 includes sgl-project/sglang#46404. Raise the Dynamo 1.7 backend admission limits as on the disagg rows. TP2/EP2 c16 sits far below the line from TP4 c16 to the disaggregated points, so it is removed with its override.
weireweire added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Oct 28, 2026
… 20261028 nightly and retire the Mooncake-linker disagg key

The hicache key now runs on nightly-dev-cu13-20261028-586355a8, which includes
sgl-project/sglang#46404. On that image the KV pool sizes at 890 B/token, so
hicache-ratio drops to 1.2 (1.8 exceeds host memory) and decode
mem-fraction-static to 0.75 (0.85 runs out of memory at graph capture).

Points: 5P c2884 (bs40), 8P c4096 (bs64) and 10P c7480 (bs160). The 9P and 10P
bs128 c7480 points are dropped; bs160 is higher at the same interactivity. The
dspark-disagg key on 65633129 and its recipe are retired.
cquil11 added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Oct 30, 2026
…-fixed nightlies with prefill-HiCache disagg points and TP4 c1 megakernel / [NVIDIA][AgentX] GB300 DSV4.1-Flash Dynamo + SGLang 升级到含 bonus 修复的 nightly,分离式改用 prefill HiCache,TP4 c1 启用 megakernel (#4595)

* Add GB300 DSV4.1-Flash SGLang DEP16 disagg c7480 points with prefill HiCache

Root cause: at c7480 the Mooncake external linker reads L3 hits layer by
layer on the forward path; after the AgentX turn burst, prefill stalls on
those reads (about 40 % of wall time), device prefixes age out, and the
deployment does not recover.

Fix: a new config key on nightly-dev-cu13-20261027-5884e4fe (Dynamo
1.7.0.dev20261026) with 9 and 10 DEP2 prefill workers and one DEP16
decode. Prefill keeps evicted prefixes in a local host-DRAM HiCache tier
(ratio 1.8, write-back, direct I/O); decode keeps SGLang's default polling
interval of 1. Dynamo 1.7 backend admission limits are raised so turns
queue in the scheduler, and NVIDIA/srt-slurm#629 is backported so the
readiness wait asks /health?instances=true when /health is status-only.
The agg and dspark-disagg keys are listed so they rerun in the same sweep.

Validation: on GB300 with the same recipe and AgentX client, 10 x DEP2
batch 128 gave 924,652 tok/s/GPU at p90 117.2 tok/s/user over 3600 s.

* Set perf-changelog pr-link to #4595

* Restore historical perf-changelog entries; set only this entry's pr-link

* Move the GB300 DSV4.1-Flash Dynamo + SGLang agg row to the 20261028 nightly and drop TP2/EP2 c16

nightly-dev-cu13-20261028-586355a8 includes sgl-project/sglang#46404. Raise the Dynamo 1.7 backend admission limits as on the disagg rows. TP2/EP2 c16 sits far below the line from TP4 c16 to the disaggregated points, so it is removed with its override.

* Move the GB300 DSV4.1-Flash Dynamo + SGLang HiCache disagg key to the 20261028 nightly and retire the Mooncake-linker disagg key

The hicache key now runs on nightly-dev-cu13-20261028-586355a8, which includes
sgl-project/sglang#46404. On that image the KV pool sizes at 890 B/token, so
hicache-ratio drops to 1.2 (1.8 exceeds host memory) and decode
mem-fraction-static to 0.75 (0.85 runs out of memory at graph capture).

Points: 5P c2884 (bs40), 8P c4096 (bs64) and 10P c7480 (bs160). The 9P and 10P
bs128 c7480 points are dropped; bs160 is higher at the same interactivity. The
dspark-disagg key on 65633129 and its recipe are retired.

* Use 9 prefill workers for the c7480 HiCache point

At decode batch 160, 9 x DEP2 reaches the same p90 interactivity as 10 x DEP2
with two fewer GPUs, so its throughput per GPU is higher.

* Name the KV layout, precision and Engram changes against the retired recipe in the changelog

* Move the GB300 DSV4.1-Flash Dynamo + SGLang agg row to the 20261029 nightly and use the megakernel runtime-optimization switch at TP4 c1

nightly-dev-cu13-20261029-c10e0e17 bundles sgl-nvidia-mega 54b428921749
(megakernel API 2, sgl-project/sglang#47145), so TP4 c1 starts with
--enable-nvidia-megakernel again. SGLANG_ENABLE_NV_MEGA_RUNTIME_OPTIMIZATION
(sgl-project/sglang#47105) replaces the 16 runtime-optimization variables the
recipe set one by one.

* Drop the srt-slurm #629 backport patch now that main pins srt-slurm v2.65.2

Main bumped inferencex-e2e/utils/srt-slurm to v2.65.2 (#4601), which includes
NVIDIA/srt-slurm#629, so runners/srt-slurm/patches/629-dynamo-health-instances.patch
and its README row are removed and the changelog entry no longer mentions it.

* chore: refresh PR #4595 for sweep reuse [skip-sweep]

---------

Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com>
Co-authored-by: cquil11 <60715037+cquil11@users.noreply.github.com>
cquil11 added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Oct 30, 2026
…6 and 3P1D c1228 points (append-only, follows #4595) / [NVIDIA][AgentX] 追加 GB300 DSV4.1-Flash Dynamo + SGLang TP2/EP2 c16 和 3P1D c1228 点(仅追加,接续 #4595) (#4679)

* Add GB300 DSV4.1-Flash SGLang DEP16 disagg c7480 points with prefill HiCache

Root cause: at c7480 the Mooncake external linker reads L3 hits layer by
layer on the forward path; after the AgentX turn burst, prefill stalls on
those reads (about 40 % of wall time), device prefixes age out, and the
deployment does not recover.

Fix: a new config key on nightly-dev-cu13-20261027-5884e4fe (Dynamo
1.7.0.dev20261026) with 9 and 10 DEP2 prefill workers and one DEP16
decode. Prefill keeps evicted prefixes in a local host-DRAM HiCache tier
(ratio 1.8, write-back, direct I/O); decode keeps SGLang's default polling
interval of 1. Dynamo 1.7 backend admission limits are raised so turns
queue in the scheduler, and NVIDIA/srt-slurm#629 is backported so the
readiness wait asks /health?instances=true when /health is status-only.
The agg and dspark-disagg keys are listed so they rerun in the same sweep.

Validation: on GB300 with the same recipe and AgentX client, 10 x DEP2
batch 128 gave 924,652 tok/s/GPU at p90 117.2 tok/s/user over 3600 s.

* Set perf-changelog pr-link to #4595

* Restore historical perf-changelog entries; set only this entry's pr-link

* Move the GB300 DSV4.1-Flash Dynamo + SGLang agg row to the 20261028 nightly and drop TP2/EP2 c16

nightly-dev-cu13-20261028-586355a8 includes sgl-project/sglang#46404. Raise the Dynamo 1.7 backend admission limits as on the disagg rows. TP2/EP2 c16 sits far below the line from TP4 c16 to the disaggregated points, so it is removed with its override.

* Move the GB300 DSV4.1-Flash Dynamo + SGLang HiCache disagg key to the 20261028 nightly and retire the Mooncake-linker disagg key

The hicache key now runs on nightly-dev-cu13-20261028-586355a8, which includes
sgl-project/sglang#46404. On that image the KV pool sizes at 890 B/token, so
hicache-ratio drops to 1.2 (1.8 exceeds host memory) and decode
mem-fraction-static to 0.75 (0.85 runs out of memory at graph capture).

Points: 5P c2884 (bs40), 8P c4096 (bs64) and 10P c7480 (bs160). The 9P and 10P
bs128 c7480 points are dropped; bs160 is higher at the same interactivity. The
dspark-disagg key on 65633129 and its recipe are retired.

* Use 9 prefill workers for the c7480 HiCache point

At decode batch 160, 9 x DEP2 reaches the same p90 interactivity as 10 x DEP2
with two fewer GPUs, so its throughput per GPU is higher.

* Name the KV layout, precision and Engram changes against the retired recipe in the changelog

* Move the GB300 DSV4.1-Flash Dynamo + SGLang agg row to the 20261029 nightly and use the megakernel runtime-optimization switch at TP4 c1

nightly-dev-cu13-20261029-c10e0e17 bundles sgl-nvidia-mega 54b428921749
(megakernel API 2, sgl-project/sglang#47145), so TP4 c1 starts with
--enable-nvidia-megakernel again. SGLANG_ENABLE_NV_MEGA_RUNTIME_OPTIMIZATION
(sgl-project/sglang#47105) replaces the 16 runtime-optimization variables the
recipe set one by one.

* Drop the srt-slurm #629 backport patch now that main pins srt-slurm v2.65.2

Main bumped inferencex-e2e/utils/srt-slurm to v2.65.2 (#4601), which includes
NVIDIA/srt-slurm#629, so runners/srt-slurm/patches/629-dynamo-health-instances.patch
and its README row are removed and the changelog entry no longer mentions it.

* Add back GB300 DSV4.1-Flash Dynamo+SGLang TP2/EP2 c16 and 3P1D c1228 points (append-only)

Fill the interactivity/E2EL gap between agg TP4 c16 and disagg 5P1D c2884
on each config's existing image: agg TP2/EP2 c16 with its previous
per-point settings, and a 3 x DEP2 / 1 x DEP16 c1228 point on the prefill
HiCache disagg recipe with the previous c1228 decode sizing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Set the append-only changelog pr-link to #4679

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: drop history clauses from DSV4.1-Flash hicache decode comments

---------

Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: cquil11 <60715037+cquil11@users.noreply.github.com>
nvpohanh added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Nov 3, 2026
…ly with the tuned low-latency settings and TP4 c1 megakernel / [NVIDIA][AgentX] GB300 DSV4.1-Flash SGLang 升级到 20261030 nightly,采用调优后的低延迟配置并在 TP4 c1 启用 megakernel (#4603)

* Update GB300 DSV4.1-Flash SGLang AgentX image to nightly-dev-cu13-20261025-65633129

Move dsv41flash-fp4-gb300-sglang-agentic-dspark from nightly-dev-cu13-20261021-c1c287a4 to the digest-pinned nightly-dev-cu13-20261025-65633129, the image the Dynamo + SGLang agg key already uses. Recipes and search space are unchanged.

* Set perf-changelog pr-link to #4603

* Move GB300 DSV4.1-Flash SGLang AgentX to 20261028 nightly with the tuned low-latency settings

Use nightly-dev-cu13-20261028-586355a8, which includes sgl-project/sglang#46404 and #46891. TP4 c1 runs as the batch-1 NVIDIA megakernel point of the Dynamo + SGLang aggregated row, without SIM_DRAW_AHEAD so the simulated step does the same work as a real one. TP4 c4/c16 and TP2/EP2 c16/c32 pin the flashmla attention backend instead of auto (MegaAttention pads per-rank heads to 64).

* Keep SIM_DRAW_AHEAD on the TP4 c1 megakernel point

The gate only moves the simulated-acceptance draw, a host-side sample plus a fill that real acceptance never runs, ahead of the draft launch. It does not skip work that a real-acceptance step performs.

* Keep TP4 c1 off the megakernel until the image ships a megakernel API 2 plugin

nightly-dev-cu13-20261028-586355a8 bundles sgl-nvidia-mega 6fd7ac30 (megakernel API 1), while its SGLang requires API 2 since sgl-project/sglang#45266, so --enable-nvidia-megakernel fails at startup. TP4 c1 keeps its previous engine settings and pins flashmla like the other TP-sharded points.

* Move the GB300 DSV4.1-Flash SGLang curve to the 20261029 nightly and run TP4 c1 on the megakernel

nightly-dev-cu13-20261029-c10e0e17 bundles sgl-nvidia-mega 54b428921749
(megakernel API 2), so TP4 c1 returns to its megakernel config. The 16
runtime-optimization variables it set one by one are replaced by
SGLANG_ENABLE_NV_MEGA_RUNTIME_OPTIMIZATION (sgl-project/sglang#47105).

* Move the GB300 DSV4.1-Flash SGLang key to the 20261030 nightly for the simulated verify accept cost fix

Root cause: nightly-dev-cu13-20261029-c10e0e17 predates sgl-project/sglang#46840,
so simulated acceptance skipped the real verify accept cost and the TP4 c1 point
was not measured on the methodology-conformant path.

Fix: pin lmsysorg/sglang:nightly-dev-cu13-20261030-302b78cc by digest. It is
63 commits past the #46840 merge and bundles the same sgl-nvidia-mega
54b428921749 (megakernel API 2). Update this PR's changelog entry to name the image
and #46840.

Validation: Dynamo + SGLang TP4 c1 megakernel, c4 and c16 started and completed
on 302b78cc with target and draft megakernels active on all four ranks.

---------

Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants