Repository navigation
Update benchmark scripts - #8
Merged
Merged
Conversation
5 tasks done
5 tasks done
5 tasks done
Closed
5 tasks done
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
chunyuan-w
added a commit
to chunyuan-w/sglang
that referenced
this pull request
Mar 12, 2025
* Use fused_experts_cpu and add weight packing * add check on whether AMX is supported * move utils to cpu_utils.py * address comment * no need to pass in is_vnni since it's True by default; change inplace to True * refactor prepack_weight_if_needed * Only import sgl_kernel.cpu once
chunyuan-w
added a commit
to chunyuan-w/sglang
that referenced
this pull request
Mar 14, 2025
* Use fused_experts_cpu and add weight packing * add check on whether AMX is supported * move utils to cpu_utils.py * address comment * no need to pass in is_vnni since it's True by default; change inplace to True * refactor prepack_weight_if_needed * Only import sgl_kernel.cpu once
chunyuan-w
added a commit
to chunyuan-w/sglang
that referenced
this pull request
Mar 14, 2025
* Use fused_experts_cpu and add weight packing * add check on whether AMX is supported * move utils to cpu_utils.py * address comment * no need to pass in is_vnni since it's True by default; change inplace to True * refactor prepack_weight_if_needed * Only import sgl_kernel.cpu once
chunyuan-w
added a commit
to chunyuan-w/sglang
that referenced
this pull request
Mar 14, 2025
* Use fused_experts_cpu and add weight packing * add check on whether AMX is supported * move utils to cpu_utils.py * address comment * no need to pass in is_vnni since it's True by default; change inplace to True * refactor prepack_weight_if_needed * Only import sgl_kernel.cpu once
This was referenced Apr 16, 2025
ch-wan
pushed a commit
to ch-wan/sglang
that referenced
this pull request
Apr 25, 2025
support fp8 dispatch
ch-wan
added a commit
that referenced
this pull request
May 26, 2026
…asserts Six audit-report fixes from the recent independent review: 1. **GDN / KDA / Lightning prefix recurrent state was never populated** (audit finding #1). The pool default zero state made every `prefix_lens > 0` case match the zero-prefix case trivially — the prefix only entered via `seq_lens` metadata, never via real recurrent history. Add `_populate_{gdn,kda,lightning}_prefix_state` helpers that seed each request's `temporal[mamba_idx]` with case-derived random values scaled by 0.05 (small enough to stay within the bf16 recurrent-kernel accumulation tolerance, large enough that "ignore prefix state" mutations show up as a numerical mismatch). Helpers save/restore the global RNG so they don't perturb downstream consumers. 2. **`mla/test_cutlass_mla.py` was using `ForwardMode.EXTEND`** (audit finding #2) — `CutlassMLABackend` only overrides `forward_decode`, so EXTEND silently falls through to the FlashInfer-MLA parent and never exercises cutlass code. Switch the single case to a DECODE page-boundary layout. 3. **RNG seeding in `eagle_draft_runner._make_*_draft_inputs` / `_make_*_draft_extend_inputs` was permanent** (audit finding #13). Wrap each in a shared `_seeded_rng` context manager that save/restores CPU+CUDA RNG. Pattern matches DSV4's `_populate_extra_kv_cache`. 4. **CUDA-graph capture mocked `get_tensor_model_parallel_rank` as 1** (audit finding #7). The only rank-0 branches in production `CudaGraphRunner.capture` are tqdm progress display and avail-mem logging, but a future rank-0-gated state initialiser would be silently skipped. Switch to `lambda: 0` across all three `_capture_*_graph_runner` helpers. 5. **No guard against replay prefix < capture prefix in cuda-graph decode harness** (audit finding #8). `make_dense_capture_inputs` writes random KV into cache slots up to `capture_prefix_len`; if any future replay case had `prefix_lens[r] < capture_prefix_len`, the trailing capture-time random KV would leak into the replay output. Add an explicit assertion in `_run_cuda_graph_decode_case` once `capture_prefix_len` is known. 6. **`_draft_tree_mask` for `topk > 1` only handled `draft_token_num == 3` despite the prior assertion saying `>= 3`** (audit finding #10). The mask only sets `mask[1, :2] = True` for the row=1, leaving rows >= 2 with just identity + root, which is the wrong tree shape for `draft_token_num > 3`. Tighten the assertion to `!= 3` so future tree-verify cases either match the supported shape or fail loudly instead of silently producing a malformed mask. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
rucnyz
added a commit
to rucnyz/sglang
that referenced
this pull request
Jun 2, 2026
…/forensic Round-15 audit found 16 items. All applied. Most consequential is replacing the round-14-part-2 'assert feasible' with a fatal() forensic-dump halt + new §10 two-fault-class invariant. MAJOR fixes: - #1 joint_decide infeasibility: was assert (round-14 part 2). Now fatal('joint_decide_infeasible', ...) writes structured JSON to <daemon-data>/forensic/joint_decide_infeasible_<ts>.json capturing event, state snapshot, candidates, DP inputs, traceback, then sys.exit(1). User mandate: deployment-bug-class faults expose themselves with full forensic context; load-class faults absorb. The fault classification is now an explicit §10 invariant. - #2 unavailability_cost signature fix: was transfer_time(add_tiers, u, state) — wrong signature (transfer_time is per σ→τ pair). Rewrote as sum over add_tiers of p_hat × P(serve fails) × reload. Caller passes state correctly. - #3 bw_free cold-start: pinned time_since_last_sample_s = +Inf at boot (trips the > LINK_IDLE_SECONDS branch → returns peak_bw_bps, the correct value for an unused link). - #4 multi-rank throughput aggregation: added per-field aggregation rule table to §6 — sum for byte fields, sum for link bandwidth (each rank has own slice), MEAN for throughput EMAs (same logical work observed N×), max for time_since_last_sample_s, identical for static config + n_bytes + subpool keys. - sgl-project#5 cross-rank subpool key consistency: explicit fatal() on disagreement. CLARITY: - sgl-project#6 PLAN.md drift: removed 'PUT updates cache file atomically' (cache was dropped in round-14), removed 'feasibility-fallback' (now fatal), 'samples_in_window' → 'time_since_last_sample_s'. - sgl-project#7 forecast_horizon: explicit branch for recent_event_rate=0 cold- start (returns heartbeat_s). - sgl-project#8 mean_unit_bytes empty-HBM guard: short-circuit returns ∞ to collapse top_k_pressure to K_MIN. - sgl-project#9 peak_bw_bps > 0 / h_max > 0 / prefill_bps > 0 as deployment-bug invariant (§10 Physical inputs); fatal() on violation. - sgl-project#10 APPLY_FAILED unknown batch_id: explicit silent drop (load-class, not deployment-bug); next state-fetch reconverges via always-fresh + idempotent invariants. - sgl-project#11 program state intermediates: explicit '4 states are exhaustive, intermediate states collapsed at dump time' in §5 schema comment. - sgl-project#12 'policy is not learned': softened 'in scope as an extension' → 'neither in scope nor ruled out'; consistent with paper which also dropped the PPO/DQN paragraph. - sgl-project#14 §12 S4 axis count: 7 → 9 (3 HBM + 3 DRAM-cap + 3 DISK-cap axes; the subtractive arithmetic in the original was wrong). - sgl-project#15 §12 scenario composition: explicit note that S1-S5 compose by subpool-key union; S2+S5 = {full, swa, draft}, S3+S5 = {full, mamba, draft}, etc. PLAN.md additions: - Daemon impl item 11: fatal(reason, **context) helper as the shared entry point for every deployment-bug-class halt; enumerates the call sites. Items not surfacing as a doc change: - sgl-project#13 forecast recompute overhead: noted in audit but the spec already says always-fresh covers this; no edit needed. - sgl-project#16 meta-invariant deployment-bug vs load-class: now covered by the new §10 'Two fault classes' invariant (above).
rucnyz
added a commit
to rucnyz/sglang
that referenced
this pull request
Jun 7, 2026
… bs>1 tests Adversarial audit verified the DP core correct (~270k randomized trials incl. parent-pointer reconstruction, bs>1 multi-axis, ties, over-relief clamping) — no correctness bug. Closed: * sgl-project#9 (real, perf/DoS) — the auditor broke the DESIGN "microseconds" claim: |dp| reaches ~1.8M cells / 34s with large distinct relief/ acquire bucket-deltas, and nothing guarded it, so a pathological candidate set could stall the daemon event loop for tens of seconds. Added a max_dp_cells ceiling (default 1e6, ~10x DESIGN's real-case estimate): past it both DPs raise KnapsackBudgetExceededError (with forensic ctx) → joint_decide maps to fatal (crash-only, DESIGN §10), failing loud instead of stalling. Stage E1 trips it; DESIGN §9 "microseconds" claim corrected. * sgl-project#8 — KnapsackInfeasibleError.context now carries the candidate items (DESIGN's fatal(candidates=...)) so ops can see WHICH candidates were available (top-k undersizing vs a dropped candidate). C0 asserts it. * sgl-project#10/sgl-project#11/sgl-project#12 (test depth) — E0: exact-vs-brute at bucket>1 with MULTIPLE relief AND cap axes (A2 was 1+1 at bucket=1), via a quantising oracle; E2: empty items / zero need / zero budget. * contract/NITs — docstrings state the phase precondition (min-cost takes Migrate/Pause, max-value takes Resume), bucket_size>0, and the safe-direction quantisation-can-halt note. verify/t34: 15 stages (12 + E0/E1/E2), all green. No DP-core change.
vstone-w
pushed a commit
to vstone-w/sglang
that referenced
this pull request
Jun 23, 2026
implement msa by triton to support Minimax m3
alexzms
added a commit
to FoundationResearch/sglang
that referenced
this pull request
Jun 25, 2026
Picks 6 of the 10 alignment items from colleague's audit (dev/hsa_compressedlmkk_37eabce0b_semantics(1).md). Skipped sgl-project#9 (chunked-prefill stable rid) and sgl-project#2 (already partially handled) per user direction. R40+R41 fast extend kernels preserved. R42 — Item sgl-project#10: decode LMK insertion off-by-one. hsa_should_insert_lmk_next and hsa_decode_postprocess_sampled_token now take current_seq_len, fed by the scheduler from batch.seq_lens[i]. Previously len(self.fill_ids) could diverge by one from the engine-visible KV length, mis-timing the internal LMK step. R43 — Item sgl-project#5: apply RoPE to lmk_q (selection query) when apply_hsa_rope=True and enable_lmk_q_proj=True (and nope_retrieval is False). sglang was documented as "NO RoPE" but TRM applies it; the unrotated selection query produces a different top-k chunk set. R44 — Item sgl-project#4: HoPE / InRange RoPE. New _HoPERotaryEmbedding and _InRangeRotaryEmbedding subclasses of RotaryEmbedding plus a _make_rope_for_hsa factory honoring use_hope, enable_inrange_rope, rope_context_length, rope_period_multiplier, hope_partial_ratio, hope_context_length, hope_theta. Both rotary_emb construction sites in FlashHSAInnerXAttention (SWA) and FlashHSAInnerXHierarchicalSparseAttention (HSA) now go through the factory; HSA passes for_hsa_layer=True so the use_hope+enable_inrange+SWA mixed mode falls back to standard RoPE on the SWA branch only. R45 — Item sgl-project#3: lmk_q_lora_dim 2-layer LoRA projection. When lmk_q_lora_dim > 0 and lmk_q_full_dim, self.lmk_q_proj becomes nn.ModuleList([ReplicatedLinear(d, lora_dim), ColumnParallelLinear (lora_dim, full)]) loading param names lmk_q_proj.0 / lmk_q_proj.1, and forward adds hsa_q_raw_for_lmk as the residual: lmk_q + hsa_q. Raw HSA q is captured pre-norm/RoPE in both fused and un-fused projection paths. Fusion path naturally skips ModuleList (no .weight attr → early return). R46 — Item sgl-project#1: external lmk_embed. _get_flashhsa_padded_vocab_size returns exactly vocab_size when enable_external_lmk_embed=True (vs the legacy next_of_y(vocab_size+1, 32) padding). FlashHSAInnerXModel grows self.lmk_embed: nn.Parameter([hidden_size]) and self._lmk_id; forward masks lmk positions to 0 before embed_tokens then torch.where-substitutes self.lmk_embed at those positions. No-op when flag is False. R47 — Item sgl-project#8 (audit error correction): multi-layer LMK pool slot reuse. gather_slots existed but was never CALLED — every layer ran lmk_k_pool.alloc(N) for the same logical (req, chunk), so layer-N's assign() overwrote layer-0's (req, chunk)->slot mapping. Reads through the mapping then returned a slot that layer-0 had never written to. Fix in _maybe_write_chunk_lmk_k: gather existing slots first, alloc only the missing ones, assign only those. R48 — Item sgl-project#6: enable per_qhead_prior_b in selector. Was guarded by a "deliberately NOT add — empirically WORSE KL" comment. That measurement was made BEFORE R47, so the entropy bias was being read from the wrong slot (mostly zeros). With R47 + R45 + R43 in place, prior_b is now consistent with TRM's chunk_attn_pool output. Alignment results (dev/align/compare.py, 64 stress requests): R41 R42-R48 delta KL prefill mean 0.003650 0.003654 +0.4% (noise) KL prefill max 0.003898 0.003894 ~flat KL decode mean 0.004414 0.004372 -1.0% KL decode p95 0.005420 0.005361 -1.1% KL decode max 0.008377 0.006102 **-27.2%** The decode-max improvement is the meaningful one — it's a config that exercises the per-q-head prior_b path. Single-request prefill on the 345M test ckpt doesn't trigger most of the new code paths (lmk_q_lora, external_lmk_embed, HoPE/InRange), so prefill KL is unchanged. Those fixes guard against silent regressions on production checkpoints with the corresponding config flags set.
2 tasks done
alexzms
added a commit
to FoundationResearch/sglang
that referenced
this pull request
Jun 29, 2026
…3 md)
Adds alignment support for enable_prior_query=true G=1 (MHA, hq_hsa == hk_hsa)
checkpoints, which sglang previously could not run correctly: every per-q-head /
prior_b path was gated on `hq_hsa > hk_hsa` (G>1 GQA only), so a G=1 model
skipped the chunk-level lmk_k/prior_b writer + selection entirely and read
zero-padded slots -> Step-0 selection already wrong (~5/200 token match).
Implements the friend's hsa_h200_v3_minimal_alignment_fixes.md (plan A — env
override only, default unchanged, so the G=8 paper path is untouched):
flash_hsa.py
sgl-project#2 writer activation: hq_hsa>hk_hsa -> enable_prior_query (G=1 also writes)
sgl-project#7 layerwise_lmkq_norm: RMSNorm lmk_q over full hq_hsa*head_dim width
(gated on the flag; default False keeps the per-head fast path)
sgl-project#8 pass `config` into FlashHSAInnerXAttention (was NameError in
_make_rope_for_hsa for interleaved SWA/full layers)
hsa_backend.py
sgl-project#1 SGLANG_HSA_HEADWISE_TOPK_SOFTMAX env override (env>server_args>config>
default False; default NOT flipped to True so G=8 speed is unchanged)
sgl-project#3 decode + extend per_qhead_active: > -> >= (G=1 uses the pool path)
sgl-project#5 extend pure-PyTorch selection fallback re-adds prior_b (maxpool +
softmax-kernel siblings already pass it via bias=)
sgl-project#6 only require _online_topk_group on the non-per-q-head path
+ robustness: route G=1 off the maxpool tilelang kernel (its
GEMM_M = BLOCK_L*G violates M%16 at G==1, a hard tvm crash independent
of head count) onto the softmax-topk kernel
selector.py
sgl-project#4 per-q-head decode topk branch: G>1 -> G>=1 (G=1 max-over-G is identity
but prior_b must still apply)
(sgl-project#9 q/k/v->hsa_q/k/v weight-name remap already present from d22f020.)
Verified vs dev/_verify_run/baseline_g8_pre_v3md_2a2884271.md:
G=8 alignment (friend verify, CG ON, PT=512): 12/12, KL bit-identical to base.
G=8 speed sweep 8K-512K: within +/-2.5% (noise); no OOM. No regression.
G=1 (config_345m_g1, 16-head proxy) compare: SHORT/MID/LONG all IDENTICAL/PASS
on BOTH default and headwise=1 paths (was M%16 crash before the maxpool
fallback). layerwise_lmkq_norm path needs the friend's 32-head ckpt to
validate end-to-end (no local ckpt sets the flag).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F7uq9ekiTtwehR1B4N92mJ
5 tasks done
chromecast56
referenced
this pull request
in modal-projects/sglang
Jul 4, 2026
The DFLASH draft KV pool is replicated at LOGICAL capacity under DCP (max_total_num_tokens * dcp_size, _apply_memory_pool_config) but the target's KV sizing only budgeted 1x draft bytes per physical token (scale_kv_cell_size_per_token_for_dflash with the raw draft layer count). The extra (dcp-1)x draft bytes were allocated outside the mem-fraction-static budget and consumed the activation headroom: 4xB200 Kimi-K2.6-NVFP4 dcp=4 @0.9455 booted ~4.3 GiB less free per rank than the same-build dcp=1 twin and hard-OOM'd on the first 16k-chunk prefill's trtllm_fp4_block_scale_moe 2.40 GiB workspace alloc (1.15 GiB free). Design-notes risk #8, now measured. Scale the draft layer budget by dcp_size so the pool configurator reserves the true aggregate; physical max_total shrinks accordingly and the headroom returns to (1 - mem_frac) parity with dcp=1. No behavior change when dcp_size == 1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
5 tasks done
jianyingzhu
added a commit
to jianyingzhu/sglang
that referenced
this pull request
Jul 15, 2026
FlexKV connector update, restore SWA only via single-page rollback, and drop the separate SWA mask
4 tasks done
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Jul 23, 2026
… side now homogeneous TP=2 + one fixed drafter The upstream comparison side of section sgl-project#3 was drawn as TP=1 (a single card), which is a pointless comparison against the fork's TP=3 (different scale). Change the upstream reference to a homogeneous even TP=2 (2 identical 24 GB cards, assumption named, whole side estimated/hatched) running ONE FIXED drafter (e.g. NEXTN k=3): single draft residence, no per-k rung tags, no dual-residence, no runtime routing (upstream cannot switch draft algorithm). Reframe the trade-off neutrally: the fork holds BOTH drafters resident plus the routing machinery (solo-draft pool 4.58 GiB + rung tags DFLASH_k16 662 / EAGLE_k3 634 / k2 554 MiB, measured) = more draft VRAM but runtime-adaptive; upstream TP=2 holds one fixed drafter = less draft VRAM but no runtime choice. A capability / VRAM trade-off, not a verdict (upstream is not worse, just fixed). Fork left side (TP=3 uneven, dual-residence) unchanged. Audited the other feature sections for the same Fork-TP=3-vs-upstream-TP=1 mismatch: only sgl-project#3 was affected. Sections sgl-project#4 (session spill), sgl-project#8 (measured VRAM budget) and sgl-project#9 (fast-lane) intentionally show one representative card on BOTH sides — they are per-rank / per-card mechanism/behaviour illustrations, not hardware-scale claims, so 1-vs-1 is legitimate and left as-is. TP=1 that is by design (weightless head-card, solo-5090) also left as-is. SVG regenerated via gen.py, alt text updated; validated well-formed and overflow-free, all img references resolve.
1 of 3 tasks
4 tasks done
5 tasks done
Merged
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.