Skip to content

Update benchmark scripts - #8

Merged
merrymercy merged 9 commits into
mainfrom
benchmark
Jan 16, 2024
Merged

merrymercy merged 9 commits into
mainfrom
benchmark

Conversation

@merrymercy

Copy link
Copy Markdown
Collaborator

No description provided.

@merrymercy
merrymercy merged commit 70359bf into main Jan 16, 2024
@merrymercy
merrymercy deleted the benchmark branch January 16, 2024 00:13
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
chunyuan-w added a commit to chunyuan-w/sglang that referenced this pull request Mar 12, 2025
* Use fused_experts_cpu and add weight packing

* add check on whether AMX is supported

* move utils to cpu_utils.py

* address comment

* no need to pass in is_vnni since it's True by default; change inplace to True

* refactor prepack_weight_if_needed

* Only import sgl_kernel.cpu once
chunyuan-w added a commit to chunyuan-w/sglang that referenced this pull request Mar 14, 2025
* Use fused_experts_cpu and add weight packing

* add check on whether AMX is supported

* move utils to cpu_utils.py

* address comment

* no need to pass in is_vnni since it's True by default; change inplace to True

* refactor prepack_weight_if_needed

* Only import sgl_kernel.cpu once
chunyuan-w added a commit to chunyuan-w/sglang that referenced this pull request Mar 14, 2025
* Use fused_experts_cpu and add weight packing

* add check on whether AMX is supported

* move utils to cpu_utils.py

* address comment

* no need to pass in is_vnni since it's True by default; change inplace to True

* refactor prepack_weight_if_needed

* Only import sgl_kernel.cpu once
chunyuan-w added a commit to chunyuan-w/sglang that referenced this pull request Mar 14, 2025
* Use fused_experts_cpu and add weight packing

* add check on whether AMX is supported

* move utils to cpu_utils.py

* address comment

* no need to pass in is_vnni since it's True by default; change inplace to True

* refactor prepack_weight_if_needed

* Only import sgl_kernel.cpu once
ch-wan pushed a commit to ch-wan/sglang that referenced this pull request Apr 25, 2025
ch-wan added a commit that referenced this pull request May 26, 2026
…asserts

Six audit-report fixes from the recent independent review:

1. **GDN / KDA / Lightning prefix recurrent state was never populated**
   (audit finding #1). The pool default zero state made every
   `prefix_lens > 0` case match the zero-prefix case trivially — the
   prefix only entered via `seq_lens` metadata, never via real recurrent
   history. Add `_populate_{gdn,kda,lightning}_prefix_state` helpers that
   seed each request's `temporal[mamba_idx]` with case-derived random
   values scaled by 0.05 (small enough to stay within the bf16
   recurrent-kernel accumulation tolerance, large enough that "ignore
   prefix state" mutations show up as a numerical mismatch). Helpers
   save/restore the global RNG so they don't perturb downstream consumers.

2. **`mla/test_cutlass_mla.py` was using `ForwardMode.EXTEND`** (audit
   finding #2) — `CutlassMLABackend` only overrides `forward_decode`,
   so EXTEND silently falls through to the FlashInfer-MLA parent and
   never exercises cutlass code. Switch the single case to a DECODE
   page-boundary layout.

3. **RNG seeding in `eagle_draft_runner._make_*_draft_inputs` /
   `_make_*_draft_extend_inputs` was permanent** (audit finding #13).
   Wrap each in a shared `_seeded_rng` context manager that
   save/restores CPU+CUDA RNG. Pattern matches DSV4's
   `_populate_extra_kv_cache`.

4. **CUDA-graph capture mocked `get_tensor_model_parallel_rank` as 1**
   (audit finding #7). The only rank-0 branches in production
   `CudaGraphRunner.capture` are tqdm progress display and avail-mem
   logging, but a future rank-0-gated state initialiser would be
   silently skipped. Switch to `lambda: 0` across all three
   `_capture_*_graph_runner` helpers.

5. **No guard against replay prefix < capture prefix in cuda-graph
   decode harness** (audit finding #8). `make_dense_capture_inputs` writes
   random KV into cache slots up to `capture_prefix_len`; if any future
   replay case had `prefix_lens[r] < capture_prefix_len`, the trailing
   capture-time random KV would leak into the replay output. Add an
   explicit assertion in `_run_cuda_graph_decode_case` once
   `capture_prefix_len` is known.

6. **`_draft_tree_mask` for `topk > 1` only handled `draft_token_num == 3`
   despite the prior assertion saying `>= 3`** (audit finding #10). The
   mask only sets `mask[1, :2] = True` for the row=1, leaving rows >= 2
   with just identity + root, which is the wrong tree shape for
   `draft_token_num > 3`. Tighten the assertion to `!= 3` so future
   tree-verify cases either match the supported shape or fail loudly
   instead of silently producing a malformed mask.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
rucnyz added a commit to rucnyz/sglang that referenced this pull request Jun 2, 2026
…/forensic

Round-15 audit found 16 items. All applied. Most consequential is
replacing the round-14-part-2 'assert feasible' with a fatal()
forensic-dump halt + new §10 two-fault-class invariant.

MAJOR fixes:
- #1 joint_decide infeasibility: was assert (round-14 part 2). Now
  fatal('joint_decide_infeasible', ...) writes structured JSON to
  <daemon-data>/forensic/joint_decide_infeasible_<ts>.json
  capturing event, state snapshot, candidates, DP inputs, traceback,
  then sys.exit(1). User mandate: deployment-bug-class faults
  expose themselves with full forensic context; load-class faults
  absorb. The fault classification is now an explicit §10 invariant.
- #2 unavailability_cost signature fix: was transfer_time(add_tiers,
  u, state) — wrong signature (transfer_time is per σ→τ pair).
  Rewrote as sum over add_tiers of p_hat × P(serve fails) × reload.
  Caller passes state correctly.
- #3 bw_free cold-start: pinned time_since_last_sample_s = +Inf at
  boot (trips the > LINK_IDLE_SECONDS branch → returns peak_bw_bps,
  the correct value for an unused link).
- #4 multi-rank throughput aggregation: added per-field aggregation
  rule table to §6 — sum for byte fields, sum for link bandwidth
  (each rank has own slice), MEAN for throughput EMAs (same logical
  work observed N×), max for time_since_last_sample_s, identical
  for static config + n_bytes + subpool keys.
- sgl-project#5 cross-rank subpool key consistency: explicit fatal() on
  disagreement.

CLARITY:
- sgl-project#6 PLAN.md drift: removed 'PUT updates cache file atomically' (cache
  was dropped in round-14), removed 'feasibility-fallback' (now
  fatal), 'samples_in_window' → 'time_since_last_sample_s'.
- sgl-project#7 forecast_horizon: explicit branch for recent_event_rate=0 cold-
  start (returns heartbeat_s).
- sgl-project#8 mean_unit_bytes empty-HBM guard: short-circuit returns ∞ to
  collapse top_k_pressure to K_MIN.
- sgl-project#9 peak_bw_bps > 0 / h_max > 0 / prefill_bps > 0 as deployment-bug
  invariant (§10 Physical inputs); fatal() on violation.
- sgl-project#10 APPLY_FAILED unknown batch_id: explicit silent drop (load-class,
  not deployment-bug); next state-fetch reconverges via always-fresh
  + idempotent invariants.
- sgl-project#11 program state intermediates: explicit '4 states are exhaustive,
  intermediate states collapsed at dump time' in §5 schema comment.
- sgl-project#12 'policy is not learned': softened 'in scope as an extension' →
  'neither in scope nor ruled out'; consistent with paper which
  also dropped the PPO/DQN paragraph.
- sgl-project#14 §12 S4 axis count: 7 → 9 (3 HBM + 3 DRAM-cap + 3 DISK-cap
  axes; the subtractive arithmetic in the original was wrong).
- sgl-project#15 §12 scenario composition: explicit note that S1-S5 compose
  by subpool-key union; S2+S5 = {full, swa, draft}, S3+S5 =
  {full, mamba, draft}, etc.

PLAN.md additions:
- Daemon impl item 11: fatal(reason, **context) helper as the
  shared entry point for every deployment-bug-class halt;
  enumerates the call sites.

Items not surfacing as a doc change:
- sgl-project#13 forecast recompute overhead: noted in audit but the spec
  already says always-fresh covers this; no edit needed.
- sgl-project#16 meta-invariant deployment-bug vs load-class: now covered by
  the new §10 'Two fault classes' invariant (above).
rucnyz added a commit to rucnyz/sglang that referenced this pull request Jun 7, 2026
… bs>1 tests

Adversarial audit verified the DP core correct (~270k randomized
trials incl. parent-pointer reconstruction, bs>1 multi-axis, ties,
over-relief clamping) — no correctness bug.  Closed:

* sgl-project#9 (real, perf/DoS) — the auditor broke the DESIGN "microseconds"
  claim: |dp| reaches ~1.8M cells / 34s with large distinct relief/
  acquire bucket-deltas, and nothing guarded it, so a pathological
  candidate set could stall the daemon event loop for tens of seconds.
  Added a max_dp_cells ceiling (default 1e6, ~10x DESIGN's real-case
  estimate): past it both DPs raise KnapsackBudgetExceededError (with
  forensic ctx) → joint_decide maps to fatal (crash-only, DESIGN §10),
  failing loud instead of stalling.  Stage E1 trips it; DESIGN §9
  "microseconds" claim corrected.
* sgl-project#8 — KnapsackInfeasibleError.context now carries the candidate items
  (DESIGN's fatal(candidates=...)) so ops can see WHICH candidates were
  available (top-k undersizing vs a dropped candidate).  C0 asserts it.
* sgl-project#10/sgl-project#11/sgl-project#12 (test depth) — E0: exact-vs-brute at bucket>1 with
  MULTIPLE relief AND cap axes (A2 was 1+1 at bucket=1), via a
  quantising oracle; E2: empty items / zero need / zero budget.
* contract/NITs — docstrings state the phase precondition (min-cost
  takes Migrate/Pause, max-value takes Resume), bucket_size>0, and the
  safe-direction quantisation-can-halt note.

verify/t34: 15 stages (12 + E0/E1/E2), all green.  No DP-core change.
vstone-w pushed a commit to vstone-w/sglang that referenced this pull request Jun 23, 2026
implement msa by triton to support Minimax m3
alexzms added a commit to FoundationResearch/sglang that referenced this pull request Jun 25, 2026
Picks 6 of the 10 alignment items from colleague's audit
(dev/hsa_compressedlmkk_37eabce0b_semantics(1).md). Skipped sgl-project#9
(chunked-prefill stable rid) and sgl-project#2 (already partially handled) per
user direction. R40+R41 fast extend kernels preserved.

R42 — Item sgl-project#10: decode LMK insertion off-by-one. hsa_should_insert_lmk_next
  and hsa_decode_postprocess_sampled_token now take current_seq_len, fed by
  the scheduler from batch.seq_lens[i]. Previously len(self.fill_ids) could
  diverge by one from the engine-visible KV length, mis-timing the internal
  LMK step.

R43 — Item sgl-project#5: apply RoPE to lmk_q (selection query) when apply_hsa_rope=True
  and enable_lmk_q_proj=True (and nope_retrieval is False). sglang was
  documented as "NO RoPE" but TRM applies it; the unrotated selection query
  produces a different top-k chunk set.

R44 — Item sgl-project#4: HoPE / InRange RoPE. New _HoPERotaryEmbedding and
  _InRangeRotaryEmbedding subclasses of RotaryEmbedding plus a
  _make_rope_for_hsa factory honoring use_hope, enable_inrange_rope,
  rope_context_length, rope_period_multiplier, hope_partial_ratio,
  hope_context_length, hope_theta. Both rotary_emb construction sites in
  FlashHSAInnerXAttention (SWA) and FlashHSAInnerXHierarchicalSparseAttention
  (HSA) now go through the factory; HSA passes for_hsa_layer=True so the
  use_hope+enable_inrange+SWA mixed mode falls back to standard RoPE on the
  SWA branch only.

R45 — Item sgl-project#3: lmk_q_lora_dim 2-layer LoRA projection. When
  lmk_q_lora_dim > 0 and lmk_q_full_dim, self.lmk_q_proj becomes
  nn.ModuleList([ReplicatedLinear(d, lora_dim), ColumnParallelLinear
  (lora_dim, full)]) loading param names lmk_q_proj.0 / lmk_q_proj.1, and
  forward adds hsa_q_raw_for_lmk as the residual: lmk_q + hsa_q. Raw HSA q
  is captured pre-norm/RoPE in both fused and un-fused projection paths.
  Fusion path naturally skips ModuleList (no .weight attr → early return).

R46 — Item sgl-project#1: external lmk_embed. _get_flashhsa_padded_vocab_size returns
  exactly vocab_size when enable_external_lmk_embed=True (vs the legacy
  next_of_y(vocab_size+1, 32) padding). FlashHSAInnerXModel grows
  self.lmk_embed: nn.Parameter([hidden_size]) and self._lmk_id; forward
  masks lmk positions to 0 before embed_tokens then torch.where-substitutes
  self.lmk_embed at those positions. No-op when flag is False.

R47 — Item sgl-project#8 (audit error correction): multi-layer LMK pool slot reuse.
  gather_slots existed but was never CALLED — every layer ran
  lmk_k_pool.alloc(N) for the same logical (req, chunk), so layer-N's
  assign() overwrote layer-0's (req, chunk)->slot mapping. Reads through
  the mapping then returned a slot that layer-0 had never written to. Fix
  in _maybe_write_chunk_lmk_k: gather existing slots first, alloc only the
  missing ones, assign only those.

R48 — Item sgl-project#6: enable per_qhead_prior_b in selector. Was guarded by a
  "deliberately NOT add — empirically WORSE KL" comment. That measurement
  was made BEFORE R47, so the entropy bias was being read from the wrong
  slot (mostly zeros). With R47 + R45 + R43 in place, prior_b is now
  consistent with TRM's chunk_attn_pool output.

Alignment results (dev/align/compare.py, 64 stress requests):
                       R41        R42-R48      delta
  KL prefill mean    0.003650   0.003654     +0.4% (noise)
  KL prefill max     0.003898   0.003894     ~flat
  KL decode mean     0.004414   0.004372     -1.0%
  KL decode p95      0.005420   0.005361     -1.1%
  KL decode max      0.008377   0.006102     **-27.2%**

The decode-max improvement is the meaningful one — it's a config that
exercises the per-q-head prior_b path. Single-request prefill on the
345M test ckpt doesn't trigger most of the new code paths (lmk_q_lora,
external_lmk_embed, HoPE/InRange), so prefill KL is unchanged. Those
fixes guard against silent regressions on production checkpoints with
the corresponding config flags set.
@harmya harmya mentioned this pull request Jun 25, 2026
5 tasks done
alexzms added a commit to FoundationResearch/sglang that referenced this pull request Jun 29, 2026
…3 md)

Adds alignment support for enable_prior_query=true G=1 (MHA, hq_hsa == hk_hsa)
checkpoints, which sglang previously could not run correctly: every per-q-head /
prior_b path was gated on `hq_hsa > hk_hsa` (G>1 GQA only), so a G=1 model
skipped the chunk-level lmk_k/prior_b writer + selection entirely and read
zero-padded slots -> Step-0 selection already wrong (~5/200 token match).

Implements the friend's hsa_h200_v3_minimal_alignment_fixes.md (plan A — env
override only, default unchanged, so the G=8 paper path is untouched):

  flash_hsa.py
    sgl-project#2  writer activation: hq_hsa>hk_hsa -> enable_prior_query (G=1 also writes)
    sgl-project#7  layerwise_lmkq_norm: RMSNorm lmk_q over full hq_hsa*head_dim width
        (gated on the flag; default False keeps the per-head fast path)
    sgl-project#8  pass `config` into FlashHSAInnerXAttention (was NameError in
        _make_rope_for_hsa for interleaved SWA/full layers)
  hsa_backend.py
    sgl-project#1  SGLANG_HSA_HEADWISE_TOPK_SOFTMAX env override (env>server_args>config>
        default False; default NOT flipped to True so G=8 speed is unchanged)
    sgl-project#3  decode + extend per_qhead_active: > -> >= (G=1 uses the pool path)
    sgl-project#5  extend pure-PyTorch selection fallback re-adds prior_b (maxpool +
        softmax-kernel siblings already pass it via bias=)
    sgl-project#6  only require _online_topk_group on the non-per-q-head path
    +   robustness: route G=1 off the maxpool tilelang kernel (its
        GEMM_M = BLOCK_L*G violates M%16 at G==1, a hard tvm crash independent
        of head count) onto the softmax-topk kernel
  selector.py
    sgl-project#4  per-q-head decode topk branch: G>1 -> G>=1 (G=1 max-over-G is identity
        but prior_b must still apply)
  (sgl-project#9 q/k/v->hsa_q/k/v weight-name remap already present from d22f020.)

Verified vs dev/_verify_run/baseline_g8_pre_v3md_2a2884271.md:
  G=8 alignment (friend verify, CG ON, PT=512): 12/12, KL bit-identical to base.
  G=8 speed sweep 8K-512K: within +/-2.5% (noise); no OOM. No regression.
  G=1 (config_345m_g1, 16-head proxy) compare: SHORT/MID/LONG all IDENTICAL/PASS
  on BOTH default and headwise=1 paths (was M%16 crash before the maxpool
  fallback). layerwise_lmkq_norm path needs the friend's 32-head ckpt to
  validate end-to-end (no local ckpt sets the flag).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F7uq9ekiTtwehR1B4N92mJ
chromecast56 referenced this pull request in modal-projects/sglang Jul 4, 2026
The DFLASH draft KV pool is replicated at LOGICAL capacity under DCP
(max_total_num_tokens * dcp_size, _apply_memory_pool_config) but the
target's KV sizing only budgeted 1x draft bytes per physical token
(scale_kv_cell_size_per_token_for_dflash with the raw draft layer
count). The extra (dcp-1)x draft bytes were allocated outside the
mem-fraction-static budget and consumed the activation headroom:
4xB200 Kimi-K2.6-NVFP4 dcp=4 @0.9455 booted ~4.3 GiB less free per
rank than the same-build dcp=1 twin and hard-OOM'd on the first
16k-chunk prefill's trtllm_fp4_block_scale_moe 2.40 GiB workspace
alloc (1.15 GiB free). Design-notes risk #8, now measured.

Scale the draft layer budget by dcp_size so the pool configurator
reserves the true aggregate; physical max_total shrinks accordingly
and the headroom returns to (1 - mem_frac) parity with dcp=1.
No behavior change when dcp_size == 1.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jianyingzhu added a commit to jianyingzhu/sglang that referenced this pull request Jul 15, 2026
FlexKV connector update, restore SWA only via single-page rollback, and drop the separate SWA mask
efschu added a commit to efschu/htsglang that referenced this pull request Jul 23, 2026
… side now homogeneous TP=2 + one fixed drafter

The upstream comparison side of section sgl-project#3 was drawn as TP=1 (a single card),
which is a pointless comparison against the fork's TP=3 (different scale).
Change the upstream reference to a homogeneous even TP=2 (2 identical 24 GB
cards, assumption named, whole side estimated/hatched) running ONE FIXED
drafter (e.g. NEXTN k=3): single draft residence, no per-k rung tags, no
dual-residence, no runtime routing (upstream cannot switch draft algorithm).

Reframe the trade-off neutrally: the fork holds BOTH drafters resident plus the
routing machinery (solo-draft pool 4.58 GiB + rung tags DFLASH_k16 662 /
EAGLE_k3 634 / k2 554 MiB, measured) = more draft VRAM but runtime-adaptive;
upstream TP=2 holds one fixed drafter = less draft VRAM but no runtime choice.
A capability / VRAM trade-off, not a verdict (upstream is not worse, just
fixed). Fork left side (TP=3 uneven, dual-residence) unchanged.

Audited the other feature sections for the same Fork-TP=3-vs-upstream-TP=1
mismatch: only sgl-project#3 was affected. Sections sgl-project#4 (session spill), sgl-project#8 (measured VRAM
budget) and sgl-project#9 (fast-lane) intentionally show one representative card on BOTH
sides — they are per-rank / per-card mechanism/behaviour illustrations, not
hardware-scale claims, so 1-vs-1 is legitimate and left as-is. TP=1 that is by
design (weightless head-card, solo-5090) also left as-is.

SVG regenerated via gen.py, alt text updated; validated well-formed and
overflow-free, all img references resolve.
@vstone-w vstone-w mentioned this pull request Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant