Skip to content

Update quick start examples - #120

Merged
merrymercy merged 4 commits into
mainfrom
docs
Jan 30, 2024
Merged

merrymercy merged 4 commits into
mainfrom
docs

Conversation

@merrymercy

Copy link
Copy Markdown
Collaborator

No description provided.

@merrymercy
merrymercy merged commit 0617528 into main Jan 30, 2024
@merrymercy
merrymercy deleted the docs branch January 30, 2024 12:29
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
NorthmanPKU pushed a commit to NorthmanPKU/sglang that referenced this pull request May 16, 2025
pi314ever pushed a commit to pi314ever/sglang that referenced this pull request Aug 14, 2025
pi314ever pushed a commit to pi314ever/sglang that referenced this pull request Sep 15, 2025
…ct#135)

- Restrict optimisation in PR sgl-project#120 to Llama-3.1-8B due to
regression in Llama4 and Llama3 70b

Signed-off-by: Rahul Vijayaraghavan <rahul.vijayaraghavan@intel.com>
vschandramourya pushed a commit to vschandramourya/sglang that referenced this pull request Feb 3, 2026
AndyLi429 added a commit to randgun/sglang that referenced this pull request Mar 23, 2026
…ect#120)

* support prefill context parallism for deepseek v3 and qwen3

* fix and clear code

* fix tp rank

* refactor atten_cp to cover pcp

* add PCP usage docs and ASCEND_USE_FIA env for Qwen3 MoE on Ascend NPU

* pre-commit and fix AI review

* fix code

* Port ascend_kv_transfer for Qwen3 MoE PD+CP

* Fix PD bootstrap and Qwen3 PCP metadata handling

* fix code formatting via pre-commit hooks

* FIX enable cp

* Fix review comments

* ADD TEST

* fix test

---------

Co-authored-by: AndyLi429 <124504189@qq.com>
Co-authored-by: t00937989 <tanlei33@huawei.com>
Co-authored-by: AndyLi429 <68410213+AndyLi429@users.noreply.github.com>
Co-authored-by: root <root@localhost.localdomain>
efschu added a commit to efschu/htsglang that referenced this pull request Jul 19, 2026
efschu added a commit to efschu/htsglang that referenced this pull request Jul 20, 2026
…ministic

The FP8-triton expert-offload was documented as byte-identical to the
no-offload run. That overclaim was retracted by the implementing commit
0fb3d80 itself: the compacted resident+scratch slot layout re-associates
the FP reduction versus the full-N layout, so cross-config (offload vs
no-offload) output agrees only on confident tokens and diverges at near-ties
(sub-ULP), while the +0.15% perplexity delta is that same reduction-order
signature. The honest property is SELF-DETERMINISM (run==run byte-identical
at a fixed fraction) plus quality-neutrality (sgl-project#120), NOT byte-identity to the
no-offload path.

This puts FP8 and marlin-Int4 in the SAME correctness class (self-det +
near-tie divergence only, not bit-exact), differing only in magnitude:
FP8 sub-ULP vs marlin ~1e-2. FEATURES_VS_UPSTREAM.md §6 and TOPOLOGIES.md
(1.5-note, 2.4, footnotes) corrected accordingly.
efschu added a commit to efschu/htsglang that referenced this pull request Jul 22, 2026
…reuse)

Redesign the per-expert MoE offload from refetch-all to FIXED-RESIDENT + SCRATCH
so the host pinned pool holds only the SPILL experts (~spill, not the full set)
-> caps load-time host RAM, AND resident experts are reused across waves (no
re-fetch -> throughput recovery). Unifies the 122B load-time RAM fix with the
deterministic-reuse perf item (they are the same change).

Model (expert_offload.py):
- ExpertResidencyPlanner: experts [0,R) fixed-resident on GPU at slot==id
  (never fetched); a wave's SPILL experts (>=R), sorted, fetch into scratch
  slots [R, R+C). Deterministic: layout is a pure function of the wave's needed
  set + fixed buffer size (R+C) -> self-det holds (validated model on the 35B
  next). resident_slot_count(R) + scratch_slot_count(C=max(8,R//4), env
  SGLANG_MOE_SCRATCH_SLOTS override).
- plan_token_waves: split on unique SPILL <= scratch (resident impose no budget).
- MoEExpertOffloadCache: [R+C] GPU buffer ([0,R) fixed + scratch) + [E-R] pinned
  host spill; _fetch indexes spill[e-R]; consumes a load-time presplit if present.
- presplit_expert_offload_after_repack(layer): the LOAD-TIME RAM cap. Runs right
  after the marlin repack (repacked on GPU, inside device_loading_context):
  splits each expert-major tensor into the [R+C] GPU buffer + pinned-host spill,
  replaces the param with a 0-row placeholder so device_loading_context pins
  ~nothing back to host -> the full [E] stack never sits in host RAM.

Wiring: gptq_moe.py + awq_moe.py process_weights_after_loading call the presplit
after the kernel repack; layer.py install log updated. Planner unit tests
rewritten for the new model (15/15 pass). Default path (fraction>=1.0)
untouched. Needs GPU re-validation on the 35B GPTQ at the sgl-project#77/sgl-project#120 bar.
efschu added a commit to efschu/htsglang that referenced this pull request Jul 22, 2026
…34%)

Follow-up (a): the 'graph + hot-set together' best config on the 122B end-state,
the compounding win the 35B predicted (static +75% -> hot-set +123%). No code
delta -- hot-set-from-file was landed in 5d5494d; this records the 122B
best-config validation.

Method: booted the 122B eager (offload f=0.25) with SGLANG_MOE_OFFLOAD_TRACE,
drove 10 representative prompts, aggregated top-64/layer over all 48 layers via
/tmp/s3_make_hotset.py (149760 routes, 63.1% routed-mass captured vs ~25%
uniform) -> /tmp/s3_122b_hotset.json. Then eager-hotset (baseline) and
graph-hotset boots, residency-matched.

122B tok/s ladder (TP=3/DCP=3, f=0.25, 5090+2x3080, float16):
  eager-static (baseline)  6.97
  graph-static             10.61  (+52%)
  eager-hotset              9.25  (+33%)
  graph-hotset  (BEST)     16.34  (+134% over baseline; +54% over graph-static)
Compounds exactly like the 35B (+123%). This is the full 'drum und dran' number.

BEST-config gate (all PASS):
- Coherent ' Paris' (id 11751); FROZEN FROM FILE x144 (48 layers x 3 ranks);
  captures buckets [1,2]; 366 decode-graph replays; self-det 5/5; VRAM fits
  (5090 28.9/32.6, 3080s 20.0/18.9 of 20.5 GB).
- Machinery Delta=0: GPU isolation test covers the hot-set LAYOUT
  (E=256/R=64/C=16) bit-identical eager-vs-captured -> our code adds 0.
- Tokens identical (32 pos) + first-flip None (argmax never flips).
- End-to-end Delta bounded & explained by decomposition (nearties_compare):
    graph-hot vs eager-hot (capture floor @ hot-set residency): max 7.68e-2
    eager-hot vs eager-static (PURE residency re-tiling, NO capture): max 1.08e-1
    graph-static vs eager-static (capture floor @ static): max 3.34e-2
  The capture floor at the hot-set residency (7.68e-2) is BELOW the pure-eager
  residency-permutation floor (1.08e-1) -- the Stage-1 marlin slot-permutation
  floor already established and accepted (sgl-project#77/sgl-project#120). So graph+hot-set adds
  nothing beyond the already-accepted marlin-Int4 intrinsic regime; argmax is
  preserved everywhere and self-det holds.

Captures: /tmp/s3_122b_hot_{eager,graph}.json, hotset /tmp/s3_122b_hotset.json.
tests/moe_offload 49/49.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant