Repository navigation
Update quick start examples - #120
Merged
Merged
Conversation
5 tasks done
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
NorthmanPKU
pushed a commit
to NorthmanPKU/sglang
that referenced
this pull request
May 16, 2025
pi314ever
pushed a commit
to pi314ever/sglang
that referenced
this pull request
Aug 14, 2025
pi314ever
pushed a commit
to pi314ever/sglang
that referenced
this pull request
Sep 15, 2025
…ct#135) - Restrict optimisation in PR sgl-project#120 to Llama-3.1-8B due to regression in Llama4 and Llama3 70b Signed-off-by: Rahul Vijayaraghavan <rahul.vijayaraghavan@intel.com>
vschandramourya
pushed a commit
to vschandramourya/sglang
that referenced
this pull request
Feb 3, 2026
AndyLi429
added a commit
to randgun/sglang
that referenced
this pull request
Mar 23, 2026
…ect#120) * support prefill context parallism for deepseek v3 and qwen3 * fix and clear code * fix tp rank * refactor atten_cp to cover pcp * add PCP usage docs and ASCEND_USE_FIA env for Qwen3 MoE on Ascend NPU * pre-commit and fix AI review * fix code * Port ascend_kv_transfer for Qwen3 MoE PD+CP * Fix PD bootstrap and Qwen3 PCP metadata handling * fix code formatting via pre-commit hooks * FIX enable cp * Fix review comments * ADD TEST * fix test --------- Co-authored-by: AndyLi429 <124504189@qq.com> Co-authored-by: t00937989 <tanlei33@huawei.com> Co-authored-by: AndyLi429 <68410213+AndyLi429@users.noreply.github.com> Co-authored-by: root <root@localhost.localdomain>
6 tasks done
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Jul 19, 2026
…loss) + PCIe-BW throughput caveats
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Jul 20, 2026
…ministic The FP8-triton expert-offload was documented as byte-identical to the no-offload run. That overclaim was retracted by the implementing commit 0fb3d80 itself: the compacted resident+scratch slot layout re-associates the FP reduction versus the full-N layout, so cross-config (offload vs no-offload) output agrees only on confident tokens and diverges at near-ties (sub-ULP), while the +0.15% perplexity delta is that same reduction-order signature. The honest property is SELF-DETERMINISM (run==run byte-identical at a fixed fraction) plus quality-neutrality (sgl-project#120), NOT byte-identity to the no-offload path. This puts FP8 and marlin-Int4 in the SAME correctness class (self-det + near-tie divergence only, not bit-exact), differing only in magnitude: FP8 sub-ULP vs marlin ~1e-2. FEATURES_VS_UPSTREAM.md §6 and TOPOLOGIES.md (1.5-note, 2.4, footnotes) corrected accordingly.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Jul 22, 2026
…reuse) Redesign the per-expert MoE offload from refetch-all to FIXED-RESIDENT + SCRATCH so the host pinned pool holds only the SPILL experts (~spill, not the full set) -> caps load-time host RAM, AND resident experts are reused across waves (no re-fetch -> throughput recovery). Unifies the 122B load-time RAM fix with the deterministic-reuse perf item (they are the same change). Model (expert_offload.py): - ExpertResidencyPlanner: experts [0,R) fixed-resident on GPU at slot==id (never fetched); a wave's SPILL experts (>=R), sorted, fetch into scratch slots [R, R+C). Deterministic: layout is a pure function of the wave's needed set + fixed buffer size (R+C) -> self-det holds (validated model on the 35B next). resident_slot_count(R) + scratch_slot_count(C=max(8,R//4), env SGLANG_MOE_SCRATCH_SLOTS override). - plan_token_waves: split on unique SPILL <= scratch (resident impose no budget). - MoEExpertOffloadCache: [R+C] GPU buffer ([0,R) fixed + scratch) + [E-R] pinned host spill; _fetch indexes spill[e-R]; consumes a load-time presplit if present. - presplit_expert_offload_after_repack(layer): the LOAD-TIME RAM cap. Runs right after the marlin repack (repacked on GPU, inside device_loading_context): splits each expert-major tensor into the [R+C] GPU buffer + pinned-host spill, replaces the param with a 0-row placeholder so device_loading_context pins ~nothing back to host -> the full [E] stack never sits in host RAM. Wiring: gptq_moe.py + awq_moe.py process_weights_after_loading call the presplit after the kernel repack; layer.py install log updated. Planner unit tests rewritten for the new model (15/15 pass). Default path (fraction>=1.0) untouched. Needs GPU re-validation on the 35B GPTQ at the sgl-project#77/sgl-project#120 bar.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Jul 22, 2026
…34%) Follow-up (a): the 'graph + hot-set together' best config on the 122B end-state, the compounding win the 35B predicted (static +75% -> hot-set +123%). No code delta -- hot-set-from-file was landed in 5d5494d; this records the 122B best-config validation. Method: booted the 122B eager (offload f=0.25) with SGLANG_MOE_OFFLOAD_TRACE, drove 10 representative prompts, aggregated top-64/layer over all 48 layers via /tmp/s3_make_hotset.py (149760 routes, 63.1% routed-mass captured vs ~25% uniform) -> /tmp/s3_122b_hotset.json. Then eager-hotset (baseline) and graph-hotset boots, residency-matched. 122B tok/s ladder (TP=3/DCP=3, f=0.25, 5090+2x3080, float16): eager-static (baseline) 6.97 graph-static 10.61 (+52%) eager-hotset 9.25 (+33%) graph-hotset (BEST) 16.34 (+134% over baseline; +54% over graph-static) Compounds exactly like the 35B (+123%). This is the full 'drum und dran' number. BEST-config gate (all PASS): - Coherent ' Paris' (id 11751); FROZEN FROM FILE x144 (48 layers x 3 ranks); captures buckets [1,2]; 366 decode-graph replays; self-det 5/5; VRAM fits (5090 28.9/32.6, 3080s 20.0/18.9 of 20.5 GB). - Machinery Delta=0: GPU isolation test covers the hot-set LAYOUT (E=256/R=64/C=16) bit-identical eager-vs-captured -> our code adds 0. - Tokens identical (32 pos) + first-flip None (argmax never flips). - End-to-end Delta bounded & explained by decomposition (nearties_compare): graph-hot vs eager-hot (capture floor @ hot-set residency): max 7.68e-2 eager-hot vs eager-static (PURE residency re-tiling, NO capture): max 1.08e-1 graph-static vs eager-static (capture floor @ static): max 3.34e-2 The capture floor at the hot-set residency (7.68e-2) is BELOW the pure-eager residency-permutation floor (1.08e-1) -- the Stage-1 marlin slot-permutation floor already established and accepted (sgl-project#77/sgl-project#120). So graph+hot-set adds nothing beyond the already-accepted marlin-Int4 intrinsic regime; argmax is preserved everywhere and self-det holds. Captures: /tmp/s3_122b_hot_{eager,graph}.json, hotset /tmp/s3_122b_hotset.json. tests/moe_offload 49/49.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.