Repository navigation
Fix model loading & format code - #125
Merged
Merged
Conversation
5 tasks done
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
pi314ever
pushed a commit
to pi314ever/sglang
that referenced
this pull request
Sep 5, 2025
* init fp8 kv cache support for deepseek-r1 * fix lint * fix code spell * remove DECODE_BLOCK_BUCKET_MAX hardcode on g2
sammysun0711
added a commit
to sammysun0711/sglang
that referenced
this pull request
Dec 31, 2025
…t#125) * fused quant kv with mrops rms kernel * update ci kvcache dtype * update ci config --------- Co-authored-by: Xiake Sun <xisun@amd.com>
4 tasks
vschandramourya
pushed a commit
to vschandramourya/sglang
that referenced
this pull request
Feb 3, 2026
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Jue WANG <zjuwangjue@gmail.com>
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Jul 29, 2026
…t mit MoE-Experten-Spill-Maschinerie (sgl-project#77/sgl-project#123/sgl-project#125/sgl-project#236/sgl-project#242) — ein generisches VRAM-Posten-Tiering mit Klassen-Registrierung statt Parallel-Mechanismus; Slice beginnt mit Schnittstellen-Audit statt Neubau
Closed
6 tasks done
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
… objective Four coordinator answers folded in; two of them close holes in the first draft. R9 (NEW, user 2026-08-05) -- every prefill/draft/decode(verify) cut is itself a load change, and an ENFORCER must actually cause a different layout per load. Flip granularity becomes a PRICED AXIS rather than a blanket rule: per knob, quantify the flip cost and classify by two checkable criteria. A knob is tick-flippable exactly when flipping it costs neither a CUDA-graph recapture nor a byte moved; everything else flips at regime boundaries with hysteresis. Grounded, not asserted: sgl-project#439 cold-expert compute assignment is verified VRAM-neutral in tree (expert_heat_migration.py:31 names "the same size before and after" as the sgl-project#439 sizing latch's invariant), which is precisely the zero-data-movement property that puts it on the tick side. Within a tick the layout is fixed by definition, so the right tools are overlap/behaviour adaptation (sgl-project#128/sgl-project#199, sgl-project#125, sgl-project#274, sgl-project#156) -- not lesser substitutes for flipping, but the correct instrument at that timescale. F9 is the falsifier for the silent error: a wrongly-regime-classified zero-move knob never demonstrates the gain it could have made, so nothing ever surfaces it. F9b guards the opposite mistake (a 3-6 s recapture at tick granularity would spend the whole tick budget on recapture). F9c asserts on ACTUATED state, not on a logged decision, because sgl-project#578 is exactly the case where the decision existed and the actuation did not. SLICE 0 (sgl-project#578) -- verified in tree, not taken on report: regime_runtime.py:911 calls planner_candidates(server_args) without solve_fn, so regime_stages.py:383 takes the None branch, returns [], and the stage table permanently holds the booted stage alone. The docstring at regime_stages.py:357 says it outright -- "it is not broken, it is unfed". The enforcer R9(i) requires therefore EXISTS AS CODE AND ACTUATES NEVER, which made every actuated requirement in the first draft unreachable. Promoted from unknown risk to a named prerequisite slice. The work is wiring plus a SolverAnswer->Stage mapping (key_solver.solve already exists and is objective-aware), but it needs plan_inputs, base plan, per-rank budgets and RigRates -- i.e. a card probe. So slice 0 shares TICKET_582 gate (a) as a prerequisite and should be scheduled into the same window; recorded. R4 -- answered, [CONFIRM] removed. The objective is JOINT (maximum compute AND maximum VRAM throughput/latency at all times), expressed as two axes rather than collapsed into one scalar on the user's behalf. Dominance on disagreement is decided by the running task's phase: decode is memory-bound so bandwidth IS throughput there, prefill is compute-bound. The weights come from the same regime signal R7 consumes. F4 is a pair for the same reason as F7: a fixed scalar objective passes exactly one of the two regimes. sgl-project#551 -- resolved as in-tree sgl-project#553, and my find is confirmed: DESIGN/ANALYSE only, no mechanism. Slice 5 therefore BUILDS elastic co-residency as the planner's tenant-event actuator. Stated plainly in the slice plan and §5 so the plan cannot be misread as driving something that exists. Also recorded: R9's knob inventory is incomplete by construction (only the knobs the addendum named are classified); a full sweep with a cost per knob is part of slice 4b, since an unclassified knob is an unpriced assumption and R9 forbids those. And "3-6 s" for recapture is an observed range, not a model -- R5's movement-cost work must cover it too.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
…anularity Fifth addendum. R7 made EVACUATION ORDERING phase-dependent; R10 says the same of the WEIGHTS themselves, and at a finer grain than "the model": individual layers and slices of layers. The named first-class case is the draft/target ping-pong. During DRAFT, target layers or slices may be spilled to any tier for any reason -- including simply that the video and sound pipelines want the VRAM and this is the only way everything is held at once. During PREFILL/VERIFY the draft spills and the target returns. The two models take turns owning VRAM because they take turns computing. (a) Every combination must be EXPRESSIBLE, over the product (layer|slice) x tier x phase. R3 applied to residency: a design that enumerates "draft spilled" and "target spilled" as two supported modes has already failed, because the third thing wanted is a half-spilled target during a long prefill while the diffusion lane holds a card. (b) The planner computes WHAT moves WHEN as a movement-bound decision on R9's priced flip axis -- never a structural cannot. A decline must be arithmetic and recoverable under R8. (c) Executed automatically, and the freed VRAM REFILLED automatically. An empty hole is a DEFECT, not a neutral outcome: it is the "surplus sits idle" failure sgl-project#582 removed from the boot path, reappearing at runtime. F10c therefore asserts on the CLAIM of the freed bytes, not on the free. OVERLAP MUST BE PRICED. While the draft computes the target's layers are idle, so their reload hides behind draft compute and vice versa. The cost model charges the unhidden remainder, effective_cost = max(0, transfer_time - overlappable_compute_time), not raw bytes/bandwidth. Charging raw transfer would price every ping-pong as unaffordable and the planner would never take a swap that is in fact nearly free -- the same error shape as pricing a restore at capture cost (F9e): a cost model wrong upward never exercises the mechanism, so nothing surfaces it. CARRIERS, verified with an honest split. Present: sgl-project#77/sgl-project#123/sgl-project#125 weight streaming really does H2D fetch of weight rows from a host tier (expert_offload.py, expert_heat_migration.py, fused_moe_triton/layer.py, cold_tier_fetch.py with identity-keyed host DRAM segments and zero-copy peer views, plus offload_register/short_term_offload_register/offload_movement). NOT present: it is EXPERT-granular. I found no layer-granular streaming and no per-layer addressing in offload_movement.py, and could not locate the double-buffered prefetch-behind-previous-layer-compute structure as a general mechanism. Slice 5 therefore EXTENDS this carrier rather than driving it -- same posture as sgl-project#553 -- and the overlap pricing in R10 DEPENDS on that structure being real, so slice 5 must confirm or build it. F10 (planned AND executed, not refused), F10b (the verify phase after a spill must be BYTE-IDENTICAL to a never-spilled reference -- residency is a placement decision, never a numerics decision, or the mechanism is unusable under the determinism arm), F10c (freed bytes must be claimed). Slice plan gains 4d (express + price, desk-provable) and extends slice 5. Also recorded in gaps: the 2026-08-05 window's measured out-of-budget demand (2434 MiB on a 3080, 1305 on the 5090) already contradicts the inherited stock activation heuristic (3968 MiB), so R2's "exact" is not yet met on that term. Fixing it is a prerequisite for trusting any objective computed from it.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 22, 2026
…the order does not matter sgl-project#809 asks for a hybrid residency of the phase-flip weight images: a small pinned share (a "hot head, in wave order"), the rest file-backed as today, and a reload with prefetch overlap on the sgl-project#125 double-buffer pattern. The prior-art gate falsified two of those three, and the third rests on a comparison nobody has run. Recording that before writing code, because the cheapest outcome here is the one where nothing is rebuilt. THE OVERLAP IS ALREADY SHIPPED. `_staged_file_refill` (sgl-project#802, landed 2026-08-22, weights_arena.py:522) reads the file-backed image with bounded preadv into sgl-project#720's pinned ring and overlaps the next chunk's read with the previous chunk's H2D DMA -- depth streams, depth events, an inflight flag, and a synchronize before a buffer is refilled. The depth is SGLANG_PHASE_FLIP_REFILL_DEPTH, default 2: literally a double buffer. Rebuilding it would be the sgl-project#720-verbatim mistake in a new place. THERE IS NO HOT HEAD. `arena_refill` copies the whole payload and only then checksums it; nothing reads the arena before that checksum passes. There is no partial publication and no per-wave consumption, so WHICH bytes are pinned cannot matter -- only HOW MANY. sgl-project#254's wave order exists because a MoE forward consumes experts wave by wave; a flip consumes the arena once, whole. A "hot head selection" would be a knob with no effect. TWO OF THE THREE CITED NUMBERS ARE NOT IN THIS TREE. Grepped as rates over python/sglang/srt/ and docs/dev/: zero hits for 2850, 4263, 1763 and 1844 as a MiB/s, MB/s or GB/s figure, and the near-misses are named in the note so the next reader does not "find" them by accident (1763 is a byte budget in an affordability check and an NVML corridor floor; 1844 is a seam-staging free memory delta). The pinned figure IS real: _PINNED_REF_LO_GBPS / _HI_GBPS = 4.93 / 8.88, phase_flip_boot.py:502-509. What the numbers that DO exist say: pinned 4.93/7.08/8.88 GB/s per rank (sgl-project#690) against staged O_DIRECT 2651/2602/ 3751 MiB/s per rank in the real flip on metal (sgl-project#802), i.e. roughly 1.8-2.6x -- but from two separate campaigns, never against each other, and the note says so rather than presenting it as an A/B. COMPRESSION IS DISCARDED, WITH THE ARITHMETIC, not deferred. This tree's own ANALYSE_306 already ends "do not build sgl-project#306 as a codec", and sgl-project#456's sparse write records a 0 byte win on /spinning because ZFS folds the same holes -- which is where the flip images live. Independently, at ratio 1.145 the bytes saved are 12.7%, so a serial win needs decompression at 7.9x the read rate (64 GiB/s synthetic, 20-29 GiB/s at metal rates) and even a fully overlapped one needs 9.29 GiB/s per rank, 10.07 GiB/s aggregate across three concurrent ranks. The measured decompression ceiling in ANALYSE_306 is 4.3-4.8 GB/s. Both bars are out of reach by 2-13x. THE DANGER DIRECTION, AND THE ONE POSITION THIS DOES NOT REVERSE. The file-backed arm exists so ~68.7 GiB of unreclaimable host RAM cannot OOM-kill a swapless boot, and it refuses a missing or tmpfs image dir rather than "silently allocating a pinned image the host ledger would then double-count as reclaimable". The tree has ALREADY decided that image posts are registered but not checked -- "a new refusal path here could break a boot that works today", weights_arena.py:918-924. That decision stands. It does not bind a new, opt-in, default-off pinned share, for its own stated reason: refusing a share no current boot requests cannot break a boot that works today. So only the new share may be checked, and a hybrid image must register exactly its pinned head and nothing else, because the file-backed bytes are deliberately unregistered (the registry sums NON-reclaimable bytes). THE HAZARD ANY IMPLEMENTATION MUST HANDLE, recorded because it is silent: _staged_file_refill takes O_DIRECT only when `at % 4096 == 0`. If reads resume at an unaligned pinned-head boundary, EVERY chunk misses that test, the whole refill falls back to the buffered fd -- 8304 -> 2595 MiB/s, a 3.2x regression with no error and no log line -- and the pinned head costs more than it saves. The head must be floored to _DIRECT_ALIGN. NOT BUILT HERE, deliberately: the share's size is the whole feature, and the comparison that would set it (pinned vs staged-O_DIRECT on ONE binary, same load, same bytes) has never been run. A size derived from the two-campaign mixture above would be the rig-fit the planner rules forbid, and shipping the actuator with the size left as an unmeasured parameter would put a knob in the tree nobody can set -- while a helper with no caller is exactly the defect removed from planner/hicache_staging.py in this same branch. The A/B is named as a window item, with the instrumentation that already exists for it (_timed_arena_refill / refill_report). Analysis only: no code changes, no test changes, no behaviour change. codespell clean.
Jiminator
added a commit
that referenced
this pull request
Oct 16, 2026
[Beam] Optimize mixed-chunk decode with FP8 KV support
zcnrex
added a commit
to zcnrex/sglang
that referenced
this pull request
Oct 28, 2026
Recovers the ~2.5% prefill regression on CC 10.7 (DeepGEMM sgl-project#125). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.