Skip to content

Fix model loading & format code - #125

Merged
merrymercy merged 1 commit into
mainfrom
fix
Jan 31, 2024
Merged

merrymercy merged 1 commit into
mainfrom
fix

Conversation

@merrymercy

Copy link
Copy Markdown
Collaborator

No description provided.

@merrymercy
merrymercy merged commit ad82bac into main Jan 31, 2024
@merrymercy
merrymercy deleted the fix branch January 31, 2024 07:49
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
pi314ever pushed a commit to pi314ever/sglang that referenced this pull request Sep 5, 2025
* init fp8 kv cache support for deepseek-r1

* fix lint

* fix code spell

* remove DECODE_BLOCK_BUCKET_MAX hardcode on g2
sammysun0711 added a commit to sammysun0711/sglang that referenced this pull request Dec 31, 2025
…t#125)

* fused quant kv with mrops rms kernel

* update ci kvcache dtype

* update ci config

---------

Co-authored-by: Xiake Sun <xisun@amd.com>
vschandramourya pushed a commit to vschandramourya/sglang that referenced this pull request Feb 3, 2026
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jue WANG <zjuwangjue@gmail.com>
efschu added a commit to efschu/htsglang that referenced this pull request Jul 29, 2026
…t mit MoE-Experten-Spill-Maschinerie (sgl-project#77/sgl-project#123/sgl-project#125/sgl-project#236/sgl-project#242) — ein generisches VRAM-Posten-Tiering mit Klassen-Registrierung statt Parallel-Mechanismus; Slice beginnt mit Schnittstellen-Audit statt Neubau
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
… objective

Four coordinator answers folded in; two of them close holes in the first draft.

R9 (NEW, user 2026-08-05) -- every prefill/draft/decode(verify) cut is itself a
load change, and an ENFORCER must actually cause a different layout per load.
Flip granularity becomes a PRICED AXIS rather than a blanket rule: per knob,
quantify the flip cost and classify by two checkable criteria. A knob is
tick-flippable exactly when flipping it costs neither a CUDA-graph recapture
nor a byte moved; everything else flips at regime boundaries with hysteresis.
Grounded, not asserted: sgl-project#439 cold-expert compute assignment is verified
VRAM-neutral in tree (expert_heat_migration.py:31 names "the same size before
and after" as the sgl-project#439 sizing latch's invariant), which is precisely the
zero-data-movement property that puts it on the tick side. Within a tick the
layout is fixed by definition, so the right tools are overlap/behaviour
adaptation (sgl-project#128/sgl-project#199, sgl-project#125, sgl-project#274, sgl-project#156) -- not lesser substitutes for
flipping, but the correct instrument at that timescale.

F9 is the falsifier for the silent error: a wrongly-regime-classified zero-move
knob never demonstrates the gain it could have made, so nothing ever surfaces
it. F9b guards the opposite mistake (a 3-6 s recapture at tick granularity
would spend the whole tick budget on recapture). F9c asserts on ACTUATED state,
not on a logged decision, because sgl-project#578 is exactly the case where the decision
existed and the actuation did not.

SLICE 0 (sgl-project#578) -- verified in tree, not taken on report: regime_runtime.py:911
calls planner_candidates(server_args) without solve_fn, so regime_stages.py:383
takes the None branch, returns [], and the stage table permanently holds the
booted stage alone. The docstring at regime_stages.py:357 says it outright --
"it is not broken, it is unfed". The enforcer R9(i) requires therefore EXISTS
AS CODE AND ACTUATES NEVER, which made every actuated requirement in the first
draft unreachable. Promoted from unknown risk to a named prerequisite slice.
The work is wiring plus a SolverAnswer->Stage mapping (key_solver.solve already
exists and is objective-aware), but it needs plan_inputs, base plan, per-rank
budgets and RigRates -- i.e. a card probe. So slice 0 shares TICKET_582 gate
(a) as a prerequisite and should be scheduled into the same window; recorded.

R4 -- answered, [CONFIRM] removed. The objective is JOINT (maximum compute AND
maximum VRAM throughput/latency at all times), expressed as two axes rather
than collapsed into one scalar on the user's behalf. Dominance on disagreement
is decided by the running task's phase: decode is memory-bound so bandwidth IS
throughput there, prefill is compute-bound. The weights come from the same
regime signal R7 consumes. F4 is a pair for the same reason as F7: a fixed
scalar objective passes exactly one of the two regimes.

sgl-project#551 -- resolved as in-tree sgl-project#553, and my find is confirmed: DESIGN/ANALYSE
only, no mechanism. Slice 5 therefore BUILDS elastic co-residency as the
planner's tenant-event actuator. Stated plainly in the slice plan and §5 so the
plan cannot be misread as driving something that exists.

Also recorded: R9's knob inventory is incomplete by construction (only the
knobs the addendum named are classified); a full sweep with a cost per knob is
part of slice 4b, since an unclassified knob is an unpriced assumption and R9
forbids those. And "3-6 s" for recapture is an observed range, not a model --
R5's movement-cost work must cover it too.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
…anularity

Fifth addendum. R7 made EVACUATION ORDERING phase-dependent; R10 says the same
of the WEIGHTS themselves, and at a finer grain than "the model": individual
layers and slices of layers.

The named first-class case is the draft/target ping-pong. During DRAFT, target
layers or slices may be spilled to any tier for any reason -- including simply
that the video and sound pipelines want the VRAM and this is the only way
everything is held at once. During PREFILL/VERIFY the draft spills and the
target returns. The two models take turns owning VRAM because they take turns
computing.

(a) Every combination must be EXPRESSIBLE, over the product
(layer|slice) x tier x phase. R3 applied to residency: a design that enumerates
"draft spilled" and "target spilled" as two supported modes has already failed,
because the third thing wanted is a half-spilled target during a long prefill
while the diffusion lane holds a card.

(b) The planner computes WHAT moves WHEN as a movement-bound decision on R9's
priced flip axis -- never a structural cannot. A decline must be arithmetic and
recoverable under R8.

(c) Executed automatically, and the freed VRAM REFILLED automatically. An empty
hole is a DEFECT, not a neutral outcome: it is the "surplus sits idle" failure
sgl-project#582 removed from the boot path, reappearing at runtime. F10c therefore asserts
on the CLAIM of the freed bytes, not on the free.

OVERLAP MUST BE PRICED. While the draft computes the target's layers are idle,
so their reload hides behind draft compute and vice versa. The cost model
charges the unhidden remainder,
  effective_cost = max(0, transfer_time - overlappable_compute_time),
not raw bytes/bandwidth. Charging raw transfer would price every ping-pong as
unaffordable and the planner would never take a swap that is in fact nearly
free -- the same error shape as pricing a restore at capture cost (F9e): a cost
model wrong upward never exercises the mechanism, so nothing surfaces it.

CARRIERS, verified with an honest split. Present: sgl-project#77/sgl-project#123/sgl-project#125 weight
streaming really does H2D fetch of weight rows from a host tier
(expert_offload.py, expert_heat_migration.py, fused_moe_triton/layer.py,
cold_tier_fetch.py with identity-keyed host DRAM segments and zero-copy peer
views, plus offload_register/short_term_offload_register/offload_movement).
NOT present: it is EXPERT-granular. I found no layer-granular streaming and no
per-layer addressing in offload_movement.py, and could not locate the
double-buffered prefetch-behind-previous-layer-compute structure as a general
mechanism. Slice 5 therefore EXTENDS this carrier rather than driving it -- same
posture as sgl-project#553 -- and the overlap pricing in R10 DEPENDS on that structure
being real, so slice 5 must confirm or build it.

F10 (planned AND executed, not refused), F10b (the verify phase after a spill
must be BYTE-IDENTICAL to a never-spilled reference -- residency is a placement
decision, never a numerics decision, or the mechanism is unusable under the
determinism arm), F10c (freed bytes must be claimed).

Slice plan gains 4d (express + price, desk-provable) and extends slice 5.

Also recorded in gaps: the 2026-08-05 window's measured out-of-budget demand
(2434 MiB on a 3080, 1305 on the 5090) already contradicts the inherited stock
activation heuristic (3968 MiB), so R2's "exact" is not yet met on that term.
Fixing it is a prerequisite for trusting any objective computed from it.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 22, 2026
…the order does not matter

sgl-project#809 asks for a hybrid residency of the phase-flip weight images: a small
pinned share (a "hot head, in wave order"), the rest file-backed as today, and
a reload with prefetch overlap on the sgl-project#125 double-buffer pattern. The prior-art
gate falsified two of those three, and the third rests on a comparison nobody
has run. Recording that before writing code, because the cheapest outcome here
is the one where nothing is rebuilt.

THE OVERLAP IS ALREADY SHIPPED. `_staged_file_refill` (sgl-project#802, landed
2026-08-22, weights_arena.py:522) reads the file-backed image with bounded
preadv into sgl-project#720's pinned ring and overlaps the next chunk's read with the
previous chunk's H2D DMA -- depth streams, depth events, an inflight flag, and
a synchronize before a buffer is refilled. The depth is
SGLANG_PHASE_FLIP_REFILL_DEPTH, default 2: literally a double buffer.
Rebuilding it would be the sgl-project#720-verbatim mistake in a new place.

THERE IS NO HOT HEAD. `arena_refill` copies the whole payload and only then
checksums it; nothing reads the arena before that checksum passes. There is no
partial publication and no per-wave consumption, so WHICH bytes are pinned
cannot matter -- only HOW MANY. sgl-project#254's wave order exists because a MoE forward
consumes experts wave by wave; a flip consumes the arena once, whole. A "hot
head selection" would be a knob with no effect.

TWO OF THE THREE CITED NUMBERS ARE NOT IN THIS TREE. Grepped as rates over
python/sglang/srt/ and docs/dev/: zero hits for 2850, 4263, 1763 and 1844 as a
MiB/s, MB/s or GB/s figure, and the near-misses are named in the note so the
next reader does not "find" them by accident (1763 is a byte budget in an
affordability check and an NVML corridor floor; 1844 is a seam-staging free
memory delta). The pinned figure IS real: _PINNED_REF_LO_GBPS / _HI_GBPS =
4.93 / 8.88, phase_flip_boot.py:502-509. What the numbers that DO exist say:
pinned 4.93/7.08/8.88 GB/s per rank (sgl-project#690) against staged O_DIRECT 2651/2602/
3751 MiB/s per rank in the real flip on metal (sgl-project#802), i.e. roughly 1.8-2.6x --
but from two separate campaigns, never against each other, and the note says so
rather than presenting it as an A/B.

COMPRESSION IS DISCARDED, WITH THE ARITHMETIC, not deferred. This tree's own
ANALYSE_306 already ends "do not build sgl-project#306 as a codec", and sgl-project#456's sparse
write records a 0 byte win on /spinning because ZFS folds the same holes --
which is where the flip images live. Independently, at ratio 1.145 the bytes
saved are 12.7%, so a serial win needs decompression at 7.9x the read rate (64
GiB/s synthetic, 20-29 GiB/s at metal rates) and even a fully overlapped one
needs 9.29 GiB/s per rank, 10.07 GiB/s aggregate across three concurrent ranks.
The measured decompression ceiling in ANALYSE_306 is 4.3-4.8 GB/s. Both bars
are out of reach by 2-13x.

THE DANGER DIRECTION, AND THE ONE POSITION THIS DOES NOT REVERSE. The
file-backed arm exists so ~68.7 GiB of unreclaimable host RAM cannot OOM-kill a
swapless boot, and it refuses a missing or tmpfs image dir rather than
"silently allocating a pinned image the host ledger would then double-count as
reclaimable". The tree has ALREADY decided that image posts are registered but
not checked -- "a new refusal path here could break a boot that works today",
weights_arena.py:918-924. That decision stands. It does not bind a new,
opt-in, default-off pinned share, for its own stated reason: refusing a share
no current boot requests cannot break a boot that works today. So only the new
share may be checked, and a hybrid image must register exactly its pinned head
and nothing else, because the file-backed bytes are deliberately unregistered
(the registry sums NON-reclaimable bytes).

THE HAZARD ANY IMPLEMENTATION MUST HANDLE, recorded because it is silent:
_staged_file_refill takes O_DIRECT only when `at % 4096 == 0`. If reads resume
at an unaligned pinned-head boundary, EVERY chunk misses that test, the whole
refill falls back to the buffered fd -- 8304 -> 2595 MiB/s, a 3.2x regression
with no error and no log line -- and the pinned head costs more than it saves.
The head must be floored to _DIRECT_ALIGN.

NOT BUILT HERE, deliberately: the share's size is the whole feature, and the
comparison that would set it (pinned vs staged-O_DIRECT on ONE binary, same
load, same bytes) has never been run. A size derived from the two-campaign
mixture above would be the rig-fit the planner rules forbid, and shipping the
actuator with the size left as an unmeasured parameter would put a knob in the
tree nobody can set -- while a helper with no caller is exactly the defect
removed from planner/hicache_staging.py in this same branch. The A/B is named
as a window item, with the instrumentation that already exists for it
(_timed_arena_refill / refill_report).

Analysis only: no code changes, no test changes, no behaviour change.
codespell clean.
Jiminator added a commit that referenced this pull request Oct 16, 2026
[Beam] Optimize mixed-chunk decode with FP8 KV support
zcnrex added a commit to zcnrex/sglang that referenced this pull request Oct 28, 2026
Recovers the ~2.5% prefill regression on CC 10.7 (DeepGEMM sgl-project#125).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants