Skip to content

fix: resolve the fp8 bug introduced by vLLM 0.5.5 - #1276

Merged
zhyncs merged 1 commit into
mainfrom
fix-deepseek-v2-fp8
Aug 31, 2024
Merged

zhyncs merged 1 commit into
mainfrom
fix-deepseek-v2-fp8

Conversation

@zhyncs

@zhyncs zhyncs commented Aug 31, 2024

Copy link
Copy Markdown
Contributor

Motivation

Modifications

Checklist

  • Format your code according to the Contributor Guide.
  • Add unit tests as outlined in the Contributor Guide.
  • Update documentation as needed, including docstrings or example tutorials.

@zhyncs
zhyncs merged commit 52cefdb into main Aug 31, 2024
@zhyncs
zhyncs deleted the fix-deepseek-v2-fp8 branch August 31, 2024 14:44
@zhyncs zhyncs mentioned this pull request Sep 1, 2024
3 tasks
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 8, 2026
Boot weg2sb4 (e4f1b9f) ran 1,499.8 HICACHE-ROUND/s on group P and
303.1/s on group D for 34 minutes with an empty queue and a live front.
Measured from that boot's own logs: P 677,214 lines / n=5,643,450 over
3,504 s, D 105,354 lines / n=877,950 over 3,628 s.

ROOT, and it is not missing machinery.  The sgl-project#547 blocking-poll ladder is
fully built and correct (idle_sleeper.py: rungs 0/1/10 ms, 50 ms cap).  It
was never CONSTRUCTED: scheduler.py:1243 init_idle_sleeper arms only under
--sleep-on-idle or SGLANG_IDLE_BLOCKING_POLL, and the weg-2 launcher sets
neither, so `idle_sleeper` was None on all six ranks of both groups and the
loop never blocked.  At the ladder's 50 ms cap an idle origin ticks at
~20 Hz instead of ~1,500 Hz -- a 75x reduction.

The RANK GATE is not the defect and is kept.  Only the request origin owns
the zmq sockets the ladder polls; ranks above it take a blocking chain
receive / broadcast (request_receiver.py: "ranks 1..n-1 from
point_to_point_pyobj") and are driven at the origin's cadence, which is why
all three PP stages logged byte-identical round counts.  Parking the origin
parks the whole group: one wait point, not six, and no rank-local decision
(RAENGE-NIE-UNEINS).  Arming is scoped to the weg-2 form via
SGLANG_WEG2_GROUP -- the launcher's own established discriminator -- so the
stock default is untouched.

Second, smaller allocator on the same path: the sgl-project#1028 ROUND CENSUS line was
written every 25th round, ~60 lines/s/rank of "nothing changed", each one a
format tuple + formatted string + LogRecord on the round path.  The counter
still counts every round; the LINE is now written on a state change, plus a
60 s heartbeat.

Tests hermetic (CUDA_VISIBLE_DEVICES=""), 14 passed.  Red-first proven: 3
of them fail against the unpatched tree (weg-2 arming, quiet rounds, state
change).  test_every_pass_still_does_its_work is the regression guard for
the fix-1c / fix-2 per-pass semantics -- a queued control object still gets
its lap within one pass; only the log line changed, never the work.
Ruff parity exact: scheduler.py 106 -> 106, unified_radix_cache.py 1 -> 1.

NOT BOOTED.  Rounds/s at idle is an arithmetic consequence of the ladder
cap, not a measurement; the next boot's WEG2-IDLE-CENSUS line is what
settles it.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 8, 2026
…line

Boot weg2sb4 grew ~19.0 MiB/min of host anon across its six ranks while
completely idle, against an independently observed +21.3 MiB/min.  Measured
per rank over two /proc samples 140 s apart: P PP0 +5.4, PP1 +5.4, PP2 +0.0,
each D rank +2.8 MiB/min; everything else (front, detokenizers, launcher
parents, inductor workers) exactly 0.0.

WHERE IT IS.  64 MiB-aligned [anon] rw-p glibc malloc arenas, extended in
place plus new arenas appearing (PP1 7d51f4000000 63.7->64.0, 7d51e8000000
19.6->20.4, new 7d51ec000000 0->11.43 MiB).  It is NOT the host ring --
dShmem was 0.0 on every process -- NOT CUDA pinned memory (/dev/zero flat)
and NOT the store.

MALLOC_ARENA_MAX=4, chosen from this rig's measurement rather than folklore:
glibc defaults to 8 x ncores arenas and CPU affinity here is 0-31, so the
default ceiling is 256.  The allocating threads on the steady-state path are
the scheduler thread plus the three the reset path names by hand ("#N RESET
JOIN threads=['prefetch', 'backup', 'prefetch_io_aux']"), so 4 gives each one
its own arena without the default's fragmentation surface.  It is a CEILING,
published with setdefault so an operator experiment still wins.

malloc_trim runs at the idle point, BOUNDED: only when this rank actually
grew WEG2_TRIM_MIN_GROWTH_MIB since the last trim, because trim walks the
arena free lists and a flat rank must not pay for it.

WEG2-IDLE-CENSUS, per rank, once a minute while idle:
  rounds_per_s=<n> rss_anon_mib=<n> d_anon_mib_per_min=<n> threads=<n>
  arena_max=<s> trimmed=<n> group=<P|D> rank=<n>
from /proc/self/status only -- no smaps walk (these ranks carry ~3,300
mappings).  It exists because NEITHER fact was in the log: the round rate had
to be reconstructed by counting sgl-project#1028 lines with an external script, and the
growth was only visible because a desk agent read /proc minutes before a host
OOM killed the processes.  The next boot's acceptance can now gate on
rounds_per_s ~= 0 and d_anon_mib_per_min ~= 0 with no external tool.

`threads=` is in the line deliberately: the per-rank thread count was NOT
captured before the OOM, so the arena value is derived from the reset line's
thread names and the next boot confirms or corrects it.

EXPECTED EFFECT, stated as arithmetic and not as a measurement: the sgl-project#1276
ladder cap takes an idle origin from ~1,500 to ~20 rounds/s (75x), and the
growth is per-round allocator churn, so the arena bound plus the reduced
round rate should take +19.0 MiB/min toward ~0.  UNPROVEN until a boot.

WHAT REMAINS UNEXPLAINED: in the same 140 s window cgroup anon rose +815 MiB
while the tracked processes explain only +44.3.  That window was NOT idle --
a desk probe was reading the checkpoint inside it -- so the +815 is
confounded and must not be quoted as an sglang rate; the +19.0 MiB/min
per-process figure is the defensible one and it is the one that matches the
independent idle observation.  The full-system attribution of the remainder
was never taken: the scan was killed by the OOM before its second sample.

Tests hermetic, 11 passed, red-first proven (the launcher publication test
fails against the unpatched tree).  Ruff parity exact: launcher.py 0 -> 0,
scheduler.py 106 -> 106 (the census import is function-local, which also
keeps weg2 from reaching back into managers at module scope).

NOT BOOTED.  No GPU used; cards were free and nothing was booked.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 8, 2026
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 12, 2026
…ED emitter fields

(1) ORIGIN FLOOR. The front samples a group's run-moment residual at its FIRST
SLEEP, i.e. inside a flip, so that reading has already realised the permanent
step `flip_ratchet_gib` charges separately -- weg2xsn24's winning floor 7.49 GiB
was taken at 19:44:31Z, THREE SECONDS before that boot's `WEG2-FLIP done
epoch=2`. `run_origin_gib` now REJECTS an interleaved run-moment sample whenever
a launch-moment reading exists, prints the rejected figure with its reason
(`origin=6.45 source=launch ... 7.49 ... REJECTED: interleaved`), and does NOT
promote the dk7 stand-in in its place: that stand-in answers "no record at all",
not "the record was refused", and at 24.50 GiB it would be a WORSE origin than
the one just refused. W95 is narrowed, not removed -- it still refuses the sum
of both residual rules, and an origin with no launch reading still has none.

(2) RUN-PEAK RESIDUAL. On ratchet-priced arms the residual is the max of the
five sgl-project#1350 replays (xsn20 -1.513, xsn21b -0.170, xsn22 +0.375, xsn23 +0.908,
xsn24 +0.046 -> +0.908) plus its own measured steady-state drift
(RATCHET_RESIDUAL_DRIFT_MIB_PER_MIN = 0.07, sgl-project#1276 max) over the planned window,
both printed with their source. The pre-ratchet weg2sb4 5.16 row is NOT
hand-edited and still binds every arm that does not charge the ratchet; summing
the two remains W95. Boot margin 8.60 -> 1.47, hard bound 87.30 -> 94.43.

(3) #1350d NEED EMITTER, so xsn25 can be the ring-anchor measuring boot:
`leg=<epoch>/<src>-><dst>` on every WEG2-RING NEED line (`set_leg`), and on an
arm with no ring `need_mib=NOT-APPLICABLE saved_mib=<n>` instead of
`need_mib=0` -- measured on xsn24, where 32 of group P's 60 NEED lines read 0
because no ring was published, not because the tag had nothing to save. The
third field the anchor needs (one line per SAVED tag, including the
non-weights tags) sits in the caller that passes the tag list, not in
ring_guard, and is left OPEN and named in the order's NACHTRAG.

DRY RUN, xsn25 form, M=150: run_peak 90.43 GiB against the 94.43 bound -- THE
PEAK NOW FUNDS, with 4.00 GiB of room, exactly the predicted effect. rc=2 all
the same, and the binding term is a DIFFERENT one: `launch_sum = -6.33 GiB`
against the #1317n safety-net conjunct (`launch_worst_case >= -margin`), while
both moment leftovers are positive (launch +5.67, run +1.62). That exposes a
coupling worth naming: ONE margin serves TWO gates with opposite sensitivity,
so shrinking it in the funding direction for the peak TIGHTENS the launch
worst-case gate. Not a defect of this change -- a relation that was inert while
the margin was large. The decision (its own named margin for the #1317n gate,
or a fresh measurement of LOAD_TRANSIENT_GIB) is the operator's and is stated
in the NACHTRAG.

Five of this branch's own earlier assertions are SUPERSEDED by (1) and (2) and
are updated in place with the reason, not deleted.

LOCAL: 263 passed / 4 failed; all four are the known pre-existing
test_weg2_ring_ledger_1235 reds at bfb5e12. No boot, no GPU, no /dev/shm;
xchg/weight_updater, S_D and the seed untouched.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 12, 2026
…ED emitter fields

(1) ORIGIN FLOOR. The front samples a group's run-moment residual at its FIRST
SLEEP, i.e. inside a flip, so that reading has already realised the permanent
step `flip_ratchet_gib` charges separately -- weg2xsn24's winning floor 7.49 GiB
was taken at 19:44:31Z, THREE SECONDS before that boot's `WEG2-FLIP done
epoch=2`. `run_origin_gib` now REJECTS an interleaved run-moment sample whenever
a launch-moment reading exists, prints the rejected figure with its reason
(`origin=6.45 source=launch ... 7.49 ... REJECTED: interleaved`), and does NOT
promote the dk7 stand-in in its place: that stand-in answers "no record at all",
not "the record was refused", and at 24.50 GiB it would be a WORSE origin than
the one just refused. W95 is narrowed, not removed -- it still refuses the sum
of both residual rules, and an origin with no launch reading still has none.

(2) RUN-PEAK RESIDUAL. On ratchet-priced arms the residual is the max of the
five sgl-project#1350 replays (xsn20 -1.513, xsn21b -0.170, xsn22 +0.375, xsn23 +0.908,
xsn24 +0.046 -> +0.908) plus its own measured steady-state drift
(RATCHET_RESIDUAL_DRIFT_MIB_PER_MIN = 0.07, sgl-project#1276 max) over the planned window,
both printed with their source. The pre-ratchet weg2sb4 5.16 row is NOT
hand-edited and still binds every arm that does not charge the ratchet; summing
the two remains W95. Boot margin 8.60 -> 1.47, hard bound 87.30 -> 94.43.

(3) #1350d NEED EMITTER, so xsn25 can be the ring-anchor measuring boot:
`leg=<epoch>/<src>-><dst>` on every WEG2-RING NEED line (`set_leg`), and on an
arm with no ring `need_mib=NOT-APPLICABLE saved_mib=<n>` instead of
`need_mib=0` -- measured on xsn24, where 32 of group P's 60 NEED lines read 0
because no ring was published, not because the tag had nothing to save. The
third field the anchor needs (one line per SAVED tag, including the
non-weights tags) sits in the caller that passes the tag list, not in
ring_guard, and is left OPEN and named in the order's NACHTRAG.

DRY RUN, xsn25 form, M=150: run_peak 90.43 GiB against the 94.43 bound -- THE
PEAK NOW FUNDS, with 4.00 GiB of room, exactly the predicted effect. rc=2 all
the same, and the binding term is a DIFFERENT one: `launch_sum = -6.33 GiB`
against the #1317n safety-net conjunct (`launch_worst_case >= -margin`), while
both moment leftovers are positive (launch +5.67, run +1.62). That exposes a
coupling worth naming: ONE margin serves TWO gates with opposite sensitivity,
so shrinking it in the funding direction for the peak TIGHTENS the launch
worst-case gate. Not a defect of this change -- a relation that was inert while
the margin was large. The decision (its own named margin for the #1317n gate,
or a fresh measurement of LOAD_TRANSIENT_GIB) is the operator's and is stated
in the NACHTRAG.

Five of this branch's own earlier assertions are SUPERSEDED by (1) and (2) and
are updated in place with the reason, not deleted.

LOCAL: 263 passed / 4 failed; all four are the known pre-existing
test_weg2_ring_ledger_1235 reds at bfb5e12. No boot, no GPU, no /dev/shm;
xchg/weight_updater, S_D and the seed untouched.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants