Repository navigation
fix: resolve the fp8 bug introduced by vLLM 0.5.5 - #1276
Merged
Merged
Conversation
ispobock
approved these changes
Aug 31, 2024
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 8, 2026
Boot weg2sb4 (e4f1b9f) ran 1,499.8 HICACHE-ROUND/s on group P and 303.1/s on group D for 34 minutes with an empty queue and a live front. Measured from that boot's own logs: P 677,214 lines / n=5,643,450 over 3,504 s, D 105,354 lines / n=877,950 over 3,628 s. ROOT, and it is not missing machinery. The sgl-project#547 blocking-poll ladder is fully built and correct (idle_sleeper.py: rungs 0/1/10 ms, 50 ms cap). It was never CONSTRUCTED: scheduler.py:1243 init_idle_sleeper arms only under --sleep-on-idle or SGLANG_IDLE_BLOCKING_POLL, and the weg-2 launcher sets neither, so `idle_sleeper` was None on all six ranks of both groups and the loop never blocked. At the ladder's 50 ms cap an idle origin ticks at ~20 Hz instead of ~1,500 Hz -- a 75x reduction. The RANK GATE is not the defect and is kept. Only the request origin owns the zmq sockets the ladder polls; ranks above it take a blocking chain receive / broadcast (request_receiver.py: "ranks 1..n-1 from point_to_point_pyobj") and are driven at the origin's cadence, which is why all three PP stages logged byte-identical round counts. Parking the origin parks the whole group: one wait point, not six, and no rank-local decision (RAENGE-NIE-UNEINS). Arming is scoped to the weg-2 form via SGLANG_WEG2_GROUP -- the launcher's own established discriminator -- so the stock default is untouched. Second, smaller allocator on the same path: the sgl-project#1028 ROUND CENSUS line was written every 25th round, ~60 lines/s/rank of "nothing changed", each one a format tuple + formatted string + LogRecord on the round path. The counter still counts every round; the LINE is now written on a state change, plus a 60 s heartbeat. Tests hermetic (CUDA_VISIBLE_DEVICES=""), 14 passed. Red-first proven: 3 of them fail against the unpatched tree (weg-2 arming, quiet rounds, state change). test_every_pass_still_does_its_work is the regression guard for the fix-1c / fix-2 per-pass semantics -- a queued control object still gets its lap within one pass; only the log line changed, never the work. Ruff parity exact: scheduler.py 106 -> 106, unified_radix_cache.py 1 -> 1. NOT BOOTED. Rounds/s at idle is an arithmetic consequence of the ladder cap, not a measurement; the next boot's WEG2-IDLE-CENSUS line is what settles it.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 8, 2026
…line
Boot weg2sb4 grew ~19.0 MiB/min of host anon across its six ranks while
completely idle, against an independently observed +21.3 MiB/min. Measured
per rank over two /proc samples 140 s apart: P PP0 +5.4, PP1 +5.4, PP2 +0.0,
each D rank +2.8 MiB/min; everything else (front, detokenizers, launcher
parents, inductor workers) exactly 0.0.
WHERE IT IS. 64 MiB-aligned [anon] rw-p glibc malloc arenas, extended in
place plus new arenas appearing (PP1 7d51f4000000 63.7->64.0, 7d51e8000000
19.6->20.4, new 7d51ec000000 0->11.43 MiB). It is NOT the host ring --
dShmem was 0.0 on every process -- NOT CUDA pinned memory (/dev/zero flat)
and NOT the store.
MALLOC_ARENA_MAX=4, chosen from this rig's measurement rather than folklore:
glibc defaults to 8 x ncores arenas and CPU affinity here is 0-31, so the
default ceiling is 256. The allocating threads on the steady-state path are
the scheduler thread plus the three the reset path names by hand ("#N RESET
JOIN threads=['prefetch', 'backup', 'prefetch_io_aux']"), so 4 gives each one
its own arena without the default's fragmentation surface. It is a CEILING,
published with setdefault so an operator experiment still wins.
malloc_trim runs at the idle point, BOUNDED: only when this rank actually
grew WEG2_TRIM_MIN_GROWTH_MIB since the last trim, because trim walks the
arena free lists and a flat rank must not pay for it.
WEG2-IDLE-CENSUS, per rank, once a minute while idle:
rounds_per_s=<n> rss_anon_mib=<n> d_anon_mib_per_min=<n> threads=<n>
arena_max=<s> trimmed=<n> group=<P|D> rank=<n>
from /proc/self/status only -- no smaps walk (these ranks carry ~3,300
mappings). It exists because NEITHER fact was in the log: the round rate had
to be reconstructed by counting sgl-project#1028 lines with an external script, and the
growth was only visible because a desk agent read /proc minutes before a host
OOM killed the processes. The next boot's acceptance can now gate on
rounds_per_s ~= 0 and d_anon_mib_per_min ~= 0 with no external tool.
`threads=` is in the line deliberately: the per-rank thread count was NOT
captured before the OOM, so the arena value is derived from the reset line's
thread names and the next boot confirms or corrects it.
EXPECTED EFFECT, stated as arithmetic and not as a measurement: the sgl-project#1276
ladder cap takes an idle origin from ~1,500 to ~20 rounds/s (75x), and the
growth is per-round allocator churn, so the arena bound plus the reduced
round rate should take +19.0 MiB/min toward ~0. UNPROVEN until a boot.
WHAT REMAINS UNEXPLAINED: in the same 140 s window cgroup anon rose +815 MiB
while the tracked processes explain only +44.3. That window was NOT idle --
a desk probe was reading the checkpoint inside it -- so the +815 is
confounded and must not be quoted as an sglang rate; the +19.0 MiB/min
per-process figure is the defensible one and it is the one that matches the
independent idle observation. The full-system attribution of the remainder
was never taken: the scan was killed by the OOM before its second sample.
Tests hermetic, 11 passed, red-first proven (the launcher publication test
fails against the unpatched tree). Ruff parity exact: launcher.py 0 -> 0,
scheduler.py 106 -> 106 (the census import is function-local, which also
keeps weg2 from reaching back into managers at module scope).
NOT BOOTED. No GPU used; cards were free and nothing was booked.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 8, 2026
…sgl-project#1269 malloc arenas + host watermark hard bound)
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 12, 2026
…ED emitter fields (1) ORIGIN FLOOR. The front samples a group's run-moment residual at its FIRST SLEEP, i.e. inside a flip, so that reading has already realised the permanent step `flip_ratchet_gib` charges separately -- weg2xsn24's winning floor 7.49 GiB was taken at 19:44:31Z, THREE SECONDS before that boot's `WEG2-FLIP done epoch=2`. `run_origin_gib` now REJECTS an interleaved run-moment sample whenever a launch-moment reading exists, prints the rejected figure with its reason (`origin=6.45 source=launch ... 7.49 ... REJECTED: interleaved`), and does NOT promote the dk7 stand-in in its place: that stand-in answers "no record at all", not "the record was refused", and at 24.50 GiB it would be a WORSE origin than the one just refused. W95 is narrowed, not removed -- it still refuses the sum of both residual rules, and an origin with no launch reading still has none. (2) RUN-PEAK RESIDUAL. On ratchet-priced arms the residual is the max of the five sgl-project#1350 replays (xsn20 -1.513, xsn21b -0.170, xsn22 +0.375, xsn23 +0.908, xsn24 +0.046 -> +0.908) plus its own measured steady-state drift (RATCHET_RESIDUAL_DRIFT_MIB_PER_MIN = 0.07, sgl-project#1276 max) over the planned window, both printed with their source. The pre-ratchet weg2sb4 5.16 row is NOT hand-edited and still binds every arm that does not charge the ratchet; summing the two remains W95. Boot margin 8.60 -> 1.47, hard bound 87.30 -> 94.43. (3) #1350d NEED EMITTER, so xsn25 can be the ring-anchor measuring boot: `leg=<epoch>/<src>-><dst>` on every WEG2-RING NEED line (`set_leg`), and on an arm with no ring `need_mib=NOT-APPLICABLE saved_mib=<n>` instead of `need_mib=0` -- measured on xsn24, where 32 of group P's 60 NEED lines read 0 because no ring was published, not because the tag had nothing to save. The third field the anchor needs (one line per SAVED tag, including the non-weights tags) sits in the caller that passes the tag list, not in ring_guard, and is left OPEN and named in the order's NACHTRAG. DRY RUN, xsn25 form, M=150: run_peak 90.43 GiB against the 94.43 bound -- THE PEAK NOW FUNDS, with 4.00 GiB of room, exactly the predicted effect. rc=2 all the same, and the binding term is a DIFFERENT one: `launch_sum = -6.33 GiB` against the #1317n safety-net conjunct (`launch_worst_case >= -margin`), while both moment leftovers are positive (launch +5.67, run +1.62). That exposes a coupling worth naming: ONE margin serves TWO gates with opposite sensitivity, so shrinking it in the funding direction for the peak TIGHTENS the launch worst-case gate. Not a defect of this change -- a relation that was inert while the margin was large. The decision (its own named margin for the #1317n gate, or a fresh measurement of LOAD_TRANSIENT_GIB) is the operator's and is stated in the NACHTRAG. Five of this branch's own earlier assertions are SUPERSEDED by (1) and (2) and are updated in place with the reason, not deleted. LOCAL: 263 passed / 4 failed; all four are the known pre-existing test_weg2_ring_ledger_1235 reds at bfb5e12. No boot, no GPU, no /dev/shm; xchg/weight_updater, S_D and the seed untouched.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 12, 2026
…ED emitter fields (1) ORIGIN FLOOR. The front samples a group's run-moment residual at its FIRST SLEEP, i.e. inside a flip, so that reading has already realised the permanent step `flip_ratchet_gib` charges separately -- weg2xsn24's winning floor 7.49 GiB was taken at 19:44:31Z, THREE SECONDS before that boot's `WEG2-FLIP done epoch=2`. `run_origin_gib` now REJECTS an interleaved run-moment sample whenever a launch-moment reading exists, prints the rejected figure with its reason (`origin=6.45 source=launch ... 7.49 ... REJECTED: interleaved`), and does NOT promote the dk7 stand-in in its place: that stand-in answers "no record at all", not "the record was refused", and at 24.50 GiB it would be a WORSE origin than the one just refused. W95 is narrowed, not removed -- it still refuses the sum of both residual rules, and an origin with no launch reading still has none. (2) RUN-PEAK RESIDUAL. On ratchet-priced arms the residual is the max of the five sgl-project#1350 replays (xsn20 -1.513, xsn21b -0.170, xsn22 +0.375, xsn23 +0.908, xsn24 +0.046 -> +0.908) plus its own measured steady-state drift (RATCHET_RESIDUAL_DRIFT_MIB_PER_MIN = 0.07, sgl-project#1276 max) over the planned window, both printed with their source. The pre-ratchet weg2sb4 5.16 row is NOT hand-edited and still binds every arm that does not charge the ratchet; summing the two remains W95. Boot margin 8.60 -> 1.47, hard bound 87.30 -> 94.43. (3) #1350d NEED EMITTER, so xsn25 can be the ring-anchor measuring boot: `leg=<epoch>/<src>-><dst>` on every WEG2-RING NEED line (`set_leg`), and on an arm with no ring `need_mib=NOT-APPLICABLE saved_mib=<n>` instead of `need_mib=0` -- measured on xsn24, where 32 of group P's 60 NEED lines read 0 because no ring was published, not because the tag had nothing to save. The third field the anchor needs (one line per SAVED tag, including the non-weights tags) sits in the caller that passes the tag list, not in ring_guard, and is left OPEN and named in the order's NACHTRAG. DRY RUN, xsn25 form, M=150: run_peak 90.43 GiB against the 94.43 bound -- THE PEAK NOW FUNDS, with 4.00 GiB of room, exactly the predicted effect. rc=2 all the same, and the binding term is a DIFFERENT one: `launch_sum = -6.33 GiB` against the #1317n safety-net conjunct (`launch_worst_case >= -margin`), while both moment leftovers are positive (launch +5.67, run +1.62). That exposes a coupling worth naming: ONE margin serves TWO gates with opposite sensitivity, so shrinking it in the funding direction for the peak TIGHTENS the launch worst-case gate. Not a defect of this change -- a relation that was inert while the margin was large. The decision (its own named margin for the #1317n gate, or a fresh measurement of LOAD_TRANSIENT_GIB) is the operator's and is stated in the NACHTRAG. Five of this branch's own earlier assertions are SUPERSEDED by (1) and (2) and are updated in place with the reason, not deleted. LOCAL: 263 passed / 4 failed; all four are the known pre-existing test_weg2_ring_ledger_1235 reds at bfb5e12. No boot, no GPU, no /dev/shm; xchg/weight_updater, S_D and the seed untouched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Modifications
Checklist