Repository navigation
Organize public APIs - #809
Merged
Merged
Conversation
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 22, 2026
…the order does not matter sgl-project#809 asks for a hybrid residency of the phase-flip weight images: a small pinned share (a "hot head, in wave order"), the rest file-backed as today, and a reload with prefetch overlap on the sgl-project#125 double-buffer pattern. The prior-art gate falsified two of those three, and the third rests on a comparison nobody has run. Recording that before writing code, because the cheapest outcome here is the one where nothing is rebuilt. THE OVERLAP IS ALREADY SHIPPED. `_staged_file_refill` (sgl-project#802, landed 2026-08-22, weights_arena.py:522) reads the file-backed image with bounded preadv into sgl-project#720's pinned ring and overlaps the next chunk's read with the previous chunk's H2D DMA -- depth streams, depth events, an inflight flag, and a synchronize before a buffer is refilled. The depth is SGLANG_PHASE_FLIP_REFILL_DEPTH, default 2: literally a double buffer. Rebuilding it would be the sgl-project#720-verbatim mistake in a new place. THERE IS NO HOT HEAD. `arena_refill` copies the whole payload and only then checksums it; nothing reads the arena before that checksum passes. There is no partial publication and no per-wave consumption, so WHICH bytes are pinned cannot matter -- only HOW MANY. sgl-project#254's wave order exists because a MoE forward consumes experts wave by wave; a flip consumes the arena once, whole. A "hot head selection" would be a knob with no effect. TWO OF THE THREE CITED NUMBERS ARE NOT IN THIS TREE. Grepped as rates over python/sglang/srt/ and docs/dev/: zero hits for 2850, 4263, 1763 and 1844 as a MiB/s, MB/s or GB/s figure, and the near-misses are named in the note so the next reader does not "find" them by accident (1763 is a byte budget in an affordability check and an NVML corridor floor; 1844 is a seam-staging free memory delta). The pinned figure IS real: _PINNED_REF_LO_GBPS / _HI_GBPS = 4.93 / 8.88, phase_flip_boot.py:502-509. What the numbers that DO exist say: pinned 4.93/7.08/8.88 GB/s per rank (sgl-project#690) against staged O_DIRECT 2651/2602/ 3751 MiB/s per rank in the real flip on metal (sgl-project#802), i.e. roughly 1.8-2.6x -- but from two separate campaigns, never against each other, and the note says so rather than presenting it as an A/B. COMPRESSION IS DISCARDED, WITH THE ARITHMETIC, not deferred. This tree's own ANALYSE_306 already ends "do not build sgl-project#306 as a codec", and sgl-project#456's sparse write records a 0 byte win on /spinning because ZFS folds the same holes -- which is where the flip images live. Independently, at ratio 1.145 the bytes saved are 12.7%, so a serial win needs decompression at 7.9x the read rate (64 GiB/s synthetic, 20-29 GiB/s at metal rates) and even a fully overlapped one needs 9.29 GiB/s per rank, 10.07 GiB/s aggregate across three concurrent ranks. The measured decompression ceiling in ANALYSE_306 is 4.3-4.8 GB/s. Both bars are out of reach by 2-13x. THE DANGER DIRECTION, AND THE ONE POSITION THIS DOES NOT REVERSE. The file-backed arm exists so ~68.7 GiB of unreclaimable host RAM cannot OOM-kill a swapless boot, and it refuses a missing or tmpfs image dir rather than "silently allocating a pinned image the host ledger would then double-count as reclaimable". The tree has ALREADY decided that image posts are registered but not checked -- "a new refusal path here could break a boot that works today", weights_arena.py:918-924. That decision stands. It does not bind a new, opt-in, default-off pinned share, for its own stated reason: refusing a share no current boot requests cannot break a boot that works today. So only the new share may be checked, and a hybrid image must register exactly its pinned head and nothing else, because the file-backed bytes are deliberately unregistered (the registry sums NON-reclaimable bytes). THE HAZARD ANY IMPLEMENTATION MUST HANDLE, recorded because it is silent: _staged_file_refill takes O_DIRECT only when `at % 4096 == 0`. If reads resume at an unaligned pinned-head boundary, EVERY chunk misses that test, the whole refill falls back to the buffered fd -- 8304 -> 2595 MiB/s, a 3.2x regression with no error and no log line -- and the pinned head costs more than it saves. The head must be floored to _DIRECT_ALIGN. NOT BUILT HERE, deliberately: the share's size is the whole feature, and the comparison that would set it (pinned vs staged-O_DIRECT on ONE binary, same load, same bytes) has never been run. A size derived from the two-campaign mixture above would be the rig-fit the planner rules forbid, and shipping the actuator with the size left as an unmeasured parameter would put a knob in the tree nobody can set -- while a helper with no caller is exactly the defect removed from planner/hicache_staging.py in this same branch. The A/B is named as a window item, with the instrumentation that already exists for it (_timed_arena_refill / refill_report). Analysis only: no code changes, no test changes, no behaviour change. codespell clean.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 24, 2026
The user asked why a tp_to_pp flip takes 11.6 s, and whether 16 waves over
116502 live slots could not just be moved in one parallel shot. The capture
answers the first half and refutes the premise of the second.
THE 116502 LIVE SLOTS AND THE 16 WAVES ARE NOT THE COST. W25's own seam
census names one segment:
seam-census] timing tp_to_pp rank 0: 10466.8 ms across 448 segment(s),
worst 'refill_highwater->weights_refill' 9516.2 ms (91% of the walk)
kv_write->gdn_state 174.3 | cutover->done 152.4 |
flip_writeback->hicache_quiesce 74.8 | kv_pack->kv_local_read 57.4
Three consecutive tp_to_pp flips: 9516.2 / 9496.5 / 12108.2 ms in that one
segment, against totals of 10466.8 / 10568.0 / 13181.2. Meanwhile the ENTIRE
KV movement at the largest observed reshard (epoch 11, 116502 live slots,
995.31 MiB sent / 825.03 MiB received) is read 19.9 + exchange 486.0 + write
394.8 = 901 ms, and the GDN blob is 11.69-18.70 MiB. Across the window live
slots grew 947x (123 -> 116502) while the flip grew 1.26x (5077.9 -> 6416.0
ms). The ranks are already parallel: totals agree within ~40 ms while movers
legs differ by ~600 ms, and the peers absorb PP0's overrun as CUTOVER WAIT
(1313.3 / 1220.8 ms vs PP0's 225.3). A rendezvous, exactly as sgl-project#690 recorded.
SO THE SEAM IS A WEIGHTS-REFILL RATE PROBLEM, and the rate is direction-
asymmetric for reasons nothing in the tree explains. Same rank, within 2.7%
of the same bytes:
pp_to_tp 15925.8 MiB 3214-3915 MiB/s 4.07-4.96 s
tp_to_pp 16362.7 MiB 1351-1723 MiB/s 9.50-12.11 s
FOUR CANDIDATE EXPLANATIONS WERE CHECKED AND ALL FOUR FAIL, and they are
recorded so nobody re-derives them:
* not a path fallback -- SGLANG_PHASE_FLIP_REFILL_STAGED defaults True
(environ.py:348), unset in the boot, and arena_refill dispatches on the
image being file-backed, not on direction. Both take _staged_file_refill.
* not a missing fd -- both sgl-project#802 warning paths ("could not open a read fd",
"O_DIRECT unavailable") appear ZERO times in the 3.45 MB capture, against
9 "flip host image FILE-BACKED" registrations.
* not the O_DIRECT alignment cliff (sgl-project#809's 8304 -> 2595 MiB/s) -- the loop
only issues aligned offsets: chunks are 32 MiB multiples of
_DIRECT_ALIGN=4096 and `want` is rounded down to it; only the trailing
checksum tail is buffered, by design.
* not the pre-sgl-project#802 fault path -- on sgl-project#802's own discriminator the fault path
makes rank rates CONVERGE (821/775 MiB/s on links differing 1.80x). W25
DIVERGES with the link in both directions (1.59x and 1.40x).
THE INSTRUMENT IS WHAT IS MISSING, NOT THE MECHANISM. The leg reports ONE
aggregate MiB/s. The read (os.preadv) and the H2D (copy_ on a stream, depth
2) are pipelined, so that aggregate is min(read_rate, h2d_rate) with no way
to see which bound it hit. That is the sgl-project#851 class -- one number with several
meanings -- sitting inside the term that is 91% of the seam, which is why two
independent readers could not attribute the gap from code.
WHAT SHIPPED. `RefillLegTiming` accumulates, on the existing path:
read_s wall time inside preadv -- storage/ARC bound
h2d_wait_s wall time blocked on a prior DMA -- PCIe/link bound
drain_s the pipeline tail, counted apart so it can never be mistaken
for either bound
They are near-exclusive by construction: the ring only waits on a buffer
whose copy has not landed, so a read-bound leg never blocks there and a
link-bound leg blocks almost every turn.
`refill_bound_phrase` is a PURE function over that record -- no GPU -- so
both directions are falsifiable off metal, the same split sgl-project#852 used for the
allocator-cache estimator and for the same reason: a rule that can only be
exercised on metal is one this corpus has repeatedly shipped inert.
THE CAN-FAIL DIRECTION IS THE WHOLE RISK and is pinned. A phrase that always
named a bound would satisfy every "it says something" assertion while being
exactly as useless as the aggregate it replaces. So "unattributed" (not
instrumented, or no time accounted) and "MIXED" (neither half dominates) are
first-class outcomes, asserted directly, and a 101-point sweep across the
whole read-share range proves the three verdicts neither overlap nor leave a
hole.
NO BEHAVIOUR CHANGE: the timing record is optional and defaults to None, so
every caller that does not pass one runs byte-identically.
ONE PINNING TEST HAD TO MOVE, and it is worth naming because widening it
blindly was the tempting wrong answer. `test_arena_high_water_631.py`'s
`fake_arena_refill` stub replaces `arena_refill` to prove the arena is
committed before it is copied into; it did not accept the new kwarg and all
six of its refill tests failed with TypeError -- the sgl-project#624 stub-drift shape.
The stub now accepts `timing=None` (which is the whole adaptation, since the
record is an instrument with no semantics the test pins), AND it captures
what it was handed, with a new `test_the_refill_leg_is_instrumented`
asserting the caller really does pass a `RefillLegTiming`. Without that, the
signature widening would have silently tolerated the instrument being
unwired again -- a stub that accepts anything proves nothing.
TESTS (hermetic, CUDA_VISIBLE_DEVICES=""):
test_refill_bound_856.py 9 passed, 101 subtests
test/registered/unit/model_executor/ 15 failed / 763 passed, against a
VERIFIED HEAD baseline of 15 failed / 754 passed -- identical failure
set, pre-existing, unrelated to this path (bf16 vendor fallback,
coresidence budget, expert-offload kv regain). Checked by reverting this
change and re-running, not asserted.
ruff format clean; the one F401 in phase_flip_boot.py is pre-existing at
HEAD (verified against git show HEAD:...).
Full ledger, with every figure and every failed search set:
docs/dev/NOTE_856_seam_cost_ledger.md
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 24, 2026
…/851-consolidated W26 resolves the open root sgl-project#856(a) left named rather than guessed. My ledger recorded the 2.5x direction gap as UNATTRIBUTED with the four candidates I had ruled out and the search sets that failed; W26 measured it with the instrument that commit shipped (RefillLegTiming / refill_bound_phrase) and the answer is none of the four -- it is the storage read, and the gap is per-image POOL SERVICE RATE, not a code path at all. 39 paired samples, 11 flip epochs, both directions, all three ranks, EVERY sample STORAGE-BOUND: disk read 99.8-100% of the leg, H2D wait 3-7 ms, arena commit ~0 ms, with the census segment and the refill timer agreeing to the ms. Two things this settles for my line: * READ PARALLELISM IS FALSIFIED (1.15x, flat from two readers). No async or overlap work belongs in this seam: a leg waiting on unread bytes cannot be overlapped away. That closes a lever I would otherwise have costed. * The only remaining lever is sgl-project#809 section 8's PARTIAL pinned share. Whole-image pinning is not merely expensive, it is impossible here -- both W26 pin arms were OOM-killed during the launch phase, before any flip, which is itself the sizing constraint. So my standing line -- that sgl-project#856's retirement projects ~10.5 s and must NOT be scored as a seam-time fix -- now has a named successor rather than an open question. W27 carries that constraint explicitly. Doc-only merge; no code changes on either side of it.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 24, 2026
…all three falsifiers First slice of the user's flip-image design: RAM holds ONE layout image plus a small overshoot, and at the flip the incoming layout streams RAM -> VRAM while the outgoing one streams VRAM -> RAM into the pages just freed. PCIe is full duplex, so the copy-back rides the idle return direction. THE COPY-BACK IS NOT WRITE-BACK. The weights are immutable and nothing is saved; it is residency PLACEMENT for the next flip, which is what a single-layout RAM budget requires. Written into the module docstring because a later reader who mistakes it for a write-back will optimise it away and break the following flip. WHY THIS AND NOT THE PARTIAL PIN: W26 proved the dual pin impossible here -- both pin arms OOM-killed in the LAUNCH phase, before any flip. One layout plus eps (~30 GiB vs ~68.7 GiB) fits AND takes the disk off the steady-state critical path, which is what reaches the physics floor; a partial pin leaves a disk share behind, and W26 measured the leg 99.8-100 % storage-bound. THIS SLICE IS THE ARITHMETIC ONLY, deliberately. The overshoot sizing and the interleaved schedule are pure functions over byte counts, so every invariant the scheme rests on is falsifiable WITHOUT a GPU -- the same split sgl-project#852's estimator and sgl-project#856(a)'s bound phrase use, and for the same reason. OVERSHOOT = size asymmetry + in-flight window, sized from the LARGER direction. The asymmetry is W26's measured one (PP0 15925.8/16362.7, PP1 8573.8/8961.3, PP2 8573.8/9481.6 MiB); a single fixed reservation has to cover whichever direction the next flip takes, so a mean is the OOM. The in-flight term is separate and pinned: an implementation returning only the asymmetry gives 0 for equal layouts and stalls immediately. ALL THREE NAMED FALSIFIERS ARE ASSERTED: * no actual overlap -- `rotation_totals` counts co-scheduled steps; a real rotation must have them and must have them as the DOMINANT shape (>90 % of steps), not as an accident of the tails. Its can-fail partner: a one-sided rotation must report zero overlap. * RAM leak across cycles -- three full A->B->A cycles must return host occupancy exactly to its start. Three, because W27-retry's leak fired on the THIRD cycle, not the first. * checksum -- verified against the real source: the image is `payload = image[:layout.total_bytes]` plus an int64 trailer, checked with `uint8_checksum(dst)` over the ARENA. That last part is what makes a D2H reproducible: bytes returned from VRAM verify exactly as bytes read from disk do, so only the 8-byte trailer is new. A FINDING THE TESTS PRODUCED, and it is why the budget test first passed vacuously: THE RAM BUDGET BINDS IN ONLY ONE DIRECTION. Pressure exists solely when the OUTGOING layout is LARGER than the incoming one, because only then does the copy-back need more RAM than the H2D frees -- PP0 copying back its 16362.7 MiB tp image while the smaller 15925.8 MiB pp image streams in leaves 436.9 MiB with nowhere to go. The opposite direction schedules cleanly at zero overshoot. Both halves are now asserted so the asymmetry is recorded rather than rediscovered. Under-sizing STALLS LOUDLY rather than proceeding: a scheduler that kept going would be holding both layouts, which is precisely the state that OOM-killed W26's pin arms. GATE (foreground, family-batched): managers core (45 PP files excluded) 3554 passed, 336 subtests, 0 failed model_executor 777 passed / 15 failed (+14 new) The 15 are the pre-existing sgl-project#815 family. ZERO new failures. NOT YET BUILT, and not claimed: the device-side execution (streams, the pinned ring registered once per sgl-project#720/sgl-project#729, the planner-priced host post per sgl-project#721/sgl-project#770), and the separately-instrumented priming flip. This slice is the plan those will execute.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 24, 2026
…the aliasing defect the arithmetic could not see Slice 1 fixed the arithmetic. This is the executor that runs it, and building it surfaced a defect in the plan that no byte-count model can express. THE ARENA IS ONE BUFFER AND THE ROTATION IS AN IN-PLACE PERMUTATION. `allocate_arena` returns a single contiguous device tensor sized max(pp, tp) and `arena_refill` overwrites `arena[: layout.total_bytes]` in place (weights_arena.py:1184,1192). Under a single-layout RAM budget the host image is likewise ONE buffer. So at every chunk offset k the two directions are CIRCULARLY dependent: the H2D wants to write arena[k], which the D2H has not read yet, and the D2H wants to write image[k], which the H2D reads. Slice 1's plan emits h2d_offset == d2h_offset while both are active, so a literal execution of it aliases on every step. Serialising removes the duplex the scheme exists for; running concurrently corrupts the image -- and it corrupts it in the direction THIS flip's checksum cannot catch, because the damage lands in the image the NEXT flip streams in. THE RING IS THEREFORE LOAD-BEARING, not an optimisation, and this is the part a later reader is most likely to undo. Per chunk: save image[k] into a ring slot, D2H arena[k] -> image[k], then H2D ring slot -> arena[k] gated on that D2H. Chunk k+1's D2H is enqueued before chunk k's H2D is waited on, so the lanes genuinely run together with no aliasing anywhere in the pipeline. The one host-to-host memcpy per chunk is the intrinsic cost of an in-place rotation, not an accident of this implementation. AND THE TWO SLICES THEN AGREE BY CONSTRUCTION rather than by coincidence: one max-sized host buffer (one image PLUS the size asymmetry) plus depth*chunk of ring IS `rotation_overshoot_bytes`. The same number reached twice, from two directions. OVERLAP IS MEASURED ON THE EXECUTOR, NOT THE PLAN. A step counts as overlapped when, at the instant its D2H is enqueued, an earlier H2D has not been waited on. Slice 1 could only count co-scheduled steps, which a serialised implementation would also produce. Its can-fail partner is pinned: a ring of depth 1 must report exactly zero overlap and must still be byte-correct. PRIOR ART REUSED, NOT REBUILT. The ring is sgl-project#720's ReadBufferPool, which charges the pinned-host registry BEFORE allocating (sgl-project#729), exactly as `weights_arena._refill_staging_pool` already composes it; registered once per process and reused by every flip. The readout is sgl-project#856(a)'s RefillLegTiming / refill_bound_phrase -- no new telemetry. The launcher PRICES the ring with the pure `joint_pinned_host_error` and does NOT register it: a planner-side registration of bytes the launcher never pins is the helper commit 272d0d9 deleted, and reintroducing it was the obvious wrong move here. THE DEFERRED WAVE-LOOP SUCCESSOR, FOLDED IN. W27-retry measured 16 empty waves for ~314 ms. Skipping the loop was deferred because `finalize_wave` is what marks the destination pool resident, and a bare skip leaves `backing_is_resident` answering no. The replacement is the whole-pool swap ALREADY on the same object (`WavedBackingSwap.__call__`): release source, reclaim, restore destination. It is not merely equivalent, it is strictly cheaper -- waving exists to bound the transient of holding a source layer live while its destination is written, and with no bytes crossing there is nothing to bracket, while `__call__` releases before it restores so its peak is max(src, dst). Gated on a new `PhaseFlipTransition.moves_nothing` that checks ALL THREE legs: a predicate reading only the peer exchange would skip a plan that still has a LOCAL move and drop KV silently, which is the one outcome worse than the 314 ms. R1's directional budget law is carried forward unregressed: pressure exists only when the outgoing layout is the larger, and both halves stay pinned. THE PRIMING FLIP IS INSTRUMENTED APART (P4), with its own stats and its own timing record, so a steady-state mean can never absorb it. WHAT IS DESK-PROVEN vs WHAT NEEDS THE WINDOW. Every test here executes the REAL executor over REAL byte patterns on CPU tensors: byte-exactness in both directions, the checksum, the absence of drift across three A->B->A cycles, and the ring returning to full are EXECUTED, not modelled. Only the CUDA lane mapping needs metal. A mutant that bypasses the ring save kills 13 of the tests, so the suite detects the corruption it is written against. GATES (foreground, family-batched, PYTHONPATH pinned to this worktree): model_executor 15 failed / 803 passed (baseline 15 / 780) -> 0 new managers batch A (240 f) 2640 passed, 1 failed + 1 collect error, both identical at HEAD managers batch B (63 PP files, one process each) 684 passed, 0 failed mem_cache (chunked) 31 failed / 2108 passed, byte-identical to HEAD; one chunk segfaults at HEAD too (pre-existing) server_args 4 failed / 793 passed, the same 4 at HEAD new suites 48 passed (rotation executor 23, plan 14, wave 11) ZERO new failures. ruff F/E7/E9 clean on new files; the two large touched files are 358 -> 358, unchanged. NOT CLAIMED: nothing here has run on a GPU. The boot-side change that allocates ONE max-sized host image instead of two, and the wiring of the executor into PhaseFlipStacks.refill, are the remaining steps before the proof window.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 24, 2026
…NE image
The wiring slice. Slice 2 built and proved the executor; this makes it the real
path and removes the two-image allocation it was built to replace.
BOOT ALLOCATES ONE HOST IMAGE, NOT TWO. `PhaseFlipStacks` carried `image_pp`
AND `image_tp` for the life of the process -- that is the dual pin, and W26
OOM-killed BOTH its arms in the LAUNCH phase, before any flip ran. It now
carries a single `rotation_image`, sized max(pp, tp) + trailer, plus
`image_holds` naming which layout is resting in it. `image_from_tensors` grew
an `out=` arm so the boot snapshots straight INTO that buffer instead of
beside it, and the PP image is released the moment the arena carries the PP
layout. The boot's two-image PEAK is unchanged (the PP snapshot necessarily
predates layout_tp, so the max cannot be known earlier); what changes is the
STEADY STATE, which is where the design's budget lives: one layout + overshoot.
`refill()` ROTATES. Both directions go through `rotate_arena`: the target
layout streams RAM -> VRAM while the outgoing one is placed back into the pages
it frees. The high-water commit stays exactly where it was and means MORE now
than it did -- the leg touches both layouts structurally rather than only via a
recovery arm.
ONE PATH, NOT TWO. The priming fill is the same call with `outgoing_bytes=0`:
nothing in the arena worth keeping, so the copy-back has zero length and it
degenerates to the plain contiguous H2D it always was, carrying `priming=True`
so it can never be averaged into a warm number (P4). A buffer holding the wrong
layout is NOT a fallback -- under a single-image budget there is no second
source to fall back to, so it is an invariant violation and it refuses. The
three-leg `moves_nothing` guard and the directional budget law are untouched.
WHAT THIS GIVES UP, stated because it is a real loss and not an oversight.
`arena_refill`'s `restore=(other_layout, other_image)` rewrote the ACTIVE
layout from its own separate image on a checksum mismatch, so an abort left
both layouts byte-exact. That arm NEEDS a second lifetime image, which is
precisely the pin W26 proved impossible here. With one buffer it cannot exist:
a mismatch now declares the arena undefined and refuses loudly instead of
silently serving it. `test_restore_never_rebuild_677` is rewritten to pin the
replacement rather than deleted -- its actual invariant (a flip may pay COPY
time, never BUILD time) is unchanged and now also covers the copy-back, which
is the new most-likely place to reach for a rebuild.
TEST CHURN, all of it contract-driven and none of it widened away:
* test_restore_never_rebuild_677 -- restore arm -> refusal; new pin that the
copy-back places the outgoing layout by COPY under the build fence.
* test_arena_high_water_631 -- stub moved from `arena_refill` to
`rotate_arena`; the high-water need is now max(incoming, outgoing)
structurally instead of via `restore[0]`.
* test_acceptance_emitters_758 -- same seam move; the sgl-project#758 emitter is
still asserted to fire, which is what that file exists for.
* test_phase_flip_boot (scheduler) -- one max-sized image; the arena is now
PRIMED with PP, because the first refill copies those bytes back and the
second streams them in, so an unprimed arena would round-trip garbage.
GATES (foreground, family-batched, PYTHONPATH pinned to this worktree):
managers batch B (64 PP files, one process each) 697 passed, 0 failed
model_executor 15 failed / 803 passed -- the same three sgl-project#815 files
managers batch A 2641 passed; 1 failure + 1 collect error, both
identical at HEAD (+1 pass = the split restore test)
mem_cache (chunked) 442 / 7f+474p / segfault / 24f+1192p -- byte-identical
to HEAD, segfaulting chunk segfaults at HEAD too
scheduler test_phase_flip_boot 31 passed
new wiring suite 13 passed
ZERO new failures. ruff F/E7/E9 clean on every new and changed file;
phase_flip_boot 1 -> 1 (pre-existing unused `pack_into_arena`). server_args was
not re-run: nothing in this slice touches it, and its ledger post landed and
was gated in the previous commit.
DESK-PROVEN: the rotation is byte-exact through the REAL `PhaseFlipStacks.
refill` over CPU tensors, in both directions and across three full A->B->A
cycles, with the buffer and the arena returning to their exact starting content
each time; the wrong-layout refusal and the corrupt-image refusal both fire and
both have passing can-fail partners.
NOT CLAIMED: nothing here has run on a GPU. The W26-instrument re-measurement
(warm 1-2 s both directions, read_s ~0, the verdict leaving STORAGE-BOUND) is
what the proof window is for.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 24, 2026
…aHostUnregister before the pages go back
FOUND ON METAL, in the W28 proof window (specimen
/spinning/evidence-665-f1/SPECIMEN_w28_a1_hostregister712.log). The boot died
during HiCache's host KV allocation:
cudaHostRegister failed (rc=712, part or all of the requested memory range
is already mapped) for ptr=0x76a88d7d8000 size=5612142592
MY DEFECT, introduced by the wiring slice. `_alloc_host_image` page-locks
through `cudaHostRegister`. Until W28 every host image lived for the life of
the process, so nothing ever had to be given back and no release path existed.
The chunk rotation frees the PP boot image once the arena carries that layout
(`del image_pp`), which returned the pages to the process allocator while CUDA
still had the RANGE registered. The next large host allocation landed on it and
its own registration failed.
The registration is a process-wide fact about an ADDRESS RANGE, not a property
of the tensor. Freeing without unregistering leaves a landmine at an address
nobody owns any more, and the victim is whoever mallocs next -- which is why
the traceback pointed at HiCache, a component with nothing to do with the flip.
`release_host_image` undoes the allocation in the right order: cudaHostUnregister
first, then the caller drops the reference. It also returns the pinned-registry
post, which previously could only be given back on an allocation FAILURE
(`_LIVE_IMAGE_POSTS` tracks it per live image) -- correct while images were
lifetime objects, wrong once one is freed at boot.
WHY NO UNIT TEST PINS THIS: the failure needs a real CUDA host registration and
a subsequent large host allocation landing on the freed range. Both hermetic
halves (post accounting, call order) are cheap to assert and the ordering is
stated in the function; the RACE ITSELF is a metal property. Recorded here
rather than claimed as covered.
GATE: model_executor 15 failed / 803 passed -- the same three pre-existing sgl-project#815
files, ZERO new. scheduler test_phase_flip_boot 31 passed, rotation wiring 13
passed, restore_never_rebuild 8 passed. ruff F/E7/E9 unchanged (1 pre-existing
unused `pack_into_arena`); format clean.
PROVEN BY THE FIX: attempt 2 booted past this point, reached readiness, served a
real generation and executed a warm rotation. The window's remaining findings are
in /spinning/gpu-arb/W28-RESULT.md.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 24, 2026
…ath, and name where the leg's time goes
CLOSES A GAP I FILED AGAINST MYSELF. `_timed_arena_refill` discarded the
`RotationStats` it got back, so on metal `overlapped_steps` -- the duplex
falsifier this whole scheme rests on -- was NOT OBSERVABLE for a warm flip. W28
could therefore score it only as NO DATA. Both the stats line and a phase line
are now logged per flip leg.
THE SECOND HALF IS THE HONEST ONE. W28's leg reported
LINK-BOUND (read 0.000s / h2d-wait 0.114s over 512 chunk(s), drain 0.001s)
on a 4.833 s rotation. Those terms are 2.4 % of it. The phrase named a bound
nobody had observed -- one number wearing several meanings, inside the dominant
term of the seam, which is the class sgl-project#851 exists to remove.
`RotationPhases` splits the leg into save / d2h-issue / h2d-issue / wait / ring
/ checksum / plan and, crucially, a NAMED `UNACCOUNTED` residual, so the
unexplained mass has to land somewhere instead of being distributed over the
parts that happen to be measured. `dominant()` lets the residual COMPETE with
the real phases: if the leftover is the biggest term, the instrument says it
does not know rather than crowning the largest thing it can see.
`phases_reconcile` refuses a leg leaving more than 10 % unexplained -- chosen,
not derived, and chosen so that W28's own 97.6 % could not have passed it.
HOST-SIDE PHASES ARE WHAT RECONCILE, and that follows from the measurement, not
from taste: `h2d_wait_s` was near zero, so the host was never blocked on the
device, so whatever consumed the wall clock was work the host thread did
itself. Device spans answer a different question and are deliberately excluded
from the sum -- adding a device span to a host total is how a reconciliation is
made to "pass" without meaning anything.
A CPU replay puts `d2h_issue` on top. That is a LEAD, not a finding, and it is
labelled as one: on CPU those copies are eager by construction.
DEFERRED BY USER DECISION (2026-08-24): ~4 s per flip is acceptable for now, so
the 4 s -> 1-2 s hunt is filed as W30 rather than chased here. What W30 still
needs is written into the ticket: CUDA-event device spans (staggered read after
the drain, never sync-in-loop) and the two can-fail mutants.
ALSO FIXED, and it was mine from the previous slice: the sgl-project#758 emitter test
replaced `rotation_executor.rotate_arena` and never restored it, so a green
file made the scheduler suite fail behind it. setUp/tearDown now restore it.
`Tuple` was used unimported (masked at runtime by `from __future__ import
annotations`, caught by ruff).
W29 IS TURNKEY, and the runnable path is the point of this slice:
* /spinning/gpu-arb/w29_load.sh -- ALTERNATES concurrent large-prompt BURSTS
with low-concurrency decode HOLDS. W28's driver was serial deep decode, which
builds no prefill backlog at all, so the policy went TP-sticky after one
flip; the user caught that from the outside. Bursts aim well above the bar.
* /spinning/gpu-arb/boot_w29.sh -- the sgl-project#630 PP x disk-HiCache warmup wedge is
AVOIDED with `--skip-server-warmup`, and the alternative is named and
rejected: dropping `--hicache-storage-backend file` would delete the
geometry-free disk STORE that is a prefix's only route across the flip
(sgl-project#706), i.e. the premise of the no-KV design.
* BREAK-EVEN REPRICED FROM THE MEASURED SEAM. N ~= 2188 tokens per second of
seam (18614/8.50 = 2190 and 39500/18.06 = 2187 agree). The measured flip is
6.410 s slowest rank, round trip 12.82 s -> N ~= 28,050, not ~39,500. The
bar is ~29 % LOWER, so the policy should flip MORE readily. Derived by
proportion from the two known pairs, said plainly so it is not mistaken for
a first-principles repricing.
GATE (foreground, family-batched):
model_executor 15 failed / 803 passed -- the same three pre-existing sgl-project#815
files, ZERO new
managers (chunked) 3184 passed; 1 failure + 1 collect error, both identical
at HEAD
mem_cache (chunked) 7f+474p / 24f+1192p -- byte-identical to HEAD
scheduler test_phase_flip_boot 31 passed; rotation suites 95 passed combined
ruff F/E7/E9 clean on the changed files; format clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.
Motivation
Organize public APIs which can be directly used by
from sglang import xxxModification
Checklist
pre-commit run --all-filesor other linting tools are used to fix potential lint issues.