Skip to content

Organize public APIs - #809

Merged
hnyls2002 merged 2 commits into
mainfrom
organize-public-apis
Jul 29, 2024
Merged

hnyls2002 merged 2 commits into
mainfrom
organize-public-apis

Conversation

@hnyls2002

Copy link
Copy Markdown
Collaborator

Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.

Motivation

Organize public APIs which can be directly used by from sglang import xxx

Modification

  • Move core benchmarks into sglang/benchmarks
  • Adjust the benchmark scripts.

Checklist

  1. Ensure pre-commit pre-commit run --all-files or other linting tools are used to fix potential lint issues.
  2. Confirm that modifications are covered by complete unit tests. If not, please add more unit tests for correctness.
  3. Modify documentation as needed, such as docstrings or example tutorials.

@hnyls2002
hnyls2002 merged commit c8e9fed into main Jul 29, 2024
@hnyls2002
hnyls2002 deleted the organize-public-apis branch July 29, 2024 22:34
Ying1123 added a commit that referenced this pull request Jul 30, 2024
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 22, 2026
…the order does not matter

sgl-project#809 asks for a hybrid residency of the phase-flip weight images: a small
pinned share (a "hot head, in wave order"), the rest file-backed as today, and
a reload with prefetch overlap on the sgl-project#125 double-buffer pattern. The prior-art
gate falsified two of those three, and the third rests on a comparison nobody
has run. Recording that before writing code, because the cheapest outcome here
is the one where nothing is rebuilt.

THE OVERLAP IS ALREADY SHIPPED. `_staged_file_refill` (sgl-project#802, landed
2026-08-22, weights_arena.py:522) reads the file-backed image with bounded
preadv into sgl-project#720's pinned ring and overlaps the next chunk's read with the
previous chunk's H2D DMA -- depth streams, depth events, an inflight flag, and
a synchronize before a buffer is refilled. The depth is
SGLANG_PHASE_FLIP_REFILL_DEPTH, default 2: literally a double buffer.
Rebuilding it would be the sgl-project#720-verbatim mistake in a new place.

THERE IS NO HOT HEAD. `arena_refill` copies the whole payload and only then
checksums it; nothing reads the arena before that checksum passes. There is no
partial publication and no per-wave consumption, so WHICH bytes are pinned
cannot matter -- only HOW MANY. sgl-project#254's wave order exists because a MoE forward
consumes experts wave by wave; a flip consumes the arena once, whole. A "hot
head selection" would be a knob with no effect.

TWO OF THE THREE CITED NUMBERS ARE NOT IN THIS TREE. Grepped as rates over
python/sglang/srt/ and docs/dev/: zero hits for 2850, 4263, 1763 and 1844 as a
MiB/s, MB/s or GB/s figure, and the near-misses are named in the note so the
next reader does not "find" them by accident (1763 is a byte budget in an
affordability check and an NVML corridor floor; 1844 is a seam-staging free
memory delta). The pinned figure IS real: _PINNED_REF_LO_GBPS / _HI_GBPS =
4.93 / 8.88, phase_flip_boot.py:502-509. What the numbers that DO exist say:
pinned 4.93/7.08/8.88 GB/s per rank (sgl-project#690) against staged O_DIRECT 2651/2602/
3751 MiB/s per rank in the real flip on metal (sgl-project#802), i.e. roughly 1.8-2.6x --
but from two separate campaigns, never against each other, and the note says so
rather than presenting it as an A/B.

COMPRESSION IS DISCARDED, WITH THE ARITHMETIC, not deferred. This tree's own
ANALYSE_306 already ends "do not build sgl-project#306 as a codec", and sgl-project#456's sparse
write records a 0 byte win on /spinning because ZFS folds the same holes --
which is where the flip images live. Independently, at ratio 1.145 the bytes
saved are 12.7%, so a serial win needs decompression at 7.9x the read rate (64
GiB/s synthetic, 20-29 GiB/s at metal rates) and even a fully overlapped one
needs 9.29 GiB/s per rank, 10.07 GiB/s aggregate across three concurrent ranks.
The measured decompression ceiling in ANALYSE_306 is 4.3-4.8 GB/s. Both bars
are out of reach by 2-13x.

THE DANGER DIRECTION, AND THE ONE POSITION THIS DOES NOT REVERSE. The
file-backed arm exists so ~68.7 GiB of unreclaimable host RAM cannot OOM-kill a
swapless boot, and it refuses a missing or tmpfs image dir rather than
"silently allocating a pinned image the host ledger would then double-count as
reclaimable". The tree has ALREADY decided that image posts are registered but
not checked -- "a new refusal path here could break a boot that works today",
weights_arena.py:918-924. That decision stands. It does not bind a new,
opt-in, default-off pinned share, for its own stated reason: refusing a share
no current boot requests cannot break a boot that works today. So only the new
share may be checked, and a hybrid image must register exactly its pinned head
and nothing else, because the file-backed bytes are deliberately unregistered
(the registry sums NON-reclaimable bytes).

THE HAZARD ANY IMPLEMENTATION MUST HANDLE, recorded because it is silent:
_staged_file_refill takes O_DIRECT only when `at % 4096 == 0`. If reads resume
at an unaligned pinned-head boundary, EVERY chunk misses that test, the whole
refill falls back to the buffered fd -- 8304 -> 2595 MiB/s, a 3.2x regression
with no error and no log line -- and the pinned head costs more than it saves.
The head must be floored to _DIRECT_ALIGN.

NOT BUILT HERE, deliberately: the share's size is the whole feature, and the
comparison that would set it (pinned vs staged-O_DIRECT on ONE binary, same
load, same bytes) has never been run. A size derived from the two-campaign
mixture above would be the rig-fit the planner rules forbid, and shipping the
actuator with the size left as an unmeasured parameter would put a knob in the
tree nobody can set -- while a helper with no caller is exactly the defect
removed from planner/hicache_staging.py in this same branch. The A/B is named
as a window item, with the instrumentation that already exists for it
(_timed_arena_refill / refill_report).

Analysis only: no code changes, no test changes, no behaviour change.
codespell clean.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 24, 2026
The user asked why a tp_to_pp flip takes 11.6 s, and whether 16 waves over
116502 live slots could not just be moved in one parallel shot. The capture
answers the first half and refutes the premise of the second.

THE 116502 LIVE SLOTS AND THE 16 WAVES ARE NOT THE COST. W25's own seam
census names one segment:

    seam-census] timing tp_to_pp rank 0: 10466.8 ms across 448 segment(s),
      worst 'refill_highwater->weights_refill' 9516.2 ms (91% of the walk)
      kv_write->gdn_state 174.3 | cutover->done 152.4 |
      flip_writeback->hicache_quiesce 74.8 | kv_pack->kv_local_read 57.4

Three consecutive tp_to_pp flips: 9516.2 / 9496.5 / 12108.2 ms in that one
segment, against totals of 10466.8 / 10568.0 / 13181.2. Meanwhile the ENTIRE
KV movement at the largest observed reshard (epoch 11, 116502 live slots,
995.31 MiB sent / 825.03 MiB received) is read 19.9 + exchange 486.0 + write
394.8 = 901 ms, and the GDN blob is 11.69-18.70 MiB. Across the window live
slots grew 947x (123 -> 116502) while the flip grew 1.26x (5077.9 -> 6416.0
ms). The ranks are already parallel: totals agree within ~40 ms while movers
legs differ by ~600 ms, and the peers absorb PP0's overrun as CUTOVER WAIT
(1313.3 / 1220.8 ms vs PP0's 225.3). A rendezvous, exactly as sgl-project#690 recorded.

SO THE SEAM IS A WEIGHTS-REFILL RATE PROBLEM, and the rate is direction-
asymmetric for reasons nothing in the tree explains. Same rank, within 2.7%
of the same bytes:

    pp_to_tp   15925.8 MiB   3214-3915 MiB/s   4.07-4.96 s
    tp_to_pp   16362.7 MiB   1351-1723 MiB/s   9.50-12.11 s

FOUR CANDIDATE EXPLANATIONS WERE CHECKED AND ALL FOUR FAIL, and they are
recorded so nobody re-derives them:
* not a path fallback -- SGLANG_PHASE_FLIP_REFILL_STAGED defaults True
  (environ.py:348), unset in the boot, and arena_refill dispatches on the
  image being file-backed, not on direction. Both take _staged_file_refill.
* not a missing fd -- both sgl-project#802 warning paths ("could not open a read fd",
  "O_DIRECT unavailable") appear ZERO times in the 3.45 MB capture, against
  9 "flip host image FILE-BACKED" registrations.
* not the O_DIRECT alignment cliff (sgl-project#809's 8304 -> 2595 MiB/s) -- the loop
  only issues aligned offsets: chunks are 32 MiB multiples of
  _DIRECT_ALIGN=4096 and `want` is rounded down to it; only the trailing
  checksum tail is buffered, by design.
* not the pre-sgl-project#802 fault path -- on sgl-project#802's own discriminator the fault path
  makes rank rates CONVERGE (821/775 MiB/s on links differing 1.80x). W25
  DIVERGES with the link in both directions (1.59x and 1.40x).

THE INSTRUMENT IS WHAT IS MISSING, NOT THE MECHANISM. The leg reports ONE
aggregate MiB/s. The read (os.preadv) and the H2D (copy_ on a stream, depth
2) are pipelined, so that aggregate is min(read_rate, h2d_rate) with no way
to see which bound it hit. That is the sgl-project#851 class -- one number with several
meanings -- sitting inside the term that is 91% of the seam, which is why two
independent readers could not attribute the gap from code.

WHAT SHIPPED. `RefillLegTiming` accumulates, on the existing path:
  read_s      wall time inside preadv          -- storage/ARC bound
  h2d_wait_s  wall time blocked on a prior DMA -- PCIe/link bound
  drain_s     the pipeline tail, counted apart so it can never be mistaken
              for either bound
They are near-exclusive by construction: the ring only waits on a buffer
whose copy has not landed, so a read-bound leg never blocks there and a
link-bound leg blocks almost every turn.

`refill_bound_phrase` is a PURE function over that record -- no GPU -- so
both directions are falsifiable off metal, the same split sgl-project#852 used for the
allocator-cache estimator and for the same reason: a rule that can only be
exercised on metal is one this corpus has repeatedly shipped inert.

THE CAN-FAIL DIRECTION IS THE WHOLE RISK and is pinned. A phrase that always
named a bound would satisfy every "it says something" assertion while being
exactly as useless as the aggregate it replaces. So "unattributed" (not
instrumented, or no time accounted) and "MIXED" (neither half dominates) are
first-class outcomes, asserted directly, and a 101-point sweep across the
whole read-share range proves the three verdicts neither overlap nor leave a
hole.

NO BEHAVIOUR CHANGE: the timing record is optional and defaults to None, so
every caller that does not pass one runs byte-identically.

ONE PINNING TEST HAD TO MOVE, and it is worth naming because widening it
blindly was the tempting wrong answer. `test_arena_high_water_631.py`'s
`fake_arena_refill` stub replaces `arena_refill` to prove the arena is
committed before it is copied into; it did not accept the new kwarg and all
six of its refill tests failed with TypeError -- the sgl-project#624 stub-drift shape.
The stub now accepts `timing=None` (which is the whole adaptation, since the
record is an instrument with no semantics the test pins), AND it captures
what it was handed, with a new `test_the_refill_leg_is_instrumented`
asserting the caller really does pass a `RefillLegTiming`. Without that, the
signature widening would have silently tolerated the instrument being
unwired again -- a stub that accepts anything proves nothing.

TESTS (hermetic, CUDA_VISIBLE_DEVICES=""):
  test_refill_bound_856.py            9 passed, 101 subtests
  test/registered/unit/model_executor/  15 failed / 763 passed, against a
    VERIFIED HEAD baseline of 15 failed / 754 passed -- identical failure
    set, pre-existing, unrelated to this path (bf16 vendor fallback,
    coresidence budget, expert-offload kv regain). Checked by reverting this
    change and re-running, not asserted.
  ruff format clean; the one F401 in phase_flip_boot.py is pre-existing at
    HEAD (verified against git show HEAD:...).

Full ledger, with every figure and every failed search set:
docs/dev/NOTE_856_seam_cost_ledger.md
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 24, 2026
…/851-consolidated

W26 resolves the open root sgl-project#856(a) left named rather than guessed. My ledger
recorded the 2.5x direction gap as UNATTRIBUTED with the four candidates I had
ruled out and the search sets that failed; W26 measured it with the instrument
that commit shipped (RefillLegTiming / refill_bound_phrase) and the answer is
none of the four -- it is the storage read, and the gap is per-image POOL
SERVICE RATE, not a code path at all.

39 paired samples, 11 flip epochs, both directions, all three ranks, EVERY
sample STORAGE-BOUND: disk read 99.8-100% of the leg, H2D wait 3-7 ms, arena
commit ~0 ms, with the census segment and the refill timer agreeing to the ms.

Two things this settles for my line:

* READ PARALLELISM IS FALSIFIED (1.15x, flat from two readers). No async or
  overlap work belongs in this seam: a leg waiting on unread bytes cannot be
  overlapped away. That closes a lever I would otherwise have costed.
* The only remaining lever is sgl-project#809 section 8's PARTIAL pinned share. Whole-image
  pinning is not merely expensive, it is impossible here -- both W26 pin arms
  were OOM-killed during the launch phase, before any flip, which is itself the
  sizing constraint.

So my standing line -- that sgl-project#856's retirement projects ~10.5 s and must NOT be
scored as a seam-time fix -- now has a named successor rather than an open
question. W27 carries that constraint explicitly.

Doc-only merge; no code changes on either side of it.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 24, 2026
…all three falsifiers

First slice of the user's flip-image design: RAM holds ONE layout image plus a
small overshoot, and at the flip the incoming layout streams RAM -> VRAM while
the outgoing one streams VRAM -> RAM into the pages just freed. PCIe is full
duplex, so the copy-back rides the idle return direction.

THE COPY-BACK IS NOT WRITE-BACK. The weights are immutable and nothing is
saved; it is residency PLACEMENT for the next flip, which is what a
single-layout RAM budget requires. Written into the module docstring because a
later reader who mistakes it for a write-back will optimise it away and break
the following flip.

WHY THIS AND NOT THE PARTIAL PIN: W26 proved the dual pin impossible here --
both pin arms OOM-killed in the LAUNCH phase, before any flip. One layout plus
eps (~30 GiB vs ~68.7 GiB) fits AND takes the disk off the steady-state
critical path, which is what reaches the physics floor; a partial pin leaves a
disk share behind, and W26 measured the leg 99.8-100 % storage-bound.

THIS SLICE IS THE ARITHMETIC ONLY, deliberately. The overshoot sizing and the
interleaved schedule are pure functions over byte counts, so every invariant
the scheme rests on is falsifiable WITHOUT a GPU -- the same split sgl-project#852's
estimator and sgl-project#856(a)'s bound phrase use, and for the same reason.

OVERSHOOT = size asymmetry + in-flight window, sized from the LARGER
direction. The asymmetry is W26's measured one (PP0 15925.8/16362.7, PP1
8573.8/8961.3, PP2 8573.8/9481.6 MiB); a single fixed reservation has to cover
whichever direction the next flip takes, so a mean is the OOM. The in-flight
term is separate and pinned: an implementation returning only the asymmetry
gives 0 for equal layouts and stalls immediately.

ALL THREE NAMED FALSIFIERS ARE ASSERTED:
  * no actual overlap -- `rotation_totals` counts co-scheduled steps; a real
    rotation must have them and must have them as the DOMINANT shape (>90 %
    of steps), not as an accident of the tails. Its can-fail partner: a
    one-sided rotation must report zero overlap.
  * RAM leak across cycles -- three full A->B->A cycles must return host
    occupancy exactly to its start. Three, because W27-retry's leak fired on
    the THIRD cycle, not the first.
  * checksum -- verified against the real source: the image is
    `payload = image[:layout.total_bytes]` plus an int64 trailer, checked with
    `uint8_checksum(dst)` over the ARENA. That last part is what makes a D2H
    reproducible: bytes returned from VRAM verify exactly as bytes read from
    disk do, so only the 8-byte trailer is new.

A FINDING THE TESTS PRODUCED, and it is why the budget test first passed
vacuously: THE RAM BUDGET BINDS IN ONLY ONE DIRECTION. Pressure exists solely
when the OUTGOING layout is LARGER than the incoming one, because only then
does the copy-back need more RAM than the H2D frees -- PP0 copying back its
16362.7 MiB tp image while the smaller 15925.8 MiB pp image streams in leaves
436.9 MiB with nowhere to go. The opposite direction schedules cleanly at zero
overshoot. Both halves are now asserted so the asymmetry is recorded rather
than rediscovered.

Under-sizing STALLS LOUDLY rather than proceeding: a scheduler that kept going
would be holding both layouts, which is precisely the state that OOM-killed
W26's pin arms.

GATE (foreground, family-batched):
  managers core (45 PP files excluded)  3554 passed, 336 subtests, 0 failed
  model_executor                        777 passed / 15 failed (+14 new)
The 15 are the pre-existing sgl-project#815 family. ZERO new failures.

NOT YET BUILT, and not claimed: the device-side execution (streams, the pinned
ring registered once per sgl-project#720/sgl-project#729, the planner-priced host post per
sgl-project#721/sgl-project#770), and the separately-instrumented priming flip. This slice is the
plan those will execute.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 24, 2026
…the aliasing defect the arithmetic could not see

Slice 1 fixed the arithmetic. This is the executor that runs it, and building it
surfaced a defect in the plan that no byte-count model can express.

THE ARENA IS ONE BUFFER AND THE ROTATION IS AN IN-PLACE PERMUTATION.
`allocate_arena` returns a single contiguous device tensor sized max(pp, tp) and
`arena_refill` overwrites `arena[: layout.total_bytes]` in place
(weights_arena.py:1184,1192). Under a single-layout RAM budget the host image is
likewise ONE buffer. So at every chunk offset k the two directions are
CIRCULARLY dependent: the H2D wants to write arena[k], which the D2H has not
read yet, and the D2H wants to write image[k], which the H2D reads. Slice 1's
plan emits h2d_offset == d2h_offset while both are active, so a literal
execution of it aliases on every step. Serialising removes the duplex the scheme
exists for; running concurrently corrupts the image -- and it corrupts it in the
direction THIS flip's checksum cannot catch, because the damage lands in the
image the NEXT flip streams in.

THE RING IS THEREFORE LOAD-BEARING, not an optimisation, and this is the part a
later reader is most likely to undo. Per chunk: save image[k] into a ring slot,
D2H arena[k] -> image[k], then H2D ring slot -> arena[k] gated on that D2H.
Chunk k+1's D2H is enqueued before chunk k's H2D is waited on, so the lanes
genuinely run together with no aliasing anywhere in the pipeline. The one
host-to-host memcpy per chunk is the intrinsic cost of an in-place rotation, not
an accident of this implementation.

AND THE TWO SLICES THEN AGREE BY CONSTRUCTION rather than by coincidence: one
max-sized host buffer (one image PLUS the size asymmetry) plus depth*chunk of
ring IS `rotation_overshoot_bytes`. The same number reached twice, from two
directions.

OVERLAP IS MEASURED ON THE EXECUTOR, NOT THE PLAN. A step counts as overlapped
when, at the instant its D2H is enqueued, an earlier H2D has not been waited on.
Slice 1 could only count co-scheduled steps, which a serialised implementation
would also produce. Its can-fail partner is pinned: a ring of depth 1 must
report exactly zero overlap and must still be byte-correct.

PRIOR ART REUSED, NOT REBUILT. The ring is sgl-project#720's ReadBufferPool, which charges
the pinned-host registry BEFORE allocating (sgl-project#729), exactly as
`weights_arena._refill_staging_pool` already composes it; registered once per
process and reused by every flip. The readout is sgl-project#856(a)'s RefillLegTiming /
refill_bound_phrase -- no new telemetry. The launcher PRICES the ring with the
pure `joint_pinned_host_error` and does NOT register it: a planner-side
registration of bytes the launcher never pins is the helper commit 272d0d9
deleted, and reintroducing it was the obvious wrong move here.

THE DEFERRED WAVE-LOOP SUCCESSOR, FOLDED IN. W27-retry measured 16 empty waves
for ~314 ms. Skipping the loop was deferred because `finalize_wave` is what marks
the destination pool resident, and a bare skip leaves `backing_is_resident`
answering no. The replacement is the whole-pool swap ALREADY on the same object
(`WavedBackingSwap.__call__`): release source, reclaim, restore destination. It
is not merely equivalent, it is strictly cheaper -- waving exists to bound the
transient of holding a source layer live while its destination is written, and
with no bytes crossing there is nothing to bracket, while `__call__` releases
before it restores so its peak is max(src, dst). Gated on a new
`PhaseFlipTransition.moves_nothing` that checks ALL THREE legs: a predicate
reading only the peer exchange would skip a plan that still has a LOCAL move and
drop KV silently, which is the one outcome worse than the 314 ms.

R1's directional budget law is carried forward unregressed: pressure exists only
when the outgoing layout is the larger, and both halves stay pinned.

THE PRIMING FLIP IS INSTRUMENTED APART (P4), with its own stats and its own
timing record, so a steady-state mean can never absorb it.

WHAT IS DESK-PROVEN vs WHAT NEEDS THE WINDOW. Every test here executes the REAL
executor over REAL byte patterns on CPU tensors: byte-exactness in both
directions, the checksum, the absence of drift across three A->B->A cycles, and
the ring returning to full are EXECUTED, not modelled. Only the CUDA lane
mapping needs metal. A mutant that bypasses the ring save kills 13 of the tests,
so the suite detects the corruption it is written against.

GATES (foreground, family-batched, PYTHONPATH pinned to this worktree):
  model_executor           15 failed / 803 passed  (baseline 15 / 780) -> 0 new
  managers batch A (240 f) 2640 passed, 1 failed + 1 collect error, both
                           identical at HEAD
  managers batch B (63 PP files, one process each)  684 passed, 0 failed
  mem_cache (chunked)      31 failed / 2108 passed, byte-identical to HEAD;
                           one chunk segfaults at HEAD too (pre-existing)
  server_args              4 failed / 793 passed, the same 4 at HEAD
  new suites               48 passed (rotation executor 23, plan 14, wave 11)
ZERO new failures. ruff F/E7/E9 clean on new files; the two large touched files
are 358 -> 358, unchanged.

NOT CLAIMED: nothing here has run on a GPU. The boot-side change that allocates
ONE max-sized host image instead of two, and the wiring of the executor into
PhaseFlipStacks.refill, are the remaining steps before the proof window.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 24, 2026
…NE image

The wiring slice. Slice 2 built and proved the executor; this makes it the real
path and removes the two-image allocation it was built to replace.

BOOT ALLOCATES ONE HOST IMAGE, NOT TWO. `PhaseFlipStacks` carried `image_pp`
AND `image_tp` for the life of the process -- that is the dual pin, and W26
OOM-killed BOTH its arms in the LAUNCH phase, before any flip ran. It now
carries a single `rotation_image`, sized max(pp, tp) + trailer, plus
`image_holds` naming which layout is resting in it. `image_from_tensors` grew
an `out=` arm so the boot snapshots straight INTO that buffer instead of
beside it, and the PP image is released the moment the arena carries the PP
layout. The boot's two-image PEAK is unchanged (the PP snapshot necessarily
predates layout_tp, so the max cannot be known earlier); what changes is the
STEADY STATE, which is where the design's budget lives: one layout + overshoot.

`refill()` ROTATES. Both directions go through `rotate_arena`: the target
layout streams RAM -> VRAM while the outgoing one is placed back into the pages
it frees. The high-water commit stays exactly where it was and means MORE now
than it did -- the leg touches both layouts structurally rather than only via a
recovery arm.

ONE PATH, NOT TWO. The priming fill is the same call with `outgoing_bytes=0`:
nothing in the arena worth keeping, so the copy-back has zero length and it
degenerates to the plain contiguous H2D it always was, carrying `priming=True`
so it can never be averaged into a warm number (P4). A buffer holding the wrong
layout is NOT a fallback -- under a single-image budget there is no second
source to fall back to, so it is an invariant violation and it refuses. The
three-leg `moves_nothing` guard and the directional budget law are untouched.

WHAT THIS GIVES UP, stated because it is a real loss and not an oversight.
`arena_refill`'s `restore=(other_layout, other_image)` rewrote the ACTIVE
layout from its own separate image on a checksum mismatch, so an abort left
both layouts byte-exact. That arm NEEDS a second lifetime image, which is
precisely the pin W26 proved impossible here. With one buffer it cannot exist:
a mismatch now declares the arena undefined and refuses loudly instead of
silently serving it. `test_restore_never_rebuild_677` is rewritten to pin the
replacement rather than deleted -- its actual invariant (a flip may pay COPY
time, never BUILD time) is unchanged and now also covers the copy-back, which
is the new most-likely place to reach for a rebuild.

TEST CHURN, all of it contract-driven and none of it widened away:
  * test_restore_never_rebuild_677  -- restore arm -> refusal; new pin that the
    copy-back places the outgoing layout by COPY under the build fence.
  * test_arena_high_water_631       -- stub moved from `arena_refill` to
    `rotate_arena`; the high-water need is now max(incoming, outgoing)
    structurally instead of via `restore[0]`.
  * test_acceptance_emitters_758    -- same seam move; the sgl-project#758 emitter is
    still asserted to fire, which is what that file exists for.
  * test_phase_flip_boot (scheduler) -- one max-sized image; the arena is now
    PRIMED with PP, because the first refill copies those bytes back and the
    second streams them in, so an unprimed arena would round-trip garbage.

GATES (foreground, family-batched, PYTHONPATH pinned to this worktree):
  managers batch B (64 PP files, one process each)  697 passed, 0 failed
  model_executor        15 failed / 803 passed -- the same three sgl-project#815 files
  managers batch A      2641 passed; 1 failure + 1 collect error, both
                        identical at HEAD (+1 pass = the split restore test)
  mem_cache (chunked)   442 / 7f+474p / segfault / 24f+1192p -- byte-identical
                        to HEAD, segfaulting chunk segfaults at HEAD too
  scheduler test_phase_flip_boot                    31 passed
  new wiring suite                                  13 passed
ZERO new failures. ruff F/E7/E9 clean on every new and changed file;
phase_flip_boot 1 -> 1 (pre-existing unused `pack_into_arena`). server_args was
not re-run: nothing in this slice touches it, and its ledger post landed and
was gated in the previous commit.

DESK-PROVEN: the rotation is byte-exact through the REAL `PhaseFlipStacks.
refill` over CPU tensors, in both directions and across three full A->B->A
cycles, with the buffer and the arena returning to their exact starting content
each time; the wrong-layout refusal and the corrupt-image refusal both fire and
both have passing can-fail partners.

NOT CLAIMED: nothing here has run on a GPU. The W26-instrument re-measurement
(warm 1-2 s both directions, read_s ~0, the verdict leaving STORAGE-BOUND) is
what the proof window is for.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 24, 2026
…aHostUnregister before the pages go back

FOUND ON METAL, in the W28 proof window (specimen
/spinning/evidence-665-f1/SPECIMEN_w28_a1_hostregister712.log). The boot died
during HiCache's host KV allocation:

    cudaHostRegister failed (rc=712, part or all of the requested memory range
    is already mapped) for ptr=0x76a88d7d8000 size=5612142592

MY DEFECT, introduced by the wiring slice. `_alloc_host_image` page-locks
through `cudaHostRegister`. Until W28 every host image lived for the life of
the process, so nothing ever had to be given back and no release path existed.
The chunk rotation frees the PP boot image once the arena carries that layout
(`del image_pp`), which returned the pages to the process allocator while CUDA
still had the RANGE registered. The next large host allocation landed on it and
its own registration failed.

The registration is a process-wide fact about an ADDRESS RANGE, not a property
of the tensor. Freeing without unregistering leaves a landmine at an address
nobody owns any more, and the victim is whoever mallocs next -- which is why
the traceback pointed at HiCache, a component with nothing to do with the flip.

`release_host_image` undoes the allocation in the right order: cudaHostUnregister
first, then the caller drops the reference. It also returns the pinned-registry
post, which previously could only be given back on an allocation FAILURE
(`_LIVE_IMAGE_POSTS` tracks it per live image) -- correct while images were
lifetime objects, wrong once one is freed at boot.

WHY NO UNIT TEST PINS THIS: the failure needs a real CUDA host registration and
a subsequent large host allocation landing on the freed range. Both hermetic
halves (post accounting, call order) are cheap to assert and the ordering is
stated in the function; the RACE ITSELF is a metal property. Recorded here
rather than claimed as covered.

GATE: model_executor 15 failed / 803 passed -- the same three pre-existing sgl-project#815
files, ZERO new. scheduler test_phase_flip_boot 31 passed, rotation wiring 13
passed, restore_never_rebuild 8 passed. ruff F/E7/E9 unchanged (1 pre-existing
unused `pack_into_arena`); format clean.

PROVEN BY THE FIX: attempt 2 booted past this point, reached readiness, served a
real generation and executed a warm rotation. The window's remaining findings are
in /spinning/gpu-arb/W28-RESULT.md.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 24, 2026
…ath, and name where the leg's time goes

CLOSES A GAP I FILED AGAINST MYSELF. `_timed_arena_refill` discarded the
`RotationStats` it got back, so on metal `overlapped_steps` -- the duplex
falsifier this whole scheme rests on -- was NOT OBSERVABLE for a warm flip. W28
could therefore score it only as NO DATA. Both the stats line and a phase line
are now logged per flip leg.

THE SECOND HALF IS THE HONEST ONE. W28's leg reported

    LINK-BOUND (read 0.000s / h2d-wait 0.114s over 512 chunk(s), drain 0.001s)

on a 4.833 s rotation. Those terms are 2.4 % of it. The phrase named a bound
nobody had observed -- one number wearing several meanings, inside the dominant
term of the seam, which is the class sgl-project#851 exists to remove.

`RotationPhases` splits the leg into save / d2h-issue / h2d-issue / wait / ring
/ checksum / plan and, crucially, a NAMED `UNACCOUNTED` residual, so the
unexplained mass has to land somewhere instead of being distributed over the
parts that happen to be measured. `dominant()` lets the residual COMPETE with
the real phases: if the leftover is the biggest term, the instrument says it
does not know rather than crowning the largest thing it can see.
`phases_reconcile` refuses a leg leaving more than 10 % unexplained -- chosen,
not derived, and chosen so that W28's own 97.6 % could not have passed it.

HOST-SIDE PHASES ARE WHAT RECONCILE, and that follows from the measurement, not
from taste: `h2d_wait_s` was near zero, so the host was never blocked on the
device, so whatever consumed the wall clock was work the host thread did
itself. Device spans answer a different question and are deliberately excluded
from the sum -- adding a device span to a host total is how a reconciliation is
made to "pass" without meaning anything.

A CPU replay puts `d2h_issue` on top. That is a LEAD, not a finding, and it is
labelled as one: on CPU those copies are eager by construction.

DEFERRED BY USER DECISION (2026-08-24): ~4 s per flip is acceptable for now, so
the 4 s -> 1-2 s hunt is filed as W30 rather than chased here. What W30 still
needs is written into the ticket: CUDA-event device spans (staggered read after
the drain, never sync-in-loop) and the two can-fail mutants.

ALSO FIXED, and it was mine from the previous slice: the sgl-project#758 emitter test
replaced `rotation_executor.rotate_arena` and never restored it, so a green
file made the scheduler suite fail behind it. setUp/tearDown now restore it.
`Tuple` was used unimported (masked at runtime by `from __future__ import
annotations`, caught by ruff).

W29 IS TURNKEY, and the runnable path is the point of this slice:
  * /spinning/gpu-arb/w29_load.sh -- ALTERNATES concurrent large-prompt BURSTS
    with low-concurrency decode HOLDS. W28's driver was serial deep decode, which
    builds no prefill backlog at all, so the policy went TP-sticky after one
    flip; the user caught that from the outside. Bursts aim well above the bar.
  * /spinning/gpu-arb/boot_w29.sh -- the sgl-project#630 PP x disk-HiCache warmup wedge is
    AVOIDED with `--skip-server-warmup`, and the alternative is named and
    rejected: dropping `--hicache-storage-backend file` would delete the
    geometry-free disk STORE that is a prefix's only route across the flip
    (sgl-project#706), i.e. the premise of the no-KV design.
  * BREAK-EVEN REPRICED FROM THE MEASURED SEAM. N ~= 2188 tokens per second of
    seam (18614/8.50 = 2190 and 39500/18.06 = 2187 agree). The measured flip is
    6.410 s slowest rank, round trip 12.82 s -> N ~= 28,050, not ~39,500. The
    bar is ~29 % LOWER, so the policy should flip MORE readily. Derived by
    proportion from the two known pairs, said plainly so it is not mistaken for
    a first-principles repricing.

GATE (foreground, family-batched):
  model_executor    15 failed / 803 passed -- the same three pre-existing sgl-project#815
                    files, ZERO new
  managers (chunked) 3184 passed; 1 failure + 1 collect error, both identical
                    at HEAD
  mem_cache (chunked) 7f+474p / 24f+1192p -- byte-identical to HEAD
  scheduler test_phase_flip_boot 31 passed; rotation suites 95 passed combined
ruff F/E7/E9 clean on the changed files; format clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant