Skip to content

Fix bugs (fp8 checkpoints, triton cache manager) - #729

Merged
Ying1123 merged 7 commits into
mainfrom
fix-bugs
Jul 25, 2024
Merged

Ying1123 merged 7 commits into
mainfrom
fix-bugs

Conversation

@Ying1123

Copy link
Copy Markdown
Contributor

No description provided.

@Ying1123
Ying1123 merged commit 8fbba3d into main Jul 25, 2024
@Ying1123
Ying1123 deleted the fix-bugs branch July 25, 2024 14:42
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
efschu added a commit to efschu/htsglang that referenced this pull request Aug 17, 2026
The window is now shut at all eight producers.

WHY A DECORATOR AND NOT SIX TRY BLOCKS. All six remaining sites are __init__
bodies whose allocation runs to the END of the constructor, so wrapping each
would mean re-indenting six constructors -- a large diff for a small property,
on a train branch, in the KV-pool hot path. revert_pinned_posts_on_failure is
added to pinned_host_budget (the same authority that owns
unregister_pinned_post, which stays the only cleanup primitive) and applied as
ONE LINE per site:

  MambaPoolHost, DeepSeekV4PagedHostPool, DeepSeekV4StateHostPool,
  DSAIndexerPoolHost   (memory_pool_host.py)
  MHATokenToKOnlyPoolHost (pool_host/mha.py)
  HostKVCache             (pool_host/base.py)

On success the wrapper does nothing at all, so every success path is
byte-identical. On failure it undoes exactly the posts that appeared during the
call -- a set difference, so a post registered by anyone else is untouched --
and re-raises the ORIGINAL exception (sgl-project#386): cleanup never substitutes the
diagnosis.

NESTING IS CORRECT BY CONSTRUCTION, and it matters because HostKVCache is the
BASE CLASS of the other five: a subclass __init__ that fails after
super().__init__() registered undoes BOTH, which is right, because the object
as a whole failed. A dedicated case pins exactly that.

LIMIT STATED AT THE DEFINITION rather than discovered later: the set difference
is not safe against a CONCURRENT registration from another thread inside the
same extent. Every producer wrapped here is a boot-time constructor on one
thread; a future off-thread producer must not use this.

ONE TEST OVER THE SHAPE, not six copies, as asked. test_every_site_carries_the
_guard subtests all six and asserts each __init__ is actually wrapped -- the
half that catches a site quietly losing its decorator, and a seventh producer
is one tuple away from being covered. Three behaviour cases drive the real
decorator: a failing call reverts only what IT registered, a successful call
keeps its post (so the guard cannot pass by never registering), and a nested
failure undoes the super() call too. No site needed splitting out: all six are
constructors with the identical shape.

CAN-FAIL, three mutations, each redding only its own case:
  one site loses its decorator      -> the wiring subtest (1 of 6)
  decorator stops unregistering     -> the failing-call AND nested cases
  it undoes everything, not just new-> test_a_failing_call_reverts_only_what_it

SUITES, zero new failures:
  unit/mem_cache        1148 -> 1152 passed, 0 failed
  unit/model_executor   15 failed / 660 passed  (identical)
  unit/managers         22 failed / 2413 passed (identical)
ruff: memory_pool_host.py is back to its 4 PRE-EXISTING E402s -- the added
import carries an explicit noqa rather than making a fifth.

FEATURE_CATALOG's "STILL OPEN, six sites" entry is replaced with the closed
state and the mechanism, in the same step.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 22, 2026
…by hand

--hicache-host-role staging (bfabd2d) refuses --hicache-ratio and requires
an explicit --hicache-size. That refusal is only worth having if the number
it demands comes from somewhere; a hand-picked --hicache-size is exactly the
pool pin the planner's sole authority over the memory budget (sgl-project#584/sgl-project#785)
exists to prevent.

planner/hicache_staging.py derives it. Two consumers, and sizing for one of
them silently breaks the other:

  * WRITE STAGING, a bandwidth-delay product: a page stays pinned from the
    moment it is handed to the storage backend until the write acks, so the
    resident set is drain rate x that residency, widened by a burst margin
    (write-through arrives in bursts, the drain is steady).
  * READ LANDING, which scales with CONCURRENCY, not bandwidth: a prefetch
    takes its destination slot from mem_pool_host.alloc() BEFORE the storage
    read is issued (hiradix_cache.py's _prefetch path). Sizing from the write
    side alone yields a tier that is correct and quietly serialises prefetch
    -- invisible to every capacity metric, visible only as latency.

The tier takes the larger, rounded UP: rounding down would emit a tier
smaller than the derivation justifying it.

NO RIG FIT. Drain rate and latency are PARAMETERS, not constants. The planner
probes rather than carrying a measured vector from one box to another, so
nothing in this module encodes what this rig happens to do (~0.5 GB/s on its
ZFS-backed file tier); that figure appears only as a test input.

WHAT THIS DELIBERATELY REFUSES TO DO. If write-through produces bytes faster
than the backend drains them, NO finite size holds -- residency is set by the
drain, so a faster producer grows the in-flight set without bound. sustainable()
answers that question so a caller can refuse instead of emitting a number that
cannot be correct. The remedy there is backpressure at the producer, a runtime
mechanism and not a size. Note sgl-project#720's ReadBufferPool made the OPPOSITE choice
on purpose for reads -- on exhaustion it allocates a fresh uncounted buffer
rather than blocking, trading "a bounded spike for an unbounded latency" --
which is right for a prefetch worker and wrong for a write path under a RAM
budget, where re-inflating the pinned footprint is the failure the budget
exists to prevent. That ring is the next posten, not this one.

fits_pinned_host_budget() asks the sgl-project#729 authority and registers the post like
every other pinned producer, so a tier that does not fit is refused where it
is derived rather than discovered at allocation.

Also: docs/dev/NOTE_810_dangling_design_706_c1.md records that
DESIGN_706_BOOT cites a DESIGN_706_constraints document (C1/C1a) that exists
nowhere in the tree. sgl-project#810's premise is quoted verbatim at the citing site and
does not depend on it, but the supporting argument is unrecoverable. Recorded
rather than reconstructed: rebuilding it from four citation fragments would
read as primary evidence while being inference, and the next reader would
cite it as the original. The tasked "add to DESIGN_706-C1" cannot be executed
until the target exists.

Tests: test/registered/unit/planner/test_hicache_staging_810.py, 19 new cases,
hermetic (CUDA_VISIBLE_DEVICES=""). FIVE mutants, all dying on the intended
tests:
  read consumer dropped (write-only sizing) -> 1 failed
  rounds down instead of up                 -> 1 failed
  zero-drain guard removed                  -> 2 failed
  burst-margin floor removed                -> 1 failed
  sustainability predicate inverted         -> 2 failed
Source green again at 19 passed; 28 passed together with the role gate.

Suite: test/registered/unit/planner/ is 2889 passed / 1 failed after this
change. The single failure is test_rejected_evidence_pins.py's sgl-project#797 evidence
assertion, confirmed PRE-EXISTING by re-running it on a clean checkout of
dd6967c with these files removed -- same test, same assertion. No
pre-existing test changed status. ruff/black/codespell clean.

Not wired to a probe yet, and named so it is not assumed: nothing measures
the drain rate automatically, so a staging profile still needs the number
supplied. Measuring it needs load against a running server, which belongs to
another strand's boot window right now.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 22, 2026
…edger

THE PREFLIGHT COULD NOT SEE THE LARGEST PINNED HOST CONSUMER ON THIS RIG. The
launcher-side joint pinned-host check (sgl-project#550/sgl-project#729) exists, but it is reached
only inside the --enable-kv-session-offload x hierarchical-cache branch of
ServerArgs validation, and only when the spill pool is non-zero. A boot without
kv-session-offload prices no host tier at parse time at all. On top of that the
default sizing is --hicache-ratio, which hicache_configured_host_bytes
correctly declines to price before the device pool exists. So the standing
boot's 22.01 GB of MHATokenToKVPoolHost across three PP ranks reached the
preflight as nothing.

Under --hicache-host-role staging both obstacles are gone: --hicache-size is
mandatory and absolute, so the number is exact at parse time. Declaring it is
what turns a smaller tier into headroom somebody has summed, rather than into a
figure that first appears in a boot log after the RAM is already gone.

MULTIPLIED BY THE RANKS, deliberately. --hicache-size sizes ONE rank's tier and
every scheduler process allocates its own (sync_fixed_hicache_size syncs the
token COUNT to the group minimum; each rank still allocates its own per-rank
buffer for that count). The joint check is about this machine's RAM, which all
ranks share, so the group product is the honest demand. NOTE for the next
reader, and left uncorrected on purpose: the pre-existing check in the
kv-session-offload branch passes a SINGLE tier's bytes and therefore
under-counts a multi-rank boot by the rank factor -- three on this rig.
Correcting it would change what a RETENTION boot accepts, and this role's
contract is that retention stays byte-identical. It belongs to sgl-project#729/sgl-project#550.

PRICED, NOT REGISTERED. joint_pinned_host_error is a pure function. The runtime
registry in pinned_host_budget is PROCESS-LOCAL and the pools are allocated in
the worker processes, where HostKVCache registers them already; a
register_pinned_post here would create a post in the launcher for bytes the
launcher never pins.

The check degrades to no guard when pinned_host_memory_bytes() has no honest
figure, per that module's own rule: refusing a boot on a fabricated number is
worse than not checking.

Default unchanged: _handle_hicache_host_role returns before any of this under
retention, so no launch that does not name the role is priced, refused, or
logged differently. The diff is purely additive.

Tests: test/registered/unit/server_args/test_hicache_staging_host_ledger_810.py,
8 cases, hermetic (CUDA_VISIBLE_DEVICES=""), with the host-memory reading
injected -- a guard whose verdict depends on how much RAM the box happens to
have free is not a test of the guard. Both directions: a tier that fits is
accepted and one that cannot is refused; the rank product is shown by a size
that fits once and not three times, for PP and for TP separately; the spill
pool is shown to be summed with it; the no-honest-number path is shown not to
refuse; and the retention default is shown to price nothing even at a size that
could not possibly fit.

Mutants, 5, all confirmed dying:
  ledger call removed                -> 4 failed
  no rank multiplication             -> 2 failed
  spill pool left out of the sum     -> 1 failed
  refusal removed                    -> 4 failed
  ranks counted from pp only         -> 1 failed
Source green again at 8 passed. 16b's 9 role cases stay green unchanged.

SUITE DIFF FOR ALL THREE sgl-project#810 COMMITS, EMPTY:
test/registered/unit/{mem_cache,distributed,server_args} on a clean detached
checkout of 66cf0f6 is 103 failed / 5217 passed / 1663 skipped / 1632
subtests; with these three commits it is 103 failed / 5271 passed / 1663
skipped / 1632 subtests. 5271 - 5217 = 54 = exactly the 33 + 13 + 8 new cases.
The 100 failure ids compare byte-identical between the two runs, so no
pre-existing test changes status. ruff/black/codespell clean on every new file
and on every hunk touched in the modified ones.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 22, 2026
…ost nothing allocates

`fits_pinned_host_budget()` had no caller and no test anywhere in the tree --
not even in its own module's test file, which imports the other eight names and
not this one. That alone would make it dead code. What makes it worth a commit
rather than a deletion in passing is its SIDE EFFECT: it called
`check_and_register_pinned_post`, i.e. it registered a sgl-project#729 post in the PLANNER
process.

THAT REGISTRY IS FOR PROCESSES THAT ALLOCATE. It exists to make the second
allocation in a process see the first, and it credits earlier posts back
against live availability -- `available + already_allocated` -- on the stated
precondition that those posts' bytes are ALREADY RESIDENT and therefore already
missing from `available` (pinned_host_budget.py, "the already-allocated posts
must be credited back", measured 2026-08-17). A planner allocates nothing, so
this post is exactly what the same comment goes on to name as "the real
hazard": registered and then never allocated, hence credited back without ever
having been resident, so the NEXT admission is charged too little and the
registry waves through the over-commitment it exists to refuse. sgl-project#550 closed
that window for the image posts with `_unregister_image_post`; the same comment
adds "a future producer that can leave a post registered with no allocation
behind it must do the same". This helper was that producer and did not.

Unreached today, so nothing is broken today. It is removed rather than left
because the whole cost of the defect is paid by whoever wires it -- and a
helper named "fits the budget" is exactly what a future caller reaches for.

NOT REPLACED BY A PURE VERSION, and the reason is that the question already has
a better answer elsewhere. `ServerArgs._post_hicache_staging_host_ledger`
prices the staging tier through `joint_pinned_host_error` without registering
anything, jointly with the spill pool, and by the RANK PRODUCT -- every rank
allocates its own tier, so the per-rank figure this helper used under-counts a
multi-rank boot by the rank factor, three on this rig. Keeping a second,
per-rank, planner-side reading of the same question would be two readings free
to drift, which is the shape refused in the sibling commit for the file-tier
watermark. The module docstring now says where the question is answered and
what a planner-side pre-check would have to be (pure, rank-aware) if one is
ever wanted.

Tests: two cases added to
test/registered/unit/planner/test_hicache_staging_810.py, hermetic
(CUDA_VISIBLE_DEVICES=""). The property is pinned BEHAVIOURALLY rather than by
absence: the full derivation path is exercised and `registered_posts()` is
asserted empty afterwards, so a re-added registering helper fails even if it is
spelled differently; the second case pins the removed name itself and points a
future reader at the replacement.

Mutants, 2, both confirmed dying:
  the removed helper is put back (verbatim pre-removal source) -> 1 failed
  the derivation itself registers a post                       -> 1 failed
This is the red-first evidence for a deletion: the first mutant IS the old
code, and the new guard fails on it.

Suite diff empty: test/registered/unit/planner/ is 2889 passed / 1 failed
before (16b's recorded figure, the failure being the pre-existing
test_rejected_evidence_pins.py sgl-project#797 case) and 2891 passed / 1 failed after =
2889 + the 2 new cases, same single failure id. ruff/black/codespell clean.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 24, 2026
…all three falsifiers

First slice of the user's flip-image design: RAM holds ONE layout image plus a
small overshoot, and at the flip the incoming layout streams RAM -> VRAM while
the outgoing one streams VRAM -> RAM into the pages just freed. PCIe is full
duplex, so the copy-back rides the idle return direction.

THE COPY-BACK IS NOT WRITE-BACK. The weights are immutable and nothing is
saved; it is residency PLACEMENT for the next flip, which is what a
single-layout RAM budget requires. Written into the module docstring because a
later reader who mistakes it for a write-back will optimise it away and break
the following flip.

WHY THIS AND NOT THE PARTIAL PIN: W26 proved the dual pin impossible here --
both pin arms OOM-killed in the LAUNCH phase, before any flip. One layout plus
eps (~30 GiB vs ~68.7 GiB) fits AND takes the disk off the steady-state
critical path, which is what reaches the physics floor; a partial pin leaves a
disk share behind, and W26 measured the leg 99.8-100 % storage-bound.

THIS SLICE IS THE ARITHMETIC ONLY, deliberately. The overshoot sizing and the
interleaved schedule are pure functions over byte counts, so every invariant
the scheme rests on is falsifiable WITHOUT a GPU -- the same split sgl-project#852's
estimator and sgl-project#856(a)'s bound phrase use, and for the same reason.

OVERSHOOT = size asymmetry + in-flight window, sized from the LARGER
direction. The asymmetry is W26's measured one (PP0 15925.8/16362.7, PP1
8573.8/8961.3, PP2 8573.8/9481.6 MiB); a single fixed reservation has to cover
whichever direction the next flip takes, so a mean is the OOM. The in-flight
term is separate and pinned: an implementation returning only the asymmetry
gives 0 for equal layouts and stalls immediately.

ALL THREE NAMED FALSIFIERS ARE ASSERTED:
  * no actual overlap -- `rotation_totals` counts co-scheduled steps; a real
    rotation must have them and must have them as the DOMINANT shape (>90 %
    of steps), not as an accident of the tails. Its can-fail partner: a
    one-sided rotation must report zero overlap.
  * RAM leak across cycles -- three full A->B->A cycles must return host
    occupancy exactly to its start. Three, because W27-retry's leak fired on
    the THIRD cycle, not the first.
  * checksum -- verified against the real source: the image is
    `payload = image[:layout.total_bytes]` plus an int64 trailer, checked with
    `uint8_checksum(dst)` over the ARENA. That last part is what makes a D2H
    reproducible: bytes returned from VRAM verify exactly as bytes read from
    disk do, so only the 8-byte trailer is new.

A FINDING THE TESTS PRODUCED, and it is why the budget test first passed
vacuously: THE RAM BUDGET BINDS IN ONLY ONE DIRECTION. Pressure exists solely
when the OUTGOING layout is LARGER than the incoming one, because only then
does the copy-back need more RAM than the H2D frees -- PP0 copying back its
16362.7 MiB tp image while the smaller 15925.8 MiB pp image streams in leaves
436.9 MiB with nowhere to go. The opposite direction schedules cleanly at zero
overshoot. Both halves are now asserted so the asymmetry is recorded rather
than rediscovered.

Under-sizing STALLS LOUDLY rather than proceeding: a scheduler that kept going
would be holding both layouts, which is precisely the state that OOM-killed
W26's pin arms.

GATE (foreground, family-batched):
  managers core (45 PP files excluded)  3554 passed, 336 subtests, 0 failed
  model_executor                        777 passed / 15 failed (+14 new)
The 15 are the pre-existing sgl-project#815 family. ZERO new failures.

NOT YET BUILT, and not claimed: the device-side execution (streams, the pinned
ring registered once per sgl-project#720/sgl-project#729, the planner-priced host post per
sgl-project#721/sgl-project#770), and the separately-instrumented priming flip. This slice is the
plan those will execute.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 24, 2026
…the aliasing defect the arithmetic could not see

Slice 1 fixed the arithmetic. This is the executor that runs it, and building it
surfaced a defect in the plan that no byte-count model can express.

THE ARENA IS ONE BUFFER AND THE ROTATION IS AN IN-PLACE PERMUTATION.
`allocate_arena` returns a single contiguous device tensor sized max(pp, tp) and
`arena_refill` overwrites `arena[: layout.total_bytes]` in place
(weights_arena.py:1184,1192). Under a single-layout RAM budget the host image is
likewise ONE buffer. So at every chunk offset k the two directions are
CIRCULARLY dependent: the H2D wants to write arena[k], which the D2H has not
read yet, and the D2H wants to write image[k], which the H2D reads. Slice 1's
plan emits h2d_offset == d2h_offset while both are active, so a literal
execution of it aliases on every step. Serialising removes the duplex the scheme
exists for; running concurrently corrupts the image -- and it corrupts it in the
direction THIS flip's checksum cannot catch, because the damage lands in the
image the NEXT flip streams in.

THE RING IS THEREFORE LOAD-BEARING, not an optimisation, and this is the part a
later reader is most likely to undo. Per chunk: save image[k] into a ring slot,
D2H arena[k] -> image[k], then H2D ring slot -> arena[k] gated on that D2H.
Chunk k+1's D2H is enqueued before chunk k's H2D is waited on, so the lanes
genuinely run together with no aliasing anywhere in the pipeline. The one
host-to-host memcpy per chunk is the intrinsic cost of an in-place rotation, not
an accident of this implementation.

AND THE TWO SLICES THEN AGREE BY CONSTRUCTION rather than by coincidence: one
max-sized host buffer (one image PLUS the size asymmetry) plus depth*chunk of
ring IS `rotation_overshoot_bytes`. The same number reached twice, from two
directions.

OVERLAP IS MEASURED ON THE EXECUTOR, NOT THE PLAN. A step counts as overlapped
when, at the instant its D2H is enqueued, an earlier H2D has not been waited on.
Slice 1 could only count co-scheduled steps, which a serialised implementation
would also produce. Its can-fail partner is pinned: a ring of depth 1 must
report exactly zero overlap and must still be byte-correct.

PRIOR ART REUSED, NOT REBUILT. The ring is sgl-project#720's ReadBufferPool, which charges
the pinned-host registry BEFORE allocating (sgl-project#729), exactly as
`weights_arena._refill_staging_pool` already composes it; registered once per
process and reused by every flip. The readout is sgl-project#856(a)'s RefillLegTiming /
refill_bound_phrase -- no new telemetry. The launcher PRICES the ring with the
pure `joint_pinned_host_error` and does NOT register it: a planner-side
registration of bytes the launcher never pins is the helper commit 272d0d9
deleted, and reintroducing it was the obvious wrong move here.

THE DEFERRED WAVE-LOOP SUCCESSOR, FOLDED IN. W27-retry measured 16 empty waves
for ~314 ms. Skipping the loop was deferred because `finalize_wave` is what marks
the destination pool resident, and a bare skip leaves `backing_is_resident`
answering no. The replacement is the whole-pool swap ALREADY on the same object
(`WavedBackingSwap.__call__`): release source, reclaim, restore destination. It
is not merely equivalent, it is strictly cheaper -- waving exists to bound the
transient of holding a source layer live while its destination is written, and
with no bytes crossing there is nothing to bracket, while `__call__` releases
before it restores so its peak is max(src, dst). Gated on a new
`PhaseFlipTransition.moves_nothing` that checks ALL THREE legs: a predicate
reading only the peer exchange would skip a plan that still has a LOCAL move and
drop KV silently, which is the one outcome worse than the 314 ms.

R1's directional budget law is carried forward unregressed: pressure exists only
when the outgoing layout is the larger, and both halves stay pinned.

THE PRIMING FLIP IS INSTRUMENTED APART (P4), with its own stats and its own
timing record, so a steady-state mean can never absorb it.

WHAT IS DESK-PROVEN vs WHAT NEEDS THE WINDOW. Every test here executes the REAL
executor over REAL byte patterns on CPU tensors: byte-exactness in both
directions, the checksum, the absence of drift across three A->B->A cycles, and
the ring returning to full are EXECUTED, not modelled. Only the CUDA lane
mapping needs metal. A mutant that bypasses the ring save kills 13 of the tests,
so the suite detects the corruption it is written against.

GATES (foreground, family-batched, PYTHONPATH pinned to this worktree):
  model_executor           15 failed / 803 passed  (baseline 15 / 780) -> 0 new
  managers batch A (240 f) 2640 passed, 1 failed + 1 collect error, both
                           identical at HEAD
  managers batch B (63 PP files, one process each)  684 passed, 0 failed
  mem_cache (chunked)      31 failed / 2108 passed, byte-identical to HEAD;
                           one chunk segfaults at HEAD too (pre-existing)
  server_args              4 failed / 793 passed, the same 4 at HEAD
  new suites               48 passed (rotation executor 23, plan 14, wave 11)
ZERO new failures. ruff F/E7/E9 clean on new files; the two large touched files
are 358 -> 358, unchanged.

NOT CLAIMED: nothing here has run on a GPU. The boot-side change that allocates
ONE max-sized host image instead of two, and the wiring of the executor into
PhaseFlipStacks.refill, are the remaining steps before the proof window.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant