Repository navigation
Conversation
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 17, 2026
The window is now shut at all eight producers. WHY A DECORATOR AND NOT SIX TRY BLOCKS. All six remaining sites are __init__ bodies whose allocation runs to the END of the constructor, so wrapping each would mean re-indenting six constructors -- a large diff for a small property, on a train branch, in the KV-pool hot path. revert_pinned_posts_on_failure is added to pinned_host_budget (the same authority that owns unregister_pinned_post, which stays the only cleanup primitive) and applied as ONE LINE per site: MambaPoolHost, DeepSeekV4PagedHostPool, DeepSeekV4StateHostPool, DSAIndexerPoolHost (memory_pool_host.py) MHATokenToKOnlyPoolHost (pool_host/mha.py) HostKVCache (pool_host/base.py) On success the wrapper does nothing at all, so every success path is byte-identical. On failure it undoes exactly the posts that appeared during the call -- a set difference, so a post registered by anyone else is untouched -- and re-raises the ORIGINAL exception (sgl-project#386): cleanup never substitutes the diagnosis. NESTING IS CORRECT BY CONSTRUCTION, and it matters because HostKVCache is the BASE CLASS of the other five: a subclass __init__ that fails after super().__init__() registered undoes BOTH, which is right, because the object as a whole failed. A dedicated case pins exactly that. LIMIT STATED AT THE DEFINITION rather than discovered later: the set difference is not safe against a CONCURRENT registration from another thread inside the same extent. Every producer wrapped here is a boot-time constructor on one thread; a future off-thread producer must not use this. ONE TEST OVER THE SHAPE, not six copies, as asked. test_every_site_carries_the _guard subtests all six and asserts each __init__ is actually wrapped -- the half that catches a site quietly losing its decorator, and a seventh producer is one tuple away from being covered. Three behaviour cases drive the real decorator: a failing call reverts only what IT registered, a successful call keeps its post (so the guard cannot pass by never registering), and a nested failure undoes the super() call too. No site needed splitting out: all six are constructors with the identical shape. CAN-FAIL, three mutations, each redding only its own case: one site loses its decorator -> the wiring subtest (1 of 6) decorator stops unregistering -> the failing-call AND nested cases it undoes everything, not just new-> test_a_failing_call_reverts_only_what_it SUITES, zero new failures: unit/mem_cache 1148 -> 1152 passed, 0 failed unit/model_executor 15 failed / 660 passed (identical) unit/managers 22 failed / 2413 passed (identical) ruff: memory_pool_host.py is back to its 4 PRE-EXISTING E402s -- the added import carries an explicit noqa rather than making a fifth. FEATURE_CATALOG's "STILL OPEN, six sites" entry is replaced with the closed state and the mechanism, in the same step.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 22, 2026
…by hand --hicache-host-role staging (bfabd2d) refuses --hicache-ratio and requires an explicit --hicache-size. That refusal is only worth having if the number it demands comes from somewhere; a hand-picked --hicache-size is exactly the pool pin the planner's sole authority over the memory budget (sgl-project#584/sgl-project#785) exists to prevent. planner/hicache_staging.py derives it. Two consumers, and sizing for one of them silently breaks the other: * WRITE STAGING, a bandwidth-delay product: a page stays pinned from the moment it is handed to the storage backend until the write acks, so the resident set is drain rate x that residency, widened by a burst margin (write-through arrives in bursts, the drain is steady). * READ LANDING, which scales with CONCURRENCY, not bandwidth: a prefetch takes its destination slot from mem_pool_host.alloc() BEFORE the storage read is issued (hiradix_cache.py's _prefetch path). Sizing from the write side alone yields a tier that is correct and quietly serialises prefetch -- invisible to every capacity metric, visible only as latency. The tier takes the larger, rounded UP: rounding down would emit a tier smaller than the derivation justifying it. NO RIG FIT. Drain rate and latency are PARAMETERS, not constants. The planner probes rather than carrying a measured vector from one box to another, so nothing in this module encodes what this rig happens to do (~0.5 GB/s on its ZFS-backed file tier); that figure appears only as a test input. WHAT THIS DELIBERATELY REFUSES TO DO. If write-through produces bytes faster than the backend drains them, NO finite size holds -- residency is set by the drain, so a faster producer grows the in-flight set without bound. sustainable() answers that question so a caller can refuse instead of emitting a number that cannot be correct. The remedy there is backpressure at the producer, a runtime mechanism and not a size. Note sgl-project#720's ReadBufferPool made the OPPOSITE choice on purpose for reads -- on exhaustion it allocates a fresh uncounted buffer rather than blocking, trading "a bounded spike for an unbounded latency" -- which is right for a prefetch worker and wrong for a write path under a RAM budget, where re-inflating the pinned footprint is the failure the budget exists to prevent. That ring is the next posten, not this one. fits_pinned_host_budget() asks the sgl-project#729 authority and registers the post like every other pinned producer, so a tier that does not fit is refused where it is derived rather than discovered at allocation. Also: docs/dev/NOTE_810_dangling_design_706_c1.md records that DESIGN_706_BOOT cites a DESIGN_706_constraints document (C1/C1a) that exists nowhere in the tree. sgl-project#810's premise is quoted verbatim at the citing site and does not depend on it, but the supporting argument is unrecoverable. Recorded rather than reconstructed: rebuilding it from four citation fragments would read as primary evidence while being inference, and the next reader would cite it as the original. The tasked "add to DESIGN_706-C1" cannot be executed until the target exists. Tests: test/registered/unit/planner/test_hicache_staging_810.py, 19 new cases, hermetic (CUDA_VISIBLE_DEVICES=""). FIVE mutants, all dying on the intended tests: read consumer dropped (write-only sizing) -> 1 failed rounds down instead of up -> 1 failed zero-drain guard removed -> 2 failed burst-margin floor removed -> 1 failed sustainability predicate inverted -> 2 failed Source green again at 19 passed; 28 passed together with the role gate. Suite: test/registered/unit/planner/ is 2889 passed / 1 failed after this change. The single failure is test_rejected_evidence_pins.py's sgl-project#797 evidence assertion, confirmed PRE-EXISTING by re-running it on a clean checkout of dd6967c with these files removed -- same test, same assertion. No pre-existing test changed status. ruff/black/codespell clean. Not wired to a probe yet, and named so it is not assumed: nothing measures the drain rate automatically, so a staging profile still needs the number supplied. Measuring it needs load against a running server, which belongs to another strand's boot window right now.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 22, 2026
…edger THE PREFLIGHT COULD NOT SEE THE LARGEST PINNED HOST CONSUMER ON THIS RIG. The launcher-side joint pinned-host check (sgl-project#550/sgl-project#729) exists, but it is reached only inside the --enable-kv-session-offload x hierarchical-cache branch of ServerArgs validation, and only when the spill pool is non-zero. A boot without kv-session-offload prices no host tier at parse time at all. On top of that the default sizing is --hicache-ratio, which hicache_configured_host_bytes correctly declines to price before the device pool exists. So the standing boot's 22.01 GB of MHATokenToKVPoolHost across three PP ranks reached the preflight as nothing. Under --hicache-host-role staging both obstacles are gone: --hicache-size is mandatory and absolute, so the number is exact at parse time. Declaring it is what turns a smaller tier into headroom somebody has summed, rather than into a figure that first appears in a boot log after the RAM is already gone. MULTIPLIED BY THE RANKS, deliberately. --hicache-size sizes ONE rank's tier and every scheduler process allocates its own (sync_fixed_hicache_size syncs the token COUNT to the group minimum; each rank still allocates its own per-rank buffer for that count). The joint check is about this machine's RAM, which all ranks share, so the group product is the honest demand. NOTE for the next reader, and left uncorrected on purpose: the pre-existing check in the kv-session-offload branch passes a SINGLE tier's bytes and therefore under-counts a multi-rank boot by the rank factor -- three on this rig. Correcting it would change what a RETENTION boot accepts, and this role's contract is that retention stays byte-identical. It belongs to sgl-project#729/sgl-project#550. PRICED, NOT REGISTERED. joint_pinned_host_error is a pure function. The runtime registry in pinned_host_budget is PROCESS-LOCAL and the pools are allocated in the worker processes, where HostKVCache registers them already; a register_pinned_post here would create a post in the launcher for bytes the launcher never pins. The check degrades to no guard when pinned_host_memory_bytes() has no honest figure, per that module's own rule: refusing a boot on a fabricated number is worse than not checking. Default unchanged: _handle_hicache_host_role returns before any of this under retention, so no launch that does not name the role is priced, refused, or logged differently. The diff is purely additive. Tests: test/registered/unit/server_args/test_hicache_staging_host_ledger_810.py, 8 cases, hermetic (CUDA_VISIBLE_DEVICES=""), with the host-memory reading injected -- a guard whose verdict depends on how much RAM the box happens to have free is not a test of the guard. Both directions: a tier that fits is accepted and one that cannot is refused; the rank product is shown by a size that fits once and not three times, for PP and for TP separately; the spill pool is shown to be summed with it; the no-honest-number path is shown not to refuse; and the retention default is shown to price nothing even at a size that could not possibly fit. Mutants, 5, all confirmed dying: ledger call removed -> 4 failed no rank multiplication -> 2 failed spill pool left out of the sum -> 1 failed refusal removed -> 4 failed ranks counted from pp only -> 1 failed Source green again at 8 passed. 16b's 9 role cases stay green unchanged. SUITE DIFF FOR ALL THREE sgl-project#810 COMMITS, EMPTY: test/registered/unit/{mem_cache,distributed,server_args} on a clean detached checkout of 66cf0f6 is 103 failed / 5217 passed / 1663 skipped / 1632 subtests; with these three commits it is 103 failed / 5271 passed / 1663 skipped / 1632 subtests. 5271 - 5217 = 54 = exactly the 33 + 13 + 8 new cases. The 100 failure ids compare byte-identical between the two runs, so no pre-existing test changes status. ruff/black/codespell clean on every new file and on every hunk touched in the modified ones.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 22, 2026
…ost nothing allocates `fits_pinned_host_budget()` had no caller and no test anywhere in the tree -- not even in its own module's test file, which imports the other eight names and not this one. That alone would make it dead code. What makes it worth a commit rather than a deletion in passing is its SIDE EFFECT: it called `check_and_register_pinned_post`, i.e. it registered a sgl-project#729 post in the PLANNER process. THAT REGISTRY IS FOR PROCESSES THAT ALLOCATE. It exists to make the second allocation in a process see the first, and it credits earlier posts back against live availability -- `available + already_allocated` -- on the stated precondition that those posts' bytes are ALREADY RESIDENT and therefore already missing from `available` (pinned_host_budget.py, "the already-allocated posts must be credited back", measured 2026-08-17). A planner allocates nothing, so this post is exactly what the same comment goes on to name as "the real hazard": registered and then never allocated, hence credited back without ever having been resident, so the NEXT admission is charged too little and the registry waves through the over-commitment it exists to refuse. sgl-project#550 closed that window for the image posts with `_unregister_image_post`; the same comment adds "a future producer that can leave a post registered with no allocation behind it must do the same". This helper was that producer and did not. Unreached today, so nothing is broken today. It is removed rather than left because the whole cost of the defect is paid by whoever wires it -- and a helper named "fits the budget" is exactly what a future caller reaches for. NOT REPLACED BY A PURE VERSION, and the reason is that the question already has a better answer elsewhere. `ServerArgs._post_hicache_staging_host_ledger` prices the staging tier through `joint_pinned_host_error` without registering anything, jointly with the spill pool, and by the RANK PRODUCT -- every rank allocates its own tier, so the per-rank figure this helper used under-counts a multi-rank boot by the rank factor, three on this rig. Keeping a second, per-rank, planner-side reading of the same question would be two readings free to drift, which is the shape refused in the sibling commit for the file-tier watermark. The module docstring now says where the question is answered and what a planner-side pre-check would have to be (pure, rank-aware) if one is ever wanted. Tests: two cases added to test/registered/unit/planner/test_hicache_staging_810.py, hermetic (CUDA_VISIBLE_DEVICES=""). The property is pinned BEHAVIOURALLY rather than by absence: the full derivation path is exercised and `registered_posts()` is asserted empty afterwards, so a re-added registering helper fails even if it is spelled differently; the second case pins the removed name itself and points a future reader at the replacement. Mutants, 2, both confirmed dying: the removed helper is put back (verbatim pre-removal source) -> 1 failed the derivation itself registers a post -> 1 failed This is the red-first evidence for a deletion: the first mutant IS the old code, and the new guard fails on it. Suite diff empty: test/registered/unit/planner/ is 2889 passed / 1 failed before (16b's recorded figure, the failure being the pre-existing test_rejected_evidence_pins.py sgl-project#797 case) and 2891 passed / 1 failed after = 2889 + the 2 new cases, same single failure id. ruff/black/codespell clean.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 24, 2026
…all three falsifiers First slice of the user's flip-image design: RAM holds ONE layout image plus a small overshoot, and at the flip the incoming layout streams RAM -> VRAM while the outgoing one streams VRAM -> RAM into the pages just freed. PCIe is full duplex, so the copy-back rides the idle return direction. THE COPY-BACK IS NOT WRITE-BACK. The weights are immutable and nothing is saved; it is residency PLACEMENT for the next flip, which is what a single-layout RAM budget requires. Written into the module docstring because a later reader who mistakes it for a write-back will optimise it away and break the following flip. WHY THIS AND NOT THE PARTIAL PIN: W26 proved the dual pin impossible here -- both pin arms OOM-killed in the LAUNCH phase, before any flip. One layout plus eps (~30 GiB vs ~68.7 GiB) fits AND takes the disk off the steady-state critical path, which is what reaches the physics floor; a partial pin leaves a disk share behind, and W26 measured the leg 99.8-100 % storage-bound. THIS SLICE IS THE ARITHMETIC ONLY, deliberately. The overshoot sizing and the interleaved schedule are pure functions over byte counts, so every invariant the scheme rests on is falsifiable WITHOUT a GPU -- the same split sgl-project#852's estimator and sgl-project#856(a)'s bound phrase use, and for the same reason. OVERSHOOT = size asymmetry + in-flight window, sized from the LARGER direction. The asymmetry is W26's measured one (PP0 15925.8/16362.7, PP1 8573.8/8961.3, PP2 8573.8/9481.6 MiB); a single fixed reservation has to cover whichever direction the next flip takes, so a mean is the OOM. The in-flight term is separate and pinned: an implementation returning only the asymmetry gives 0 for equal layouts and stalls immediately. ALL THREE NAMED FALSIFIERS ARE ASSERTED: * no actual overlap -- `rotation_totals` counts co-scheduled steps; a real rotation must have them and must have them as the DOMINANT shape (>90 % of steps), not as an accident of the tails. Its can-fail partner: a one-sided rotation must report zero overlap. * RAM leak across cycles -- three full A->B->A cycles must return host occupancy exactly to its start. Three, because W27-retry's leak fired on the THIRD cycle, not the first. * checksum -- verified against the real source: the image is `payload = image[:layout.total_bytes]` plus an int64 trailer, checked with `uint8_checksum(dst)` over the ARENA. That last part is what makes a D2H reproducible: bytes returned from VRAM verify exactly as bytes read from disk do, so only the 8-byte trailer is new. A FINDING THE TESTS PRODUCED, and it is why the budget test first passed vacuously: THE RAM BUDGET BINDS IN ONLY ONE DIRECTION. Pressure exists solely when the OUTGOING layout is LARGER than the incoming one, because only then does the copy-back need more RAM than the H2D frees -- PP0 copying back its 16362.7 MiB tp image while the smaller 15925.8 MiB pp image streams in leaves 436.9 MiB with nowhere to go. The opposite direction schedules cleanly at zero overshoot. Both halves are now asserted so the asymmetry is recorded rather than rediscovered. Under-sizing STALLS LOUDLY rather than proceeding: a scheduler that kept going would be holding both layouts, which is precisely the state that OOM-killed W26's pin arms. GATE (foreground, family-batched): managers core (45 PP files excluded) 3554 passed, 336 subtests, 0 failed model_executor 777 passed / 15 failed (+14 new) The 15 are the pre-existing sgl-project#815 family. ZERO new failures. NOT YET BUILT, and not claimed: the device-side execution (streams, the pinned ring registered once per sgl-project#720/sgl-project#729, the planner-priced host post per sgl-project#721/sgl-project#770), and the separately-instrumented priming flip. This slice is the plan those will execute.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 24, 2026
…the aliasing defect the arithmetic could not see Slice 1 fixed the arithmetic. This is the executor that runs it, and building it surfaced a defect in the plan that no byte-count model can express. THE ARENA IS ONE BUFFER AND THE ROTATION IS AN IN-PLACE PERMUTATION. `allocate_arena` returns a single contiguous device tensor sized max(pp, tp) and `arena_refill` overwrites `arena[: layout.total_bytes]` in place (weights_arena.py:1184,1192). Under a single-layout RAM budget the host image is likewise ONE buffer. So at every chunk offset k the two directions are CIRCULARLY dependent: the H2D wants to write arena[k], which the D2H has not read yet, and the D2H wants to write image[k], which the H2D reads. Slice 1's plan emits h2d_offset == d2h_offset while both are active, so a literal execution of it aliases on every step. Serialising removes the duplex the scheme exists for; running concurrently corrupts the image -- and it corrupts it in the direction THIS flip's checksum cannot catch, because the damage lands in the image the NEXT flip streams in. THE RING IS THEREFORE LOAD-BEARING, not an optimisation, and this is the part a later reader is most likely to undo. Per chunk: save image[k] into a ring slot, D2H arena[k] -> image[k], then H2D ring slot -> arena[k] gated on that D2H. Chunk k+1's D2H is enqueued before chunk k's H2D is waited on, so the lanes genuinely run together with no aliasing anywhere in the pipeline. The one host-to-host memcpy per chunk is the intrinsic cost of an in-place rotation, not an accident of this implementation. AND THE TWO SLICES THEN AGREE BY CONSTRUCTION rather than by coincidence: one max-sized host buffer (one image PLUS the size asymmetry) plus depth*chunk of ring IS `rotation_overshoot_bytes`. The same number reached twice, from two directions. OVERLAP IS MEASURED ON THE EXECUTOR, NOT THE PLAN. A step counts as overlapped when, at the instant its D2H is enqueued, an earlier H2D has not been waited on. Slice 1 could only count co-scheduled steps, which a serialised implementation would also produce. Its can-fail partner is pinned: a ring of depth 1 must report exactly zero overlap and must still be byte-correct. PRIOR ART REUSED, NOT REBUILT. The ring is sgl-project#720's ReadBufferPool, which charges the pinned-host registry BEFORE allocating (sgl-project#729), exactly as `weights_arena._refill_staging_pool` already composes it; registered once per process and reused by every flip. The readout is sgl-project#856(a)'s RefillLegTiming / refill_bound_phrase -- no new telemetry. The launcher PRICES the ring with the pure `joint_pinned_host_error` and does NOT register it: a planner-side registration of bytes the launcher never pins is the helper commit 272d0d9 deleted, and reintroducing it was the obvious wrong move here. THE DEFERRED WAVE-LOOP SUCCESSOR, FOLDED IN. W27-retry measured 16 empty waves for ~314 ms. Skipping the loop was deferred because `finalize_wave` is what marks the destination pool resident, and a bare skip leaves `backing_is_resident` answering no. The replacement is the whole-pool swap ALREADY on the same object (`WavedBackingSwap.__call__`): release source, reclaim, restore destination. It is not merely equivalent, it is strictly cheaper -- waving exists to bound the transient of holding a source layer live while its destination is written, and with no bytes crossing there is nothing to bracket, while `__call__` releases before it restores so its peak is max(src, dst). Gated on a new `PhaseFlipTransition.moves_nothing` that checks ALL THREE legs: a predicate reading only the peer exchange would skip a plan that still has a LOCAL move and drop KV silently, which is the one outcome worse than the 314 ms. R1's directional budget law is carried forward unregressed: pressure exists only when the outgoing layout is the larger, and both halves stay pinned. THE PRIMING FLIP IS INSTRUMENTED APART (P4), with its own stats and its own timing record, so a steady-state mean can never absorb it. WHAT IS DESK-PROVEN vs WHAT NEEDS THE WINDOW. Every test here executes the REAL executor over REAL byte patterns on CPU tensors: byte-exactness in both directions, the checksum, the absence of drift across three A->B->A cycles, and the ring returning to full are EXECUTED, not modelled. Only the CUDA lane mapping needs metal. A mutant that bypasses the ring save kills 13 of the tests, so the suite detects the corruption it is written against. GATES (foreground, family-batched, PYTHONPATH pinned to this worktree): model_executor 15 failed / 803 passed (baseline 15 / 780) -> 0 new managers batch A (240 f) 2640 passed, 1 failed + 1 collect error, both identical at HEAD managers batch B (63 PP files, one process each) 684 passed, 0 failed mem_cache (chunked) 31 failed / 2108 passed, byte-identical to HEAD; one chunk segfaults at HEAD too (pre-existing) server_args 4 failed / 793 passed, the same 4 at HEAD new suites 48 passed (rotation executor 23, plan 14, wave 11) ZERO new failures. ruff F/E7/E9 clean on new files; the two large touched files are 358 -> 358, unchanged. NOT CLAIMED: nothing here has run on a GPU. The boot-side change that allocates ONE max-sized host image instead of two, and the wiring of the executor into PhaseFlipStacks.refill, are the remaining steps before the proof window.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.