Repository navigation
Optimize radix tree matching - #364
Merged
Merged
Conversation
Collaborator
|
@ispobock This seems to be an interesting optimization. |
Collaborator
Author
|
@merrymercy I tested on mmlu dataset with different parallel, the accuracy is the same with the main branch. The total latency has slight improvement.
I also test on sharegpt dataset with 1000 prompts and found the throughput has slight improvement. with optimization: Test env:
Reproduce: |
Collaborator
Author
|
For 5000 prompts of sharegpt dataset benchmark, the throughput is improved 11.2% with this optimization. with optimization: |
Collaborator
|
Awesome! cc @hnyls2002 @Ying1123 |
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
shiyu7
pushed a commit
to shiyu7/sglang
that referenced
this pull request
Dec 26, 2025
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Jul 31, 2026
…mplete GDN state blob (incl. ReplaySSM rings + cursor), idle-only vacate planner (sgl-project#364 slice 1)
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Jul 31, 2026
…window guard, newcomers-as-demand fix, on-card bit-identical vacate/restore (sgl-project#364 slice 2)
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 1, 2026
…from the concurrency ceiling, admission sized from session budget (pre-cap profiled), over-admission interlock (sgl-project#364 slice 3)
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 1, 2026
…e machine with enforced sgl-project#364 quiesce boundary (TypeError without a QuiesceReport), DETACH_PENDING serves / ATTACH_PENDING refuses, MANUAL > ROUTED > CONTROLLER > BOOT with per-resolution source, tag routing from config with no built-in names; executor/endpoints deliberately unbuilt pending the card ticket (sgl-project#309 decision layer)
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
…and a five-cut list Desk analysis only, no build, as scoped. docs/dev/ANALYSE_553_elastic_coresidence.md. What the four named building blocks actually are, read from the code rather than from their titles: - sgl-project#330 VRAM dial: a REAL mechanism, and its decisive property for sgl-project#553 is that store_bound_rows is a LIFETIME bound baked into captured graphs. So the dial has a FREE BAND (shrink to any floor, re-grow to the boot bound) and a HARD WALL above it. The free band is exactly where CUDA-graph compatibility is preserved. - sgl-project#364 GDN slot ladder: a real, executing hot/cold mover, and graph-neutral by construction (it moves state between slots, not geometry). It is the existence proof that a hot/cold event need not touch capture, and its inventory seam -- the owner enumerates its live set, consumers never read a container directly -- is the pattern the other consumers need (sgl-project#551). - sgl-project#287 KV pressure ladder: PLANNER ONLY. Seventeen NotImplementedError sites; the docstring says so plainly. But the ordering invariants are already enforced in code and the relief rungs reference existing features BY NAME, so the relief half can be made to execute by delegation without new mechanism. - sgl-project#286 register: the right vocabulary, but exactly one ledger exists and it is wired to the translator. There is no generic tenant to raise an event against. The pattern across the event table: both directions are largely built, and what is missing is ARBITRATION. Two registries exist (dial participants, sgl-project#286 asset classes) and neither can see the other, so "free 4 GiB for the video tenant" has no addressee. Cut list, ordered so each cut is provable alone and no cut before the last touches capture: (1) one registry or an explicit bridge; (2) a generic tenant mover factored out of AudioAssetLedger; (3) make sgl-project#287's RELIEF rungs execute by delegation -- the largest block of stubs, and it never approaches the graph wall because relief rungs are declared "no KV layout change"; (4) the arbitration policy on top; (5) LAST and only with the rig, cross the graph wall via the per-rung pre-captured graphs sgl-project#287 already specifies, with cold rungs parked as register class graph_rungs. Graph compatibility is treated as the target state throughout, per directive: every cut states its effect on graph state, and cut 5 carries the explicit route back to full coverage. Until cut 5 lands, sgl-project#553 must state its band -- "elastic within the boot capture bound" -- rather than implying unlimited elasticity. Instrumentation named honestly: SGLANG_KVSO_TICK_TRACE=1 (throttled per-session regulator trace: effective interval, measured tick_cost, binding headroom ratio, host-tail size; pure logging, no control effect). The analysis states what it is NOT -- a regulator trace, not a per-phase breakdown of device time -- and proposes no measurement it has not checked exists. One decision deliberately left open and named as such: whether sgl-project#553 owns the arbitration policy or whether it belongs to sgl-project#363. That fork needs the sgl-project#363 stage feed bound first, since an arbitration policy fed by a table holding one stage would have nothing to choose between -- the same failure sgl-project#363/S8 documents one level up. codespell: clean.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 9, 2026
… and the route that does compose As specified -- reserves for requests 2-4 spill at runtime so a single session's usable KV grows -- the leg cannot be built from existing machinery: 1. sgl-project#364's own comment (model_runner_kv_cache_mixin.py:2136-2140) says a vacated GDN slot is reused by another session and never returned to the KV pool; the pool is fixed at boot. 2. kv_session_offload.py:1076-1085 states the by-design consequence that a single running session is always the oldest, hence untouchable, hence never a spill victim: a lone request over the device budget truncates rather than spilling. 3. phase_flip_runtime.py:585-612 guards the flip against kv-session- offload, so enabling it boots but permanently disarms every flip. What composes with --enable-phase-flip: --gdn-resident-state-slots N (boot-time KV gain, GDN state crosses the flip via gdn_flip_mover) and an explicit --kv-pressure-ladder relief vector without the session_offload rung. Combined with the measurement that the lone session is capped by context_length rather than by a reserve, the route to 'one session gets max KV' is the context ceiling plus a boot-time state-pool cap. The YaRN boot did not run: the prod_boot single-instance guard refused while the previous instance was still dying and the call budget expired inside that refusal. Serving is back on the proven config.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 9, 2026
…top the flip spending the VRAM reserve
Two defects found on metal while opening the long-context leg, both in the
composition of --gdn-resident-state-slots with --enable-phase-flip.
1. THE CAP AND THE FLIP COULD NOT BOOT TOGETHER.
--gdn-resident-state-slots 10 --max-mamba-cache-size 20 refused:
PhaseFlipBootError: req_to_token shapes diverge between stacks:
PP (5, 393224) vs TP (4, 393224)
Applying the cap OVERRIDES server_args.max_mamba_cache_size, while the
pre-cap profiled count -- which sgl-project#364 slice 3 sizes the concurrency
ceiling from -- was remembered on the MODEL RUNNER. phase_flip_boot
builds its TP stack from copy.deepcopy(server_args) taken AFTER that
override, so that stack had no runner attribute and read an already
capped pool: it sized its ceiling from 10 where the PP stack used 20
(ratio 3 -> 3 vs 4 requests). The crater slice 3 exists to prevent,
reintroduced through the copy.
The pre-cap count now lives on the ARGS, write-once, so the deepcopy
carries it. Underscore-prefixed, which is what ServerArgs.__setattr__
exempts from the strict post-resolution mutation guard.
2. A FLIP COULD KILL THE INSTANCE WHEN THE KV POOL WAS FULL.
One session of 276214 tokens (0.995 of the PP pool), then a routine
policy flip:
kv_reshard._exchange -> torch.empty(584 MiB)
-> torch.OutOfMemoryError ("600.38 MiB is free")
-> Fatal Python error: Aborted, all three ranks down
The flip is not free of memory: it packs its outgoing rows and
pre-allocates a receive buffer per peer, from the same device memory the
KV pool fills. Nothing checked that they fit. The free VRAM at the crash
was 600 MiB, already under the 1024 MiB corridor floor, so a flip that
had squeezed through would have broken the corridor anyway.
Affordability is now a second term in the EXISTING pre-flight verdict --
the one that already abandons a flip whose live set does not fit the
target pool. Same properties on purpose: computed from the plan before a
byte is allocated, reduced across ranks so the abandon is unanimous, and
it abandons the FLIP, never the server. Spendable memory is
cached_free + max(0, driver_free - reserve): allocator-cached bytes are
invisible to NVML and cannot move the corridor, driver bytes can. The
reserve comes from --rank-user-reserve-mib rather than a second opinion.
TESTS
bash scripts/run_631_flip_family.sh -> 559 passed (was 540).
New, both in the family list:
test/registered/unit/mem_cache/test_gdn_cap_flip_two_stacks_631.py
11 cases incl. the can-fail that reproduces PP 4 vs TP 3 exactly.
test/registered/unit/managers/test_phase_flip_staging_reserve_631.py
10 cases incl. the measured 584-vs-600 MiB refusal and the
falsifier for "just add driver-free and cache".
METAL
Defect 1: the boot that refused now completes and flips (PP->TP->PP)
during warmup; ctx 393216, PP pool 277468 tokens, max_running_requests
still 4. Defect 2's guard is code+tests, metal proof pending the next
full-occupancy run.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 13, 2026
…ever engaged on the flip path
Ordered mid-shift: prove that idle GDN/Mamba state (~147 MiB per slot,
~440 MiB for three idle slots) vacates into the KV pool and that the sizer
banks it as reclaimable mass.
IT DOES NOT FIRE. Zero occurrences of "vacat" and zero of "resident state
slots" across all three flip boots of this shift (boot_f, boot_g, boot_j,
/spinning/evidence-631/kvuniverse-r2). The gate is
if self.server_args.gdn_resident_state_slots is not None:
and --gdn-resident-state-slots appears in NEITHER the ship argv capture nor
the uncapped flip argv. Flag unset, gdn_slot_executor stays None, the
ladder never runs. The machinery (gdn_slot_runtime, gdn_slot_ladder,
gdn_slot_executor) is real and wired to the between-tick boundary; it is
simply switched off for this configuration. So "unused Mamba states are
spilled during bs1 time" is, here, unbacked -- the states sit as dead
reservation.
AND THE SIZER MUST NOT BANK THEM YET. The mamba reservation is its own
budget post subtracted before KV, and the seam reserve's measured anchor is
read with every slot reserved -- so those bytes are counted unavailable
twice and reclaimable never. Making the sizer spend them while the vacate
is off would size a pool against memory nothing releases. Engaging the
vacate comes first; counting it second.
Not fixed in this shift and nothing is claimed about the bs1 leg: no boot
with the flag, no vacated-byte count, no wake-on-slot-2 determined-answer
probe. Booked as register entry 57 with the numbers that make it a live
candidate for the budget re-solve: ~440 MiB parked against seam shortfalls
of 476 MiB (rank1) and 931 MiB (rank2) measured on boot G.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 13, 2026
…he wall is the layout Adding --gdn-resident-state-slots 4 to the flip argv makes the cap fire on all three ranks and banks measurable memory, but the RUNTIME vacate still cannot fire: kvso, the ladder's only standing population, is refused at parse time on pp_size>1 and the flip instance is PP=3 by construction. Slice 1 (boot-time pool cap) delivers the bytes without it; slice 3 (admission above the cap) depends on it and is unbacked here. Credit measured at an identical pinned pool (563974) against boot J: have_m 3744->4306 / 1224->1574 / 1514->1822 MiB. That is a quarter of the estimate carried in, because the cap correctly does not shrink intermediate_ssm_state_cache, which is sized from admission. The budget re-solve the previous shift specified cannot work: have_bytes is physical free VRAM, and the binding rank's pool is stopped by its seam floor rather than by its ceiling, so raising the ceiling frees nothing. The pool falls out of physical arithmetic on rank2 -- 2846 free, 1455 floor, 1024 law, 367 MiB spendable, 46976 tokens -- and the sizer derived 610942 against that hand figure of 610950. The lever is rank2's 1455 MiB arena tail, a function of the PP stage ratio against the TP vector. T4 is answered without a boot: the spilled-session-across-a-PP-flip composition is refused before the server starts, which is the safest of the three possible verdicts. Test results: - 77 passed (test_phase_flip_seam_reserve, test_gdn_slot_executor, test_gdn_ladder_decouple, test_gdn_cap_flip_two_stacks_631, test_gdn_slot_ladder, test_yarn_rope_cache_growth). - Boot K1 (cap engaged, pool pinned 563974): 30 completed cutovers (15/15), 0 abandons, 0 refusals, 0 tracebacks, 64001-token prefill, corridor continuous minimum under load 2082/3341/2128 MiB, 0 breaches. - Boot K3 (derived pool 610942, no operator number): 30 completed cutovers (15/15), 0 abandons, 0 refusals, 0 tracebacks, 64001-token prefill, corridor 1510/2715/1994 MiB, 0 breaches. 610942 < 620000, so the quarantine stays. - Serving restored on 30030 and verified with health 200 plus three real generations. Evidence: /spinning/evidence-631/kvuniverse-r3/
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 13, 2026
…ool, seam reserve, YaRN 1M) Merge source origin/feat/kv-universe-656 @ 4157aad into the R6 tip 481411a. Conflicts: 0. 28 files, 4684 insertions, 37 deletions. Carries: derived KV pool with the 620000-token quarantine constant DELETED and replaced by phase_flip_seam_reserve (solver reserves each rank's seam staging bytes on top of the user corridor, ENV_MARGIN_MIB default 192 MiB, seam_reserve_enabled() defaults True); phase-policy control-loop fix (refusal outcomes reach the policy, bounded stand-down instead of silent re-arm at the dwell interval); sgl-project#364 idle-vacate engagement; YaRN 1M ceiling plus the RoPE growth-path correctness fix; seam fingerprint widened with gdn_resident_state_slots / enable_kv_session_offload / pp_stage_ratio / phase_flip_tp_vector (keyed only when non-default); cgroup-escape boot fix in scripts/s33_boot_from_capture.sh (transient systemd-run --scope); and the lazy RoPE reserve, BUILT BUT DEFAULT OFF -- falsified on metal (buys 240 MiB per rank of written bytes but zero pool, and breaches the 1024 MiB corridor under deep prefill). SGLANG_ROPE_LAZY_CACHE = EnvBool(False) verified after the merge, both at the definition site and at runtime (unset, resolves False). TEST RESULTS -- CPU-only desk protocol, CUDA_VISIBLE_DEVICES=99, PYTHONPATH=<worktree>/python, pytest --color=no, one directory per invocation. Baseline is this worktree at 481411a before the merge; its counts are identical to MERGE-R6's post-step-3 column, so the chain back to the frozen /spinning/wt-merge-r6-base @ 598f570 is continuous. | suite | BASE 481411a | after this merge | |----------------------|-------------------|-------------------| | sgl-project#631 flip family | 1116 passed | 1116 passed | | unit/managers | 9F 1357P 18S | 9F 1357P 18S | | unit/mem_ledger | 1F 359P | 1F 359P | | unit/model_executor | 15F 594P | 15F 594P | | unit/server_args | 615P | 615P | | unit/turnkey | 116P | 116P | | unit/utils | 46F 348P 4S | 46F 348P 4S | | unit/docker | 4P | 4P | | new-656 arm | (files absent) | 4F 56P | Every failure count identical to baseline. The branch's six new test files are not named in the explicit run_631_flip_family.sh list, so they were run as a separate arm rather than assumed covered. Its 4 failures are INHERITED, not introduced: the same arm run against the source worktree /spinning/wt-kv-universe gives the identical 4 failed / 56 passed. All four are test/srt/test_phase_flip_serving_proof_gate.py raising "No accelerator (CUDA, XPU, HPU, NPU, MUSA, MPS) or platform plugin is available" -- the file needs a visible device and cannot run under the CPU-only merge protocol.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 17, 2026
…authorities only WIRED VS PRICED-ONLY, answered by grep before building: nothing imported coresidency_registry outside its own module, so a tenant COLD event did NOT actuate -- the bridge priced and no one called it. Same for HOT. That is the gap Cut 3 closes. managers/coresidency_policy.py is the caller and introduces NO NEW ACTUATOR. Every move is an existing authority's own API: vram_dial.apply_budget_request (replicated grow/shrink, already enforces the floor, "rejections carry the exact floor arithmetic and change nothing"), vram_dial. verify_pool_reached_capacity (the read-back), GdnSlotRuntime.unbind (sgl-project#364), and the Cut 1/2 bridge. A source pin forbids this module from touching torch.cuda / cuMem / set_active_prefix / runtime_set_ at all. THE TWO DIRECTIONS FAIL DIFFERENTLY AND THE CODE IS SHAPED BY THAT. COLD -- shrink must not strand bytes. Every source drawn on is recorded with what it was asked for and what it REPORTED giving. A source asked that reports nothing is carried as STRANDED: never counted as zero, never dropped. Bytes that left one ledger and entered none are the shape that goes unnoticed for weeks. A delivered zero is an accounting; silence is not, and the result keeps them distinguishable. Per sgl-project#694, totals come only from actuator reports -- nothing increments a "reclaimed" figure from the plan. An unfundable ask refuses and actuates NOTHING rather than drawing what is there, and the refusal states that unavailable bytes did not count toward it. HOT -- grow must not exceed the floor, and that is deliberately NOT enforced here: the dial enforces it. This module's duty is to not paper over the refusal, so a floor refusal is returned unchanged and is NEVER RETRIED SMALLER -- a floor refusal is a statement about the rig, not a negotiation. sgl-project#217 SHAPES THE HOT PATH. A restore that "came back" was measured at 23% of target. So a grow is followed by a read-back and the result reports what was MEASURED, never what was requested. An absent read-back leaves reached_bytes None -- "not measured", distinct from "reached nothing" (sgl-project#606). A read-back that raises REFUSES, because a tenant must not be reported warm on an unverified grow. Tests, hermetic (CUDA_VISIBLE_DEVICES="", real bridge objects, injected actuators, no scheduler, no CUDA): test_coresidency_policy_553.py 16 passed + 4 subtests MUTATIONS: hiding stranding, and reporting the request instead of the measurement, fail 4 test/registered/unit/managers/ 13 failed / 2263 passed -- failures unchanged from this branch's pre-existing 13, passes 2247 -> 2263, exactly the 16 new pins ruff + codespell clean. STILL PRICED-ONLY, and this is the honest seam: the policy takes injected release_fn / grow_fn / measure_fn. Binding them to the live dial and slot runtime needs a scheduler and belongs with the live proof, not a desk fake. WINDOW TICKET: /spinning/GPU_WINDOWS.md. There was NO prior sgl-project#553 entry (grepped; none), so this CREATES one rather than appending to an existing ticket, and says so in the entry. The 142 pre-existing lines are verified byte-identical by sha256 before and after (18e21da5dda7ce55).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 17, 2026
The analysis's tenant-COLD row named the gap exactly: parking 'works for the translator (ledger.park_all) and for GDN slots (sgl-project#364). No generic per-tenant mover; no single caller that can address tenant X.' The first half wired the two byte probes so the bridge stops answering all-unavailable; this is the actuator a cold event calls once that bridge has ranked a plan. I CHECKED THE TABLE'S GATING COLUMN RATHER THAN TRUSTING IT, because §10 had just caught the same table lying about numbering. test_ledger.py runs 22 passed under CUDA_VISIBLE_DEVICES=: the ledger's park/restore is already exercised hermetically with CPU tensors, so a mover over the same protocol is desk- provable and the marking holds. Had those tests needed a device this would have been a window item and the table would have been wrong twice. NOT A SECOND LEDGER. AudioAssetLedger already parks and restores and its ParkRoute protocol is already generic; what was missing was never the parking. TenantMover registers TENANTS over that same protocol and reimplements nothing. A parallel asset ledger would have been the two-authorities defect the sgl-project#553 bridge exists to reconcile. The vocabulary is per tenant, which is the whole factoring: the translator's ASR -> talker -> codec order is one tenant's physics, not a property of moving tenants. register(tenant, routes, ranks=...) takes it as configuration, and a test asserts a second tenant restoring y, x -- nonsense for the translator, correct for that tenant. An unranked route sorts last: unlisted is unknown, not urgent. Stranding keeps the shape the policy layer already uses. Released bytes come from what a route RETURNED (sgl-project#694); a route that reports nothing, or raises, is STRANDED and never zero, while a route reporting 0 is an accounting. release_fn() hands cold_event a None in exactly that case, which is how that layer reads 'did not report', so a stranded tenant surfaces there instead of being counted as a delivered zero. parked_bytes_by_device() omits a device with nothing parked rather than reporting 0 (sgl-project#606). An unknown tenant RAISES and names who is registered: 'no such tenant' and 'that tenant had nothing to give' are different answers. A defect in my own test, recorded in the analysis: the first _Route stub used parked_bytes=None for 'not specified', so 'reports nothing' was inexpressible and two stranding tests failed against correct code. A stub whose default and whose absence-case share a value cannot test an absence rule. 12 tests red first, mutation-proven (collapsing stranding to zero reds the two stranding pins; ignoring the supplied ranks reds all three vocabulary pins). 57 passed across the mover, the rung executor, the coresidency policy and the translator ledger -- the ledger's 22 unchanged, it was not touched. ruff clean. Desk only, no boots. sgl-project#553's desk surface is now exhausted: cut 4 is gated on sgl-project#305's multi-model binding, cut 5 is rig-gated by its own text, G1 is filed, and the live bindings plus any live number's correctness remain the named window items.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 17, 2026
… exist, twice Prior-art gate outcome: do NOT build steps (2) and (3). Periodic GDN snapshots at chunk boundaries and resume-from-nearest-checkpoint are both implemented, in two lineages, and verified at file:line. Resume-from-nearest (step 3): _match_prefix_helper tracks the deepest node on the matched path still carrying a state, gated on the checkpoint grid (mamba_radix_cache.py:1513-1517); _match_post_processor truncates the KV prefix to it (:1685), COWs the state into a fresh slot (:1611-1654), and the scheduler re-prefills the tail (schedule_batch.py:2644-2660, :2719-2733). Snapshot writer (step 2): retention at cache_unfinished_req (:771, the chunked-prefill boundary) and cache_finished_req (:603). Cadence flag: --mamba-checkpoint-interval (server_args.py:4106/:14048). Host tier for mamba state: HiMambaRadixCache write_backup (hi_mamba_radix_cache.py:373) / init_load_back (:553) into mamba_pool_host. The real gap is an explicit refusal plus two switches that are off: server_args.py:14066-14071 rejects --mamba-checkpoint-interval together with --enable-hierarchical-cache, so today you get deterministic grid-pinned checkpoints (VRAM-only) OR a host tier, never both. Live boot has mamba_checkpoint_interval=None, enable_hierarchical_cache=False, gdn_resident_state_slots=None, max_mamba_cache_size=12. This also RETRACTS my own sgl-project#740 mechanism claim (NOTE_740 section 4a). I said misses were "no state at that position". Resume-from-nearest exists, so a full re-prefill needs no state ANYWHERE on the path. The cause is EVICTION: 12 slots shared between running requests and every retained checkpoint under LRU (mamba_radix_cache.py:1090). Same conclusion as sgl-project#745 from the other end -- the states need somewhere to live -- but the missing piece is tiering and survival, not a writer. sgl-project#212 determination as asked: sgl-project#555's canonical blob does NOT obsolete it, because the blocker was never the format. The encoded objection (gdn_slot_ladder.py:26-30) is that a recurrent state cannot be TRUNCATED to a shorter prefix. sgl-project#745 never truncates -- it anchors at exact positions and replays the tail -- so it is the composition sgl-project#212 leaves open, not a re-litigation of it. Step (5) reconciled: FEATURE_CATALOG.md:514-521 (idle slots free an INDEX, not memory) and sgl-project#364 (gdn_slot_ladder.py:10-15, pool boot-sized to the cap, surplus bytes land in the KV pool) are both true in different domains. Lowering the cap at BOOT returns real bytes via handle_max_mamba_cache spending from total_rest_memory (model_runner_kv_cache_mixin.py:1999, :2050-2068); vacating at runtime returns an index. Framed throughout as raising the binding stage's token ceiling, never as a KV share rising. Idle-vacate x resume is already safe: vacate_plan takes resumed_ids and makes restores unconditional before vacates (gdn_slot_ladder.py:169-215). The obligation on the resume path is wiring, not machinery. Recommended revised order is in section 7: measure an --enable-hierarchical-cache arm FIRST, since it already tiers mamba state to host; only lift the :14066 refusal if deterministic resume points are also wanted. Nothing boot-verified: source plus one read-only get_server_info probe. Docs only.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 30, 2026
…her premise holds
(1) "WHERE DID THE CONVERTED FREE GO" -- IT DIDN'T. The premise mixes two
instruments, two cuts and two moments.
The KV budget posts are BYTE-IDENTICAL between pass 1 and pass 3, every post,
both phases:
PP-phase PP0 13.572 / PP1 7.408 / PP2 8.342 weights, holdback 1.000,
mamba 0.731/0.426/0.304, spec 0.877/0.511/0.365, prefill 1.000
TP-phase weights 4.283 / 2.734 / 3.678, rest 24.164 / 13.821 / 13.855
Nothing in the ledger moved. And on the SAME instrument across passes,
available_gpu_mem is unchanged too: PP0 1.62 / PP1 3.01 / PP2 2.36 GB in pass
1 AND pass 3.
The "before" figures (PP1 3522 / PP2 2171 MiB) are available_gpu_mem READ OFF
THE solveRR BOOT -- a DIFFERENT cut ([32,17,15]) -- reported as 3.44 / 2.12 GB.
The "after" figures (1239/1078/1643) are NVML free sampled at runtime after
the conv3 boot. Different quantity, different cut, different moment. So the
free was never consumed and the 674k projection was never funded by it: on
this cut the headroom is 3.01 GB on PP1 and 2.36 GB on PP2 and it is STILL
THERE, unspent, right now.
WHAT DOES MOVE is the seam STAGING POST, and it existed in pass 1 too -- the
"vorher --" in the question is an artefact of which lines I had quoted
earlier, not of the boot:
pass rank0 seam B/row rank0 STAGING rank1/2 B/row rank1/2 STAGING world
1 3620.4 1794 MiB 1167.1 701 MiB 615,434
2 3954.3 1990 MiB 1167.1 701 MiB 583,030
3 3961.7 1903 MiB 1202.9 685 MiB 616,194
rank0 B/row rises monotonically (3620 -> 3954 -> 3962) but STAGING does not
(1794 -> 1990 -> 1903), and the world figure follows STAGING, not B/row. So
this is not a sgl-project#782 monotone ratchet. Consistent with that: trapsafe finds ZERO
latch/refusal lines of any kind, and the rank1/2 staging change of -16 MiB is
itself below the 64 MiB latch threshold and happened anyway -- i.e. no latch
is arming on this path at all.
CONSEQUENCE: the +9.42 % projection is NOT falsified and NOT demonstrated. It
was never attempted, because the headroom it would draw on is still sitting
free. Pushing the staging post back is a live lever; it is just not the thing
that ate anything.
(2) LOAN-CAP HYPOTHESIS -- FALSIFIED. The pool does not refuse the rows; it
never asks for them.
Log check on the TP-decode phase, conv3 boot: RESERVATION-CAPPED 0,
"reservation capped" 0, extent refusals 0, "capped at the id space" 0,
"beyond the id space" 0. Zero refusals of any kind.
The 39 BACKING-DIAL calls show why. Fields carried are request / prev_size /
uniform_backed_rows / reserved_backing_rows / store_bound_rows. First call:
request=575920 against prev_size=616194 -- the dial asks for LESS than the
pool already is. Last call: request=334134, prev_size=228134,
uniform_backed_rows=229376 -- growing, but to 334k against an id space of
616,194. Across the run the dial operates far BELOW the existing ceiling and
its requests track demand.
Also worth recording against the cap theory: reserved_backing_rows=763651 and
store_bound_rows=763652 are ABOVE the 616,194 id space, so the backing
bookkeeping is not clamped to the id space in the first place.
So the LEND is not unconsumable BY CONSTRUCTION. It is unconsumed because
demand at the tested load (agent-shaped, 5 sessions) never reached the
existing id space, so the loaned extension was never needed. That is a
weaker and more accurate statement than the cap hypothesis, and it changes
the handover: the open question for sgl-project#778 is not "fix the cap" but "does a
load that actually saturates 616k rows take the loan". My measurement does
not exclude that; it only shows this load never got near it.
sgl-project#778 therefore stays OUT of the ladder -- unchanged verdict, corrected reason.
LADDER STATUS with today's deletions: fixed point 616k MEASURED (pass 1 and
pass 3 agree to 0.12 %), +/-5.3 % boot-to-boot noise mandatory on every future
pool figure, sgl-project#364 out (both gates), sgl-project#778 out (not demonstrated), 674k
projection unattempted with its headroom still free.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Use the first token to find child, avoiding traversing all children.