Skip to content

Optimize radix tree matching - #364

Merged
Ying1123 merged 1 commit into
sgl-project:mainfrom
ispobock:optimize_radix_tree
Apr 17, 2024
Merged

Ying1123 merged 1 commit into
sgl-project:mainfrom
ispobock:optimize_radix_tree

Conversation

@ispobock

Copy link
Copy Markdown
Collaborator

Use the first token to find child, avoiding traversing all children.

@merrymercy

Copy link
Copy Markdown
Collaborator

@ispobock This seems to be an interesting optimization.
Did you check the correctness of many workloads? Did you notice any performance improvements?

@ispobock

ispobock commented Apr 17, 2024 •

Copy link
Copy Markdown
Collaborator Author

@merrymercy I tested on mmlu dataset with different parallel, the accuracy is the same with the main branch. The total latency has slight improvement.

with optimization parallel total latency avg accuracy
N 64 258.204 0.528
Y 64 257.729 0.528
N 128 254.929 0.528
Y 128 254.867 0.528

I also test on sharegpt dataset with 1000 prompts and found the throughput has slight improvement.
without optimization:

Total time: 175.29 s
Throughput: 5.70 requests/s
Average latency: 65.24 s
Average latency per token: 0.18 s
Average latency per output token: 1.10 s

with optimization:

Total time: 173.21 s
Throughput: 5.77 requests/s
Average latency: 64.15 s
Average latency per token: 0.17 s
Average latency per output token: 1.08 s

Test env:

  • model: llama2_13b_chat
  • gpu: 1 A100 80G

Reproduce:

python -m sglang.launch_server --model-path /workdir/llama2_13b_chat/ --port 30000

python benchmark/mmlu/bench_sglang.py

python benchmark/latency_throughput/bench_throughput.py --backend srt --tokenizer /workdir/llama2_13b_chat/ --dataset /workdir/ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 1000 --request-rate 128 --port 30000

@ispobock

Copy link
Copy Markdown
Collaborator Author

For 5000 prompts of sharegpt dataset benchmark, the throughput is improved 11.2% with this optimization.
without optimization:

Total time: 973.67 s
Throughput: 5.14 requests/s
Average latency: 439.68 s
Average latency per token: 1.17 s
Average latency per output token: 10.23 s

with optimization:

Total time: 874.84 s
Throughput: 5.72 requests/s
Average latency: 382.35 s
Average latency per token: 1.01 s
Average latency per output token: 8.91 s

@merrymercy

Copy link
Copy Markdown
Collaborator

Awesome! cc @hnyls2002 @Ying1123

@Ying1123
Ying1123 merged commit e822e59 into sgl-project:main Apr 17, 2024
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
shiyu7 pushed a commit to shiyu7/sglang that referenced this pull request Dec 26, 2025
efschu added a commit to efschu/htsglang that referenced this pull request Jul 31, 2026
…mplete GDN state blob (incl. ReplaySSM rings + cursor), idle-only vacate planner (sgl-project#364 slice 1)
efschu added a commit to efschu/htsglang that referenced this pull request Jul 31, 2026
…window guard, newcomers-as-demand fix, on-card bit-identical vacate/restore (sgl-project#364 slice 2)
efschu added a commit to efschu/htsglang that referenced this pull request Aug 1, 2026
…from the concurrency ceiling, admission sized from session budget (pre-cap profiled), over-admission interlock (sgl-project#364 slice 3)
efschu added a commit to efschu/htsglang that referenced this pull request Aug 1, 2026
…e machine with enforced sgl-project#364 quiesce boundary (TypeError without a QuiesceReport), DETACH_PENDING serves / ATTACH_PENDING refuses, MANUAL > ROUTED > CONTROLLER > BOOT with per-resolution source, tag routing from config with no built-in names; executor/endpoints deliberately unbuilt pending the card ticket (sgl-project#309 decision layer)
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
…and a five-cut list

Desk analysis only, no build, as scoped. docs/dev/ANALYSE_553_elastic_coresidence.md.

What the four named building blocks actually are, read from the code rather
than from their titles:

- sgl-project#330 VRAM dial: a REAL mechanism, and its decisive property for sgl-project#553 is that
  store_bound_rows is a LIFETIME bound baked into captured graphs. So the dial
  has a FREE BAND (shrink to any floor, re-grow to the boot bound) and a HARD
  WALL above it. The free band is exactly where CUDA-graph compatibility is
  preserved.
- sgl-project#364 GDN slot ladder: a real, executing hot/cold mover, and graph-neutral by
  construction (it moves state between slots, not geometry). It is the
  existence proof that a hot/cold event need not touch capture, and its
  inventory seam -- the owner enumerates its live set, consumers never read a
  container directly -- is the pattern the other consumers need (sgl-project#551).
- sgl-project#287 KV pressure ladder: PLANNER ONLY. Seventeen NotImplementedError sites;
  the docstring says so plainly. But the ordering invariants are already
  enforced in code and the relief rungs reference existing features BY NAME, so
  the relief half can be made to execute by delegation without new mechanism.
- sgl-project#286 register: the right vocabulary, but exactly one ledger exists and it is
  wired to the translator. There is no generic tenant to raise an event
  against.

The pattern across the event table: both directions are largely built, and
what is missing is ARBITRATION. Two registries exist (dial participants, sgl-project#286
asset classes) and neither can see the other, so "free 4 GiB for the video
tenant" has no addressee.

Cut list, ordered so each cut is provable alone and no cut before the last
touches capture: (1) one registry or an explicit bridge; (2) a generic tenant
mover factored out of AudioAssetLedger; (3) make sgl-project#287's RELIEF rungs execute by
delegation -- the largest block of stubs, and it never approaches the graph
wall because relief rungs are declared "no KV layout change"; (4) the
arbitration policy on top; (5) LAST and only with the rig, cross the graph wall
via the per-rung pre-captured graphs sgl-project#287 already specifies, with cold rungs
parked as register class graph_rungs.

Graph compatibility is treated as the target state throughout, per directive:
every cut states its effect on graph state, and cut 5 carries the explicit
route back to full coverage. Until cut 5 lands, sgl-project#553 must state its band --
"elastic within the boot capture bound" -- rather than implying unlimited
elasticity.

Instrumentation named honestly: SGLANG_KVSO_TICK_TRACE=1 (throttled per-session
regulator trace: effective interval, measured tick_cost, binding headroom
ratio, host-tail size; pure logging, no control effect). The analysis states
what it is NOT -- a regulator trace, not a per-phase breakdown of device time --
and proposes no measurement it has not checked exists.

One decision deliberately left open and named as such: whether sgl-project#553 owns the
arbitration policy or whether it belongs to sgl-project#363. That fork needs the sgl-project#363
stage feed bound first, since an arbitration policy fed by a table holding one
stage would have nothing to choose between -- the same failure sgl-project#363/S8
documents one level up.

codespell: clean.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 9, 2026
… and the route that does compose

As specified -- reserves for requests 2-4 spill at runtime so a single
session's usable KV grows -- the leg cannot be built from existing
machinery:

1. sgl-project#364's own comment (model_runner_kv_cache_mixin.py:2136-2140) says a
   vacated GDN slot is reused by another session and never returned to
   the KV pool; the pool is fixed at boot.
2. kv_session_offload.py:1076-1085 states the by-design consequence that
   a single running session is always the oldest, hence untouchable,
   hence never a spill victim: a lone request over the device budget
   truncates rather than spilling.
3. phase_flip_runtime.py:585-612 guards the flip against kv-session-
   offload, so enabling it boots but permanently disarms every flip.

What composes with --enable-phase-flip: --gdn-resident-state-slots N
(boot-time KV gain, GDN state crosses the flip via gdn_flip_mover) and an
explicit --kv-pressure-ladder relief vector without the session_offload
rung.

Combined with the measurement that the lone session is capped by
context_length rather than by a reserve, the route to 'one session gets
max KV' is the context ceiling plus a boot-time state-pool cap. The YaRN
boot did not run: the prod_boot single-instance guard refused while the
previous instance was still dying and the call budget expired inside that
refusal. Serving is back on the proven config.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 9, 2026
…top the flip spending the VRAM reserve

Two defects found on metal while opening the long-context leg, both in the
composition of --gdn-resident-state-slots with --enable-phase-flip.

1. THE CAP AND THE FLIP COULD NOT BOOT TOGETHER.

   --gdn-resident-state-slots 10 --max-mamba-cache-size 20 refused:

     PhaseFlipBootError: req_to_token shapes diverge between stacks:
     PP (5, 393224) vs TP (4, 393224)

   Applying the cap OVERRIDES server_args.max_mamba_cache_size, while the
   pre-cap profiled count -- which sgl-project#364 slice 3 sizes the concurrency
   ceiling from -- was remembered on the MODEL RUNNER. phase_flip_boot
   builds its TP stack from copy.deepcopy(server_args) taken AFTER that
   override, so that stack had no runner attribute and read an already
   capped pool: it sized its ceiling from 10 where the PP stack used 20
   (ratio 3 -> 3 vs 4 requests). The crater slice 3 exists to prevent,
   reintroduced through the copy.

   The pre-cap count now lives on the ARGS, write-once, so the deepcopy
   carries it. Underscore-prefixed, which is what ServerArgs.__setattr__
   exempts from the strict post-resolution mutation guard.

2. A FLIP COULD KILL THE INSTANCE WHEN THE KV POOL WAS FULL.

   One session of 276214 tokens (0.995 of the PP pool), then a routine
   policy flip:

     kv_reshard._exchange -> torch.empty(584 MiB)
     -> torch.OutOfMemoryError ("600.38 MiB is free")
     -> Fatal Python error: Aborted, all three ranks down

   The flip is not free of memory: it packs its outgoing rows and
   pre-allocates a receive buffer per peer, from the same device memory the
   KV pool fills. Nothing checked that they fit. The free VRAM at the crash
   was 600 MiB, already under the 1024 MiB corridor floor, so a flip that
   had squeezed through would have broken the corridor anyway.

   Affordability is now a second term in the EXISTING pre-flight verdict --
   the one that already abandons a flip whose live set does not fit the
   target pool. Same properties on purpose: computed from the plan before a
   byte is allocated, reduced across ranks so the abandon is unanimous, and
   it abandons the FLIP, never the server. Spendable memory is
   cached_free + max(0, driver_free - reserve): allocator-cached bytes are
   invisible to NVML and cannot move the corridor, driver bytes can. The
   reserve comes from --rank-user-reserve-mib rather than a second opinion.

TESTS
  bash scripts/run_631_flip_family.sh -> 559 passed (was 540).
  New, both in the family list:
    test/registered/unit/mem_cache/test_gdn_cap_flip_two_stacks_631.py
      11 cases incl. the can-fail that reproduces PP 4 vs TP 3 exactly.
    test/registered/unit/managers/test_phase_flip_staging_reserve_631.py
      10 cases incl. the measured 584-vs-600 MiB refusal and the
      falsifier for "just add driver-free and cache".

METAL
  Defect 1: the boot that refused now completes and flips (PP->TP->PP)
  during warmup; ctx 393216, PP pool 277468 tokens, max_running_requests
  still 4. Defect 2's guard is code+tests, metal proof pending the next
  full-occupancy run.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 13, 2026
…ever engaged on the flip path

Ordered mid-shift: prove that idle GDN/Mamba state (~147 MiB per slot,
~440 MiB for three idle slots) vacates into the KV pool and that the sizer
banks it as reclaimable mass.

IT DOES NOT FIRE. Zero occurrences of "vacat" and zero of "resident state
slots" across all three flip boots of this shift (boot_f, boot_g, boot_j,
/spinning/evidence-631/kvuniverse-r2). The gate is

    if self.server_args.gdn_resident_state_slots is not None:

and --gdn-resident-state-slots appears in NEITHER the ship argv capture nor
the uncapped flip argv. Flag unset, gdn_slot_executor stays None, the
ladder never runs. The machinery (gdn_slot_runtime, gdn_slot_ladder,
gdn_slot_executor) is real and wired to the between-tick boundary; it is
simply switched off for this configuration. So "unused Mamba states are
spilled during bs1 time" is, here, unbacked -- the states sit as dead
reservation.

AND THE SIZER MUST NOT BANK THEM YET. The mamba reservation is its own
budget post subtracted before KV, and the seam reserve's measured anchor is
read with every slot reserved -- so those bytes are counted unavailable
twice and reclaimable never. Making the sizer spend them while the vacate
is off would size a pool against memory nothing releases. Engaging the
vacate comes first; counting it second.

Not fixed in this shift and nothing is claimed about the bs1 leg: no boot
with the flag, no vacated-byte count, no wake-on-slot-2 determined-answer
probe. Booked as register entry 57 with the numbers that make it a live
candidate for the budget re-solve: ~440 MiB parked against seam shortfalls
of 476 MiB (rank1) and 931 MiB (rank2) measured on boot G.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 13, 2026
…he wall is the layout

Adding --gdn-resident-state-slots 4 to the flip argv makes the cap fire on
all three ranks and banks measurable memory, but the RUNTIME vacate still
cannot fire: kvso, the ladder's only standing population, is refused at parse
time on pp_size>1 and the flip instance is PP=3 by construction. Slice 1
(boot-time pool cap) delivers the bytes without it; slice 3 (admission above
the cap) depends on it and is unbacked here.

Credit measured at an identical pinned pool (563974) against boot J:
have_m 3744->4306 / 1224->1574 / 1514->1822 MiB. That is a quarter of the
estimate carried in, because the cap correctly does not shrink
intermediate_ssm_state_cache, which is sized from admission.

The budget re-solve the previous shift specified cannot work: have_bytes is
physical free VRAM, and the binding rank's pool is stopped by its seam floor
rather than by its ceiling, so raising the ceiling frees nothing. The pool
falls out of physical arithmetic on rank2 -- 2846 free, 1455 floor, 1024 law,
367 MiB spendable, 46976 tokens -- and the sizer derived 610942 against that
hand figure of 610950. The lever is rank2's 1455 MiB arena tail, a function
of the PP stage ratio against the TP vector.

T4 is answered without a boot: the spilled-session-across-a-PP-flip
composition is refused before the server starts, which is the safest of the
three possible verdicts.

Test results:
- 77 passed (test_phase_flip_seam_reserve, test_gdn_slot_executor,
  test_gdn_ladder_decouple, test_gdn_cap_flip_two_stacks_631,
  test_gdn_slot_ladder, test_yarn_rope_cache_growth).
- Boot K1 (cap engaged, pool pinned 563974): 30 completed cutovers (15/15),
  0 abandons, 0 refusals, 0 tracebacks, 64001-token prefill, corridor
  continuous minimum under load 2082/3341/2128 MiB, 0 breaches.
- Boot K3 (derived pool 610942, no operator number): 30 completed cutovers
  (15/15), 0 abandons, 0 refusals, 0 tracebacks, 64001-token prefill,
  corridor 1510/2715/1994 MiB, 0 breaches. 610942 < 620000, so the
  quarantine stays.
- Serving restored on 30030 and verified with health 200 plus three real
  generations.

Evidence: /spinning/evidence-631/kvuniverse-r3/
efschu added a commit to efschu/htsglang that referenced this pull request Aug 13, 2026
…ool, seam reserve, YaRN 1M)

Merge source origin/feat/kv-universe-656 @ 4157aad into the R6 tip
481411a. Conflicts: 0. 28 files, 4684 insertions, 37 deletions.

Carries: derived KV pool with the 620000-token quarantine constant DELETED
and replaced by phase_flip_seam_reserve (solver reserves each rank's seam
staging bytes on top of the user corridor, ENV_MARGIN_MIB default 192 MiB,
seam_reserve_enabled() defaults True); phase-policy control-loop fix (refusal
outcomes reach the policy, bounded stand-down instead of silent re-arm at the
dwell interval); sgl-project#364 idle-vacate engagement; YaRN 1M ceiling plus the RoPE
growth-path correctness fix; seam fingerprint widened with
gdn_resident_state_slots / enable_kv_session_offload / pp_stage_ratio /
phase_flip_tp_vector (keyed only when non-default); cgroup-escape boot fix in
scripts/s33_boot_from_capture.sh (transient systemd-run --scope); and the
lazy RoPE reserve, BUILT BUT DEFAULT OFF -- falsified on metal (buys 240 MiB
per rank of written bytes but zero pool, and breaches the 1024 MiB corridor
under deep prefill). SGLANG_ROPE_LAZY_CACHE = EnvBool(False) verified after
the merge, both at the definition site and at runtime (unset, resolves False).

TEST RESULTS -- CPU-only desk protocol, CUDA_VISIBLE_DEVICES=99,
PYTHONPATH=<worktree>/python, pytest --color=no, one directory per invocation.
Baseline is this worktree at 481411a before the merge; its counts are
identical to MERGE-R6's post-step-3 column, so the chain back to the frozen
/spinning/wt-merge-r6-base @ 598f570 is continuous.

| suite                | BASE 481411a   | after this merge  |
|----------------------|-------------------|-------------------|
| sgl-project#631 flip family     | 1116 passed       | 1116 passed       |
| unit/managers        | 9F 1357P 18S      | 9F 1357P 18S      |
| unit/mem_ledger      | 1F 359P           | 1F 359P           |
| unit/model_executor  | 15F 594P          | 15F 594P          |
| unit/server_args     | 615P              | 615P              |
| unit/turnkey         | 116P              | 116P              |
| unit/utils           | 46F 348P 4S       | 46F 348P 4S       |
| unit/docker          | 4P                | 4P                |
| new-656 arm          | (files absent)    | 4F 56P            |

Every failure count identical to baseline. The branch's six new test files are
not named in the explicit run_631_flip_family.sh list, so they were run as a
separate arm rather than assumed covered. Its 4 failures are INHERITED, not
introduced: the same arm run against the source worktree
/spinning/wt-kv-universe gives the identical 4 failed / 56 passed. All four
are test/srt/test_phase_flip_serving_proof_gate.py raising "No accelerator
(CUDA, XPU, HPU, NPU, MUSA, MPS) or platform plugin is available" -- the file
needs a visible device and cannot run under the CPU-only merge protocol.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 17, 2026
…authorities only

WIRED VS PRICED-ONLY, answered by grep before building: nothing imported
coresidency_registry outside its own module, so a tenant COLD event did NOT
actuate -- the bridge priced and no one called it. Same for HOT. That is the
gap Cut 3 closes.

managers/coresidency_policy.py is the caller and introduces NO NEW ACTUATOR.
Every move is an existing authority's own API: vram_dial.apply_budget_request
(replicated grow/shrink, already enforces the floor, "rejections carry the
exact floor arithmetic and change nothing"), vram_dial.
verify_pool_reached_capacity (the read-back), GdnSlotRuntime.unbind (sgl-project#364),
and the Cut 1/2 bridge. A source pin forbids this module from touching
torch.cuda / cuMem / set_active_prefix / runtime_set_ at all.

THE TWO DIRECTIONS FAIL DIFFERENTLY AND THE CODE IS SHAPED BY THAT.

COLD -- shrink must not strand bytes. Every source drawn on is recorded with
what it was asked for and what it REPORTED giving. A source asked that reports
nothing is carried as STRANDED: never counted as zero, never dropped. Bytes
that left one ledger and entered none are the shape that goes unnoticed for
weeks. A delivered zero is an accounting; silence is not, and the result keeps
them distinguishable. Per sgl-project#694, totals come only from actuator reports --
nothing increments a "reclaimed" figure from the plan. An unfundable ask
refuses and actuates NOTHING rather than drawing what is there, and the
refusal states that unavailable bytes did not count toward it.

HOT -- grow must not exceed the floor, and that is deliberately NOT enforced
here: the dial enforces it. This module's duty is to not paper over the
refusal, so a floor refusal is returned unchanged and is NEVER RETRIED SMALLER
-- a floor refusal is a statement about the rig, not a negotiation.

sgl-project#217 SHAPES THE HOT PATH. A restore that "came back" was measured at 23% of
target. So a grow is followed by a read-back and the result reports what was
MEASURED, never what was requested. An absent read-back leaves reached_bytes
None -- "not measured", distinct from "reached nothing" (sgl-project#606). A read-back
that raises REFUSES, because a tenant must not be reported warm on an
unverified grow.

Tests, hermetic (CUDA_VISIBLE_DEVICES="", real bridge objects, injected
actuators, no scheduler, no CUDA):
  test_coresidency_policy_553.py  16 passed + 4 subtests
    MUTATIONS: hiding stranding, and reporting the request instead of the
    measurement, fail 4
  test/registered/unit/managers/  13 failed / 2263 passed -- failures
    unchanged from this branch's pre-existing 13, passes 2247 -> 2263, exactly
    the 16 new pins
  ruff + codespell clean.

STILL PRICED-ONLY, and this is the honest seam: the policy takes injected
release_fn / grow_fn / measure_fn. Binding them to the live dial and slot
runtime needs a scheduler and belongs with the live proof, not a desk fake.

WINDOW TICKET: /spinning/GPU_WINDOWS.md. There was NO prior sgl-project#553 entry
(grepped; none), so this CREATES one rather than appending to an existing
ticket, and says so in the entry. The 142 pre-existing lines are verified
byte-identical by sha256 before and after (18e21da5dda7ce55).
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 17, 2026
The analysis's tenant-COLD row named the gap exactly: parking 'works for the
translator (ledger.park_all) and for GDN slots (sgl-project#364). No generic per-tenant
mover; no single caller that can address tenant X.' The first half wired the two
byte probes so the bridge stops answering all-unavailable; this is the actuator
a cold event calls once that bridge has ranked a plan.

I CHECKED THE TABLE'S GATING COLUMN RATHER THAN TRUSTING IT, because §10 had
just caught the same table lying about numbering. test_ledger.py runs 22 passed
under CUDA_VISIBLE_DEVICES=: the ledger's park/restore is already exercised
hermetically with CPU tensors, so a mover over the same protocol is desk-
provable and the marking holds. Had those tests needed a device this would have
been a window item and the table would have been wrong twice.

NOT A SECOND LEDGER. AudioAssetLedger already parks and restores and its
ParkRoute protocol is already generic; what was missing was never the parking.
TenantMover registers TENANTS over that same protocol and reimplements nothing.
A parallel asset ledger would have been the two-authorities defect the sgl-project#553
bridge exists to reconcile.

The vocabulary is per tenant, which is the whole factoring: the translator's
ASR -> talker -> codec order is one tenant's physics, not a property of moving
tenants. register(tenant, routes, ranks=...) takes it as configuration, and a
test asserts a second tenant restoring y, x -- nonsense for the translator,
correct for that tenant. An unranked route sorts last: unlisted is unknown, not
urgent.

Stranding keeps the shape the policy layer already uses. Released bytes come
from what a route RETURNED (sgl-project#694); a route that reports nothing, or raises, is
STRANDED and never zero, while a route reporting 0 is an accounting.
release_fn() hands cold_event a None in exactly that case, which is how that
layer reads 'did not report', so a stranded tenant surfaces there instead of
being counted as a delivered zero. parked_bytes_by_device() omits a device with
nothing parked rather than reporting 0 (sgl-project#606).

An unknown tenant RAISES and names who is registered: 'no such tenant' and 'that
tenant had nothing to give' are different answers.

A defect in my own test, recorded in the analysis: the first _Route stub used
parked_bytes=None for 'not specified', so 'reports nothing' was inexpressible
and two stranding tests failed against correct code. A stub whose default and
whose absence-case share a value cannot test an absence rule.

12 tests red first, mutation-proven (collapsing stranding to zero reds the two
stranding pins; ignoring the supplied ranks reds all three vocabulary pins).
57 passed across the mover, the rung executor, the coresidency policy and the
translator ledger -- the ledger's 22 unchanged, it was not touched. ruff clean.
Desk only, no boots.

sgl-project#553's desk surface is now exhausted: cut 4 is gated on sgl-project#305's multi-model
binding, cut 5 is rig-gated by its own text, G1 is filed, and the live bindings
plus any live number's correctness remain the named window items.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 17, 2026
… exist, twice

Prior-art gate outcome: do NOT build steps (2) and (3). Periodic GDN
snapshots at chunk boundaries and resume-from-nearest-checkpoint are both
implemented, in two lineages, and verified at file:line.

Resume-from-nearest (step 3): _match_prefix_helper tracks the deepest node
on the matched path still carrying a state, gated on the checkpoint grid
(mamba_radix_cache.py:1513-1517); _match_post_processor truncates the KV
prefix to it (:1685), COWs the state into a fresh slot (:1611-1654), and
the scheduler re-prefills the tail (schedule_batch.py:2644-2660,
:2719-2733). Snapshot writer (step 2): retention at cache_unfinished_req
(:771, the chunked-prefill boundary) and cache_finished_req (:603).
Cadence flag: --mamba-checkpoint-interval (server_args.py:4106/:14048).
Host tier for mamba state: HiMambaRadixCache write_backup
(hi_mamba_radix_cache.py:373) / init_load_back (:553) into mamba_pool_host.

The real gap is an explicit refusal plus two switches that are off:
server_args.py:14066-14071 rejects --mamba-checkpoint-interval together
with --enable-hierarchical-cache, so today you get deterministic
grid-pinned checkpoints (VRAM-only) OR a host tier, never both. Live boot
has mamba_checkpoint_interval=None, enable_hierarchical_cache=False,
gdn_resident_state_slots=None, max_mamba_cache_size=12.

This also RETRACTS my own sgl-project#740 mechanism claim (NOTE_740 section 4a).
I said misses were "no state at that position". Resume-from-nearest
exists, so a full re-prefill needs no state ANYWHERE on the path. The
cause is EVICTION: 12 slots shared between running requests and every
retained checkpoint under LRU (mamba_radix_cache.py:1090). Same
conclusion as sgl-project#745 from the other end -- the states need somewhere to
live -- but the missing piece is tiering and survival, not a writer.

sgl-project#212 determination as asked: sgl-project#555's canonical blob does NOT obsolete it,
because the blocker was never the format. The encoded objection
(gdn_slot_ladder.py:26-30) is that a recurrent state cannot be TRUNCATED
to a shorter prefix. sgl-project#745 never truncates -- it anchors at exact
positions and replays the tail -- so it is the composition sgl-project#212 leaves
open, not a re-litigation of it.

Step (5) reconciled: FEATURE_CATALOG.md:514-521 (idle slots free an
INDEX, not memory) and sgl-project#364 (gdn_slot_ladder.py:10-15, pool boot-sized to
the cap, surplus bytes land in the KV pool) are both true in different
domains. Lowering the cap at BOOT returns real bytes via
handle_max_mamba_cache spending from total_rest_memory
(model_runner_kv_cache_mixin.py:1999, :2050-2068); vacating at runtime
returns an index. Framed throughout as raising the binding stage's token
ceiling, never as a KV share rising.

Idle-vacate x resume is already safe: vacate_plan takes resumed_ids and
makes restores unconditional before vacates (gdn_slot_ladder.py:169-215).
The obligation on the resume path is wiring, not machinery.

Recommended revised order is in section 7: measure an
--enable-hierarchical-cache arm FIRST, since it already tiers mamba state
to host; only lift the :14066 refusal if deterministic resume points are
also wanted.

Nothing boot-verified: source plus one read-only get_server_info probe.
Docs only.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 30, 2026
…her premise holds

(1) "WHERE DID THE CONVERTED FREE GO" -- IT DIDN'T. The premise mixes two
    instruments, two cuts and two moments.

The KV budget posts are BYTE-IDENTICAL between pass 1 and pass 3, every post,
both phases:
  PP-phase  PP0 13.572 / PP1 7.408 / PP2 8.342 weights, holdback 1.000,
            mamba 0.731/0.426/0.304, spec 0.877/0.511/0.365, prefill 1.000
  TP-phase  weights 4.283 / 2.734 / 3.678, rest 24.164 / 13.821 / 13.855
Nothing in the ledger moved. And on the SAME instrument across passes,
available_gpu_mem is unchanged too: PP0 1.62 / PP1 3.01 / PP2 2.36 GB in pass
1 AND pass 3.

The "before" figures (PP1 3522 / PP2 2171 MiB) are available_gpu_mem READ OFF
THE solveRR BOOT -- a DIFFERENT cut ([32,17,15]) -- reported as 3.44 / 2.12 GB.
The "after" figures (1239/1078/1643) are NVML free sampled at runtime after
the conv3 boot. Different quantity, different cut, different moment. So the
free was never consumed and the 674k projection was never funded by it: on
this cut the headroom is 3.01 GB on PP1 and 2.36 GB on PP2 and it is STILL
THERE, unspent, right now.

WHAT DOES MOVE is the seam STAGING POST, and it existed in pass 1 too -- the
"vorher --" in the question is an artefact of which lines I had quoted
earlier, not of the boot:
  pass   rank0 seam B/row   rank0 STAGING   rank1/2 B/row   rank1/2 STAGING   world
   1         3620.4            1794 MiB        1167.1          701 MiB      615,434
   2         3954.3            1990 MiB        1167.1          701 MiB      583,030
   3         3961.7            1903 MiB        1202.9          685 MiB      616,194
rank0 B/row rises monotonically (3620 -> 3954 -> 3962) but STAGING does not
(1794 -> 1990 -> 1903), and the world figure follows STAGING, not B/row. So
this is not a sgl-project#782 monotone ratchet. Consistent with that: trapsafe finds ZERO
latch/refusal lines of any kind, and the rank1/2 staging change of -16 MiB is
itself below the 64 MiB latch threshold and happened anyway -- i.e. no latch
is arming on this path at all.

CONSEQUENCE: the +9.42 % projection is NOT falsified and NOT demonstrated. It
was never attempted, because the headroom it would draw on is still sitting
free. Pushing the staging post back is a live lever; it is just not the thing
that ate anything.

(2) LOAN-CAP HYPOTHESIS -- FALSIFIED. The pool does not refuse the rows; it
    never asks for them.

Log check on the TP-decode phase, conv3 boot: RESERVATION-CAPPED 0,
"reservation capped" 0, extent refusals 0, "capped at the id space" 0,
"beyond the id space" 0. Zero refusals of any kind.

The 39 BACKING-DIAL calls show why. Fields carried are request / prev_size /
uniform_backed_rows / reserved_backing_rows / store_bound_rows. First call:
request=575920 against prev_size=616194 -- the dial asks for LESS than the
pool already is. Last call: request=334134, prev_size=228134,
uniform_backed_rows=229376 -- growing, but to 334k against an id space of
616,194. Across the run the dial operates far BELOW the existing ceiling and
its requests track demand.

Also worth recording against the cap theory: reserved_backing_rows=763651 and
store_bound_rows=763652 are ABOVE the 616,194 id space, so the backing
bookkeeping is not clamped to the id space in the first place.

So the LEND is not unconsumable BY CONSTRUCTION. It is unconsumed because
demand at the tested load (agent-shaped, 5 sessions) never reached the
existing id space, so the loaned extension was never needed. That is a
weaker and more accurate statement than the cap hypothesis, and it changes
the handover: the open question for sgl-project#778 is not "fix the cap" but "does a
load that actually saturates 616k rows take the loan". My measurement does
not exclude that; it only shows this load never got near it.

sgl-project#778 therefore stays OUT of the ladder -- unchanged verdict, corrected reason.

LADDER STATUS with today's deletions: fixed point 616k MEASURED (pass 1 and
pass 3 agree to 0.12 %), +/-5.3 % boot-to-boot noise mandatory on every future
pool figure, sgl-project#364 out (both gates), sgl-project#778 out (not demonstrated), 674k
projection unattempted with its headroom still free.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants