Skip to content

misc: update e2e test benchmark config - #825

Merged
zhyncs merged 5 commits into
sgl-project:mainfrom
zhyncs:new
Jul 30, 2024
Merged

zhyncs merged 5 commits into
sgl-project:mainfrom
zhyncs:new

Conversation

@zhyncs

@zhyncs zhyncs commented Jul 30, 2024

Copy link
Copy Markdown
Contributor

Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.

Motivation

Please explain the motivation behind this PR and the goal you aim to achieve with it.

Modification

Briefly describe the changes made in this PR.

Checklist

  1. Ensure pre-commit pre-commit run --all-files or other linting tools are used to fix potential lint issues.
  2. Confirm that modifications are covered by complete unit tests. If not, please add more unit tests for correctness.
  3. Modify documentation as needed, such as docstrings or example tutorials.

@zhyncs
zhyncs merged commit f52eda3 into sgl-project:main Jul 30, 2024
@zhyncs
zhyncs deleted the new branch July 30, 2024 09:19
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
amote-i pushed a commit to amote-i/sglang that referenced this pull request Dec 8, 2025
SGL-Router adds and modifies the health status interface.
A new service parameter --router-url is added.
When a service process exits, the health status interface is called to change the health status to False.

Co-authored-by: wangguoyan <wangguoyan6@h-partners.com>
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 23, 2026
… on resident lock refs

FALSIFIED ON METAL, boot_826_review_0823_0912, 08:55:45, all three ranks
~1 minute after ready and 36 s into load:

  08:55:24  sgl-project#825 TREE DIVERGENCE ONSET   (all 3 ranks)
  08:55:42  sgl-project#825 TREE RECONCILE          (all 3 ranks, pp_to_tp cutover)
  08:55:45  AttributeError: 'NoneType' object has no attribute 'id'
            full_component.py:239 release_component_lock
  08:55:50  kill_process_tree

Mine, not the ambient root-D family: the previous boot on the same load
profile, same duration, without sgl-project#825, has zero release_component_lock
occurrences.

My premise was "requests are parked between the movers and the cutover, so
dropping the tree is safe". PARKED IS NOT UNREFERENCED. The cutover carries
RESIDENT requests across and each holds a last_node with a lock ref; reset()
rebuilds the root, orphaning those nodes, so dec_lock_ref's parent walk no
longer terminates at the live root.

So the ACTION is withdrawn behind SGLANG_TREE_RECONCILE, default off. The
DETECTION half stays on and is what earned its place: it found real
divergence on all three ranks within 35 s of load, which is the first direct
measurement of the B.10 defect rather than an inference from #cached-token.
A reconcile that must not run while any node is locked has to be built
against the lock refs -- evicting only the unlocked portion, or reconciling
where no resident reqs exist -- and that is a design, not a flag flip.

sgl-project#827, independent of the above: the walk's only exit was reaching root, so
ANY broken ancestor chain -- detach, evict, rebuilt root -- ran off the top
and died on None.id, naming an attribute instead of the actual fault. Now a
named RuntimeError carrying the node, the component and the hop count. NOT a
break: swallowing it would leave component_protected_size_ overcounted for
the life of the process, trading a loud fault for a slow leak. Mutant
(raise -> break) killed by 3 arms.

Tests: 48 green across the four suites. sgl-project#827 red first, with the two
crash-pattern arms failing on the real AttributeError before the fix.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 23, 2026
Wave 3, stage 1 -- the first stage of the batched-window tree. Carries
WINDOW-QUEUE tickets W1 (sgl-project#825/sgl-project#827, release-walk crash guard) and W2 (sgl-project#826,
solved arming floor), which the queue requires to ship in the SAME tree: W1
lets the boot survive the flip walk and W2 lets the flip actually arm, so
validating either alone repeats boot_827's wasted window.

Four commits, base 21ff075 (the feat/822 stage):

  bb09574  [sgl-project#825] Prove the prefix trees congruent at the one aligned point
  80ca356  [sgl-project#826] Adopt the solved arming floor, or refuse the boot by name
  e0895cf  [sgl-project#825/sgl-project#827] Withdraw the tree reset: it crashed on resident lock refs
  500be7e  [W2] Pre-flight: arming reachability must include the measured seam draw

ANCESTRY COMPUTED, NOT INFERRED FROM THE BRANCH NAME. merge-base with the tip
is 21ff075, i.e. this branch does not carry fix/815-rest-stubdrift. That
matters only if the two touch the same files, and they do not: sgl-project#815 is
test-only across ten helper files, none of which are the five test files here.

CONFLICT LAGE: none, and checked rather than hoped. `git merge-tree
--write-tree` against the tip returned a single tree with no conflict block
before the merge was attempted, and the merge itself reported "Automatic merge
went well". The file surface is DISJOINT from wave 3's other stage
(fix/801-void-send-contract, which touches only scheduler_pp_mixin.py), so the
two stages cannot interact textually.

W2's ticket is preflight_pass=N and that is NOT a reason to keep it off the
line. The queue's N is CONDITIONAL: the solved arming floor is reachable only
while the seam's measured draw stays at or below the solved 218 MiB reserve,
and the measured draws on this rig are 162/387/436/907 MiB. That gates the
WINDOW, not the merge -- the actuator sits behind SGLANG_ARMING_FLOOR_SOLVED,
so merging it changes no default. Delivering one bootable tree is this
strand's job; whether the window runs is the operator's.

The branch's own pre-flight suite is worth naming because it refuses a false
green rather than manufacturing one: test_arming_reachability_preflight_w2.py
asserts `band_floor + max(reserve, measured_draw) + margin <= band_ceiling`
and records that the obvious W2 assertion -- 819 + 218 + 192 = 1229 <= 1229 --
is true and useless, because it silently assumes a measured draw of zero,
which is the case that does not occur.

SECTION 18 DUTY, DISCHARGED IN THIS MERGE. This stage adds a new reusable
module, managers/tree_congruence.py, so rule 5 applies to this merge the same
way it applied to the four modules catalogued in 9f1af20. The entry is in
this commit (docs/dev/FEATURE_CATALOG.md, section 18.5). The reusable part is
not the prefix-tree logic but the equality-by-MIN-reduce trick: a digest sent
as the pair (d, -d) makes the group agree iff group_min == -group_neg_min,
which turns "are all ranks equal?" into two MIN-reduces over a collective that
offers only min. The entry states the gate honestly: DETECTION is ungated and
always runs, while the RECOVERY is behind SGLANG_TREE_RECONCILE
(phase_flip_runtime.py:3777, unset = OFF) because the reset it performs is
what crashed on resident lock references and was withdrawn rather than
repaired in place.

GATE. Battery test/registered/unit/{managers,planner,server_args,mem_cache},
hermetic under CUDA_VISIBLE_DEVICES="", PYTHONPATH set to this worktree,
one battery at a time.

    baseline (tip 4f2072a)   7 failed, 8585 passed, 1852 skipped, 887 s
    this stage                  7 failed, 8638 passed, 1852 skipped, 868 s
    NEW failure ids             NONE (comm against the baseline list is empty)
    fixed vs baseline           0

The +53 passes are this stage's own new cases; the failure set is unchanged
and is still exactly the seven known ones.

The baseline was re-measured in this worktree rather than inherited from
strand 17c, and came out byte-identical to its list: the same seven cases, all
in test_pp_flip_slot_hold_631.py, left red on purpose (see the stub-drift
family in section 12 of the catalog).

The gate is not vacuum-green: all five suites this stage adds land INSIDE the
battery directories and are therefore actually collected --
test_arming_floor_actuator_826.py, test_arming_reachability_preflight_w2.py,
test_tree_congruence_825.py and test_tree_congruence_wiring_825.py under
managers, test_lock_release_walk_detached_827.py under mem_cache.

Section-18 pointer checker (test_building_blocks_catalog_538.py): 104 passed,
up from 101, the three new cases being the three new tree_congruence
citations.

COVERAGE CHECK, because a battery that silently fails to collect is a gate
that measures nothing. The W4/W5 strand reported that its worktree cannot
collect the managers suite at all -- 92 modules import `datasets`, which is
not installed there. That does NOT apply to this gate, and it was verified in
the positive direction rather than inferred from the absence of an error
message: the venv here has datasets 5.0.0, the battery log contains zero
collection errors and zero ModuleNotFoundError, and an explicit --collect-only
over test/registered/unit/managers reports 3436 tests collected. The numbers
above therefore cover the managers suite in full.

No boot. Serving stays down; this tree is the input to ONE batched window, not
a reason to open one.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 24, 2026
…le, not by new machinery

Follow-up to 3b0c031. The open item was the correctness core: after a
cutover that moves no KV, the new phase's device pool holds no valid KV while
the radix tree still maps prefixes to row ids, so a lookup must MISS and fall
through to the host tier. Which function makes it miss was unanswered.

THE OBVIOUS ACTION IS ALREADY KNOWN TO CRASH, and finding that out from the
tree rather than from a boot is the point of asking first.
phase_flip_runtime.py:4590-4620 records sgl-project#825 trying exactly this on
2026-08-23, on all three ranks at once:

    cache_finished_req -> dec_lock_ref
    -> full_component.py:239  `if cur.id in skip_lock_node_ids`
    AttributeError: 'NoneType' object has no attribute 'id'

with the cause in its own words: "PARKED IS NOT UNREFERENCED. The cutover
carries RESIDENT requests across, and each holds a `last_node` with a lock
ref. `reset()` rebuilds the root, orphaning those nodes." sgl-project#825 withdrew the
ACTION and kept detection only, noting the real fix "needs to be built against
the lock refs ... and that is a design, not a flag flip."

RESOLVED, AND BY THE USER'S OWN RULE. The stale `req_pool_idx` (the
resident-carry blocker) and the orphaned `last_node` (sgl-project#825's crash) are the
same fact wearing two hats: A RESIDENT REQUEST CARRIED ACROSS THE CUTOVER.
The no-carry rule removes it, and both blockers go with it.

The mechanism exists and needs no new code: `retract_all`
(schedule_batch.py:1812) walks `release_req` (:1783) over every request, and
`release_req` does exactly the two releases required --
`release_kv_cache(req, tree_cache, is_insert=False)` and
`req.reset_for_retract()` -- taking precisely the objects the flip already
holds.

SEAM ORDER: fence (maybe_flip_writeback + demotion on) -> retract all ->
tree reset (now safe, sgl-project#825's precondition gone) -> cutover + weights refill ->
re-admit, served by HiCache read-through. That is the user's sequence verbatim.

THE CARRY ONLY EVER EXISTED TO AVOID A RE-PREFILL. With read-through that
re-prefill is a cache hit, which is why "no carry" is correct rather than a
simplification -- and it is exactly the "honest warm-up cost as served-request
latency" named as the validation metric. That cost is REAL and must be
measured, not assumed small: it is what this design pays to delete the mover,
the staging reserve and a crash class.

TWO CAVEATS RECORDED AS UNVERIFIED rather than glossed: that
`release_kv_cache` releases the lock ref along the whole parent chain (read
from release_req's body, not from release_kv_cache's own), and that retraction
at the seam is compatible with the flip's quiescence predicate. Both must be
confirmed before this is built.

No code changes. Documentation only.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 24, 2026
… and retraction feeds the policy a lie

Follow-up to 4b6ee1e, which recorded two unverified caveats rather than
glossing them. Both are now answered, and only one of them was clean.

CAVEAT 1 -- DISCHARGED, AND IT PROMOTES THE ORDER TO A LAW.
`release_kv_cache` (mem_cache/common.py:1749) routes to
`tree_cache.cache_finished_req(req, is_insert=False)`, and the live
`dec_lock_ref` (mem_cache/hi_mamba_radix_cache.py:1610) is

    while node != self.root_node:
        ...
        node = node.parent

so the lock ref is released along the whole parent chain -- PROVIDED the walk
still terminates at the live root. That is exactly the loop sgl-project#825 crashed in
once `reset()` had rebuilt the root and orphaned the nodes.

RETRACT STRICTLY BEFORE RESET. Reversing the two reproduces the 2026-08-23
three-rank crash. A red-first test must pin the ORDER, not merely the outcome.
Two bypass paths are recorded for the build to handle, both already named in
that function: `req_pool_idx is None` under MambaRadixCache, and
`kv_spill_state == "host"` routing to the kv-session-offload release.

CAVEAT 2 -- NOT A CLEAN PASS. A NEW DEFECT THIS DESIGN WOULD CREATE.
Retracted requests return to the waiting queue, so their full context
reappears as `pending_prefill_tokens` -- the exact quantity the flip policy
compares against N. But N is priced from X and P, the UNCACHED prefill
throughputs. Retracted tokens are CACHED by construction: the fence just
persisted them and read-through serves them. Their real cost is a small
fraction of what the bar prices them at.

So every cutover would hand the policy a large pending figure whose true cost
is far smaller, in BOTH directions. That is a thrash pathway CREATED by this
design, not inherited -- today's carry keeps those tokens out of the pending
count entirely. Recorded as unresolved, with three candidate directions and
none of them yet evidenced. The honest one is to price the pending figure by
cache residency, which is the same class as sgl-project#856(b): a decision is only as
good as the quantity it compares.

Explicitly NOT claimed as mitigation: sgl-project#856(b) raises N (8.50 -> 18.06 s,
18614 -> ~39500), which makes this harder to trigger. That is a mitigating
accident and must not be cited as a fix.

No code changes. Documentation only.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 24, 2026
… cannot argue for a flip

PREREQUISITE for the seam rebuild, and its own defect. The phase-flip seam
will RETRACT resident requests rather than carry them -- the carry is what
made sgl-project#825's tree reset crash, and read-through makes it unnecessary. But
retraction puts each full prompt back in the waiting queue, and
`_pending_prefill_tokens` sums `len(req.origin_input_ids)`.

That figure is compared against N = C / (1/X - 1/P), where X and P are
UNCACHED prefill throughputs. Retracted tokens are not uncached: their KV was
computed and the fence persisted it, so re-prefilling them is a CACHE READ --
and a cache read costs the same in TP as in PP. Equal cost on both sides of an
inequality cancels, so those tokens cannot make PP cheaper than TP and have no
business in the comparison. Pricing them as cold prefill would hand the policy
a huge backlog after every cutover, in BOTH directions.

sgl-project#731 measured that outcome from a different cause: "51,369 -> 102,307 tokens
across one cutover ... six cutovers, nothing served." Its fix (the carry
consumes the queue entry) cannot catch this route, because nothing is
double-counted here -- the tokens are counted ONCE, at the wrong price. On
W25's numbers the shipped sum prices eight retracted 20k prompts at 160,000
against a live bar of 18,614: 8.6x over, from a cutover alone.

THE OPTION THAT WAS CHOSEN, AND THE TWO THAT WERE NOT.

* CHOSEN -- stamp the residency at retraction. `reset_for_retract` records the
  fill boundary (`extend_range.end`, the same notion of "computed" the pending
  counter already uses for a chunked remainder; a request with output has
  finished its prefill by construction) BEFORE clearing it. Exact, and free.
* REJECTED -- residency lookup at the counter. `prefix_indices`,
  `num_matched_prefix_tokens` and `extend_range` are ALL cleared by
  `reset_for_retract`, so after the fact the information does not exist
  anywhere and only a fresh `match_prefix` walk could recover it -- per queued
  request, per policy round, under the tree lock, duplicating the matching
  admission already does.
* REJECTED -- a dwell-window exclusion. Coarser in the wrong direction: it
  also excludes tokens that genuinely need computing, so a request retracted
  mid-chunked-prefill would have its real remainder deleted from the backlog.
  The stamp distinguishes them (6000 of 20000 computed -> 14000 still count).

DEFAULT PATH BYTE-IDENTICAL. A request that was never retracted carries no
stamp and is counted in full, so every pre-sgl-project#856 caller gets the same number.
A request retracted for a reason OTHER than the flip seam -- priority
preemption, sgl-project#731's own path -- has no fence behind it and also carries no
stamp, which the can-fail test pins: zeroing anything merely flagged retracted
would delete real backlog and pass every other assertion here.

ERROR DIRECTION IS THE SAFE ONE. If the cache is evicted under the credit, the
stamp overstates residency and this UNDER-reports pending, making the policy
LESS eager to flip -- not more. It cannot cause the thrash it exists to
prevent.

TESTS (hermetic, CUDA_VISIBLE_DEVICES=""):
  test_uncached_pending_856.py                     12 passed, 3 subtests
    incl. the shipped-sum reproduction of the sgl-project#731 shape, and a source-order
    assertion that the stamp precedes the clear (afterwards it is
    unrecoverable)
  pending-counter consumers (713/677/689/701/819/854/phase_policy)
                                                  156 passed
  retraction consumers (retract_decode_fcfs, prefill_adder, 798, 791b)
                                                   28 passed
  ruff check + format clean
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 24, 2026
The correctness core of the no-KV flip, encoded before the retirement that
needs it. Once the flip carries no KV, the new phase's device pool holds no
valid rows while the radix tree still maps prefixes to row ids, so the tree
must be dropped for a lookup to MISS and fall through to the host tier.

That action is already known to be fatal in the wrong order. sgl-project#825 tried it and
took the instance down on all three ranks (2026-08-23):

    cache_finished_req -> dec_lock_ref
    -> full_component.py:239  `if cur.id in skip_lock_node_ids`
    AttributeError: 'NoneType' object has no attribute 'id'

    "PARKED IS NOT UNREFERENCED ... each holds a `last_node` with a lock ref.
     `reset()` rebuilds the root, orphaning those nodes, so the parent walk in
     `dec_lock_ref` no longer terminates at the live root and runs off the top
     into None."

THE NO-CARRY RULE REMOVES THE PRECONDITION rather than working around it.
Retraction releases each resident request's rows AND its tree lock ref
(release_req -> release_kv_cache -> cache_finished_req -> dec_lock_ref, whose
loop is `while node != self.root_node: node = node.parent`). Run it FIRST and
every walk terminates at a live root; run it after reset() and it walks off
exactly as sgl-project#825 did. So the ordering is not a preference, and it is now a
named function instead of a comment two callers must remember.

REFUSES RATHER THAN REPAIRS when either step is missing, and the asymmetry is
deliberate: skipping the retraction leaves locked nodes for the reset to
orphan (the crash, loud); skipping the reset leaves the tree naming rows that
hold no KV (a WRONG ANSWER, silent) -- which is worse, so neither is allowed
to be optional.

THE TEST PINS THE ORDER, NOT THE OUTCOME, and that is the whole point. A test
asserting only "the tree ended up empty" would pass against the fatal
ordering. So the FATAL ordering is reproduced hermetically against a faithful
model of the real walk -- nodes with a parent, a reset() that installs a NEW
root object, and a release that walks parents until it reaches the CURRENT
root, which is the entire mechanism of the crash -- and
`test_reset_before_retract_reproduces_the_825_crash` asserts it still raises
AttributeError('NoneType'). If that ever stops raising, the model has drifted
from the crash it represents and every other assertion in the file is
worthless.

TESTS (hermetic, CUDA_VISIBLE_DEVICES=""):
  test_seam_order_856.py     8 passed
  ruff format clean

Correction to the previous commit's message (3ff19e7), recorded rather than
force-pushed: it said "ruff check + format clean" for scheduler.py and
schedule_batch.py. Format is clean and that change added ZERO ruff errors --
verified, 144 at HEAD~1 and 144 after, ~90 of them the E402s scheduler.py's
own comment already documents -- but `ruff check` on those two files is not
zero and never was. The claim was imprecise.

Coordination: the refill call sites (weights_arena.py, phase_flip_boot.py) are
untouched here and stay that way -- W26's refill-root diagnostic owns them on
fix/856-refill-root, building on the sgl-project#856(a) instrument from 7500c85.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 24, 2026
…ure what warm-up costs

TWO PIECES THE DESIGN OWES, both consequences of carrying no KV.

1. VALIDATE-EARLY REFUSAL (sgl-project#806 pattern, ServerArgs.__post_init__).
The flip retracts every resident request and DROPS the prefix tree, so the
next phase starts with an empty device tier and restores prefixes by
read-through. With --enable-hierarchical-cache off there is nothing to read
through: the sgl-project#703 flip-time writeback has nowhere to persist, the retracted
prefixes are gone, and every conversation re-prefills from scratch on every
flip.

That is a correctness-shaped cost, not a tuning one, and it is INVISIBLE at
runtime -- the flip completes, the requests complete, and only the token bill
says anything happened. A silent, expensive, correct-looking failure is what a
launch gate is for.

Placed beside sgl-project#806's own check and after `materialize_declarations`, for the
reason sgl-project#806 documents: the hierarchical cache is switched by handlers above,
so an earlier check would pass a launch this one refuses. sgl-project#806 refuses the
flip that cannot ENUMERATE what it must move; this refuses the flip that
cannot RESTORE what it deliberately drops.

NO FALLBACK IS OFFERED AND THE MESSAGE SAYS SO -- a test asserts the phrase.
Reviving the mover here would reintroduce the seam this ticket retired, the
staging reserve behind W25's 33 refused arms, and the resident carry that
crashed three ranks in sgl-project#825. Both exits are named instead, because which one
is right depends on what the operator wanted.

2. THE WARM-UP LEDGER (managers/warmup_latency.py), the metric the user
named: not "rows carried" but served-request latency by rounds since cutover.

PRIOR ART CHECKED AND IT IS A NEAR MISS, not a hit:
`regime_classifier.PhaseDwellGate.rounds_since_flip` has the rounds-since-flip
concept but is a GATE deciding whether a flip may happen, and carries no
latency. Also searched: phase_flip_runtime for warm/latency/post-cutover, the
seam census (times the seam, not what follows it), the sgl-project#605 flight recorder
(the seam's own peaks). Nothing measures this.

Geometric bands (<=1, <=4, <=16, <=64, steady) because the CLAIM is that the
cost concentrates in the first rounds and decays; a mean cannot be right or
wrong in a way that shows that. Every band reports as a ratio against THIS
instance's own steady state, so the figure needs no remembered number from
another boot to be readable.

The can-fail directions are the file's point. `None` survives wherever
nothing was compared -- a ratio against an absent control reads as "no
warm-up cost" while meaning "nothing was compared", the sgl-project#606 shape in the one
number this ticket is judged on. "Has not flipped" never folds into "has
flipped and settled". An empty ledger still says it has nothing to say.

HALF-WIRED, AND SAID PLAINLY. `note_cutover` is wired at the cutover. The
request feed is NOT: request latency is assembled in `tokenizer_manager`, a
DIFFERENT PROCESS, so feeding it is a cross-process change rather than a line.
Recorded as an open integration point instead of guessed at -- a
wrongly-wired instrument reports a number, which is worse than reporting none,
and this is the number the proof window judges the design on.

TESTS (hermetic, CUDA_VISIBLE_DEVICES=""):
  test_phase_flip_needs_hicache_856.py   in server_args/ -- 776 passed overall
  test_warmup_latency_856.py             13 passed, 5 subtests
  managers -k "phase_flip or seam or warmup or uncached"
                                         564 passed, 28 subtests, 0 failed
  ruff format clean; ruff check adds ZERO errors -- verified per file from the
  repo root against HEAD: server_args 358 -> 358, phase_flip_runtime 0,
  warmup_latency 0, corridor_admission 0.

RUFF MEASUREMENT CORRECTION, since a previous commit reasoned from it: ruff
must be run FROM THE REPO ROOT or it does not find pyproject.toml and reports
a different rule set entirely (phase_flip_runtime.py read as "110 errors" from
a stray cwd and "All checks passed" from the root). The earlier 144-vs-144
comparison was taken with the same cwd on both sides, so its conclusion --
zero added -- stands; the absolute number was just scheduler.py's 102 plus
schedule_batch.py's 42.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 24, 2026
…NCOMPLETE

The proof boot did what a proof boot is for: it falsified a claim this note
was making. Pin 3111539, boot_w27_0824_1510.log, operator-granted window,
died on all three ranks 15:14:45Z. Full result:
/spinning/gpu-arb/W27-RESULT.md

WHAT PASSED, and it is the one that mattered most. At 15:14:44 all three ranks
logged `RESIDENTS RELEASED ... 1 request(s) retracted and the prefix tree
dropped, in that order`. Zero `NoneType' object has no attribute 'id'`, zero
`dec_lock_ref` -- with the tree dropped while a request was LIVE. That is
exactly the sgl-project#825 three-rank crash the no-carry rule was designed to remove,
and it did not occur. The order law is now vindicated on metal and not only in
its hermetic reproduction.

WHAT FAILED, one second later:

    resident_mamba_slots (gdn_flip_mover.py:620)
    KvReshardError: PHASE-FLIP-GDN live request da65cfe4... has no mamba slot
      -- refusing to flip past unmoved linear state

I retired the KV mover by rebuilding the transfer plan EMPTY after retraction.
I did NOT retire `GdnFlipMover.move()` -- which this note's own retirement
inventory lists as a RETIRE entry (3b0c031). So it still runs, still
enumerates live requests, and finds one whose mamba slot the retraction just
freed. The guard is CORRECT: refusing to flip past unmoved linear state is
right, and a loud abort beats the alternative.

A SECOND FINDING RIDES ON IT, and it is the more general one: `retract_all`
releases rows, mamba slots and the tree lock ref, but the scheduler's batch
structures still reference the `Req`. Every seam-side consumer of "live
requests" therefore sees a live request with freed resources. The GDN mover is
the first to hit it; the sweep for others is a build item.

NO RETRY WAS ATTEMPTED, on evidence rather than caution. The obvious in-window
fix -- drop the GDN mover from the flip path -- trades a LOUD CRASH for SILENT
LINEAR-STATE LOSS, because the GDN anchor resume (sgl-project#745/sgl-project#747) is listed NOT
BUILT in this very note. Removing a guard's caller without building what
replaces it is the class of change this strand exists to refuse.

THE NOTE IS CORRECTED RATHER THAN APPENDED TO. Its "BUILT AND HERMETICALLY
PINNED" list claimed "the retirement"; that entry now reads "the KV
retirement" and carries the falsification inline. A build-state list that
survives its own disproof is worse than no list.

FIVE CRITERIA HAVE NO DATA and are recorded as no-data, not as passes: no
`PHASE-FLIP DONE` line was reached, so zero abandons and zero staging
refusals are absence of evidence, NOT the funding claim C2 makes.

Window hygiene: heartbeat stopped and its exit VERIFIED BEFORE holder release;
holder -> holder.released-w27; cards 0/0/0 before and after; router PID 142
untouched (it is itself an sglang process, which is why a bare `pgrep -f
sglang` matches it). `choom -n 1000` verified on the launcher AND all three
rank PIDs before the weights load -- no OOM, so W26's failure mode did not
recur.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 24, 2026
…rse, and the GDN mover then retires by construction

W27 killed the boot in `resident_mamba_slots` and the guard was RIGHT. This is
the root behind it, and it is not the GDN mover.

THE ROOT. `retract_all` frees a request's KV rows, its mamba slot and its tree
lock ref -- and the scheduler's batch structures keep REFERENCING the `Req`.
`_live_reqs` is the one authority for "who is resident" and reads exactly four
places: every `running_mbs` slot, `running_batch`, `last_batch`, and the
out-of-batch `chunked_req`. Retraction touched none of them, so every seam
consumer after it was handed a live request whose resources were gone. The GDN
mover was simply the first to look.

FREEING A RESOURCE AND RETIRING THE REFERENCE TO IT ARE DIFFERENT JOBS. Doing
only the first leaves a live object that every reader has to special-case, and
the next reader added reintroduces the bug. Same shape as sgl-project#731's fix, where
the carry had to CONSUME the queue entry rather than leave one request counted
in two places.

`consume_retracted_from_live_universe` retires the reference out of all four,
using `filter_batch(keep_indices=...)` and NOT a raw `.reqs` edit -- a batch
carries per-request tensors beside the list and a list edit desynchronises
them. Pinned by a test that asserts filter_batch was the mechanism, because
the raw edit is the tempting shortcut and it fails silently, later.

FIXED AT THE AUTHORITY, AND THE SWEEP SAYS THAT IS ENOUGH. Every seam reader
of the live set goes through `_live_reqs`: resident_mamba_slots
(gdn_flip_mover.py:617), the KV enumeration (:835), the sgl-project#822 census (:1431),
:5457, the output trace (phase_flip_output_trace.py:266), and the release
itself (:8129). Grepping the seam modules for direct running_mbs /
running_batch / last_batch / chunked_req reads returns comments and docstrings
ONLY -- no live code bypasses the authority.

THE GDN MOVER NOW RETIRES BY CONSTRUCTION, WHICH IS THE POINT. W27's no-retry
refused dropping `GdnFlipMover.move()` because doing that with live linear
state trades a loud crash for SILENT linear-state loss. Downstream of this fix
the trade is gone: with the live universe consumed and the tree dropped, both
halves of `flip_mamba_slots` -- resident slots UNION tree checkpoints -- are
empty, so the mover moves nothing. No deletion; the same way the KV mover was
retired by emptying its input.

THE ORDER IS THE SAFETY PROPERTY, and it is pinned against the REAL guard, not
a stand-in: `resident_mamba_slots` no longer refuses AFTER the consume, STILL
refuses without it, and a genuinely resident request still yields its slot. So
the guard is SATISFIED, never weakened -- if that middle test ever stops
raising, the silent-loss trade has been made after all.

W27's CONFIRMED RESULTS, kept out of the FAIL headline's shadow:
  * C6 PASS ON METAL: the sgl-project#825 three-rank crash did NOT occur with the prefix
    tree dropped under a live request -- 0 `NoneType ... has no attribute
    'id'`, 0 `dec_lock_ref`. The retract-before-reset order is vindicated on
    hardware, not only in its hermetic reproduction.
  * `choom -n 1000` held on the launcher AND all three rank PIDs before the
    weights load; no OOM, so W26's failure mode did not recur.

TESTS (hermetic, CUDA_VISIBLE_DEVICES=""):
  test_retracted_leaves_live_universe_856.py   15 passed
    incl. the W27 specimen reproduced (a freed request still enumerated),
    all four live-universe routes, the filter_batch mechanism, seam-safety
    (no targets / no match / a refusing filter_batch / a bare scheduler),
    and the three real-guard derivation tests.
  ruff check + format clean on phase_flip_runtime.py

Full window record: /spinning/gpu-arb/W27-RESULT.md
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 24, 2026
… tree drop leaks 152 rows

The root fix HOLDS. 9 flips, both directions, and the two things that killed
the first attempt did not happen once:

    has no mamba slot          0     (W27: killed all three ranks)
    NoneType ... attribute id  0
    dec_lock_ref               0

`resident_mamba_slots` never fired. The live-universe consume removed it and
the GDN mover retired BY CONSTRUCTION, exactly as derived -- the derivation is
now confirmed on hardware and not only against the real guard in a unit test.

C1 IS PROVEN ON METAL, which is the headline this ticket exists for. Every
`PHASE-FLIP DONE` line, all ranks, both directions:

    0 live slots, sent 0 cells / 0.00 MiB, received 0 cells / 0.00 MiB

against W25's 116502 live slots and 995.31 MiB sent on PP0.

TWO NEW COSTS, MEASURED RATHER THAN ESTIMATED, and both are mine:
`hicache_quiesce->resident_release` is 753.6 ms (the retract+consume+drop), and
the wave loop still walks 16 EMPTY waves for ~314 ms -- the price of retiring
the mover by emptying its input instead of deleting it. That number is what
turns "should we delete the dead loop" into a decidable question.

THE NEW DEFECT, NAMED:

    ValueError: pool memory leak detected! [full] total=472864,
      available=126802, evictable=22, protected=0, ... withheld=345888

126802 + 22 + 345888 = 472712 against 472864 -> 152 rows belong to nobody, and
it accumulates per cycle (fired on the third retract+drop). The retraction
frees the RESIDENT REQUESTS' rows; the prefix tree additionally holds CACHED
PREFIX rows, and `tree_cache.reset()` rebuilds the root and ORPHANS them
instead of returning them to the allocator.

Before sgl-project#856 the tree was never reset at the seam -- sgl-project#825 withdrew exactly that
action -- so this is NEW and it is mine. Attribution checked rather than
assumed: W25 ran 33 cutovers on this rig with no such error, and the broken
load driver cannot be the cause because its requests failed to connect and
allocated no rows.

NO RETRY. The fix means freeing the tree's own rows at the drop, which means
choosing the right API (`flush_cache` is a wipe, `reset` is not) and PROVING
it returns rows to the allocator. Guessing a tree API at a live seam is how a
loud leak becomes a silent double-free. Red-first at the desk, against this
leak arithmetic; it gates the next window.

OPERATOR HONESTY: my `load_w27r.sh` was broken -- curl exit 7, `code:"000"`,
spinning to round 640 in 45 s. So the "responses" it logged are FAILURES, C2
and C7 were never exercised under real load, and the zero abandons above are
NOT the funding claim. The staging-limiter question I left open in the funding
inventory therefore stays open. The 9 flips came from the boot's own warm-up
traffic.

Window hygiene: heartbeat stopped and its exit VERIFIED BEFORE holder release;
holder -> holder.released-w27r; cards 0/0/0; router PID 142 untouched;
`choom -n 1000` held on the launcher before the weights load, no OOM.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 28, 2026
… was in the tree

ROOT, per-rank coverage of window-958 boot 2 (pin 78d030e) plus both boot
logs. Under pure PP the HiCache storage-prefetch veto is a RANK-LOCAL verdict:
prefetch_ballot.prefetch_done_under_ballot returns the local value when the
ballot is None, and the ballot rides _update_uniform_pool_budget's reduce on
tp_cpu_group -- a group the PP loop never passes, and which has world 1 under
pp_size>1 in any case. The ranks therefore decide independently. scheduler.py
:9048 (`continue` after _note_skip("prefetch_pending")) is the LAST line PP0 and
PP1 both ran; :9050 is the first PP0 ran ALONE. Everything downstream --
add_one_req, `self.chunked_req = adder.new_chunked_req`, the unconditional stash
at :7010-7011 -> stash_chunked_request:5449 -> mem_cache/common.py:169 ->
cache_unfinished_req -- is PP0-only. PP0's radix tree gains a 1250-token prefix
PP1's and PP2's never received.

From there it is self-sustaining. PP0 matches its own tree and offers told=1250;
PP1 measures local=0 HONESTLY against its own; sgl-project#791 retracts, sgl-project#797 voids, the
requeue resets the REQUEST and nothing resets the TREE. PP1 can only acquire the
prefix by running the batch it is refused for lacking the prefix.

WHY THE EXISTING BOUND COULD NOT REACH IT. PPAdmissionCongruenceGuard is, in its
own words, "RID-SCOPED, ONE-SHOT", and its termination argument is per rid: each
new retraction for THAT rid lowers THAT rid's floor. The argument is sound and
silent about the POPULATION. The shortfall is a property of the TREE, so every
fresh rid over the same prefix starts unclamped and buys its own voided pass --
six distinct rids in one second, all told=1250 local=0, which is why
_learned_floor was measured RUNNING and LOWERING on PP0 and never bound.

FIX: the same actuator, scoped to the prefix the offer was made over. No new
mechanism and no new collective -- the recorded fatal (the 2026-08-17 HiCache
ack-count reduction) rules out a collective on this path, and a group-uniform
INSERT is structurally impossible: the radix value is a tensor of that rank's own
KV slot ids, dereferenced as real memory by four consumers (allocator free at
unified_radix_cache.py:1713, evict at full_component.py:96, HiCache backup at
:2284, write-back into req_to_token at :1325), and the GDN mamba component
donates a live state slot a non-computing rank does not have.

offered_prefix_key() names the prefix by a blake2b fingerprint of its tokens --
NOT hash(), which is PYTHONHASHSEED-salted and would disagree between the very
ranks this keeps congruent (tree_congruence's constraint 3, same lesson). Length
is mixed in so a prefix and its extension cannot share a floor. prefix_key=None
leaves the pre-sgl-project#963 rid-scoped path byte-identical.

PRIOR ART, gated rather than duplicated. #616g's group-MIN reduces
available_size(), a different quantity, and is switched OFF in the PP phase
(tp_cpu_group world=1). sgl-project#823's tp_head_congruence MINs the right quantity but
only REORDERS, and its enforcer_gate returns GATE_OFF_TP_WORLD_OF_ONE here.
sgl-project#825's tree_congruence detects exactly this divergence and emitted 0 in BOTH
boots: in the PP phase on_round is reached only via scheduler_pp_mixin.py:2477
with require_armed_and_parked=True, so it samples only at an armed flip -- which
the livelock prevents. That is this instance's compensator-reachability gap.

RED-FIRST, and the first red was rejected as worthless: all six cases failed on
the SIGNATURE (TypeError), not on behaviour. Four mutants were then run against
the finished fix and their errors READ:
  consultation withdrawn  -> assert 6 == 1   (boot 2's six voided passes, exactly)
  learn withdrawn         -> assert 6 == 1
  clear-on-serve withdrawn-> assert 0 == 1250 (permanent loss on a healed prefix)
  key-blind clamp         -> assert 0 == 1250 (loss on a prefix nobody reported)
The last two are the danger direction: this fix must never discard a prefix
every rank holds. It clamps only against an OBSERVED shortfall on a SPECIFIC
prefix, and clears the moment the group serves that prefix, so the cost is one
voided pass once rather than one per rid for ever. Double-prefill law: the
requeue loses 0 today and still loses 0; what changes is that it terminates.

An UNRESOLVED miss (observed_local=None) teaches the prefix floor NOTHING -- the
stakes are strictly higher than for the rid scope, since a floor invented from a
number nobody measured would cap every request over that prefix. The sgl-project#944
consumer ratchet CAUGHT this file on its first gate run and is now registered.

NUMBERS, desk gate scripts/gate_tier2_partitioned.py, CVD="" :
  BEFORE 2 genuine (test_collective_family_siblings_610.py x2), 685.00 s
  AFTER  2 genuine (the same two), plus 14 new tests green
  count probe: 2 named == 2 summary, SUBFAILED and ERRORS included
  ruff clean on both touched files.

NOT CLAIMED: no boot, no metal. The divergence SOURCE (the rank-local prefetch
verdict under PP) is named here and left open -- closing it needs the sgl-project#791 ring
lap to carry a prefetch-pending fact home, which is its own posten. This makes
the system self-healing against a divergence however caused.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant