Skip to content

#590 Increase default , track changes in examples and documentation - #971

Merged
Ying1123 merged 3 commits into
sgl-project:mainfrom
foszto:issue-590
Aug 8, 2024
Merged

Ying1123 merged 3 commits into
sgl-project:mainfrom
foszto:issue-590

Conversation

@foszto

@foszto foszto commented Aug 7, 2024

Copy link
Copy Markdown
Contributor

No description provided.

Comment thread README.md Resolved
Comment thread docs/en/sampling_params.md Resolved
@Ying1123
Ying1123 enabled auto-merge (squash) August 8, 2024 00:20
@Ying1123
Ying1123 merged commit c62d560 into sgl-project:main Aug 8, 2024
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
…mentation (sgl-project#971)

Co-authored-by: Ying Sheng <sqy1415@gmail.com>
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 28, 2026
…n it was handed

Root (window-flip-0828 boot 1, pin 0caab9a): scheduler.py hands the
final-extend continuation from self.chunked_req into adder.can_run_list
(add_chunked_req appends and returns None); every PPScheduleRefused exit
discarded the adder without a restore. Both existing restores are
unreachable from that path: _pp_void_own_batch early-returns on
'batch is None' above its restore (#797d = 0 lines in the whole boot log)
and _pp_absorb_void_output's restore is keyed on an output expectation a
pass that built nothing never made. The request was then in NONE of the
four lookup places: 507x 'sgl-project#944 UNRESOLVED told=8192 local=UNKNOWN',
512 voids, sgl-project#801-spin guard death.

The class, stated at the junction: ownership of a chunked continuation
may pass from scheduler-owned state into the pass-owned adder only if
EVERY exit of the pass re-homes it.

Fixes:
1. pp_rehome_refused_chunked_req, called from the one funnel
   _pp_refuse_forwarded_schedule (slot = amended.mb_id, verified
   preserved by void_pp_admission_decision; fallback _pp_live_mb_id).
   The park is GEOMETRY-ONLY via new pass_allocated=False: the refusal
   raises above both the KV allocation and the inflight_middle_chunks
   increment, so the give-backs would double-free / over-decrement --
   caught by the accounting arm during red-first, pinned by
   test_every_refusal_raise_is_above_the_inflight_increment. Zero tokens
   lost at this junction; the status quo lost 8192 per recompute.
2. pp_request_locations answers the container shape instead of
   excepting it away: production's LIST-shaped slot ring made the slot
   place silently empty for every consumer of the helper. The masking
   dict fixture in test_chunked_continuation_clamp_946.py now uses the
   production list shape.
3. The admission skip census is written before all three refusal raise
   sites, so a refused pass logs its skips instead of 'reason=-'.

Tests: test_pp_refused_pass_keeps_continuation_971.py -- red-first
11 failed / 9 passed on the pristine tree, 25 passed with the fixes;
the can-fail arm neuters only the restore and flips 3 assertions.
Gate: 11/11 neighbouring registered modules green, failure sets
before == after == empty, extraction count probe OK on all 11.
ruff / codespell: 0 new findings.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 28, 2026
…dow at flip) into merge/flip-window-0828

Terrains disjoint: sgl-project#971 (scheduler.py, scheduler_pp_mixin.py) x roundtrip
(cache_controller.py, hicache_phase_binding.py, hicache_storage.py,
canonical_page_store.py). Gate on the merge tip recorded in the register
before push.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 28, 2026
…it travels

Boot-2 ring wedge (R3a root H3): the output ring's intermediate-hop
predicates serve the LAGGED slot -- the sender forwards on `if pp_outputs:`
(what it took off the wire last iteration), the receiver reads its own
`mbs[next_mb_id]`, committed passes earlier -- and sgl-project#951's launched posting
is a SAME-SLOT closure that cannot retract a batch already resident there.
The uncovered path is the void relay: `pp_void_relay_stop_rank` derives its
stop from the RETRACTION structure, so a void that names no retraction
(sgl-project#944's zero-offer escape: a rank that lost the request retracts nothing
and launches nothing) was forwarded to a rank whose slot for it was empty,
taken off the wire by that rank's next ADMISSION receive, stashed, and
served POSITIONALLY to a healthy later generation's receive -- which
`_pp_absorb_void_output` then emptied on one rank only, leaving the ring
one message short for ever. PP2 sat in `_do_recv` unbounded because sgl-project#971's
busy wire kept the sgl-project#789 gate early-returning (consumed<posted); boot 1
only survived on a counter another defect had frozen.

Fix: the launched CHAIN -- the same per-hop statement sgl-project#951 consumes, kept
per generation. `_PP_LAUNCHED_CHAIN_KEY` accumulates one bool per rank on
the admission decision (PP0 starts it, every hop appends its own
`mbs[mb_id] is not None`, the last rank records without sending, sgl-project#796),
rides back on the slot's output/void via
`pp_output_payload_with_return_trip`, and `pp_void_forward_payload`
consults `pp_void_relay_launched_verdict`: a void travels exactly to the
ranks whose own admission-time statement says they launched, never to its
source. Absent chain falls back to the legacy rule byte for byte. Log-only
`sgl-project#978 STALE VOID` tripwire in `_pp_absorb_void_output`.

Red-first, measured 2026-08-28 against the unfixed tree
(TheLaggedSlotIsBeyondTheLaunchedPosting):
test_the_launched_chain_relay_stop_closes_the_lagged_slot FAILED -- stuck
[0,1,2], PP2 parked in output_exchange, PP1 event `pass=7 void_absorbed
slot=2` against hazard slot 1 (the mispair), with sgl-project#951 wired in AND
CHAN_DICT counters armed (the readiness gate ran and early-returned:
boot-2 fidelity, not boot-1's frozen-counter accident). Count check:
1 failed, 1 passed of 2. After the fix: 2 passed; the wedge stays
measurable via fix_off (spawn worker forces the verdict back to legacy --
mutant on the hazard direction, reproduces the wedge). The verdict pure
function is pinned exhaustively over rings 2..8; three boundary mutants
(hand back to source / read own entry / short chain forwards) all fall.

Suites: neighbor baseline before == after. Before: 95 passed (801, 791b,
795, 797, 951, idle-void, void-send families, 262s). After: 100 collected,
99 passed + 1 failed, the 1 a harness-stub interface drift in
test_pp_retracted_pass_void_797.py (local recorder lacking the new kwarg;
repaired, no assertion touched; file then 31 passed). ruff: only the one
finding already present on HEAD; codespell clean.

sgl-project#981 boundary, checked before push (CPG_DERIVATION_0828 (b)): this fix's
predicate does NOT stand on the `_pp_output_expected_incoming` memo. The
chain's write (decision recv), reset (top-of-body block) and consumption
(decision send) all live inside full body iterations; the resume-slot
jump's `continue` sits before the reset block but consumes nothing before
the next full body resets. The by-slot chain is written only at a slot's
admission and read while that generation is resident. K3's ungated
admission recv and the clear-site-behind-skip fragility remain sgl-project#981.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 28, 2026
Root (R3b, window-flip-0828 boot 2): batch_is_full is a PASS-scoped memo
(written at the adder's stop condition) stored on the scheduler-lifetime
running_batch. Every clear site is either a FINISH path the
requeue-without-finish family (sgl-project#797/sgl-project#798/#791b/sgl-project#971) destroys before it
runs, or sits behind 'if not running_batch.is_empty()' -- unreachable at
exactly running=0, the defect state. Boot 2: three
gate=batch_full_or_empty_queue(batch_is_full=1,queue=N) declines at
running=0 with free seats and a waiting queue, self-sustaining until an
external kill. The #888b relief lies on the path but its verdict
producer sits behind the same emptiness guard (registered #962b).

Fix: ONE junction at the gate read -- batch_is_full AND is_empty() is
re-derived from the seat test (get_num_allocatable_reqs), not trusted.
Re-derive, not clear: a genuinely exhausted seat table comes straight
back True. Not an eighth clear site: any future requeue path inherits
this without knowing the flag exists. In-tree precedent: the
retract-RPC handler. #888b relief and the seam hook untouched; arm 2 of
the new test pins that #962b's producer stays unreachable at running=0
so the posten stays open rather than silently closed.

Tests: test_batch_full_empty_invariant_962b.py, 8 arms -- red-first on
the REAL void path (gate declined at running=0 with seats free,
note=gate=batch_full_or_empty_queue -- boot 2's own line shape), void
family flag hole filled, green-guard for the 8-resident full-table case
both sides, can-fail mutant. Baselined modules byte-identical.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 28, 2026
…ny traffic

Root (R3a, measured boot 1 vs boot 2 of window-flip-0828):
_pp_wait_for_dict_readiness decided progress from the SHARED CHAN_DICT
posted/consumed counters. CHAN_DICT is one wire demultiplexed by
__msg_type__ (admission_decision / proxy / output). Boot 1's livelock
froze ALL traffic, the counters froze, the gate fired twice -- loud
death. Boot 2's sgl-project#971 fix (correctly) kept the upstream busy posting
admission_decision messages, so consumed < posted held at every poll,
the gate early-returned every time, and PP2 sailed into the unbounded
output receive -- silent wedge, zero firings. A compensator whose
trigger condition was supplied by a foreign defect, disarmed by the
foreign fix.

Fix: per-KIND accounting riding the EXISTING chan axis as a sub-channel
(dict|<kind>, KIND_SEP) -- same single writer, same atomic replace, same
sweep; no second mechanism. The drains keep the shared wire counter as
sole authority (counting kinds separately there would leave unmatched
messages behind -- the module docstring reconciles the two questions).
Only the readiness gate reads the kind axis, and only after
kind_axis_covers proves the sender labels everything it posts;
unlabelled traffic degrades the gate to byte-identical pre-sgl-project#974
behaviour. Helpers module-level, not methods: the method form broke the
one-name-at-a-time stand-in holders (797: 31 -> 24, measured), now
pinned by its own test arm.

Tests: test_pp_readiness_per_kind_974.py, 11 arms in-process with two
real PhaseFlipCounters (no gloo pair -- Corpse-F structurally
unreachable): boot 2's disarm shape red-first (8 failed / 2 passed
pre-fix, the 2 being the directions that must not move), healthy-kind
progress unchanged, no-upstream still raises, kind-axis mutant re-arms
the disarm, sweep covers sub-channel files. 13 baselined modules
byte-identical + 14 counter-importers green, count probes OK.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 28, 2026
…and PP0 names it

Boot 3 of window-flip-0828: after a sgl-project#797 void the follower holds a
sgl-project#971-parked chunked continuation in self.chunked_req (never in any
queue); PP0 cannot see it, its uniform cap is MIN-reduced before
occupancy is loaded, and the void's requeue tail-rotates PP0's copy so
it never reaches a seat. The follower's add_chunked_req appends its
continuation unconditionally, so every forwarded schedule is illegal on
the follower: 514 refusals (191 missing + 323 'admitted rid(s) the
decision does not name'), 512 voids, sgl-project#801-spin guard death in 2 min.

Fix, the register's named sgl-project#968 direction -- the fact rides EXISTING
messages, no new wire, no collective: (rid, executed_len) of a parked
continuation is stamped on (a) the per-pass decision downstream and (b)
the output/void return lap homeward. Named deviation from the design
brief: the ring is asymmetric (the last rank sends output DIRECTly to
PP0, so a middle parker never gets onto the output lap alone) -- hence
both messages, and facts are MERGED per hop, never overwritten,
or a middle rank's fact dies one hop before its consumer. PP0 absorbs
before the admission loop, REORDERS its own waiting_queue so the
ordinary loop admits the rid (reorder, never admit: no second admission
authority, no seat pricing -- once named, the continuation is a
cap-counted member on every rank: no extra, no missing), and clears the
fact when a decision names the rid (self-correcting: the next lap
re-stamps if still parked). No prefix is ever reduced (Kein-Doppel-
Prefill); all seven helpers module-level (holder lesson, test-pinned).

Single-slot hypothesis CONFIRMED (double-sourced): chunked_req is one
attribute, overwritten per truncation and cleared on launch -- PP0
cycles it (518 admits / 7 rids, the stuck rid named 2x), the follower
pins it (514x). That is why the carry must exist.

Tests: test_pp_parked_continuation_carry_968.py, 21 tests -- red-first
GENUINE against the unmodified pin 3c470fb in a detached worktree:
18 failed / 3 passed (names == summary 18==18), 21 passed on the fixed
tree; legacy arm byte-identical without the stamp; no-double-prefill
arm asserts zero prefix reduction; can-fail arm neuters the absorb.
Gate: 42 modules BEFORE/AFTER byte-identical 510P/9F, the 9
pre-existing name-identical both sides (prefetch_symmetry_580 7 +
ring_commit_973 2 -- the 973 pair is the known gloo hang-shape arms,
reproduced solo on the unmodified tree; environmental, posten noted).
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 28, 2026
…w speaks

Boot 4 proved the sgl-project#968 carry chain worked L1-L3 (absorb ran 514x) and
failed at L4 ACT, twofold: the act permutes only waiting_queue, where a
parked continuation never lives, and PP0's copy was DROPPED at the
cross-slot void restore -- self.chunked_req is ONE field, the per-slot
restores write it back to back, last slot wins, and the clear at the
old :8382 justified itself with a claim ('re-admitted from the waiting
queue') that is false for continuations by this file's own rules.
Live witness: boot 4 line 408 clearing 4077b704, one of the 107
missing-rids; 407 of 514 refusals were the parked rid as an unnamed
extra.

Fix at all four unguarded void writers (:8729/:7449 rehome-before-
overwrite, :8823/:7486 requeue-on-clear, checked not asserted: only a
healthy parked continuation not already reachable, not finished, never
reset -- reset shapes are the disposal loop's, which releases AND
queues in one act). scheduler.py:9506/:9545 untouched (exonerated:
preempt requeue / assert-guarded mint). ACT unchanged; its docstring's
false premise and two dead citations are kept as a NAMED retraction
plus the real requeue sites. Writer census: 11, four now guarded,
seven sighted clean and listed in the helper docstring.

Reachability (CPG/call-graph, standing order): every new site closes at
_event_loop_pp_body (:3200 -> :8982 absorb -> :8729/:8823; :3200 ->
:3415 own-void -> :7449/:7486; :3430 -> :7745 -> twins); ACT via
get_next_batch_to_run:6962 -> scheduler.py:9130.

Instruments (standing order, 22 lines, rate-limited first+64th, zero
cases PRINTED): sgl-project#968 MINT/FORWARD/ABSORB/ACT (ACT carries carry>0-
front=0 with an explicit NOT-REACHABLE clause -- the line that names
boot 4 in one pass), sgl-project#968 PARK + sgl-project#971 REHOME-ON-REFUSAL in both
previously-silent park actuators, #968b REHOME-ON-DISPLACE /
REQUEUE-ON-CLEAR, and #791b/#797d now print released=/kept=/displaced=
rid lists (all 514 boot-4 void lines were rid-free).

Test evidence (desk, built before the tests-off-critical-path order
landed; the boot with instruments is the acceptance):
test_pp_continuation_cross_slot_rehome_968b.py red-first GENUINE
22F/9P at b070296 -> 31 passed; 10 neighbouring modules green
(968 21, 797 31, 971 25, 798 7, 631 6, 801 14, 973 5, 962b 8, 974 11);
one honest update: 797:1578 asserted 'dropped, not retracted' -- the
exact belief this commit refutes.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 28, 2026
…nstead

Boot 5's third continuation-loss exit was #968b's own gate: the
park-shape equality in pp_rehome_displaced_chunked_req refused the
UN-parked mid-plan occupant (end=7939 prefix=4096, named verbatim by its
own 'sgl-project#968 MINT none' line), returned None, and the caller's next
statement nulled the live occupant -- a drop, not a release. The fix
that closed the cross-slot displacement carried the class it closed: a
predicate assumed where pp_chunked_req_is_reachable already tests.
Evidence chain one second wide in the boot-5 slice (R6, double-checked):
ADMIT 8192,4096 -> MINT none (last provable hold) -> next line's void
restore with chunk parked=False + displaced=None -> rid never on PP0
again; 514 refusals, PP1 spinning 323x on the unnamed extra it took over
via the sgl-project#971 route.

Fix: the un-parked shape is PARKED first (_park_chunked_prefill_chunk,
void-default pass_allocated=True -- its pass was built, so KV rows and
the inflight increment are owed back), then falls through to the SAME
reachability+queue junction as the parked shape. Zero cached tokens
discarded, nothing reset. Parker-side alternative rejected: retiring the
orphan discards 7939 tokens against the one-chunk law and treats the
spin instead of the drop.

Instruments: the formerly-ambiguous displaced=None (empty field vs
refused occupant were indistinguishable) disappears for a present
occupant -- 'sgl-project#968 PARK route=void-give-backs' plus REHOME-ON-DISPLACE
now fire at exactly the line boot 5 went silent.

Execution proof (standing order): the strongest available -- boot 5's
own log shows this exact line running 514x (every ACT NOT-REACHABLE pass
traversed the refusing gate); reachability is measured, not argued.
Import smoke green; suites live in the parallel test-agent lane per the
tests-off-critical-path order.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 28, 2026
…d acquired

Boot 10 (b27d7c2) died on PP0 at 19:58:17 with
`AttributeError: 'NoneType' object has no attribute 'unsqueeze'` at
mem_cache/unified_cache_components/mamba_component.py:1056, reached via
_pp_process_batch_result -> process_batch_result_prefill ->
maybe_cache_unfinished_req -> cache_unfinished_req, rid
e3d6a9ff11c5460cb115f266f2b187a3.

ROOT (class, not effect). The admission loop's revert site in
managers/scheduler.py frees `req.mamba_pool_idx` whenever a request is not
added to `can_run_list`, guarded by `not req.session`. That predicate is
upstream's PROXY for "nobody else already owned this slot" -- correct
upstream, where a session was the only carrier of a pre-existing slot. This
fork invalidated it: sgl-project#984 (void-park), #968b (rehome-on-displace) and sgl-project#971
(rehome-on-refusal) re-queue requests UNRESET, holding the live mamba slot
of a prefill batch that is still IN FLIGHT. The revert then hands a live
owner's slot back to the allocator and nulls the field; the in-flight
batch's result arrives and dereferences it.

The silent direction is the same defect: when the free lands outside the
in-flight window, HybridReqToTokenPool.alloc re-acquires a fresh slot with
mamba_needs_clear=True -- a ZEROED recurrent state in place of the
request's own (sgl-project#767's shape), with no crash and no log line. Both close at
the provenance test.

FIX, at the one junction. Stamp provenance where the slot is acquired
speculatively by an admission round's prefix match --
unified_cache_components/mamba_component.py (COW resume + host load-back),
mamba_radix_cache.py, hi_mamba_radix_cache.py (both siblings) -- and free
at the revert site only when that stamp is set. The stamp is cleared the
moment HybridReqToTokenPool.alloc takes the slot into a batch, and on every
release/reset path (free_mamba_cache, _rollback_alloc, release_kv_cache's
early branch, reset_for_retract, Req.__init__ default). This is sgl-project#984's own
"pages versus claim" doctrine applied to the third increment: the lock ref
(sgl-project#990) and the chunk (sgl-project#971) already carry provenance; the mamba slot was
the one that did not. It opens neither sgl-project#929 (deficit) nor sgl-project#924 (surplus) --
the stamp is one-shot and dies with the slot.

AFFIRMATIVE REPORTING. None of the five mamba free sites named a rid, which
is why the specimen could only be reached by elimination. The refused
give-back now logs rid + rejection result, so the next boot is the metal
proof of the root rather than a yes/no on the crash.

BACKSTOP, named as a defect. mamba_component.prepare_for_caching_req now
declines retention with a counted, rid-bearing warning instead of
dereferencing a None slot. This is NOT the fix and is stated as such in
place: it costs the step's retention and re-prefills the chunk, which the
standing no-double-prefill order forbids as a routine. It exists because a
crash here takes the whole instance while a decline takes one step.

SIBLING SWEEP: all five writers of `mamba_pool_idx = None` audited and the
stamp cleared at each. gdn_slot_runtime.py:136 is not wired on this build
(the gdn_resident_state_slots gate is off, no "armed" line in any boot log)
and is left untouched. streaming_session.py's clears stay covered by the
retained `not req.session` term.
FUTURE CHECK: a new producer of a pre-existing slot cannot reopen this
silently -- it either stamps at its acquire (and is given back correctly)
or it does not (and is kept), and the backstop's counter names it with a
rid if it manages to drop one anyway.

Evidence: desk. py_compile + import smoke on all seven touched files;
lifecycle probe (Req default False, reset_for_retract clears). Belegstufe:
DESK-BEWIESEN, not metal -- boot 11 is the test.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 28, 2026
… its seat too

Boot 11 (aae7e3b) reached 3m47s -- sgl-project#991 held, no AttributeError -- and
died on `sgl-project#801-spin PP IDLE-VOID LIVELOCK REFUSED` (pp_rank=2, 512
consecutive voided passes, scheduler_pp_mixin.py:8214). The guard is right
that the defect is not on rank 2.

ROOT. Uniform membership -- "a rank executing a forwarded schedule may admit
only what the decision names" -- is enforced at scheduler.py's `pp_not_named`
skip for every candidate that comes out of `waiting_queue`, and NOWHERE for
the one candidate that never appears in a queue. `add_chunked_req` runs ~250
lines earlier and appends `self.chunked_req` unconditionally, before the
decision is consulted at all.

MEASURED, 512 byte-identical rounds (21:27-21:31): rank 1 held rid
8a330526c7b9410f963232874adc451b at executed=3332 as its parked
continuation, in NO queue on ANY rank, so rank 0's decision could not name
it. That site gave it the seat; the decision's own rid (dfd22a9a… /
9528ff20…) could then not be reached; `sgl-project#791 FORWARDED SCHEDULE UNEXECUTABLE`
voided the pass; `sgl-project#971 REHOME-ON-REFUSAL` put the continuation straight back
into `self.chunked_req`; the next pass was identical. Rank 2 voided every
pass behind it until the bound fired.

THE CLASS: a compensator that restores the precondition of its own trigger.
The cut therefore belongs upstream of the rehome, at the admission -- fixing
sgl-project#971 would only move the latch.

FIX. At the one junction, gate `add_chunked_req` on the same membership test
the queue loop already applies: when this rank is executing a forwarded
schedule that does not name its continuation, refuse the SEAT. Not a drop --
the precedent is the sgl-project#906 gate immediately below it: the request stays
`self.chunked_req`, keeps prefix, pages and mid-chunk position, and resumes
on the first pass whose decision names it. PP0 has no incoming decision, so
`incoming is None` there and the default path is byte-identical.

SIBLING SWEEP: the two candidate sources into `can_run_list` under a
forwarded schedule are the queue loop and this one; the queue loop was
already gated. `_note_skip("batch_full_break")` and `pp_not_named` are the
two symptoms this produced in the census and both are downstream of it.

WHAT THIS DOES NOT CLOSE, stated rather than assumed: sgl-project#968's half. sgl-project#968
makes the parked-continuation fact ride the return lap so PP0 CAN name such
a rid, but its actuator `pp_parked_continuation_priority` only REORDERS
PP0's queue and is a no-op exactly when the rid is in no queue on PP0 --
which is this specimen. The new log line prints the rid, its executed
extent and the decision's full name set, so boot 12 MEASURES whether the
continuation ever becomes named instead of leaving it to inference.

Evidence: desk. py_compile + import smoke. Belegstufe: DESK-BEWIESEN.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 2, 2026
… group STOP, not a void

Root (HANDOVER_1153_0902, PROVEN, pre-existing at the pin 228a66d):
scheduler.py _pp_refuse_forwarded_schedule answered PPScheduleRefused
('sgl-project#791 FORWARDED SCHEDULE UNEXECUTABLE', PP1 reached 1 of the 2 rids PP0
named, batch_full_break) with a rank-local compensation:
_pp_admission_pass_voided = True + emptied decision dicts -> mixin
_pp_void_own_batch -> 'sgl-project#631 ROW-DELIVER BATCH NULLED slot=0
pass_voided=True' (boot_855_weg1b2 log 65000-65004) -> PP1 sent no proxy
while PP0's slot stayed set -> nothing carried the void upstream (sgl-project#797
return trip: zero call sites since CUT V; sgl-project#1072 deleted the void relay)
-> PP0's blocking _do_recv consumed PP2's NEXT output under this slot's
label (log 65119) -> one output ahead for the rest of the boot -> the
pp_to_tp arm at 21:42:05 turned the debt into an unproducible output ->
sgl-project#980 ObjectRecvStalled 60 s / sgl-project#1071 PpChainRecvStalled 90 s. Same form
sgl-project#1071 (169f53c) deleted for _pp_void_retracted_pass; this was the
second writer of the same flag.

F1 ROOT: the compensation is deleted, the detector is kept, the refusal
is a group STOP. scheduler.py get_new_batch_prefill's except now does
`raise self._pp_forwarded_schedule_stop(refusal) from refusal`;
_pp_refuse_forwarded_schedule (flag write, dict emptying, sgl-project#971 re-home)
is removed; the new _pp_forwarded_schedule_stop logs the kept
'sgl-project#791 PP-ADMISSION forwarded schedule REFUSED on rank N' line and returns
a RuntimeError formatted by pp_admission_congruence
.forwarded_schedule_stop_message:
  'sgl-project#791 FORWARDED SCHEDULE UNEXECUTABLE STOP rank={rank} slot={slot}
   told=[{told}] reached=[{reached}] census={census} local={local}
   limiter={limiter} running_bs={running_bs} parked={parked}
   r2t_avail={r2t_avail} headroom={headroom} group_limit={group_limit}
   batch_full_setter={batch_full_setter}
   batch_full_at_loop_entry={batch_full_at_loop_entry}: {refusal}'
Every probe is guarded (n/a on an unreadable value); the reached rids
are recorded after the admission loop before any of the three raises;
the batch_is_full setter site is recorded at each writer of the pass
(no_allocatable_reqs_gate, count_arm, disagg_prefill_r2t_avail,
add_one_req_NO_TOKEN). Group stop mechanism (existing, no new
collective): the RuntimeError leaves run_event_loop; run_scheduler_process
(scheduler.py 'except Exception') logs 'Scheduler hit an exception' and
parent_process.send_signal(SIGQUIT) (+ killpg/kill_process_tree under
SGLANG_KILLPG_ON_SCHEDULER_EXCEPTION); peers end in the bounded sgl-project#980 /
sgl-project#1071 receives or the barlink dead-peer probe.

F2 TRIGGER CLASS (PP0 order): on a forwarded schedule the follower's
rank-local seat-count veto is not a verdict. pp_admission_congruence
.rank_local_count_veto_applies(scheduled_extents) is True on PP0 and on
every non-PP boot (unchanged expression) and False on a rank > 0
executing PP0's decision; the sgl-project#823 count arm, its batch_full_break, and
the three pre-loop count gates of the same arithmetic
(batch_full_or_empty_queue's batch_is_full half, min_free_slots_delay,
no_allocatable_reqs) are gated on it. The physical allocator still
refuses (NO_TOKEN -> membership refusal -> STOP naming the numbers).

F3 SIBLINGS: mixin _pp_void_pass_without_upstream_launch's writer of the
flag is provably unreachable (pp_upstream_void_pending returns False on
every path; its final statement is `return False`) -- left with the
proof in a comment. The row-authority _row_skip_plan exit now nulls a
slot through _pp_null_frameless_slot, which applies the same named sgl-project#1020
'VOID REFUSED ON A LAUNCHED SLOT' guard as the void path; the guard is
factored into _pp_slot_holds_unconsumed_launch(mb_id, site) and used by
both sites.

Matched check (error class: a follower still ends a PP0-launched pass
silently): grep -rn '_pp_admission_pass_voided = True' python/sglang/srt
-> only scheduler_pp_mixin.py (the unreachable sgl-project#801 writer); chain
scheduler.py raise PPScheduleRefused (:11633/:11646, :11606) ->
except PPScheduleRefused (:9441) -> raise self._pp_forwarded_schedule_stop
(:9472) -> RuntimeError (:9623); the flag's only readers are
_event_loop_pp_body (:4519 -> _pp_void_own_batch) and the scheduler.py
void guard (:8955), neither reachable from the refusal.

Tests (hermetic, CUDA_VISIBLE_DEVICES=""): new
test_pp_forwarded_refusal_stop_1153.py (T1 STOP form + no void, T2 the
count-veto helper + source pin, T3 the sgl-project#1020 guard on the frameless
null, + the mixin:9389 unreachability proof): 17 passed after; on the
parent ca4c6b7 (git worktree) it is a collection error (new names).
test_pp_refused_pass_keeps_continuation_971.py: 12 tests that pinned the
old compensation INVERTED with the withdrawal named in each docstring
(24 passed after; the inverted 12 are red on the parent: 16 failed /
8 passed / 1 error across both files). Mutants: M1 flag write restored
in the STOP builder -> 2 T1 red; M2a helper returns True -> 1 T2 red;
M2b count-arm gate dropped in the loop -> 1 T2 red; M3 guard dropped from
_pp_null_frameless_slot -> 1 T3 red. Bounded suite (7 unit/managers
files + test/registered/scheduler): before 103 failed / 527 passed /
1 skipped / 27 errors; after 103 failed / 544 passed / 1 skipped /
27 errors -- per-file identical except the new file (+17). ruff check
clean on all touched files; ruff format --diff hunk count unchanged vs
parent (15/14/3, pre-existing) and the two test files formatted.

Evidence tier: DESK-PROVEN. Boot-3 acceptance: grep -F
'sgl-project#791 FORWARDED SCHEDULE UNEXECUTABLE STOP' in the same second as
'sgl-project#791 PP-ADMISSION forwarded schedule REFUSED on rank', and zero
'ROW-DELIVER BATCH NULLED ... pass_voided=True' lines.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 28, 2026
…de the decode

NF rc12z22 D TP0 14:56:26-31: weg2-10-16 and weg2-11-18 were released from the
sgl-project#1471 settle at 14:56:26 and waited 5 s while every pass ran DECODE bs=1 with
admission_ms=-1; the log cannot say why -- WEG2 X-DEFER speaks only while its
GLOBAL pass count is <= 5 (weg2-4-11 used those up) and the admission's skip
census (sgl-project#971 _admission_decline_note) is not printed on this path.
- WEG2-POST-WAKE-PASS gains waiting=<queue> admit=<the pass's admission
  gate / skip census> (loop_skips(kind=n(first=rid)), gate=...).
- the two branches that skip the admission entirely name themselves there
  (gate=phase_prefill_blocked, gate=congruent_lane_cadence) instead of
  leaving the previous pass's note.
- WEG2 X-DEFER speaks the first defer of every rid as well (the repeats stay
  under the global cap).
Instrument only: no admission verdict, order or X rule changes (over X stays
P's, under X D may take it beside the decode as before).

Tests: test_weg2_wake_tail_nowriter_0928 +2 (the line names the census and
the skip branches; X-DEFER speaks each rid's first defer past the global cap)
-> 9 passed. Tests pinning these lines/paths: the same 15 failures on
e7715b0 (leg2_store_probe_1290, sched_fix7_0908, store_grid_claim_1298
-- red on the base too, byte-identical list), the rest green.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 28, 2026
…de the decode

NF rc12z22 D TP0 14:56:26-31: weg2-10-16 and weg2-11-18 were released from the
sgl-project#1471 settle at 14:56:26 and waited 5 s while every pass ran DECODE bs=1 with
admission_ms=-1; the log cannot say why -- WEG2 X-DEFER speaks only while its
GLOBAL pass count is <= 5 (weg2-4-11 used those up) and the admission's skip
census (sgl-project#971 _admission_decline_note) is not printed on this path.
- WEG2-POST-WAKE-PASS gains waiting=<queue> admit=<the pass's admission
  gate / skip census> (loop_skips(kind=n(first=rid)), gate=...).
- the two branches that skip the admission entirely name themselves there
  (gate=phase_prefill_blocked, gate=congruent_lane_cadence) instead of
  leaving the previous pass's note.
- WEG2 X-DEFER speaks the first defer of every rid as well (the repeats stay
  under the global cap).
Instrument only: no admission verdict, order or X rule changes (over X stays
P's, under X D may take it beside the decode as before).

Tests: test_weg2_wake_tail_nowriter_0928 +2 (the line names the census and
the skip branches; X-DEFER speaks each rid's first defer past the global cap)
-> 9 passed. Tests pinning these lines/paths: the same 15 failures on
e7715b0 (leg2_store_probe_1290, sched_fix7_0908, store_grid_claim_1298
-- red on the base too, byte-identical list), the rest green.
efschu pushed a commit to efschu/htsglang that referenced this pull request Oct 16, 2026
…robe findet die END-ANCHOR-Tails -- sie liest die gemeinsame Aufzaehlung pp_request_locations statt ihrer eigenen vier Orte

Boots 1016_150003 (weg2-6-94) und 1016_151644 (weg2-0-20), derselbe Client-Request nach jedem Neustart:
89664 Tokens, 89600 gecacht, uncached Rest 64. Der Rest passt ganz in einen Chunk, also teilt die
END-ANCHOR-Regel (sgl-project#1233, Anker bei floor_grain(N-1), grain 4) ihn in Body 60 + Tail 4. Mit fnFL2 H42
(multi anchor tails) lebt der Tail in Scheduler.anchor_tails -- absichtlich NICHT im einzigen
chunked_req-Feld (das fuehrte weiter weg2-0-18: 'sgl-project#992 CHUNKED CONTINUATION NOT NAMED rid=weg2-0-18
decision_names=weg2-0-20'). PP1/PP2 mintet denselben Tail aus der weitergeleiteten Geometrie
(ANCHOR-TAILS minted_rids=weg2-0-20 bei slot 1). PP0s slot-2-Zeile nennt den Tail; die sgl-project#631-Row-Probe
in _pp_proxy_frame_pending zaehlte aber nur waiting_queue, chunked_req, den Slot-Chunk und running_batch
-> "not yet locatable" ohne Aenderung -> 4 Laps -> PpRowDeferCapExceeded, RANK_EXCEPTION. Der
Early-Settle (equal/short_satisfied) ist nur der Weg zum ganzen-Chunk-Rest, nicht die Ursache.

Fix: _known = pp_request_locations(self) (sgl-project#946: jede Stelle, an der ein Request leben kann, inkl.
anchor_tails und Slot-Ring) plus der Chunk des genannten Slots. Keine Lap-Grenze geaendert; eine rid,
die an keinem Ort liegt, wird weiter verschoben und endet benannt.

Tests: test_pp_row_probe_anchor_tail_1016.py 3 gruen (weg2-6-94 ueber die Lap-Grenze hinaus
lokalisierbar, weg2-0-20 nach equal-Early-Settle, eine rid an keinem Ort wird weiter verschoben); auf
1f4b3e7 rot 2. Mutant: anchor_tails aus pp_request_locations entfernt -> ROT 1. Regression 9 Dateien
(Row-Probe/sgl-project#1180/#791T/sgl-project#968/sgl-project#946/sgl-project#961/sgl-project#971/H42/K14/244): Kopf 221/1 rot = Basis 221/1 rot (vorbestehend
test_weg2_p_multi_anchor_tails_h42::test_count_arm_uses_the_fresh_admission_count). Kaltstart-Rig s7s8
28/28 GRUEN.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants