Repository navigation
#590 Increase default , track changes in examples and documentation - #971
Merged
Merged
Conversation
Ying1123
reviewed
Aug 8, 2024
Ying1123
enabled auto-merge (squash)
August 8, 2024 00:20
Ying1123
approved these changes
Aug 8, 2024
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
…mentation (sgl-project#971) Co-authored-by: Ying Sheng <sqy1415@gmail.com>
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 28, 2026
…n it was handed Root (window-flip-0828 boot 1, pin 0caab9a): scheduler.py hands the final-extend continuation from self.chunked_req into adder.can_run_list (add_chunked_req appends and returns None); every PPScheduleRefused exit discarded the adder without a restore. Both existing restores are unreachable from that path: _pp_void_own_batch early-returns on 'batch is None' above its restore (#797d = 0 lines in the whole boot log) and _pp_absorb_void_output's restore is keyed on an output expectation a pass that built nothing never made. The request was then in NONE of the four lookup places: 507x 'sgl-project#944 UNRESOLVED told=8192 local=UNKNOWN', 512 voids, sgl-project#801-spin guard death. The class, stated at the junction: ownership of a chunked continuation may pass from scheduler-owned state into the pass-owned adder only if EVERY exit of the pass re-homes it. Fixes: 1. pp_rehome_refused_chunked_req, called from the one funnel _pp_refuse_forwarded_schedule (slot = amended.mb_id, verified preserved by void_pp_admission_decision; fallback _pp_live_mb_id). The park is GEOMETRY-ONLY via new pass_allocated=False: the refusal raises above both the KV allocation and the inflight_middle_chunks increment, so the give-backs would double-free / over-decrement -- caught by the accounting arm during red-first, pinned by test_every_refusal_raise_is_above_the_inflight_increment. Zero tokens lost at this junction; the status quo lost 8192 per recompute. 2. pp_request_locations answers the container shape instead of excepting it away: production's LIST-shaped slot ring made the slot place silently empty for every consumer of the helper. The masking dict fixture in test_chunked_continuation_clamp_946.py now uses the production list shape. 3. The admission skip census is written before all three refusal raise sites, so a refused pass logs its skips instead of 'reason=-'. Tests: test_pp_refused_pass_keeps_continuation_971.py -- red-first 11 failed / 9 passed on the pristine tree, 25 passed with the fixes; the can-fail arm neuters only the restore and flips 3 assertions. Gate: 11/11 neighbouring registered modules green, failure sets before == after == empty, extraction count probe OK on all 11. ruff / codespell: 0 new findings.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 28, 2026
…dow at flip) into merge/flip-window-0828 Terrains disjoint: sgl-project#971 (scheduler.py, scheduler_pp_mixin.py) x roundtrip (cache_controller.py, hicache_phase_binding.py, hicache_storage.py, canonical_page_store.py). Gate on the merge tip recorded in the register before push.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 28, 2026
…it travels Boot-2 ring wedge (R3a root H3): the output ring's intermediate-hop predicates serve the LAGGED slot -- the sender forwards on `if pp_outputs:` (what it took off the wire last iteration), the receiver reads its own `mbs[next_mb_id]`, committed passes earlier -- and sgl-project#951's launched posting is a SAME-SLOT closure that cannot retract a batch already resident there. The uncovered path is the void relay: `pp_void_relay_stop_rank` derives its stop from the RETRACTION structure, so a void that names no retraction (sgl-project#944's zero-offer escape: a rank that lost the request retracts nothing and launches nothing) was forwarded to a rank whose slot for it was empty, taken off the wire by that rank's next ADMISSION receive, stashed, and served POSITIONALLY to a healthy later generation's receive -- which `_pp_absorb_void_output` then emptied on one rank only, leaving the ring one message short for ever. PP2 sat in `_do_recv` unbounded because sgl-project#971's busy wire kept the sgl-project#789 gate early-returning (consumed<posted); boot 1 only survived on a counter another defect had frozen. Fix: the launched CHAIN -- the same per-hop statement sgl-project#951 consumes, kept per generation. `_PP_LAUNCHED_CHAIN_KEY` accumulates one bool per rank on the admission decision (PP0 starts it, every hop appends its own `mbs[mb_id] is not None`, the last rank records without sending, sgl-project#796), rides back on the slot's output/void via `pp_output_payload_with_return_trip`, and `pp_void_forward_payload` consults `pp_void_relay_launched_verdict`: a void travels exactly to the ranks whose own admission-time statement says they launched, never to its source. Absent chain falls back to the legacy rule byte for byte. Log-only `sgl-project#978 STALE VOID` tripwire in `_pp_absorb_void_output`. Red-first, measured 2026-08-28 against the unfixed tree (TheLaggedSlotIsBeyondTheLaunchedPosting): test_the_launched_chain_relay_stop_closes_the_lagged_slot FAILED -- stuck [0,1,2], PP2 parked in output_exchange, PP1 event `pass=7 void_absorbed slot=2` against hazard slot 1 (the mispair), with sgl-project#951 wired in AND CHAN_DICT counters armed (the readiness gate ran and early-returned: boot-2 fidelity, not boot-1's frozen-counter accident). Count check: 1 failed, 1 passed of 2. After the fix: 2 passed; the wedge stays measurable via fix_off (spawn worker forces the verdict back to legacy -- mutant on the hazard direction, reproduces the wedge). The verdict pure function is pinned exhaustively over rings 2..8; three boundary mutants (hand back to source / read own entry / short chain forwards) all fall. Suites: neighbor baseline before == after. Before: 95 passed (801, 791b, 795, 797, 951, idle-void, void-send families, 262s). After: 100 collected, 99 passed + 1 failed, the 1 a harness-stub interface drift in test_pp_retracted_pass_void_797.py (local recorder lacking the new kwarg; repaired, no assertion touched; file then 31 passed). ruff: only the one finding already present on HEAD; codespell clean. sgl-project#981 boundary, checked before push (CPG_DERIVATION_0828 (b)): this fix's predicate does NOT stand on the `_pp_output_expected_incoming` memo. The chain's write (decision recv), reset (top-of-body block) and consumption (decision send) all live inside full body iterations; the resume-slot jump's `continue` sits before the reset block but consumes nothing before the next full body resets. The by-slot chain is written only at a slot's admission and read while that generation is resident. K3's ungated admission recv and the clear-site-behind-skip fragility remain sgl-project#981.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 28, 2026
Root (R3b, window-flip-0828 boot 2): batch_is_full is a PASS-scoped memo (written at the adder's stop condition) stored on the scheduler-lifetime running_batch. Every clear site is either a FINISH path the requeue-without-finish family (sgl-project#797/sgl-project#798/#791b/sgl-project#971) destroys before it runs, or sits behind 'if not running_batch.is_empty()' -- unreachable at exactly running=0, the defect state. Boot 2: three gate=batch_full_or_empty_queue(batch_is_full=1,queue=N) declines at running=0 with free seats and a waiting queue, self-sustaining until an external kill. The #888b relief lies on the path but its verdict producer sits behind the same emptiness guard (registered #962b). Fix: ONE junction at the gate read -- batch_is_full AND is_empty() is re-derived from the seat test (get_num_allocatable_reqs), not trusted. Re-derive, not clear: a genuinely exhausted seat table comes straight back True. Not an eighth clear site: any future requeue path inherits this without knowing the flag exists. In-tree precedent: the retract-RPC handler. #888b relief and the seam hook untouched; arm 2 of the new test pins that #962b's producer stays unreachable at running=0 so the posten stays open rather than silently closed. Tests: test_batch_full_empty_invariant_962b.py, 8 arms -- red-first on the REAL void path (gate declined at running=0 with seats free, note=gate=batch_full_or_empty_queue -- boot 2's own line shape), void family flag hole filled, green-guard for the 8-resident full-table case both sides, can-fail mutant. Baselined modules byte-identical.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 28, 2026
…ny traffic Root (R3a, measured boot 1 vs boot 2 of window-flip-0828): _pp_wait_for_dict_readiness decided progress from the SHARED CHAN_DICT posted/consumed counters. CHAN_DICT is one wire demultiplexed by __msg_type__ (admission_decision / proxy / output). Boot 1's livelock froze ALL traffic, the counters froze, the gate fired twice -- loud death. Boot 2's sgl-project#971 fix (correctly) kept the upstream busy posting admission_decision messages, so consumed < posted held at every poll, the gate early-returned every time, and PP2 sailed into the unbounded output receive -- silent wedge, zero firings. A compensator whose trigger condition was supplied by a foreign defect, disarmed by the foreign fix. Fix: per-KIND accounting riding the EXISTING chan axis as a sub-channel (dict|<kind>, KIND_SEP) -- same single writer, same atomic replace, same sweep; no second mechanism. The drains keep the shared wire counter as sole authority (counting kinds separately there would leave unmatched messages behind -- the module docstring reconciles the two questions). Only the readiness gate reads the kind axis, and only after kind_axis_covers proves the sender labels everything it posts; unlabelled traffic degrades the gate to byte-identical pre-sgl-project#974 behaviour. Helpers module-level, not methods: the method form broke the one-name-at-a-time stand-in holders (797: 31 -> 24, measured), now pinned by its own test arm. Tests: test_pp_readiness_per_kind_974.py, 11 arms in-process with two real PhaseFlipCounters (no gloo pair -- Corpse-F structurally unreachable): boot 2's disarm shape red-first (8 failed / 2 passed pre-fix, the 2 being the directions that must not move), healthy-kind progress unchanged, no-upstream still raises, kind-axis mutant re-arms the disarm, sweep covers sub-channel files. 13 baselined modules byte-identical + 14 counter-importers green, count probes OK.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 28, 2026
…and PP0 names it Boot 3 of window-flip-0828: after a sgl-project#797 void the follower holds a sgl-project#971-parked chunked continuation in self.chunked_req (never in any queue); PP0 cannot see it, its uniform cap is MIN-reduced before occupancy is loaded, and the void's requeue tail-rotates PP0's copy so it never reaches a seat. The follower's add_chunked_req appends its continuation unconditionally, so every forwarded schedule is illegal on the follower: 514 refusals (191 missing + 323 'admitted rid(s) the decision does not name'), 512 voids, sgl-project#801-spin guard death in 2 min. Fix, the register's named sgl-project#968 direction -- the fact rides EXISTING messages, no new wire, no collective: (rid, executed_len) of a parked continuation is stamped on (a) the per-pass decision downstream and (b) the output/void return lap homeward. Named deviation from the design brief: the ring is asymmetric (the last rank sends output DIRECTly to PP0, so a middle parker never gets onto the output lap alone) -- hence both messages, and facts are MERGED per hop, never overwritten, or a middle rank's fact dies one hop before its consumer. PP0 absorbs before the admission loop, REORDERS its own waiting_queue so the ordinary loop admits the rid (reorder, never admit: no second admission authority, no seat pricing -- once named, the continuation is a cap-counted member on every rank: no extra, no missing), and clears the fact when a decision names the rid (self-correcting: the next lap re-stamps if still parked). No prefix is ever reduced (Kein-Doppel- Prefill); all seven helpers module-level (holder lesson, test-pinned). Single-slot hypothesis CONFIRMED (double-sourced): chunked_req is one attribute, overwritten per truncation and cleared on launch -- PP0 cycles it (518 admits / 7 rids, the stuck rid named 2x), the follower pins it (514x). That is why the carry must exist. Tests: test_pp_parked_continuation_carry_968.py, 21 tests -- red-first GENUINE against the unmodified pin 3c470fb in a detached worktree: 18 failed / 3 passed (names == summary 18==18), 21 passed on the fixed tree; legacy arm byte-identical without the stamp; no-double-prefill arm asserts zero prefix reduction; can-fail arm neuters the absorb. Gate: 42 modules BEFORE/AFTER byte-identical 510P/9F, the 9 pre-existing name-identical both sides (prefetch_symmetry_580 7 + ring_commit_973 2 -- the 973 pair is the known gloo hang-shape arms, reproduced solo on the unmodified tree; environmental, posten noted).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 28, 2026
…w speaks Boot 4 proved the sgl-project#968 carry chain worked L1-L3 (absorb ran 514x) and failed at L4 ACT, twofold: the act permutes only waiting_queue, where a parked continuation never lives, and PP0's copy was DROPPED at the cross-slot void restore -- self.chunked_req is ONE field, the per-slot restores write it back to back, last slot wins, and the clear at the old :8382 justified itself with a claim ('re-admitted from the waiting queue') that is false for continuations by this file's own rules. Live witness: boot 4 line 408 clearing 4077b704, one of the 107 missing-rids; 407 of 514 refusals were the parked rid as an unnamed extra. Fix at all four unguarded void writers (:8729/:7449 rehome-before- overwrite, :8823/:7486 requeue-on-clear, checked not asserted: only a healthy parked continuation not already reachable, not finished, never reset -- reset shapes are the disposal loop's, which releases AND queues in one act). scheduler.py:9506/:9545 untouched (exonerated: preempt requeue / assert-guarded mint). ACT unchanged; its docstring's false premise and two dead citations are kept as a NAMED retraction plus the real requeue sites. Writer census: 11, four now guarded, seven sighted clean and listed in the helper docstring. Reachability (CPG/call-graph, standing order): every new site closes at _event_loop_pp_body (:3200 -> :8982 absorb -> :8729/:8823; :3200 -> :3415 own-void -> :7449/:7486; :3430 -> :7745 -> twins); ACT via get_next_batch_to_run:6962 -> scheduler.py:9130. Instruments (standing order, 22 lines, rate-limited first+64th, zero cases PRINTED): sgl-project#968 MINT/FORWARD/ABSORB/ACT (ACT carries carry>0- front=0 with an explicit NOT-REACHABLE clause -- the line that names boot 4 in one pass), sgl-project#968 PARK + sgl-project#971 REHOME-ON-REFUSAL in both previously-silent park actuators, #968b REHOME-ON-DISPLACE / REQUEUE-ON-CLEAR, and #791b/#797d now print released=/kept=/displaced= rid lists (all 514 boot-4 void lines were rid-free). Test evidence (desk, built before the tests-off-critical-path order landed; the boot with instruments is the acceptance): test_pp_continuation_cross_slot_rehome_968b.py red-first GENUINE 22F/9P at b070296 -> 31 passed; 10 neighbouring modules green (968 21, 797 31, 971 25, 798 7, 631 6, 801 14, 973 5, 962b 8, 974 11); one honest update: 797:1578 asserted 'dropped, not retracted' -- the exact belief this commit refutes.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 28, 2026
…nstead Boot 5's third continuation-loss exit was #968b's own gate: the park-shape equality in pp_rehome_displaced_chunked_req refused the UN-parked mid-plan occupant (end=7939 prefix=4096, named verbatim by its own 'sgl-project#968 MINT none' line), returned None, and the caller's next statement nulled the live occupant -- a drop, not a release. The fix that closed the cross-slot displacement carried the class it closed: a predicate assumed where pp_chunked_req_is_reachable already tests. Evidence chain one second wide in the boot-5 slice (R6, double-checked): ADMIT 8192,4096 -> MINT none (last provable hold) -> next line's void restore with chunk parked=False + displaced=None -> rid never on PP0 again; 514 refusals, PP1 spinning 323x on the unnamed extra it took over via the sgl-project#971 route. Fix: the un-parked shape is PARKED first (_park_chunked_prefill_chunk, void-default pass_allocated=True -- its pass was built, so KV rows and the inflight increment are owed back), then falls through to the SAME reachability+queue junction as the parked shape. Zero cached tokens discarded, nothing reset. Parker-side alternative rejected: retiring the orphan discards 7939 tokens against the one-chunk law and treats the spin instead of the drop. Instruments: the formerly-ambiguous displaced=None (empty field vs refused occupant were indistinguishable) disappears for a present occupant -- 'sgl-project#968 PARK route=void-give-backs' plus REHOME-ON-DISPLACE now fire at exactly the line boot 5 went silent. Execution proof (standing order): the strongest available -- boot 5's own log shows this exact line running 514x (every ACT NOT-REACHABLE pass traversed the refusing gate); reachability is measured, not argued. Import smoke green; suites live in the parallel test-agent lane per the tests-off-critical-path order.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 28, 2026
…d acquired Boot 10 (b27d7c2) died on PP0 at 19:58:17 with `AttributeError: 'NoneType' object has no attribute 'unsqueeze'` at mem_cache/unified_cache_components/mamba_component.py:1056, reached via _pp_process_batch_result -> process_batch_result_prefill -> maybe_cache_unfinished_req -> cache_unfinished_req, rid e3d6a9ff11c5460cb115f266f2b187a3. ROOT (class, not effect). The admission loop's revert site in managers/scheduler.py frees `req.mamba_pool_idx` whenever a request is not added to `can_run_list`, guarded by `not req.session`. That predicate is upstream's PROXY for "nobody else already owned this slot" -- correct upstream, where a session was the only carrier of a pre-existing slot. This fork invalidated it: sgl-project#984 (void-park), #968b (rehome-on-displace) and sgl-project#971 (rehome-on-refusal) re-queue requests UNRESET, holding the live mamba slot of a prefill batch that is still IN FLIGHT. The revert then hands a live owner's slot back to the allocator and nulls the field; the in-flight batch's result arrives and dereferences it. The silent direction is the same defect: when the free lands outside the in-flight window, HybridReqToTokenPool.alloc re-acquires a fresh slot with mamba_needs_clear=True -- a ZEROED recurrent state in place of the request's own (sgl-project#767's shape), with no crash and no log line. Both close at the provenance test. FIX, at the one junction. Stamp provenance where the slot is acquired speculatively by an admission round's prefix match -- unified_cache_components/mamba_component.py (COW resume + host load-back), mamba_radix_cache.py, hi_mamba_radix_cache.py (both siblings) -- and free at the revert site only when that stamp is set. The stamp is cleared the moment HybridReqToTokenPool.alloc takes the slot into a batch, and on every release/reset path (free_mamba_cache, _rollback_alloc, release_kv_cache's early branch, reset_for_retract, Req.__init__ default). This is sgl-project#984's own "pages versus claim" doctrine applied to the third increment: the lock ref (sgl-project#990) and the chunk (sgl-project#971) already carry provenance; the mamba slot was the one that did not. It opens neither sgl-project#929 (deficit) nor sgl-project#924 (surplus) -- the stamp is one-shot and dies with the slot. AFFIRMATIVE REPORTING. None of the five mamba free sites named a rid, which is why the specimen could only be reached by elimination. The refused give-back now logs rid + rejection result, so the next boot is the metal proof of the root rather than a yes/no on the crash. BACKSTOP, named as a defect. mamba_component.prepare_for_caching_req now declines retention with a counted, rid-bearing warning instead of dereferencing a None slot. This is NOT the fix and is stated as such in place: it costs the step's retention and re-prefills the chunk, which the standing no-double-prefill order forbids as a routine. It exists because a crash here takes the whole instance while a decline takes one step. SIBLING SWEEP: all five writers of `mamba_pool_idx = None` audited and the stamp cleared at each. gdn_slot_runtime.py:136 is not wired on this build (the gdn_resident_state_slots gate is off, no "armed" line in any boot log) and is left untouched. streaming_session.py's clears stay covered by the retained `not req.session` term. FUTURE CHECK: a new producer of a pre-existing slot cannot reopen this silently -- it either stamps at its acquire (and is given back correctly) or it does not (and is kept), and the backstop's counter names it with a rid if it manages to drop one anyway. Evidence: desk. py_compile + import smoke on all seven touched files; lifecycle probe (Req default False, reset_for_retract clears). Belegstufe: DESK-BEWIESEN, not metal -- boot 11 is the test.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 28, 2026
… its seat too Boot 11 (aae7e3b) reached 3m47s -- sgl-project#991 held, no AttributeError -- and died on `sgl-project#801-spin PP IDLE-VOID LIVELOCK REFUSED` (pp_rank=2, 512 consecutive voided passes, scheduler_pp_mixin.py:8214). The guard is right that the defect is not on rank 2. ROOT. Uniform membership -- "a rank executing a forwarded schedule may admit only what the decision names" -- is enforced at scheduler.py's `pp_not_named` skip for every candidate that comes out of `waiting_queue`, and NOWHERE for the one candidate that never appears in a queue. `add_chunked_req` runs ~250 lines earlier and appends `self.chunked_req` unconditionally, before the decision is consulted at all. MEASURED, 512 byte-identical rounds (21:27-21:31): rank 1 held rid 8a330526c7b9410f963232874adc451b at executed=3332 as its parked continuation, in NO queue on ANY rank, so rank 0's decision could not name it. That site gave it the seat; the decision's own rid (dfd22a9a… / 9528ff20…) could then not be reached; `sgl-project#791 FORWARDED SCHEDULE UNEXECUTABLE` voided the pass; `sgl-project#971 REHOME-ON-REFUSAL` put the continuation straight back into `self.chunked_req`; the next pass was identical. Rank 2 voided every pass behind it until the bound fired. THE CLASS: a compensator that restores the precondition of its own trigger. The cut therefore belongs upstream of the rehome, at the admission -- fixing sgl-project#971 would only move the latch. FIX. At the one junction, gate `add_chunked_req` on the same membership test the queue loop already applies: when this rank is executing a forwarded schedule that does not name its continuation, refuse the SEAT. Not a drop -- the precedent is the sgl-project#906 gate immediately below it: the request stays `self.chunked_req`, keeps prefix, pages and mid-chunk position, and resumes on the first pass whose decision names it. PP0 has no incoming decision, so `incoming is None` there and the default path is byte-identical. SIBLING SWEEP: the two candidate sources into `can_run_list` under a forwarded schedule are the queue loop and this one; the queue loop was already gated. `_note_skip("batch_full_break")` and `pp_not_named` are the two symptoms this produced in the census and both are downstream of it. WHAT THIS DOES NOT CLOSE, stated rather than assumed: sgl-project#968's half. sgl-project#968 makes the parked-continuation fact ride the return lap so PP0 CAN name such a rid, but its actuator `pp_parked_continuation_priority` only REORDERS PP0's queue and is a no-op exactly when the rid is in no queue on PP0 -- which is this specimen. The new log line prints the rid, its executed extent and the decision's full name set, so boot 12 MEASURES whether the continuation ever becomes named instead of leaving it to inference. Evidence: desk. py_compile + import smoke. Belegstufe: DESK-BEWIESEN.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 2, 2026
… group STOP, not a void Root (HANDOVER_1153_0902, PROVEN, pre-existing at the pin 228a66d): scheduler.py _pp_refuse_forwarded_schedule answered PPScheduleRefused ('sgl-project#791 FORWARDED SCHEDULE UNEXECUTABLE', PP1 reached 1 of the 2 rids PP0 named, batch_full_break) with a rank-local compensation: _pp_admission_pass_voided = True + emptied decision dicts -> mixin _pp_void_own_batch -> 'sgl-project#631 ROW-DELIVER BATCH NULLED slot=0 pass_voided=True' (boot_855_weg1b2 log 65000-65004) -> PP1 sent no proxy while PP0's slot stayed set -> nothing carried the void upstream (sgl-project#797 return trip: zero call sites since CUT V; sgl-project#1072 deleted the void relay) -> PP0's blocking _do_recv consumed PP2's NEXT output under this slot's label (log 65119) -> one output ahead for the rest of the boot -> the pp_to_tp arm at 21:42:05 turned the debt into an unproducible output -> sgl-project#980 ObjectRecvStalled 60 s / sgl-project#1071 PpChainRecvStalled 90 s. Same form sgl-project#1071 (169f53c) deleted for _pp_void_retracted_pass; this was the second writer of the same flag. F1 ROOT: the compensation is deleted, the detector is kept, the refusal is a group STOP. scheduler.py get_new_batch_prefill's except now does `raise self._pp_forwarded_schedule_stop(refusal) from refusal`; _pp_refuse_forwarded_schedule (flag write, dict emptying, sgl-project#971 re-home) is removed; the new _pp_forwarded_schedule_stop logs the kept 'sgl-project#791 PP-ADMISSION forwarded schedule REFUSED on rank N' line and returns a RuntimeError formatted by pp_admission_congruence .forwarded_schedule_stop_message: 'sgl-project#791 FORWARDED SCHEDULE UNEXECUTABLE STOP rank={rank} slot={slot} told=[{told}] reached=[{reached}] census={census} local={local} limiter={limiter} running_bs={running_bs} parked={parked} r2t_avail={r2t_avail} headroom={headroom} group_limit={group_limit} batch_full_setter={batch_full_setter} batch_full_at_loop_entry={batch_full_at_loop_entry}: {refusal}' Every probe is guarded (n/a on an unreadable value); the reached rids are recorded after the admission loop before any of the three raises; the batch_is_full setter site is recorded at each writer of the pass (no_allocatable_reqs_gate, count_arm, disagg_prefill_r2t_avail, add_one_req_NO_TOKEN). Group stop mechanism (existing, no new collective): the RuntimeError leaves run_event_loop; run_scheduler_process (scheduler.py 'except Exception') logs 'Scheduler hit an exception' and parent_process.send_signal(SIGQUIT) (+ killpg/kill_process_tree under SGLANG_KILLPG_ON_SCHEDULER_EXCEPTION); peers end in the bounded sgl-project#980 / sgl-project#1071 receives or the barlink dead-peer probe. F2 TRIGGER CLASS (PP0 order): on a forwarded schedule the follower's rank-local seat-count veto is not a verdict. pp_admission_congruence .rank_local_count_veto_applies(scheduled_extents) is True on PP0 and on every non-PP boot (unchanged expression) and False on a rank > 0 executing PP0's decision; the sgl-project#823 count arm, its batch_full_break, and the three pre-loop count gates of the same arithmetic (batch_full_or_empty_queue's batch_is_full half, min_free_slots_delay, no_allocatable_reqs) are gated on it. The physical allocator still refuses (NO_TOKEN -> membership refusal -> STOP naming the numbers). F3 SIBLINGS: mixin _pp_void_pass_without_upstream_launch's writer of the flag is provably unreachable (pp_upstream_void_pending returns False on every path; its final statement is `return False`) -- left with the proof in a comment. The row-authority _row_skip_plan exit now nulls a slot through _pp_null_frameless_slot, which applies the same named sgl-project#1020 'VOID REFUSED ON A LAUNCHED SLOT' guard as the void path; the guard is factored into _pp_slot_holds_unconsumed_launch(mb_id, site) and used by both sites. Matched check (error class: a follower still ends a PP0-launched pass silently): grep -rn '_pp_admission_pass_voided = True' python/sglang/srt -> only scheduler_pp_mixin.py (the unreachable sgl-project#801 writer); chain scheduler.py raise PPScheduleRefused (:11633/:11646, :11606) -> except PPScheduleRefused (:9441) -> raise self._pp_forwarded_schedule_stop (:9472) -> RuntimeError (:9623); the flag's only readers are _event_loop_pp_body (:4519 -> _pp_void_own_batch) and the scheduler.py void guard (:8955), neither reachable from the refusal. Tests (hermetic, CUDA_VISIBLE_DEVICES=""): new test_pp_forwarded_refusal_stop_1153.py (T1 STOP form + no void, T2 the count-veto helper + source pin, T3 the sgl-project#1020 guard on the frameless null, + the mixin:9389 unreachability proof): 17 passed after; on the parent ca4c6b7 (git worktree) it is a collection error (new names). test_pp_refused_pass_keeps_continuation_971.py: 12 tests that pinned the old compensation INVERTED with the withdrawal named in each docstring (24 passed after; the inverted 12 are red on the parent: 16 failed / 8 passed / 1 error across both files). Mutants: M1 flag write restored in the STOP builder -> 2 T1 red; M2a helper returns True -> 1 T2 red; M2b count-arm gate dropped in the loop -> 1 T2 red; M3 guard dropped from _pp_null_frameless_slot -> 1 T3 red. Bounded suite (7 unit/managers files + test/registered/scheduler): before 103 failed / 527 passed / 1 skipped / 27 errors; after 103 failed / 544 passed / 1 skipped / 27 errors -- per-file identical except the new file (+17). ruff check clean on all touched files; ruff format --diff hunk count unchanged vs parent (15/14/3, pre-existing) and the two test files formatted. Evidence tier: DESK-PROVEN. Boot-3 acceptance: grep -F 'sgl-project#791 FORWARDED SCHEDULE UNEXECUTABLE STOP' in the same second as 'sgl-project#791 PP-ADMISSION forwarded schedule REFUSED on rank', and zero 'ROW-DELIVER BATCH NULLED ... pass_voided=True' lines.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 28, 2026
…de the decode NF rc12z22 D TP0 14:56:26-31: weg2-10-16 and weg2-11-18 were released from the sgl-project#1471 settle at 14:56:26 and waited 5 s while every pass ran DECODE bs=1 with admission_ms=-1; the log cannot say why -- WEG2 X-DEFER speaks only while its GLOBAL pass count is <= 5 (weg2-4-11 used those up) and the admission's skip census (sgl-project#971 _admission_decline_note) is not printed on this path. - WEG2-POST-WAKE-PASS gains waiting=<queue> admit=<the pass's admission gate / skip census> (loop_skips(kind=n(first=rid)), gate=...). - the two branches that skip the admission entirely name themselves there (gate=phase_prefill_blocked, gate=congruent_lane_cadence) instead of leaving the previous pass's note. - WEG2 X-DEFER speaks the first defer of every rid as well (the repeats stay under the global cap). Instrument only: no admission verdict, order or X rule changes (over X stays P's, under X D may take it beside the decode as before). Tests: test_weg2_wake_tail_nowriter_0928 +2 (the line names the census and the skip branches; X-DEFER speaks each rid's first defer past the global cap) -> 9 passed. Tests pinning these lines/paths: the same 15 failures on e7715b0 (leg2_store_probe_1290, sched_fix7_0908, store_grid_claim_1298 -- red on the base too, byte-identical list), the rest green.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 28, 2026
…de the decode NF rc12z22 D TP0 14:56:26-31: weg2-10-16 and weg2-11-18 were released from the sgl-project#1471 settle at 14:56:26 and waited 5 s while every pass ran DECODE bs=1 with admission_ms=-1; the log cannot say why -- WEG2 X-DEFER speaks only while its GLOBAL pass count is <= 5 (weg2-4-11 used those up) and the admission's skip census (sgl-project#971 _admission_decline_note) is not printed on this path. - WEG2-POST-WAKE-PASS gains waiting=<queue> admit=<the pass's admission gate / skip census> (loop_skips(kind=n(first=rid)), gate=...). - the two branches that skip the admission entirely name themselves there (gate=phase_prefill_blocked, gate=congruent_lane_cadence) instead of leaving the previous pass's note. - WEG2 X-DEFER speaks the first defer of every rid as well (the repeats stay under the global cap). Instrument only: no admission verdict, order or X rule changes (over X stays P's, under X D may take it beside the decode as before). Tests: test_weg2_wake_tail_nowriter_0928 +2 (the line names the census and the skip branches; X-DEFER speaks each rid's first defer past the global cap) -> 9 passed. Tests pinning these lines/paths: the same 15 failures on e7715b0 (leg2_store_probe_1290, sched_fix7_0908, store_grid_claim_1298 -- red on the base too, byte-identical list), the rest green.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Oct 16, 2026
…robe findet die END-ANCHOR-Tails -- sie liest die gemeinsame Aufzaehlung pp_request_locations statt ihrer eigenen vier Orte Boots 1016_150003 (weg2-6-94) und 1016_151644 (weg2-0-20), derselbe Client-Request nach jedem Neustart: 89664 Tokens, 89600 gecacht, uncached Rest 64. Der Rest passt ganz in einen Chunk, also teilt die END-ANCHOR-Regel (sgl-project#1233, Anker bei floor_grain(N-1), grain 4) ihn in Body 60 + Tail 4. Mit fnFL2 H42 (multi anchor tails) lebt der Tail in Scheduler.anchor_tails -- absichtlich NICHT im einzigen chunked_req-Feld (das fuehrte weiter weg2-0-18: 'sgl-project#992 CHUNKED CONTINUATION NOT NAMED rid=weg2-0-18 decision_names=weg2-0-20'). PP1/PP2 mintet denselben Tail aus der weitergeleiteten Geometrie (ANCHOR-TAILS minted_rids=weg2-0-20 bei slot 1). PP0s slot-2-Zeile nennt den Tail; die sgl-project#631-Row-Probe in _pp_proxy_frame_pending zaehlte aber nur waiting_queue, chunked_req, den Slot-Chunk und running_batch -> "not yet locatable" ohne Aenderung -> 4 Laps -> PpRowDeferCapExceeded, RANK_EXCEPTION. Der Early-Settle (equal/short_satisfied) ist nur der Weg zum ganzen-Chunk-Rest, nicht die Ursache. Fix: _known = pp_request_locations(self) (sgl-project#946: jede Stelle, an der ein Request leben kann, inkl. anchor_tails und Slot-Ring) plus der Chunk des genannten Slots. Keine Lap-Grenze geaendert; eine rid, die an keinem Ort liegt, wird weiter verschoben und endet benannt. Tests: test_pp_row_probe_anchor_tail_1016.py 3 gruen (weg2-6-94 ueber die Lap-Grenze hinaus lokalisierbar, weg2-0-20 nach equal-Early-Settle, eine rid an keinem Ort wird weiter verschoben); auf 1f4b3e7 rot 2. Mutant: anchor_tails aus pp_request_locations entfernt -> ROT 1. Regression 9 Dateien (Row-Probe/sgl-project#1180/#791T/sgl-project#968/sgl-project#946/sgl-project#961/sgl-project#971/H42/K14/244): Kopf 221/1 rot = Basis 221/1 rot (vorbestehend test_weg2_p_multi_anchor_tails_h42::test_count_arm_uses_the_fresh_admission_count). Kaltstart-Rig s7s8 28/28 GRUEN.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.