Repository navigation
Chunked prefill support - #797
Merged
Merged
Conversation
Collaborator
Author
|
Also thanks @vikranth22446 for his help with chunked prefill. |
Ying1123
requested changes
Jul 29, 2024
Ying1123
reviewed
Jul 29, 2024
Ying1123
reviewed
Jul 29, 2024
Merged
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
amote-i
pushed a commit
to amote-i/sglang
that referenced
this pull request
Dec 8, 2025
* Update test_utils.py * Update test_utils.py * Update test_ascend_debug_tensor_input_file.py * Update run_suite.py * Update run_suite.py * Change lint job to run on linux-amd64-cpu-1 * Update lint.yml * Refactor lint workflow for improved configuration * Update daily-build-test-npu-innersource.yml * Fix syntax error in merged PR benchmark workflow * Update pr-test-npu-debug.yml * Clean up blank lines in activation.py Removed unnecessary blank lines in activation.py. * Fix indentation in attention layer methods * Update io_struct.py * Update dbrx.py * Update phi.py * Update test_utils.py * Clean up unused imports in test_vlm_utils.py Removed unused imports from the test file. * Add blank line before main execution block * Add model path to TestLing class * Update test_ascend_llm_models_phi_4_multimodal.py * Update test_ascend_llm_models_smollm_1_7b.py * Update test_ascend_sleep_on_idle.py * Update test_ascend_llm_models_Qwen3_32B.py * Update test_serving_chat.py * Update run_suite.py * Update test_server_args.py * Update daily-build-test-npu-innersource.yml * Update pr-test-npu-debug.yml * Update activation.py * Update torch_native_backend.py * Update io_struct.py * Update phi.py * Update test_utils.py * Update test_ascend_llm_models_Stablelm_2_1_6b.py * Update test_ascend_llm_models_ling.py * Update test_ascend_llm_models_smollm_1_7b.py * Update merged-pr-benchmark.yml * Update run_suite.py * Update daily-build-test-npu-innersource.yml * Create pr-test-npu-debug-1.yml * Update run_suite.py * Update pr-test-npu-debug-1.yml * Update pr-test-npu-debug-1.yml * Create test_ascend_hicache_mha.py * Create test_ascend_sampling_backend.py * Update run_suite.py * Update pr-test-npu-debug-1.yml * Update run_suite.py * Update and rename test_ascend_deepep.py to test_ascend_deepep_mtp.py * Update test_ascend_deepep_mtp.py * Update test_ascend_deepep_mtp.py * Update test_ascend_mla_w8a8int8.py * Update test_ascend_tp4_bf16.py * Update daily-build-test-npu-innersource.yml * Create daily-build-test-npu-opensource.yml * Update daily-build-test-npu-opensource.yml * Update test_ascend_deepep_mtp.py * Update test_ascend_mla_w8a8int8.py * Update run_suite.py * Update pr-test-npu-debug.yml
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 21, 2026
…fix floor back to rank 0 #791c made a downstream REFUSE a proxy for a pass it had narrowed. It detected; it did not prevent, and the frequency data is why that mattered: boot 'unhonourable prefix' fatal width mismatch instr15 661 1 instr16 1651 1 instr17 1718 1 instr17 retracted 1718 times and only ONE differed in width enough for model_runner.py:4178-4191 to see. The other ~1717 were SAME-WIDTH divergences that computed silently, because chunked prefill caps every chunk at the same size, so two ranks running different request sets routinely present equal widths. TRIGGER, measured: retractions are FLIP-triggered. Boots driven with short prompts only, where no runtime flip ever arms, logged ZERO retractions (boot_coh, boot_detval); every flipping boot logged hundreds. The cutover cold-starts the downstream radix cache, so PP0 offers a prefix the downstream has nothing for. BOTH HALVES, decided together as they must be. (b) THE RETRACTION VOIDS THE PASS, NOT THE RID. Three membership outcomes exist and two are physically unavailable: the rank cannot admit the rid (it has no KV for the prefix its upstream reused, and the upstream sent hidden states only for the extend tokens), and the upstream cannot be amended (it sent its decision and launched earlier in the same pass). What remains is to run the pass nowhere. `effective` is emptied and every surviving entry becomes admitted=False, retracted=False -- the third state, which record_return_trip correctly leaves alone (it teaches on `retracted`, clears on `admitted`). _PP_PASS_VOIDED_KEY carries the fact downstream because the entries alone cannot: a rank with nothing to prefill falls through to its running DECODE batch and would pair that with the upstream's prefill batch. (a) THE FLOOR COMES BACK ON THE OUTPUT MESSAGE. PP0's request state is resynced by #791b's existing void-output absorb, which rides the chain-reconciled decision home so the guard learns observed_local and prefix_len_for clamps the next offer -- strictly decreasing, non-negative, terminating. sgl-project#797 additionally puts that payload on a SUCCESSFUL output, the half that lets a floor be CLEARED: sgl-project#796 removed the only feeder for that and #791b restored only the learning half. A GAP FOUND AND CLOSED, or this would have traded corruption for a HANG: #791b's void stops at the first rank. With the retraction on rank r, ranks 1..r-1 also hold launched batches with output receives posted, so PP0 absorbing and forwarding nothing would block them. Pre-sgl-project#797 that shape mispaired instead (the retractor narrowed rather than emptied, so a real output still went out), which is why it never showed. r=2 is reachable at pp_size=3, i.e. on this rig. The void now travels last->0->1->...->r-1 and stops, gated on pp_first_retracting_rank read off the decision the void already carries. No new key. WHAT THIS DOES NOT DO, stated because the acceptance gate depends on it: IT DOES NOT REACH ZERO RETRACTIONS. Expected is ~one per rid whose first post-cutover offer is unhonourable, against 661/1651/1718 -- the count stops being per-PASS and becomes per-RID-per-CUTOVER, because the floor now comes back and clamps the next offer. Zero is unreachable this way: PP0 cannot know a downstream's cache state before it offers, and "offer and find out" is the only channel that exists without a new collective. The counter that MUST be zero with this fix is the #791c divergence tripwire, which proves no narrowed pass was ever created. Reaching zero retractions would require clamping PP0's first offer per rid per flip epoch to told=0 -- sound, ~15 lines in prefix_len_for, but it pays away exactly what prefix caching buys, so it is not taken unilaterally. ITEM 2, and the answer is NO: the two rank-local decode-retraction paths (scheduler.py:7489/:7517 and :7211-7312) cannot produce this class. They CAN diverge in membership -- _update_uniform_pool_budget reduces on tp_cpu_group, world=1 under TP=1/PP=3, so the floors are off, as the boot line at scheduler.py:4733-4740 says. But not SILENTLY: _get_decode_retraction_order (schedule_batch.py:3038-3082) sorts on replicated per-request state under replicated server args, retract_decode pops only from the END of it (schedule_batch.py:2961) so victim sets are NESTED and never disjoint, a decode batch's row count is a strict function of its request count, and retract_decode entered with >1 request always retracts >=1. So every divergence there is a WIDTH divergence, raised loudly every time. Real defect, different root (the reduce group), and sgl-project#797's remedy structurally does not fit it -- voiding a pass repairs per-pass membership, while a retracted decode victim mutates long-lived state. Filed at the site (scheduler.py:7554-7600), not folded in. TESTS. test_pp_retracted_pass_void_797.py: 11 passed (7 three-process gloo arms + 4 pure), 61 s. Three separate can-fail neuters, each a single return-value rebind through scheduler_pp_mixin's module globals IN THE CHILD: blind entries_retracted_by_rank -> 3/3 green assertions fail; victim reports rows=128 foreign_rows=64, the same-width mispair accepted blind pp_pass_should_void only -> 3/3 fail, victim raises #791c PROXY BATCH DIVERGED (detection, not prevention) blind the per-hop launched key -> 1/3 fails, drained=False stashed=1, the stranded proxy corpse Neighbours 631/791/791b/791c/795/796 re-measured AT HEAD: 4 failed / 67 passed; after: 4 failed / 78 passed, failure names byte-identical. Zero regressions, +11. With 791b and 791c together: 20 passed. ruff/ruff-format/codespell identical to HEAD.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 21, 2026
Boot instr19 ran sgl-project#797 on metal and died after 53 s. Both roots are here. #797b -- THE CRASH. A void is a PARK, not a retraction. sgl-project#797's release/re-queue tore down chunked_req, so a voided pass never reached prepare_for_extend while self.chunked_req stayed set with extend_range None, and the next pass died at scheduler.py:5572: AttributeError: 'NoneType' object has no attribute 'end' get_next_batch_to_run <- _event_loop_pp_body:1132 (PP0 08:47:27) One root, two victims, established per rank rather than assumed: PP1 and PP2 carry ONLY "Connection closed by peer" in recv_tensor_dict, and PP0 took a second, consequential gloo error after its peers went down. So it was neither the floor-feedback message nor the void itself -- the void executed correctly. #797c -- THE LIVELOCK, and it falsifies sgl-project#797's own termination claim. All three retractions on instr19 are the SAME rid and `told` GROWS by exactly one chunk: 08:47:26 rid=d4f59edf... told=512 local=0 08:47:27 rid=d4f59edf... told=1024 local=0 08:47:27 rid=d4f59edf... told=1536 local=0 sgl-project#797 asserted the offer is "strictly decreasing, non-negative, terminating". It is strictly INCREASING. Root: local=0 WAS NEVER A MEASUREMENT. _pp_reconcile_incoming_admission looks rids up in waiting_queue; scheduler.py:7075 removes admitted requests from it; a chunked request then lives in self.chunked_req -- so from round 2 the lookup structurally reports 0 for a prefix the rank genuinely holds. Log line 1737 proves the rank had it: all three ranks admitted the first chunk CONGRUENTLY one second before the first retraction. The rank was retracting a prefix it owned, every round, forever. TERMINATION IS NOW PROVEN IN A TEST, not asserted in a docstring -- the specific lesson of this round. PPChunkedOfferLivelock797c drives one rid through repeated offers ACROSS CHUNK BOUNDARIES: told advances exactly one chunk per round, 0 retractions, and the request finishes. Two red arms, including the one that matters: the PARK ALONE gives flat-but-never- finishing, i.e. #797b is NECESSARY BUT NOT SUFFICIENT and neither fix stands without the other. SIBLING STATE, enumerated rather than fixed only where the traceback pointed: seven call sites audited with fixed / safe / named verdicts. waiting_queue removal (:7075) is the one that bites; adder.preempt_list is NAMED, not fixed; several are safe today but become siblings the day they are wired, because a voided pass would spend a chunk that never ran and commit raises on re-commit. TESTS. test_pp_retracted_pass_void_797.py: 23 passed (11 before), 58.6 s. Red-first for #797b: blinding pp_void_keeps_request reproduces the exact traceback line. Neighbour set 4 failed / 90 passed, failure names byte-identical to 6f42d7f and all pre-existing. ruff and codespell identical to 6f42d7f (the 98 scheduler.py findings are pre-existing). OPERATIONAL NOTE, my error not the agent's: the standing sub-briefing line "Qwen bevorzugt" is unsatisfiable while this strand holds the GPU, because the qwen/local-model agent types run on the local htsglang server at 127.0.0.1:30030 -- the exact server the boots kill. Measured: serving_30030 DOWN, router_30099 alive. A Qwen sub-subagent spawned in that window produced one write and nothing for 32 minutes. While the claim is held, sub-briefings must say Sonnet/Haiku; the never-opus/never-fable half is unchanged.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 22, 2026
…ion gate Follow-up to 1ccfaac, prompted by strand 10's b71dc7b, which instruments the install-capable CALL SITE while 1ccfaac instruments the four gates INSIDE the method. The two are complementary and neither duplicates the other, but strand 10's analysis exposed a real weakness in the gate-one line: it reported only "uneven DCP is not active for this dcp_size", and that collapses three different causes into one sentence. uneven_dcp_active refuses in three ways (utils.py:326-330), and they call for three different fixes: * no vector installed yet (ratios is None) * a uniform vector (the even modulo fast path, deliberately not "uneven") * a length mismatch (len(ratios) != dcp_size) MY EARLIER READING NAMED THE WRONG ONE, and that is corrected here rather than quietly overwritten. 1ccfaac's comment asserted the length mismatch -- a three-entry vector against dcp_size 1 -- as the dominant suspect. On a phase-flip boot the real cause is the FIRST: the only install-capable site runs from init_memory_pools (scheduler.py:1429), which is BEFORE build_phase_flip_tp_stack (scheduler.py:1446) performs the first set_cp_token_ratios (phase_flip_boot.py:971-990). At that site there is no vector installed at all to improve on. Whoever may install does not yet have the numbers; whoever has the numbers (post_capture_resize_kv_pool) may no longer install. A line reporting a length mismatch there would have sent the next reader after the wrong fix -- the precise failure mode this instrumentation exists to prevent, so it is worth the extra branch. The installed vector is now printed, so the three are separable from a log alone. Still logging only: every gate returns exactly where it did. Tests: test_the_three_causes_of_the_first_gate_are_told_apart drives all three and asserts the three lines are PAIRWISE DISTINCT, not merely that each contains its own keyword -- three reasons sharing a substring would pass the weaker check. test_the_length_mismatch_gate_names_itself additionally pins "3 entries against dcp_size 1". 30 passed in 8.66s. ruff and codespell clean.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 22, 2026
…le, as a rule The rejected register answers "was this CONFIGURATION tried and settled". It has no answer for a different question that turns out to matter more: is the number I am about to run traceable to work that was later withdrawn. An investigation can be retracted while its outputs keep riding, because nothing connects the two -- the value is just a number in a launch line by then, with no lineage attached to notice. The shipped uneven-DCP token vector is the proof. 29,19,16 comes from sgl-project#602, whose per-card attribution was withdrawn and re-derived more than once (NOTE_602_fill_side_attribution.md:67, "already had to retract twice"), so the slack figures it was proportioned from no longer stand. It has been the active vector on every boot since, and every one of those boots measured its own per-rank capacity, computed a better vector, printed it, and threw it away. The rule, stated so it can be enforced rather than remembered: AN ACTIVE VECTOR MUST NEVER ORIGINATE FROM A RETRACTED INVESTIGATION. "Active" carries the weight. A retracted value that is merely present is harmless -- as a sizing seed it is superseded in-process by the measured optimum before anything serves on it. A retracted value that is PINNED, or that is seeded and then fails to be superseded, is the number the server actually runs, and that is refused. So the gate belongs where a vector becomes active, and the seed path gets re-checked after the install attempt instead of waved through. Matching is layered, because a rule that fires only when the operator volunteers the incriminating fact is not a rule. Mode 1: a declared --*-provenance naming a retracted investigation is refused whatever the value, which is the durable general form. Mode 2: with no provenance declared, the gcd-reduced vector is matched against the values the retraction recorded -- without this the rule would be silent on precisely the vector that motivated it, since the launch shipping 29,19,16 declares no lineage at all. Mode 2 matches gcd-reduced so 58,38,32 cannot dodge it, and a stated clean lineage short-circuits it, so it stays a fallback for an unstated lineage rather than an override of a stated one. A measured install stamps PROVENANCE_MEASURED and is never refused -- the remedy must not be refused by the rule it satisfies. This commit is the register and the predicate. The boot-path refusals that consume them are the next commit; landed separately so the gate arrives with its can-fail evidence rather than alongside a wiring change. Tests: test/registered/unit/planner/test_retracted_provenance_797.py, 21 passed. Can-fail EXECUTED, both directions: neutralising the mode-2 value loop turns 2 red (the shipped vector and its scaled spelling stop being caught); making the predicate ignore its declared provenance turns 3 red. Restored and re-verified 21 passed. ruff check clean and ruff format clean on both files. No behaviour change yet: nothing calls into this module, so every existing path is byte-identical.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 22, 2026
…worker's label
The measured token-vector install has been refused on this rig by its last
conjunct, and the conjunct is a mis-picked member of a pair the codebase
already defines. model_runner.py:512:
self.is_draft_pool_worker = is_draft_worker and not is_phase_flip_tp_stack
The two flags coincide everywhere except on the phase-flip TP stack, where a
draft runner is a draft worker whose POOLS take the target-model treatment.
That runner is the one that owns the real KV pool, and _resolve_memory_pool_config
reaches it precisely because the sizing branch at :7272 keys on
is_draft_pool_worker. Twenty lines further down the install gate keyed on the
other member of the pair and refused the install to the very runner that had
just resolved the pool config.
The exclusion itself is still right and still enforced. A runner with its OWN
draft pool must not install the target's token vector, because its capacity is
not the target's capacity. It is now asked about the pool, which is what the
vector is about, instead of about the worker's label.
MEASURED, not deduced from the source alone. boot_798_0822_0629.log carries
the instrumentation from b71dc7b and 4bf3de1. Six occurrences of the
call-site line, three per stack. The PP stack, dcp_size=1 active_vector=None,
skips with "uneven DCP is not active: no token vector is installed at this
point yet" -- init_memory_pools (scheduler.py:1429) runs before
build_phase_flip_tp_stack (scheduler.py:1446) installs the vector at
phase_flip_boot.py:971-990. The TP stack, on all three ranks:
sgl-project#797 install-capable sizing reached: post_capture_kv_active=False
derived_mode=False token_vector_is_seed=True role='seed' dcp_size=3
active_vector=[29, 19, 16]
No SKIP line follows it. The calibration ran to completion, computed the
better vector, and still printed "restart with
SGLANG_UNEVEN_TOKEN_VECTOR=30,17,17 to raise max_total_num_tokens from 1213952
to ~1356736 (per-rank profiled capacity [636730, 360392, 375560])". The
PINNED-VECTOR warning did not fire, so pinned_vector was False and seed_role
True; allow_install was True by the same line. That leaves the draft conjunct
as the only one that can have been False.
This also settles the shape of the fix. The install-capable site is REACHED on
the TP stack with the DCP layout active and the vector present, so a third
install point is not needed: the site was unlocked all along and one predicate
was refusing it.
Two other candidate sites were excluded by mechanism first, so they are not
quietly still open:
- post_capture_resize_kv_pool cannot grow the pool. _config_from_budget applies
min(max_tokens, cap_tokens) at :6984 and the caller passes
cap_tokens=self.max_total_num_tokens (:2627), so a better vector computed
there is clamped away.
- The flip cutover cannot install a different vector. dcp_compact_pool_rows =
(global_tokens // cp_S + 1) * cp_ratio at layers/dcp/owner.py:181 is
evaluated ONCE at pool construction with cp_S = sum(ratios), so the pool's
physical row geometry is a function of the vector's sum; installing a
different one desyncs the slot space from the allocation.
Tests: test_draft_pool_worker_gate_797.py, 6 passed, new. Can-fail EXECUTED:
restoring the gate to `not self.is_draft_worker` turns 2 red, including the
falsifier that asserts the flip-TP draft runner installs; restored and
re-verified 6 passed. The sibling suite test_uneven_token_vector_role_797.py
goes 30 passed, unchanged in count.
The sibling suite's stubs gained is_draft_pool_worker, mirroring
model_runner.py:512. They model a NON-flip runner, where the two flags
coincide, so every existing case keeps its exact meaning; _run gained an
optional draft_pool_worker to let the new suite drive the divergence. That the
stubs raised AttributeError before this was added is itself the point -- every
case in that suite lives in the coinciding region, which is why the defect
could not be seen from there.
ruff check and ruff format clean on all three files.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 22, 2026
…a retracted study 214a9f6 landed the register and the predicate with nothing calling them. This wires them into resolve_cp_token_ratios, the single resolver every boot passes through, so the rule stops being a fact about a module and becomes a refusal. A PINNED vector whose provenance names a retracted investigation -- or, with no provenance declared, whose gcd-reduced value matches one a retraction recorded -- now raises RetractedProvenanceError before anything is sized. Both explicit doors are covered: SGLANG_UNEVEN_TOKEN_VECTOR and the --rank-kv-ratio a,b,c pin. The falsifier is the vector this rig shipped: 29,19,16 from sgl-project#602, pinned, with no lineage declared, refused by value. A SEED is permitted past this point and warned about instead, and that is the rule applied consistently rather than softened. The rule is about the ACTIVE vector; a seed is superseded in-process by the measured optimum before anything serves on it, so for a seed the question is only answerable AFTER the install attempt. That check is the remaining piece and it is named here rather than implied: a seed that fails to be superseded is precisely the boot this task exists to end, and it must be refused at the install site. Two flags carry the lineage, both published to the environment like the role and for the same reason -- the resolver runs inside the flip's second stack build, where the boot's ServerArgs object is not the one consulted. --uneven-token-vector-provenance is published ONLY when stated, because "not stated" is a meaningful answer: it is what ARMS the fallback value match. Publishing "None" as a string would read as a declared lineage and disarm the gate, which is the failure mode this whole task is about. _token_vector_role gets one definition, env over flag, and an EMPTY env value reads as "not stated here, ask the flag" rather than as "pin". That is deliberate and it is the 2026-08-19 lesson: an empty override silently meaning the default is how a blank SGLANG_UNEVEN_TOKEN_VECTOR rode along for days unnoticed. A gate that switches itself off when handed an empty string is the same defect wearing different clothes. Tests: test_retracted_vector_boot_refusal_797.py, 12 passed, new, all driving the real resolver rather than the predicate -- the sgl-project#182 lesson, where a token-vector honesty guard was correct and sat on a branch no server could reach. Can-fail EXECUTED twice: deleting the refusal call in the env branch turns 5 red; making _token_vector_role hard-default an empty env to "pin" turns the empty-override test red. Restored and re-verified 12 passed. The sibling suite test_retracted_provenance_797.py stays at 21 passed. Regression, matched HEAD, both runs hermetic (CUDA_VISIBLE_DEVICES="") and identical selection: test/registered/unit/distributed/ at HEAD 57778d4 in a clean worktree gives 81 failed / 2997 passed; with this change, 81 failed / 3009 passed. Same failure count, and the +12 passed are exactly this commit's new tests. Sampled failures are pre-existing GroupCoordinator/parallel_state attribute errors unrelated to the token vector, and 81 matches the pre-existing baseline 95fdc54 recorded. Noted honestly: this repo's reporter emits no "FAILED id" summary lines for these, so the sets were compared by count and by sampling rather than id-by-id. ruff check and ruff format clean on the touched files; the two format hunks ruff still reports in distributed/utils.py are pre-existing and outside this change. Default path unchanged: with no explicit vector, no provenance and no retraction match, the resolver behaves exactly as before.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 22, 2026
The retracted-provenance gate refuses a vector whose LINEAGE is bad. This is
the other half: a vector admitted as a SEED made a claim about the future --
"I am a pre-boot estimate and the measured per-rank capacity supersedes me
in-process" -- and a boot where that never happened is now refused rather than
warned about.
That case is not hypothetical. boot_798_0822_0629 reached three install-capable
sizing sites at dcp_size=3 with allow_install=True and role='seed', declined
every one on a predicate about the worker's LABEL instead of its pool, and
served the seed [29, 19, 16] as if it were a decision. Nothing failed. The
advisory printed and was ignored, which is the defect class this task is about:
a path that is logged and never enforced. A warning here would be the same
thing again, so this raises.
Three signals, because refusing on fewer would refuse correct boots:
armed -- a seed was actually resolved (a pin asserts its value and
arms nothing)
calibration -- a site that could REALLY have superseded it was reached
(dcp_size > 1 AND allow_install), so "never superseded" means
declined and not "never had the chance"
superseded -- a verdict was reached
Refuse iff armed AND calibration AND NOT superseded.
Two design points that each prevent a false refusal. The latch is disarmed on
the VERDICT, not on the write: the install site writes nothing when the
measured optimum equals the active vector, so disarming on the write would
refuse a boot whose seed happened to be right. And the latch is process-global,
so a genuine draft-pool worker declining its own install does not trip it --
its target runs in the same process and its verdict disarms the claim for both.
Placed at scheduler.py:1448, after build_phase_flip_tp_stack rather than after
init_memory_pools, because under the flip it is the TP stack that carries the
DCP layout; the earlier position would refuse every flip boot.
CAN-FAIL, EXECUTED, both directions, driving the real sizing method in the
boot-0629 configuration (is_draft_worker=True, is_draft_pool_worker=False):
predicate `not self.is_draft_worker` -> installs [29, 19, 16] -> FIRED
predicate `not self.is_draft_pool_worker` -> installs [29, 17, 18] -> silent
So the gate would have refused the boot that shipped and admits the one that is
fixed. Neutering assert_seed_superseded to return unconditionally turns both
falsifiers red; verified separately.
15 new tests; 84 green across the six sgl-project#797 suites. ruff clean on the new files
and unchanged in count on the edited ones (utils.py 1, scheduler.py 99, mixin
0, all pre-existing); the two codespell hits in utils.py are pre-existing and
unchanged in count.
WHAT sgl-project#797 IS WORTH, corrected. The commit that landed the install fix quoted
this runtime's own projection, "max_total_num_tokens 1213952 -> ~1335296". That
number is not deliverable and should not be cited. Measured on boot
_798_0822_0646: installing [29, 17, 18] in place of [29, 19, 16] moved the
world pool by ZERO tokens, 990656 either way. The phase-flip boot caps every TP
rank at the PP id space (448910 tokens, phase_flip_boot.py:206), clipping rank 0
from its profiled 620560 and making rank 0 the binding rank under both vectors,
since both give it ratio 29. The threshold is cap0 > 29 * min(360392//17,
375560//18) = 605056, about 35 % above today's operating point; lifting it is
the PP cut's problem (sgl-project#723/sgl-project#702), where cut and vector have to be solved
together. The DEFECT remains real on any cut without that ceiling. This commit
also annotates the install log line, whose "-> ~N" is scored on uncapped
capacity and is what made a non-deliverable figure quotable, and drops the
"roughly 10 % of the KV pool" line from the suite docstring.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 22, 2026
…or it produced The seed-liveness gate refuses a boot, so its dangerous direction is a MISSED disarm, not a missed refusal. It recorded "the calibration reached a verdict" implicitly, as "the recorded vector is not None", which means a verdict that carried an empty or None vector would have failed to disarm silently and refused a boot that was doing everything right. Unreachable from today's install site, where `optimal` is always a non-empty list (partition_units, or list(active)). That is the reason to pin it rather than the reason to leave it: an invariant held by a distant call site's good behaviour is not an invariant, and the gate should not need `optimal` to keep a promise it never states. The verdict is now its own boolean. seed_liveness_state() reports it separately from the vector, because a reader that could not tell "verdict, no vector" from "no verdict" would face exactly the ambiguity the gate no longer has. CAN-FAIL, EXECUTED: restoring the `_SEED_SUPERSEDED_BY is not None` test turns test_a_verdict_disarms_even_when_it_carries_no_vector red, and it fails with SeedNotSupersededError -- the wrong refusal, in the flesh. 13 tests in the suite, green. ruff unchanged at 1 pre-existing finding in utils.py, clean on the test.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 22, 2026
…n a send nobody owes
The PP output ring wedged with all three ranks alive and none of them
able to move (specimen /spinning/evidence-665-f1/wedge_802f_1712/, PP=3,
--enable-phase-flip, py-spy of all three schedulers):
PP0 _pp_commit_comm_work <- _pp_commit_pending_req_work (:4071/:2262)
PP1 _pp_recv_dict_from_prev_stage <- _do_recv (:5608/:6069)
PP2 PpChainReceiver.recv <- recv_requests (pp_chain_receiver.py:329)
PP0 is flushing the request chain, which PP1 can only take at the top of
its next pass; PP1 never reaches that top because it is blocked in the
output receive; PP2 waits for the chain send PP1 has not reached. Three
arcs, one cycle, no timeout anywhere.
THE ASYMMETRY. The two ends of the intermediate hop apply unrelated
predicates. `_do_recv` decides to receive from THIS rank's own slot
state; the non-last sender in `_pp_send_output_to_next_stage` decides to
forward on `if pp_outputs:`, which is whatever it received LAST
iteration. The last-rank hop is matched by construction because it
consults `_pp_output_expected_for_slot` -- but that flag is the FIRST
rank's verdict, published for PP0's arc. Nothing publishes the same
thing for the intermediate hop.
WHY THE EXISTING VOID CONTRACT DOES NOT COVER IT. `pp_void_forward_
payload` (sgl-project#797) already forwards a void along this hop and stops at
`pp_first_retracting_rank`, on the argument that rank r and everything
after it has an empty slot whose receive early-returns. That holds for
the VOIDED slot -- `_pp_void_own_batch` empties it. It does not hold for
a slot the resident decode path still occupies, and both void paths keep
resident requests on purpose (`_pp_absorb_void_output` refuses to release
them as a double-free; `_pp_void_own_batch` deliberately leaves
`running_mbs` alone). The specimen shows exactly that: PP0's last act was
absorbing a void for slot 0, leaving `pp_outputs` None, while PP1 logged
`running=1 chunked=1` and re-entered the receive for a slot PP0 would
never send to. PP0 cannot know PP1's resident set, so closing this needs
a per-slot expectation every non-first rank publishes to its predecessor
-- a protocol extension, and not this commit.
WHAT THIS COMMIT DOES. The sgl-project#789 readiness gate is parameterised by wire
kind and the output receive now passes through it. The CHAN_DICT counter
was never proxy-specific -- one counter per wire, demultiplexed by
`__msg_type__` after it comes off -- so the gate reads the same true
statement about the same wire either way; `kind` selects only the
per-(src, kind) inbox peek. The ring is cut at its one cuttable arc: PP1
waits boundedly and refuses by name instead of for ever. It does not make
the missing send appear, and the error text says so.
AN ALIAS, NOT A WRAPPER. `_pp_wait_for_proxy_readiness` is now a
class-level alias for the same function object. About ten stand-in
holders across the sgl-project#631/sgl-project#757/sgl-project#787/sgl-project#789/sgl-project#791/sgl-project#795/sgl-project#797/sgl-project#798 test family
bind that name one method at a time; a delegating wrapper resolved the
second name on the HOLDER and turned 9 green tests into 5 failures.
Measured, then fixed.
Tests: test/registered/unit/managers/test_pp_output_readiness_ring_802.py
-- three real gloo processes, real PhaseFlipCounters, neutering done in
the child (spawn re-executes the module, not the test body).
* red arm, gate neutered: stuck_ranks == [0, 1, 2], specimen reproduced
* green arm: stuck == [], PP1 reports "sgl-project#789 OUTPUT READINESS TIMEOUT"
and "sgl-project#802-ring", PP0 chain-flushed, PP2 requests-received
* false-positive direction: upstream really posts -> receive succeeds
* no-op without counters, so the non-phase-flip default path is unchanged
Mutants, both die: call site removed -> green arm red (all ranks stuck);
gate raises unconditionally -> false-positive test red.
Regression: baseline 787+789 = 9 passed / 0 failed; with this change
791b + 789 + 787 + 802 = 17 passed / 0 failed.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 22, 2026
…by hand --hicache-host-role staging (bfabd2d) refuses --hicache-ratio and requires an explicit --hicache-size. That refusal is only worth having if the number it demands comes from somewhere; a hand-picked --hicache-size is exactly the pool pin the planner's sole authority over the memory budget (sgl-project#584/sgl-project#785) exists to prevent. planner/hicache_staging.py derives it. Two consumers, and sizing for one of them silently breaks the other: * WRITE STAGING, a bandwidth-delay product: a page stays pinned from the moment it is handed to the storage backend until the write acks, so the resident set is drain rate x that residency, widened by a burst margin (write-through arrives in bursts, the drain is steady). * READ LANDING, which scales with CONCURRENCY, not bandwidth: a prefetch takes its destination slot from mem_pool_host.alloc() BEFORE the storage read is issued (hiradix_cache.py's _prefetch path). Sizing from the write side alone yields a tier that is correct and quietly serialises prefetch -- invisible to every capacity metric, visible only as latency. The tier takes the larger, rounded UP: rounding down would emit a tier smaller than the derivation justifying it. NO RIG FIT. Drain rate and latency are PARAMETERS, not constants. The planner probes rather than carrying a measured vector from one box to another, so nothing in this module encodes what this rig happens to do (~0.5 GB/s on its ZFS-backed file tier); that figure appears only as a test input. WHAT THIS DELIBERATELY REFUSES TO DO. If write-through produces bytes faster than the backend drains them, NO finite size holds -- residency is set by the drain, so a faster producer grows the in-flight set without bound. sustainable() answers that question so a caller can refuse instead of emitting a number that cannot be correct. The remedy there is backpressure at the producer, a runtime mechanism and not a size. Note sgl-project#720's ReadBufferPool made the OPPOSITE choice on purpose for reads -- on exhaustion it allocates a fresh uncounted buffer rather than blocking, trading "a bounded spike for an unbounded latency" -- which is right for a prefetch worker and wrong for a write path under a RAM budget, where re-inflating the pinned footprint is the failure the budget exists to prevent. That ring is the next posten, not this one. fits_pinned_host_budget() asks the sgl-project#729 authority and registers the post like every other pinned producer, so a tier that does not fit is refused where it is derived rather than discovered at allocation. Also: docs/dev/NOTE_810_dangling_design_706_c1.md records that DESIGN_706_BOOT cites a DESIGN_706_constraints document (C1/C1a) that exists nowhere in the tree. sgl-project#810's premise is quoted verbatim at the citing site and does not depend on it, but the supporting argument is unrecoverable. Recorded rather than reconstructed: rebuilding it from four citation fragments would read as primary evidence while being inference, and the next reader would cite it as the original. The tasked "add to DESIGN_706-C1" cannot be executed until the target exists. Tests: test/registered/unit/planner/test_hicache_staging_810.py, 19 new cases, hermetic (CUDA_VISIBLE_DEVICES=""). FIVE mutants, all dying on the intended tests: read consumer dropped (write-only sizing) -> 1 failed rounds down instead of up -> 1 failed zero-drain guard removed -> 2 failed burst-margin floor removed -> 1 failed sustainability predicate inverted -> 2 failed Source green again at 19 passed; 28 passed together with the role gate. Suite: test/registered/unit/planner/ is 2889 passed / 1 failed after this change. The single failure is test_rejected_evidence_pins.py's sgl-project#797 evidence assertion, confirmed PRE-EXISTING by re-running it on a clean checkout of dd6967c with these files removed -- same test, same assertion. No pre-existing test changed status. ruff/black/codespell clean. Not wired to a probe yet, and named so it is not assumed: nothing measures the drain rate automatically, so a staging profile still needs the number supplied. Measuring it needs load against a running server, which belongs to another strand's boot window right now.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 22, 2026
…ost nothing allocates `fits_pinned_host_budget()` had no caller and no test anywhere in the tree -- not even in its own module's test file, which imports the other eight names and not this one. That alone would make it dead code. What makes it worth a commit rather than a deletion in passing is its SIDE EFFECT: it called `check_and_register_pinned_post`, i.e. it registered a sgl-project#729 post in the PLANNER process. THAT REGISTRY IS FOR PROCESSES THAT ALLOCATE. It exists to make the second allocation in a process see the first, and it credits earlier posts back against live availability -- `available + already_allocated` -- on the stated precondition that those posts' bytes are ALREADY RESIDENT and therefore already missing from `available` (pinned_host_budget.py, "the already-allocated posts must be credited back", measured 2026-08-17). A planner allocates nothing, so this post is exactly what the same comment goes on to name as "the real hazard": registered and then never allocated, hence credited back without ever having been resident, so the NEXT admission is charged too little and the registry waves through the over-commitment it exists to refuse. sgl-project#550 closed that window for the image posts with `_unregister_image_post`; the same comment adds "a future producer that can leave a post registered with no allocation behind it must do the same". This helper was that producer and did not. Unreached today, so nothing is broken today. It is removed rather than left because the whole cost of the defect is paid by whoever wires it -- and a helper named "fits the budget" is exactly what a future caller reaches for. NOT REPLACED BY A PURE VERSION, and the reason is that the question already has a better answer elsewhere. `ServerArgs._post_hicache_staging_host_ledger` prices the staging tier through `joint_pinned_host_error` without registering anything, jointly with the spill pool, and by the RANK PRODUCT -- every rank allocates its own tier, so the per-rank figure this helper used under-counts a multi-rank boot by the rank factor, three on this rig. Keeping a second, per-rank, planner-side reading of the same question would be two readings free to drift, which is the shape refused in the sibling commit for the file-tier watermark. The module docstring now says where the question is answered and what a planner-side pre-check would have to be (pure, rank-aware) if one is ever wanted. Tests: two cases added to test/registered/unit/planner/test_hicache_staging_810.py, hermetic (CUDA_VISIBLE_DEVICES=""). The property is pinned BEHAVIOURALLY rather than by absence: the full derivation path is exercised and `registered_posts()` is asserted empty afterwards, so a re-added registering helper fails even if it is spelled differently; the second case pins the removed name itself and points a future reader at the replacement. Mutants, 2, both confirmed dying: the removed helper is put back (verbatim pre-removal source) -> 1 failed the derivation itself registers a post -> 1 failed This is the red-first evidence for a deletion: the first mutant IS the old code, and the new guard fails on it. Suite diff empty: test/registered/unit/planner/ is 2889 passed / 1 failed before (16b's recorded figure, the failure being the pre-existing test_rejected_evidence_pins.py sgl-project#797 case) and 2891 passed / 1 failed after = 2889 + the 2 new cases, same single failure id. ruff/black/codespell clean.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 22, 2026
…uard actually sits on The pp_with_spec registry entry cited server_args.py:17583 and :17598. Those lines now carry sgl-project#797's provenance block; the guard itself moved to server_args.py:18958 (if self.pp_size > 1) and the spec assert to server_args.py:18973 (verified by reading the block, not by search alone). The overrides.py:2163 half of the cite is still correct and is untouched. This is the drift test_rejected_evidence_pins.py exists to catch, and the entry's own text records the previous occurrence ('the earlier :16240-16245 cite had drifted'). A rejection reason nobody can follow back to the code is a reason that stops being checkable. test_rejected_evidence_pins.py 1 failed, 5 passed -> 6 passed
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 23, 2026
[sgl-project#797] Read the token vector back after the TP stack is built. Clean merge, sole production file phase_flip_boot.py, untouched by the 808/739/810/814/806 line since merge-base 203ca8a.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 23, 2026
…t#816, sgl-project#810, sgl-project#806, sgl-project#797, sgl-project#790/sgl-project#777, sgl-project#817/sgl-project#820, sgl-project#818 Merge-checklist duty for the nine stages on this branch. Entries follow each section's own house style, and the mechanism text is taken from the commits' measured evidence rather than restated from the ticket titles. §3 KV backing relief + the allocator cap -- UPDATED IN PLACE rather than given a second bullet, because sgl-project#814 and sgl-project#816 are follow-on defects OF the KvRowCap mechanism that bullet already describes: the census reading the withheld block as a leak (340262 of 465190 ids), the lift being reachable only from a cutover (one boot at 26.8% of its id space for the life of the process), and exposure exceeding the backing (417850 rows over 105413 committed, the device-side assert in masked_set_kv_buffer_kernel). §3 HiCache staging write-through ring (sgl-project#810), new bullet, plus its two companion refusals -- the unbounded-file-tier refusal and the boot preflight ledger entry, the latter being why 22.01 GB of MHATokenToKVPoolHost across three PP ranks previously reached the preflight as nothing. §7 BAR1 deadline + loud abort -- appended the sgl-project#818 peer-liveness half to the existing narrative: the gate could wait forever on a peer that no longer exists, and neither Bar1CollectiveStalled (reset by every resolved read) nor defer_stall_for_building_peer (900 s off a build marker) caught it. §12 Robustness canon -- three new families: contradictory-flag (sgl-project#806), read-back-after-construction (sgl-project#797), denylist-of-reasons (sgl-project#817, sgl-project#820). §18.3 hicache staging sizing (sgl-project#810) -- §18's own rule is that a merge adding a reusable module adds its entry in the SAME merge, and this module had none. Records the removal of fits_pinned_host_budget so it is not reintroduced. §18.6 mamba carry instrument (sgl-project#767, gated by sgl-project#790) and flip break-even N (sgl-project#777). No existing entry was contradicted. Checked before writing: none of sgl-project#814, sgl-project#810, sgl-project#806, sgl-project#772, sgl-project#797, sgl-project#790, sgl-project#777, sgl-project#817, sgl-project#820, sgl-project#818 had a catalog entry, and the one sgl-project#677 line (§19.2, RESTORE-NEVER-REBUILD) describes a different mechanism than the sgl-project#677 layout hold, so it is not stale and was left alone. Gates for the tree this documents (hermetic, CUDA_VISIBLE_DEVICES=""): battery test/registered/unit/{managers,planner,server_args,mem_cache} baseline integ @ 78d27da 44 failed c56d238 (through sgl-project#818) 30 failed, 8452 passed 0 new ids, 14 fixed 1c4eadb (through sgl-project#816) 30 failed, 8461 passed 0 new ids, same set test_barlink_abort_gate_liveness_818.py (outside the battery dirs) 10 passed ruff --select=F401,F821,UP037 and codespell: 0 new findings vs the same file set on 78d27da (16 ruff / 6 codespell exist identically on base) docs-only change; codespell clean on the catalog itself
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 23, 2026
…g said it once The #791b ballot ALREADY detects that the TP replicas' waiting-queue heads have parted -- it rides an all_reduce(MIN) that runs once per TP-loop iteration, so the detector and its carrier both exist. On a mismatch it latched _prefetch_ballot_mismatch_logged, logged ONCE PER PROCESS, and fell back to the rank-local verdict. Its own comment called a divergent queue head "a deeper breakage than a divergent prefetch verdict" and then recorded it exactly once. WHAT THAT COST, from specimen /spinning/evidence-816-18f/wedge_0823_055757 (boot 0516, PP=3 with --enable-phase-flip): 05:55:19 sgl-project#797 void storm begins on PP1 05:55:38 this warning fires -- one line per rank, the last the process ever emits about it. PP0 digest 887126098, PP1+PP2 1471852626. 05:55:53 void storm core: 7710 #797d/sgl-project#798 lines in seven seconds 05:56:13 PHASE-FLIP DONE pp_to_tp (epoch 5) 05:56:15 flip back ABANDONED: "live slot set divergence cannot be repaired this round: the group's union reaches row 240831 and the poorest rank has only 1208" 05:56:15 last real forward progress 05:56:18 the three ranks each build a DIFFERENT prefill batch (#new-seq 1 vs 3, #cached-token 0 vs 16384, #queue-req 6 vs 3) 05:57:57 py-spy: PP0+PP1 in the spec VERIFY arm (eagle_worker_v2.py:2246), PP2 in the EXTEND arm (:2151); all three GPUs pinned at 100%, stacks frozen across three samples 06:00:43 external SIGTERM -- the wedge never resolved itself That 05:55:38 line was the earliest evidence in the whole chain, and the latch made it a single event with no duration attached. "It healed after N passes" and "it never healed" are the same silence to a latched logger, and they call for opposite responses. THIS CHANGES NO DECISION. The fall-back to the rank-local prefetch verdict is untouched; prefetch_done_under_ballot is not modified and its behaviour is pinned by test, because "only logging" is a claim rather than a licence. What changes is that the divergence is counted, its onset is reported, a persisting one is re-reported on a bounded cadence, and the RECOVERY EDGE is reported with the length of the divergence that just ended. NOT UNBOUNDED LOGGING, and the same boot is the argument: 7710 void lines in seven seconds is what a per-iteration warning on a hot loop becomes. The cadence reports streaks 1, 2, 4, 8, ... to a cap and then every cap-many passes -- a handful of lines per minute for a divergence that never heals. THE DECISION IS A PURE FUNCTION, AND A SURVIVING MUTANT IS WHY. advance_mismatch_streak lives in prefetch_ballot.py rather than inline. With it inline the recovery edge could only be checked by grepping the shipped source, and a source grep cannot tell a live branch from `elif False:` -- measured: mutant M2 (recovery edge disabled) left all 14 tests GREEN. After extraction the same mutant kills two tests that drive the machine directly. Same lesson as the sgl-project#820 M5 survivor: a probe that measures presence is not measuring reachability. Tests, hermetic (CUDA_VISIBLE_DEVICES=""): 20 passed, 7.7s. RED-FIRST on unchanged 587e4c2 in a scratch worktree: 10 of 14 red (before the state-machine class was added). The 4 green are the "nothing decides differently" pins plus the detector's own can-fail arm, which hold on base by construction. Noted as a limit, not a strength. Mutants, after extraction, each killed by the arm that owns it: M1 latch restored DEAD (1) M2 recovery edge disabled DEAD (2) -- SURVIVED before extraction M3 cadence logs every pass DEAD (4) M4 total resets across recovery DEAD (1) Regression: every registered unit test touching prefetch_ballot or the uniform reduce -- 40 failed / 65 passed on branch, 40 failed / 45 passed on unchanged base (the 20 difference is this commit's new file). Identical failures, all pre-existing holder drift. Zero regressions. black clean; ruff (F401,F821,E731) and codespell report the same pre-existing hits on base and branch, none in the changed region. isort wanted to renormalise this file's whole import block, which is pre-existing disorder and not this change; reverted, so the diff is the block and nothing else (0 import lines touched). Root cause of the wedge remains OPEN. This makes the earliest signal in the chain legible; it does not stop the divergence. Acting on the mismatch (refusing admission, or a uniform named abort off the already-agreed reduce) needs metal and belongs to the 18 lane.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 23, 2026
Wave 3, stage 2 of the batched-window tree. WINDOW-QUEUE ticket W3 (strand 20a, preflight_pass=Y: 14 new + 31 passing, red-first over three gloo processes, 0 new ids). One commit, base 481c732 (the wave-1 head): 49f39b1 [sgl-project#801] Every slot carries exactly one message, void included THE DEFECT IS A DEFAULT, NOT A MISSING BRANCH, in the commit's own words: `pp_first_retracting_rank` answered None for two unrelated states -- a decision naming no retraction, and a void carrying no decision at all -- and sgl-project#797 read both as "stop forwarding here". On a ring, "stop" is the one answer that cannot be taken back, because the successor's blocking receive is already posted and no later pass can satisfy it. `pp_void_relay_stop_rank` turns the two absences into a ring position (pp_size - 1, one hop short of the void's own source) so the ring becomes a strict relay: a non-last rank that took one message off the wire for a generation puts exactly one back on it, void included. THE CROSS-SURFACE, CHECKED RATHER THAN ASSUMED. This branch's base predates ALL of wave 2, and it edits scheduler_pp_mixin.py -- the same file the already -merged fix/801-admission-recv-liveness edits. Two different sgl-project#801 branches in one file is exactly the case where a clean auto-merge is not by itself an argument, so the hunks were compared instead of trusted: this branch (old side) :846-903 (pp_first_retracting_rank, pp_void_forward_payload), :5883 (new method) wave 2 on the line :34-41 (import block) :4331-4337 (sgl-project#821's recv marker inside _pp_recv_typed_dict) Nearest approach ~1546 lines, and they are different functions. The wave-2 import hunk is an isort REORDER -- RelayPayload moves above phase_flip_counters, nothing is removed -- which was the one way a textually clean merge could still have broken this branch's new code by deleting a name it needs. The two compose rather than collide, and the composition is the useful part: sgl-project#821 marks the one place a PP rank can disappear silently (the blocking dict receive, marked as a DURATION before it blocks), while this change guarantees there is always a message to receive. One instruments the wait; the other removes the reason to wait forever. Verified on the merged tree, not on the diff: pp_void_relay_stop_rank present 4x and _pp_blocked_recv_since present 2x in the merged scheduler_pp_mixin.py, and the merge diffstat (106 / 64 / 596) is identical to the branch's standalone diff, i.e. the merge took the change whole with no fallout in the marker region. GATE. Battery test/registered/unit/{managers,planner,server_args,mem_cache}, hermetic under CUDA_VISIBLE_DEVICES="", one battery at a time. baseline (tip 4f2072a) 7 failed, 8585 passed, 1852 skipped, 887 s stage 1 (W1+W2) 7 failed, 8638 passed, 1852 skipped, 868 s this stage 7 failed, 8653 passed, 1852 skipped, 922 s NEW failure ids NONE (comm against the baseline list is empty) fixed vs baseline 0 The +15 over stage 1 are this stage's own cases; the failure set is unchanged and is still exactly the seven known ones in test_pp_flip_slot_hold_631.py. The gate is not vacuum-green: this stage's new suite, test_pp_void_send_contract_801.py, lands under test/registered/unit/managers and is therefore actually collected, as is the modified test_pp_retracted_pass_void_797.py. Coverage is full rather than assumed: this venv has datasets 5.0.0, both battery logs carry zero collection errors, and an explicit --collect-only over test/registered/unit/managers reports 3436 tests collected. (The W4/W5 strand reported that ITS worktree cannot collect that suite at all because 92 modules import datasets, which is not installed there -- that limitation is not present in this gate.) No boot. This tree is the input to ONE batched window.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 26, 2026
…alue match sgl-project#797 layered two matching modes and returned out of the first one. A declared provenance naming an investigation that is NOT in the retraction register hit `return by_investigation(declared)` -> None, and mode 2's value match never ran. So the shipped 29,19,16 -- from retracted sgl-project#602, the vector the whole gate exists for -- passed under any unrelated lineage. Not an exotic input. The env twin SGLANG_UNEVEN_TOKEN_VECTOR_PROVENANCE outlives the process that set it, so one earlier launch in the same shell disarms every later one. The gate protects the tree-spec-DCP class from serving on withdrawn evidence; a value-blind gate protects nothing. FIX DIRECTION: the modes ADD, they do not replace. Mode 1 may convict a vector whose value looks innocent; it may never acquit one whose value does not. A declared, non-retracted provenance now falls THROUGH to the value match instead of returning. SCOPE, and the boundary is deliberate rather than an oversight. PROVENANCE_MEASURED keeps its short-circuit. It names no investigation -- it asserts that this boot profiled the value itself -- so a value match there would refuse a measurement for resembling the withdrawn estimate it replaced. That exemption is sgl-project#797's own and is tested directly at test_retracted_vector_boot_refusal_797.py::TestWhatMustNotBeRefused797; widening past it would have turned a green sibling red, which is a different ticket, not this one. Named here rather than changed silently. RED FIRST, both axes. Before the fix: 6 failed / 6 passed -- the 6 red are the new-behaviour cases, the 6 green are mode 1, the no-false-positive side and the measured exemption, so the file is not blanket-red. After: 36 passed across the new suite plus both sgl-project#797 suites (boot refusal, seed liveness), i.e. the widened refusal regresses neither sibling. THE MUTANT, as the ticket names it: provenance = 'sgl-project#797' (real, not retracted) over vector 29,19,16 (retracted). Caught, at the predicate and at the boot path through resolve_cp_token_ratios -- sgl-project#182's rule, a gate no boot arrives at is not a gate. The gcd-written form 58,38,32 is caught on the same path, so the fix is not one rewrite wide. CLASS SWEEP. The shape is "a declared lineage short-circuits a value check". planner/rejected.py, the register next door, has no provenance concept at all and cannot carry it. distributed/utils.py:690 reads like the same shape and is not: it is the env-over-flag fallback that PRODUCES the declaration, and None there arms the value match rather than disarming it. retracted.py was the only instance. Two comments the fix made false are corrected in the same commit: the module docstring's "mode 1 short-circuits before any value comparison happens" and mode 2's description as the UNDECLARED-provenance mode.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 27, 2026
…ker per abort The abort path is the leading candidate for the rank-local slot divergence behind the PP output-ring wedges, and the code says why itself. `abort_request` carries sgl-project#631 pin 4 verbatim: "an abort applied on one rank before its peers DIVERGES THE REPLICATED LIVE SET mid-flip" -- and the guard that prevents it is conditional, `if window is not None and window.active`. The `phase_flip_abort_window` is activated only on arming and drained after the cutover, so OUTSIDE an armed flip `_abort_request_now` runs directly, on whichever rank processes it, in whichever pass it lands. The hazard is recognised, named, and closed only inside the flip window. THE CHECK COULD NOT BE MADE, WHICH IS THE POINT OF THIS COMMIT. Grepping the three wedge logs for an abort near the onset returns nothing -- and that nothing is worthless. Every marker inside `_abort_request_now` is `logger.debug` and the boots run `log_level='info'`: measured, ZERO DEBUG lines in boot_accept2e0827_0827_0454, boot_accept2e0827b6_0827_0613 and boot_802f_staged1_0822_1716. So "no abort before the wedge" is a LOGGING GAP, not a finding. This is the marker-absence trap sgl-project#843 hit on this same path ("the only refusal marker was a logger.debug on a boot running log_level='info', so it could never appear"), and reporting the null as evidence would have been the second instance. ONE SUMMARY LINE, not the per-request ones. `abort_all` can name the whole live set, so promoting those would trade a blind spot for a flood -- the sgl-project#801 void-streak shape in reverse. This says an abort happened, when, whether the flip window deferred it, and what it targeted, which is everything the discriminator needs to place it against a wedge onset. `deferred` is now read once into a local and used for both the log and the branch, so the line cannot disagree with the path taken. WHAT THE SAME SWEEP DID FIND, recorded because it is real and does NOT support the abort hypothesis: * Boot 1's wedge onset (last progress 05:08:59) is preceded by EIGHT SECONDS by `sgl-project#905 HOST-POOL DOUBLE-FREE about to raise ... 8192 of 8192 index(es) are in range but not allocated, span [3530, 14937]` on all three ranks, and the `[sgl-project#703 flip-writeback]` drain fails with it. Boot-1-specific: zero such lines in boot 6 and zero in 1712. The magnitude is the same 8192 the sgl-project#927 hit prefix carries, on the HOST pool rather than the device one, which is worth linking rather than filing twice. * Specimen 1712 shows a rank-local admission-congruence retraction (`sgl-project#797 PP-ADMISSION pass voided on rank 1 ... told=147456 local=139264`) -- that is the in-loop site, already covered by the void relay whose default sgl-project#801 widened. * So the three wedges do NOT share one proximate cause in these logs, and the single-class assumption should not be carried further without evidence. NO BEHAVIOUR CHANGE. The branch is byte-identical; only the marker is new. Import smoke + wiring check green (the marker precedes both branches, and the branch reads the same `deferred` the line printed).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 27, 2026
…t causes is bounded THE CLASS, AND THIS IS ITS THIRD INSTANCE. `_pp_reconcile_incoming_admission` resolves a rid through a chain of lookups and, on a total miss, wrote 0 into `local_match_lens`. `reconcile_pp_admission_decision` then read that 0 as a MEASUREMENT and voided the pass. #797c patched the `chunked_req` miss, sgl-project#798 patched the wrong-slot miss -- each added a lookup and left the miss answering with a number, which is the only reason each looked like a fresh defect instead of the same one again. sgl-project#944 is the RUNNING BATCH miss, and it is where a request lives once its chunked prefill finishes: not in the waiting queue, not in `chunked_req`, not in the slot's chunked req. `self.running_batch` is on the same object and read elsewhere in the same file; this resolution chain referenced it zero times. Measured under real agent-shaped load (long SHARED prefixes): 2106 `unhonourable prefix` events, 2107 voided passes, and a three-rank hang 35 s after health, watchdog kill at 300 s. Live py-spy at wedge onset, not the post-mortem: PP0 computing, PP1 blocked in `_pp_drain_voided_proxy`, PP2 blocked in `hicache_collective.join`. WHAT LANDS, AND WHY IT IS ONE COMMIT AND NOT THREE. 1. `running_batch` is the fourth lookup, and a rid found in NONE of the four is `UNKNOWN_MATCH` (-1), never 0. The sentinel idiom is adopted, not invented: `tp_head_congruence._ABSENT_MATCH` is the same -1 one file over. 2. `PPAdmissionEntry.unresolved` -- a field of its own, on the wire. Encoding the miss as a special value of `observed_local` would be the same class one level up: that field's readers treat it as a LENGTH and feed it to `_learned_floor`, which clamps the next round's offer, and a floor learned from a number nobody measured is exactly the defect. It crosses the wire because the rank that OBSERVES the miss is never the rank that can ACT on it -- only PP0 chooses `told`. 3. `told <= 0` is honourable unconditionally. A zero offer demands no prefix reuse, so no lookup result -- not even a failed one -- can make it unhonourable. This fell out of the arithmetic for free while a miss answered 0 (`0 >= 0`); at -1 it stops falling out, and leaving it implicit retracts the FIRST, congruent round of every request. For a rank that did resolve it is a no-op (`local >= 0 >= told`). 4. `UNRESOLVED_DEFER_CAP = 3` with a per-rid count in `PPAdmissionCongruenceGuard`. THIS IS WHY (1)-(3) COULD NOT SHIP ALONE: `_learned_floor` is what damped the re-offer, it is fed from `observed_local`, and a miss must not set it -- so the sentinel by itself makes the 2106-loop WORSE than the 0 did. The old 0 was a false measurement, but it at least clamped. At the cap PP0 emits ONE loud refusal naming the rid and all four lookup locations, and pins the next offer to `told=0` -- the only offer honourable without a measurement, hence the only terminator available once the measurable one is gone. The count clears only on a pass that actually SERVED the rid (sgl-project#552's lesson: a defer that resets its own counter makes the bound unreachable). `<= 0` disables the bound, so it can be neutered on its own in a can-fail proof. THE GROUP DEFERS, NOT ONE RANK. A defer that only one rank takes IS the next divergence. Downstream ranks only REPORT `unresolved`; the pass is voided by the existing sgl-project#797 mechanism, which is already group-uniform (`pp_pass_should_void` ORs the incoming flag and never clears it). Whether to defer again or escalate is decided once, by PP0, from a count that rode the wire. No new collective, no new send. TESTS. New, `test_pp_unresolved_defer_cap_944.py` (22 tests + 3 subtests): the told=0 hoist, the two populations separable on the wire and through the codec, the cap's counting/escalation/clearing/per-rid scoping, and TERMINATION with a can-fail arm that neuters ONLY the cap and shows the offer standing at 4096 for 23 rounds with nothing admitted. New, `test_pp_unresolved_group_defer_gloo_944.py` (4 tests, 3 real gloo processes, deadline-bounded): (a) one rank's miss defers the group and then resolves, with no floor invented on any round and the sgl-project#944 line logged where the sgl-project#791 line must not be; (b) unresolved everywhere -> exactly one loud refusal, `told` ends at 0, served exactly at the cap, never a hang; (c) the danger mutant (`observed_local=0` + `unresolved=False`, the exact pre-sgl-project#944 shape) is caught. (c) also records that the MUTANT TERMINATES FASTER than the fix -- the false 0 IS a clamp, which is why this defect twice looked like it was working, and why "it terminated" is worthless as evidence here. `TheConsumerSweepRatchet` is the class fix's future half. #797c and sgl-project#798 were each fixed by adding a lookup, and neither asked who READS the value; that is the only reason the same defect survived twice. It pins the file set reading `observed_local` and `UNKNOWN_MATCH`, in src AND test. The four contract inversions this change required were found by grep; the fifth will be found by a red test. Four contract inversions in `test_pp_retracted_pass_void_797.py` and `test_pp_reconcile_slot_blind_798.py`, each with the reasoning recorded at the site and none deleted: they pinned the miss as the specimen's exact 0, correct while a miss was SPELLED as a measurement. The behaviour they reproduce is unchanged -- still retracted, still absent from `effective`. One test renamed (`..._still_reads_as_zero` -> `..._is_reported_as_unresolved`): its name outlived its contract by one ticket and would have kept teaching the sentence that cost three boots. TEST RESULTS. Frozen before/after over the derived blast radius (every registered module mentioning `pp_admission_congruence`, `PPAdmissionEntry`, the wire codec, the three reconcile/send/recv mixin methods, `observed_local`, or the return-trip pair), each module ALONE IN A FRESH PROCESS, identical script both sides, baseline in a detached worktree at 70bbd98: BEFORE 20 modules 209 passed 0 failed AFTER 22 modules 235 passed 0 failed Every shared module matches its baseline count exactly; the two deliberately red pins are green at their original assertions and 797 is back to 31/31. Extraction count probe passes on both sides (extracted FAILED names == summary failed count). ruff (F401,F821,UP037), ruff format and codespell clean on all six touched files.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 28, 2026
… it had to prevent DETERMINISTIC, 2/2, AT EXACTLY 7 BATCHES. Pin dc4895e, boots boot_943bx_dc4895e1dc_0828_000240.log and _001113.log, no CUDA error involved: scheduler.py:9286 in _get_new_batch_prefill_raw assert self.chunked_req is None AssertionError immediately after `sgl-project#798 PP-ADMISSION pass voided on slot N` and `#797d own pass voided on slot N`. THE ASYMMETRY. `_event_loop_pp_body` runs two voids and calls the second "AND THE MIRROR OF IT". They were not mirrored. sgl-project#797 (this rank retracted) sets `_pp_admission_pass_voided` in `_pp_void_retracted_pass`, BEFORE `get_next_batch_to_run`, so scheduler.py's guard refuses the pass and it builds nothing -- which is what that guard's own comment demands: "The retraction voids the pass, so the pass must build nothing at all." sgl-project#798 (this rank's UPSTREAM did not launch) set the same flag AFTER the call, so its pass ran the whole of `_get_new_batch_prefill_raw` -- advancing `chunked_req`, possibly adopting a fresh `new_chunked_req`, taking lock refs, emptying the waiting queue -- and was then unwound retroactively by `_pp_void_own_batch`. THE UNWIND CANNOT BE COMPLETED, so this is a guard and not a bigger unwind. `_get_new_batch_prefill_raw` reaches `_retract_decode_and_requeue` (the sgl-project#679 relief ladder, the #888b seat yield), which sends `AbortReq` to the tokenizer over `ipc_channels` at scheduler.py:9626-9632. A message already delivered to another process is not scheduler state and no handler can put it back. Every other unrestored item is a matter of writing more restore code; this one is not. THE CONDITION WAS KNOWABLE EARLY ALL ALONG, which is what makes the guard possible. `_pp_upstream_launched_incoming` is written by `_pp_recv_admission_decision` (scheduler_pp_mixin.py:5482), whose own docstring records the placement -- "Positioned in `_event_loop_pp_body` strictly BEFORE `get_next_batch_to_run`" -- and the call site agrees (:1959 receive, :2073 plan). One shared predicate, `pp_upstream_void_pending`, is now read at both moments, so the refusal and the forward cannot drift apart. The void is still FORWARDED from where sgl-project#798 always forwarded it. CARRIED, NOT DROPPED. Once the guard empties the slot, `self.mbs[mb_id]` is None on exactly the passes the sgl-project#798 site used to find non-empty, so two instruments would have gone quiet unnoticed: * the sgl-project#801-spin livelock streak, which raises at 512. The guard now records `_pp_upstream_void_withheld_work` (this rank HELD work and was refused anyway) so the streak counts what it always counted. It is a pre-plan superset of "derived a batch" -- the safe direction: it can make a genuine streak visible sooner, never hide one, and an idle rank still clears it. * `_pp_idle_void_suppress_log`, whose contract is that it "can never outlive this pass". Its consume sat AFTER the empty-slot early return, so it would have leaked a True into the next, unrelated sgl-project#797 void and silenced a record nothing asked to silence. Read and cleared at the top now. has_chunked_req DELETED, not left to mislead again. `add_one_req` carried it as a parameter its body never read; it only ever forwarded to `add_one_req_ignore_eos`, and upstream removed that last consumer in 8cc7726 ("Super tiny remove unused argument"). Reading it as the guard against a second chunked request cost a boot window. What actually holds that invariant is budget arithmetic, and that is now recorded at the assert -- including its measured reachability: witnesses under /spinning/evidence-665-f1/witness_951/ drive the real PrefillAdder into three states where `add_chunked_req` returns the request while leaving `rem_chunk_tokens` positive, and a mid-pass replenishment of `rem_total_tokens` then lets the loop mint a second chunked request. witness_941_d2 is the negative control and does not reproduce without it. That general case is a SEPARATE posten -- it needs no PP void and its danger direction (wedging a request mid-prefill, the sgl-project#858 shape) needs its own analysis. The assert stays: it is the honest watcher. TESTS. test_pp_upstream_void_before_formation_951.py, 11 tests, red-first: 3 fail on dc4895e and pass here. The five over-fire arms (first rank, pp_size<=1, gapped wire, healthy upstream, healthy chunked continuation) are green in BOTH states, which is what makes the red mean something -- a guard that over-fires voids every pass on PP0 and serves nothing at all. Two mutants on the carried instruments, each killed by exactly its own test: consume-after-return revives the suppress leak, emptiness-only revives the streak reset. Desk gate scripts/gate_tier2_partitioned.py, frozen both sides on this tree: BEFORE 4757 passed / 2 failed / 18 skipped, AFTER 4768 passed / 2 failed / 18 skipped. Delta +11 is exactly this commit's new tests. Identical failure set both sides (test_collective_family_siblings_610.py, 2 genuine, pre-existing at the pin); count check 2 == 2. The lane shift (wide -15, narrow -30, serial +56) is the gate's own sha256 rule demoting the four touched test modules to the serial lane. ruff parity exact per file (105/105, 3/3, 21/21, 1/1) with no finding on an added line; black run only on files that were clean at the pin. NOT BOOTED. Boot half is /spinning/gpu-arb/TICKET_951_WINDOW.md.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 28, 2026
… was in the tree ROOT, per-rank coverage of window-958 boot 2 (pin 78d030e) plus both boot logs. Under pure PP the HiCache storage-prefetch veto is a RANK-LOCAL verdict: prefetch_ballot.prefetch_done_under_ballot returns the local value when the ballot is None, and the ballot rides _update_uniform_pool_budget's reduce on tp_cpu_group -- a group the PP loop never passes, and which has world 1 under pp_size>1 in any case. The ranks therefore decide independently. scheduler.py :9048 (`continue` after _note_skip("prefetch_pending")) is the LAST line PP0 and PP1 both ran; :9050 is the first PP0 ran ALONE. Everything downstream -- add_one_req, `self.chunked_req = adder.new_chunked_req`, the unconditional stash at :7010-7011 -> stash_chunked_request:5449 -> mem_cache/common.py:169 -> cache_unfinished_req -- is PP0-only. PP0's radix tree gains a 1250-token prefix PP1's and PP2's never received. From there it is self-sustaining. PP0 matches its own tree and offers told=1250; PP1 measures local=0 HONESTLY against its own; sgl-project#791 retracts, sgl-project#797 voids, the requeue resets the REQUEST and nothing resets the TREE. PP1 can only acquire the prefix by running the batch it is refused for lacking the prefix. WHY THE EXISTING BOUND COULD NOT REACH IT. PPAdmissionCongruenceGuard is, in its own words, "RID-SCOPED, ONE-SHOT", and its termination argument is per rid: each new retraction for THAT rid lowers THAT rid's floor. The argument is sound and silent about the POPULATION. The shortfall is a property of the TREE, so every fresh rid over the same prefix starts unclamped and buys its own voided pass -- six distinct rids in one second, all told=1250 local=0, which is why _learned_floor was measured RUNNING and LOWERING on PP0 and never bound. FIX: the same actuator, scoped to the prefix the offer was made over. No new mechanism and no new collective -- the recorded fatal (the 2026-08-17 HiCache ack-count reduction) rules out a collective on this path, and a group-uniform INSERT is structurally impossible: the radix value is a tensor of that rank's own KV slot ids, dereferenced as real memory by four consumers (allocator free at unified_radix_cache.py:1713, evict at full_component.py:96, HiCache backup at :2284, write-back into req_to_token at :1325), and the GDN mamba component donates a live state slot a non-computing rank does not have. offered_prefix_key() names the prefix by a blake2b fingerprint of its tokens -- NOT hash(), which is PYTHONHASHSEED-salted and would disagree between the very ranks this keeps congruent (tree_congruence's constraint 3, same lesson). Length is mixed in so a prefix and its extension cannot share a floor. prefix_key=None leaves the pre-sgl-project#963 rid-scoped path byte-identical. PRIOR ART, gated rather than duplicated. #616g's group-MIN reduces available_size(), a different quantity, and is switched OFF in the PP phase (tp_cpu_group world=1). sgl-project#823's tp_head_congruence MINs the right quantity but only REORDERS, and its enforcer_gate returns GATE_OFF_TP_WORLD_OF_ONE here. sgl-project#825's tree_congruence detects exactly this divergence and emitted 0 in BOTH boots: in the PP phase on_round is reached only via scheduler_pp_mixin.py:2477 with require_armed_and_parked=True, so it samples only at an armed flip -- which the livelock prevents. That is this instance's compensator-reachability gap. RED-FIRST, and the first red was rejected as worthless: all six cases failed on the SIGNATURE (TypeError), not on behaviour. Four mutants were then run against the finished fix and their errors READ: consultation withdrawn -> assert 6 == 1 (boot 2's six voided passes, exactly) learn withdrawn -> assert 6 == 1 clear-on-serve withdrawn-> assert 0 == 1250 (permanent loss on a healed prefix) key-blind clamp -> assert 0 == 1250 (loss on a prefix nobody reported) The last two are the danger direction: this fix must never discard a prefix every rank holds. It clamps only against an OBSERVED shortfall on a SPECIFIC prefix, and clears the moment the group serves that prefix, so the cost is one voided pass once rather than one per rid for ever. Double-prefill law: the requeue loses 0 today and still loses 0; what changes is that it terminates. An UNRESOLVED miss (observed_local=None) teaches the prefix floor NOTHING -- the stakes are strictly higher than for the rid scope, since a floor invented from a number nobody measured would cap every request over that prefix. The sgl-project#944 consumer ratchet CAUGHT this file on its first gate run and is now registered. NUMBERS, desk gate scripts/gate_tier2_partitioned.py, CVD="" : BEFORE 2 genuine (test_collective_family_siblings_610.py x2), 685.00 s AFTER 2 genuine (the same two), plus 14 new tests green count probe: 2 named == 2 summary, SUBFAILED and ERRORS included ruff clean on both touched files. NOT CLAIMED: no boot, no metal. The divergence SOURCE (the rank-local prefetch verdict under PP) is named here and left open -- closing it needs the sgl-project#791 ring lap to carry a prefetch-pending fact home, which is its own posten. This makes the system self-healing against a divergence however caused.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 28, 2026
Root (R3b, window-flip-0828 boot 2): batch_is_full is a PASS-scoped memo (written at the adder's stop condition) stored on the scheduler-lifetime running_batch. Every clear site is either a FINISH path the requeue-without-finish family (sgl-project#797/sgl-project#798/#791b/sgl-project#971) destroys before it runs, or sits behind 'if not running_batch.is_empty()' -- unreachable at exactly running=0, the defect state. Boot 2: three gate=batch_full_or_empty_queue(batch_is_full=1,queue=N) declines at running=0 with free seats and a waiting queue, self-sustaining until an external kill. The #888b relief lies on the path but its verdict producer sits behind the same emptiness guard (registered #962b). Fix: ONE junction at the gate read -- batch_is_full AND is_empty() is re-derived from the seat test (get_num_allocatable_reqs), not trusted. Re-derive, not clear: a genuinely exhausted seat table comes straight back True. Not an eighth clear site: any future requeue path inherits this without knowing the flag exists. In-tree precedent: the retract-RPC handler. #888b relief and the seam hook untouched; arm 2 of the new test pins that #962b's producer stays unreachable at running=0 so the posten stays open rather than silently closed. Tests: test_batch_full_empty_invariant_962b.py, 8 arms -- red-first on the REAL void path (gate declined at running=0 with seats free, note=gate=batch_full_or_empty_queue -- boot 2's own line shape), void family flag hole filled, green-guard for the 8-resident full-table case both sides, can-fail mutant. Baselined modules byte-identical.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 28, 2026
…and PP0 names it Boot 3 of window-flip-0828: after a sgl-project#797 void the follower holds a sgl-project#971-parked chunked continuation in self.chunked_req (never in any queue); PP0 cannot see it, its uniform cap is MIN-reduced before occupancy is loaded, and the void's requeue tail-rotates PP0's copy so it never reaches a seat. The follower's add_chunked_req appends its continuation unconditionally, so every forwarded schedule is illegal on the follower: 514 refusals (191 missing + 323 'admitted rid(s) the decision does not name'), 512 voids, sgl-project#801-spin guard death in 2 min. Fix, the register's named sgl-project#968 direction -- the fact rides EXISTING messages, no new wire, no collective: (rid, executed_len) of a parked continuation is stamped on (a) the per-pass decision downstream and (b) the output/void return lap homeward. Named deviation from the design brief: the ring is asymmetric (the last rank sends output DIRECTly to PP0, so a middle parker never gets onto the output lap alone) -- hence both messages, and facts are MERGED per hop, never overwritten, or a middle rank's fact dies one hop before its consumer. PP0 absorbs before the admission loop, REORDERS its own waiting_queue so the ordinary loop admits the rid (reorder, never admit: no second admission authority, no seat pricing -- once named, the continuation is a cap-counted member on every rank: no extra, no missing), and clears the fact when a decision names the rid (self-correcting: the next lap re-stamps if still parked). No prefix is ever reduced (Kein-Doppel- Prefill); all seven helpers module-level (holder lesson, test-pinned). Single-slot hypothesis CONFIRMED (double-sourced): chunked_req is one attribute, overwritten per truncation and cleared on launch -- PP0 cycles it (518 admits / 7 rids, the stuck rid named 2x), the follower pins it (514x). That is why the carry must exist. Tests: test_pp_parked_continuation_carry_968.py, 21 tests -- red-first GENUINE against the unmodified pin 3c470fb in a detached worktree: 18 failed / 3 passed (names == summary 18==18), 21 passed on the fixed tree; legacy arm byte-identical without the stamp; no-double-prefill arm asserts zero prefix reduction; can-fail arm neuters the absorb. Gate: 42 modules BEFORE/AFTER byte-identical 510P/9F, the 9 pre-existing name-identical both sides (prefetch_symmetry_580 7 + ring_commit_973 2 -- the 973 pair is the known gloo hang-shape arms, reproduced solo on the unmodified tree; environmental, posten noted).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 28, 2026
…e class behind both The 797 suite went red at 8be86f5 in two arms. Neither is a regression; both are sgl-project#984's premise arriving in a suite written against the old one. test_a_chunk_started_this_round_is_un_started_not_kept asserted that a chunk started this round is 'retracted like any member'. The void is now symmetric: members are PARKED and re-queued with prefix and pages intact, and only an already-finished request still takes the release path. The arm's actual point is its other two assertions -- the chunk is UN-STARTED and is back in the queue, so no rank is left uniquely mid-chunk -- and those never moved. How it gets back to the queue is the disposal's business; that it gets there is sgl-project#797's. test_instr19s_raise_returns_when_the_pre_admission_value_is_blinded was a DEAD CAN-FAIL. Its mutant blinded pp_void_keeps_request to force a retraction, hence the reset shape, hence instr19's AttributeError. After sgl-project#984 an un-kept member is parked rather than reset, so the mutant injected no fault at all and the arm failed on assertTrue(chunked.retracted) with nothing having retracted anything. THE CLASS, since this is the second instance in one commit (the first being test_the_underflow_detector_can_actually_fail in test_pp_void_lock_ref_969.py, aimed at a cache_finished_req the void no longer calls): a change that converts RETRACTION into PARKING disarms every can-fail proof whose injected fault worked by causing a retraction. Such an arm does not report a safe system -- it reports nothing, while looking like a red a tired reader would 'fix' by deleting the assertion. The re-aim needed TWO neuters, and that is a result rather than a concession: aiming at the park verdict alone left the arm red, because pp_void_keeps_request still guards the top of the loop and continues past the chunked request before any verdict is asked. The instr19 shape now requires defeating two independent guards where one sufficed before. Both are neutered visibly, because a can-fail proof must state the true depth of what it defeats rather than pick the smallest edit that goes red. Test: 31 defs / 31 passed at 8be86f5 (count probe), 89.51s.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 28, 2026
…resident at all Boot 14 (cf16281) died on PP1 after 39 s: `assert self.chunked_req is None`, scheduler.py:9784 in `_get_new_batch_prefill_raw`. Third recorded fundstelle of this family after :9286 (sgl-project#951) and :9367 (sgl-project#959). ROOT, and it corrects my own attribution in the boot-14 register entry. The invariant behind that assert is held "by ARITHMETIC, not by a check" (scheduler.py's own comment). sgl-project#959 therefore gave the two `add_one_req*` mint sites an explicit `chunked_req_outstanding` check, and SKIPPED the third -- `PrefillAdder._add_scheduled_req`, the forwarded-schedule execution path -- reasoning at schedule_policy.py that it "already has its own (`carried_chunk`)". That is the guard-comment-names-the-hazard trap. `carried_chunk` answers "is THIS request the resident continuation". The invariant needs "is there a resident continuation AT ALL". It covers a request being re-announced; it does not cover a DIFFERENT named request becoming a second continuation while the first is resident. Two of three sites guarded, one not. Boot 14 is that gap on metal: the resident continuation survived `add_chunked_req`, so `chunked_req_outstanding` was True and both sibling sites correctly refused -- and this site minted anyway, on another rid the same forwarded schedule named. sgl-project#994 EXPOSED THIS, IT DID NOT CREATE IT. Boot 13 never reached the line because the sgl-project#791 geometry refusal killed every pass before a batch was built; sgl-project#994 removed that refusal. My register entry attributed the death to sgl-project#994's effect on `rem_chunk_tokens` arithmetic. That was wrong: both sibling sites were already guarded and did refuse. The arithmetic was not the hole; the missing third guard was. Withdrawn here rather than left standing. FIX: the sibling guard, at the site that lacked it. WHY A PASS REFUSAL AND NOT A REQUEST SKIP -- the danger direction, which is the whole question here. On a forwarded schedule this rank may NOT drop a named request: the upstream's hidden states for it are already on the wire, which is what this same method already raises `PPScheduleRefused` for a few lines above. Running the chunk WITHOUT announcing it is worse -- the continuation would be untracked and re-prefilled next pass, the double prefill the standing law forbids. So the disposal is the one this path already owns: refuse the PASS by name, let sgl-project#791/sgl-project#797 void and re-derive. It cannot starve: the resident continuation is consuming chunks, and when it finishes `chunked_req` is None and the schedule is executable. This is the same direction sgl-project#959 chose ("the resident continuation is never the one to give way; the fresh admission is") -- not the sgl-project#858 mid-prefill wedge, which would be clearing `scheduler.chunked_req` instead. FUTURE CHECK: `grep -c 'if self.chunked_req_outstanding:'` over schedule_policy.py is now 3 and equals the number of `new_chunked_req` writers. A fourth writer that skips it is visible as an inequality rather than as a boot death. Execution proof (speed mode, one instrument per link): `note_second_continuation_refused(req, "_add_scheduled_req")` counts it and the refusal text names itself, so boot 15 measures whether the guard fires at all -- absence of the assert alone would be green-by-absence. Evidence: desk. py_compile; local-import dominance and guard-before-raise- before-mint verified by source order in the loaded module; guarded-site count 3 == mint-site count 3. Belegstufe: DESK-BEWIESEN.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 30, 2026
…serve has NO post, and its two guards share one gate Operator order mid-turn: stop chasing the OOM symptom rank by rank, identify every CONSUMER and RELEASER of the budget structurally. Done with the index tools; this replaces the four-arm symptom chase of 3bf2e78 with its root. TOOLS/QUERIES (reporting duty): devindex `where rank_gpu_memory_mib` -> the single profiling site; `symbol_profile ModelRunnerKVCacheMixin._profile_available_bytes` (83 outgoing calls); `trace_value available_bytes direction=back` (the full deduction chain); `where budget_posts`; `where post_capture_kv_active`; `symbol_profile activation_reserve_mb` / `post_capture_kv_sizing_planned`. Read only as the last mile at the file:line those produced. THE LEDGER IS CLOSED, so the post list is EXHAUSTIVE, not a sample: `budget_posts` has ONE writer (mixin:769) and all 18 reads inside `_profile_available_bytes` (mixin:710-1028). Posts, in order: 1. "weights + runtime state" :792 2. "gapped corridor holdback" :795 / :837 (the user's free column) 3. "mamba pre-capture reserve" :815 -- SEE GATE BELOW 4. mamba state pool (decomposed) :841 5. "GGUF dequant scratch" :870 (0.0 here, non-GGUF) 6. measured KV budget correction :877 (0 unless SGLANG_MEASURED_KV_BUDGET) Everything left becomes KV pool. THERE IS NO POST FOR CUDA-GRAPH CAPTURE. THE GATE, and why this is a defect shape rather than a missing line. Two independent mechanisms could each price the 810 MiB decode-graph capture, and BOTH hang off the SAME predicate `post_capture_kv_active` (mixin:2622): - `post_capture_resize_kv_pool` (:2629) re-sizes from MEASURED free AFTER capture -- capture bytes are already spent when `free_gb` is read; - the pre-capture post sgl-project#3 at :807 `if self.mambaish_config is not None and self.post_capture_kv_active:`. So one flag disables a mechanism AND its own fallback. Measured on the live boot, from the log's own sgl-project#797 instrument, all three ranks, both sizing passes: `post_capture_kv_active=False`. Neither guard is in force. WHY THE BUDGET PATH SPECIFICALLY. The else-branch (:817) opens with `slack_gb = pre_model_load_memory * (1 - mem_fraction_static)` -- that IS the upstream graph reserve, and `post_capture_kv_sizing_planned`'s own docstring calls it exactly that ("whether the mem_fraction heuristic may skip the graph reserve"). The `--rank-gpu-memory-mib` branch (:785-815) has no equivalent line. UPSTREAM-MINIMAL READING: the fork's absolute-MiB budget REPLACED mem_fraction_static and silently dropped the reserve that mem_fraction_static implicitly carried. Second bookkeeping beside an upstream truth -- so the fix is to make the budget path carry the upstream reserve (or enable post-capture sizing on it), NOT to invent a third fork-own reserve. INSTRUMENT-TEXT-LIES (that memory class, again, and it is why the reserve looked covered): the boot line "[auto-mamba] ... activation_reserve=1.00 GB; remaining VRAM -> KV pool" reads as "1 GB is being withheld". It is not. The number is the module constant MAMBA_AUTO_ACTIVATION_RESERVE_MIB=1024 (mixin:145), passed only into `_fit_mamba_pool_to_budget` (:2125) as a FIT CONSTRAINT on the mamba pool, whose own docstring says it "engages ONLY when the size computed by the caller would leave less than MAMBA_CEILING_FIT_MIN_KV_MIB for the KV pool". Nothing is deducted; `handle_max_mamba_cache` returns `total_rest_memory - mamba_state_memory` (:2415) and no reserve term. THIS EXPLAINS ALL FOUR ARMS of 3bf2e78 without a single new measurement: - std/gdncov boots because the weights are LARGER, so less flows to the pool and the UNBUDGETED residual happens to exceed 810 MiB. Luck, not design -- the reserve was never in force there either. - lm_head OOMs because the freed 1212 MiB went entirely to the pool and the residual fell under 810 MiB. - the knob is non-monotone because nothing TARGETS a capture reserve: free-at-capture is an uncontrolled residual, and the uneven token split redistributes whatever a rank gives up. EVIDENCE TIER: DESK-PROVEN on the code (closed-ledger argument, exhaustive by construction) + BOOT-PROVEN on the gate value (sgl-project#797 line, 3 ranks x 2 passes, boot_855_makespan_0840f82601_0830_093650.log). NOT DONE: no fix written, no capture-cost measurement of the reserve that should be booked. sgl-project#1025 now has its file:line and its shape.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 30, 2026
…lden statt bauen (Koordinator-Klausel) Koordinator-Entscheid: post-capture sizing auf den Budget-Pfad bringen, mit der Klausel "Wenn der Fix mehr als S/M wird: melden statt bauen". Gescoped; die Klausel greift. Kein Code geschrieben. WARUM DER ENV-SCHALTER ALLEIN NICHT REICHT. `SGLANG_ENABLE_POST_CAPTURE_KV_SIZING` ist EnvBool(False) (environ.py:489) und in keiner Boot-Env gesetzt -- das ist der erste Verweigerer. Der zweite ist `dcp_size == 1` in `post_capture_kv_sizing_planned` (server_args.py:14345), und der bindet genau dort, wo es weh tut. Gemessen an der Boot-Reihenfolge (boot_855_lmhead5_..._093325.log): 09:33:55 Sizing-Pass 1, sgl-project#797 sagt dcp_size=1 (PP-Layout) 09:34:07 cuda_graph_capture: 0.000-0.001 s <- de facto NO-OP 09:35:53 Sizing-Pass 2, sgl-project#797 sagt dcp_size=3 (TP/Flip-Layout) 09:36:10 cuda_graph_capture: 6.34-6.39 s <- DIE echte Capture 09:36:11 OOM, 810 MiB Die 810-MiB-Capture liegt VOLLSTAENDIG im dcp_size=3-Regime. Der env-Schalter wuerde also nur den Pass freischalten, dessen Capture 1 ms kostet, und den teuren Pass weiter verweigern lassen. WAS OPTION 2 DAMIT WIRKLICH VERLANGT -- die sgl-project#592-DCP-Sperre heben. Der Kommentar an server_args.py:14296-14344 benennt die fehlende Arbeit selbst und praezise: `pool.finalize_backing()` reicht `config.max_total_num_tokens` direkt als Zeilenzahl an `_finalize_backing_tokens` weiter, waehrend alle fuenf Pool-Konstruktionsstellen ueber `_dcp_token_sharded_pool_rows` gehen; die Identitaet gilt nur bei dcp_size == 1. Ein Heben OHNE diese Uebersetzung "would back C rows per rank instead of the rank's compact rows and set pool.size to a number the owner rule never addresses" -- also STILL falsche Groesse, kein Crash. Auf der Achse uneven-DCP-weighted, die laut Nutzergesetz nie abgeschaltet wird. ZWEITER BEFUND, der die Kostenschaetzung des Kommentars schwaecht: derselbe Kommentar beruft sich auf `test_post_capture_dcp_gap_592`, der "beides pinnt, die Sperre und die fehlende Uebersetzung, damit wer es angeht von der Evidenz aus startet". DIESER TEST EXISTIERT NICHT. `git grep` ueber den ganzen Baum: EIN Treffer, und das ist der Kommentar selbst. Das Sicherheitsnetz, auf das "bounded work" sich stuetzt, ist nicht da. (Klasse: PRAESENT-ABER-UNVERDRAHTET vs ABWESEND -- hier schlicht abwesend.) DRITTER BEFUND, aus dem sgl-project#797-Kommentar (mixin:7236-7248): auf einem Phase-Flip-Boot laeuft dieses Sizing VOR `build_phase_flip_tp_stack`, der Token-Vektor ist noch nicht installiert -- und "the site that DOES have the measured capacities (post_capture_resize_kv_pool) is HINT-ONLY". Die Stelle, die der Entscheid autoritativ machen will, ist auf der Flip-Form ausdruecklich nur Hinweisgeber. Sie autoritativ zu machen ist ein dritter Arbeitsblock, nicht Teil der Zeilen-Uebersetzung. SUMME: env-Default + sgl-project#592-Zeilen-Uebersetzung + Hint-only->autoritativ auf der Flip-Form, ohne den Test, den der Kommentar als vorhanden annimmt, auf einer Achse mit stiller Fehlgroesse als Fehlermodus. Das ist L, nicht S/M. ZUR BEGRUENDUNG (a) DES ENTSCHEIDS, unveraendert richtig: ein portierter slack_gb-Fraction-Term auf dem MiB-Pfad waere die Schaetz-Form, die der Pfad abschaffen sollte. Der Einwand gegen Option 1 bleibt gueltig -- er macht Option 2 nur nicht kleiner. VORSCHLAG ZUR ENTSCHEIDUNG (nicht gebaut): den 810-MiB-Posten als PROVISORISCHE Schranke auf dem Budget-Pfad buchen, ausdruecklich als Uebergang kommentiert und mit dem gemessenen Wert statt einer Fraction -- das ist S und macht das lm_head-Artefakt sofort bootbar -- und die sgl-project#592-Hebung als eigenen Posten fuehren, beginnend mit dem fehlenden Test. Entscheid liegt beim Operator. BELEG-STUFE: DESK-BEWIESEN (Praedikat-Zeilen, git grep) + BOOT-GEMESSEN (Capture-Dauer je Pass, sgl-project#797-dcp_size, beides aus den Logs dieses Strangs).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 1, 2026
…ot every rank's Boot 1065umbau wedged at 06:26:07Z: pp_to_tp armed one pass after PP0 launched the final 118-token chunk mb. PP1 had RECEIVED and reconciled that pass's frame (ROW-PROBE delivered=31, BATCH-NULLED occurrence=1 with all void flags False) but the armed park in get_next_batch_to_run withheld the build; PP2 stayed at delivered=30; PP0 blocked forever in the output recv (mb slots [0, 2] in flight), so the 30 s abandon carrier never ran. The deadman's BUSY-STARVED verdict took the boot down. Pre-sgl-project#1065 the strict chunk clause kept the park open during a chunk ladder and masked this: a follower's park is a rank-local verdict that stops the very drain the park exists to wait for. Per the PP0-authority order, followers execute what the row tells them and decide nothing: in the PP layout (pp_to_tp armed) only PP0 parks -- it stops SOURCING new work while in-flight mbs keep draining downstream. In the TP layout every rank is a replica of one decision from replicated state, so the park stays group-wide (tp_to_pp unchanged). Sibling sweep: the sgl-project#797 voided-pass hold below the park is row-carried (PP0-decided) and stays; the sgl-project#1066 PP0-wait is admission-side on PP0 only. Desk-proven: py_compile; metal falsifier is the same drain scenario -- chunk ladder + arm mid-ladder must now flip and decode.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 1, 2026
…ot end a pass PP0 launched `_pp_void_retracted_pass` let ONE downstream rank decide that the GROUP's pass ran nowhere. Its own docstring named the return trip that made that safe -- "the void output carries the observed local match home, and PP0's guard learns it as a floor" -- and sgl-project#969 CUT V had already deleted that emitter (`_PP_VOID_OUTPUT_KEY`: zero originating senders at ca0ee3a). The verdict therefore travelled downstream only. PP0's `mbs[slot]` stayed set while the last rank's did not, so no output was ever sent and PP0 blocked in `_do_recv` until the deadman -- the exact invariant `_do_recv`'s own comment relies on ("sender and receiver ask one question of one batch"). Measured, twice, and the second is on the stall second itself: 1068cap 07:34:02-09 sgl-project#797 void on rank 1 ONLY (no void/retract line on PP0 or PP2); PP0 parked, PP2 spinning. 1069cohort 08:00:54/55 sgl-project#791 unhonourable on PP1, told=12493 local=8397 then told=13399 local=12493 -- `local` exactly one pass behind `told`. After 08:00:55 only ranks 1 and 2 emit at all (slot_occupant / output_fill / width_agreement run to the takedown at 08:13:32, 4177 and 7973 lines; rank 0 emits nothing). That last measurement also refutes the occupant-sleep node as the halting member: ranks 1 and 2 are alive and turning; the `sgl-project#1000 SLOT-OCCUPANT reasons={'no-statement'}` spin is an INERT probe whose carrier sgl-project#1015 EDIT-F made permanently None. The only halting member is PP0's unbounded output receive. Repairing the return trip would repair a compensation layer for a rank-local verdict, which is the arc the sgl-project#968 order forbids continuing; under upstream-minimal the repair carries the burden of proof and the deletion does not. So the verdict is deleted and the disagreement is DETECTED instead (RAENGE-NIE-UNEINS: a detected divergence stops the group, never a compensating wait): * `_pp_assert_told_honourable` replaces it -- an unhonourable told names rank, slot, rid, told and local and raises. No clamp: clamping to this rank's own local match is rank-local geometry, i.e. sgl-project#631. * The chain-receive throttle arm gets a horizon (SGLANG_PP_OCCUPANT_HORIZON_S, default 90 s, 0 disables) and a named stop. Taking the arm is legitimate and frequent; outliving it never is. * `PpChainReceiver.recv` is bounded the way its sibling `consume_up_to` already was (runaway guard + reported counter), and the launcher now sets SGLANG_PP_CHAIN_RECV_STALL_S=60 -- the sgl-project#824 mechanism has existed since 2026-08-24 and shipped disabled by default, which is why it never fired in either stall. (A-i) is WITHDRAWN rather than built: its counter proof is blind at that site (the rendezvous bumps `sent` only on recv entry) and the DEFER one-shot is itself compensation for the rank-local verdict this commit deletes, so its trigger is removed at the source. Its two red-first tests are replaced by zombie tests for the deletion. Desk evidence: hermetic import + AST (deleted verdict absent, watchman and horizon present, recv bounded); test_968_deletion_falsifiers 24 passed, 1 failed -- test_C_the_void_relay_is_wired_or_deleted_but_never_half_built, which stays red until the relay SYMBOLS are swept out too. The relay is already unreachable at runtime (no rank originates a void any more, and `_pp_absorb_void_output` has no caller in production), so that sweep is a dead-code deletion scheduled beside this commit, not a runtime dependency of it. Belegstufe: DESK-BEWIESEN. Boot pending.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 1, 2026
…sends, nobody relays, one rank absorbs sgl-project#969 CUT V deleted the void-output EMITTER on the premise that the batch is the verdict ("sender and receiver ask one question of one batch"). What it left behind was the rest of the arc, and the arc has been dead ever since: _PP_VOID_OUTPUT_KEY: True constructed at exactly ONE site -- inside _pp_absorb_void_output, i.e. only when re-forwarding a void already received. Zero originating senders. _pp_void_forward_payload 1 writer, 0 readers. Computed and never sent. _pp_absorb_void_output 0 call sites (ast over python/sglang). pp_void_forward_payload called only from the absorber. pp_void_relay_stop_rank called only from the forward payload. pp_void_relay_launched_verdict same. pp_first_retracting_rank called only from the forward payload. A closed orphan subtree, whose own docstring states the sgl-project#801 relay invariant it can no longer keep: "a non-last rank that took exactly one message off this wire for a ring generation must put exactly one back on it, void included". WHY DELETED AND NOT WIRED. Wiring it back means rebuilding the second bookkeeping sgl-project#969 CUT V removed, and under the upstream-minimal law (user order 2026-08-29) repair carries the burden of proof that upstream semantics cannot carry the goal. It can: PP0 owns the verdict and the travelling batch is the verdict. Half a relay is the worst of the three states -- PP0 parks on an output the mid-rank void swallowed and its successor throttles on an occupant nobody will speak for, which is the shape both measured stalls took (1068cap 07:34:09 and 1069cohort 08:00:55, roles swapped, same halting node). Tests of the deleted mechanism go with it rather than being patched green: the five files whose SUBJECT is the void relay (sgl-project#797 retracted-pass void, sgl-project#798 chunked-retracted void slot, both sgl-project#801 relay files, #791b output-ring retraction), plus test_pp_continuation_cross_slot_rehome_968b (28 of its 30 tests drive the deleted absorber transitively) and test_pp_void_lock_ref_969 (its harness is the deleted method). Files that merely mentioned it keep their pins; only the individual test that exercised the deleted site is removed. Comments naming a deleted symbol are rewritten rather than left to send the next reader after a ghost. Desk-proven: py_compile + import smoke (scheduler and mixin), the sgl-project#968 deletion falsifiers 25/25 with the wired-or-deleted-but-never-half-built contract now green by the deletion arm, ruff unchanged at 104 pre-existing findings on both touched files (baseline taken from HEAD under the same config).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 1, 2026
…line positions
`local < told` was never a shortfall on this rig. It is the pipeline stagger,
and the comparison that read it as a defect had its two operands at different
pipeline positions and different times.
MEASURED, boot_855_1071cut, 12:09:58, one minute after READY, on the FIRST
chunked prefill of a FRESH boot against an EMPTY store (the launcher created
/tmp/hicache_855_1071cut that same minute):
PP1 rid=0d43edbc told=8192 local=4096
Nothing was in the store, so nothing could be reused and no rank could be
SHORT of anything -- a cache shortfall is constructively impossible in that
state. `told` is chunk 3's `extend_range.start`; `local` is chunk 1's
`extend_range.end`; the gap is exactly one --chunked-prefill-size. The check
was reading PP0's forward-most plan against a downstream rank's rear-most
completed extent.
This also retracts this strand's reading of the 1068cap and 1069cohort events
as "store skew" / "a moving told" (told=12493 local=8397, told=13399
local=12493): the same structural false positive later in the run, then
AMPLIFIED by the void it triggered -- `_pp_void_own_batch` restored
`chunked_req` to its pre-admission value, freezing `local` while PP0's `told`
ran on. Cause and amplifier were one mechanism.
DELETED (each verified as an absent BINDING by AST/hasattr, never by grepping
prose -- the first attempt at this check matched the function's own docstring
and had to be replaced):
* both retraction branches of `reconcile_pp_admission_decision`. The sgl-project#944
lookup-miss branch had to go WITH the sgl-project#791 one: its own comment rests its
safety on the sgl-project#797 void being group-uniform, sgl-project#1071 deleted that void, and
a retraction with no void left no longer stops the group -- it drops the
rid on ONE rank and the pass runs with divergent membership, i.e. sgl-project#631.
sgl-project#1071 had turned a covered branch into an uncovered one.
* `_pp_assert_told_honourable`, sgl-project#1071's own watchpost. It fired once, named
its own premise, and goes with the comparison it guarded; a watchpost over
a deleted check is second bookkeeping of its own.
* `pp_pass_should_void`, `pp_proxy_pass_retraction_reason`,
`_pp_pass_retraction_reason` (no callers), the `entries_retracted_by_rank`
import.
* the #995c proxy-width probe (141 lines) and its #995e/#995f census (34).
P2 measured why: it skipped 100% of its population on exactly the
downstream ranks where sgl-project#791 saw anything -- 1069cohort PP1 evaluated=54
agree=0 disagree=0 skipped=54, PP2 52/0/0/52; 1071cut 3/0/0/3 and 3/0/0/3;
reason always `no_local_input_ids`, because at proxy-receive time this
rank's `input_ids` do not exist yet. A counter printing `disagree=0` while
never comparing reads as "checked, all good" on the next pass. sgl-project#998 is
rank-local by construction and cannot see rank against rank.
NOTHING IS LOST, and this is the load-bearing half. The same length comparison
already exists where its operands are finally comparable and it CRASHES rather
than counting: model_runner.py:4233-4236, the one funnel every PP stage's
forward passes through, compares received hidden-states rows against
`forward_batch.input_ids.shape[0]` after `input_ids` is materialised and
raises with a sender stamp that separates a pairing error from a payload
error. sgl-project#791, #995c and that gate are the same check at three different
moments; this deletes the two that cannot fire and keeps the one that
executes. Ranks are held in agreement by sgl-project#631 ROW AUTHORITY (PP0 geometry
bulletin, 642b99c), not by a downstream veto.
`pp_rehome_displaced_chunked_req` deliberately STAYS (live caller at
scheduler_pp_mixin.py:9022).
Evidence: matched AST/binding check green (no retraction minted, 7 symbols
unbound, #995c counters gone from the recv path);
test_968_deletion_falsifiers 25/25 including the void-relay census. The 5
collection errors in test/registered/unit/managers are PRE-EXISTING, proven by
A/B in a fresh worktree at 3730c8e without these edits, not asserted.
Belegstufe: DESK-BEWIESEN. Acceptance boot pending.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 2, 2026
… group STOP, not a void Root (HANDOVER_1153_0902, PROVEN, pre-existing at the pin 228a66d): scheduler.py _pp_refuse_forwarded_schedule answered PPScheduleRefused ('sgl-project#791 FORWARDED SCHEDULE UNEXECUTABLE', PP1 reached 1 of the 2 rids PP0 named, batch_full_break) with a rank-local compensation: _pp_admission_pass_voided = True + emptied decision dicts -> mixin _pp_void_own_batch -> 'sgl-project#631 ROW-DELIVER BATCH NULLED slot=0 pass_voided=True' (boot_855_weg1b2 log 65000-65004) -> PP1 sent no proxy while PP0's slot stayed set -> nothing carried the void upstream (sgl-project#797 return trip: zero call sites since CUT V; sgl-project#1072 deleted the void relay) -> PP0's blocking _do_recv consumed PP2's NEXT output under this slot's label (log 65119) -> one output ahead for the rest of the boot -> the pp_to_tp arm at 21:42:05 turned the debt into an unproducible output -> sgl-project#980 ObjectRecvStalled 60 s / sgl-project#1071 PpChainRecvStalled 90 s. Same form sgl-project#1071 (169f53c) deleted for _pp_void_retracted_pass; this was the second writer of the same flag. F1 ROOT: the compensation is deleted, the detector is kept, the refusal is a group STOP. scheduler.py get_new_batch_prefill's except now does `raise self._pp_forwarded_schedule_stop(refusal) from refusal`; _pp_refuse_forwarded_schedule (flag write, dict emptying, sgl-project#971 re-home) is removed; the new _pp_forwarded_schedule_stop logs the kept 'sgl-project#791 PP-ADMISSION forwarded schedule REFUSED on rank N' line and returns a RuntimeError formatted by pp_admission_congruence .forwarded_schedule_stop_message: 'sgl-project#791 FORWARDED SCHEDULE UNEXECUTABLE STOP rank={rank} slot={slot} told=[{told}] reached=[{reached}] census={census} local={local} limiter={limiter} running_bs={running_bs} parked={parked} r2t_avail={r2t_avail} headroom={headroom} group_limit={group_limit} batch_full_setter={batch_full_setter} batch_full_at_loop_entry={batch_full_at_loop_entry}: {refusal}' Every probe is guarded (n/a on an unreadable value); the reached rids are recorded after the admission loop before any of the three raises; the batch_is_full setter site is recorded at each writer of the pass (no_allocatable_reqs_gate, count_arm, disagg_prefill_r2t_avail, add_one_req_NO_TOKEN). Group stop mechanism (existing, no new collective): the RuntimeError leaves run_event_loop; run_scheduler_process (scheduler.py 'except Exception') logs 'Scheduler hit an exception' and parent_process.send_signal(SIGQUIT) (+ killpg/kill_process_tree under SGLANG_KILLPG_ON_SCHEDULER_EXCEPTION); peers end in the bounded sgl-project#980 / sgl-project#1071 receives or the barlink dead-peer probe. F2 TRIGGER CLASS (PP0 order): on a forwarded schedule the follower's rank-local seat-count veto is not a verdict. pp_admission_congruence .rank_local_count_veto_applies(scheduled_extents) is True on PP0 and on every non-PP boot (unchanged expression) and False on a rank > 0 executing PP0's decision; the sgl-project#823 count arm, its batch_full_break, and the three pre-loop count gates of the same arithmetic (batch_full_or_empty_queue's batch_is_full half, min_free_slots_delay, no_allocatable_reqs) are gated on it. The physical allocator still refuses (NO_TOKEN -> membership refusal -> STOP naming the numbers). F3 SIBLINGS: mixin _pp_void_pass_without_upstream_launch's writer of the flag is provably unreachable (pp_upstream_void_pending returns False on every path; its final statement is `return False`) -- left with the proof in a comment. The row-authority _row_skip_plan exit now nulls a slot through _pp_null_frameless_slot, which applies the same named sgl-project#1020 'VOID REFUSED ON A LAUNCHED SLOT' guard as the void path; the guard is factored into _pp_slot_holds_unconsumed_launch(mb_id, site) and used by both sites. Matched check (error class: a follower still ends a PP0-launched pass silently): grep -rn '_pp_admission_pass_voided = True' python/sglang/srt -> only scheduler_pp_mixin.py (the unreachable sgl-project#801 writer); chain scheduler.py raise PPScheduleRefused (:11633/:11646, :11606) -> except PPScheduleRefused (:9441) -> raise self._pp_forwarded_schedule_stop (:9472) -> RuntimeError (:9623); the flag's only readers are _event_loop_pp_body (:4519 -> _pp_void_own_batch) and the scheduler.py void guard (:8955), neither reachable from the refusal. Tests (hermetic, CUDA_VISIBLE_DEVICES=""): new test_pp_forwarded_refusal_stop_1153.py (T1 STOP form + no void, T2 the count-veto helper + source pin, T3 the sgl-project#1020 guard on the frameless null, + the mixin:9389 unreachability proof): 17 passed after; on the parent ca4c6b7 (git worktree) it is a collection error (new names). test_pp_refused_pass_keeps_continuation_971.py: 12 tests that pinned the old compensation INVERTED with the withdrawal named in each docstring (24 passed after; the inverted 12 are red on the parent: 16 failed / 8 passed / 1 error across both files). Mutants: M1 flag write restored in the STOP builder -> 2 T1 red; M2a helper returns True -> 1 T2 red; M2b count-arm gate dropped in the loop -> 1 T2 red; M3 guard dropped from _pp_null_frameless_slot -> 1 T3 red. Bounded suite (7 unit/managers files + test/registered/scheduler): before 103 failed / 527 passed / 1 skipped / 27 errors; after 103 failed / 544 passed / 1 skipped / 27 errors -- per-file identical except the new file (+17). ruff check clean on all touched files; ruff format --diff hunk count unchanged vs parent (15/14/3, pre-existing) and the two test files formatted. Evidence tier: DESK-PROVEN. Boot-3 acceptance: grep -F 'sgl-project#791 FORWARDED SCHEDULE UNEXECUTABLE STOP' in the same second as 'sgl-project#791 PP-ADMISSION forwarded schedule REFUSED on rank', and zero 'ROW-DELIVER BATCH NULLED ... pass_voided=True' lines.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.
Motivation
Support chunked prefill.
Modification
--chunked-prefill-sizeparameter.Checklist
pre-commit run --all-filesor other linting tools are used to fix potential lint issues.