Skip to content

Chunked prefill support - #797

Merged
hnyls2002 merged 13 commits into
mainfrom
chunked-prefill
Jul 29, 2024
Merged

hnyls2002 merged 13 commits into
mainfrom
chunked-prefill

Conversation

@hnyls2002

Copy link
Copy Markdown
Collaborator

Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.

Motivation

Support chunked prefill.

Modification

  1. Add an --chunked-prefill-size parameter.
  2. Add an inflight request field in tp worker.

Checklist

  1. Ensure pre-commit pre-commit run --all-files or other linting tools are used to fix potential lint issues.
  2. Confirm that modifications are covered by complete unit tests. If not, please add more unit tests for correctness.
  3. Modify documentation as needed, such as docstrings or example tutorials.

@Ying1123 Ying1123 self-assigned this Jul 29, 2024
@hnyls2002

Copy link
Copy Markdown
Collaborator Author

Also thanks @vikranth22446 for his help with chunked prefill.

Comment thread python/sglang/srt/server_args.py Resolved
Comment thread python/sglang/srt/server_args.py Resolved
Comment thread python/sglang/srt/server.py Resolved
Comment thread python/sglang/srt/server_args.py Resolved
Comment thread python/sglang/srt/server_args.py Resolved
@hnyls2002
hnyls2002 merged commit 2ec39ab into main Jul 29, 2024
@hnyls2002
hnyls2002 deleted the chunked-prefill branch July 29, 2024 09:21
Ying1123 added a commit that referenced this pull request Jul 29, 2024
@hnyls2002
hnyls2002 restored the chunked-prefill branch July 29, 2024 09:45
@hnyls2002 hnyls2002 mentioned this pull request Jul 29, 2024
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
amote-i pushed a commit to amote-i/sglang that referenced this pull request Dec 8, 2025
* Update test_utils.py

* Update test_utils.py

* Update test_ascend_debug_tensor_input_file.py

* Update run_suite.py

* Update run_suite.py

* Change lint job to run on linux-amd64-cpu-1

* Update lint.yml

* Refactor lint workflow for improved configuration

* Update daily-build-test-npu-innersource.yml

* Fix syntax error in merged PR benchmark workflow

* Update pr-test-npu-debug.yml

* Clean up blank lines in activation.py

Removed unnecessary blank lines in activation.py.

* Fix indentation in attention layer methods

* Update io_struct.py

* Update dbrx.py

* Update phi.py

* Update test_utils.py

* Clean up unused imports in test_vlm_utils.py

Removed unused imports from the test file.

* Add blank line before main execution block

* Add model path to TestLing class

* Update test_ascend_llm_models_phi_4_multimodal.py

* Update test_ascend_llm_models_smollm_1_7b.py

* Update test_ascend_sleep_on_idle.py

* Update test_ascend_llm_models_Qwen3_32B.py

* Update test_serving_chat.py

* Update run_suite.py

* Update test_server_args.py

* Update daily-build-test-npu-innersource.yml

* Update pr-test-npu-debug.yml

* Update activation.py

* Update torch_native_backend.py

* Update io_struct.py

* Update phi.py

* Update test_utils.py

* Update test_ascend_llm_models_Stablelm_2_1_6b.py

* Update test_ascend_llm_models_ling.py

* Update test_ascend_llm_models_smollm_1_7b.py

* Update merged-pr-benchmark.yml

* Update run_suite.py

* Update daily-build-test-npu-innersource.yml

* Create pr-test-npu-debug-1.yml

* Update run_suite.py

* Update pr-test-npu-debug-1.yml

* Update pr-test-npu-debug-1.yml

* Create test_ascend_hicache_mha.py

* Create test_ascend_sampling_backend.py

* Update run_suite.py

* Update pr-test-npu-debug-1.yml

* Update run_suite.py

* Update and rename test_ascend_deepep.py to test_ascend_deepep_mtp.py

* Update test_ascend_deepep_mtp.py

* Update test_ascend_deepep_mtp.py

* Update test_ascend_mla_w8a8int8.py

* Update test_ascend_tp4_bf16.py

* Update daily-build-test-npu-innersource.yml

* Create daily-build-test-npu-opensource.yml

* Update daily-build-test-npu-opensource.yml

* Update test_ascend_deepep_mtp.py

* Update test_ascend_mla_w8a8int8.py

* Update run_suite.py

* Update pr-test-npu-debug.yml
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 21, 2026
…fix floor back to rank 0

#791c made a downstream REFUSE a proxy for a pass it had narrowed. It detected; it did
not prevent, and the frequency data is why that mattered:

  boot      'unhonourable prefix'   fatal width mismatch
  instr15          661                      1
  instr16         1651                      1
  instr17         1718                      1

instr17 retracted 1718 times and only ONE differed in width enough for
model_runner.py:4178-4191 to see. The other ~1717 were SAME-WIDTH divergences that
computed silently, because chunked prefill caps every chunk at the same size, so two
ranks running different request sets routinely present equal widths.

TRIGGER, measured: retractions are FLIP-triggered. Boots driven with short prompts only,
where no runtime flip ever arms, logged ZERO retractions (boot_coh, boot_detval); every
flipping boot logged hundreds. The cutover cold-starts the downstream radix cache, so
PP0 offers a prefix the downstream has nothing for.

BOTH HALVES, decided together as they must be.

(b) THE RETRACTION VOIDS THE PASS, NOT THE RID. Three membership outcomes exist and two
are physically unavailable: the rank cannot admit the rid (it has no KV for the prefix
its upstream reused, and the upstream sent hidden states only for the extend tokens),
and the upstream cannot be amended (it sent its decision and launched earlier in the
same pass). What remains is to run the pass nowhere. `effective` is emptied and every
surviving entry becomes admitted=False, retracted=False -- the third state, which
record_return_trip correctly leaves alone (it teaches on `retracted`, clears on
`admitted`). _PP_PASS_VOIDED_KEY carries the fact downstream because the entries alone
cannot: a rank with nothing to prefill falls through to its running DECODE batch and
would pair that with the upstream's prefill batch.

(a) THE FLOOR COMES BACK ON THE OUTPUT MESSAGE. PP0's request state is resynced by
#791b's existing void-output absorb, which rides the chain-reconciled decision home so
the guard learns observed_local and prefix_len_for clamps the next offer -- strictly
decreasing, non-negative, terminating. sgl-project#797 additionally puts that payload on a
SUCCESSFUL output, the half that lets a floor be CLEARED: sgl-project#796 removed the only feeder
for that and #791b restored only the learning half.

A GAP FOUND AND CLOSED, or this would have traded corruption for a HANG: #791b's void
stops at the first rank. With the retraction on rank r, ranks 1..r-1 also hold launched
batches with output receives posted, so PP0 absorbing and forwarding nothing would block
them. Pre-sgl-project#797 that shape mispaired instead (the retractor narrowed rather than emptied,
so a real output still went out), which is why it never showed. r=2 is reachable at
pp_size=3, i.e. on this rig. The void now travels last->0->1->...->r-1 and stops, gated
on pp_first_retracting_rank read off the decision the void already carries. No new key.

WHAT THIS DOES NOT DO, stated because the acceptance gate depends on it: IT DOES NOT
REACH ZERO RETRACTIONS. Expected is ~one per rid whose first post-cutover offer is
unhonourable, against 661/1651/1718 -- the count stops being per-PASS and becomes
per-RID-per-CUTOVER, because the floor now comes back and clamps the next offer. Zero is
unreachable this way: PP0 cannot know a downstream's cache state before it offers, and
"offer and find out" is the only channel that exists without a new collective. The
counter that MUST be zero with this fix is the #791c divergence tripwire, which proves no
narrowed pass was ever created. Reaching zero retractions would require clamping PP0's
first offer per rid per flip epoch to told=0 -- sound, ~15 lines in prefix_len_for, but it
pays away exactly what prefix caching buys, so it is not taken unilaterally.

ITEM 2, and the answer is NO: the two rank-local decode-retraction paths
(scheduler.py:7489/:7517 and :7211-7312) cannot produce this class. They CAN diverge in
membership -- _update_uniform_pool_budget reduces on tp_cpu_group, world=1 under
TP=1/PP=3, so the floors are off, as the boot line at scheduler.py:4733-4740 says. But
not SILENTLY: _get_decode_retraction_order (schedule_batch.py:3038-3082) sorts on
replicated per-request state under replicated server args, retract_decode pops only from
the END of it (schedule_batch.py:2961) so victim sets are NESTED and never disjoint, a
decode batch's row count is a strict function of its request count, and retract_decode
entered with >1 request always retracts >=1. So every divergence there is a WIDTH
divergence, raised loudly every time. Real defect, different root (the reduce group), and
sgl-project#797's remedy structurally does not fit it -- voiding a pass repairs per-pass membership,
while a retracted decode victim mutates long-lived state. Filed at the site
(scheduler.py:7554-7600), not folded in.

TESTS. test_pp_retracted_pass_void_797.py: 11 passed (7 three-process gloo arms + 4
pure), 61 s. Three separate can-fail neuters, each a single return-value rebind through
scheduler_pp_mixin's module globals IN THE CHILD:
  blind entries_retracted_by_rank  -> 3/3 green assertions fail; victim reports rows=128
                                      foreign_rows=64, the same-width mispair accepted
  blind pp_pass_should_void only   -> 3/3 fail, victim raises #791c PROXY BATCH DIVERGED
                                      (detection, not prevention)
  blind the per-hop launched key   -> 1/3 fails, drained=False stashed=1, the stranded
                                      proxy corpse
Neighbours 631/791/791b/791c/795/796 re-measured AT HEAD: 4 failed / 67 passed; after:
4 failed / 78 passed, failure names byte-identical. Zero regressions, +11.
With 791b and 791c together: 20 passed. ruff/ruff-format/codespell identical to HEAD.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 21, 2026
Boot instr19 ran sgl-project#797 on metal and died after 53 s. Both roots are here.

#797b -- THE CRASH. A void is a PARK, not a retraction. sgl-project#797's release/re-queue tore down
chunked_req, so a voided pass never reached prepare_for_extend while self.chunked_req stayed
set with extend_range None, and the next pass died at scheduler.py:5572:

  AttributeError: 'NoneType' object has no attribute 'end'
    get_next_batch_to_run <- _event_loop_pp_body:1132        (PP0 08:47:27)

One root, two victims, established per rank rather than assumed: PP1 and PP2 carry ONLY
"Connection closed by peer" in recv_tensor_dict, and PP0 took a second, consequential gloo
error after its peers went down. So it was neither the floor-feedback message nor the void
itself -- the void executed correctly.

#797c -- THE LIVELOCK, and it falsifies sgl-project#797's own termination claim. All three retractions
on instr19 are the SAME rid and `told` GROWS by exactly one chunk:

  08:47:26 rid=d4f59edf... told=512  local=0
  08:47:27 rid=d4f59edf... told=1024 local=0
  08:47:27 rid=d4f59edf... told=1536 local=0

sgl-project#797 asserted the offer is "strictly decreasing, non-negative, terminating". It is strictly
INCREASING. Root: local=0 WAS NEVER A MEASUREMENT. _pp_reconcile_incoming_admission looks
rids up in waiting_queue; scheduler.py:7075 removes admitted requests from it; a chunked
request then lives in self.chunked_req -- so from round 2 the lookup structurally reports 0
for a prefix the rank genuinely holds. Log line 1737 proves the rank had it: all three ranks
admitted the first chunk CONGRUENTLY one second before the first retraction. The rank was
retracting a prefix it owned, every round, forever.

TERMINATION IS NOW PROVEN IN A TEST, not asserted in a docstring -- the specific lesson of
this round. PPChunkedOfferLivelock797c drives one rid through repeated offers ACROSS CHUNK
BOUNDARIES: told advances exactly one chunk per round, 0 retractions, and the request
finishes. Two red arms, including the one that matters: the PARK ALONE gives flat-but-never-
finishing, i.e. #797b is NECESSARY BUT NOT SUFFICIENT and neither fix stands without the
other.

SIBLING STATE, enumerated rather than fixed only where the traceback pointed: seven call
sites audited with fixed / safe / named verdicts. waiting_queue removal (:7075) is the one
that bites; adder.preempt_list is NAMED, not fixed; several are safe today but become
siblings the day they are wired, because a voided pass would spend a chunk that never ran and
commit raises on re-commit.

TESTS. test_pp_retracted_pass_void_797.py: 23 passed (11 before), 58.6 s. Red-first for
#797b: blinding pp_void_keeps_request reproduces the exact traceback line. Neighbour set
4 failed / 90 passed, failure names byte-identical to 6f42d7f and all pre-existing.
ruff and codespell identical to 6f42d7f (the 98 scheduler.py findings are pre-existing).

OPERATIONAL NOTE, my error not the agent's: the standing sub-briefing line "Qwen bevorzugt"
is unsatisfiable while this strand holds the GPU, because the qwen/local-model agent types
run on the local htsglang server at 127.0.0.1:30030 -- the exact server the boots kill.
Measured: serving_30030 DOWN, router_30099 alive. A Qwen sub-subagent spawned in that window
produced one write and nothing for 32 minutes. While the claim is held, sub-briefings must
say Sonnet/Haiku; the never-opus/never-fable half is unchanged.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 22, 2026
…ion gate

Follow-up to 1ccfaac, prompted by strand 10's b71dc7b, which
instruments the install-capable CALL SITE while 1ccfaac instruments the
four gates INSIDE the method. The two are complementary and neither
duplicates the other, but strand 10's analysis exposed a real weakness in the
gate-one line: it reported only "uneven DCP is not active for this dcp_size",
and that collapses three different causes into one sentence.

uneven_dcp_active refuses in three ways (utils.py:326-330), and they call for
three different fixes:
  * no vector installed yet (ratios is None)
  * a uniform vector (the even modulo fast path, deliberately not "uneven")
  * a length mismatch (len(ratios) != dcp_size)

MY EARLIER READING NAMED THE WRONG ONE, and that is corrected here rather
than quietly overwritten. 1ccfaac's comment asserted the length mismatch
-- a three-entry vector against dcp_size 1 -- as the dominant suspect. On a
phase-flip boot the real cause is the FIRST: the only install-capable site
runs from init_memory_pools (scheduler.py:1429), which is BEFORE
build_phase_flip_tp_stack (scheduler.py:1446) performs the first
set_cp_token_ratios (phase_flip_boot.py:971-990). At that site there is no
vector installed at all to improve on. Whoever may install does not yet have
the numbers; whoever has the numbers (post_capture_resize_kv_pool) may no
longer install. A line reporting a length mismatch there would have sent the
next reader after the wrong fix -- the precise failure mode this
instrumentation exists to prevent, so it is worth the extra branch.

The installed vector is now printed, so the three are separable from a log
alone. Still logging only: every gate returns exactly where it did.

Tests: test_the_three_causes_of_the_first_gate_are_told_apart drives all
three and asserts the three lines are PAIRWISE DISTINCT, not merely that each
contains its own keyword -- three reasons sharing a substring would pass the
weaker check. test_the_length_mismatch_gate_names_itself additionally pins
"3 entries against dcp_size 1". 30 passed in 8.66s. ruff and codespell clean.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 22, 2026
…le, as a rule

The rejected register answers "was this CONFIGURATION tried and settled". It
has no answer for a different question that turns out to matter more: is the
number I am about to run traceable to work that was later withdrawn. An
investigation can be retracted while its outputs keep riding, because nothing
connects the two -- the value is just a number in a launch line by then, with
no lineage attached to notice.

The shipped uneven-DCP token vector is the proof. 29,19,16 comes from sgl-project#602,
whose per-card attribution was withdrawn and re-derived more than once
(NOTE_602_fill_side_attribution.md:67, "already had to retract twice"), so the
slack figures it was proportioned from no longer stand. It has been the active
vector on every boot since, and every one of those boots measured its own
per-rank capacity, computed a better vector, printed it, and threw it away.

The rule, stated so it can be enforced rather than remembered: AN ACTIVE
VECTOR MUST NEVER ORIGINATE FROM A RETRACTED INVESTIGATION. "Active" carries
the weight. A retracted value that is merely present is harmless -- as a
sizing seed it is superseded in-process by the measured optimum before
anything serves on it. A retracted value that is PINNED, or that is seeded and
then fails to be superseded, is the number the server actually runs, and that
is refused. So the gate belongs where a vector becomes active, and the seed
path gets re-checked after the install attempt instead of waved through.

Matching is layered, because a rule that fires only when the operator
volunteers the incriminating fact is not a rule. Mode 1: a declared
--*-provenance naming a retracted investigation is refused whatever the value,
which is the durable general form. Mode 2: with no provenance declared, the
gcd-reduced vector is matched against the values the retraction recorded --
without this the rule would be silent on precisely the vector that motivated
it, since the launch shipping 29,19,16 declares no lineage at all. Mode 2
matches gcd-reduced so 58,38,32 cannot dodge it, and a stated clean lineage
short-circuits it, so it stays a fallback for an unstated lineage rather than
an override of a stated one. A measured install stamps PROVENANCE_MEASURED and
is never refused -- the remedy must not be refused by the rule it satisfies.

This commit is the register and the predicate. The boot-path refusals that
consume them are the next commit; landed separately so the gate arrives with
its can-fail evidence rather than alongside a wiring change.

Tests: test/registered/unit/planner/test_retracted_provenance_797.py, 21
passed. Can-fail EXECUTED, both directions: neutralising the mode-2 value loop
turns 2 red (the shipped vector and its scaled spelling stop being caught);
making the predicate ignore its declared provenance turns 3 red. Restored and
re-verified 21 passed. ruff check clean and ruff format clean on both files.

No behaviour change yet: nothing calls into this module, so every existing
path is byte-identical.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 22, 2026
…worker's label

The measured token-vector install has been refused on this rig by its last
conjunct, and the conjunct is a mis-picked member of a pair the codebase
already defines. model_runner.py:512:

    self.is_draft_pool_worker = is_draft_worker and not is_phase_flip_tp_stack

The two flags coincide everywhere except on the phase-flip TP stack, where a
draft runner is a draft worker whose POOLS take the target-model treatment.
That runner is the one that owns the real KV pool, and _resolve_memory_pool_config
reaches it precisely because the sizing branch at :7272 keys on
is_draft_pool_worker. Twenty lines further down the install gate keyed on the
other member of the pair and refused the install to the very runner that had
just resolved the pool config.

The exclusion itself is still right and still enforced. A runner with its OWN
draft pool must not install the target's token vector, because its capacity is
not the target's capacity. It is now asked about the pool, which is what the
vector is about, instead of about the worker's label.

MEASURED, not deduced from the source alone. boot_798_0822_0629.log carries
the instrumentation from b71dc7b and 4bf3de1. Six occurrences of the
call-site line, three per stack. The PP stack, dcp_size=1 active_vector=None,
skips with "uneven DCP is not active: no token vector is installed at this
point yet" -- init_memory_pools (scheduler.py:1429) runs before
build_phase_flip_tp_stack (scheduler.py:1446) installs the vector at
phase_flip_boot.py:971-990. The TP stack, on all three ranks:

  sgl-project#797 install-capable sizing reached: post_capture_kv_active=False
  derived_mode=False token_vector_is_seed=True role='seed' dcp_size=3
  active_vector=[29, 19, 16]

No SKIP line follows it. The calibration ran to completion, computed the
better vector, and still printed "restart with
SGLANG_UNEVEN_TOKEN_VECTOR=30,17,17 to raise max_total_num_tokens from 1213952
to ~1356736 (per-rank profiled capacity [636730, 360392, 375560])". The
PINNED-VECTOR warning did not fire, so pinned_vector was False and seed_role
True; allow_install was True by the same line. That leaves the draft conjunct
as the only one that can have been False.

This also settles the shape of the fix. The install-capable site is REACHED on
the TP stack with the DCP layout active and the vector present, so a third
install point is not needed: the site was unlocked all along and one predicate
was refusing it.

Two other candidate sites were excluded by mechanism first, so they are not
quietly still open:
- post_capture_resize_kv_pool cannot grow the pool. _config_from_budget applies
  min(max_tokens, cap_tokens) at :6984 and the caller passes
  cap_tokens=self.max_total_num_tokens (:2627), so a better vector computed
  there is clamped away.
- The flip cutover cannot install a different vector. dcp_compact_pool_rows =
  (global_tokens // cp_S + 1) * cp_ratio at layers/dcp/owner.py:181 is
  evaluated ONCE at pool construction with cp_S = sum(ratios), so the pool's
  physical row geometry is a function of the vector's sum; installing a
  different one desyncs the slot space from the allocation.

Tests: test_draft_pool_worker_gate_797.py, 6 passed, new. Can-fail EXECUTED:
restoring the gate to `not self.is_draft_worker` turns 2 red, including the
falsifier that asserts the flip-TP draft runner installs; restored and
re-verified 6 passed. The sibling suite test_uneven_token_vector_role_797.py
goes 30 passed, unchanged in count.

The sibling suite's stubs gained is_draft_pool_worker, mirroring
model_runner.py:512. They model a NON-flip runner, where the two flags
coincide, so every existing case keeps its exact meaning; _run gained an
optional draft_pool_worker to let the new suite drive the divergence. That the
stubs raised AttributeError before this was added is itself the point -- every
case in that suite lives in the coinciding region, which is why the defect
could not be seen from there.

ruff check and ruff format clean on all three files.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 22, 2026
…a retracted study

214a9f6 landed the register and the predicate with nothing calling them.
This wires them into resolve_cp_token_ratios, the single resolver every boot
passes through, so the rule stops being a fact about a module and becomes a
refusal.

A PINNED vector whose provenance names a retracted investigation -- or, with no
provenance declared, whose gcd-reduced value matches one a retraction recorded
-- now raises RetractedProvenanceError before anything is sized. Both explicit
doors are covered: SGLANG_UNEVEN_TOKEN_VECTOR and the --rank-kv-ratio a,b,c
pin. The falsifier is the vector this rig shipped: 29,19,16 from sgl-project#602, pinned,
with no lineage declared, refused by value.

A SEED is permitted past this point and warned about instead, and that is the
rule applied consistently rather than softened. The rule is about the ACTIVE
vector; a seed is superseded in-process by the measured optimum before anything
serves on it, so for a seed the question is only answerable AFTER the install
attempt. That check is the remaining piece and it is named here rather than
implied: a seed that fails to be superseded is precisely the boot this task
exists to end, and it must be refused at the install site.

Two flags carry the lineage, both published to the environment like the role
and for the same reason -- the resolver runs inside the flip's second stack
build, where the boot's ServerArgs object is not the one consulted.
--uneven-token-vector-provenance is published ONLY when stated, because "not
stated" is a meaningful answer: it is what ARMS the fallback value match.
Publishing "None" as a string would read as a declared lineage and disarm the
gate, which is the failure mode this whole task is about.

_token_vector_role gets one definition, env over flag, and an EMPTY env value
reads as "not stated here, ask the flag" rather than as "pin". That is
deliberate and it is the 2026-08-19 lesson: an empty override silently meaning
the default is how a blank SGLANG_UNEVEN_TOKEN_VECTOR rode along for days
unnoticed. A gate that switches itself off when handed an empty string is the
same defect wearing different clothes.

Tests: test_retracted_vector_boot_refusal_797.py, 12 passed, new, all driving
the real resolver rather than the predicate -- the sgl-project#182 lesson, where a
token-vector honesty guard was correct and sat on a branch no server could
reach. Can-fail EXECUTED twice: deleting the refusal call in the env branch
turns 5 red; making _token_vector_role hard-default an empty env to "pin"
turns the empty-override test red. Restored and re-verified 12 passed.
The sibling suite test_retracted_provenance_797.py stays at 21 passed.

Regression, matched HEAD, both runs hermetic (CUDA_VISIBLE_DEVICES="") and
identical selection: test/registered/unit/distributed/ at HEAD 57778d4 in a
clean worktree gives 81 failed / 2997 passed; with this change, 81 failed /
3009 passed. Same failure count, and the +12 passed are exactly this commit's
new tests. Sampled failures are pre-existing GroupCoordinator/parallel_state
attribute errors unrelated to the token vector, and 81 matches the pre-existing
baseline 95fdc54 recorded. Noted honestly: this repo's reporter emits no
"FAILED id" summary lines for these, so the sets were compared by count and by
sampling rather than id-by-id.

ruff check and ruff format clean on the touched files; the two format hunks
ruff still reports in distributed/utils.py are pre-existing and outside this
change.

Default path unchanged: with no explicit vector, no provenance and no
retraction match, the resolver behaves exactly as before.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 22, 2026
The retracted-provenance gate refuses a vector whose LINEAGE is bad. This is
the other half: a vector admitted as a SEED made a claim about the future --
"I am a pre-boot estimate and the measured per-rank capacity supersedes me
in-process" -- and a boot where that never happened is now refused rather than
warned about.

That case is not hypothetical. boot_798_0822_0629 reached three install-capable
sizing sites at dcp_size=3 with allow_install=True and role='seed', declined
every one on a predicate about the worker's LABEL instead of its pool, and
served the seed [29, 19, 16] as if it were a decision. Nothing failed. The
advisory printed and was ignored, which is the defect class this task is about:
a path that is logged and never enforced. A warning here would be the same
thing again, so this raises.

Three signals, because refusing on fewer would refuse correct boots:
  armed        -- a seed was actually resolved (a pin asserts its value and
                  arms nothing)
  calibration  -- a site that could REALLY have superseded it was reached
                  (dcp_size > 1 AND allow_install), so "never superseded" means
                  declined and not "never had the chance"
  superseded   -- a verdict was reached
Refuse iff armed AND calibration AND NOT superseded.

Two design points that each prevent a false refusal. The latch is disarmed on
the VERDICT, not on the write: the install site writes nothing when the
measured optimum equals the active vector, so disarming on the write would
refuse a boot whose seed happened to be right. And the latch is process-global,
so a genuine draft-pool worker declining its own install does not trip it --
its target runs in the same process and its verdict disarms the claim for both.

Placed at scheduler.py:1448, after build_phase_flip_tp_stack rather than after
init_memory_pools, because under the flip it is the TP stack that carries the
DCP layout; the earlier position would refuse every flip boot.

CAN-FAIL, EXECUTED, both directions, driving the real sizing method in the
boot-0629 configuration (is_draft_worker=True, is_draft_pool_worker=False):
  predicate `not self.is_draft_worker`      -> installs [29, 19, 16] -> FIRED
  predicate `not self.is_draft_pool_worker` -> installs [29, 17, 18] -> silent
So the gate would have refused the boot that shipped and admits the one that is
fixed. Neutering assert_seed_superseded to return unconditionally turns both
falsifiers red; verified separately.

15 new tests; 84 green across the six sgl-project#797 suites. ruff clean on the new files
and unchanged in count on the edited ones (utils.py 1, scheduler.py 99, mixin
0, all pre-existing); the two codespell hits in utils.py are pre-existing and
unchanged in count.

WHAT sgl-project#797 IS WORTH, corrected. The commit that landed the install fix quoted
this runtime's own projection, "max_total_num_tokens 1213952 -> ~1335296". That
number is not deliverable and should not be cited. Measured on boot
_798_0822_0646: installing [29, 17, 18] in place of [29, 19, 16] moved the
world pool by ZERO tokens, 990656 either way. The phase-flip boot caps every TP
rank at the PP id space (448910 tokens, phase_flip_boot.py:206), clipping rank 0
from its profiled 620560 and making rank 0 the binding rank under both vectors,
since both give it ratio 29. The threshold is cap0 > 29 * min(360392//17,
375560//18) = 605056, about 35 % above today's operating point; lifting it is
the PP cut's problem (sgl-project#723/sgl-project#702), where cut and vector have to be solved
together. The DEFECT remains real on any cut without that ceiling. This commit
also annotates the install log line, whose "-> ~N" is scored on uncapped
capacity and is what made a non-deliverable figure quotable, and drops the
"roughly 10 % of the KV pool" line from the suite docstring.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 22, 2026
…or it produced

The seed-liveness gate refuses a boot, so its dangerous direction is a MISSED
disarm, not a missed refusal. It recorded "the calibration reached a verdict"
implicitly, as "the recorded vector is not None", which means a verdict that
carried an empty or None vector would have failed to disarm silently and
refused a boot that was doing everything right.

Unreachable from today's install site, where `optimal` is always a non-empty
list (partition_units, or list(active)). That is the reason to pin it rather
than the reason to leave it: an invariant held by a distant call site's good
behaviour is not an invariant, and the gate should not need `optimal` to keep
a promise it never states.

The verdict is now its own boolean. seed_liveness_state() reports it separately
from the vector, because a reader that could not tell "verdict, no vector" from
"no verdict" would face exactly the ambiguity the gate no longer has.

CAN-FAIL, EXECUTED: restoring the `_SEED_SUPERSEDED_BY is not None` test turns
test_a_verdict_disarms_even_when_it_carries_no_vector red, and it fails with
SeedNotSupersededError -- the wrong refusal, in the flesh.

13 tests in the suite, green. ruff unchanged at 1 pre-existing finding in
utils.py, clean on the test.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 22, 2026
…n a send nobody owes

The PP output ring wedged with all three ranks alive and none of them
able to move (specimen /spinning/evidence-665-f1/wedge_802f_1712/, PP=3,
--enable-phase-flip, py-spy of all three schedulers):

    PP0  _pp_commit_comm_work <- _pp_commit_pending_req_work  (:4071/:2262)
    PP1  _pp_recv_dict_from_prev_stage <- _do_recv            (:5608/:6069)
    PP2  PpChainReceiver.recv <- recv_requests   (pp_chain_receiver.py:329)

PP0 is flushing the request chain, which PP1 can only take at the top of
its next pass; PP1 never reaches that top because it is blocked in the
output receive; PP2 waits for the chain send PP1 has not reached. Three
arcs, one cycle, no timeout anywhere.

THE ASYMMETRY. The two ends of the intermediate hop apply unrelated
predicates. `_do_recv` decides to receive from THIS rank's own slot
state; the non-last sender in `_pp_send_output_to_next_stage` decides to
forward on `if pp_outputs:`, which is whatever it received LAST
iteration. The last-rank hop is matched by construction because it
consults `_pp_output_expected_for_slot` -- but that flag is the FIRST
rank's verdict, published for PP0's arc. Nothing publishes the same
thing for the intermediate hop.

WHY THE EXISTING VOID CONTRACT DOES NOT COVER IT. `pp_void_forward_
payload` (sgl-project#797) already forwards a void along this hop and stops at
`pp_first_retracting_rank`, on the argument that rank r and everything
after it has an empty slot whose receive early-returns. That holds for
the VOIDED slot -- `_pp_void_own_batch` empties it. It does not hold for
a slot the resident decode path still occupies, and both void paths keep
resident requests on purpose (`_pp_absorb_void_output` refuses to release
them as a double-free; `_pp_void_own_batch` deliberately leaves
`running_mbs` alone). The specimen shows exactly that: PP0's last act was
absorbing a void for slot 0, leaving `pp_outputs` None, while PP1 logged
`running=1 chunked=1` and re-entered the receive for a slot PP0 would
never send to. PP0 cannot know PP1's resident set, so closing this needs
a per-slot expectation every non-first rank publishes to its predecessor
-- a protocol extension, and not this commit.

WHAT THIS COMMIT DOES. The sgl-project#789 readiness gate is parameterised by wire
kind and the output receive now passes through it. The CHAN_DICT counter
was never proxy-specific -- one counter per wire, demultiplexed by
`__msg_type__` after it comes off -- so the gate reads the same true
statement about the same wire either way; `kind` selects only the
per-(src, kind) inbox peek. The ring is cut at its one cuttable arc: PP1
waits boundedly and refuses by name instead of for ever. It does not make
the missing send appear, and the error text says so.

AN ALIAS, NOT A WRAPPER. `_pp_wait_for_proxy_readiness` is now a
class-level alias for the same function object. About ten stand-in
holders across the sgl-project#631/sgl-project#757/sgl-project#787/sgl-project#789/sgl-project#791/sgl-project#795/sgl-project#797/sgl-project#798 test family
bind that name one method at a time; a delegating wrapper resolved the
second name on the HOLDER and turned 9 green tests into 5 failures.
Measured, then fixed.

Tests: test/registered/unit/managers/test_pp_output_readiness_ring_802.py
-- three real gloo processes, real PhaseFlipCounters, neutering done in
the child (spawn re-executes the module, not the test body).
  * red arm, gate neutered: stuck_ranks == [0, 1, 2], specimen reproduced
  * green arm: stuck == [], PP1 reports "sgl-project#789 OUTPUT READINESS TIMEOUT"
    and "sgl-project#802-ring", PP0 chain-flushed, PP2 requests-received
  * false-positive direction: upstream really posts -> receive succeeds
  * no-op without counters, so the non-phase-flip default path is unchanged
Mutants, both die: call site removed -> green arm red (all ranks stuck);
gate raises unconditionally -> false-positive test red.
Regression: baseline 787+789 = 9 passed / 0 failed; with this change
791b + 789 + 787 + 802 = 17 passed / 0 failed.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 22, 2026
…by hand

--hicache-host-role staging (bfabd2d) refuses --hicache-ratio and requires
an explicit --hicache-size. That refusal is only worth having if the number
it demands comes from somewhere; a hand-picked --hicache-size is exactly the
pool pin the planner's sole authority over the memory budget (sgl-project#584/sgl-project#785)
exists to prevent.

planner/hicache_staging.py derives it. Two consumers, and sizing for one of
them silently breaks the other:

  * WRITE STAGING, a bandwidth-delay product: a page stays pinned from the
    moment it is handed to the storage backend until the write acks, so the
    resident set is drain rate x that residency, widened by a burst margin
    (write-through arrives in bursts, the drain is steady).
  * READ LANDING, which scales with CONCURRENCY, not bandwidth: a prefetch
    takes its destination slot from mem_pool_host.alloc() BEFORE the storage
    read is issued (hiradix_cache.py's _prefetch path). Sizing from the write
    side alone yields a tier that is correct and quietly serialises prefetch
    -- invisible to every capacity metric, visible only as latency.

The tier takes the larger, rounded UP: rounding down would emit a tier
smaller than the derivation justifying it.

NO RIG FIT. Drain rate and latency are PARAMETERS, not constants. The planner
probes rather than carrying a measured vector from one box to another, so
nothing in this module encodes what this rig happens to do (~0.5 GB/s on its
ZFS-backed file tier); that figure appears only as a test input.

WHAT THIS DELIBERATELY REFUSES TO DO. If write-through produces bytes faster
than the backend drains them, NO finite size holds -- residency is set by the
drain, so a faster producer grows the in-flight set without bound. sustainable()
answers that question so a caller can refuse instead of emitting a number that
cannot be correct. The remedy there is backpressure at the producer, a runtime
mechanism and not a size. Note sgl-project#720's ReadBufferPool made the OPPOSITE choice
on purpose for reads -- on exhaustion it allocates a fresh uncounted buffer
rather than blocking, trading "a bounded spike for an unbounded latency" --
which is right for a prefetch worker and wrong for a write path under a RAM
budget, where re-inflating the pinned footprint is the failure the budget
exists to prevent. That ring is the next posten, not this one.

fits_pinned_host_budget() asks the sgl-project#729 authority and registers the post like
every other pinned producer, so a tier that does not fit is refused where it
is derived rather than discovered at allocation.

Also: docs/dev/NOTE_810_dangling_design_706_c1.md records that
DESIGN_706_BOOT cites a DESIGN_706_constraints document (C1/C1a) that exists
nowhere in the tree. sgl-project#810's premise is quoted verbatim at the citing site and
does not depend on it, but the supporting argument is unrecoverable. Recorded
rather than reconstructed: rebuilding it from four citation fragments would
read as primary evidence while being inference, and the next reader would
cite it as the original. The tasked "add to DESIGN_706-C1" cannot be executed
until the target exists.

Tests: test/registered/unit/planner/test_hicache_staging_810.py, 19 new cases,
hermetic (CUDA_VISIBLE_DEVICES=""). FIVE mutants, all dying on the intended
tests:
  read consumer dropped (write-only sizing) -> 1 failed
  rounds down instead of up                 -> 1 failed
  zero-drain guard removed                  -> 2 failed
  burst-margin floor removed                -> 1 failed
  sustainability predicate inverted         -> 2 failed
Source green again at 19 passed; 28 passed together with the role gate.

Suite: test/registered/unit/planner/ is 2889 passed / 1 failed after this
change. The single failure is test_rejected_evidence_pins.py's sgl-project#797 evidence
assertion, confirmed PRE-EXISTING by re-running it on a clean checkout of
dd6967c with these files removed -- same test, same assertion. No
pre-existing test changed status. ruff/black/codespell clean.

Not wired to a probe yet, and named so it is not assumed: nothing measures
the drain rate automatically, so a staging profile still needs the number
supplied. Measuring it needs load against a running server, which belongs to
another strand's boot window right now.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 22, 2026
…ost nothing allocates

`fits_pinned_host_budget()` had no caller and no test anywhere in the tree --
not even in its own module's test file, which imports the other eight names and
not this one. That alone would make it dead code. What makes it worth a commit
rather than a deletion in passing is its SIDE EFFECT: it called
`check_and_register_pinned_post`, i.e. it registered a sgl-project#729 post in the PLANNER
process.

THAT REGISTRY IS FOR PROCESSES THAT ALLOCATE. It exists to make the second
allocation in a process see the first, and it credits earlier posts back
against live availability -- `available + already_allocated` -- on the stated
precondition that those posts' bytes are ALREADY RESIDENT and therefore already
missing from `available` (pinned_host_budget.py, "the already-allocated posts
must be credited back", measured 2026-08-17). A planner allocates nothing, so
this post is exactly what the same comment goes on to name as "the real
hazard": registered and then never allocated, hence credited back without ever
having been resident, so the NEXT admission is charged too little and the
registry waves through the over-commitment it exists to refuse. sgl-project#550 closed
that window for the image posts with `_unregister_image_post`; the same comment
adds "a future producer that can leave a post registered with no allocation
behind it must do the same". This helper was that producer and did not.

Unreached today, so nothing is broken today. It is removed rather than left
because the whole cost of the defect is paid by whoever wires it -- and a
helper named "fits the budget" is exactly what a future caller reaches for.

NOT REPLACED BY A PURE VERSION, and the reason is that the question already has
a better answer elsewhere. `ServerArgs._post_hicache_staging_host_ledger`
prices the staging tier through `joint_pinned_host_error` without registering
anything, jointly with the spill pool, and by the RANK PRODUCT -- every rank
allocates its own tier, so the per-rank figure this helper used under-counts a
multi-rank boot by the rank factor, three on this rig. Keeping a second,
per-rank, planner-side reading of the same question would be two readings free
to drift, which is the shape refused in the sibling commit for the file-tier
watermark. The module docstring now says where the question is answered and
what a planner-side pre-check would have to be (pure, rank-aware) if one is
ever wanted.

Tests: two cases added to
test/registered/unit/planner/test_hicache_staging_810.py, hermetic
(CUDA_VISIBLE_DEVICES=""). The property is pinned BEHAVIOURALLY rather than by
absence: the full derivation path is exercised and `registered_posts()` is
asserted empty afterwards, so a re-added registering helper fails even if it is
spelled differently; the second case pins the removed name itself and points a
future reader at the replacement.

Mutants, 2, both confirmed dying:
  the removed helper is put back (verbatim pre-removal source) -> 1 failed
  the derivation itself registers a post                       -> 1 failed
This is the red-first evidence for a deletion: the first mutant IS the old
code, and the new guard fails on it.

Suite diff empty: test/registered/unit/planner/ is 2889 passed / 1 failed
before (16b's recorded figure, the failure being the pre-existing
test_rejected_evidence_pins.py sgl-project#797 case) and 2891 passed / 1 failed after =
2889 + the 2 new cases, same single failure id. ruff/black/codespell clean.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 22, 2026
…uard actually sits on

The pp_with_spec registry entry cited server_args.py:17583 and :17598.
Those lines now carry sgl-project#797's provenance block; the guard itself moved to
server_args.py:18958 (if self.pp_size > 1) and the spec assert to
server_args.py:18973 (verified by reading the block, not by search alone).
The overrides.py:2163 half of the cite is still correct and is untouched.

This is the drift test_rejected_evidence_pins.py exists to catch, and the
entry's own text records the previous occurrence ('the earlier
:16240-16245 cite had drifted'). A rejection reason nobody can follow back
to the code is a reason that stops being checkable.

  test_rejected_evidence_pins.py   1 failed, 5 passed  ->  6 passed
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 23, 2026
[sgl-project#797] Read the token vector back after the TP stack is built. Clean merge,
sole production file phase_flip_boot.py, untouched by the 808/739/810/814/806
line since merge-base 203ca8a.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 23, 2026
…t#816, sgl-project#810, sgl-project#806, sgl-project#797, sgl-project#790/sgl-project#777, sgl-project#817/sgl-project#820, sgl-project#818

Merge-checklist duty for the nine stages on this branch. Entries follow each
section's own house style, and the mechanism text is taken from the commits'
measured evidence rather than restated from the ticket titles.

§3 KV backing relief + the allocator cap -- UPDATED IN PLACE rather than given
a second bullet, because sgl-project#814 and sgl-project#816 are follow-on defects OF the KvRowCap
mechanism that bullet already describes: the census reading the withheld block
as a leak (340262 of 465190 ids), the lift being reachable only from a cutover
(one boot at 26.8% of its id space for the life of the process), and exposure
exceeding the backing (417850 rows over 105413 committed, the device-side
assert in masked_set_kv_buffer_kernel).

§3 HiCache staging write-through ring (sgl-project#810), new bullet, plus its two
companion refusals -- the unbounded-file-tier refusal and the boot preflight
ledger entry, the latter being why 22.01 GB of MHATokenToKVPoolHost across
three PP ranks previously reached the preflight as nothing.

§7 BAR1 deadline + loud abort -- appended the sgl-project#818 peer-liveness half to the
existing narrative: the gate could wait forever on a peer that no longer
exists, and neither Bar1CollectiveStalled (reset by every resolved read) nor
defer_stall_for_building_peer (900 s off a build marker) caught it.

§12 Robustness canon -- three new families: contradictory-flag (sgl-project#806),
read-back-after-construction (sgl-project#797), denylist-of-reasons (sgl-project#817, sgl-project#820).

§18.3 hicache staging sizing (sgl-project#810) -- §18's own rule is that a merge adding a
reusable module adds its entry in the SAME merge, and this module had none.
Records the removal of fits_pinned_host_budget so it is not reintroduced.

§18.6 mamba carry instrument (sgl-project#767, gated by sgl-project#790) and flip break-even N
(sgl-project#777).

No existing entry was contradicted. Checked before writing: none of sgl-project#814,
sgl-project#810, sgl-project#806, sgl-project#772, sgl-project#797, sgl-project#790, sgl-project#777, sgl-project#817, sgl-project#820, sgl-project#818 had a catalog entry, and
the one sgl-project#677 line (§19.2, RESTORE-NEVER-REBUILD) describes a different
mechanism than the sgl-project#677 layout hold, so it is not stale and was left alone.

Gates for the tree this documents (hermetic, CUDA_VISIBLE_DEVICES=""):

  battery test/registered/unit/{managers,planner,server_args,mem_cache}
    baseline integ @ 78d27da       44 failed
    c56d238 (through sgl-project#818)         30 failed, 8452 passed   0 new ids, 14 fixed
    1c4eadb (through sgl-project#816)         30 failed, 8461 passed   0 new ids, same set
  test_barlink_abort_gate_liveness_818.py (outside the battery dirs)  10 passed
  ruff --select=F401,F821,UP037 and codespell: 0 new findings vs the same
    file set on 78d27da (16 ruff / 6 codespell exist identically on base)
  docs-only change; codespell clean on the catalog itself
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 23, 2026
…g said it once

The #791b ballot ALREADY detects that the TP replicas' waiting-queue heads
have parted -- it rides an all_reduce(MIN) that runs once per TP-loop
iteration, so the detector and its carrier both exist. On a mismatch it
latched _prefetch_ballot_mismatch_logged, logged ONCE PER PROCESS, and fell
back to the rank-local verdict. Its own comment called a divergent queue
head "a deeper breakage than a divergent prefetch verdict" and then recorded
it exactly once.

WHAT THAT COST, from specimen /spinning/evidence-816-18f/wedge_0823_055757
(boot 0516, PP=3 with --enable-phase-flip):

  05:55:19  sgl-project#797 void storm begins on PP1
  05:55:38  this warning fires -- one line per rank, the last the process
            ever emits about it. PP0 digest 887126098, PP1+PP2 1471852626.
  05:55:53  void storm core: 7710 #797d/sgl-project#798 lines in seven seconds
  05:56:13  PHASE-FLIP DONE pp_to_tp (epoch 5)
  05:56:15  flip back ABANDONED: "live slot set divergence cannot be
            repaired this round: the group's union reaches row 240831 and
            the poorest rank has only 1208"
  05:56:15  last real forward progress
  05:56:18  the three ranks each build a DIFFERENT prefill batch
            (#new-seq 1 vs 3, #cached-token 0 vs 16384, #queue-req 6 vs 3)
  05:57:57  py-spy: PP0+PP1 in the spec VERIFY arm (eagle_worker_v2.py:2246),
            PP2 in the EXTEND arm (:2151); all three GPUs pinned at 100%,
            stacks frozen across three samples
  06:00:43  external SIGTERM -- the wedge never resolved itself

That 05:55:38 line was the earliest evidence in the whole chain, and the
latch made it a single event with no duration attached. "It healed after N
passes" and "it never healed" are the same silence to a latched logger, and
they call for opposite responses.

THIS CHANGES NO DECISION. The fall-back to the rank-local prefetch verdict is
untouched; prefetch_done_under_ballot is not modified and its behaviour is
pinned by test, because "only logging" is a claim rather than a licence. What
changes is that the divergence is counted, its onset is reported, a
persisting one is re-reported on a bounded cadence, and the RECOVERY EDGE is
reported with the length of the divergence that just ended.

NOT UNBOUNDED LOGGING, and the same boot is the argument: 7710 void lines in
seven seconds is what a per-iteration warning on a hot loop becomes. The
cadence reports streaks 1, 2, 4, 8, ... to a cap and then every cap-many
passes -- a handful of lines per minute for a divergence that never heals.

THE DECISION IS A PURE FUNCTION, AND A SURVIVING MUTANT IS WHY.
advance_mismatch_streak lives in prefetch_ballot.py rather than inline. With
it inline the recovery edge could only be checked by grepping the shipped
source, and a source grep cannot tell a live branch from `elif False:` --
measured: mutant M2 (recovery edge disabled) left all 14 tests GREEN. After
extraction the same mutant kills two tests that drive the machine directly.
Same lesson as the sgl-project#820 M5 survivor: a probe that measures presence is not
measuring reachability.

Tests, hermetic (CUDA_VISIBLE_DEVICES=""): 20 passed, 7.7s.
  RED-FIRST on unchanged 587e4c2 in a scratch worktree: 10 of 14 red
    (before the state-machine class was added). The 4 green are the
    "nothing decides differently" pins plus the detector's own can-fail arm,
    which hold on base by construction. Noted as a limit, not a strength.
  Mutants, after extraction, each killed by the arm that owns it:
    M1 latch restored               DEAD (1)
    M2 recovery edge disabled       DEAD (2)  -- SURVIVED before extraction
    M3 cadence logs every pass      DEAD (4)
    M4 total resets across recovery DEAD (1)
  Regression: every registered unit test touching prefetch_ballot or the
    uniform reduce -- 40 failed / 65 passed on branch, 40 failed / 45 passed
    on unchanged base (the 20 difference is this commit's new file).
    Identical failures, all pre-existing holder drift. Zero regressions.
  black clean; ruff (F401,F821,E731) and codespell report the same
    pre-existing hits on base and branch, none in the changed region.
  isort wanted to renormalise this file's whole import block, which is
    pre-existing disorder and not this change; reverted, so the diff is the
    block and nothing else (0 import lines touched).

Root cause of the wedge remains OPEN. This makes the earliest signal in the
chain legible; it does not stop the divergence. Acting on the mismatch
(refusing admission, or a uniform named abort off the already-agreed reduce)
needs metal and belongs to the 18 lane.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 23, 2026
Wave 3, stage 2 of the batched-window tree. WINDOW-QUEUE ticket W3 (strand
20a, preflight_pass=Y: 14 new + 31 passing, red-first over three gloo
processes, 0 new ids).

One commit, base 481c732 (the wave-1 head):

  49f39b1  [sgl-project#801] Every slot carries exactly one message, void included

THE DEFECT IS A DEFAULT, NOT A MISSING BRANCH, in the commit's own words:
`pp_first_retracting_rank` answered None for two unrelated states -- a
decision naming no retraction, and a void carrying no decision at all -- and
sgl-project#797 read both as "stop forwarding here". On a ring, "stop" is the one answer
that cannot be taken back, because the successor's blocking receive is already
posted and no later pass can satisfy it. `pp_void_relay_stop_rank` turns the
two absences into a ring position (pp_size - 1, one hop short of the void's own
source) so the ring becomes a strict relay: a non-last rank that took one
message off the wire for a generation puts exactly one back on it, void
included.

THE CROSS-SURFACE, CHECKED RATHER THAN ASSUMED. This branch's base predates
ALL of wave 2, and it edits scheduler_pp_mixin.py -- the same file the already
-merged fix/801-admission-recv-liveness edits. Two different sgl-project#801 branches in
one file is exactly the case where a clean auto-merge is not by itself an
argument, so the hunks were compared instead of trusted:

    this branch (old side)  :846-903  (pp_first_retracting_rank,
                            pp_void_forward_payload), :5883 (new method)
    wave 2 on the line      :34-41    (import block)
                            :4331-4337 (sgl-project#821's recv marker inside
                            _pp_recv_typed_dict)

Nearest approach ~1546 lines, and they are different functions. The wave-2
import hunk is an isort REORDER -- RelayPayload moves above
phase_flip_counters, nothing is removed -- which was the one way a textually
clean merge could still have broken this branch's new code by deleting a name
it needs.

The two compose rather than collide, and the composition is the useful part:
sgl-project#821 marks the one place a PP rank can disappear silently (the blocking dict
receive, marked as a DURATION before it blocks), while this change guarantees
there is always a message to receive. One instruments the wait; the other
removes the reason to wait forever.

Verified on the merged tree, not on the diff: pp_void_relay_stop_rank present
4x and _pp_blocked_recv_since present 2x in the merged
scheduler_pp_mixin.py, and the merge diffstat (106 / 64 / 596) is identical to
the branch's standalone diff, i.e. the merge took the change whole with no
fallout in the marker region.

GATE. Battery test/registered/unit/{managers,planner,server_args,mem_cache},
hermetic under CUDA_VISIBLE_DEVICES="", one battery at a time.

    baseline (tip 4f2072a)   7 failed, 8585 passed, 1852 skipped, 887 s
    stage 1 (W1+W2)             7 failed, 8638 passed, 1852 skipped, 868 s
    this stage                  7 failed, 8653 passed, 1852 skipped, 922 s
    NEW failure ids             NONE (comm against the baseline list is empty)
    fixed vs baseline           0

The +15 over stage 1 are this stage's own cases; the failure set is unchanged
and is still exactly the seven known ones in test_pp_flip_slot_hold_631.py.

The gate is not vacuum-green: this stage's new suite,
test_pp_void_send_contract_801.py, lands under test/registered/unit/managers
and is therefore actually collected, as is the modified
test_pp_retracted_pass_void_797.py.

Coverage is full rather than assumed: this venv has datasets 5.0.0, both
battery logs carry zero collection errors, and an explicit --collect-only over
test/registered/unit/managers reports 3436 tests collected. (The W4/W5 strand
reported that ITS worktree cannot collect that suite at all because 92 modules
import datasets, which is not installed there -- that limitation is not
present in this gate.)

No boot. This tree is the input to ONE batched window.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 26, 2026
…alue match

sgl-project#797 layered two matching modes and returned out of the first one. A
declared provenance naming an investigation that is NOT in the retraction
register hit `return by_investigation(declared)` -> None, and mode 2's
value match never ran. So the shipped 29,19,16 -- from retracted sgl-project#602, the
vector the whole gate exists for -- passed under any unrelated lineage.

Not an exotic input. The env twin SGLANG_UNEVEN_TOKEN_VECTOR_PROVENANCE
outlives the process that set it, so one earlier launch in the same shell
disarms every later one. The gate protects the tree-spec-DCP class from
serving on withdrawn evidence; a value-blind gate protects nothing.

FIX DIRECTION: the modes ADD, they do not replace. Mode 1 may convict a
vector whose value looks innocent; it may never acquit one whose value does
not. A declared, non-retracted provenance now falls THROUGH to the value
match instead of returning.

SCOPE, and the boundary is deliberate rather than an oversight.
PROVENANCE_MEASURED keeps its short-circuit. It names no investigation --
it asserts that this boot profiled the value itself -- so a value match
there would refuse a measurement for resembling the withdrawn estimate it
replaced. That exemption is sgl-project#797's own and is tested directly at
test_retracted_vector_boot_refusal_797.py::TestWhatMustNotBeRefused797;
widening past it would have turned a green sibling red, which is a
different ticket, not this one. Named here rather than changed silently.

RED FIRST, both axes. Before the fix: 6 failed / 6 passed -- the 6 red are
the new-behaviour cases, the 6 green are mode 1, the no-false-positive side
and the measured exemption, so the file is not blanket-red. After: 36
passed across the new suite plus both sgl-project#797 suites (boot refusal, seed
liveness), i.e. the widened refusal regresses neither sibling.

THE MUTANT, as the ticket names it: provenance = 'sgl-project#797' (real, not
retracted) over vector 29,19,16 (retracted). Caught, at the predicate and
at the boot path through resolve_cp_token_ratios -- sgl-project#182's rule, a gate no
boot arrives at is not a gate. The gcd-written form 58,38,32 is caught on
the same path, so the fix is not one rewrite wide.

CLASS SWEEP. The shape is "a declared lineage short-circuits a value
check". planner/rejected.py, the register next door, has no provenance
concept at all and cannot carry it. distributed/utils.py:690 reads like the
same shape and is not: it is the env-over-flag fallback that PRODUCES the
declaration, and None there arms the value match rather than disarming it.
retracted.py was the only instance.

Two comments the fix made false are corrected in the same commit: the
module docstring's "mode 1 short-circuits before any value comparison
happens" and mode 2's description as the UNDECLARED-provenance mode.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 27, 2026
…ker per abort

The abort path is the leading candidate for the rank-local slot divergence
behind the PP output-ring wedges, and the code says why itself. `abort_request`
carries sgl-project#631 pin 4 verbatim: "an abort applied on one rank before its peers
DIVERGES THE REPLICATED LIVE SET mid-flip" -- and the guard that prevents it is
conditional, `if window is not None and window.active`. The
`phase_flip_abort_window` is activated only on arming and drained after the
cutover, so OUTSIDE an armed flip `_abort_request_now` runs directly, on
whichever rank processes it, in whichever pass it lands. The hazard is
recognised, named, and closed only inside the flip window.

THE CHECK COULD NOT BE MADE, WHICH IS THE POINT OF THIS COMMIT. Grepping the
three wedge logs for an abort near the onset returns nothing -- and that
nothing is worthless. Every marker inside `_abort_request_now` is
`logger.debug` and the boots run `log_level='info'`: measured, ZERO DEBUG
lines in boot_accept2e0827_0827_0454, boot_accept2e0827b6_0827_0613 and
boot_802f_staged1_0822_1716. So "no abort before the wedge" is a LOGGING GAP,
not a finding. This is the marker-absence trap sgl-project#843 hit on this same path
("the only refusal marker was a logger.debug on a boot running
log_level='info', so it could never appear"), and reporting the null as
evidence would have been the second instance.

ONE SUMMARY LINE, not the per-request ones. `abort_all` can name the whole
live set, so promoting those would trade a blind spot for a flood -- the sgl-project#801
void-streak shape in reverse. This says an abort happened, when, whether the
flip window deferred it, and what it targeted, which is everything the
discriminator needs to place it against a wedge onset. `deferred` is now read
once into a local and used for both the log and the branch, so the line cannot
disagree with the path taken.

WHAT THE SAME SWEEP DID FIND, recorded because it is real and does NOT support
the abort hypothesis:

* Boot 1's wedge onset (last progress 05:08:59) is preceded by EIGHT SECONDS
  by `sgl-project#905 HOST-POOL DOUBLE-FREE about to raise ... 8192 of 8192 index(es) are
  in range but not allocated, span [3530, 14937]` on all three ranks, and the
  `[sgl-project#703 flip-writeback]` drain fails with it. Boot-1-specific: zero such
  lines in boot 6 and zero in 1712. The magnitude is the same 8192 the sgl-project#927
  hit prefix carries, on the HOST pool rather than the device one, which is
  worth linking rather than filing twice.
* Specimen 1712 shows a rank-local admission-congruence retraction (`sgl-project#797
  PP-ADMISSION pass voided on rank 1 ... told=147456 local=139264`) -- that is
  the in-loop site, already covered by the void relay whose default sgl-project#801
  widened.
* So the three wedges do NOT share one proximate cause in these logs, and the
  single-class assumption should not be carried further without evidence.

NO BEHAVIOUR CHANGE. The branch is byte-identical; only the marker is new.

Import smoke + wiring check green (the marker precedes both branches, and the
branch reads the same `deferred` the line printed).
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 27, 2026
…t causes is bounded

THE CLASS, AND THIS IS ITS THIRD INSTANCE.
`_pp_reconcile_incoming_admission` resolves a rid through a chain of lookups
and, on a total miss, wrote 0 into `local_match_lens`.
`reconcile_pp_admission_decision` then read that 0 as a MEASUREMENT and voided
the pass. #797c patched the `chunked_req` miss, sgl-project#798 patched the wrong-slot
miss -- each added a lookup and left the miss answering with a number, which is
the only reason each looked like a fresh defect instead of the same one again.

sgl-project#944 is the RUNNING BATCH miss, and it is where a request lives once its
chunked prefill finishes: not in the waiting queue, not in `chunked_req`, not
in the slot's chunked req. `self.running_batch` is on the same object and read
elsewhere in the same file; this resolution chain referenced it zero times.
Measured under real agent-shaped load (long SHARED prefixes): 2106
`unhonourable prefix` events, 2107 voided passes, and a three-rank hang 35 s
after health, watchdog kill at 300 s. Live py-spy at wedge onset, not the
post-mortem: PP0 computing, PP1 blocked in `_pp_drain_voided_proxy`, PP2
blocked in `hicache_collective.join`.

WHAT LANDS, AND WHY IT IS ONE COMMIT AND NOT THREE.

1. `running_batch` is the fourth lookup, and a rid found in NONE of the four is
   `UNKNOWN_MATCH` (-1), never 0. The sentinel idiom is adopted, not invented:
   `tp_head_congruence._ABSENT_MATCH` is the same -1 one file over.

2. `PPAdmissionEntry.unresolved` -- a field of its own, on the wire. Encoding
   the miss as a special value of `observed_local` would be the same class one
   level up: that field's readers treat it as a LENGTH and feed it to
   `_learned_floor`, which clamps the next round's offer, and a floor learned
   from a number nobody measured is exactly the defect. It crosses the wire
   because the rank that OBSERVES the miss is never the rank that can ACT on it
   -- only PP0 chooses `told`.

3. `told <= 0` is honourable unconditionally. A zero offer demands no prefix
   reuse, so no lookup result -- not even a failed one -- can make it
   unhonourable. This fell out of the arithmetic for free while a miss answered
   0 (`0 >= 0`); at -1 it stops falling out, and leaving it implicit retracts
   the FIRST, congruent round of every request. For a rank that did resolve it
   is a no-op (`local >= 0 >= told`).

4. `UNRESOLVED_DEFER_CAP = 3` with a per-rid count in
   `PPAdmissionCongruenceGuard`. THIS IS WHY (1)-(3) COULD NOT SHIP ALONE:
   `_learned_floor` is what damped the re-offer, it is fed from
   `observed_local`, and a miss must not set it -- so the sentinel by itself
   makes the 2106-loop WORSE than the 0 did. The old 0 was a false measurement,
   but it at least clamped. At the cap PP0 emits ONE loud refusal naming the
   rid and all four lookup locations, and pins the next offer to `told=0` --
   the only offer honourable without a measurement, hence the only terminator
   available once the measurable one is gone. The count clears only on a pass
   that actually SERVED the rid (sgl-project#552's lesson: a defer that resets its own
   counter makes the bound unreachable). `<= 0` disables the bound, so it can
   be neutered on its own in a can-fail proof.

THE GROUP DEFERS, NOT ONE RANK. A defer that only one rank takes IS the next
divergence. Downstream ranks only REPORT `unresolved`; the pass is voided by
the existing sgl-project#797 mechanism, which is already group-uniform
(`pp_pass_should_void` ORs the incoming flag and never clears it). Whether to
defer again or escalate is decided once, by PP0, from a count that rode the
wire. No new collective, no new send.

TESTS.

New, `test_pp_unresolved_defer_cap_944.py` (22 tests + 3 subtests): the told=0
hoist, the two populations separable on the wire and through the codec, the
cap's counting/escalation/clearing/per-rid scoping, and TERMINATION with a
can-fail arm that neuters ONLY the cap and shows the offer standing at 4096 for
23 rounds with nothing admitted.

New, `test_pp_unresolved_group_defer_gloo_944.py` (4 tests, 3 real gloo
processes, deadline-bounded): (a) one rank's miss defers the group and then
resolves, with no floor invented on any round and the sgl-project#944 line logged where
the sgl-project#791 line must not be; (b) unresolved everywhere -> exactly one loud
refusal, `told` ends at 0, served exactly at the cap, never a hang; (c) the
danger mutant (`observed_local=0` + `unresolved=False`, the exact pre-sgl-project#944
shape) is caught. (c) also records that the MUTANT TERMINATES FASTER than the
fix -- the false 0 IS a clamp, which is why this defect twice looked like it
was working, and why "it terminated" is worthless as evidence here.

`TheConsumerSweepRatchet` is the class fix's future half. #797c and sgl-project#798 were
each fixed by adding a lookup, and neither asked who READS the value; that is
the only reason the same defect survived twice. It pins the file set reading
`observed_local` and `UNKNOWN_MATCH`, in src AND test. The four contract
inversions this change required were found by grep; the fifth will be found by
a red test.

Four contract inversions in `test_pp_retracted_pass_void_797.py` and
`test_pp_reconcile_slot_blind_798.py`, each with the reasoning recorded at the
site and none deleted: they pinned the miss as the specimen's exact 0, correct
while a miss was SPELLED as a measurement. The behaviour they reproduce is
unchanged -- still retracted, still absent from `effective`. One test renamed
(`..._still_reads_as_zero` -> `..._is_reported_as_unresolved`): its name
outlived its contract by one ticket and would have kept teaching the sentence
that cost three boots.

TEST RESULTS. Frozen before/after over the derived blast radius (every
registered module mentioning `pp_admission_congruence`, `PPAdmissionEntry`, the
wire codec, the three reconcile/send/recv mixin methods, `observed_local`, or
the return-trip pair), each module ALONE IN A FRESH PROCESS, identical script
both sides, baseline in a detached worktree at 70bbd98:

    BEFORE  20 modules  209 passed  0 failed
    AFTER   22 modules  235 passed  0 failed

Every shared module matches its baseline count exactly; the two deliberately
red pins are green at their original assertions and 797 is back to 31/31.
Extraction count probe passes on both sides (extracted FAILED names == summary
failed count). ruff (F401,F821,UP037), ruff format and codespell clean on all
six touched files.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 28, 2026
… it had to prevent

DETERMINISTIC, 2/2, AT EXACTLY 7 BATCHES. Pin dc4895e, boots
boot_943bx_dc4895e1dc_0828_000240.log and _001113.log, no CUDA error
involved:

    scheduler.py:9286 in _get_new_batch_prefill_raw
      assert self.chunked_req is None
    AssertionError

immediately after `sgl-project#798 PP-ADMISSION pass voided on slot N` and
`#797d own pass voided on slot N`.

THE ASYMMETRY. `_event_loop_pp_body` runs two voids and calls the second
"AND THE MIRROR OF IT". They were not mirrored. sgl-project#797 (this rank
retracted) sets `_pp_admission_pass_voided` in `_pp_void_retracted_pass`,
BEFORE `get_next_batch_to_run`, so scheduler.py's guard refuses the pass
and it builds nothing -- which is what that guard's own comment demands:
"The retraction voids the pass, so the pass must build nothing at all."
sgl-project#798 (this rank's UPSTREAM did not launch) set the same flag AFTER the
call, so its pass ran the whole of `_get_new_batch_prefill_raw` --
advancing `chunked_req`, possibly adopting a fresh `new_chunked_req`,
taking lock refs, emptying the waiting queue -- and was then unwound
retroactively by `_pp_void_own_batch`.

THE UNWIND CANNOT BE COMPLETED, so this is a guard and not a bigger
unwind. `_get_new_batch_prefill_raw` reaches `_retract_decode_and_requeue`
(the sgl-project#679 relief ladder, the #888b seat yield), which sends `AbortReq` to
the tokenizer over `ipc_channels` at scheduler.py:9626-9632. A message
already delivered to another process is not scheduler state and no
handler can put it back. Every other unrestored item is a matter of
writing more restore code; this one is not.

THE CONDITION WAS KNOWABLE EARLY ALL ALONG, which is what makes the guard
possible. `_pp_upstream_launched_incoming` is written by
`_pp_recv_admission_decision` (scheduler_pp_mixin.py:5482), whose own
docstring records the placement -- "Positioned in `_event_loop_pp_body`
strictly BEFORE `get_next_batch_to_run`" -- and the call site agrees
(:1959 receive, :2073 plan). One shared predicate,
`pp_upstream_void_pending`, is now read at both moments, so the refusal
and the forward cannot drift apart. The void is still FORWARDED from
where sgl-project#798 always forwarded it.

CARRIED, NOT DROPPED. Once the guard empties the slot, `self.mbs[mb_id]`
is None on exactly the passes the sgl-project#798 site used to find non-empty, so
two instruments would have gone quiet unnoticed:
  * the sgl-project#801-spin livelock streak, which raises at 512. The guard now
    records `_pp_upstream_void_withheld_work` (this rank HELD work and was
    refused anyway) so the streak counts what it always counted. It is a
    pre-plan superset of "derived a batch" -- the safe direction: it can
    make a genuine streak visible sooner, never hide one, and an idle rank
    still clears it.
  * `_pp_idle_void_suppress_log`, whose contract is that it "can never
    outlive this pass". Its consume sat AFTER the empty-slot early return,
    so it would have leaked a True into the next, unrelated sgl-project#797 void and
    silenced a record nothing asked to silence. Read and cleared at the
    top now.

has_chunked_req DELETED, not left to mislead again. `add_one_req` carried
it as a parameter its body never read; it only ever forwarded to
`add_one_req_ignore_eos`, and upstream removed that last consumer in
8cc7726 ("Super tiny remove unused argument"). Reading it as the
guard against a second chunked request cost a boot window. What actually
holds that invariant is budget arithmetic, and that is now recorded at
the assert -- including its measured reachability: witnesses under
/spinning/evidence-665-f1/witness_951/ drive the real PrefillAdder into
three states where `add_chunked_req` returns the request while leaving
`rem_chunk_tokens` positive, and a mid-pass replenishment of
`rem_total_tokens` then lets the loop mint a second chunked request.
witness_941_d2 is the negative control and does not reproduce without it.
That general case is a SEPARATE posten -- it needs no PP void and its
danger direction (wedging a request mid-prefill, the sgl-project#858 shape) needs
its own analysis. The assert stays: it is the honest watcher.

TESTS. test_pp_upstream_void_before_formation_951.py, 11 tests, red-first:
3 fail on dc4895e and pass here. The five over-fire arms (first rank,
pp_size<=1, gapped wire, healthy upstream, healthy chunked continuation)
are green in BOTH states, which is what makes the red mean something -- a
guard that over-fires voids every pass on PP0 and serves nothing at all.
Two mutants on the carried instruments, each killed by exactly its own
test: consume-after-return revives the suppress leak, emptiness-only
revives the streak reset.

Desk gate scripts/gate_tier2_partitioned.py, frozen both sides on this
tree: BEFORE 4757 passed / 2 failed / 18 skipped, AFTER 4768 passed /
2 failed / 18 skipped. Delta +11 is exactly this commit's new tests.
Identical failure set both sides (test_collective_family_siblings_610.py,
2 genuine, pre-existing at the pin); count check 2 == 2. The lane shift
(wide -15, narrow -30, serial +56) is the gate's own sha256 rule
demoting the four touched test modules to the serial lane. ruff parity
exact per file (105/105, 3/3, 21/21, 1/1) with no finding on an added
line; black run only on files that were clean at the pin.

NOT BOOTED. Boot half is /spinning/gpu-arb/TICKET_951_WINDOW.md.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 28, 2026
… was in the tree

ROOT, per-rank coverage of window-958 boot 2 (pin 78d030e) plus both boot
logs. Under pure PP the HiCache storage-prefetch veto is a RANK-LOCAL verdict:
prefetch_ballot.prefetch_done_under_ballot returns the local value when the
ballot is None, and the ballot rides _update_uniform_pool_budget's reduce on
tp_cpu_group -- a group the PP loop never passes, and which has world 1 under
pp_size>1 in any case. The ranks therefore decide independently. scheduler.py
:9048 (`continue` after _note_skip("prefetch_pending")) is the LAST line PP0 and
PP1 both ran; :9050 is the first PP0 ran ALONE. Everything downstream --
add_one_req, `self.chunked_req = adder.new_chunked_req`, the unconditional stash
at :7010-7011 -> stash_chunked_request:5449 -> mem_cache/common.py:169 ->
cache_unfinished_req -- is PP0-only. PP0's radix tree gains a 1250-token prefix
PP1's and PP2's never received.

From there it is self-sustaining. PP0 matches its own tree and offers told=1250;
PP1 measures local=0 HONESTLY against its own; sgl-project#791 retracts, sgl-project#797 voids, the
requeue resets the REQUEST and nothing resets the TREE. PP1 can only acquire the
prefix by running the batch it is refused for lacking the prefix.

WHY THE EXISTING BOUND COULD NOT REACH IT. PPAdmissionCongruenceGuard is, in its
own words, "RID-SCOPED, ONE-SHOT", and its termination argument is per rid: each
new retraction for THAT rid lowers THAT rid's floor. The argument is sound and
silent about the POPULATION. The shortfall is a property of the TREE, so every
fresh rid over the same prefix starts unclamped and buys its own voided pass --
six distinct rids in one second, all told=1250 local=0, which is why
_learned_floor was measured RUNNING and LOWERING on PP0 and never bound.

FIX: the same actuator, scoped to the prefix the offer was made over. No new
mechanism and no new collective -- the recorded fatal (the 2026-08-17 HiCache
ack-count reduction) rules out a collective on this path, and a group-uniform
INSERT is structurally impossible: the radix value is a tensor of that rank's own
KV slot ids, dereferenced as real memory by four consumers (allocator free at
unified_radix_cache.py:1713, evict at full_component.py:96, HiCache backup at
:2284, write-back into req_to_token at :1325), and the GDN mamba component
donates a live state slot a non-computing rank does not have.

offered_prefix_key() names the prefix by a blake2b fingerprint of its tokens --
NOT hash(), which is PYTHONHASHSEED-salted and would disagree between the very
ranks this keeps congruent (tree_congruence's constraint 3, same lesson). Length
is mixed in so a prefix and its extension cannot share a floor. prefix_key=None
leaves the pre-sgl-project#963 rid-scoped path byte-identical.

PRIOR ART, gated rather than duplicated. #616g's group-MIN reduces
available_size(), a different quantity, and is switched OFF in the PP phase
(tp_cpu_group world=1). sgl-project#823's tp_head_congruence MINs the right quantity but
only REORDERS, and its enforcer_gate returns GATE_OFF_TP_WORLD_OF_ONE here.
sgl-project#825's tree_congruence detects exactly this divergence and emitted 0 in BOTH
boots: in the PP phase on_round is reached only via scheduler_pp_mixin.py:2477
with require_armed_and_parked=True, so it samples only at an armed flip -- which
the livelock prevents. That is this instance's compensator-reachability gap.

RED-FIRST, and the first red was rejected as worthless: all six cases failed on
the SIGNATURE (TypeError), not on behaviour. Four mutants were then run against
the finished fix and their errors READ:
  consultation withdrawn  -> assert 6 == 1   (boot 2's six voided passes, exactly)
  learn withdrawn         -> assert 6 == 1
  clear-on-serve withdrawn-> assert 0 == 1250 (permanent loss on a healed prefix)
  key-blind clamp         -> assert 0 == 1250 (loss on a prefix nobody reported)
The last two are the danger direction: this fix must never discard a prefix
every rank holds. It clamps only against an OBSERVED shortfall on a SPECIFIC
prefix, and clears the moment the group serves that prefix, so the cost is one
voided pass once rather than one per rid for ever. Double-prefill law: the
requeue loses 0 today and still loses 0; what changes is that it terminates.

An UNRESOLVED miss (observed_local=None) teaches the prefix floor NOTHING -- the
stakes are strictly higher than for the rid scope, since a floor invented from a
number nobody measured would cap every request over that prefix. The sgl-project#944
consumer ratchet CAUGHT this file on its first gate run and is now registered.

NUMBERS, desk gate scripts/gate_tier2_partitioned.py, CVD="" :
  BEFORE 2 genuine (test_collective_family_siblings_610.py x2), 685.00 s
  AFTER  2 genuine (the same two), plus 14 new tests green
  count probe: 2 named == 2 summary, SUBFAILED and ERRORS included
  ruff clean on both touched files.

NOT CLAIMED: no boot, no metal. The divergence SOURCE (the rank-local prefetch
verdict under PP) is named here and left open -- closing it needs the sgl-project#791 ring
lap to carry a prefetch-pending fact home, which is its own posten. This makes
the system self-healing against a divergence however caused.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 28, 2026
Root (R3b, window-flip-0828 boot 2): batch_is_full is a PASS-scoped memo
(written at the adder's stop condition) stored on the scheduler-lifetime
running_batch. Every clear site is either a FINISH path the
requeue-without-finish family (sgl-project#797/sgl-project#798/#791b/sgl-project#971) destroys before it
runs, or sits behind 'if not running_batch.is_empty()' -- unreachable at
exactly running=0, the defect state. Boot 2: three
gate=batch_full_or_empty_queue(batch_is_full=1,queue=N) declines at
running=0 with free seats and a waiting queue, self-sustaining until an
external kill. The #888b relief lies on the path but its verdict
producer sits behind the same emptiness guard (registered #962b).

Fix: ONE junction at the gate read -- batch_is_full AND is_empty() is
re-derived from the seat test (get_num_allocatable_reqs), not trusted.
Re-derive, not clear: a genuinely exhausted seat table comes straight
back True. Not an eighth clear site: any future requeue path inherits
this without knowing the flag exists. In-tree precedent: the
retract-RPC handler. #888b relief and the seam hook untouched; arm 2 of
the new test pins that #962b's producer stays unreachable at running=0
so the posten stays open rather than silently closed.

Tests: test_batch_full_empty_invariant_962b.py, 8 arms -- red-first on
the REAL void path (gate declined at running=0 with seats free,
note=gate=batch_full_or_empty_queue -- boot 2's own line shape), void
family flag hole filled, green-guard for the 8-resident full-table case
both sides, can-fail mutant. Baselined modules byte-identical.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 28, 2026
…and PP0 names it

Boot 3 of window-flip-0828: after a sgl-project#797 void the follower holds a
sgl-project#971-parked chunked continuation in self.chunked_req (never in any
queue); PP0 cannot see it, its uniform cap is MIN-reduced before
occupancy is loaded, and the void's requeue tail-rotates PP0's copy so
it never reaches a seat. The follower's add_chunked_req appends its
continuation unconditionally, so every forwarded schedule is illegal on
the follower: 514 refusals (191 missing + 323 'admitted rid(s) the
decision does not name'), 512 voids, sgl-project#801-spin guard death in 2 min.

Fix, the register's named sgl-project#968 direction -- the fact rides EXISTING
messages, no new wire, no collective: (rid, executed_len) of a parked
continuation is stamped on (a) the per-pass decision downstream and (b)
the output/void return lap homeward. Named deviation from the design
brief: the ring is asymmetric (the last rank sends output DIRECTly to
PP0, so a middle parker never gets onto the output lap alone) -- hence
both messages, and facts are MERGED per hop, never overwritten,
or a middle rank's fact dies one hop before its consumer. PP0 absorbs
before the admission loop, REORDERS its own waiting_queue so the
ordinary loop admits the rid (reorder, never admit: no second admission
authority, no seat pricing -- once named, the continuation is a
cap-counted member on every rank: no extra, no missing), and clears the
fact when a decision names the rid (self-correcting: the next lap
re-stamps if still parked). No prefix is ever reduced (Kein-Doppel-
Prefill); all seven helpers module-level (holder lesson, test-pinned).

Single-slot hypothesis CONFIRMED (double-sourced): chunked_req is one
attribute, overwritten per truncation and cleared on launch -- PP0
cycles it (518 admits / 7 rids, the stuck rid named 2x), the follower
pins it (514x). That is why the carry must exist.

Tests: test_pp_parked_continuation_carry_968.py, 21 tests -- red-first
GENUINE against the unmodified pin 3c470fb in a detached worktree:
18 failed / 3 passed (names == summary 18==18), 21 passed on the fixed
tree; legacy arm byte-identical without the stamp; no-double-prefill
arm asserts zero prefix reduction; can-fail arm neuters the absorb.
Gate: 42 modules BEFORE/AFTER byte-identical 510P/9F, the 9
pre-existing name-identical both sides (prefetch_symmetry_580 7 +
ring_commit_973 2 -- the 973 pair is the known gloo hang-shape arms,
reproduced solo on the unmodified tree; environmental, posten noted).
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 28, 2026
…e class behind both

The 797 suite went red at 8be86f5 in two arms. Neither is a regression;
both are sgl-project#984's premise arriving in a suite written against the old one.

test_a_chunk_started_this_round_is_un_started_not_kept asserted that a chunk
started this round is 'retracted like any member'. The void is now symmetric:
members are PARKED and re-queued with prefix and pages intact, and only an
already-finished request still takes the release path. The arm's actual point
is its other two assertions -- the chunk is UN-STARTED and is back in the
queue, so no rank is left uniquely mid-chunk -- and those never moved. How it
gets back to the queue is the disposal's business; that it gets there is
sgl-project#797's.

test_instr19s_raise_returns_when_the_pre_admission_value_is_blinded was a
DEAD CAN-FAIL. Its mutant blinded pp_void_keeps_request to force a retraction,
hence the reset shape, hence instr19's AttributeError. After sgl-project#984 an un-kept
member is parked rather than reset, so the mutant injected no fault at all and
the arm failed on assertTrue(chunked.retracted) with nothing having retracted
anything.

THE CLASS, since this is the second instance in one commit (the first being
test_the_underflow_detector_can_actually_fail in test_pp_void_lock_ref_969.py,
aimed at a cache_finished_req the void no longer calls): a change that
converts RETRACTION into PARKING disarms every can-fail proof whose injected
fault worked by causing a retraction. Such an arm does not report a safe
system -- it reports nothing, while looking like a red a tired reader would
'fix' by deleting the assertion.

The re-aim needed TWO neuters, and that is a result rather than a concession:
aiming at the park verdict alone left the arm red, because pp_void_keeps_request
still guards the top of the loop and continues past the chunked request before
any verdict is asked. The instr19 shape now requires defeating two independent
guards where one sufficed before. Both are neutered visibly, because a
can-fail proof must state the true depth of what it defeats rather than pick
the smallest edit that goes red.

Test: 31 defs / 31 passed at 8be86f5 (count probe), 89.51s.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 28, 2026
…resident at all

Boot 14 (cf16281) died on PP1 after 39 s: `assert self.chunked_req is
None`, scheduler.py:9784 in `_get_new_batch_prefill_raw`. Third recorded
fundstelle of this family after :9286 (sgl-project#951) and :9367 (sgl-project#959).

ROOT, and it corrects my own attribution in the boot-14 register entry.
The invariant behind that assert is held "by ARITHMETIC, not by a check"
(scheduler.py's own comment). sgl-project#959 therefore gave the two `add_one_req*`
mint sites an explicit `chunked_req_outstanding` check, and SKIPPED the
third -- `PrefillAdder._add_scheduled_req`, the forwarded-schedule
execution path -- reasoning at schedule_policy.py that it "already has its
own (`carried_chunk`)".

That is the guard-comment-names-the-hazard trap. `carried_chunk` answers
"is THIS request the resident continuation". The invariant needs "is there
a resident continuation AT ALL". It covers a request being re-announced; it
does not cover a DIFFERENT named request becoming a second continuation
while the first is resident. Two of three sites guarded, one not.

Boot 14 is that gap on metal: the resident continuation survived
`add_chunked_req`, so `chunked_req_outstanding` was True and both sibling
sites correctly refused -- and this site minted anyway, on another rid the
same forwarded schedule named.

sgl-project#994 EXPOSED THIS, IT DID NOT CREATE IT. Boot 13 never reached the line
because the sgl-project#791 geometry refusal killed every pass before a batch was
built; sgl-project#994 removed that refusal. My register entry attributed the death to
sgl-project#994's effect on `rem_chunk_tokens` arithmetic. That was wrong: both
sibling sites were already guarded and did refuse. The arithmetic was not
the hole; the missing third guard was. Withdrawn here rather than left
standing.

FIX: the sibling guard, at the site that lacked it.

WHY A PASS REFUSAL AND NOT A REQUEST SKIP -- the danger direction, which is
the whole question here. On a forwarded schedule this rank may NOT drop a
named request: the upstream's hidden states for it are already on the wire,
which is what this same method already raises `PPScheduleRefused` for a few
lines above. Running the chunk WITHOUT announcing it is worse -- the
continuation would be untracked and re-prefilled next pass, the double
prefill the standing law forbids. So the disposal is the one this path
already owns: refuse the PASS by name, let sgl-project#791/sgl-project#797 void and re-derive.
It cannot starve: the resident continuation is consuming chunks, and when
it finishes `chunked_req` is None and the schedule is executable. This is
the same direction sgl-project#959 chose ("the resident continuation is never the one
to give way; the fresh admission is") -- not the sgl-project#858 mid-prefill wedge,
which would be clearing `scheduler.chunked_req` instead.

FUTURE CHECK: `grep -c 'if self.chunked_req_outstanding:'` over
schedule_policy.py is now 3 and equals the number of `new_chunked_req`
writers. A fourth writer that skips it is visible as an inequality rather
than as a boot death.

Execution proof (speed mode, one instrument per link):
`note_second_continuation_refused(req, "_add_scheduled_req")` counts it and
the refusal text names itself, so boot 15 measures whether the guard fires
at all -- absence of the assert alone would be green-by-absence.

Evidence: desk. py_compile; local-import dominance and guard-before-raise-
before-mint verified by source order in the loaded module; guarded-site
count 3 == mint-site count 3. Belegstufe: DESK-BEWIESEN.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 30, 2026
…serve has NO post, and its two guards share one gate

Operator order mid-turn: stop chasing the OOM symptom rank by rank, identify
every CONSUMER and RELEASER of the budget structurally. Done with the index
tools; this replaces the four-arm symptom chase of 3bf2e78 with its root.

TOOLS/QUERIES (reporting duty): devindex `where rank_gpu_memory_mib` ->
the single profiling site; `symbol_profile
ModelRunnerKVCacheMixin._profile_available_bytes` (83 outgoing calls);
`trace_value available_bytes direction=back` (the full deduction chain);
`where budget_posts`; `where post_capture_kv_active`; `symbol_profile
activation_reserve_mb` / `post_capture_kv_sizing_planned`. Read only as the
last mile at the file:line those produced.

THE LEDGER IS CLOSED, so the post list is EXHAUSTIVE, not a sample:
`budget_posts` has ONE writer (mixin:769) and all 18 reads inside
`_profile_available_bytes` (mixin:710-1028). Posts, in order:
  1. "weights + runtime state"            :792
  2. "gapped corridor holdback"           :795 / :837  (the user's free column)
  3. "mamba pre-capture reserve"          :815  -- SEE GATE BELOW
  4. mamba state pool (decomposed)        :841
  5. "GGUF dequant scratch"               :870  (0.0 here, non-GGUF)
  6. measured KV budget correction        :877  (0 unless SGLANG_MEASURED_KV_BUDGET)
Everything left becomes KV pool. THERE IS NO POST FOR CUDA-GRAPH CAPTURE.

THE GATE, and why this is a defect shape rather than a missing line. Two
independent mechanisms could each price the 810 MiB decode-graph capture,
and BOTH hang off the SAME predicate `post_capture_kv_active` (mixin:2622):
  - `post_capture_resize_kv_pool` (:2629) re-sizes from MEASURED free AFTER
    capture -- capture bytes are already spent when `free_gb` is read;
  - the pre-capture post sgl-project#3 at :807 `if self.mambaish_config is not None and
    self.post_capture_kv_active:`.
So one flag disables a mechanism AND its own fallback. Measured on the live
boot, from the log's own sgl-project#797 instrument, all three ranks, both sizing
passes: `post_capture_kv_active=False`. Neither guard is in force.

WHY THE BUDGET PATH SPECIFICALLY. The else-branch (:817) opens with
`slack_gb = pre_model_load_memory * (1 - mem_fraction_static)` -- that IS
the upstream graph reserve, and `post_capture_kv_sizing_planned`'s own
docstring calls it exactly that ("whether the mem_fraction heuristic may
skip the graph reserve"). The `--rank-gpu-memory-mib` branch (:785-815) has
no equivalent line. UPSTREAM-MINIMAL READING: the fork's absolute-MiB budget
REPLACED mem_fraction_static and silently dropped the reserve that
mem_fraction_static implicitly carried. Second bookkeeping beside an
upstream truth -- so the fix is to make the budget path carry the upstream
reserve (or enable post-capture sizing on it), NOT to invent a third
fork-own reserve.

INSTRUMENT-TEXT-LIES (that memory class, again, and it is why the reserve
looked covered): the boot line
  "[auto-mamba] ... activation_reserve=1.00 GB; remaining VRAM -> KV pool"
reads as "1 GB is being withheld". It is not. The number is the module
constant MAMBA_AUTO_ACTIVATION_RESERVE_MIB=1024 (mixin:145), passed only
into `_fit_mamba_pool_to_budget` (:2125) as a FIT CONSTRAINT on the mamba
pool, whose own docstring says it "engages ONLY when the size computed by
the caller would leave less than MAMBA_CEILING_FIT_MIN_KV_MIB for the KV
pool". Nothing is deducted; `handle_max_mamba_cache` returns
`total_rest_memory - mamba_state_memory` (:2415) and no reserve term.

THIS EXPLAINS ALL FOUR ARMS of 3bf2e78 without a single new measurement:
  - std/gdncov boots because the weights are LARGER, so less flows to the
    pool and the UNBUDGETED residual happens to exceed 810 MiB. Luck, not
    design -- the reserve was never in force there either.
  - lm_head OOMs because the freed 1212 MiB went entirely to the pool and
    the residual fell under 810 MiB.
  - the knob is non-monotone because nothing TARGETS a capture reserve:
    free-at-capture is an uncontrolled residual, and the uneven token split
    redistributes whatever a rank gives up.

EVIDENCE TIER: DESK-PROVEN on the code (closed-ledger argument, exhaustive
by construction) + BOOT-PROVEN on the gate value (sgl-project#797 line, 3 ranks x 2
passes, boot_855_makespan_0840f82601_0830_093650.log).
NOT DONE: no fix written, no capture-cost measurement of the reserve that
should be booked. sgl-project#1025 now has its file:line and its shape.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 30, 2026
…lden statt bauen (Koordinator-Klausel)

Koordinator-Entscheid: post-capture sizing auf den Budget-Pfad bringen, mit
der Klausel "Wenn der Fix mehr als S/M wird: melden statt bauen". Gescoped;
die Klausel greift. Kein Code geschrieben.

WARUM DER ENV-SCHALTER ALLEIN NICHT REICHT. `SGLANG_ENABLE_POST_CAPTURE_KV_SIZING`
ist EnvBool(False) (environ.py:489) und in keiner Boot-Env gesetzt -- das ist
der erste Verweigerer. Der zweite ist `dcp_size == 1` in
`post_capture_kv_sizing_planned` (server_args.py:14345), und der bindet genau
dort, wo es weh tut. Gemessen an der Boot-Reihenfolge
(boot_855_lmhead5_..._093325.log):

  09:33:55  Sizing-Pass 1, sgl-project#797 sagt dcp_size=1   (PP-Layout)
  09:34:07  cuda_graph_capture: 0.000-0.001 s     <- de facto NO-OP
  09:35:53  Sizing-Pass 2, sgl-project#797 sagt dcp_size=3   (TP/Flip-Layout)
  09:36:10  cuda_graph_capture: 6.34-6.39 s       <- DIE echte Capture
  09:36:11  OOM, 810 MiB

Die 810-MiB-Capture liegt VOLLSTAENDIG im dcp_size=3-Regime. Der
env-Schalter wuerde also nur den Pass freischalten, dessen Capture 1 ms
kostet, und den teuren Pass weiter verweigern lassen.

WAS OPTION 2 DAMIT WIRKLICH VERLANGT -- die sgl-project#592-DCP-Sperre heben. Der
Kommentar an server_args.py:14296-14344 benennt die fehlende Arbeit selbst
und praezise: `pool.finalize_backing()` reicht
`config.max_total_num_tokens` direkt als Zeilenzahl an
`_finalize_backing_tokens` weiter, waehrend alle fuenf
Pool-Konstruktionsstellen ueber `_dcp_token_sharded_pool_rows` gehen; die
Identitaet gilt nur bei dcp_size == 1. Ein Heben OHNE diese Uebersetzung
"would back C rows per rank instead of the rank's compact rows and set
pool.size to a number the owner rule never addresses" -- also STILL falsche
Groesse, kein Crash. Auf der Achse uneven-DCP-weighted, die laut
Nutzergesetz nie abgeschaltet wird.

ZWEITER BEFUND, der die Kostenschaetzung des Kommentars schwaecht: derselbe
Kommentar beruft sich auf `test_post_capture_dcp_gap_592`, der "beides
pinnt, die Sperre und die fehlende Uebersetzung, damit wer es angeht von
der Evidenz aus startet". DIESER TEST EXISTIERT NICHT. `git grep` ueber den
ganzen Baum: EIN Treffer, und das ist der Kommentar selbst. Das
Sicherheitsnetz, auf das "bounded work" sich stuetzt, ist nicht da.
(Klasse: PRAESENT-ABER-UNVERDRAHTET vs ABWESEND -- hier schlicht abwesend.)

DRITTER BEFUND, aus dem sgl-project#797-Kommentar (mixin:7236-7248): auf einem
Phase-Flip-Boot laeuft dieses Sizing VOR `build_phase_flip_tp_stack`, der
Token-Vektor ist noch nicht installiert -- und "the site that DOES have the
measured capacities (post_capture_resize_kv_pool) is HINT-ONLY". Die Stelle,
die der Entscheid autoritativ machen will, ist auf der Flip-Form
ausdruecklich nur Hinweisgeber. Sie autoritativ zu machen ist ein dritter
Arbeitsblock, nicht Teil der Zeilen-Uebersetzung.

SUMME: env-Default + sgl-project#592-Zeilen-Uebersetzung + Hint-only->autoritativ auf
der Flip-Form, ohne den Test, den der Kommentar als vorhanden annimmt, auf
einer Achse mit stiller Fehlgroesse als Fehlermodus. Das ist L, nicht S/M.

ZUR BEGRUENDUNG (a) DES ENTSCHEIDS, unveraendert richtig: ein portierter
slack_gb-Fraction-Term auf dem MiB-Pfad waere die Schaetz-Form, die der
Pfad abschaffen sollte. Der Einwand gegen Option 1 bleibt gueltig -- er
macht Option 2 nur nicht kleiner.

VORSCHLAG ZUR ENTSCHEIDUNG (nicht gebaut): den 810-MiB-Posten als
PROVISORISCHE Schranke auf dem Budget-Pfad buchen, ausdruecklich als
Uebergang kommentiert und mit dem gemessenen Wert statt einer Fraction --
das ist S und macht das lm_head-Artefakt sofort bootbar -- und die
sgl-project#592-Hebung als eigenen Posten fuehren, beginnend mit dem fehlenden Test.
Entscheid liegt beim Operator.

BELEG-STUFE: DESK-BEWIESEN (Praedikat-Zeilen, git grep) + BOOT-GEMESSEN
(Capture-Dauer je Pass, sgl-project#797-dcp_size, beides aus den Logs dieses Strangs).
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 1, 2026
…ot every rank's

Boot 1065umbau wedged at 06:26:07Z: pp_to_tp armed one pass after PP0
launched the final 118-token chunk mb. PP1 had RECEIVED and reconciled that
pass's frame (ROW-PROBE delivered=31, BATCH-NULLED occurrence=1 with all
void flags False) but the armed park in get_next_batch_to_run withheld the
build; PP2 stayed at delivered=30; PP0 blocked forever in the output recv
(mb slots [0, 2] in flight), so the 30 s abandon carrier never ran. The
deadman's BUSY-STARVED verdict took the boot down.

Pre-sgl-project#1065 the strict chunk clause kept the park open during a chunk ladder
and masked this: a follower's park is a rank-local verdict that stops the
very drain the park exists to wait for. Per the PP0-authority order,
followers execute what the row tells them and decide nothing: in the PP
layout (pp_to_tp armed) only PP0 parks -- it stops SOURCING new work while
in-flight mbs keep draining downstream. In the TP layout every rank is a
replica of one decision from replicated state, so the park stays
group-wide (tp_to_pp unchanged).

Sibling sweep: the sgl-project#797 voided-pass hold below the park is row-carried
(PP0-decided) and stays; the sgl-project#1066 PP0-wait is admission-side on PP0 only.
Desk-proven: py_compile; metal falsifier is the same drain scenario --
chunk ladder + arm mid-ladder must now flip and decode.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 1, 2026
…ot end a pass PP0 launched

`_pp_void_retracted_pass` let ONE downstream rank decide that the GROUP's
pass ran nowhere. Its own docstring named the return trip that made that
safe -- "the void output carries the observed local match home, and PP0's
guard learns it as a floor" -- and sgl-project#969 CUT V had already deleted that
emitter (`_PP_VOID_OUTPUT_KEY`: zero originating senders at ca0ee3a).
The verdict therefore travelled downstream only. PP0's `mbs[slot]` stayed
set while the last rank's did not, so no output was ever sent and PP0
blocked in `_do_recv` until the deadman -- the exact invariant `_do_recv`'s
own comment relies on ("sender and receiver ask one question of one batch").

Measured, twice, and the second is on the stall second itself:
  1068cap    07:34:02-09  sgl-project#797 void on rank 1 ONLY (no void/retract line on
                          PP0 or PP2); PP0 parked, PP2 spinning.
  1069cohort 08:00:54/55  sgl-project#791 unhonourable on PP1, told=12493 local=8397
                          then told=13399 local=12493 -- `local` exactly one
                          pass behind `told`. After 08:00:55 only ranks 1
                          and 2 emit at all (slot_occupant / output_fill /
                          width_agreement run to the takedown at 08:13:32,
                          4177 and 7973 lines; rank 0 emits nothing).

That last measurement also refutes the occupant-sleep node as the halting
member: ranks 1 and 2 are alive and turning; the `sgl-project#1000 SLOT-OCCUPANT
reasons={'no-statement'}` spin is an INERT probe whose carrier sgl-project#1015 EDIT-F
made permanently None. The only halting member is PP0's unbounded output
receive.

Repairing the return trip would repair a compensation layer for a rank-local
verdict, which is the arc the sgl-project#968 order forbids continuing; under
upstream-minimal the repair carries the burden of proof and the deletion does
not. So the verdict is deleted and the disagreement is DETECTED instead
(RAENGE-NIE-UNEINS: a detected divergence stops the group, never a
compensating wait):

  * `_pp_assert_told_honourable` replaces it -- an unhonourable told names
    rank, slot, rid, told and local and raises. No clamp: clamping to this
    rank's own local match is rank-local geometry, i.e. sgl-project#631.
  * The chain-receive throttle arm gets a horizon
    (SGLANG_PP_OCCUPANT_HORIZON_S, default 90 s, 0 disables) and a named
    stop. Taking the arm is legitimate and frequent; outliving it never is.
  * `PpChainReceiver.recv` is bounded the way its sibling `consume_up_to`
    already was (runaway guard + reported counter), and the launcher now
    sets SGLANG_PP_CHAIN_RECV_STALL_S=60 -- the sgl-project#824 mechanism has existed
    since 2026-08-24 and shipped disabled by default, which is why it never
    fired in either stall.

(A-i) is WITHDRAWN rather than built: its counter proof is blind at that site
(the rendezvous bumps `sent` only on recv entry) and the DEFER one-shot is
itself compensation for the rank-local verdict this commit deletes, so its
trigger is removed at the source. Its two red-first tests are replaced by
zombie tests for the deletion.

Desk evidence: hermetic import + AST (deleted verdict absent, watchman and
horizon present, recv bounded); test_968_deletion_falsifiers 24 passed,
1 failed -- test_C_the_void_relay_is_wired_or_deleted_but_never_half_built,
which stays red until the relay SYMBOLS are swept out too. The relay is
already unreachable at runtime (no rank originates a void any more, and
`_pp_absorb_void_output` has no caller in production), so that sweep is a
dead-code deletion scheduled beside this commit, not a runtime dependency of
it. Belegstufe: DESK-BEWIESEN. Boot pending.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 1, 2026
…sends, nobody relays, one rank absorbs

sgl-project#969 CUT V deleted the void-output EMITTER on the premise that the batch is
the verdict ("sender and receiver ask one question of one batch"). What it
left behind was the rest of the arc, and the arc has been dead ever since:

  _PP_VOID_OUTPUT_KEY: True   constructed at exactly ONE site -- inside
                              _pp_absorb_void_output, i.e. only when
                              re-forwarding a void already received. Zero
                              originating senders.
  _pp_void_forward_payload    1 writer, 0 readers. Computed and never sent.
  _pp_absorb_void_output      0 call sites (ast over python/sglang).
  pp_void_forward_payload     called only from the absorber.
  pp_void_relay_stop_rank     called only from the forward payload.
  pp_void_relay_launched_verdict  same.
  pp_first_retracting_rank    called only from the forward payload.

A closed orphan subtree, whose own docstring states the sgl-project#801 relay invariant
it can no longer keep: "a non-last rank that took exactly one message off
this wire for a ring generation must put exactly one back on it, void
included".

WHY DELETED AND NOT WIRED. Wiring it back means rebuilding the second
bookkeeping sgl-project#969 CUT V removed, and under the upstream-minimal law (user
order 2026-08-29) repair carries the burden of proof that upstream semantics
cannot carry the goal. It can: PP0 owns the verdict and the travelling batch
is the verdict. Half a relay is the worst of the three states -- PP0 parks
on an output the mid-rank void swallowed and its successor throttles on an
occupant nobody will speak for, which is the shape both measured stalls took
(1068cap 07:34:09 and 1069cohort 08:00:55, roles swapped, same halting node).

Tests of the deleted mechanism go with it rather than being patched green:
the five files whose SUBJECT is the void relay (sgl-project#797 retracted-pass void,
sgl-project#798 chunked-retracted void slot, both sgl-project#801 relay files, #791b output-ring
retraction), plus test_pp_continuation_cross_slot_rehome_968b (28 of its 30
tests drive the deleted absorber transitively) and test_pp_void_lock_ref_969
(its harness is the deleted method). Files that merely mentioned it keep
their pins; only the individual test that exercised the deleted site is
removed. Comments naming a deleted symbol are rewritten rather than left to
send the next reader after a ghost.

Desk-proven: py_compile + import smoke (scheduler and mixin), the sgl-project#968
deletion falsifiers 25/25 with the wired-or-deleted-but-never-half-built
contract now green by the deletion arm, ruff unchanged at 104 pre-existing
findings on both touched files (baseline taken from HEAD under the same
config).
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 1, 2026
…line positions

`local < told` was never a shortfall on this rig. It is the pipeline stagger,
and the comparison that read it as a defect had its two operands at different
pipeline positions and different times.

MEASURED, boot_855_1071cut, 12:09:58, one minute after READY, on the FIRST
chunked prefill of a FRESH boot against an EMPTY store (the launcher created
/tmp/hicache_855_1071cut that same minute):

    PP1  rid=0d43edbc  told=8192  local=4096

Nothing was in the store, so nothing could be reused and no rank could be
SHORT of anything -- a cache shortfall is constructively impossible in that
state. `told` is chunk 3's `extend_range.start`; `local` is chunk 1's
`extend_range.end`; the gap is exactly one --chunked-prefill-size. The check
was reading PP0's forward-most plan against a downstream rank's rear-most
completed extent.

This also retracts this strand's reading of the 1068cap and 1069cohort events
as "store skew" / "a moving told" (told=12493 local=8397, told=13399
local=12493): the same structural false positive later in the run, then
AMPLIFIED by the void it triggered -- `_pp_void_own_batch` restored
`chunked_req` to its pre-admission value, freezing `local` while PP0's `told`
ran on. Cause and amplifier were one mechanism.

DELETED (each verified as an absent BINDING by AST/hasattr, never by grepping
prose -- the first attempt at this check matched the function's own docstring
and had to be replaced):

  * both retraction branches of `reconcile_pp_admission_decision`. The sgl-project#944
    lookup-miss branch had to go WITH the sgl-project#791 one: its own comment rests its
    safety on the sgl-project#797 void being group-uniform, sgl-project#1071 deleted that void, and
    a retraction with no void left no longer stops the group -- it drops the
    rid on ONE rank and the pass runs with divergent membership, i.e. sgl-project#631.
    sgl-project#1071 had turned a covered branch into an uncovered one.
  * `_pp_assert_told_honourable`, sgl-project#1071's own watchpost. It fired once, named
    its own premise, and goes with the comparison it guarded; a watchpost over
    a deleted check is second bookkeeping of its own.
  * `pp_pass_should_void`, `pp_proxy_pass_retraction_reason`,
    `_pp_pass_retraction_reason` (no callers), the `entries_retracted_by_rank`
    import.
  * the #995c proxy-width probe (141 lines) and its #995e/#995f census (34).
    P2 measured why: it skipped 100% of its population on exactly the
    downstream ranks where sgl-project#791 saw anything -- 1069cohort PP1 evaluated=54
    agree=0 disagree=0 skipped=54, PP2 52/0/0/52; 1071cut 3/0/0/3 and 3/0/0/3;
    reason always `no_local_input_ids`, because at proxy-receive time this
    rank's `input_ids` do not exist yet. A counter printing `disagree=0` while
    never comparing reads as "checked, all good" on the next pass. sgl-project#998 is
    rank-local by construction and cannot see rank against rank.

NOTHING IS LOST, and this is the load-bearing half. The same length comparison
already exists where its operands are finally comparable and it CRASHES rather
than counting: model_runner.py:4233-4236, the one funnel every PP stage's
forward passes through, compares received hidden-states rows against
`forward_batch.input_ids.shape[0]` after `input_ids` is materialised and
raises with a sender stamp that separates a pairing error from a payload
error. sgl-project#791, #995c and that gate are the same check at three different
moments; this deletes the two that cannot fire and keeps the one that
executes. Ranks are held in agreement by sgl-project#631 ROW AUTHORITY (PP0 geometry
bulletin, 642b99c), not by a downstream veto.

`pp_rehome_displaced_chunked_req` deliberately STAYS (live caller at
scheduler_pp_mixin.py:9022).

Evidence: matched AST/binding check green (no retraction minted, 7 symbols
unbound, #995c counters gone from the recv path);
test_968_deletion_falsifiers 25/25 including the void-relay census. The 5
collection errors in test/registered/unit/managers are PRE-EXISTING, proven by
A/B in a fresh worktree at 3730c8e without these edits, not asserted.
Belegstufe: DESK-BEWIESEN. Acceptance boot pending.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 2, 2026
… group STOP, not a void

Root (HANDOVER_1153_0902, PROVEN, pre-existing at the pin 228a66d):
scheduler.py _pp_refuse_forwarded_schedule answered PPScheduleRefused
('sgl-project#791 FORWARDED SCHEDULE UNEXECUTABLE', PP1 reached 1 of the 2 rids PP0
named, batch_full_break) with a rank-local compensation:
_pp_admission_pass_voided = True + emptied decision dicts -> mixin
_pp_void_own_batch -> 'sgl-project#631 ROW-DELIVER BATCH NULLED slot=0
pass_voided=True' (boot_855_weg1b2 log 65000-65004) -> PP1 sent no proxy
while PP0's slot stayed set -> nothing carried the void upstream (sgl-project#797
return trip: zero call sites since CUT V; sgl-project#1072 deleted the void relay)
-> PP0's blocking _do_recv consumed PP2's NEXT output under this slot's
label (log 65119) -> one output ahead for the rest of the boot -> the
pp_to_tp arm at 21:42:05 turned the debt into an unproducible output ->
sgl-project#980 ObjectRecvStalled 60 s / sgl-project#1071 PpChainRecvStalled 90 s. Same form
sgl-project#1071 (169f53c) deleted for _pp_void_retracted_pass; this was the
second writer of the same flag.

F1 ROOT: the compensation is deleted, the detector is kept, the refusal
is a group STOP. scheduler.py get_new_batch_prefill's except now does
`raise self._pp_forwarded_schedule_stop(refusal) from refusal`;
_pp_refuse_forwarded_schedule (flag write, dict emptying, sgl-project#971 re-home)
is removed; the new _pp_forwarded_schedule_stop logs the kept
'sgl-project#791 PP-ADMISSION forwarded schedule REFUSED on rank N' line and returns
a RuntimeError formatted by pp_admission_congruence
.forwarded_schedule_stop_message:
  'sgl-project#791 FORWARDED SCHEDULE UNEXECUTABLE STOP rank={rank} slot={slot}
   told=[{told}] reached=[{reached}] census={census} local={local}
   limiter={limiter} running_bs={running_bs} parked={parked}
   r2t_avail={r2t_avail} headroom={headroom} group_limit={group_limit}
   batch_full_setter={batch_full_setter}
   batch_full_at_loop_entry={batch_full_at_loop_entry}: {refusal}'
Every probe is guarded (n/a on an unreadable value); the reached rids
are recorded after the admission loop before any of the three raises;
the batch_is_full setter site is recorded at each writer of the pass
(no_allocatable_reqs_gate, count_arm, disagg_prefill_r2t_avail,
add_one_req_NO_TOKEN). Group stop mechanism (existing, no new
collective): the RuntimeError leaves run_event_loop; run_scheduler_process
(scheduler.py 'except Exception') logs 'Scheduler hit an exception' and
parent_process.send_signal(SIGQUIT) (+ killpg/kill_process_tree under
SGLANG_KILLPG_ON_SCHEDULER_EXCEPTION); peers end in the bounded sgl-project#980 /
sgl-project#1071 receives or the barlink dead-peer probe.

F2 TRIGGER CLASS (PP0 order): on a forwarded schedule the follower's
rank-local seat-count veto is not a verdict. pp_admission_congruence
.rank_local_count_veto_applies(scheduled_extents) is True on PP0 and on
every non-PP boot (unchanged expression) and False on a rank > 0
executing PP0's decision; the sgl-project#823 count arm, its batch_full_break, and
the three pre-loop count gates of the same arithmetic
(batch_full_or_empty_queue's batch_is_full half, min_free_slots_delay,
no_allocatable_reqs) are gated on it. The physical allocator still
refuses (NO_TOKEN -> membership refusal -> STOP naming the numbers).

F3 SIBLINGS: mixin _pp_void_pass_without_upstream_launch's writer of the
flag is provably unreachable (pp_upstream_void_pending returns False on
every path; its final statement is `return False`) -- left with the
proof in a comment. The row-authority _row_skip_plan exit now nulls a
slot through _pp_null_frameless_slot, which applies the same named sgl-project#1020
'VOID REFUSED ON A LAUNCHED SLOT' guard as the void path; the guard is
factored into _pp_slot_holds_unconsumed_launch(mb_id, site) and used by
both sites.

Matched check (error class: a follower still ends a PP0-launched pass
silently): grep -rn '_pp_admission_pass_voided = True' python/sglang/srt
-> only scheduler_pp_mixin.py (the unreachable sgl-project#801 writer); chain
scheduler.py raise PPScheduleRefused (:11633/:11646, :11606) ->
except PPScheduleRefused (:9441) -> raise self._pp_forwarded_schedule_stop
(:9472) -> RuntimeError (:9623); the flag's only readers are
_event_loop_pp_body (:4519 -> _pp_void_own_batch) and the scheduler.py
void guard (:8955), neither reachable from the refusal.

Tests (hermetic, CUDA_VISIBLE_DEVICES=""): new
test_pp_forwarded_refusal_stop_1153.py (T1 STOP form + no void, T2 the
count-veto helper + source pin, T3 the sgl-project#1020 guard on the frameless
null, + the mixin:9389 unreachability proof): 17 passed after; on the
parent ca4c6b7 (git worktree) it is a collection error (new names).
test_pp_refused_pass_keeps_continuation_971.py: 12 tests that pinned the
old compensation INVERTED with the withdrawal named in each docstring
(24 passed after; the inverted 12 are red on the parent: 16 failed /
8 passed / 1 error across both files). Mutants: M1 flag write restored
in the STOP builder -> 2 T1 red; M2a helper returns True -> 1 T2 red;
M2b count-arm gate dropped in the loop -> 1 T2 red; M3 guard dropped from
_pp_null_frameless_slot -> 1 T3 red. Bounded suite (7 unit/managers
files + test/registered/scheduler): before 103 failed / 527 passed /
1 skipped / 27 errors; after 103 failed / 544 passed / 1 skipped /
27 errors -- per-file identical except the new file (+17). ruff check
clean on all touched files; ruff format --diff hunk count unchanged vs
parent (15/14/3, pre-existing) and the two test files formatted.

Evidence tier: DESK-PROVEN. Boot-3 acceptance: grep -F
'sgl-project#791 FORWARDED SCHEDULE UNEXECUTABLE STOP' in the same second as
'sgl-project#791 PP-ADMISSION forwarded schedule REFUSED on rank', and zero
'ROW-DELIVER BATCH NULLED ... pass_voided=True' lines.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants