Repository navigation
Update install commands - #583
Merged
Merged
Conversation
merrymercy
added a commit
that referenced
this pull request
Jul 3, 2024
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
…d, plus the power-cap caveat User directive: the live hypothesis is that a captured prefill only pays IN COMBINATION WITH barlink. The mechanism is falsifiable and specific -- NCCL's collectives are host-driven, so the launch train a graph would tighten is not the part that costs, while barlink's device-side collectives can sit inside the captured graph and put the 68-75% collective share of the prefill window (sgl-project#252) within reach. If that holds, the flat NCCL result is a property of the transport, not of prefill graphs. window3_boot_floor.sh now stages STAGE=nccl|barlink|all over prefill eager prefill graphs NCCL E1, E2 G barlink BE1, BE2 BG Each transport carries its OWN eager-vs-eager floor, a delta counts only against the floor of its own row, and the reported swing counts only if it clears both. Points are the long-prompt 1900 and the bs>1 concurrency mix (256 tokens, 4 in flight), the latter scored on aggregate_tok_s because per-request rates understate a concurrent regime. The barlink row gets its own content gate too -- a transport change is exactly the kind of thing that moves reduction order. Both preconditions are enforced by the script rather than assumed: the barlink stage refuses to run unless b001d10 (the sgl-project#583 tripped-spin-kernel / CUDA context fix, merged here) is an ancestor of HEAD, and unless the operator passes BARLINK_VERDICT=confirmed from the live-load repro window. Absent arms report 'hypothesis UNTESTED at this point' instead of being guessed. Reporting was executed against the window-2 artifacts before commit, exercising the missing-floor, missing-transport and missing-point skip paths. Power caps were reduced on this rig (3080 320->200 W, 5090 525->400 W; verified power.limit 200/400/200 against default 320/575/320). Same-boot A/B arms share the caps and are unaffected, but comparisons against pre-change archive numbers -- the sgl-project#320 Messbuendel tables in particular -- are confounded. The script now prints the measured caps in its report header and NOTE_515 carries the caveat.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
…rash configuration The 2026-08-05 GPU window ran two arms. Neither reached a barlink fault, and both failed for reasons that were in the harness, not the transport. What the window did establish is that the harness now reproduces the crash boot's configuration exactly, and what it costs to get there. Four config gates had to be cleared before the server would start at all: * --speculative-algorithm NEXTN is rejected under --dcp-size. The alias is resolved by the arg hook only AFTER _handle_dcp_validation has already run SpeculativeAlgorithm.from_string on the raw string (server_args.py:7817), so the enum never sees it. EAGLE is what the crash log's server_args recorded anyway. * spec + DCP is admitted only on the uneven-hybrid weighted path, which needs SGLANG_UNEVEN_DCP=1 and SGLANG_UNEVEN_DCP_WEIGHTED=1 (server_args.py:7834). * rank_tp_ratio must already be non-uniform at validation time, but passing it explicitly collides with "--rank-auto-reserve-mib only applies with --rank-tp-ratio auto" (server_args.py:9656). auto-performance resolves the conflict and DERIVES [27107, 16680, 16680] -- byte-identical to the crash boot's recorded rank_gpu_memory_mib and rank_tp_ratio. * --max-mamba-cache-size 96, production's value. With those, the boot is faithful: barlink reports ACHIEVED=device for world:0, tp:0 and dcp:0, the same three groups as the crash log. The load generator was the second problem and is the reason no arm reached the barlink question. An unbounded growing conversation per session drove mamba usage to 1.00 and killed both arms on mamba allocation asserts -- arm 1 on "Not enough space for mamba ping pong idx" (memory_pool.py:1494) at 8 minutes, arm 2, with the larger pool, on "Not enough space for mamba cache". The crash boot ran at mamba usage 0.25-0.28 with 3-4 running requests, so the harness was measuring a regime the crash never occupied. Each session now recycles its prefix every six turns to bound the number of distinct cached prefixes, and the comment states the mamba-usage band the arm has to stay inside to be valid. Neither mamba assert is a barlink fault. Both are worth reporting separately: they kill the scheduler with SIGQUIT, and the merged "Mamba slot starvation must not kill the scheduler" fix does not cover the _alloc_ping_pong_buffer site. Rig fact worth keeping, because it silently invalidates any hardcoded mapping: torch device order is NOT NVML order here. torch 0 is the RTX 5090 while nvml 0 is an RTX 3080. --rank-gpu-id is torch order, which is why 27107 MiB on rank 0 is valid. The harness prints the NVML inventory for identification but derives nothing from its indices. Window hygiene: serving was stopped by pgid only, VRAM verified fully reclaimed (20052/32069/20052 MiB), production rebooted verbatim from /root/bin/start-serving-30030.sh, health 200, corridor restored to its pre-window values (1278/3317/2148 MiB). The router on 30099 was never touched. No barlink fault was observed in either arm: 0 occurrences of "unspecified launch failure" and 0 of DeviceCollectiveAborted across 202 prefill and 30 decode batches. That is not evidence the sgl-project#583 fix is wrong, and not evidence the trigger is gone -- the arms simply died before reaching the regime.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
…ach arm prove its own validity The 2026-08-05 window produced no barlink verdict because the load ran in a regime the crash never occupied. This rebuilds the load profile around the variable that actually saturated and adds the instrumentation that would have caught the mistake in the first minute instead of after two arms. What consumes mamba state is the number of DISTINCT cached prefixes -- the mamba radix cache holds one state per cached prefix under mamba_radix_cache_strategy='extra_buffer'. The old generator gave each session its own forever-growing conversation, so distinct prefixes grew without bound. Sessions now cycle a FIXED pool of six long prefixes, which pins the mamba footprint flat over the soak while each request still carries a large cached-token count plus a fresh 600-2000 token chunk -- the crash boot's shape. The band was then measured rather than assumed, from the two logs: crash boot 5 mamba median 0.25 max 0.80 running median 3.0 max 4 rejected arm 2 mamba median 0.18 max 1.00 running median 3.0 max 4 Two corrections follow from that, and both matter: * The crash boot PEAKED AT 0.80. A safety valve at the band ceiling would have throttled exactly the excursions the crash exhibited, and would have flagged the crash boot itself as out of regime. The valve therefore sits at near-saturation (0.88), not at the band edge. * The rejected arm's MEDIAN was lower than the crash boot's. Median is not what distinguished them -- saturation is. Validity is judged on the median band (0.20-0.35), failure on saturation (>= 0.95). Concurrency was never the problem: both runs sat at running-req median 3.0, max 4, which already matched the crash. Each arm now reports its own validity. A sampler reads the server's own log lines and the run prints MAMBA BAND / REGIME / RUNNING-REQ, so a barlink verdict from an out-of-regime arm is visibly not trustworthy rather than silently wrong. Can-fail proof, replaying the verdict logic over the two real logs: the crash boot scores REGIME OK (median 0.25, peak 0.80) and the rejected arm scores REGIME FAIL (saturated 1.00). Adds the `soak` arm (20 min) for the dual-acceptance window: sgl-project#583's abort path under the crash regime, and sgl-project#581's mamba pool floor live proof in the same run. The verdict block now reports both. sgl-project#581's half is negative evidence -- neither "Not enough space for mamba ping pong idx" nor "Not enough space for mamba cache" may reappear -- plus the regime report showing the run was actually loaded while staying clear. Pin-budget lines are counted and their ABSENCE is reported explicitly, so a soak that logged nothing about pins is recorded as leaving pin behaviour unproven rather than passing by silence. Not yet merged: fix/mamba-pool-floor-581 is still uncommitted in /spinning/wt-mamba-floor (8 modified files, 2 new, 0 commits), so there was nothing to take. The soak needs it -- without the floor the arms die on the asserts again. Validation, no GPU: bash -n, embedded-Python AST parse, codespell clean, and the regime instrument's can-fail proof above.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
…k tree The soak needs the pool floor: both 2026-08-05 arms died on the mamba allocation asserts this branch removes, before barlink was ever stressed. Merged here so one window can carry both acceptances.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
…ry markers verbatim and guard them with a test The soak reads both acceptances out of the server log by matching fixed strings, so the strings are now pinned to the sources that must contain them rather than matched by a loose pattern. Markers taken from ee8d1b5: hi_mamba_radix_cache.py "mamba write-through pin budget reached" memory_pool.py "deferring this batch" (+ mamba_evictable= / mamba_protected= starvation fields) mamba_component.py "skipping this cache insert" mamba_pool_floor.py "pinned checkpoint" (the floor derivation line) Writing the pins exposed one of my own: "deferring this batch of" is NOT contiguous in the source -- memory_pool.py wraps the format string between "batch " and "of %d request(s)". It matches the formatted log line but cannot be verified against the source, which is precisely how a pin rots unnoticed. The harness and the test now both use the shorter contiguous fragment, and the reason is written down at both sites. test_soak_harness_pins_583.py guards this in three directions: * every pinned literal must exist verbatim in its source file; * every `grep -F` in the harness must be a REGISTERED pin -- this is the direction that actually rots, where someone adds a marker, never registers it, and it quietly stops matching; * the two death asserts must still EXIST in memory_pool.py. The soak proves their absence from a run's log, and you cannot prove the absence of a string that no longer exists anywhere -- deleting them would make the acceptance pass vacuously. That third check also settled how to read the sgl-project#581 verdict. Both asserts are still live `assert` statements (memory_pool.py:1476-1482); sgl-project#581 makes them UNREACHABLE via the floor, the admission gate and graceful back-off elsewhere, rather than converting them into warnings. So their appearance in a log remains a hard failure, and the harness is right to count them that way -- while "deferring this batch" / "skipping this cache insert" are the GRACEFUL paths and are reported as back-off, not death. Pin-budget acceptance is reported honestly in both directions: observing the budget engage is the proof, and silence is printed as "pin behaviour remains UNPROVEN by this arm" rather than folded into a pass. Soak config verified against the new parse-time floor refusal: --max-mamba-cache-size 96 with --max-running-requests 4 gives a hard floor of 20 (4 x (1 active + 2 ping-pong + 1 donation + 1 pinned checkpoint)), so the soak boots with 76 slots of headroom and does not trip the refusal. Tests (no GPU): test_mamba_pool_floor.py + test_barlink_device_abort_583.py 38 passed on the merged tree; test_soak_harness_pins_583.py 4 passed / 9 subtests. ruff and codespell clean. A full test/registered/unit/mem_cache sweep is running; test_hicache_nixl_storage.py fails to collect there for a missing optional 'nixl' module, which is environmental and untouched by sgl-project#581.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
…ce instead of guessing it
Standing order: the harness finds its own values. An env knob is not an
improvement over a hand edit -- it is the same guess with better ergonomics,
and hand-picked values produced both failed arms on 2026-08-05.
scripts/repro/derive_soak_target.py parses a crash-reference log for the
ABSOLUTE regime and solves the load parameters from it. The harness calls it
by default; MAMBA_CACHE / PREFIX_POOL / SESSIONS survive only as documented
overrides.
THE RULE IT MECHANIZES. A reference's `mamba usage` means nothing except
against that reference's own `max_mamba_cache_size`. Soak arm 1 ran fraction
0.17 against the crash's 0.25 and was rejected as "too light" while its
ABSOLUTE occupancy was identical -- ~16 slots either way, 96 vs 64 denominator.
Following the fraction would have raised real load ABOVE the reference while
the displayed number moved toward it. The deriver converts to absolute slots at
parse time using the reference's own denominator and targets absolutes only. A
log without a denominator is a hard error, never a guess.
WHY NOT READ `mamba num`, WHICH IS ALREADY ABSOLUTE. It is printed on DECODE
lines only. In boot5 that is 14 samples, all 16, stdev 1.0 -- a flat regime --
while the actual peak of 51 slots occurs on PREFILL lines carrying no
`mamba num` at all. Reading it alone understates the peak by more than 3x,
which is precisely why soak arm 1 came out min=median=max. The series is
reconstructed from usage x denominator for full coverage (222 samples) and
cross-checked against `mamba num` where both appear on one line; the observed
disagreement is 0.16 slots.
SOLVED ON THE PEAK, NOT THE MEDIAN. The prefix pool is a ceiling on distinct
cached prefixes, so it sets the PEAK; the median falls out below it as prefixes
are evicted between bursts. Solving on the median caps the pool too low to ever
reach the reference's excursions.
boot5 (barlink crash): median 16, peak 51, spikiness 3.19x, running median 3
-> MAMBA_CACHE=64 SESSIONS=3 PREFIX_POOL=31 (51 - 4x5 structural)
REFERENCE SELECTION IS ITSELF GUARDED. The default is boot5, the BARLINK crash
-- not boot6, which is the mamba ping-pong crash and SATURATED its pool
(64/64). Targeting boot6 would build a harness that deliberately saturates,
which this harness's own regime gate scores REGIME FAIL. The deriver prints a
warning whenever a reference saturates, so "aim at the most recent crash log"
fails loudly instead of quietly producing an unusable arm.
Also fixes a truncation the new block exposed: the NVML section used
`tee "$LOG"` without -a and would have erased the derivation record from the
log -- the first thing a reader of any verdict needs.
Hand-derived vs auto-derived, for the record: soak arm 2 (running now) was
aimed by hand at MAMBA_CACHE=64 / PREFIX_POOL=16 / SESSIONS=4. The deriver
agrees on the denominator (64, the correction that matters) but calls for
PREFIX_POOL=31 and SESSIONS=3, so arm 2 is expected to sit at the reference
median with a lower peak. Any rerun uses the auto-derived targeting.
Tests (no GPU): test_derive_soak_target_583.py 12 passed --
denominator-from-reference, peak-not-decode-only, cross-check tightness,
peak-based pool solve, the boot5 regression pin (64/31/3), floor clearance,
saturation detection on boot6, and the hard error on a missing denominator.
Plus test_soak_harness_pins_583.py 4 passed / 9 subtests. ruff and codespell
clean; bash -n and an embedded-Python AST parse on the harness.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
…ation test prose No behaviour change; keeps codespell clean so real hits stay visible. test_derive_soak_target_583.py 12 passed.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
…sions is half a regime Gate gap found by soak arm 2 in the 2026-08-05 window. That arm matched the reference MEDIAN exactly (0.25) and was scored REGIME OK -- while its peak was also 0.25 against the reference's 0.80, i.e. spikiness 1.00x against the reference's 3.19x. The gate checked median and saturation and never checked peak fidelity, so an arm that reproduced half the regime was reported as having reproduced it. That matters because the excursion half is the part that plausibly stresses the transport: a null barlink verdict from a dead-flat arm would have been reported as if the crash conditions had been met. New REGIME PARTIAL verdict fires when the median is in band but the peak falls below 60% of the reference peak. It names the observed and reference spikiness and points at derive_soak_target.py, which computes the prefix pool from the reference PEAK precisely so this does not recur. Replay over all four recorded runs, which is the can-fail proof: crash boot5 REGIME OK med 0.25 peak 0.80 baseline arm2 REGIME FAIL saturated 1.00 soak arm1 WARNING light med 0.17 soak arm2 REGIME PARTIAL med 0.25 ok, peak 0.25, 1.00x vs 3.20x The reference fractions are supplied by the solver rather than hardcoded, and are only ever compared against the arm's own denominator -- the same rule the solver enforces. Tests: test_soak_harness_pins_583.py 6 passed / 9 subtests, including that the PARTIAL branch exists and is evaluated BEFORE the OK branch (ordering is the whole defect: OK must be the last resort, not the first match). Full sgl-project#583 suite 33 passed / 12 subtests. ruff and codespell clean, bash -n on the harness.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
…IRMED, with a regime split
TRANSPORT=barlink (production's transport), interleaved E,G,E,G,E,G, ms per
fixed unit of work, own barlink eager floor. Artifacts in
/spinning/gpu-battery-results/2026-08-05_prefill_graphs_w4{,det}/.
point eager graphs paired delta floor verdict
1900 single-stream 1196.3 ms 1320.6 ms +10.25% slower 0.12% REPORTABLE
256 x4 concurrent 230.1 ms 219.0 ms -4.62% faster 1.99% REPORTABLE
All three pairs agree in sign at both points and both deltas clear their own
A-vs-A floor by a wide margin (85x and 2.3x).
The user's hypothesis holds, and only in the regime it predicted: a captured
prefill pays where the work is launch-train bound (many small prefills, several
in flight) and loses where it is GEMM bound. With barlink's device-side
collectives inside the captured graph the concurrent short-prompt point turns a
real -4.6%, while the single long prefill has no launch overhead to recover and
pays the padding and capture cost instead. The +10.25% at 1900 is larger than
bucket padding alone explains (~3.7% for a 1974-token prefill in the 2048
bucket), so something else in the captured long-prefill path costs on top; not
localised here.
Consequence for the recommendation, now rescoped: the answer is not on/off but
workload-dependent, and a blanket flag is the wrong shape. can_run_graph is
already a per-batch decision (prefill_cuda_graph_runner.py:615), which is where
a regime gate would live. Enabling globally would trade 10% on long prefills
for 4.6% on short concurrent ones, and production's traffic mix decides whether
that nets out -- unmeasured.
DETERMINISM ARMS booted first try at the derived RESERVE 7164,5864,5864,
validating the §5a ledger derivation end to end -- right vector, no padding, no
retry. Same directional split (+7.44% / -6.31%) but REPS=1, so no floor and
therefore NO VERDICT on those two numbers, as the reporter states. Determinism
did NOT close the content divergence: 1/8 with it, 1/8 without.
CONTENT under barlink is 1/8 text (char 124) against 4/8 under NCCL, but the
gate ran on pair 1 only, so there is NO barlink eager-vs-eager content floor.
The 1/8 is not yet distinguishable from barlink boot noise, and is recorded
that way rather than as an improvement.
SOAK: 8 barlink arms booted and completed with no fault -- a clean data point
for sgl-project#583.
Serving restored pgid 231310, health 200 after 40s, corridor 1708/4131/2578 MiB,
smoke content-sane.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
…t skipped Crashes 9, 10 and 11 (2026-08-05) all ended in the same unhelpful shape: ranks 0 and 1 standing in one collective, rank 2 standing in a different one, and nothing anywhere saying which collective rank 2 had SKIPPED. The tracebacks report where each rank was STANDING, which is downstream of the defect. The per-rank counts already existed -- the prefill line prints "tp.all_reduce 243.4/129x" -- they were simply never compared ACROSS ranks. This adds that comparison as a standing instrument. It is measurement, not mitigation: it changes nothing about what the ranks do and cannot repair a divergence. It reports, so the next crash arrives with its cause attached. Counting: one dict increment per collective, at the four GroupCoordinator dispatch sites, keyed by the family names the collective clock already derives (tp.all_reduce, dcp.all_gather, ...). Placed AFTER the clock's re-entry guard so each collective is counted exactly once. Guarded by a module-level bool resolved at import, so the hot path costs one bool read plus one dict increment. Two reporting paths, for two different failure states: 1. Scheduler._census_tick -- the detector. Runs on tp_cpu_group (gloo), NEVER the device/BAR1 path, so a wedged device group cannot silence the instrument meant to explain it. Rides the same unconditional once-per-iteration point as the sgl-project#603 reduce, so its round counter is replicated and the cadence gate opens on the same round for every rank. Default every 50 iterations (~2 s in production), an order of magnitude inside the ~30 s spin deadline: the divergence is named while the ranks are still healthy, long before the abort. 2. format_local_census, called from barlink_bar1.check_aborted before it raises -- the wedge-proof half. Takes NO collective on purpose: by abort time a peer is very likely dead (in all three crashes the third rank aborted ~30 s after the first two), so a cross-rank exchange would hang exactly when the evidence is wanted. Each rank logs its own counts; the three lines are diffed afterwards. Warn-never-raise throughout: a failed comparison is logged and dropped, and the abort-path dump can never mask the real error. ARMED BY DEFAULT, like the sgl-project#605 flight-recorder marks -- an instrument that must be switched on before a crash explains no crashes. SGLANG_COLLECTIVE_CENSUS=0 disables counting; SGLANG_COLLECTIVE_CENSUS_INTERVAL tunes or (<=0) disables only the periodic comparison, leaving the abort dump armed. Tests: test/registered/unit/distributed/test_collective_census_583.py Hermetic (no CUDA, no process group, no model). The injected divergence is crashes 9/11 in numbers: [129, 129, 128] on tp.all_reduce. * 14 passed * can-fail proof: breaking only the count comparison in _diff turns 4 of them red * pins the missing-key case (a rank that skipped a family's ONLY collective has no key at all -> must read as zero, not drop the family) * pins armed-by-default, the kill switch, the replicated cadence gate, and warn-never-raise on a wedged/absent group * regression: census + retract-583 + 603 + retract-fcfs = 35 passed * ruff: 0 new findings (parallel_state 0/0, barlink_bar1 2/2, scheduler 89/89, schedule_batch 40/40); new files ruff-clean and formatted; codespell clean Crash 11 (21:26, boot 11, tree WITHOUT the retraction fix) is byte-for-byte crash 9's shape and NOT crash 10's: all three ranks are inside run_batch (scheduler.py:2032), ranks 0/1 at communicator.py:1179 and rank 2 at linear.py:2340, with check windows 1 and 4. So the `if batch:` fork is not involved there, and crashes 9/11 are a site INSIDE the forward, distinct from the retraction site closed for crash 10. The census is what will name it.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
…tbeat) Boot 13 ran the census and its log contained NO census line at all. That is not reassuring, it is unfalsifiable: an instrument that is silent when healthy AND silent when unwired cannot be told apart from a dead one, and "armed, counts uniform" reads exactly like "import path never reached on this flagset" (the sgl-project#380 test-honesty trap). Two additions, both cheap: * ONE INFO line per scheduler rank, the first time the tick runs: "collective census armed (rank N): interval=50 iterations, heartbeat=..., families tracked=K (...)". Emitted from the TICK rather than at import on purpose -- an import-time line proves only that the module loaded, while this one proves the wired production path is actually reached. Guarded by a flag so it appears once, not once per iteration. * A coarse local heartbeat (default every 10000 iterations, INFO, carrying this rank's snapshot) so a long-running boot keeps evidence the detector is still ticking. It rides the round counter that is incremented anyway, so the hot path is one extra modulo. SGLANG_COLLECTIVE_CENSUS_HEARTBEAT tunes it; <=0 disables the heartbeat only. Tests: 4 added to test_collective_census_583.py (17 total). * arming line emitted EXACTLY once, naming rank, interval, family count * heartbeat cadence rides the replicated counter * default coarse, overridable, junk-tolerant * can-fail proof: removing the once-only guard turns the arming test red
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
…etero gate) `apply_flashinfer_allreduce_fusion` gated on `is_sm90_supported() or is_sm100_supported()` -- a query against THIS RANK's device. That value reaches `should_fuse_mlp_allreduce_with_next_layer`, and a True there makes the layer SKIP `postprocess_layer`, i.e. it REMOVES a collective from that layer. So it is a rank-local predicate deciding whether a group collective is entered -- the sgl-project#583 family. The ranks that fuse issue one fewer all-reduce per layer than the ranks that do not. And because the mispaired payloads are frequently the same size, the ranks complete each other's collectives and compute garbage SILENTLY before anything hangs; the deadline abort is the lucky case, not the normal one. Dormant on a homogeneous cluster, and dormant on this rig (sm86 + sm120: both answer False, which is why it never fired here). LIVE on any heterogeneous TP group straddling the sm90 boundary -- an sm89 card beside an sm90 one is enough. Closing it now, while the context is hot, rather than waiting for the rig that has those cards. Fix, same shape and same reasoning as `decide_spec_kernel_backend` in speculative/eagle_utils.py: every rank contributes its own capability, the MINIMUM rules on the gloo CPU group, and if one rank lacks the arch then NO rank fuses. Never a mixture. Decided once in `init_torch_distributed`, where every rank arrives exactly once, the TP group exists and no CUDA graph is capturing. A conflicting second decision is refused rather than silently winning, because two disagreeing decisions in one process mean the collective ran twice with different inputs. Undecided falls back to the LOCAL value, which is correct where it is reachable (single rank, unit tests) and is not a silent per-rank restoration on a real TP boot. Behaviour on this rig is unchanged: both cards answer False, the group MIN is False, fusion stays off exactly as before. Tests: test/registered/unit/distributed/test_ar_fusion_arch_583.py, 8 tests. * falsifier: a mixed group [capable, incapable] must fuse on NEITHER rank, with the pre-fix per-rank predicate pinned as genuinely divergent first * the gate can still answer YES (a uniformly capable group fuses), so the falsifier cannot pass against something that always disables fusion * conflicting second decision refused; undecided falls back to local, not to a silent False; an unavailable group does not block boot * can-fail proof: reverting the predicate to the local value turns the mixed-group falsifier red * regression across all sgl-project#583 work: 46 passed * ruff: 0 new findings (communicator 0/0, model_runner 5/5, scheduler 89/89); new test ruff-clean and formatted; codespell clean
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
…by line number REGRESSION I CAUSED, and the fragility that let one line do so much damage. _bar1_marker_source.py parses the real emitter sources with `ast` and pulls the format strings out of them -- a good design, built precisely so BAR1 log fixtures cannot drift into hand-typed copies (sgl-project#315, the sgl-project#380 family). But it located each call by ABSOLUTE LINE NUMBER (LINE_BAR1_SETUP = 2236, ...). That coupled the module to the LAYOUT of files it does not own. Adding the collective-census counter to parallel_state.py shifted line 787, the AST lookup found no call there, and the assertion fired at IMPORT -- taking the whole BAR1 marker suite down at COLLECTION (test_gpu_battery_checks_bar1.py + test_bar1_marker_coupling.py, 136 tests). Note the actual trigger was the census BUMPS in parallel_state.py, not the abort-handler edit in barlink_bar1.py: the failing assertion names parallel_state.py:787. A line number is the same fragility class as a hand-typed fixture string: a second, unmaintained copy of a fact the source already states. So the anchors are now the MARKER TEXT itself (MARK_* instead of LINE_*), resolved by `_call_with_marker`: the unique call whose first argument contains the fragment. For the exception class the anchor is its DOCSTRING, not its name -- anchoring on the name would make `bar1_unavailable_class_name` a tautology, when its whole job is to report a rename to the consumers. This is strictly better coupling, not weaker: it survives every edit that does not touch the marker, and still fails loudly on the edit that does -- which is the drift the module exists to catch. Ambiguity (a marker matching two calls) is a loud error naming both, never a silent first-match. Tests: * test_gpu_battery_checks_bar1.py + test_bar1_marker_coupling.py: 136 passed (were: collection error) * robustness proof: inserting 40 unrelated lines above the old pin in parallel_state.py -- the exact edit class that broke it -- still 136 passed * can-fail proof: renaming the emitted marker ("barlink enabled for group " -> "barlink ACTIVATED for group ") fails loudly with a message naming the file, the marker and both things to update * WHOLE test/registered/unit/distributed directory: 2370 collected, 0 collection errors, 2358 passed / 12 skipped / 726 subtests passed * the remaining 3 subtest failures (test_dcp_context_ceiling.py, '_PoolRowStub' object has no attribute 'server_args') are PRE-EXISTING, not from this branch: verified identical at 38e7c69 in a throwaway worktree (3 failed, 17 passed there too). The call site is flashinfer_backend.py:691, which this branch does not touch. Process note: this slipped through because I ran only my own test files. The directory-wide run is the check that would have caught it, and it is what I ran here.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
…rank)
MY BUG, and the worst possible one to have: the instrument built so that
silence could be trusted was itself silently dead.
`Scheduler._census_tick` read `self.tp_rank`. That attribute does not
exist -- the Scheduler keeps its parallel identity on the ParallelState
wrapper, `self.ps.tp_rank`. So every tick raised AttributeError into the
tick's own warn-never-raise guard: nothing was counted, nothing compared,
and the abort-time dump would have been empty. Boot 13 logged the skip
2661 times; boot 14 had 693 within minutes.
The unit tests missed it because they drove `CollectiveCensus` directly and
STUBBED the scheduler surface. A stub that grants the attribute proves only
that the stub has it. Desk-written-never-executed, exactly.
Three fixes:
1. `self.ps.tp_rank`, the real path. The same mistake had already been made
a second time in the (unpushed) fallback hardening, which used
`self.tp_size`; that is corrected here too, before it could ship.
2. The skip warning is emitted ONCE, not per tick. A tick that cannot run
usually cannot run ever, so per-tick warning buries the first line under
thousands of copies.
3. The arming line now prints only AFTER a tick has completed without
raising. Previously it was announced first, so a dead census could still
have certified itself as armed -- the line exists to make the
instrument's liveness checkable, and announcing it before the work would
certify a corpse. This is the same lesson the arming line was added for,
applied one level deeper.
Tests: test/registered/unit/distributed/test_census_attribute_surface_583.py
Two independent styles, because the stub is what failed here:
* STRUCTURAL -- parses the shipped scheduler.py, asserts the identity
readers go through `self.ps` and that `self.tp_rank`/`self.tp_size`
appear nowhere in them, and that every `self.ps.X` they use is a real
ParallelState field. A stub cannot satisfy this; it reads the source.
* EXECUTION -- binds the REAL `_census_tick` to an object carrying only
the real attribute surface (ParallelState built from its OWN
annotations, so a new field cannot quietly turn this back into a stub)
and asserts the tick does not take its skip path.
* plus: the skip warning fires once in 500 calls; a broken tick never
announces itself as armed.
* can-fail proof: restoring `self.tp_rank` fails 3 of them
(structural, positive-read, and arming-after-failure).
* 9 passed; census suites 43 passed together.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
…o local sgl-project#603 and the sgl-project#583 retraction fix both decide from a once-per-iteration MIN-reduced pool headroom instead of the rank-local `available_size()`. Both kept a fallback for "reduced value not available": * `Scheduler.uniform_min_avail` -- getattr(self, "_uniform_min_avail", None), returning the local pool when the reduce had not run * `ScheduleBatch.decode_mem_avail` -- returning the local pool when `uniform_avail_floor` was unset Justified as "correct for a single rank and for tests", which is true and beside the point: on a MULTI-RANK boot they hand back exactly the rank-local predicate both fixes removed. Any path reaching a decode-mem decision without setting the floor silently restores the desync, with the protection still plainly visible in the source. That is the getattr-default trap (sgl-project#606) -- a guard that reads as armed and is absent at runtime, the same shape as a fixture string nobody ever printed (sgl-project#380). Fix: single rank keeps the local value (nothing to diverge from); a group raises with a message naming the ordering that was violated. The world-size probe (`_group_world_size`) never raises -- a probe that could fail would just move the silent-default problem one level down. Tests: test/registered/unit/distributed/test_uniform_floor_fallbacks_583.py * falsifier: pins the pre-fix silent-local behaviour first, then asserts both paths refuse on a group * single-rank behaviour preserved on both paths (else this is a regression, not a fix) * a present reduced value always wins; a ZERO floor is honoured and not treated as missing (`if not floor` would be the same bug in disguise) * 10 passed; can-fail proof: reverting either guard turns the two refusal tests red Also updates the sgl-project#603 and sgl-project#583 stubs to model `self.ps.tp_size` rather than inventing a flat `self.tp_size` -- a stub inventing a surface the real class lacks is precisely what let the census ship dead. Full run: test/registered/unit/distributed + unit/managers = 2875 passed, 30 skipped, 735 subtests passed. Remaining failures are pre-existing and not from this branch: 3 subtests in test_dcp_context_ceiling.py (verified identical at 38e7c69) and test_bar1_host_cleanup.py's pgrep self-match test, which passes in isolation (28/28) and is sensitive to concurrently running processes. Directory ruff count unchanged at 4.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
Boots 13, 14 and 16 all died with a message that reads as a contradiction:
alloc_req_slots runs out of memory. ... available_size()=4, num_reqs=1
Four request slots free, one request wanted, and it raised anyway. The same
line appeared with avail=2 (boot 13) and avail=3 (boot 16).
`available_size()` counts REQUEST slots only. On a HybridReqToTokenPool the
allocation also needs a mamba state -- and, with the extra buffer, a
ping-pong slot -- and `alloc` returns None if EITHER is unavailable. In all
three crashes the mamba pool was full (96/96 on boot 14). So the message
reported the one pool that was fine and said nothing about the one that was
not.
A fail-loud path that names the wrong resource is worse than a quiet one:
it does not merely fail to help, it points away from the cause. This cost
real diagnosis time on three separate boots.
The message now carries mamba_available / mamba_schedulable / mamba_total
whenever the pool is hybrid, labels the request-slot count as such so it
cannot be misread as "the" capacity, and names the two places to look
(--max-mamba-cache-size, or mamba slots held by unevictable cached states).
The diagnostic is wrapped so that a broken allocator cannot replace the real
OOM with its own error.
Tests: test/registered/unit/mem_cache/test_alloc_req_slots_names_the_pool_583.py
* falsifier drives boot 14's exact numbers (avail=4, num_reqs=1,
mamba 0/96) and asserts the mamba figures appear
* pins that the non-hybrid message grows no mamba noise
* pins that a throwing mamba allocator still surfaces the real OOM
* 5 passed; can-fail proof: restoring the old message fails 4 of them
Pre-existing failures, NOT from this branch (verified by running the
directory with this change reverted): test/registered/unit/mem_cache is at
937 failures on the integration tree already, and
test_hicache_nixl_storage.py fails collection for a missing optional `nixl`
module. Adding this change leaves both counts unchanged.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 6, 2026
…ilies the desync hid in The 2026-08-06 05:53:59 crash was a rank-arrival desync, and the census was armed and running through all of it. It reported nothing. Not because it failed -- because every family it tracked genuinely agreed: dcp.all_gather 10176x, dcp.all_reduce 2864x, tp.all_gather 580x, tp.all_reduce 36824x, identical on rank 0 and rank 1. The family that diverged was necessarily one nobody counted, and the abort named an 8-byte collective served by the a2a kernel. Neither broadcast nor all_to_all was censused at all. Agreement across every family you count is not health when the failure is in a family you do not count. WHAT CHANGED - collective_clock_families() returns two more names per group, "<group>.broadcast" and "<group>.all_to_all". This is the one place a group's families are named and both instruments read it. The two new names are CENSUS-ONLY: they carry no _COLLECTIVE_CLOCK span, because arming a span is a separate decision with its own cost and its own log-line surface. Naming a family there is not a claim that it is timed, and the docstring now says so. - GroupCoordinator.broadcast and .all_to_all_single bump their family, at the same seam and under the same placement rule as the existing four (after the clock's re-entry guard, before the world_size==1 bypass), so a collective is counted exactly once. all_to_all_single_v shares the all_to_all family: a rank that skips an uneven a2a has skipped an a2a. No bump was added to _all_to_all_single -- the public seam already counted the call that routes through the registered op, and bumping both would double count. - CollectiveCensus.declare_families(): the census payload width is now fixed by replicated config instead of by rank-local state (sgl-project#610). snapshot() is what all_gather_object packs, so the KEY SET is the payload width; left to grow on first use it is a function of what a rank has happened to execute, which is exactly the rank-local state this instrument exists to catch diverging. Group construction is replicated, so each group declares its families at construction and every rank packs the same keys from the first comparison onward. Second effect worth having: a declared family that never fires is visible as 0x instead of absent, so "never counted" and "not instrumented" stop looking alike. Mechanics are unchanged otherwise: same rank-cumulative dict, same gloo all_gather_object cross-check on tp_cpu_group, same union-based differ, same warn-never-raise. Hot-path cost is one further dict increment on two seams, behind the existing module-level bool. TESTS test/registered/unit/distributed/test_collective_census_broadcast_families_583.py, 12 cases, hermetic (CUDA_VISIBLE_DEVICES=99, no torch.distributed init, no group, no CUDA). They drive the REAL dispatch sites and the REAL differ, so they fail on a tree missing the bump rather than on a tree where a string changed. The harness is deliberately arity-tolerant about the family tuple so the can-fail lands on behaviour, not on fixture construction. 12 passed here. 11 of 12 FAILED on bccb87b with the same file. The headline case, test_a_broadcast_desync_is_named (129 vs 128 broadcasts across two ranks), fails there with "'tp.broadcast' not found in set()" -- the differ returns an EMPTY list, which is precisely "the divergence passes silently". test_an_all_to_all_desync_is_named fails the same way ([] != ['tp.all_to_all']). The 1 case that passes on baseline is test_agreement_on_the_new_families_stays_silent, which asserts an empty diff when the counts AGREE; it is the no-cry-wolf guard, not a can-fail case, and it passes on baseline trivially because nothing is counted there. No regressions: test/registered/unit/distributed -> 13 failed / 2404 passed here versus 13 failed / 2392 passed on bccb87b. The failing set is IDENTICAL (set difference empty); the 13 are pre-existing and unrelated (test_uniform_decode_mem_603 and siblings, _FakeScheduler fixtures). The +12 is exactly this commit's new tests. test/registered/unit/managers/test_rank_prefill_log.py: 32 passed, 4 skipped. test_rank_prefill_log.py's family-tuple pin was updated to the six names and now also asserts dcp.broadcast; its sibling that pins the four CLOCK dispatch sites is deliberately left at four, which is what keeps the census-only tail honest. ruff and codespell clean on all changed files. Out of scope, still a boot ticket: the PeerWatchdog sequence-number liveness mechanism that would catch a peer that is alive but not arriving.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 6, 2026
Two instruments armed during the wedged soak agree the ranks were in step: the sgl-project#583 census reported no divergence and its local dumps are byte-identical between ranks, and the barlink 1 Hz launch sampler shows all three ranks carrying the same last-op records at wedge time (all_gather 192512, broadcast 32, broadcast 24, all_gather 192512). So the wedge is neither a pairing shift nor a count divergence, and that line of inquiry is closed. What remains: if every rank enqueued the same collective and all three still block in the next host sync, the queue is busy rather than starved, and Bar1CollectiveAborted is a consequence of a peer's deadline expiring against a GPU that is still executing. That points at the device work just before the sync (create_flashinfer_kv_indices_triton, and the torch.empty(total) ahead of it) and at a pathological total / paged_kernel_lens arriving from replicated scheduler state -- which would hit all ranks at once, as observed. Recorded as a hypothesis with the measurement that would settle it, not as a finding. Also records why the per-site D2H removal was dropped: owner.py has a second unavoidable sync at line 566 (boolean-mask indexing calls nonzero), so removing the .item() cannot make the function sync-free; and the plumbing that would have fed total_tokens derived it from extend_prefix_lens_cpu, which is not guaranteed on every model -- a model-dependent fix in a model-independent layer, where a wrong total silently mis-sizes an attention index buffer.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 6, 2026
Every host-visible signal says the ranks agree during the wedge: sgl-project#583 census counts match, the launch sampler shows identical last ops and identical captured/eager flags, and the host kernel logs zero Xid, zero PCIe AER, zero IOMMU faults and zero GPU resets across 23 h of uptime. Meanwhile all three cards sit at 100% SM occupancy with 0% memory utilisation and power far below cap -- spinning, not computing. The one thing nothing reads is the device flag state the spin waits on, which is where the remaining explanations live (generation mismatch, or a flag written but never observed). This adds that read to the abort path only. Safe precisely there, and nowhere else: the 1 Hz launch sampler refuses device reads because it runs while a collective may be in flight and would queue behind the wedged spin; by abort time the kernel has already taken its abort path, so nothing is left to queue behind. Warn-never-raise, like the census dump beside it, so the instrument can never mask the real error. Reports the first dword of each 256-byte flag line; diffing the three ranks' snapshots names a generation mismatch. Zero cost until an abort fires.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 6, 2026
The abort-time flag snapshot fired on its first real wedge and produced the number no other instrument could read: rank 0's highest flag generation is 1190714, rank 1's is 1190715. The ranks are out of step by exactly one collective generation. Section 15 listed this as one of two untestable candidates; it is now measured. It also explains why every earlier instrument was silent. The sgl-project#583 census counts at the Python GroupCoordinator layer and the launch sampler compares op name and size - both identical across a one-generation skew. The skew lives in the device-side BAR1 generation counter, which neither reads. The objection that the pairing-shift mechanism owed an explanation for the census silence is therefore answered. And it unifies the two families: the index assert was a pairing shift of exactly one collective (accept_index receiving predict, same wire size); the wedge is a skew of exactly one generation. Same distance, and the symptom depends only on whether the mismatched pair happens to be the same size - silent substitution if yes, spin to the cycle cap if no. The accept-broadcast fusion removed the consequence for one pair, not the skew, which is why the wedge survives it. Still unknown: which collective creates the skew. Recorded, with the limitation that rank 2 died before emitting its snapshot, and a correction to the previous commit's claim - this wedge happened under agent-only load, not saturation.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 8, 2026
…e, never __trap The last live instance of the sgl-project#583 context-killing mechanism: wait_ge in the host-staged transport still called __trap() on spin expiry, which destroys the CUDA context and surfaces as sticky 'unspecified launch failure' at whatever unrelated call site runs next. Unreachable in every production configuration (host is explicitly-selected, no-fallback, bar1 pinned) — but a landmine for any host-transport measurement boot. Ported the device transport's sgl-project#583 fix honestly, not minimally: - wait_ge returns bool and writes a per-site code into the abort word on expiry (put reuse-guard=1, reduce peer-wait=2, copyout wait=3); the BARLINK_HOST_TRAP macro and its hipify plumbing are deleted. - Block-uniform exit: shared abortS written by the spinning thread 0, __syncthreads(), all threads return together before any payload loop, fence, or last_block barrier — verified at all three sites. Terminal- abort contract documented: cross-block skew and stale blk_ctr are acceptable residue of a process the host is about to kill by raise. - The transport's THREE sequence-counter families (collective, per-peer send, per-peer recv) are unified into one zero-initialized backing tensor with 2-word rows — the spec's naive grow-to-2 would have made the p2p kernels' abort store corrupt a NEIGHBOR's sequence number, and the unified shape makes check_aborted a single read over every abort word the transport can produce. - HostCollectiveAborted mirrors DeviceCollectiveAborted (rank/world/ code/where, garbage-results contract) and appends the sgl-project#650 peer statement. check_aborted is a plain in-line read (documented choice: this is a measurement vehicle, not the bs=1 hot path) wired twice: the duck-typed _after_transport seam (sgl-project#431) for eager collectives, and abort-gate registration for replay-only collectives — host is capturable, so a graph-resident collective never reaches the seam (the same gap sgl-project#583 closed for the device transport). - The load-bearing reachability claim is pinned: 'host' is in the no-fallback set via the real parallel_state lookup, and the dispatch seam is proven transport-stable mid-run — an explicit SGLANG_BARLINK_TRANSPORT=host boot is the only way this code runs. Falsifiers, red-first (executed against the pre-fix source inside the test): old wait_ge provably trapped and returned void, old seq tensor had no abort-word room; fixed source has zero traps, bool returns, all three consuming sites, distinct codes. nvcc compile smoke rc=0 (sm_86). GPU-gated on-card falsifier written and documented (put-kernel forced timeout must set the word, leave the context ALIVE, and raise named) — skips at the desk by design. Counting basis: 35 passed + 1 GPU-skip + 19 subtests in the new file; full battery with abort/stub-drift/sentinel regression suites 81 passed + 1 skipped + 19 subtests (97.9 s, my rerun); ruff and imports clean.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 12, 2026
…tion alignment
C30 was booked as "--enable-deterministic-inference and
--enable-kv-session-offload cannot both be on". That diagnosis is wrong on
every load-bearing detail, and wiring the refusal it implied would have
rejected a working configuration while leaving the real trap armed
everywhere else.
THE ACTUAL MECHANISM, verified in source. PrefillAdder.add_one_req's chunked
branch (schedule_policy.py) aligns the chunk it is about to take:
trunc_len = self.rem_chunk_tokens // self.page_size * self.page_size
if truncation_align_size is not None:
if trunc_len < truncation_align_size:
return AddReqResult.OTHER
rem_chunk_tokens is bounded above by --chunked-prefill-size, so a chunk
budget below one alignment unit refuses EVERY request longer than the budget,
on every iteration, forever. The scheduler's admission loop breaks on any
non-CONTINUE verdict, so one such request at the head of the FCFS queue
blocks the queue behind it, can_run_list stays empty and no batch is ever
built. The instance boots, prints "fired up and ready to roll", serves its
warmup prefills (short enough to take the non-chunked branch) and then admits
nothing -- the worst failure shape there is.
kv-session-offload's ONLY role is that it forces the flashinfer backend. The
refusing predicate is a third variable, --chunked-prefill-size, which the
wedged boot had at 256 against an alignment of 4096. The same two flags with
a chunk budget of 4096 serve normally, so the flag-pair refusal the register
called for would have rejected a configuration that works.
Two claims in the C30 entry do not survive re-reading its own evidence, and
both are corrected in the register:
* there is NO rank divergence. probe_v8.log prints the sgl-project#583 collective census
from both ranks throughout the wedge and the two are IDENTICAL (all_reduce
1862x on each). Frozen because no forward ran, not because a rank was stuck.
`num_queue_reqs 0` is a 30 s-cadence idle gauge sampled after the client had
already disconnected.
* the disabled radix cache is a red herring. ChunkCache inherits
evictable_size() -> 0 (never None, never raising) and _tree_evictable_size()
already tolerates it; losing the radix cache costs restore headroom, it does
not gate admission.
THE ALIGNMENT HAS TWO INDEPENDENT SOURCES and either alone arms the trap, so
the guard covers both: --enable-deterministic-inference on flashinfer/triton
(align 4096 by default), and --mamba-checkpoint-interval, which sets the
alignment on its own with no deterministic inference anywhere and is lcm-ed
into it when both are present. The check therefore sits in
init_deterministic_inference_config AFTER the lcm -- the only point where the
final alignment exists; server_args cannot see it without restating the lcm.
Side-finding, fixed in the same commit: --mamba-checkpoint-interval's own help
recommended "2048", which silently wedges any boot whose chunked-prefill-size
is below 2048 -- including this rig's ship config at 512. The help now states
the constraint and the guard enforces it.
INERT ON THE SHIP CONFIG, verified against the live process rather than
assumed: /proc/<pid>/cmdline shows --chunked-prefill-size 512, no
--mamba-checkpoint-interval and no --enable-deterministic-inference, and the
helper returns None for (512, page 1, align None).
RED-FIRST, EXECUTED NOT ASSERTED, and split so both halves are visible on the
unfixed tree. The guard import is deliberately made INSIDE the guard tests so
collection still succeeds without it. Reverse-applying the source patch:
the MECHANISM test PASSES (the trap is real and reachable through the real
add_one_req) while the WIRING test fails behaviourally -- "the scheduler
accepted --enable-deterministic-inference on flashinfer with
--chunked-prefill-size 256: that instance boots, reports ready and then
admits nothing at all". Re-applied: 3 passed.
The mechanism test measures can_run_list, not add_one_req's return value: a
successful chunked admission consumes the whole chunk budget, so budget_state
reports OTHER on the way out of a SUCCESS just as the refusal does. An
earlier draft asserted on the return value and its own control assertion
caught it -- kept as a control that proves the fixture can admit the same
request at a satisfiable alignment.
Tests: sgl-project#631 flip family 1095 passed / 0 failed. The count moves 1092 -> 1095
because the new file was ADDED to the canonical list in
scripts/run_631_flip_family.sh -- this tree has had three under-collections
on that list and an admission-path test must not be orphaned. kvso + prefill
adder families 217 passed / 0 failed. ruff: identical to baseline on all
three touched source files, clean on the new test. codespell clean.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…roject#677 to compose against The sgl-project#679 close-out said the retry net's registry is empty because the reliefs that could genuinely pay are collective and belong at admission. This is that claim made concrete, written so sgl-project#677's hysteresis-drain design can consume it without re-deriving anything. WHAT IT ESTABLISHES. Four reliefs exist and only four: radix eviction, the sgl-project#287 ladder's admission_cap, kvso try_spill, and retract_decode. For each: what it frees at the admission decision point, what it costs, and how it is invoked without splitting the group. Two of them are not what one would guess: the sgl-project#287 ladder is TOO SLOW to be a rung. Its consensus boundary is every 8 rounds; a chunked-prefill burst exhausts the pool in fewer. It is the slow outer loop -- and the decode path already uses it that way, throttling before retraction to stop the freed slots being handed straight back. kvso try_spill is the BEST rung, not retraction. It frees a bounded, chosen amount (the victim's block-aligned tail overhang), is already driven from the reduced value, and costs no request's progress. Its bound is the host region supply, and exhaustion is a reachable state the decode path already documents. THE ITERATION ORDER, VERIFIED, AND IT IS FAVOURABLE. The reduce runs at scheduler.py:4777, unconditional and pre-branch by its own comment; admission at 5089; retraction at 5950 inside update_running_batch (5818). So a ladder at admission reads an ALREADY-AGREED number and needs no collective of its own, while retraction is genuinely downstream -- which is why sgl-project#679's crash had nothing to fall back on, and what rung 3 must work around. THE ORDER, mirroring the decode-OOM branch rather than inventing a second shape: evict (baseline) -> try_spill -> throttle -> retract_decode -> PARK. Parking stays the floor of the ladder, not its replacement. THE COMPOSITION CONTRACT WITH sgl-project#677, stated in three rules because both mechanisms decide admission from pool headroom: 1. The park guard is the FINAL authority, the drain gate the prior one. A phase gate cannot make memory exist; it may narrow what the park guard allows, never widen it. 2. Any headroom quantity sgl-project#677 branches on must be the GROUP-PUBLISHED floor. A rank-local reading in the drain gate reintroduces the sgl-project#603/sgl-project#583 divergence class upstream of every safeguard sgl-project#679 added -- and a park guard reading the reduced floor does not protect a gate that reads a local one. 3. Parking must be reachable from every path that reaches alloc_for_extend. AND THE LIKELIEST COMPOSITION FAILURE, called out so it can be tested on both sides before it is met on metal: hysteresis and parking can beat against each other. A parked chunk schedules ZERO tokens, so a drain that counts admission ATTEMPTS rather than SCHEDULED TOKENS will believe work was taken, hold its hysteresis, and deadlock against the park at exactly the pressure where both are needed. Section 5 lists what this note does NOT close: the ladder is unbuilt, rung 3 is a real refactor, and rung 1's host-region bound has never been measured under the 5-lane load that produced the crash. No code change. Every claim carries its file:line.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…ore parking
Built to DESIGN_679_admission_relief_ladder.md. A park is not free -- it is a
request that made no progress this round -- and the note costed what should be
spent before accepting one. This spends it.
rung 0 radix eviction already spent by the caller. Baseline, not repeated.
rung 1 kvso.try_spill bounded, chosen, costs no request's progress.
rung 2 throttle frees NOTHING now; stops rung 3 repeating next round.
rung 3 retract_decode most tokens, loudest: the victim re-prefills.
rung 4 PARK the floor, and still the final authority.
RUNG 3 IS THE ONLY HARD PART, AND IT IS A REFACTOR, NOT A CALL. retract_decode
is one line; what surrounds it in update_running_batch is what must not drift --
the metrics, the new_token_ratio handover, the abort dispatch, and above all
for req in retracted_reqs:
self._add_request_to_queue(req, is_retracted=True)
A second implementation that forgot that line would LEAK every victim it
retracted, which is worse than the crash this ladder exists to prevent. So the
block is EXTRACTED VERBATIM into _retract_decode_and_requeue and both call sites
share it. One implementation, two callers, no drift -- and a test pins that the
shared actuator still contains the requeue.
The precondition travels with it: rung 3 sets batch.uniform_avail_floor from the
reduced value before retracting, because that bound governs the retraction loop
AND the last-survivor test. sgl-project#583 is exactly the case where the entry decision was
uniform and the loop bound was not, so ranks entered together and popped
DIFFERENT numbers of victims.
GUARD (a) -- EXHAUSTION IS AN OUTCOME, NEVER AN ERROR. try_spill returns False
when no host region is free; the ladder falls through to the next rung. Same for
a rung that frees less than asked, and same for a rung that RAISES: every rung is
wrapped, the ladder continues, and nothing here can turn a relief bug into an
instance death. That host-region bound is still unmeasured under the 5-lane load
that produced the crash, which is precisely why it is treated as ordinary.
GUARD (b) -- EVERY DECISION IS GROUP-UNIFORM. Each rung reads uniform_min_avail(),
the value the pre-branch reduce published at the top of this iteration, so no rung
takes a collective of its own and none can split the group. The shortfall is sized
from that same reduced value, so every rank asks its rungs for the same tokens --
sizing it locally would be sgl-project#583 one layer up from where sgl-project#583 was found.
GUARD (c) -- OFF BY DEFAULT. SGLANG_ADMISSION_RELIEF_LADDER unset returns 0 before
touching anything: byte-identical to c4b88e1, the boot currently serving. Rung 3
carries a SECOND flag (SGLANG_ADMISSION_RELIEF_RETRACT) because it is the only rung
that destroys progress, and it is inert unless the ladder itself is on.
THE PARK REMAINS FINAL (DESIGN_679 rule 1). The ladder changes what there is to
decide from; add_chunked_req still decides. It runs immediately before that call
and nowhere else, and a source pin refuses can_run_list / add_chunked_req /
set_extend_range inside it -- a ladder that admitted anything would be a second
admission authority.
Tests: 18 hermetic, on a Scheduler stub carrying only what the ladder touches.
Mutation-proven on the four decisions that could invert silently:
default flipped ON -> the off-by-default case fails
rung 3 without uniform_avail_floor -> the sgl-project#583 precondition case fails
exhausted spill treated as terminal -> the fall-through case fails
a paid spill no longer short-circuiting -> the ordering case fails
Each fails alone, by name. One test of my own was caught by its first run reading
the wrong frame (get_new_batch_prefill delegates to _get_new_batch_prefill_raw) --
the same stale-pin class repaired in the PP loop tests, fixed the same way.
No regressions: 1011 pass across the scheduler/prefill/seam suites, zero failures.
The extraction adds no lint (94 before, 94 after, all pre-existing).
NOT BOOTED. Desk-only; validation bundles with the sgl-project#678 acceptance.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…, by design not by accident The build brief named group-uniform rung decisions as a guard to be mutation-proven. It was implemented -- every decision reads uniform_min_avail() -- but the tests caught a rank-local read only by ACCIDENT: the stub carried no allocator, so a mutation raised AttributeError, the trigger's own except swallowed it, and the ladder silently did nothing. Tests that pass because the wrong code crashes are not tests of the property. So the stub now carries a rank-local availability that DISAGREES with the reduced one -- 10x the chunk, i.e. comfortable, while the group is starved -- and two cases assert the ladder acts on the group's number: it spends rungs when the GROUP is short even though this rank looks fine, and it sizes the shortfall from the group value (sgl-project#583 one layer up: ranks that size the ask differently retract different numbers of victims). Mutation-proven properly now. Pointing the TRIGGER at this rank's pool fails five cases including both new ones by name; pointing the ladder's internal comparison at it fails the ordering case. Neither is an AttributeError any more -- each is a wrong verdict, which is what the group would actually experience. 20 tests. No production change.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…l; the new-request gate never did sgl-project#679 taught `add_chunked_req` not to schedule a chunk the pool cannot fund, and it made that decision GROUP-UNIFORMLY: `fundable_extend_tokens` reads the published group MIN through `uniform_avail_for_evict`. Its sibling gate -- the one every FRESH request passes -- was left reading this rank's own pool: PrefillAdder.rem_total_tokens available_size() + evictable_size() - offset - dcp_avail_deficit That is the same over-promise one door further along, and it fails twice over: * A rank roomier than the binding rank admits work the GROUP cannot fund, and the batch dies where it is allocated -- `alloc_for_extend`, 2026-08-16 01:46:10, all three ranks together. * It is a rank-local BRANCH upstream of a collective. Two ranks can build different batches and enter the next extend with different token axes, which is a HANG rather than a stall -- the family sgl-project#583/sgl-project#603/#616g/sgl-project#639 was paid for. `fundable_extend_tokens`' own docstring names the first half. It was written for the chunked gate and never wired to this one; `fundable_extend_tokens` appeared nowhere in scheduler.py, which the red-first run confirms. WHAT THIS IS NOT. It is not the cause of the 01:46 traceback. That was e778276 -- an unlocked mamba tombstone leaf the eviction frontier selected and `_evict_leaf_node` could not consume, so eviction under-delivered while the counter honestly promised 65766 evictable. THE COUNTER AND THE ACTUATOR NOW AGREE; this commit is about the second consumer of that counter, which reads a rank-local copy of it and can over-promise even when it is honest. A CEILING, NOT A SUBSTITUTE. `rem_total_tokens` also subtracts reservations the floor knows nothing about (the running batch's hold, mamba gap, page overhead), so replacing the local term would talk a rank that is ITSELF short back UP. The budget is the MIN of the two. Both spend down the same `rem_total_token_offset` -- a floor consulted per request without that shared accounting is not a bound at all, since every request in the round would compare itself against the same untouched pool. `dcp_avail_deficit` is deliberately NOT subtracted from the floor term: the floor is already the group MIN, and the deficit is the other mechanism for pinning a local number to the binding rank. WHY THE MIN RESOLVES TO THE FLOOR IN PRODUCTION, and it is a test, not a claim: `floor = uniform_avail + evictable` and `local = local_avail + evictable - offset` with `uniform_avail <= local_avail` by definition of a MIN, so `floor - offset <= local` on every rank. A rank BELOW the floor is not a state the reduce can produce; the clamp's behaviour there is pinned separately because refusing to be talked up by a stale floor is the wanted behaviour. THE CEILING MUST NOT BE ABLE TO WEDGE THE INSTANCE, and this is why the wiring goes through `published_fundable_floor` rather than calling `fundable_extend_tokens` directly. That helper returns 0 for BOTH "the pool is empty" and "the pool could not be read". The chunked gate can live with the ambiguity -- 0 parks a chunk and the next round retries, a self-clearing state. As a budget ceiling a mis-read 0 admits NOTHING, for every request, on every subsequent round: a harder wedge than the crash it prevents. So the ceiling is applied only where a floor was actually PUBLISHED, which is the one state (uneven rank pools) it exists for. A published floor of genuine zero still binds -- the distinction is "was a floor published", not "is it non-zero", or the ticket's own crash state would be exempt from its own fix. DEFAULT PATH UNTOUCHED, BY CONSTRUCTION AND BY ASSERTION. With no floor published -- single rank, or pools that agree -- `fundable_extend_floor` is None and `rem_total_tokens` returns exactly what it returned before. TESTS. RED-FIRST against 7936bc4: 12 of the 18 fail there, including the 01:46 shape itself (a rank holding 100000 tokens admitting against a group that holds none) and all three wiring pins. The guard against the wedge was falsified separately by removing it -- 2 of its 4 cases go red -- because an instrument that cannot fail is not evidence. Covered: the cap; the floor spent down by admissions; group agreement across ranks with different local pools; the floor binding before the local term whenever the reduce's invariant holds; the local term still binding when it is the smaller one; inertness with no floor and with a roomy floor; the published-vs-readable distinction including a genuine zero; and three pins that this stays wired (the scheduler constructs with a floor, it comes through the guarded helper, and that helper still delegates to the group-uniform one). NO REGRESSIONS, measured against the same subset on the unpatched tree: 944 failed / 2776 passed / 725 skipped before, 944 failed / 2794 passed / 725 skipped after -- identical failures (all CUDA-gated collection, hermetic run with CUDA_VISIBLE_DEVICES=""), +18 = exactly the new cases. NOT BOOTED. Desk-only; READY-FOR-BOOT. The metal arm that would exercise it is a multi-rank boot with uneven pools under max fill -- the arm that produced the 01:46 specimen -- watching for the admission ceiling binding without a wedge.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…l; the new-request gate never did sgl-project#679 taught `add_chunked_req` not to schedule a chunk the pool cannot fund, and it made that decision GROUP-UNIFORMLY: `fundable_extend_tokens` reads the published group MIN through `uniform_avail_for_evict`. Its sibling gate -- the one every FRESH request passes -- was left reading this rank's own pool: PrefillAdder.rem_total_tokens available_size() + evictable_size() - offset - dcp_avail_deficit That is the same over-promise one door further along, and it fails twice over: * A rank roomier than the binding rank admits work the GROUP cannot fund, and the batch dies where it is allocated -- `alloc_for_extend`, 2026-08-16 01:46:10, all three ranks together. * It is a rank-local BRANCH upstream of a collective. Two ranks can build different batches and enter the next extend with different token axes, which is a HANG rather than a stall -- the family sgl-project#583/sgl-project#603/#616g/sgl-project#639 was paid for. `fundable_extend_tokens`' own docstring names the first half. It was written for the chunked gate and never wired to this one; `fundable_extend_tokens` appeared nowhere in scheduler.py, which the red-first run confirms. WHAT THIS IS NOT. It is not the cause of the 01:46 traceback. That was e778276 -- an unlocked mamba tombstone leaf the eviction frontier selected and `_evict_leaf_node` could not consume, so eviction under-delivered while the counter honestly promised 65766 evictable. THE COUNTER AND THE ACTUATOR NOW AGREE; this commit is about the second consumer of that counter, which reads a rank-local copy of it and can over-promise even when it is honest. A CEILING, NOT A SUBSTITUTE. `rem_total_tokens` also subtracts reservations the floor knows nothing about (the running batch's hold, mamba gap, page overhead), so replacing the local term would talk a rank that is ITSELF short back UP. The budget is the MIN of the two. Both spend down the same `rem_total_token_offset` -- a floor consulted per request without that shared accounting is not a bound at all, since every request in the round would compare itself against the same untouched pool. `dcp_avail_deficit` is deliberately NOT subtracted from the floor term: the floor is already the group MIN, and the deficit is the other mechanism for pinning a local number to the binding rank. WHY THE MIN RESOLVES TO THE FLOOR IN PRODUCTION, and it is a test, not a claim: `floor = uniform_avail + evictable` and `local = local_avail + evictable - offset` with `uniform_avail <= local_avail` by definition of a MIN, so `floor - offset <= local` on every rank. A rank BELOW the floor is not a state the reduce can produce; the clamp's behaviour there is pinned separately because refusing to be talked up by a stale floor is the wanted behaviour. THE CEILING MUST NOT BE ABLE TO WEDGE THE INSTANCE, and this is why the wiring goes through `published_fundable_floor` rather than calling `fundable_extend_tokens` directly. That helper returns 0 for BOTH "the pool is empty" and "the pool could not be read". The chunked gate can live with the ambiguity -- 0 parks a chunk and the next round retries, a self-clearing state. As a budget ceiling a mis-read 0 admits NOTHING, for every request, on every subsequent round: a harder wedge than the crash it prevents. So the ceiling is applied only where a floor was actually PUBLISHED, which is the one state (uneven rank pools) it exists for. A published floor of genuine zero still binds -- the distinction is "was a floor published", not "is it non-zero", or the ticket's own crash state would be exempt from its own fix. DEFAULT PATH UNTOUCHED, BY CONSTRUCTION AND BY ASSERTION. With no floor published -- single rank, or pools that agree -- `fundable_extend_floor` is None and `rem_total_tokens` returns exactly what it returned before. TESTS. RED-FIRST against 7936bc4: 12 of the 18 fail there, including the 01:46 shape itself (a rank holding 100000 tokens admitting against a group that holds none) and all three wiring pins. The guard against the wedge was falsified separately by removing it -- 2 of its 4 cases go red -- because an instrument that cannot fail is not evidence. Covered: the cap; the floor spent down by admissions; group agreement across ranks with different local pools; the floor binding before the local term whenever the reduce's invariant holds; the local term still binding when it is the smaller one; inertness with no floor and with a roomy floor; the published-vs-readable distinction including a genuine zero; and three pins that this stays wired (the scheduler constructs with a floor, it comes through the guarded helper, and that helper still delegates to the group-uniform one). NO REGRESSIONS, measured against the same subset on the unpatched tree: 944 failed / 2776 passed / 725 skipped before, 944 failed / 2794 passed / 725 skipped after -- identical failures (all CUDA-gated collection, hermetic run with CUDA_VISIBLE_DEVICES=""), +18 = exactly the new cases. NOT BOOTED. Desk-only; READY-FOR-BOOT. The metal arm that would exercise it is a multi-rank boot with uneven pools under max fill -- the arm that produced the 01:46 specimen -- watching for the admission ceiling binding without a wedge.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 22, 2026
… unified lineage The previous commit refused the sgl-project#755 floor reduction because the lineage that gets built does not implement the mechanism. This builds it, and the floor drops back to 1+P+1 -- honestly this time, because the runtime now does what the arithmetic claims. MECHANISM. cache_unfinished_req releases the OLD anchor's mamba pin BEFORE the donation allocation instead of after the insert, so the old and new anchors never coexist and a running request holds active + donated rather than active + donated + old pin. That is the slot, and on the standing boot (max_running_requests 8) it is 8 of them. STRICTLY SAFER THAN THE ORIGINAL, and this is why the port was worth doing rather than re-gating. MambaRadixCache releases the WHOLE node lock early; inc_lock_ref walks ancestors for the FULL component, so its window also leaves the request's own matched KV prefix evictable. The unified tree locks per component, so dec_mamba_lock_only drops ONLY the mamba lock: the KV path stays protected and the single state slot being un-double-counted is the only thing made evictable. dec_swa_lock_only is the same shape for SWA, so this is the file's existing idiom. PER-NODE GATE. The config predicate cannot know whether THIS anchor survives losing its slot, so MambaComponent.anchor_release_admissible asks again: the node must carry a host copy (an evicted-but-backed anchor is a valid match that triggers load_back -- is_resume_candidate's device_only=False rule), and that copy must have LANDED. sgl-project#767: write-through publishes host_value the moment the transfer is handed to the controller and records the node in ongoing_write_through in the same block; between those two facts the anchor is an intention only, and releasing there is the dead anchor the gate exists to prevent. PAIRING, which is where this class of change goes wrong (sgl-project#583). Two exit paths, both handled: * the normal path skips the mamba half at the usual dec_lock_ref, via the skip_lock_node_ids set the components already consult -- a ref that was never taken must not be given back; * the bail path (nothing inserted, so no new anchor takes over the pin and req.last_node is unchanged) RE-ACQUIRES, restoring the exact pre-call state rather than carrying an imbalance into the next call. TESTS: test_mamba_reorder_unified_773.py, 17 tests, CPU-only. The end-to-end pair is the real proof: the pool is sized so the donation allocation can only succeed if the old anchor was released first (2 slots = one active + one anchor, free = 0). A backed anchor caches; an unbacked one, identical in every other respect, does not. That is the saved slot, observed as behaviour rather than as arithmetic. CAN-FAIL, ALL EXECUTED, and three of them were earned the hard way: * unwiring the reorder in cache_unfinished_req -> 1 fails * dropping the skip set -> 1 fails * dropping the bail re-acquire -> 1 fails The last two SURVIVED the first version of this file, because every pairing test simulated the sequence by hand instead of driving the entry point. release_component_lock no-ops at zero, so a double release on a singly-held node hides completely; it only shows when the anchor has a SECOND holder, whose ref the unskipped release would steal. Both tests were rewritten to go through cache_unfinished_req, and the skip-set case needed a spare slot -- with a full pool the step bails and correctly restores the pin, which is a different path and now has its own test. Floor on the standing boot: 24 -> 16. The retention budget introduced in 703b05c becomes 8 slots instead of 0, so mamba host backups are possible again rather than uniformly declined. REGRESSION: test/registered/unit/mem_cache/ = 1473 passed, 8 failed, the same 8 as an untouched HEAD worktree, diffed name-for-name.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 4, 2026
…it in a named stop BOOT weg1b7 (03:49:20 -> 03:50:24Z) died in a circular PP ring wait: PP0 parked in recv_object[src=2], PP1 tore at the 60 s chain-stall bound, both downstream ranks ran their two ring-cut drain turns and the group died. ROOT (built against, and it is NOT the ring cut). The sgl-project#631 row probe peeks the head proxy frame's admission row and, when the row names a rid this rank cannot locate, answers False and leaves the frame in the inbox. That protection is right (boot 631row14: a frame consumed before its request hop plans nothing and the ring dies upstream-waiting) and is untouched here. What it lacked is a BOUND, and the defer is not free: PP0 has ALREADY launched the pass the frame belongs to and is ALREADY parked on its output, so an unbounded defer closes the ring BY CONSTRUCTION. The defer's own premise -- "an unlocatable rid PROVES a chain send is in flight, bounded by rendezvous+transfer" -- is false for both populations measured in weg1b7: * a rid PP0 RETRACTED after sealing the row. PP0 shipped sgl-project#631 PROXY-SEND t40 naming 772368da and c25108f3, then the discretionary #888b parked- carrier relief retracted c25108f3 in the NEXT scheduling iteration (#969AD RETRACT site=_retract_decode_and_requeue fwd_ct=157). A sealed row cannot be amended, so _missing can never empty. * a re-admitted resident (sgl-project#968 READMIT CACHED, last_queued_as=cutover- requeue) requeued rank-locally by _add_request_to_queue -- no chain hop for it exists or ever will. Ranks disagreeing about a rid is RAENGE-NIE-UNEINS, and the sanctioned answer there is a bounded, NAMED stop -- never a silent unbounded wait. sgl-project#1071's occupant horizon is the precedent this follows, down to "0 disables". CHANGE python/sglang/srt/managers/pp_row_defer_horizon.py (new, 300 lines) RowDeferHorizon: per-slot clock over the (frame stamp, missing-rid set) identity. First sighting defers. A CHANGED missing set is a hop landing and restarts the clock -- the boot 631row15 world (43k hot defers on a TURNING ring) therefore never reaches the bound. A DIFFERENT frame also restarts it, so a slot whose head alternates cannot inherit a predecessor's age. Only the same frame with the same missing set, held past SGLANG_PP_ROW_DEFER_HORIZON_S (default 20 s), answers defer=False with a message naming slot, wait, occurrences, rids, BOTH measured shapes, what is deliberately not repaired, and the remedy knob. PURE: no logger, no wire, no scheduler attribute -- the caller owns the raise, as parked_carrier_relief and min_free_slots_delayer are built. 20 s is chosen against three measured neighbours: orders of magnitude above the healthy case (weg1b7's three defers of one frame land in the same log second), BELOW the 60 s chain stall bound so THIS names the contradiction before the ring cut misreports it as "closed ring on the request chain", and BELOW the 90 s sgl-project#1071 horizon. scheduler_pp_mixin.py:9184-9243 the defer consults the horizon and raises PpRowDeferHorizonLapsed past the bound. It does NOT consume the frame (that revives boot 631row14) and does NOT emit a void (that revives the sgl-project#801 reverse corpse). The DEFER log line now carries held=%.1fs. scheduler_pp_mixin.py:9266-9271 a satisfied or unreadable row clears the slot's clock, so a resolved wait cannot age a later one into the bound. scheduler_pp_mixin.py:8912-8941 _pp_row_chain_pending's docstring corrected: its _pp_row_chain_owed arm returns True on an INFERENCE, not on proof of availability, and the receive it opens is bounded by this horizon rather than by transfer time. Do not read that True as uniform evidence of a posted message. EVIDENCE RED FIRST on parent 2615514 (scratch worktree /spinning/wt-1180-red, only the new test file copied in): 12 failed / 2 passed. Two of the reds are BEHAVIOURAL, not import errors -- test_the_defer_is_not_unbounded: "the row probe deferred the same unsatisfiable row 4096 times over an hour of its own clock without ever stopping" (imports nothing new, on purpose); test_a_satisfied_row_clears_the_clock_its_own_defer_armed: "the defer must have armed a clock". FIXED: 14 passed / 0 failed. MUTANTS 6/6 died, each patch verified applied: M1 never raise (fix inert) DIED M2 bound never lapses DIED M3 a changed missing set does not restart the clock DIED M4 satisfied row leaves its clock behind DIED M5 no frame token in the identity DIED M6 DANGER DIRECTION: stop on the FIRST defer DIED M4 survived the first round -- the locatable-row test was vacuous (no horizon object existed to inspect). The recovery test above was added to make the clear observable, and M4 then died. ruff parity: 3 pre-existing errors on the parent, the same 3 after (F841 carries_flip_arm, E741 l x2). Import smoke hermetic (CVD=""). SUITE ONCE, both trees, same command, sequentially (crowding-free box, load ~2.9), test/registered/unit/managers with the five pre-existing collection-error modules ignored (the same five on both trees: the sgl-project#1044 import-broken *_631 quartet plus test_quiescence_no_carry_858): parent 2615514 : 240 failed / 4767 passed / 2 skipped / 638 subtests fixed : 240 failed / 4781 passed / 2 skipped / 638 subtests Delta = +14 passed, exactly the new file; the FAILED NAME SETS are byte identical (diff empty, 236 names each side). Counting probe declared honestly: 236 extracted names vs the summary's 240 on BOTH trees -- a symmetric extraction gap of 4 that is not investigated here and cannot affect the A/B, since both sides carry the same 236 names and the same summary. RESIDUALS, deliberately NOT fixed here * The stop is RANK-LOCAL, exactly like sgl-project#1071's: the deferring rank raises, the others then die on their own collectives or the deadman. A group-uniform stop would need a collective on the wedged path, which is what the ring cut already refuses. * #888b's discretionary yield is untouched. Making it protect published rids is refused: the entry decision and the victim count must be rank-uniform (the sgl-project#583 precondition stated in _maybe_yield_parked_carrier itself), and only PP0 builds rows, so a PP0-only term would make ranks pop different numbers of victims -- a silent rank divergence traded for this deadlock. * The ring cut's arc asymmetry (pp_flip_drain_leftover_dicts receives only from the draining rank's own upstream, scheduler_pp_mixin.py:6796) is left as it is: it is fork-own compensation, and under upstream-minimal a defect inside a compensation layer is a deletion candidate, not a repair order -- and widening it could not clear PP1's _missing anyway.
This was referenced Oct 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.