Skip to content

Update install commands - #583

Merged
merrymercy merged 1 commit into
mainfrom
install-flash-infer
Jul 3, 2024
Merged

merrymercy merged 1 commit into
mainfrom
install-flash-infer

Conversation

@merrymercy

Copy link
Copy Markdown
Collaborator

No description provided.

@merrymercy
merrymercy merged this pull request into main Jul 3, 2024
@merrymercy
merrymercy deleted the install-flash-infer branch July 3, 2024 09:07
merrymercy added a commit that referenced this pull request Jul 3, 2024
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
efschu added a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
…d, plus the power-cap caveat

User directive: the live hypothesis is that a captured prefill only pays IN
COMBINATION WITH barlink. The mechanism is falsifiable and specific -- NCCL's
collectives are host-driven, so the launch train a graph would tighten is not
the part that costs, while barlink's device-side collectives can sit inside the
captured graph and put the 68-75% collective share of the prefill window (sgl-project#252)
within reach. If that holds, the flat NCCL result is a property of the
transport, not of prefill graphs.

window3_boot_floor.sh now stages STAGE=nccl|barlink|all over

               prefill eager   prefill graphs
  NCCL         E1, E2          G
  barlink      BE1, BE2        BG

Each transport carries its OWN eager-vs-eager floor, a delta counts only
against the floor of its own row, and the reported swing counts only if it
clears both. Points are the long-prompt 1900 and the bs>1 concurrency mix
(256 tokens, 4 in flight), the latter scored on aggregate_tok_s because
per-request rates understate a concurrent regime. The barlink row gets its own
content gate too -- a transport change is exactly the kind of thing that moves
reduction order.

Both preconditions are enforced by the script rather than assumed: the barlink
stage refuses to run unless b001d10 (the sgl-project#583 tripped-spin-kernel / CUDA
context fix, merged here) is an ancestor of HEAD, and unless the operator
passes BARLINK_VERDICT=confirmed from the live-load repro window. Absent arms
report 'hypothesis UNTESTED at this point' instead of being guessed.

Reporting was executed against the window-2 artifacts before commit, exercising
the missing-floor, missing-transport and missing-point skip paths.

Power caps were reduced on this rig (3080 320->200 W, 5090 525->400 W; verified
power.limit 200/400/200 against default 320/575/320). Same-boot A/B arms share
the caps and are unaffected, but comparisons against pre-change archive numbers
-- the sgl-project#320 Messbuendel tables in particular -- are confounded. The script now
prints the measured caps in its report header and NOTE_515 carries the caveat.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
…rash configuration

The 2026-08-05 GPU window ran two arms. Neither reached a barlink fault, and
both failed for reasons that were in the harness, not the transport. What the
window did establish is that the harness now reproduces the crash boot's
configuration exactly, and what it costs to get there.

Four config gates had to be cleared before the server would start at all:

* --speculative-algorithm NEXTN is rejected under --dcp-size. The alias is
  resolved by the arg hook only AFTER _handle_dcp_validation has already run
  SpeculativeAlgorithm.from_string on the raw string (server_args.py:7817), so
  the enum never sees it. EAGLE is what the crash log's server_args recorded
  anyway.
* spec + DCP is admitted only on the uneven-hybrid weighted path, which needs
  SGLANG_UNEVEN_DCP=1 and SGLANG_UNEVEN_DCP_WEIGHTED=1 (server_args.py:7834).
* rank_tp_ratio must already be non-uniform at validation time, but passing it
  explicitly collides with "--rank-auto-reserve-mib only applies with
  --rank-tp-ratio auto" (server_args.py:9656). auto-performance resolves the
  conflict and DERIVES [27107, 16680, 16680] -- byte-identical to the crash
  boot's recorded rank_gpu_memory_mib and rank_tp_ratio.
* --max-mamba-cache-size 96, production's value.

With those, the boot is faithful: barlink reports ACHIEVED=device for world:0,
tp:0 and dcp:0, the same three groups as the crash log.

The load generator was the second problem and is the reason no arm reached the
barlink question. An unbounded growing conversation per session drove mamba
usage to 1.00 and killed both arms on mamba allocation asserts -- arm 1 on
"Not enough space for mamba ping pong idx" (memory_pool.py:1494) at 8 minutes,
arm 2, with the larger pool, on "Not enough space for mamba cache". The crash
boot ran at mamba usage 0.25-0.28 with 3-4 running requests, so the harness was
measuring a regime the crash never occupied. Each session now recycles its
prefix every six turns to bound the number of distinct cached prefixes, and the
comment states the mamba-usage band the arm has to stay inside to be valid.

Neither mamba assert is a barlink fault. Both are worth reporting separately:
they kill the scheduler with SIGQUIT, and the merged "Mamba slot starvation
must not kill the scheduler" fix does not cover the _alloc_ping_pong_buffer
site.

Rig fact worth keeping, because it silently invalidates any hardcoded mapping:
torch device order is NOT NVML order here. torch 0 is the RTX 5090 while nvml 0
is an RTX 3080. --rank-gpu-id is torch order, which is why 27107 MiB on rank 0
is valid. The harness prints the NVML inventory for identification but derives
nothing from its indices.

Window hygiene: serving was stopped by pgid only, VRAM verified fully reclaimed
(20052/32069/20052 MiB), production rebooted verbatim from
/root/bin/start-serving-30030.sh, health 200, corridor restored to its
pre-window values (1278/3317/2148 MiB). The router on 30099 was never touched.

No barlink fault was observed in either arm: 0 occurrences of "unspecified
launch failure" and 0 of DeviceCollectiveAborted across 202 prefill and 30
decode batches. That is not evidence the sgl-project#583 fix is wrong, and not evidence
the trigger is gone -- the arms simply died before reaching the regime.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
…ach arm prove its own validity

The 2026-08-05 window produced no barlink verdict because the load ran in a
regime the crash never occupied. This rebuilds the load profile around the
variable that actually saturated and adds the instrumentation that would have
caught the mistake in the first minute instead of after two arms.

What consumes mamba state is the number of DISTINCT cached prefixes -- the
mamba radix cache holds one state per cached prefix under
mamba_radix_cache_strategy='extra_buffer'. The old generator gave each session
its own forever-growing conversation, so distinct prefixes grew without bound.
Sessions now cycle a FIXED pool of six long prefixes, which pins the mamba
footprint flat over the soak while each request still carries a large
cached-token count plus a fresh 600-2000 token chunk -- the crash boot's shape.

The band was then measured rather than assumed, from the two logs:

  crash boot 5    mamba median 0.25  max 0.80   running median 3.0  max 4
  rejected arm 2  mamba median 0.18  max 1.00   running median 3.0  max 4

Two corrections follow from that, and both matter:

* The crash boot PEAKED AT 0.80. A safety valve at the band ceiling would have
  throttled exactly the excursions the crash exhibited, and would have flagged
  the crash boot itself as out of regime. The valve therefore sits at
  near-saturation (0.88), not at the band edge.
* The rejected arm's MEDIAN was lower than the crash boot's. Median is not what
  distinguished them -- saturation is. Validity is judged on the median band
  (0.20-0.35), failure on saturation (>= 0.95).

Concurrency was never the problem: both runs sat at running-req median 3.0,
max 4, which already matched the crash.

Each arm now reports its own validity. A sampler reads the server's own log
lines and the run prints MAMBA BAND / REGIME / RUNNING-REQ, so a barlink
verdict from an out-of-regime arm is visibly not trustworthy rather than
silently wrong. Can-fail proof, replaying the verdict logic over the two real
logs: the crash boot scores REGIME OK (median 0.25, peak 0.80) and the rejected
arm scores REGIME FAIL (saturated 1.00).

Adds the `soak` arm (20 min) for the dual-acceptance window: sgl-project#583's abort path
under the crash regime, and sgl-project#581's mamba pool floor live proof in the same run.
The verdict block now reports both. sgl-project#581's half is negative evidence -- neither
"Not enough space for mamba ping pong idx" nor "Not enough space for mamba
cache" may reappear -- plus the regime report showing the run was actually
loaded while staying clear. Pin-budget lines are counted and their ABSENCE is
reported explicitly, so a soak that logged nothing about pins is recorded as
leaving pin behaviour unproven rather than passing by silence.

Not yet merged: fix/mamba-pool-floor-581 is still uncommitted in
/spinning/wt-mamba-floor (8 modified files, 2 new, 0 commits), so there was
nothing to take. The soak needs it -- without the floor the arms die on the
asserts again.

Validation, no GPU: bash -n, embedded-Python AST parse, codespell clean, and
the regime instrument's can-fail proof above.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
…k tree

The soak needs the pool floor: both 2026-08-05 arms died on the mamba
allocation asserts this branch removes, before barlink was ever stressed.
Merged here so one window can carry both acceptances.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
…ry markers verbatim and guard them with a test

The soak reads both acceptances out of the server log by matching fixed
strings, so the strings are now pinned to the sources that must contain them
rather than matched by a loose pattern. Markers taken from ee8d1b5:

  hi_mamba_radix_cache.py  "mamba write-through pin budget reached"
  memory_pool.py           "deferring this batch" (+ mamba_evictable= /
                           mamba_protected= starvation fields)
  mamba_component.py       "skipping this cache insert"
  mamba_pool_floor.py      "pinned checkpoint" (the floor derivation line)

Writing the pins exposed one of my own: "deferring this batch of" is NOT
contiguous in the source -- memory_pool.py wraps the format string between
"batch " and "of %d request(s)". It matches the formatted log line but cannot
be verified against the source, which is precisely how a pin rots unnoticed.
The harness and the test now both use the shorter contiguous fragment, and the
reason is written down at both sites.

test_soak_harness_pins_583.py guards this in three directions:

* every pinned literal must exist verbatim in its source file;
* every `grep -F` in the harness must be a REGISTERED pin -- this is the
  direction that actually rots, where someone adds a marker, never registers
  it, and it quietly stops matching;
* the two death asserts must still EXIST in memory_pool.py. The soak proves
  their absence from a run's log, and you cannot prove the absence of a string
  that no longer exists anywhere -- deleting them would make the acceptance
  pass vacuously.

That third check also settled how to read the sgl-project#581 verdict. Both asserts are
still live `assert` statements (memory_pool.py:1476-1482); sgl-project#581 makes them
UNREACHABLE via the floor, the admission gate and graceful back-off elsewhere,
rather than converting them into warnings. So their appearance in a log remains
a hard failure, and the harness is right to count them that way -- while
"deferring this batch" / "skipping this cache insert" are the GRACEFUL paths
and are reported as back-off, not death.

Pin-budget acceptance is reported honestly in both directions: observing the
budget engage is the proof, and silence is printed as "pin behaviour remains
UNPROVEN by this arm" rather than folded into a pass.

Soak config verified against the new parse-time floor refusal:
--max-mamba-cache-size 96 with --max-running-requests 4 gives a hard floor of
20 (4 x (1 active + 2 ping-pong + 1 donation + 1 pinned checkpoint)), so the
soak boots with 76 slots of headroom and does not trip the refusal.

Tests (no GPU): test_mamba_pool_floor.py + test_barlink_device_abort_583.py
38 passed on the merged tree; test_soak_harness_pins_583.py 4 passed / 9
subtests. ruff and codespell clean. A full test/registered/unit/mem_cache
sweep is running; test_hicache_nixl_storage.py fails to collect there for a
missing optional 'nixl' module, which is environmental and untouched by sgl-project#581.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
…ce instead of guessing it

Standing order: the harness finds its own values. An env knob is not an
improvement over a hand edit -- it is the same guess with better ergonomics,
and hand-picked values produced both failed arms on 2026-08-05.

scripts/repro/derive_soak_target.py parses a crash-reference log for the
ABSOLUTE regime and solves the load parameters from it. The harness calls it
by default; MAMBA_CACHE / PREFIX_POOL / SESSIONS survive only as documented
overrides.

THE RULE IT MECHANIZES. A reference's `mamba usage` means nothing except
against that reference's own `max_mamba_cache_size`. Soak arm 1 ran fraction
0.17 against the crash's 0.25 and was rejected as "too light" while its
ABSOLUTE occupancy was identical -- ~16 slots either way, 96 vs 64 denominator.
Following the fraction would have raised real load ABOVE the reference while
the displayed number moved toward it. The deriver converts to absolute slots at
parse time using the reference's own denominator and targets absolutes only. A
log without a denominator is a hard error, never a guess.

WHY NOT READ `mamba num`, WHICH IS ALREADY ABSOLUTE. It is printed on DECODE
lines only. In boot5 that is 14 samples, all 16, stdev 1.0 -- a flat regime --
while the actual peak of 51 slots occurs on PREFILL lines carrying no
`mamba num` at all. Reading it alone understates the peak by more than 3x,
which is precisely why soak arm 1 came out min=median=max. The series is
reconstructed from usage x denominator for full coverage (222 samples) and
cross-checked against `mamba num` where both appear on one line; the observed
disagreement is 0.16 slots.

SOLVED ON THE PEAK, NOT THE MEDIAN. The prefix pool is a ceiling on distinct
cached prefixes, so it sets the PEAK; the median falls out below it as prefixes
are evicted between bursts. Solving on the median caps the pool too low to ever
reach the reference's excursions.

  boot5 (barlink crash): median 16, peak 51, spikiness 3.19x, running median 3
    -> MAMBA_CACHE=64  SESSIONS=3  PREFIX_POOL=31 (51 - 4x5 structural)

REFERENCE SELECTION IS ITSELF GUARDED. The default is boot5, the BARLINK crash
-- not boot6, which is the mamba ping-pong crash and SATURATED its pool
(64/64). Targeting boot6 would build a harness that deliberately saturates,
which this harness's own regime gate scores REGIME FAIL. The deriver prints a
warning whenever a reference saturates, so "aim at the most recent crash log"
fails loudly instead of quietly producing an unusable arm.

Also fixes a truncation the new block exposed: the NVML section used
`tee "$LOG"` without -a and would have erased the derivation record from the
log -- the first thing a reader of any verdict needs.

Hand-derived vs auto-derived, for the record: soak arm 2 (running now) was
aimed by hand at MAMBA_CACHE=64 / PREFIX_POOL=16 / SESSIONS=4. The deriver
agrees on the denominator (64, the correction that matters) but calls for
PREFIX_POOL=31 and SESSIONS=3, so arm 2 is expected to sit at the reference
median with a lower peak. Any rerun uses the auto-derived targeting.

Tests (no GPU): test_derive_soak_target_583.py 12 passed --
denominator-from-reference, peak-not-decode-only, cross-check tightness,
peak-based pool solve, the boot5 regression pin (64/31/3), floor clearance,
saturation detection on boot6, and the hard error on a missing denominator.
Plus test_soak_harness_pins_583.py 4 passed / 9 subtests. ruff and codespell
clean; bash -n and an embedded-Python AST parse on the harness.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
…ation test prose

No behaviour change; keeps codespell clean so real hits stay visible.
test_derive_soak_target_583.py 12 passed.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
…sions is half a regime

Gate gap found by soak arm 2 in the 2026-08-05 window. That arm matched the
reference MEDIAN exactly (0.25) and was scored REGIME OK -- while its peak was
also 0.25 against the reference's 0.80, i.e. spikiness 1.00x against the
reference's 3.19x. The gate checked median and saturation and never checked
peak fidelity, so an arm that reproduced half the regime was reported as having
reproduced it. That matters because the excursion half is the part that
plausibly stresses the transport: a null barlink verdict from a dead-flat arm
would have been reported as if the crash conditions had been met.

New REGIME PARTIAL verdict fires when the median is in band but the peak falls
below 60% of the reference peak. It names the observed and reference spikiness
and points at derive_soak_target.py, which computes the prefix pool from the
reference PEAK precisely so this does not recur.

Replay over all four recorded runs, which is the can-fail proof:

  crash boot5            REGIME OK       med 0.25  peak 0.80
  baseline arm2          REGIME FAIL     saturated 1.00
  soak arm1              WARNING light   med 0.17
  soak arm2              REGIME PARTIAL  med 0.25 ok, peak 0.25, 1.00x vs 3.20x

The reference fractions are supplied by the solver rather than hardcoded, and
are only ever compared against the arm's own denominator -- the same rule the
solver enforces.

Tests: test_soak_harness_pins_583.py 6 passed / 9 subtests, including that the
PARTIAL branch exists and is evaluated BEFORE the OK branch (ordering is the
whole defect: OK must be the last resort, not the first match). Full sgl-project#583 suite
33 passed / 12 subtests. ruff and codespell clean, bash -n on the harness.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
…IRMED, with a regime split

TRANSPORT=barlink (production's transport), interleaved E,G,E,G,E,G, ms per
fixed unit of work, own barlink eager floor. Artifacts in
/spinning/gpu-battery-results/2026-08-05_prefill_graphs_w4{,det}/.

  point                 eager      graphs     paired delta   floor    verdict
  1900 single-stream    1196.3 ms  1320.6 ms  +10.25% slower  0.12%   REPORTABLE
  256 x4 concurrent      230.1 ms   219.0 ms   -4.62% faster  1.99%   REPORTABLE

All three pairs agree in sign at both points and both deltas clear their own
A-vs-A floor by a wide margin (85x and 2.3x).

The user's hypothesis holds, and only in the regime it predicted: a captured
prefill pays where the work is launch-train bound (many small prefills, several
in flight) and loses where it is GEMM bound. With barlink's device-side
collectives inside the captured graph the concurrent short-prompt point turns a
real -4.6%, while the single long prefill has no launch overhead to recover and
pays the padding and capture cost instead. The +10.25% at 1900 is larger than
bucket padding alone explains (~3.7% for a 1974-token prefill in the 2048
bucket), so something else in the captured long-prefill path costs on top; not
localised here.

Consequence for the recommendation, now rescoped: the answer is not on/off but
workload-dependent, and a blanket flag is the wrong shape. can_run_graph is
already a per-batch decision (prefill_cuda_graph_runner.py:615), which is where
a regime gate would live. Enabling globally would trade 10% on long prefills
for 4.6% on short concurrent ones, and production's traffic mix decides whether
that nets out -- unmeasured.

DETERMINISM ARMS booted first try at the derived RESERVE 7164,5864,5864,
validating the §5a ledger derivation end to end -- right vector, no padding, no
retry. Same directional split (+7.44% / -6.31%) but REPS=1, so no floor and
therefore NO VERDICT on those two numbers, as the reporter states. Determinism
did NOT close the content divergence: 1/8 with it, 1/8 without.

CONTENT under barlink is 1/8 text (char 124) against 4/8 under NCCL, but the
gate ran on pair 1 only, so there is NO barlink eager-vs-eager content floor.
The 1/8 is not yet distinguishable from barlink boot noise, and is recorded
that way rather than as an improvement.

SOAK: 8 barlink arms booted and completed with no fault -- a clean data point
for sgl-project#583.

Serving restored pgid 231310, health 200 after 40s, corridor 1708/4131/2578 MiB,
smoke content-sane.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
…t skipped

Crashes 9, 10 and 11 (2026-08-05) all ended in the same unhelpful shape:
ranks 0 and 1 standing in one collective, rank 2 standing in a different
one, and nothing anywhere saying which collective rank 2 had SKIPPED. The
tracebacks report where each rank was STANDING, which is downstream of the
defect. The per-rank counts already existed -- the prefill line prints
"tp.all_reduce 243.4/129x" -- they were simply never compared ACROSS ranks.

This adds that comparison as a standing instrument. It is measurement, not
mitigation: it changes nothing about what the ranks do and cannot repair a
divergence. It reports, so the next crash arrives with its cause attached.

Counting: one dict increment per collective, at the four GroupCoordinator
dispatch sites, keyed by the family names the collective clock already
derives (tp.all_reduce, dcp.all_gather, ...). Placed AFTER the clock's
re-entry guard so each collective is counted exactly once. Guarded by a
module-level bool resolved at import, so the hot path costs one bool read
plus one dict increment.

Two reporting paths, for two different failure states:

1. Scheduler._census_tick -- the detector. Runs on tp_cpu_group (gloo),
   NEVER the device/BAR1 path, so a wedged device group cannot silence the
   instrument meant to explain it. Rides the same unconditional
   once-per-iteration point as the sgl-project#603 reduce, so its round counter is
   replicated and the cadence gate opens on the same round for every rank.
   Default every 50 iterations (~2 s in production), an order of magnitude
   inside the ~30 s spin deadline: the divergence is named while the ranks
   are still healthy, long before the abort.

2. format_local_census, called from barlink_bar1.check_aborted before it
   raises -- the wedge-proof half. Takes NO collective on purpose: by abort
   time a peer is very likely dead (in all three crashes the third rank
   aborted ~30 s after the first two), so a cross-rank exchange would hang
   exactly when the evidence is wanted. Each rank logs its own counts; the
   three lines are diffed afterwards.

Warn-never-raise throughout: a failed comparison is logged and dropped, and
the abort-path dump can never mask the real error.

ARMED BY DEFAULT, like the sgl-project#605 flight-recorder marks -- an instrument that
must be switched on before a crash explains no crashes.
SGLANG_COLLECTIVE_CENSUS=0 disables counting;
SGLANG_COLLECTIVE_CENSUS_INTERVAL tunes or (<=0) disables only the periodic
comparison, leaving the abort dump armed.

Tests: test/registered/unit/distributed/test_collective_census_583.py
Hermetic (no CUDA, no process group, no model). The injected divergence is
crashes 9/11 in numbers: [129, 129, 128] on tp.all_reduce.
  * 14 passed
  * can-fail proof: breaking only the count comparison in _diff turns 4 of
    them red
  * pins the missing-key case (a rank that skipped a family's ONLY
    collective has no key at all -> must read as zero, not drop the family)
  * pins armed-by-default, the kill switch, the replicated cadence gate,
    and warn-never-raise on a wedged/absent group
  * regression: census + retract-583 + 603 + retract-fcfs = 35 passed
  * ruff: 0 new findings (parallel_state 0/0, barlink_bar1 2/2,
    scheduler 89/89, schedule_batch 40/40); new files ruff-clean and
    formatted; codespell clean

Crash 11 (21:26, boot 11, tree WITHOUT the retraction fix) is byte-for-byte
crash 9's shape and NOT crash 10's: all three ranks are inside run_batch
(scheduler.py:2032), ranks 0/1 at communicator.py:1179 and rank 2 at
linear.py:2340, with check windows 1 and 4. So the `if batch:` fork is not
involved there, and crashes 9/11 are a site INSIDE the forward, distinct
from the retraction site closed for crash 10. The census is what will name
it.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
…tbeat)

Boot 13 ran the census and its log contained NO census line at all. That is
not reassuring, it is unfalsifiable: an instrument that is silent when
healthy AND silent when unwired cannot be told apart from a dead one, and
"armed, counts uniform" reads exactly like "import path never reached on
this flagset" (the sgl-project#380 test-honesty trap).

Two additions, both cheap:

* ONE INFO line per scheduler rank, the first time the tick runs:
  "collective census armed (rank N): interval=50 iterations, heartbeat=...,
  families tracked=K (...)". Emitted from the TICK rather than at import on
  purpose -- an import-time line proves only that the module loaded, while
  this one proves the wired production path is actually reached. Guarded by
  a flag so it appears once, not once per iteration.

* A coarse local heartbeat (default every 10000 iterations, INFO, carrying
  this rank's snapshot) so a long-running boot keeps evidence the detector
  is still ticking. It rides the round counter that is incremented anyway,
  so the hot path is one extra modulo. SGLANG_COLLECTIVE_CENSUS_HEARTBEAT
  tunes it; <=0 disables the heartbeat only.

Tests: 4 added to test_collective_census_583.py (17 total).
  * arming line emitted EXACTLY once, naming rank, interval, family count
  * heartbeat cadence rides the replicated counter
  * default coarse, overridable, junk-tolerant
  * can-fail proof: removing the once-only guard turns the arming test red
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
…etero gate)

`apply_flashinfer_allreduce_fusion` gated on
`is_sm90_supported() or is_sm100_supported()` -- a query against THIS RANK's
device. That value reaches `should_fuse_mlp_allreduce_with_next_layer`, and
a True there makes the layer SKIP `postprocess_layer`, i.e. it REMOVES a
collective from that layer.

So it is a rank-local predicate deciding whether a group collective is
entered -- the sgl-project#583 family. The ranks that fuse issue one fewer all-reduce
per layer than the ranks that do not. And because the mispaired payloads
are frequently the same size, the ranks complete each other's collectives
and compute garbage SILENTLY before anything hangs; the deadline abort is
the lucky case, not the normal one.

Dormant on a homogeneous cluster, and dormant on this rig (sm86 + sm120:
both answer False, which is why it never fired here). LIVE on any
heterogeneous TP group straddling the sm90 boundary -- an sm89 card beside
an sm90 one is enough. Closing it now, while the context is hot, rather
than waiting for the rig that has those cards.

Fix, same shape and same reasoning as `decide_spec_kernel_backend` in
speculative/eagle_utils.py: every rank contributes its own capability, the
MINIMUM rules on the gloo CPU group, and if one rank lacks the arch then NO
rank fuses. Never a mixture. Decided once in `init_torch_distributed`,
where every rank arrives exactly once, the TP group exists and no CUDA
graph is capturing. A conflicting second decision is refused rather than
silently winning, because two disagreeing decisions in one process mean the
collective ran twice with different inputs. Undecided falls back to the
LOCAL value, which is correct where it is reachable (single rank, unit
tests) and is not a silent per-rank restoration on a real TP boot.

Behaviour on this rig is unchanged: both cards answer False, the group MIN
is False, fusion stays off exactly as before.

Tests: test/registered/unit/distributed/test_ar_fusion_arch_583.py, 8 tests.
  * falsifier: a mixed group [capable, incapable] must fuse on NEITHER rank,
    with the pre-fix per-rank predicate pinned as genuinely divergent first
  * the gate can still answer YES (a uniformly capable group fuses), so the
    falsifier cannot pass against something that always disables fusion
  * conflicting second decision refused; undecided falls back to local, not
    to a silent False; an unavailable group does not block boot
  * can-fail proof: reverting the predicate to the local value turns the
    mixed-group falsifier red
  * regression across all sgl-project#583 work: 46 passed
  * ruff: 0 new findings (communicator 0/0, model_runner 5/5, scheduler
    89/89); new test ruff-clean and formatted; codespell clean
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
…by line number

REGRESSION I CAUSED, and the fragility that let one line do so much damage.

_bar1_marker_source.py parses the real emitter sources with `ast` and pulls
the format strings out of them -- a good design, built precisely so BAR1 log
fixtures cannot drift into hand-typed copies (sgl-project#315, the sgl-project#380 family). But it
located each call by ABSOLUTE LINE NUMBER (LINE_BAR1_SETUP = 2236, ...).

That coupled the module to the LAYOUT of files it does not own. Adding the
collective-census counter to parallel_state.py shifted line 787, the AST
lookup found no call there, and the assertion fired at IMPORT -- taking the
whole BAR1 marker suite down at COLLECTION (test_gpu_battery_checks_bar1.py
+ test_bar1_marker_coupling.py, 136 tests).

Note the actual trigger was the census BUMPS in parallel_state.py, not the
abort-handler edit in barlink_bar1.py: the failing assertion names
parallel_state.py:787.

A line number is the same fragility class as a hand-typed fixture string: a
second, unmaintained copy of a fact the source already states. So the
anchors are now the MARKER TEXT itself (MARK_* instead of LINE_*), resolved
by `_call_with_marker`: the unique call whose first argument contains the
fragment. For the exception class the anchor is its DOCSTRING, not its name
-- anchoring on the name would make `bar1_unavailable_class_name` a
tautology, when its whole job is to report a rename to the consumers.

This is strictly better coupling, not weaker: it survives every edit that
does not touch the marker, and still fails loudly on the edit that does --
which is the drift the module exists to catch. Ambiguity (a marker matching
two calls) is a loud error naming both, never a silent first-match.

Tests:
  * test_gpu_battery_checks_bar1.py + test_bar1_marker_coupling.py: 136
    passed (were: collection error)
  * robustness proof: inserting 40 unrelated lines above the old pin in
    parallel_state.py -- the exact edit class that broke it -- still 136
    passed
  * can-fail proof: renaming the emitted marker ("barlink enabled for
    group " -> "barlink ACTIVATED for group ") fails loudly with a message
    naming the file, the marker and both things to update
  * WHOLE test/registered/unit/distributed directory: 2370 collected, 0
    collection errors, 2358 passed / 12 skipped / 726 subtests passed
  * the remaining 3 subtest failures (test_dcp_context_ceiling.py,
    '_PoolRowStub' object has no attribute 'server_args') are PRE-EXISTING,
    not from this branch: verified identical at 38e7c69 in a throwaway
    worktree (3 failed, 17 passed there too). The call site is
    flashinfer_backend.py:691, which this branch does not touch.

Process note: this slipped through because I ran only my own test files.
The directory-wide run is the check that would have caught it, and it is
what I ran here.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
…rank)

MY BUG, and the worst possible one to have: the instrument built so that
silence could be trusted was itself silently dead.

`Scheduler._census_tick` read `self.tp_rank`. That attribute does not
exist -- the Scheduler keeps its parallel identity on the ParallelState
wrapper, `self.ps.tp_rank`. So every tick raised AttributeError into the
tick's own warn-never-raise guard: nothing was counted, nothing compared,
and the abort-time dump would have been empty. Boot 13 logged the skip
2661 times; boot 14 had 693 within minutes.

The unit tests missed it because they drove `CollectiveCensus` directly and
STUBBED the scheduler surface. A stub that grants the attribute proves only
that the stub has it. Desk-written-never-executed, exactly.

Three fixes:

1. `self.ps.tp_rank`, the real path. The same mistake had already been made
   a second time in the (unpushed) fallback hardening, which used
   `self.tp_size`; that is corrected here too, before it could ship.

2. The skip warning is emitted ONCE, not per tick. A tick that cannot run
   usually cannot run ever, so per-tick warning buries the first line under
   thousands of copies.

3. The arming line now prints only AFTER a tick has completed without
   raising. Previously it was announced first, so a dead census could still
   have certified itself as armed -- the line exists to make the
   instrument's liveness checkable, and announcing it before the work would
   certify a corpse. This is the same lesson the arming line was added for,
   applied one level deeper.

Tests: test/registered/unit/distributed/test_census_attribute_surface_583.py
Two independent styles, because the stub is what failed here:
  * STRUCTURAL -- parses the shipped scheduler.py, asserts the identity
    readers go through `self.ps` and that `self.tp_rank`/`self.tp_size`
    appear nowhere in them, and that every `self.ps.X` they use is a real
    ParallelState field. A stub cannot satisfy this; it reads the source.
  * EXECUTION -- binds the REAL `_census_tick` to an object carrying only
    the real attribute surface (ParallelState built from its OWN
    annotations, so a new field cannot quietly turn this back into a stub)
    and asserts the tick does not take its skip path.
  * plus: the skip warning fires once in 500 calls; a broken tick never
    announces itself as armed.
  * can-fail proof: restoring `self.tp_rank` fails 3 of them
    (structural, positive-read, and arming-after-failure).
  * 9 passed; census suites 43 passed together.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
…o local

sgl-project#603 and the sgl-project#583 retraction fix both decide from a once-per-iteration
MIN-reduced pool headroom instead of the rank-local `available_size()`.
Both kept a fallback for "reduced value not available":

  * `Scheduler.uniform_min_avail` -- getattr(self, "_uniform_min_avail",
    None), returning the local pool when the reduce had not run
  * `ScheduleBatch.decode_mem_avail` -- returning the local pool when
    `uniform_avail_floor` was unset

Justified as "correct for a single rank and for tests", which is true and
beside the point: on a MULTI-RANK boot they hand back exactly the
rank-local predicate both fixes removed. Any path reaching a decode-mem
decision without setting the floor silently restores the desync, with the
protection still plainly visible in the source. That is the
getattr-default trap (sgl-project#606) -- a guard that reads as armed and is absent at
runtime, the same shape as a fixture string nobody ever printed (sgl-project#380).

Fix: single rank keeps the local value (nothing to diverge from); a group
raises with a message naming the ordering that was violated. The world-size
probe (`_group_world_size`) never raises -- a probe that could fail would
just move the silent-default problem one level down.

Tests: test/registered/unit/distributed/test_uniform_floor_fallbacks_583.py
  * falsifier: pins the pre-fix silent-local behaviour first, then asserts
    both paths refuse on a group
  * single-rank behaviour preserved on both paths (else this is a
    regression, not a fix)
  * a present reduced value always wins; a ZERO floor is honoured and not
    treated as missing (`if not floor` would be the same bug in disguise)
  * 10 passed; can-fail proof: reverting either guard turns the two
    refusal tests red

Also updates the sgl-project#603 and sgl-project#583 stubs to model `self.ps.tp_size` rather than
inventing a flat `self.tp_size` -- a stub inventing a surface the real class
lacks is precisely what let the census ship dead.

Full run: test/registered/unit/distributed + unit/managers = 2875 passed,
30 skipped, 735 subtests passed. Remaining failures are pre-existing and
not from this branch: 3 subtests in test_dcp_context_ceiling.py (verified
identical at 38e7c69) and test_bar1_host_cleanup.py's pgrep self-match
test, which passes in isolation (28/28) and is sensitive to concurrently
running processes. Directory ruff count unchanged at 4.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
Boots 13, 14 and 16 all died with a message that reads as a contradiction:

    alloc_req_slots runs out of memory. ... available_size()=4, num_reqs=1

Four request slots free, one request wanted, and it raised anyway. The same
line appeared with avail=2 (boot 13) and avail=3 (boot 16).

`available_size()` counts REQUEST slots only. On a HybridReqToTokenPool the
allocation also needs a mamba state -- and, with the extra buffer, a
ping-pong slot -- and `alloc` returns None if EITHER is unavailable. In all
three crashes the mamba pool was full (96/96 on boot 14). So the message
reported the one pool that was fine and said nothing about the one that was
not.

A fail-loud path that names the wrong resource is worse than a quiet one:
it does not merely fail to help, it points away from the cause. This cost
real diagnosis time on three separate boots.

The message now carries mamba_available / mamba_schedulable / mamba_total
whenever the pool is hybrid, labels the request-slot count as such so it
cannot be misread as "the" capacity, and names the two places to look
(--max-mamba-cache-size, or mamba slots held by unevictable cached states).
The diagnostic is wrapped so that a broken allocator cannot replace the real
OOM with its own error.

Tests: test/registered/unit/mem_cache/test_alloc_req_slots_names_the_pool_583.py
  * falsifier drives boot 14's exact numbers (avail=4, num_reqs=1,
    mamba 0/96) and asserts the mamba figures appear
  * pins that the non-hybrid message grows no mamba noise
  * pins that a throwing mamba allocator still surfaces the real OOM
  * 5 passed; can-fail proof: restoring the old message fails 4 of them

Pre-existing failures, NOT from this branch (verified by running the
directory with this change reverted): test/registered/unit/mem_cache is at
937 failures on the integration tree already, and
test_hicache_nixl_storage.py fails collection for a missing optional `nixl`
module. Adding this change leaves both counts unchanged.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 6, 2026
…ilies the desync hid in

The 2026-08-06 05:53:59 crash was a rank-arrival desync, and the census was
armed and running through all of it. It reported nothing. Not because it
failed -- because every family it tracked genuinely agreed: dcp.all_gather
10176x, dcp.all_reduce 2864x, tp.all_gather 580x, tp.all_reduce 36824x,
identical on rank 0 and rank 1. The family that diverged was necessarily one
nobody counted, and the abort named an 8-byte collective served by the a2a
kernel. Neither broadcast nor all_to_all was censused at all.

Agreement across every family you count is not health when the failure is in
a family you do not count.

WHAT CHANGED

- collective_clock_families() returns two more names per group,
  "<group>.broadcast" and "<group>.all_to_all". This is the one place a
  group's families are named and both instruments read it. The two new names
  are CENSUS-ONLY: they carry no _COLLECTIVE_CLOCK span, because arming a
  span is a separate decision with its own cost and its own log-line surface.
  Naming a family there is not a claim that it is timed, and the docstring
  now says so.
- GroupCoordinator.broadcast and .all_to_all_single bump their family, at the
  same seam and under the same placement rule as the existing four (after the
  clock's re-entry guard, before the world_size==1 bypass), so a collective is
  counted exactly once. all_to_all_single_v shares the all_to_all family: a
  rank that skips an uneven a2a has skipped an a2a. No bump was added to
  _all_to_all_single -- the public seam already counted the call that routes
  through the registered op, and bumping both would double count.
- CollectiveCensus.declare_families(): the census payload width is now fixed
  by replicated config instead of by rank-local state (sgl-project#610). snapshot() is
  what all_gather_object packs, so the KEY SET is the payload width; left to
  grow on first use it is a function of what a rank has happened to execute,
  which is exactly the rank-local state this instrument exists to catch
  diverging. Group construction is replicated, so each group declares its
  families at construction and every rank packs the same keys from the first
  comparison onward. Second effect worth having: a declared family that never
  fires is visible as 0x instead of absent, so "never counted" and "not
  instrumented" stop looking alike.

Mechanics are unchanged otherwise: same rank-cumulative dict, same gloo
all_gather_object cross-check on tp_cpu_group, same union-based differ, same
warn-never-raise. Hot-path cost is one further dict increment on two seams,
behind the existing module-level bool.

TESTS

test/registered/unit/distributed/test_collective_census_broadcast_families_583.py,
12 cases, hermetic (CUDA_VISIBLE_DEVICES=99, no torch.distributed init, no
group, no CUDA). They drive the REAL dispatch sites and the REAL differ, so
they fail on a tree missing the bump rather than on a tree where a string
changed. The harness is deliberately arity-tolerant about the family tuple so
the can-fail lands on behaviour, not on fixture construction.

  12 passed here.
  11 of 12 FAILED on bccb87b with the same file. The headline case,
  test_a_broadcast_desync_is_named (129 vs 128 broadcasts across two ranks),
  fails there with "'tp.broadcast' not found in set()" -- the differ returns
  an EMPTY list, which is precisely "the divergence passes silently".
  test_an_all_to_all_desync_is_named fails the same way ([] != ['tp.all_to_all']).
  The 1 case that passes on baseline is
  test_agreement_on_the_new_families_stays_silent, which asserts an empty diff
  when the counts AGREE; it is the no-cry-wolf guard, not a can-fail case, and
  it passes on baseline trivially because nothing is counted there.

No regressions: test/registered/unit/distributed -> 13 failed / 2404 passed
here versus 13 failed / 2392 passed on bccb87b. The failing set is
IDENTICAL (set difference empty); the 13 are pre-existing and unrelated
(test_uniform_decode_mem_603 and siblings, _FakeScheduler fixtures). The +12
is exactly this commit's new tests.
test/registered/unit/managers/test_rank_prefill_log.py: 32 passed, 4 skipped.

test_rank_prefill_log.py's family-tuple pin was updated to the six names and
now also asserts dcp.broadcast; its sibling that pins the four CLOCK dispatch
sites is deliberately left at four, which is what keeps the census-only tail
honest.

ruff and codespell clean on all changed files.

Out of scope, still a boot ticket: the PeerWatchdog sequence-number liveness
mechanism that would catch a peer that is alive but not arriving.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 6, 2026
Two instruments armed during the wedged soak agree the ranks were in step: the
sgl-project#583 census reported no divergence and its local dumps are byte-identical
between ranks, and the barlink 1 Hz launch sampler shows all three ranks
carrying the same last-op records at wedge time (all_gather 192512,
broadcast 32, broadcast 24, all_gather 192512).

So the wedge is neither a pairing shift nor a count divergence, and that line
of inquiry is closed. What remains: if every rank enqueued the same collective
and all three still block in the next host sync, the queue is busy rather than
starved, and Bar1CollectiveAborted is a consequence of a peer's deadline
expiring against a GPU that is still executing. That points at the device work
just before the sync (create_flashinfer_kv_indices_triton, and the
torch.empty(total) ahead of it) and at a pathological total / paged_kernel_lens
arriving from replicated scheduler state -- which would hit all ranks at once,
as observed. Recorded as a hypothesis with the measurement that would settle
it, not as a finding.

Also records why the per-site D2H removal was dropped: owner.py has a second
unavoidable sync at line 566 (boolean-mask indexing calls nonzero), so
removing the .item() cannot make the function sync-free; and the plumbing that
would have fed total_tokens derived it from extend_prefix_lens_cpu, which is
not guaranteed on every model -- a model-dependent fix in a model-independent
layer, where a wrong total silently mis-sizes an attention index buffer.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 6, 2026
Every host-visible signal says the ranks agree during the wedge: sgl-project#583 census
counts match, the launch sampler shows identical last ops and identical
captured/eager flags, and the host kernel logs zero Xid, zero PCIe AER, zero
IOMMU faults and zero GPU resets across 23 h of uptime. Meanwhile all three
cards sit at 100% SM occupancy with 0% memory utilisation and power far below
cap -- spinning, not computing.

The one thing nothing reads is the device flag state the spin waits on, which
is where the remaining explanations live (generation mismatch, or a flag
written but never observed). This adds that read to the abort path only.

Safe precisely there, and nowhere else: the 1 Hz launch sampler refuses device
reads because it runs while a collective may be in flight and would queue
behind the wedged spin; by abort time the kernel has already taken its abort
path, so nothing is left to queue behind. Warn-never-raise, like the census
dump beside it, so the instrument can never mask the real error.

Reports the first dword of each 256-byte flag line; diffing the three ranks'
snapshots names a generation mismatch. Zero cost until an abort fires.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 6, 2026
The abort-time flag snapshot fired on its first real wedge and produced the
number no other instrument could read: rank 0's highest flag generation is
1190714, rank 1's is 1190715. The ranks are out of step by exactly one
collective generation. Section 15 listed this as one of two untestable
candidates; it is now measured.

It also explains why every earlier instrument was silent. The sgl-project#583 census
counts at the Python GroupCoordinator layer and the launch sampler compares op
name and size - both identical across a one-generation skew. The skew lives in
the device-side BAR1 generation counter, which neither reads. The objection
that the pairing-shift mechanism owed an explanation for the census silence is
therefore answered.

And it unifies the two families: the index assert was a pairing shift of
exactly one collective (accept_index receiving predict, same wire size); the
wedge is a skew of exactly one generation. Same distance, and the symptom
depends only on whether the mismatched pair happens to be the same size -
silent substitution if yes, spin to the cycle cap if no. The accept-broadcast
fusion removed the consequence for one pair, not the skew, which is why the
wedge survives it.

Still unknown: which collective creates the skew. Recorded, with the limitation
that rank 2 died before emitting its snapshot, and a correction to the previous
commit's claim - this wedge happened under agent-only load, not saturation.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 8, 2026
…e, never __trap

The last live instance of the sgl-project#583 context-killing mechanism: wait_ge in
the host-staged transport still called __trap() on spin expiry, which
destroys the CUDA context and surfaces as sticky 'unspecified launch
failure' at whatever unrelated call site runs next. Unreachable in every
production configuration (host is explicitly-selected, no-fallback,
bar1 pinned) — but a landmine for any host-transport measurement boot.

Ported the device transport's sgl-project#583 fix honestly, not minimally:
- wait_ge returns bool and writes a per-site code into the abort word on
  expiry (put reuse-guard=1, reduce peer-wait=2, copyout wait=3); the
  BARLINK_HOST_TRAP macro and its hipify plumbing are deleted.
- Block-uniform exit: shared abortS written by the spinning thread 0,
  __syncthreads(), all threads return together before any payload loop,
  fence, or last_block barrier — verified at all three sites. Terminal-
  abort contract documented: cross-block skew and stale blk_ctr are
  acceptable residue of a process the host is about to kill by raise.
- The transport's THREE sequence-counter families (collective, per-peer
  send, per-peer recv) are unified into one zero-initialized backing
  tensor with 2-word rows — the spec's naive grow-to-2 would have made
  the p2p kernels' abort store corrupt a NEIGHBOR's sequence number, and
  the unified shape makes check_aborted a single read over every abort
  word the transport can produce.
- HostCollectiveAborted mirrors DeviceCollectiveAborted (rank/world/
  code/where, garbage-results contract) and appends the sgl-project#650 peer
  statement. check_aborted is a plain in-line read (documented choice:
  this is a measurement vehicle, not the bs=1 hot path) wired twice:
  the duck-typed _after_transport seam (sgl-project#431) for eager collectives,
  and abort-gate registration for replay-only collectives — host is
  capturable, so a graph-resident collective never reaches the seam
  (the same gap sgl-project#583 closed for the device transport).
- The load-bearing reachability claim is pinned: 'host' is in the
  no-fallback set via the real parallel_state lookup, and the dispatch
  seam is proven transport-stable mid-run — an explicit
  SGLANG_BARLINK_TRANSPORT=host boot is the only way this code runs.

Falsifiers, red-first (executed against the pre-fix source inside the
test): old wait_ge provably trapped and returned void, old seq tensor
had no abort-word room; fixed source has zero traps, bool returns, all
three consuming sites, distinct codes. nvcc compile smoke rc=0 (sm_86).
GPU-gated on-card falsifier written and documented (put-kernel forced
timeout must set the word, leave the context ALIVE, and raise named)
— skips at the desk by design.

Counting basis: 35 passed + 1 GPU-skip + 19 subtests in the new file;
full battery with abort/stub-drift/sentinel regression suites 81 passed
+ 1 skipped + 19 subtests (97.9 s, my rerun); ruff and imports clean.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 12, 2026
…tion alignment

C30 was booked as "--enable-deterministic-inference and
--enable-kv-session-offload cannot both be on". That diagnosis is wrong on
every load-bearing detail, and wiring the refusal it implied would have
rejected a working configuration while leaving the real trap armed
everywhere else.

THE ACTUAL MECHANISM, verified in source. PrefillAdder.add_one_req's chunked
branch (schedule_policy.py) aligns the chunk it is about to take:

    trunc_len = self.rem_chunk_tokens // self.page_size * self.page_size
    if truncation_align_size is not None:
        if trunc_len < truncation_align_size:
            return AddReqResult.OTHER

rem_chunk_tokens is bounded above by --chunked-prefill-size, so a chunk
budget below one alignment unit refuses EVERY request longer than the budget,
on every iteration, forever. The scheduler's admission loop breaks on any
non-CONTINUE verdict, so one such request at the head of the FCFS queue
blocks the queue behind it, can_run_list stays empty and no batch is ever
built. The instance boots, prints "fired up and ready to roll", serves its
warmup prefills (short enough to take the non-chunked branch) and then admits
nothing -- the worst failure shape there is.

kv-session-offload's ONLY role is that it forces the flashinfer backend. The
refusing predicate is a third variable, --chunked-prefill-size, which the
wedged boot had at 256 against an alignment of 4096. The same two flags with
a chunk budget of 4096 serve normally, so the flag-pair refusal the register
called for would have rejected a configuration that works.

Two claims in the C30 entry do not survive re-reading its own evidence, and
both are corrected in the register:
* there is NO rank divergence. probe_v8.log prints the sgl-project#583 collective census
  from both ranks throughout the wedge and the two are IDENTICAL (all_reduce
  1862x on each). Frozen because no forward ran, not because a rank was stuck.
  `num_queue_reqs 0` is a 30 s-cadence idle gauge sampled after the client had
  already disconnected.
* the disabled radix cache is a red herring. ChunkCache inherits
  evictable_size() -> 0 (never None, never raising) and _tree_evictable_size()
  already tolerates it; losing the radix cache costs restore headroom, it does
  not gate admission.

THE ALIGNMENT HAS TWO INDEPENDENT SOURCES and either alone arms the trap, so
the guard covers both: --enable-deterministic-inference on flashinfer/triton
(align 4096 by default), and --mamba-checkpoint-interval, which sets the
alignment on its own with no deterministic inference anywhere and is lcm-ed
into it when both are present. The check therefore sits in
init_deterministic_inference_config AFTER the lcm -- the only point where the
final alignment exists; server_args cannot see it without restating the lcm.

Side-finding, fixed in the same commit: --mamba-checkpoint-interval's own help
recommended "2048", which silently wedges any boot whose chunked-prefill-size
is below 2048 -- including this rig's ship config at 512. The help now states
the constraint and the guard enforces it.

INERT ON THE SHIP CONFIG, verified against the live process rather than
assumed: /proc/<pid>/cmdline shows --chunked-prefill-size 512, no
--mamba-checkpoint-interval and no --enable-deterministic-inference, and the
helper returns None for (512, page 1, align None).

RED-FIRST, EXECUTED NOT ASSERTED, and split so both halves are visible on the
unfixed tree. The guard import is deliberately made INSIDE the guard tests so
collection still succeeds without it. Reverse-applying the source patch:
the MECHANISM test PASSES (the trap is real and reachable through the real
add_one_req) while the WIRING test fails behaviourally -- "the scheduler
accepted --enable-deterministic-inference on flashinfer with
--chunked-prefill-size 256: that instance boots, reports ready and then
admits nothing at all". Re-applied: 3 passed.

The mechanism test measures can_run_list, not add_one_req's return value: a
successful chunked admission consumes the whole chunk budget, so budget_state
reports OTHER on the way out of a SUCCESS just as the refusal does. An
earlier draft asserted on the return value and its own control assertion
caught it -- kept as a control that proves the fixture can admit the same
request at a satisfiable alignment.

Tests: sgl-project#631 flip family 1095 passed / 0 failed. The count moves 1092 -> 1095
because the new file was ADDED to the canonical list in
scripts/run_631_flip_family.sh -- this tree has had three under-collections
on that list and an admission-path test must not be orphaned. kvso + prefill
adder families 217 passed / 0 failed. ruff: identical to baseline on all
three touched source files, clean on the new test. codespell clean.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…roject#677 to compose against

The sgl-project#679 close-out said the retry net's registry is empty because the reliefs
that could genuinely pay are collective and belong at admission. This is that
claim made concrete, written so sgl-project#677's hysteresis-drain design can consume it
without re-deriving anything.

WHAT IT ESTABLISHES.

Four reliefs exist and only four: radix eviction, the sgl-project#287 ladder's
admission_cap, kvso try_spill, and retract_decode. For each: what it frees at
the admission decision point, what it costs, and how it is invoked without
splitting the group. Two of them are not what one would guess:

  the sgl-project#287 ladder is TOO SLOW to be a rung. Its consensus boundary is every 8
  rounds; a chunked-prefill burst exhausts the pool in fewer. It is the slow
  outer loop -- and the decode path already uses it that way, throttling before
  retraction to stop the freed slots being handed straight back.

  kvso try_spill is the BEST rung, not retraction. It frees a bounded, chosen
  amount (the victim's block-aligned tail overhang), is already driven from the
  reduced value, and costs no request's progress. Its bound is the host region
  supply, and exhaustion is a reachable state the decode path already documents.

THE ITERATION ORDER, VERIFIED, AND IT IS FAVOURABLE. The reduce runs at
scheduler.py:4777, unconditional and pre-branch by its own comment; admission
at 5089; retraction at 5950 inside update_running_batch (5818). So a ladder at
admission reads an ALREADY-AGREED number and needs no collective of its own,
while retraction is genuinely downstream -- which is why sgl-project#679's crash had
nothing to fall back on, and what rung 3 must work around.

THE ORDER, mirroring the decode-OOM branch rather than inventing a second
shape: evict (baseline) -> try_spill -> throttle -> retract_decode -> PARK.
Parking stays the floor of the ladder, not its replacement.

THE COMPOSITION CONTRACT WITH sgl-project#677, stated in three rules because both
mechanisms decide admission from pool headroom:

  1. The park guard is the FINAL authority, the drain gate the prior one. A
     phase gate cannot make memory exist; it may narrow what the park guard
     allows, never widen it.
  2. Any headroom quantity sgl-project#677 branches on must be the GROUP-PUBLISHED floor.
     A rank-local reading in the drain gate reintroduces the sgl-project#603/sgl-project#583
     divergence class upstream of every safeguard sgl-project#679 added -- and a park
     guard reading the reduced floor does not protect a gate that reads a
     local one.
  3. Parking must be reachable from every path that reaches alloc_for_extend.

AND THE LIKELIEST COMPOSITION FAILURE, called out so it can be tested on both
sides before it is met on metal: hysteresis and parking can beat against each
other. A parked chunk schedules ZERO tokens, so a drain that counts admission
ATTEMPTS rather than SCHEDULED TOKENS will believe work was taken, hold its
hysteresis, and deadlock against the park at exactly the pressure where both
are needed.

Section 5 lists what this note does NOT close: the ladder is unbuilt, rung 3 is
a real refactor, and rung 1's host-region bound has never been measured under
the 5-lane load that produced the crash.

No code change. Every claim carries its file:line.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…ore parking

Built to DESIGN_679_admission_relief_ladder.md. A park is not free -- it is a
request that made no progress this round -- and the note costed what should be
spent before accepting one. This spends it.

    rung 0  radix eviction   already spent by the caller. Baseline, not repeated.
    rung 1  kvso.try_spill   bounded, chosen, costs no request's progress.
    rung 2  throttle         frees NOTHING now; stops rung 3 repeating next round.
    rung 3  retract_decode   most tokens, loudest: the victim re-prefills.
    rung 4  PARK             the floor, and still the final authority.

RUNG 3 IS THE ONLY HARD PART, AND IT IS A REFACTOR, NOT A CALL. retract_decode
is one line; what surrounds it in update_running_batch is what must not drift --
the metrics, the new_token_ratio handover, the abort dispatch, and above all

    for req in retracted_reqs:
        self._add_request_to_queue(req, is_retracted=True)

A second implementation that forgot that line would LEAK every victim it
retracted, which is worse than the crash this ladder exists to prevent. So the
block is EXTRACTED VERBATIM into _retract_decode_and_requeue and both call sites
share it. One implementation, two callers, no drift -- and a test pins that the
shared actuator still contains the requeue.

The precondition travels with it: rung 3 sets batch.uniform_avail_floor from the
reduced value before retracting, because that bound governs the retraction loop
AND the last-survivor test. sgl-project#583 is exactly the case where the entry decision was
uniform and the loop bound was not, so ranks entered together and popped
DIFFERENT numbers of victims.

GUARD (a) -- EXHAUSTION IS AN OUTCOME, NEVER AN ERROR. try_spill returns False
when no host region is free; the ladder falls through to the next rung. Same for
a rung that frees less than asked, and same for a rung that RAISES: every rung is
wrapped, the ladder continues, and nothing here can turn a relief bug into an
instance death. That host-region bound is still unmeasured under the 5-lane load
that produced the crash, which is precisely why it is treated as ordinary.

GUARD (b) -- EVERY DECISION IS GROUP-UNIFORM. Each rung reads uniform_min_avail(),
the value the pre-branch reduce published at the top of this iteration, so no rung
takes a collective of its own and none can split the group. The shortfall is sized
from that same reduced value, so every rank asks its rungs for the same tokens --
sizing it locally would be sgl-project#583 one layer up from where sgl-project#583 was found.

GUARD (c) -- OFF BY DEFAULT. SGLANG_ADMISSION_RELIEF_LADDER unset returns 0 before
touching anything: byte-identical to c4b88e1, the boot currently serving. Rung 3
carries a SECOND flag (SGLANG_ADMISSION_RELIEF_RETRACT) because it is the only rung
that destroys progress, and it is inert unless the ladder itself is on.

THE PARK REMAINS FINAL (DESIGN_679 rule 1). The ladder changes what there is to
decide from; add_chunked_req still decides. It runs immediately before that call
and nowhere else, and a source pin refuses can_run_list / add_chunked_req /
set_extend_range inside it -- a ladder that admitted anything would be a second
admission authority.

Tests: 18 hermetic, on a Scheduler stub carrying only what the ladder touches.
Mutation-proven on the four decisions that could invert silently:
  default flipped ON                       -> the off-by-default case fails
  rung 3 without uniform_avail_floor       -> the sgl-project#583 precondition case fails
  exhausted spill treated as terminal      -> the fall-through case fails
  a paid spill no longer short-circuiting  -> the ordering case fails
Each fails alone, by name. One test of my own was caught by its first run reading
the wrong frame (get_new_batch_prefill delegates to _get_new_batch_prefill_raw) --
the same stale-pin class repaired in the PP loop tests, fixed the same way.

No regressions: 1011 pass across the scheduler/prefill/seam suites, zero failures.
The extraction adds no lint (94 before, 94 after, all pre-existing).

NOT BOOTED. Desk-only; validation bundles with the sgl-project#678 acceptance.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…, by design not by accident

The build brief named group-uniform rung decisions as a guard to be
mutation-proven. It was implemented -- every decision reads uniform_min_avail()
-- but the tests caught a rank-local read only by ACCIDENT: the stub carried no
allocator, so a mutation raised AttributeError, the trigger's own except
swallowed it, and the ladder silently did nothing. Tests that pass because the
wrong code crashes are not tests of the property.

So the stub now carries a rank-local availability that DISAGREES with the
reduced one -- 10x the chunk, i.e. comfortable, while the group is starved --
and two cases assert the ladder acts on the group's number: it spends rungs when
the GROUP is short even though this rank looks fine, and it sizes the shortfall
from the group value (sgl-project#583 one layer up: ranks that size the ask differently
retract different numbers of victims).

Mutation-proven properly now. Pointing the TRIGGER at this rank's pool fails
five cases including both new ones by name; pointing the ladder's internal
comparison at it fails the ordering case. Neither is an AttributeError any more
-- each is a wrong verdict, which is what the group would actually experience.

20 tests. No production change.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…l; the new-request gate never did

sgl-project#679 taught `add_chunked_req` not to schedule a chunk the pool cannot fund, and
it made that decision GROUP-UNIFORMLY: `fundable_extend_tokens` reads the
published group MIN through `uniform_avail_for_evict`. Its sibling gate -- the
one every FRESH request passes -- was left reading this rank's own pool:

    PrefillAdder.rem_total_tokens
        available_size() + evictable_size() - offset - dcp_avail_deficit

That is the same over-promise one door further along, and it fails twice over:

  * A rank roomier than the binding rank admits work the GROUP cannot fund, and
    the batch dies where it is allocated -- `alloc_for_extend`, 2026-08-16
    01:46:10, all three ranks together.
  * It is a rank-local BRANCH upstream of a collective. Two ranks can build
    different batches and enter the next extend with different token axes,
    which is a HANG rather than a stall -- the family sgl-project#583/sgl-project#603/#616g/sgl-project#639 was
    paid for.

`fundable_extend_tokens`' own docstring names the first half. It was written
for the chunked gate and never wired to this one; `fundable_extend_tokens`
appeared nowhere in scheduler.py, which the red-first run confirms.

WHAT THIS IS NOT. It is not the cause of the 01:46 traceback. That was
e778276 -- an unlocked mamba tombstone leaf the eviction frontier selected
and `_evict_leaf_node` could not consume, so eviction under-delivered while the
counter honestly promised 65766 evictable. THE COUNTER AND THE ACTUATOR NOW
AGREE; this commit is about the second consumer of that counter, which reads a
rank-local copy of it and can over-promise even when it is honest.

A CEILING, NOT A SUBSTITUTE. `rem_total_tokens` also subtracts reservations the
floor knows nothing about (the running batch's hold, mamba gap, page overhead),
so replacing the local term would talk a rank that is ITSELF short back UP. The
budget is the MIN of the two. Both spend down the same `rem_total_token_offset`
-- a floor consulted per request without that shared accounting is not a bound
at all, since every request in the round would compare itself against the same
untouched pool. `dcp_avail_deficit` is deliberately NOT subtracted from the
floor term: the floor is already the group MIN, and the deficit is the other
mechanism for pinning a local number to the binding rank.

WHY THE MIN RESOLVES TO THE FLOOR IN PRODUCTION, and it is a test, not a claim:
`floor = uniform_avail + evictable` and `local = local_avail + evictable -
offset` with `uniform_avail <= local_avail` by definition of a MIN, so
`floor - offset <= local` on every rank. A rank BELOW the floor is not a state
the reduce can produce; the clamp's behaviour there is pinned separately
because refusing to be talked up by a stale floor is the wanted behaviour.

THE CEILING MUST NOT BE ABLE TO WEDGE THE INSTANCE, and this is why the wiring
goes through `published_fundable_floor` rather than calling
`fundable_extend_tokens` directly. That helper returns 0 for BOTH "the pool is
empty" and "the pool could not be read". The chunked gate can live with the
ambiguity -- 0 parks a chunk and the next round retries, a self-clearing state.
As a budget ceiling a mis-read 0 admits NOTHING, for every request, on every
subsequent round: a harder wedge than the crash it prevents. So the ceiling is
applied only where a floor was actually PUBLISHED, which is the one state
(uneven rank pools) it exists for. A published floor of genuine zero still
binds -- the distinction is "was a floor published", not "is it non-zero", or
the ticket's own crash state would be exempt from its own fix.

DEFAULT PATH UNTOUCHED, BY CONSTRUCTION AND BY ASSERTION. With no floor
published -- single rank, or pools that agree -- `fundable_extend_floor` is
None and `rem_total_tokens` returns exactly what it returned before.

TESTS. RED-FIRST against 7936bc4: 12 of the 18 fail there, including the
01:46 shape itself (a rank holding 100000 tokens admitting against a group that
holds none) and all three wiring pins. The guard against the wedge was
falsified separately by removing it -- 2 of its 4 cases go red -- because an
instrument that cannot fail is not evidence.

Covered: the cap; the floor spent down by admissions; group agreement across
ranks with different local pools; the floor binding before the local term
whenever the reduce's invariant holds; the local term still binding when it is
the smaller one; inertness with no floor and with a roomy floor; the
published-vs-readable distinction including a genuine zero; and three pins that
this stays wired (the scheduler constructs with a floor, it comes through the
guarded helper, and that helper still delegates to the group-uniform one).

NO REGRESSIONS, measured against the same subset on the unpatched tree:
944 failed / 2776 passed / 725 skipped before, 944 failed / 2794 passed / 725
skipped after -- identical failures (all CUDA-gated collection, hermetic run
with CUDA_VISIBLE_DEVICES=""), +18 = exactly the new cases.

NOT BOOTED. Desk-only; READY-FOR-BOOT. The metal arm that would exercise it is
a multi-rank boot with uneven pools under max fill -- the arm that produced the
01:46 specimen -- watching for the admission ceiling binding without a wedge.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…l; the new-request gate never did

sgl-project#679 taught `add_chunked_req` not to schedule a chunk the pool cannot fund, and
it made that decision GROUP-UNIFORMLY: `fundable_extend_tokens` reads the
published group MIN through `uniform_avail_for_evict`. Its sibling gate -- the
one every FRESH request passes -- was left reading this rank's own pool:

    PrefillAdder.rem_total_tokens
        available_size() + evictable_size() - offset - dcp_avail_deficit

That is the same over-promise one door further along, and it fails twice over:

  * A rank roomier than the binding rank admits work the GROUP cannot fund, and
    the batch dies where it is allocated -- `alloc_for_extend`, 2026-08-16
    01:46:10, all three ranks together.
  * It is a rank-local BRANCH upstream of a collective. Two ranks can build
    different batches and enter the next extend with different token axes,
    which is a HANG rather than a stall -- the family sgl-project#583/sgl-project#603/#616g/sgl-project#639 was
    paid for.

`fundable_extend_tokens`' own docstring names the first half. It was written
for the chunked gate and never wired to this one; `fundable_extend_tokens`
appeared nowhere in scheduler.py, which the red-first run confirms.

WHAT THIS IS NOT. It is not the cause of the 01:46 traceback. That was
e778276 -- an unlocked mamba tombstone leaf the eviction frontier selected
and `_evict_leaf_node` could not consume, so eviction under-delivered while the
counter honestly promised 65766 evictable. THE COUNTER AND THE ACTUATOR NOW
AGREE; this commit is about the second consumer of that counter, which reads a
rank-local copy of it and can over-promise even when it is honest.

A CEILING, NOT A SUBSTITUTE. `rem_total_tokens` also subtracts reservations the
floor knows nothing about (the running batch's hold, mamba gap, page overhead),
so replacing the local term would talk a rank that is ITSELF short back UP. The
budget is the MIN of the two. Both spend down the same `rem_total_token_offset`
-- a floor consulted per request without that shared accounting is not a bound
at all, since every request in the round would compare itself against the same
untouched pool. `dcp_avail_deficit` is deliberately NOT subtracted from the
floor term: the floor is already the group MIN, and the deficit is the other
mechanism for pinning a local number to the binding rank.

WHY THE MIN RESOLVES TO THE FLOOR IN PRODUCTION, and it is a test, not a claim:
`floor = uniform_avail + evictable` and `local = local_avail + evictable -
offset` with `uniform_avail <= local_avail` by definition of a MIN, so
`floor - offset <= local` on every rank. A rank BELOW the floor is not a state
the reduce can produce; the clamp's behaviour there is pinned separately
because refusing to be talked up by a stale floor is the wanted behaviour.

THE CEILING MUST NOT BE ABLE TO WEDGE THE INSTANCE, and this is why the wiring
goes through `published_fundable_floor` rather than calling
`fundable_extend_tokens` directly. That helper returns 0 for BOTH "the pool is
empty" and "the pool could not be read". The chunked gate can live with the
ambiguity -- 0 parks a chunk and the next round retries, a self-clearing state.
As a budget ceiling a mis-read 0 admits NOTHING, for every request, on every
subsequent round: a harder wedge than the crash it prevents. So the ceiling is
applied only where a floor was actually PUBLISHED, which is the one state
(uneven rank pools) it exists for. A published floor of genuine zero still
binds -- the distinction is "was a floor published", not "is it non-zero", or
the ticket's own crash state would be exempt from its own fix.

DEFAULT PATH UNTOUCHED, BY CONSTRUCTION AND BY ASSERTION. With no floor
published -- single rank, or pools that agree -- `fundable_extend_floor` is
None and `rem_total_tokens` returns exactly what it returned before.

TESTS. RED-FIRST against 7936bc4: 12 of the 18 fail there, including the
01:46 shape itself (a rank holding 100000 tokens admitting against a group that
holds none) and all three wiring pins. The guard against the wedge was
falsified separately by removing it -- 2 of its 4 cases go red -- because an
instrument that cannot fail is not evidence.

Covered: the cap; the floor spent down by admissions; group agreement across
ranks with different local pools; the floor binding before the local term
whenever the reduce's invariant holds; the local term still binding when it is
the smaller one; inertness with no floor and with a roomy floor; the
published-vs-readable distinction including a genuine zero; and three pins that
this stays wired (the scheduler constructs with a floor, it comes through the
guarded helper, and that helper still delegates to the group-uniform one).

NO REGRESSIONS, measured against the same subset on the unpatched tree:
944 failed / 2776 passed / 725 skipped before, 944 failed / 2794 passed / 725
skipped after -- identical failures (all CUDA-gated collection, hermetic run
with CUDA_VISIBLE_DEVICES=""), +18 = exactly the new cases.

NOT BOOTED. Desk-only; READY-FOR-BOOT. The metal arm that would exercise it is
a multi-rank boot with uneven pools under max fill -- the arm that produced the
01:46 specimen -- watching for the admission ceiling binding without a wedge.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 22, 2026
… unified lineage

The previous commit refused the sgl-project#755 floor reduction because the lineage
that gets built does not implement the mechanism. This builds it, and the
floor drops back to 1+P+1 -- honestly this time, because the runtime now
does what the arithmetic claims.

MECHANISM. cache_unfinished_req releases the OLD anchor's mamba pin BEFORE
the donation allocation instead of after the insert, so the old and new
anchors never coexist and a running request holds active + donated rather
than active + donated + old pin. That is the slot, and on the standing
boot (max_running_requests 8) it is 8 of them.

STRICTLY SAFER THAN THE ORIGINAL, and this is why the port was worth
doing rather than re-gating. MambaRadixCache releases the WHOLE node lock
early; inc_lock_ref walks ancestors for the FULL component, so its window
also leaves the request's own matched KV prefix evictable. The unified
tree locks per component, so dec_mamba_lock_only drops ONLY the mamba
lock: the KV path stays protected and the single state slot being
un-double-counted is the only thing made evictable. dec_swa_lock_only is
the same shape for SWA, so this is the file's existing idiom.

PER-NODE GATE. The config predicate cannot know whether THIS anchor
survives losing its slot, so MambaComponent.anchor_release_admissible asks
again: the node must carry a host copy (an evicted-but-backed anchor is a
valid match that triggers load_back -- is_resume_candidate's
device_only=False rule), and that copy must have LANDED. sgl-project#767: write-through
publishes host_value the moment the transfer is handed to the controller
and records the node in ongoing_write_through in the same block; between
those two facts the anchor is an intention only, and releasing there is
the dead anchor the gate exists to prevent.

PAIRING, which is where this class of change goes wrong (sgl-project#583). Two exit
paths, both handled:
  * the normal path skips the mamba half at the usual dec_lock_ref, via
    the skip_lock_node_ids set the components already consult -- a ref that
    was never taken must not be given back;
  * the bail path (nothing inserted, so no new anchor takes over the pin
    and req.last_node is unchanged) RE-ACQUIRES, restoring the exact
    pre-call state rather than carrying an imbalance into the next call.

TESTS: test_mamba_reorder_unified_773.py, 17 tests, CPU-only.
The end-to-end pair is the real proof: the pool is sized so the donation
allocation can only succeed if the old anchor was released first (2 slots
= one active + one anchor, free = 0). A backed anchor caches; an unbacked
one, identical in every other respect, does not. That is the saved slot,
observed as behaviour rather than as arithmetic.

CAN-FAIL, ALL EXECUTED, and three of them were earned the hard way:
  * unwiring the reorder in cache_unfinished_req  -> 1 fails
  * dropping the skip set                         -> 1 fails
  * dropping the bail re-acquire                  -> 1 fails
The last two SURVIVED the first version of this file, because every
pairing test simulated the sequence by hand instead of driving the entry
point. release_component_lock no-ops at zero, so a double release on a
singly-held node hides completely; it only shows when the anchor has a
SECOND holder, whose ref the unskipped release would steal. Both tests
were rewritten to go through cache_unfinished_req, and the skip-set case
needed a spare slot -- with a full pool the step bails and correctly
restores the pin, which is a different path and now has its own test.

Floor on the standing boot: 24 -> 16. The retention budget introduced in
703b05c becomes 8 slots instead of 0, so mamba host backups are
possible again rather than uniformly declined.

REGRESSION: test/registered/unit/mem_cache/ = 1473 passed, 8 failed, the
same 8 as an untouched HEAD worktree, diffed name-for-name.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 4, 2026
…it in a named stop

BOOT weg1b7 (03:49:20 -> 03:50:24Z) died in a circular PP ring wait: PP0
parked in recv_object[src=2], PP1 tore at the 60 s chain-stall bound, both
downstream ranks ran their two ring-cut drain turns and the group died.

ROOT (built against, and it is NOT the ring cut). The sgl-project#631 row probe peeks
the head proxy frame's admission row and, when the row names a rid this rank
cannot locate, answers False and leaves the frame in the inbox. That
protection is right (boot 631row14: a frame consumed before its request hop
plans nothing and the ring dies upstream-waiting) and is untouched here.
What it lacked is a BOUND, and the defer is not free: PP0 has ALREADY
launched the pass the frame belongs to and is ALREADY parked on its output,
so an unbounded defer closes the ring BY CONSTRUCTION.

The defer's own premise -- "an unlocatable rid PROVES a chain send is in
flight, bounded by rendezvous+transfer" -- is false for both populations
measured in weg1b7:

  * a rid PP0 RETRACTED after sealing the row. PP0 shipped sgl-project#631 PROXY-SEND
    t40 naming 772368da and c25108f3, then the discretionary #888b parked-
    carrier relief retracted c25108f3 in the NEXT scheduling iteration
    (#969AD RETRACT site=_retract_decode_and_requeue fwd_ct=157). A sealed
    row cannot be amended, so _missing can never empty.
  * a re-admitted resident (sgl-project#968 READMIT CACHED, last_queued_as=cutover-
    requeue) requeued rank-locally by _add_request_to_queue -- no chain hop
    for it exists or ever will.

Ranks disagreeing about a rid is RAENGE-NIE-UNEINS, and the sanctioned
answer there is a bounded, NAMED stop -- never a silent unbounded wait.
sgl-project#1071's occupant horizon is the precedent this follows, down to "0 disables".

CHANGE
  python/sglang/srt/managers/pp_row_defer_horizon.py  (new, 300 lines)
      RowDeferHorizon: per-slot clock over the (frame stamp, missing-rid set)
      identity. First sighting defers. A CHANGED missing set is a hop landing
      and restarts the clock -- the boot 631row15 world (43k hot defers on a
      TURNING ring) therefore never reaches the bound. A DIFFERENT frame also
      restarts it, so a slot whose head alternates cannot inherit a
      predecessor's age. Only the same frame with the same missing set, held
      past SGLANG_PP_ROW_DEFER_HORIZON_S (default 20 s), answers defer=False
      with a message naming slot, wait, occurrences, rids, BOTH measured
      shapes, what is deliberately not repaired, and the remedy knob.
      PURE: no logger, no wire, no scheduler attribute -- the caller owns the
      raise, as parked_carrier_relief and min_free_slots_delayer are built.
      20 s is chosen against three measured neighbours: orders of magnitude
      above the healthy case (weg1b7's three defers of one frame land in the
      same log second), BELOW the 60 s chain stall bound so THIS names the
      contradiction before the ring cut misreports it as "closed ring on the
      request chain", and BELOW the 90 s sgl-project#1071 horizon.

  scheduler_pp_mixin.py:9184-9243   the defer consults the horizon and raises
      PpRowDeferHorizonLapsed past the bound. It does NOT consume the frame
      (that revives boot 631row14) and does NOT emit a void (that revives the
      sgl-project#801 reverse corpse). The DEFER log line now carries held=%.1fs.
  scheduler_pp_mixin.py:9266-9271   a satisfied or unreadable row clears the
      slot's clock, so a resolved wait cannot age a later one into the bound.
  scheduler_pp_mixin.py:8912-8941   _pp_row_chain_pending's docstring
      corrected: its _pp_row_chain_owed arm returns True on an INFERENCE, not
      on proof of availability, and the receive it opens is bounded by this
      horizon rather than by transfer time. Do not read that True as uniform
      evidence of a posted message.

EVIDENCE
  RED FIRST on parent 2615514 (scratch worktree /spinning/wt-1180-red,
  only the new test file copied in): 12 failed / 2 passed. Two of the reds
  are BEHAVIOURAL, not import errors --
    test_the_defer_is_not_unbounded: "the row probe deferred the same
      unsatisfiable row 4096 times over an hour of its own clock without ever
      stopping" (imports nothing new, on purpose);
    test_a_satisfied_row_clears_the_clock_its_own_defer_armed: "the defer
      must have armed a clock".
  FIXED: 14 passed / 0 failed.

  MUTANTS 6/6 died, each patch verified applied:
    M1 never raise (fix inert)                                    DIED
    M2 bound never lapses                                         DIED
    M3 a changed missing set does not restart the clock           DIED
    M4 satisfied row leaves its clock behind                      DIED
    M5 no frame token in the identity                             DIED
    M6 DANGER DIRECTION: stop on the FIRST defer                  DIED
  M4 survived the first round -- the locatable-row test was vacuous (no
  horizon object existed to inspect). The recovery test above was added to
  make the clear observable, and M4 then died.

  ruff parity: 3 pre-existing errors on the parent, the same 3 after
  (F841 carries_flip_arm, E741 l x2). Import smoke hermetic (CVD="").

  SUITE ONCE, both trees, same command, sequentially (crowding-free box,
  load ~2.9), test/registered/unit/managers with the five pre-existing
  collection-error modules ignored (the same five on both trees: the sgl-project#1044
  import-broken *_631 quartet plus test_quiescence_no_carry_858):
    parent 2615514 : 240 failed / 4767 passed / 2 skipped / 638 subtests
    fixed             : 240 failed / 4781 passed / 2 skipped / 638 subtests
  Delta = +14 passed, exactly the new file; the FAILED NAME SETS are byte
  identical (diff empty, 236 names each side). Counting probe declared
  honestly: 236 extracted names vs the summary's 240 on BOTH trees -- a
  symmetric extraction gap of 4 that is not investigated here and cannot
  affect the A/B, since both sides carry the same 236 names and the same
  summary.

RESIDUALS, deliberately NOT fixed here
  * The stop is RANK-LOCAL, exactly like sgl-project#1071's: the deferring rank raises,
    the others then die on their own collectives or the deadman. A
    group-uniform stop would need a collective on the wedged path, which is
    what the ring cut already refuses.
  * #888b's discretionary yield is untouched. Making it protect published
    rids is refused: the entry decision and the victim count must be
    rank-uniform (the sgl-project#583 precondition stated in _maybe_yield_parked_carrier
    itself), and only PP0 builds rows, so a PP0-only term would make ranks
    pop different numbers of victims -- a silent rank divergence traded for
    this deadlock.
  * The ring cut's arc asymmetry (pp_flip_drain_leftover_dicts receives only
    from the draining rank's own upstream, scheduler_pp_mixin.py:6796) is left
    as it is: it is fork-own compensation, and under upstream-minimal a defect
    inside a compensation layer is a deletion candidate, not a repair order --
    and widening it could not clear PP1's _missing anyway.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant