Skip to content

Improve benchmark scripts - #615

Merged
merrymercy merged 2 commits into
mainfrom
bench
Jul 13, 2024
Merged

merrymercy merged 2 commits into
mainfrom
bench

Conversation

@merrymercy

Copy link
Copy Markdown
Collaborator

No description provided.

@merrymercy
merrymercy merged commit 0feca02 into main Jul 13, 2024
@merrymercy
merrymercy deleted the bench branch July 13, 2024 22:59
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 7, 2026
…build window — a rank entering a lazy JIT/cold build publishes a pid-checked marker (/dev/shm, path derived from the existing PeerTable tuple, no new collective); both bounded waiters (barlink_liveness.bounded_poll + barlink_bar1._wait_ctl_event) extend their deadline through one shared decision under a 900 s total cap, then raise with the peer named; decision is stat+float only — no device read, no .item(), pinned structurally; wired via cold_build_window, jit_build_guard (incl. the 18-min hicache_hash_cpp site) and _ext_cache_guarded for the three barlink load_inline exts; 31 hermetic tests, three single-line can-fail reverts proven, distributed-suite delta exactly +31 vs base, default path byte-identical; single-node scope + unmeasured cap documented)
efschu added a commit to efschu/htsglang that referenced this pull request Aug 9, 2026
…keeps cycling

Boot 10 showed the arm delivery finally correct -- all three ranks armed
identically, zero control messages -- and it still wedged:

  rank0:   on_round -> bounded_collective        (flip reduction)
  rank1/2: _pull_raw_reqs -> point_to_point_pyobj (chain recv)

Ranks 1 and 2 DID arm and DID reach on_round, but were not parked yet, so
they returned and came back to the chain recv, which is an UNBOUNDED
wait. Rank 0's park predicate went true first; it entered the blocking
reduction -- and entering it stops rank 0 forwarding the very batch its
peers are blocked on. Nobody can join, and rank 0 cannot time out because
the park deadline is evaluated before entry, never from inside the wait.

The existing safety argument ("a parked rank owes no pipeline send")
covers hidden-state sends. It does NOT cover the request-chain forward,
which a parked rank still owes its downstream stage every iteration.

An unbounded wait in a coordination path is also against this fork's
bounded-wait canon (sgl-project#610/sgl-project#615/sgl-project#653). Bound it: a stage with nothing to
receive returns empty, re-reaches its own on_round -- by then parked --
and joins. Idle coordination then needs no chain message at all, which is
what lets the policy stay message-free.

THE INVARIANT, and the reason this is not a one-line timeout: a posted
irecv is a standing promise to the sender, which will deliver into that
exact tensor. It can never be abandoned on a deadline. The Work and its
tensor therefore live in chain_recv_state and are re-polled across calls;
a second irecv is never posted while one is outstanding. Only the SIZE
header is polled -- once it lands the sender is committed to the payload,
so the body keeps the ordinary blocking wait.

Gated on chain_recv_poll_s, wired to 50 ms only when the policy is on.
Every other boot keeps the upstream unbounded path byte-identical.

Tests: 317 passed in the sgl-project#631 flip family, CPU-only. New: a bounded recv
returns empty rather than blocking and never waits an incomplete Work;
three polls of one pending receive post exactly ONE irecv (the
desynchronisation falsifier); a landed size==0 clears the state; poll 0.0
keeps the upstream path.

Also recorded: the first VALID sustained load of this task -- 4 min
agent-like mixed traffic (1 long-prefill worker at 8192/32768 + 3 decode
workers, bs<=4) against the manual-policy server: 52 requests, 52 ok,
0 aborted, post-idle 32768-tok probe 4383.7 tok/s. Earlier load attempts
measured wedged servers and were no measurement at all.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 9, 2026
Restores the arm as a chain message and fixes the ORDER that made it
deadlock, rather than removing the message. The message-free variant was
built and measured (boot 10): every rank reached an identical verdict and
armed itself with zero messages -- and it wedged solid. arms=3,
cutovers=0, 0 % GPU, /generate dead at 45 s.

THE MECHANISM, which inverts the earlier reading: THE ARM MESSAGE IS THE
WAKEUP. Ranks 1..n-1 spend idle time BLOCKED in the chain recv, and
forwarding the arm is the only thing that makes them runnable. That is
why the manual POST /phase_flip has always worked (~1.2 s cutovers,
reproduced all session) and why removing the channel removed the
synchronisation. The hazard was never the channel; it was the order.

THE INVARIANT (named in code): no rank may enter the flip's BLOCKING
consensus reduction while it still owes, or has uncommitted, chain sends.
A rank acts on an arm only in the pass AFTER _pp_commit_comm_work has
committed the forward carrying it downstream. Intermediate stages obey
the same rule on receipt, so the wake propagates strictly ahead of every
rank's block; the last rank owes no forward and its deferral is a
one-pass no-op. With this ordering the automatic path is byte-for-byte
the manual one -- same request, same chain, same wake, same
drain-park-meet -- and only the originator differs (policy on rank 0 vs
an HTTP caller). That equivalence with a mechanism proven on metal is the
design's strongest argument.

Two alternatives stay rejected, both measured: a SYNCHRONOUS forward
deadlocks worse (a peer need not be in the chain recv at all; found
blocked in the hidden-states exchange while this rank blocked in send),
and BOUNDING the chain recv decouples a 1:1 send/consume contract so the
consumer can skip a consume, unmatched sends accumulate and the SENDERS
block (reverted in 3c0115fd6d; its posted-irecv-is-a-promise test is kept
as the artifact).

(c) BOUNDED JOIN, unconditional backstop: entry to the consensus
reduction is now a bounded wait that abandons from INSIDE, disarms, logs
loudly and returns to cycling, so the policy may re-arm. Every other flip
deadline is evaluated BEFORE entry, so a rank that got in and found no
peers previously waited for ever -- an unbounded wait in a coordination
path, against sgl-project#610/sgl-project#615/sgl-project#653. The consensus channel now forwards a
per-caller timeout_s to bounded_collective. Default 45 s, generous by
intent: a peer draining a long prefill is normal and must not trip it.
This is a wedge breaker, not a latency control.

Tests: 317 passed in the sgl-project#631 flip family, CPU-only. Can-fail proven
against the boot-10 specimen: reintroducing arm-in-same-pass turns all
three delivery-before-block tests red with "acted on the arm in the pass
that forwarded it". Also pinned: the origin guard (a downstream stage
receiving a populated list injects nothing -- the 1/2/3-arm self-kill),
the last-rank no-op deferral, that a human RPC is never deferred, that
abandonment never raises into the event loop, and that the join deadline
actually reaches the channel.

Operational acceptance under load NOT yet claimed; the boot carrying this
is running.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 9, 2026
THE DESIGN LAW, from seven measured corpses:

  NO RANK MAY BLOCK ON ANY CHANNEL WHILE A PEER MAY BE IN A DIFFERENT
  BLOCKING CHANNEL.

  A   arm same-pass, async forward   rank0 in reduction, peers in chain recv
  B   arm same-pass, sync forward    rank0 in send, peers in hidden-states recv
  B'  arm same-pass, targeted commit IDENTICAL to B (boot 13)
  D   defer arm by one pass          peers never reach the acting pass (boot 12)
  --  message-free local decision    rank0 alone in the reduction (boot 10)
  --  bounded chain recv             breaks the 1:1 send/consume contract;
                                     the SENDERS block
  --  bounded join (abandon inside)  FATAL; closed the gloo pairs, aborted
                                     every rank

B' is recorded as a falsification of my own reasoning: the
targeted-vs-blanket distinction has NO force. Both block rank 0 on the
chain send while a peer may sit in the hidden-states channel, and the
induction failed at its first premise -- the downstream stage was not
"waiting for the arm" at all.

THE FIX: make ENTRY to the blocking reduction conditional on knowing
every peer is already at that entry. Then entering is safe by
CONSTRUCTION, not by argument, because no participant is anywhere else.

  * managers/phase_flip_presence.py: epoch-stamped, monotone, pollable
    per-rank markers in /dev/shm (the sgl-project#615 build-window precedent for
    group-visible state on this single-node topology). Presence is file
    EXISTENCE, written by atomic rename, so a half-written marker cannot
    be misread. Flags are NEVER cleared -- a retraction mints a new epoch
    -- which is what makes a poll safe against a racing writer without
    any clear-coordination.
  * _await_group_presence: the armed poll loop. Per iteration it only
    announces, PUMPS its outstanding arm forward non-blockingly, polls
    peers' flags, and sleeps. Nothing blocks, so the design law holds by
    construction.
  * the entry gate: enter the reduction only on all-ready.
  * PRE-ENTRY bound (60 s): if the group never assembles, disarm loudly
    naming the missing ranks and return to cycling; the policy re-arms a
    new epoch. Legal here precisely because nothing was entered -- the
    contrast with the withdrawn (c) is the point, and (c)'s fatal pin
    stays so nobody moves a bound back inside.

The blocking targeted commit is REMOVED from the forward path (corpse
B'); the arm is delivered by the pump instead, which also fixes corpse A
(an async send is otherwise progressed only by the commit at the top of
the NEXT pass, which never arrives once this rank is armed and polling).

(a) posted-and-polled all_reduce Work is REJECTED on the record: its
load-bearing premise -- that such a Work progresses while merely polled,
without explicit progress calls -- is an unverified transport assumption
of exactly the kind that has already killed designs here. This fork's
async SENDS demonstrably do not progress without an explicit commit
(corpse A). Betting the gate on the reduction transport differing,
unverified, is the same bet.

Tests: 326 passed in the sgl-project#631 flip family, CPU-only. Can-fail proven:
opening the gate with a peer missing turns three gate tests red;
reintroducing a blocking commit in the armed path turns the boot-13
specimen red. Also pinned: pre-entry timeout disarms without raising and
leaves the flip retryable; epochs are monotone and a retraction is a new
epoch; sweep never drops the epoch in flight; a failing pump cannot break
the gate; the manual path takes the same non-blocking route.

Operational acceptance NOT yet claimed.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 30, 2026
…lt ein TRIGGER. Aktionsarm ist S.

Reiner Lese-Pass, kein Boot, kein Code. Beide Quellen je Zeile.

== PFLICHT-VORFRAGE BEANTWORTET: es ist ein UNGEDECKTER TRANSPORT ==
Das haengende all_reduce laeuft auf `grp = self.tp_cpu_group`
(scheduler.py:5964) -- eine GLOO-CPU-Gruppe. Die sgl-project#650/sgl-project#653-Klassen
(Bar1CollectiveAborted / HostCollectiveAborted / Bar1CollectiveStalled)
liegen ausschliesslich in `distributed/device_communicators/barlink_*.py`
plus den zwei debug_utils-Lesern: sie decken den DEVICE/BAR1-Pfad.
Also NICHT "Deckung da, Schwellwert/Leser defekt", sondern: der Pfad hat
diese Deckung nie gehabt. Auf dem gloo-Pfad gibt es weder Deadline noch
Peer-Positions-Bericht.

== CENSUS-FAEHIGKEIT: IDENTITAET IST DA, und mehr ==
`_history.append((family, nbytes))` (:246) -- jede Eintragung traegt die
KOLLEKTIV-FAMILIE, nicht nur Zaehlstaende. `Divergence(family, counts)`
(:459) vergleicht bereits NACH FAMILIE. Und der Modulkopf beschreibt zwei
Berichtspfade:
  1. `compare_across_ranks` -- Detektor, laeuft auf gloo `tp_cpu_group`,
     replizierte Kadenz, feuert waehrend die Raenge noch gesund sind.
  2. `format_local_census` -- die WEDGE-FESTE Haelfte: NIMMT KEIN KOLLEKTIV,
     jeder Rang kippt seine EIGENEN Zaehler ins Log, die drei Dumps werden
     hinterher gediffed.
Die Identitaet fehlt also NICHT. Das ist nicht die Luecke.

== DIE LUECKE, praezise ==
Der Modulkopf sagt woertlich, der Detektor laufe auf dem CPU-Pfad,
"NEVER the device/BAR1 path, SO A WEDGED DEVICE GROUP DOES NOT DISABLE THE
INSTRUMENT THAT IS SUPPOSED TO EXPLAIN IT".
Genau diese Immunitaet greift bei uns NICHT: unser Wedge IST auf der
CPU-Gruppe. Der Detektor braucht ein `all_gather_object` auf ebendieser
Gruppe -- er kann eine Uneinigkeit der Gruppe, die er selbst benutzt, nicht
melden. Die Schutz-Annahme ist um genau einen Transport verschoben.
Und die wedge-feste Haelfte (`format_local_census`, kollektiv-frei) EXISTIERT
fuer diesen Fall -- aber sie wird nie AUSGELOEST, weil auf dem gloo-Pfad
nichts abbricht: kein Deadline, kein Abort, kein Ausloeser.
DAMIT: die Luecke ist ein TRIGGER, keine Faehigkeit.

== AKTIONSARM-DESIGN, S ==
Diskriminator: NICHT "Follower wartet lange" (Normalzustand) und NICHT "PP0
macht keinen Fortschritt" (PP0 wartet selbst) -- sondern eine BEGRENZTE
Wartezeit am gloo-Kollektiv, nach deren Ablauf jeder Rang
`format_local_census()` ruft (kollektiv-frei, schon gebaut) und die Gruppe
per CRASH/STOP stirbt. Die Beider-Seiten-Diagnose faellt dabei GRATIS an:
jeder Rang druckt seine eigene Familien-Historie, und der bestehende
Hinweistext sagt bereits, was damit zu tun ist -- "Diff this line against
the peers' to find where the sequences diverged".
DEADLINE-BEGRUENDUNG gegen die zwei Lehren:
  * sgl-project#630 (wait() treiben, nie is_completed() pollen): der bounded wait muss
    ueber das async work-handle mit Timeout laufen, NICHT ueber eine
    Poll-Schleife.
  * sgl-project#615 (Build-Fenster-Dehnung): die Deadline darf nicht in ein legitimes
    JIT-Cold-Build-Fenster feuern (bis SGLANG_BARLINK_BUILD_WINDOW_CAP_S=900s).
    Sie muss dasselbe Fenster konsultieren, das barlink schon fuehrt --
    sonst ersetzt sie einen Wedge durch einen Fehlalarm beim ersten Boot.
KEIN timeout-und-weiter: Ablauf = Tod der Gruppe, per Rang-Gesetz.

== VERDIKT: S ==
Wiederverwendet wird durchweg Vorhandenes (Census-Familien-Historie,
kollektiv-freier Dump, barlink-Build-Fenster). Neu ist nur der bounded wait
plus sein Ausloeser. Kein neues Kollektiv auf dem Pfad -- das Gesetz bleibt
gewahrt, weil die Diagnose-Haelfte ausdruecklich kollektiv-frei ist.
OFFEN und ehrlich: die Deadline-ZAHL ist nicht herleitbar aus dem, was ich
gemessen habe -- sie braucht die legitime Maximal-Wartezeit unter Last, und
die kenne ich nur als "DE-06 dauerte 95,6 s im gesunden Lauf". Das ist die
eine Groesse, die vor dem Bau gemessen gehoert.

BELEG-STUFE: DESK-BEWIESEN (Transport-Zuordnung, Deckungs-Scope, Census-
Faehigkeit -- alle am Pin gelesen, Modulkopf woertlich zitiert).
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 31, 2026
… while the peers spin

BOOT-PROVEN (boot 22, boot_855_1033b_0840f82601_0831_131955, 13:24-13:26).
The first TP forward after a pp_to_tp cutover touches a shape specialisation
never loaded in this process. Rank 0's crash-time stack (log line 53248 ff.):

  _dcp_write_scatter (flashinfer_backend.py:2574)
    run (triton/runtime/jit.py:743) -> _init_handles (compiler.py:466)
      loadBinary -> cuModuleLoadData (libcuda.so) + 11 libcuda frames

NOT the compiler -- cuModuleLoadData. The module is built; LOADING it into the
CUDA context blocks, because the load needs the device and the device is
saturated by the peers' barlink BAR1 spin kernels, which are waiting in
all_gather for this very rank. The cycle closes and the spin deadline fires:
Bar1CollectiveAborted (ranks 1/3 and 2/3, group flip_dcp:0) -> SIGQUIT.

WHY THE EXISTING MECHANISM MISSED IT, measured: 352 build-window lines in that
boot, ZERO in 13:24-13:26, all 352 carrying 'full cuda-graph capture warmup'.
cold_build_window had exactly three production callers (barlink BAR1 build,
sampling warmup #603b, capture warmup) and none is on the path a cutover
re-dispatches into. sgl-project#640 on a path sgl-project#615 never saw. The fix makes that path a
caller rather than building a second mechanism beside it.

A STALE COMMENT NEARLY REFUTED THIS FIX, and the correction ships with it.
sampler_warmup.py said wrapping a lazy build in cold_build_window "does NOT
work ... the window is PROCESS-LOCAL". True when written (8bddb93,
2026-08-06); falsified ONE DAY LATER by sgl-project#615 (38ec4fb, 2026-08-07), which
hooked publication into cold_build_window so "every existing call site
therefore becomes group-visible without moving". Never revised, both ancestors
of this pin, verified with git log -S rather than assumed. The paragraph is
annotated in place rather than deleted, because it is still right about ITS
module (warm-at-boot + barrier REMOVES the race; a window only EXTENDS it).

WHY A WINDOW OVER THE REAL FORWARDS, NOT AN ENUMERATED WARM SET. The warm set
is not knowable by inspection -- it varies with direction, spec-decode, the
sgl-project#887 one-chunk grant and any backend swap -- and missing one member reproduces
the wedge exactly, at the next first-loader. Whatever loads, loads under the
window. Residual stated in the code: this EXTENDS deadlines (900 s cap) rather
than removing the race; boot 22's stall was ~150 s, so the cap is not binding.

FIRST-LOADER CENSUS, as a standing table: devtools/CENSUS_1033c_first_loaders.md
-- every first-loader site x triggering state change x covering window x
collective proximity. Rows 1-6 verified at file:line (row 5 on metal); rows 7
(runtime recapture / drafter switch) and 8 (rung change sgl-project#704, resume-restore
sgl-project#89) are written UNVERIFIED, so the table is a lower bound on the covered set
and never a proof that nothing else is exposed. It also names the inverse
column (armed windows that may cover paths that no longer first-load -- the
352 are a count of WARMUPS, not of BUILDS, and the window carries no
modules-loaded counter, so live and dead coverage are indistinguishable from
the log) and answers the upstream-minimal question: there are not four window
mechanisms but two plus a front door, with one site (barlink_device.py:865)
bypassing the front door. Named, not rebuilt.

#1033d, same boot, INDEPENDENT DEFECT, minimal hardening only: the prefetch IO
aux thread caught only Empty, so a page whose geometry did not match the
incoming binding (mha.py:556 reshape, '[2,16,1,4,256]'=32768 against a 16384
page) ended the thread three seconds before the cutover -- killing no process,
setting no exit code, appearing in no health probe, and leaving storage
prefetch dead for the rest of the boot. It now fails the OPERATION (host slots
released, so the requester gets a refusal instead of an unreachable
completion), logs loudly with a counter, and stays alive. The broad except is
defensible only because the alternative is silent thread death with no
supervisor above the loop; the underlying two-geometry host pool across a flip
(sgl-project#718/sgl-project#719/sgl-project#875 family, and the fork's own 'sgl-project#939 RE-HOME VIA RE-READ ... source
page 16384 elems vs destination page 32768 elems' line names the same mismatch
two lines earlier while handling it correctly) is NOT fixed here and is its own
posten.

DESK CHECKS, matched to each edit's failure class:
* devtools/check_1033c_cutover_window.py -- 19 cases, hermetic, driving the real
  run_batch wrapper: RED-FIRST arm reproducing the boot-22 condition (counter
  unarmed -> no window -> nothing published), first forward covered AND
  published, window closes (the sgl-project#431 open-without-close shape), budget finite
  and steady state byte-identical, window closed and budget spent even when the
  forward raises, and an AST check that the arming sits under no `if` (the naive
  grep form of that assertion failed on a COMMENT -- fixed to ask the AST).
* AST check on the aux thread: broad handler present, logs at ERROR, releases
  the failed operation's host slots, and continues.
* Method-split check: wrapper delegates, window only on the armed path, lazy
  import off the fast path, inner body intact.
ruff F401/F821/UP037 clean.

DESK-PROVEN. Metal proof is boot 23.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 31, 2026
… counting forwards

A rank loading a Triton module needs the device; the peers' barlink spin
kernels saturate it while they wait for that rank in a collective. Closed
cycle, group wedged. Two fully captured specimens:

* BOOT 22: `_dcp_write_scatter`, cold, on the FIRST FORWARD AFTER A CUTOVER.
  #1033c answered it by arming `cold_build_window` for the first
  `_post_cutover_build_batches = 8` forwards after each cutover.
* BOOT 25: `l2norm_fwd`, cold, on an ORDINARY forward, on TWO ranks at once,
  ~90 s into load and 57 clean cutovers later -- far outside that counter.
  py-spy, two samples per rank, identical frames both times.

#1033c is not wrong. It is BOUNDED BY A COUNT, and a first-loader whose first
call is late is outside any count. `devtools/CENSUS_1033c_first_loaders.md`
predicted exactly this: rows 7 and 8 UNVERIFIED, the table calling itself a
LOWER BOUND. An enumeration is not completed by adding rows to it, and a second
counter would inherit the defect. So this cuts the class instead.

THE CHOKEPOINT IS ONE FUNCTION. `CompiledKernel._init_handles` is the only
place a kernel's module can come into existence -- `load_binary`
(`cuModuleLoadData`) is called there and nowhere else, and Triton's own
`kernel_load_start_hook`/`kernel_load_end_hook` bracket precisely that call,
which is what identifies it as the intended extension point. Wrapping it covers
EVERY future first-loader by construction: no enumeration, no census to keep
current, no counter to tune. Since sgl-project#615 `cold_build_window` publishes at every
call site, so the window opened here is group-visible with no further plumbing.

WHY THE METHOD AND NOT THOSE HOOKS -- the one real trade-off, decided on
exception safety. The END hook runs only on the success path, so a load that
RAISES would leave the window open for the life of the process and stretch
every collective deadline permanently. A `with` block closes on both paths, and
`cold_build_window` is documented re-entrant and exception-safe. The hooks stay
untouched and available.

STEADY STATE IS ONE ATTRIBUTE CHECK. `_init_handles` already opens with
`if self.module is not None: return`. The wrapper asks that FIRST and delegates
untouched, so a warm launch never constructs the window. That early return is
the zero-cost precondition, so it is ASSERTED against the installed Triton
rather than assumed: if a future build drops it, the install DECLINES and logs,
leaving the stock method in place -- a failure to instrument must never become
a failure to run. Verified True against the shipped triton here.

THE 8-BATCH COUNTER STAYS, with a reason. It is now redundant for module loads,
but `_post_cutover_build_batches` also stretches deadlines across the first
post-cutover forwards for reasons other than module loading (capture warmup,
first-touch allocations), and boot 23 measured it working (21/21 balanced). It
is cheap and it is the only thing with metal evidence behind it. Removing it in
the same commit that adds an untested broader mechanism would be trading a
proven narrow cover for an unproven wide one; it can go once this has a boot.

FUTURE CHECK: `devtools/check_1056_loader_chokepoint.py` drives the REAL
wrapped method with a stand-in kernel: a cold load opens exactly one window
naming the kernel, 50 warm launches open ZERO, a raising cold load propagates
unchanged AND leaves no window open. 7/7. It demonstrated its own can-fail on
the first run: the fixture lacked the early return, so install DECLINED and
three assertions failed -- the precondition guard firing correctly.

EVIDENCE TIER: DESK-PROVEN. Metal proof is boot 26.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant