Repository navigation
Detokenize incrementally when streaming - #653
Merged
Merged
Conversation
Contributor
|
@hnyls2002 Nice work! cc @merrymercy @Ying1123 @yzh119 model: Llama 3 8B Instruct # server
python3 -m sglang.launch_server --model /root/Meta-Llama-3-8B-Instruct --trust-remote-code --port 23333 --disable-radix-cach --stream-interval 1
# client
python3 benchmark_serving.py --backend openai --host 127.0.0.1 --port 23333 --dataset /root/ShareGPT_V3_unfiltered_cleaned_split.json --model /root/Meta-Llama-3-8B-Instruct --tokenizer /root/Meta-Llama-3-8B-Instruct --num-prompts 1000 --request-rate 128 --trust-remote-code |
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 8, 2026
…e, never __trap The last live instance of the sgl-project#583 context-killing mechanism: wait_ge in the host-staged transport still called __trap() on spin expiry, which destroys the CUDA context and surfaces as sticky 'unspecified launch failure' at whatever unrelated call site runs next. Unreachable in every production configuration (host is explicitly-selected, no-fallback, bar1 pinned) — but a landmine for any host-transport measurement boot. Ported the device transport's sgl-project#583 fix honestly, not minimally: - wait_ge returns bool and writes a per-site code into the abort word on expiry (put reuse-guard=1, reduce peer-wait=2, copyout wait=3); the BARLINK_HOST_TRAP macro and its hipify plumbing are deleted. - Block-uniform exit: shared abortS written by the spinning thread 0, __syncthreads(), all threads return together before any payload loop, fence, or last_block barrier — verified at all three sites. Terminal- abort contract documented: cross-block skew and stale blk_ctr are acceptable residue of a process the host is about to kill by raise. - The transport's THREE sequence-counter families (collective, per-peer send, per-peer recv) are unified into one zero-initialized backing tensor with 2-word rows — the spec's naive grow-to-2 would have made the p2p kernels' abort store corrupt a NEIGHBOR's sequence number, and the unified shape makes check_aborted a single read over every abort word the transport can produce. - HostCollectiveAborted mirrors DeviceCollectiveAborted (rank/world/ code/where, garbage-results contract) and appends the sgl-project#650 peer statement. check_aborted is a plain in-line read (documented choice: this is a measurement vehicle, not the bs=1 hot path) wired twice: the duck-typed _after_transport seam (sgl-project#431) for eager collectives, and abort-gate registration for replay-only collectives — host is capturable, so a graph-resident collective never reaches the seam (the same gap sgl-project#583 closed for the device transport). - The load-bearing reachability claim is pinned: 'host' is in the no-fallback set via the real parallel_state lookup, and the dispatch seam is proven transport-stable mid-run — an explicit SGLANG_BARLINK_TRANSPORT=host boot is the only way this code runs. Falsifiers, red-first (executed against the pre-fix source inside the test): old wait_ge provably trapped and returned void, old seq tensor had no abort-word room; fixed source has zero traps, bool returns, all three consuming sites, distinct codes. nvcc compile smoke rc=0 (sm_86). GPU-gated on-card falsifier written and documented (put-kernel forced timeout must set the word, leave the context ALIVE, and raise named) — skips at the desk by design. Counting basis: 35 passed + 1 GPU-skip + 19 subtests in the new file; full battery with abort/stub-drift/sentinel regression suites 81 passed + 1 skipped + 19 subtests (97.9 s, my rerun); ruff and imports clean.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 8, 2026
…port wait_ge wait_ge in the host transport writes a per-site abort code and returns instead of trapping the context; block-uniform exit via shared flag + barrier at all three sites; HostCollectiveAborted mirrors the structured device/bar1 shape and carries the sgl-project#650 peer statement. Also closes the capturable-transport abort-gate wiring gap for host, and unifies the abort/seq words into one zero-initialized backing tensor (the spec's grow-to-2 layout would have corrupted p2p slice neighbors). Falsifiers red-first against the pre-fix source; 35 passed + 1 GPU-skip + 19 subtests new; 81 passed full battery.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 9, 2026
…keeps cycling
Boot 10 showed the arm delivery finally correct -- all three ranks armed
identically, zero control messages -- and it still wedged:
rank0: on_round -> bounded_collective (flip reduction)
rank1/2: _pull_raw_reqs -> point_to_point_pyobj (chain recv)
Ranks 1 and 2 DID arm and DID reach on_round, but were not parked yet, so
they returned and came back to the chain recv, which is an UNBOUNDED
wait. Rank 0's park predicate went true first; it entered the blocking
reduction -- and entering it stops rank 0 forwarding the very batch its
peers are blocked on. Nobody can join, and rank 0 cannot time out because
the park deadline is evaluated before entry, never from inside the wait.
The existing safety argument ("a parked rank owes no pipeline send")
covers hidden-state sends. It does NOT cover the request-chain forward,
which a parked rank still owes its downstream stage every iteration.
An unbounded wait in a coordination path is also against this fork's
bounded-wait canon (sgl-project#610/sgl-project#615/sgl-project#653). Bound it: a stage with nothing to
receive returns empty, re-reaches its own on_round -- by then parked --
and joins. Idle coordination then needs no chain message at all, which is
what lets the policy stay message-free.
THE INVARIANT, and the reason this is not a one-line timeout: a posted
irecv is a standing promise to the sender, which will deliver into that
exact tensor. It can never be abandoned on a deadline. The Work and its
tensor therefore live in chain_recv_state and are re-polled across calls;
a second irecv is never posted while one is outstanding. Only the SIZE
header is polled -- once it lands the sender is committed to the payload,
so the body keeps the ordinary blocking wait.
Gated on chain_recv_poll_s, wired to 50 ms only when the policy is on.
Every other boot keeps the upstream unbounded path byte-identical.
Tests: 317 passed in the sgl-project#631 flip family, CPU-only. New: a bounded recv
returns empty rather than blocking and never waits an incomplete Work;
three polls of one pending receive post exactly ONE irecv (the
desynchronisation falsifier); a landed size==0 clears the state; poll 0.0
keeps the upstream path.
Also recorded: the first VALID sustained load of this task -- 4 min
agent-like mixed traffic (1 long-prefill worker at 8192/32768 + 3 decode
workers, bs<=4) against the manual-policy server: 52 requests, 52 ok,
0 aborted, post-idle 32768-tok probe 4383.7 tok/s. Earlier load attempts
measured wedged servers and were no measurement at all.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 9, 2026
Restores the arm as a chain message and fixes the ORDER that made it deadlock, rather than removing the message. The message-free variant was built and measured (boot 10): every rank reached an identical verdict and armed itself with zero messages -- and it wedged solid. arms=3, cutovers=0, 0 % GPU, /generate dead at 45 s. THE MECHANISM, which inverts the earlier reading: THE ARM MESSAGE IS THE WAKEUP. Ranks 1..n-1 spend idle time BLOCKED in the chain recv, and forwarding the arm is the only thing that makes them runnable. That is why the manual POST /phase_flip has always worked (~1.2 s cutovers, reproduced all session) and why removing the channel removed the synchronisation. The hazard was never the channel; it was the order. THE INVARIANT (named in code): no rank may enter the flip's BLOCKING consensus reduction while it still owes, or has uncommitted, chain sends. A rank acts on an arm only in the pass AFTER _pp_commit_comm_work has committed the forward carrying it downstream. Intermediate stages obey the same rule on receipt, so the wake propagates strictly ahead of every rank's block; the last rank owes no forward and its deferral is a one-pass no-op. With this ordering the automatic path is byte-for-byte the manual one -- same request, same chain, same wake, same drain-park-meet -- and only the originator differs (policy on rank 0 vs an HTTP caller). That equivalence with a mechanism proven on metal is the design's strongest argument. Two alternatives stay rejected, both measured: a SYNCHRONOUS forward deadlocks worse (a peer need not be in the chain recv at all; found blocked in the hidden-states exchange while this rank blocked in send), and BOUNDING the chain recv decouples a 1:1 send/consume contract so the consumer can skip a consume, unmatched sends accumulate and the SENDERS block (reverted in 3c0115fd6d; its posted-irecv-is-a-promise test is kept as the artifact). (c) BOUNDED JOIN, unconditional backstop: entry to the consensus reduction is now a bounded wait that abandons from INSIDE, disarms, logs loudly and returns to cycling, so the policy may re-arm. Every other flip deadline is evaluated BEFORE entry, so a rank that got in and found no peers previously waited for ever -- an unbounded wait in a coordination path, against sgl-project#610/sgl-project#615/sgl-project#653. The consensus channel now forwards a per-caller timeout_s to bounded_collective. Default 45 s, generous by intent: a peer draining a long prefill is normal and must not trip it. This is a wedge breaker, not a latency control. Tests: 317 passed in the sgl-project#631 flip family, CPU-only. Can-fail proven against the boot-10 specimen: reintroducing arm-in-same-pass turns all three delivery-before-block tests red with "acted on the arm in the pass that forwarded it". Also pinned: the origin guard (a downstream stage receiving a populated list injects nothing -- the 1/2/3-arm self-kill), the last-rank no-op deferral, that a human RPC is never deferred, that abandonment never raises into the event loop, and that the join deadline actually reaches the channel. Operational acceptance under load NOT yet claimed; the boot carrying this is running.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 27, 2026
…stead of clobbering it The re-issue this ticket asked for already existed. A cutover retracts every resident; the retracted request is re-queued through _add_request_to_queue, which calls _prefetch_kvcache, which re-runs prefetch_from_storage -- a fresh radix walk under the CURRENT binding generation, allocating a fresh host span from the currently bound pool. Building a second requeue path would have been redundant machinery on top of a working one. What did not exist was a guard on the registration. prefetch_from_storage ended with an unconditional `self.ongoing_prefetch[req_id] = ...`, so when the post-cutover registration arrived while the pre-cutover one was still ongoing, the incumbent was overwritten outright: its host_indices were lost to every owner (nothing else holds them) and its anchor_lock_params lock ref was never decremented, leaving that node PROTECTED for good. The next cutover's drop_prefix_tree_returning_rows then orphans exactly those device rows -- one request's allocation per cutover, which is the sgl-project#938 forensic shape, and it needs no in-flight write-through to explain it. RETIRE, DO NOT REAP. Freeing the displaced span at the registration site is not available: whether it is safe to release is answered by can_terminate_prefetch, which checks pool_transfers_done and is COLLECTIVE, and HiCacheController.terminate_prefetch is only mark_terminate() -- a flag, not a join -- so the transfer thread may still be writing into that span (sgl-project#913 IMA family). Running that collective there is not available either: whether a clobber happens is RANK-LOCAL, and the site already sits inside the sgl-project#580 participation region. So the record is retired -- terminated, moved out of the req_id slot, nothing freed -- and drain_retired_prefetch reaps it from the per-round site, where collectives are legal. That also DEFINES the ordering instead of leaving it to timing: the re-issue owns the slot, so a late completion resolves through check_prefetch_progress to the new record and cannot reach the displaced one, which is reachable only from the reap. Neither insert nor free happens on a retired record anywhere else. UNIFORMITY IS ENFORCED, NOT INFERRED. It is tempting to argue the retired list is rank-uniform (retirement follows a registration that only happens under the sgl-project#580 positive consensus), but this module refuses arguments of that shape elsewhere -- _prefetch_done_for RAISES rather than assume a request is in the replicated set -- and a wrong inference here hangs three ranks. Both halves are enforced instead: candidates are taken in sorted req_id order (canonical, never insertion order), and the ranks AGREE on the candidate before anyone reaps it, via one all_reduce contributing [d, -d] under MIN so min(d) and max(d) arrive in a single collective. Reaping proceeds only on min == max != 0. A rank that has not retired it yet contributes 0, agreement fails, and the record waits a round. A latecomer delays a reap; it never wedges one. The collective is UNCONDITIONAL -- gating it on a non-empty local list would be the rank-local participation predicate this whole design exists to avoid. Empty contributes zeros; the common case is one two-element all_reduce per round. Not micro-optimised into an existing round collective: correctness first, measure before optimising. The re-fetch budget is a REPORT, not a gate (sgl-project#939 RE-FETCH BUDGET SPENT ... recomputing). Refusing a re-issue would be a rank-local decision taken inside the participation region, which is the one thing this path may not do; the line exists so a cutover cadence faster than a fetch completes is named instead of surfacing only as an unexplained cached=0. Red-first, single process (13 behavioural + wiring pins): retire does not free, the span returns to its MINTING pool and never to the bound one, the drain empties the list with no further event from the request at all (otherwise this trades a clobber for unbounded growth), a completion cannot touch a retired record, and the registration retires before it assigns. Mutants killed: retire frees immediately (5 red), the clobber restored (1 red). MULTI-RANK, and I had this wrong first. I recorded "the agreement only bites multi-rank, which a CPU test can't exercise" -- false, and the user caught it: three gloo processes on localhost exercise it fine, as sgl-project#630/sgl-project#650/sgl-project#653/sgl-project#899 already do here. Only a SINGLE-PROCESS test cannot. The new gloo falsifier runs the real drain and the real _all_reduce_attn_groups over three real ranks with DIVERGENT retired lists. With the agreement every rank reports rounds [0, 1] -- nobody reaps while rank 2 is behind, everybody reaps once it catches up. Removing it does NOT hang, which is worse and is why this needed measuring rather than predicting: gloo pairs ranks 0/1's round-1 all_reduce with rank 2's round-2 one -- same group, same shape, nothing to distinguish them -- so no rank blocks, every rank reports success, and the ranks silently reap in different rounds. That is the sgl-project#580 signature, not a deadlock, and the next collective carrying a differently-shaped payload is the one that corrupts. The test asserts on that divergence (and accepts a timeout too, since another backend may block). Verified to kill the production mutant only once BOTH agreement gates are removed -- the second, `digest != agreed_min`, independently covers the all-but-one-rank case, so a partial mutation leaves it alive. Also fixed here, and it was mine: test_prefetch_participation_desync_580's cache stand-in drives prefetch_from_storage as a real bound method and had no _retire_ongoing_prefetch, so the new call broke 4 of its tests. Carried as the REAL bound method rather than a stub -- production instances always have it, so a defensive getattr would have hidden a genuine missing-method bug. Tests: test/registered/unit/mem_cache/ plus the gloo module -- 2875 passed, 3 failed (test_acceptance_emitters_758 RefillTiming x2, test_kv_row_ownership_822 TestCallSiteAdapters), all three identical by name and count on the pristine file at HEAD, i.e. pre-existing. ruff clean on both new modules and unchanged on the two production files (unified_radix_cache 0/0, scheduler 103/103 pre-existing); black clean. The gloo module spawns real ranks, so it belongs in the narrow RANKS lane. gate_partition.tsv is generated and sha256-checked, so it is not hand-edited here; gate_partition_build.py has to be re-run for the classification to land. Stage-A interaction, so a reader does not misdiagnose it: a retired-but-unreaped record still holds its lock ref, so sgl-project#938 PROTECTED RESIDUE ORPHANED can show it TRANSIENTLY. Healthy is a counter that RETURNS across reap rounds; monotone growth is still a leak.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 30, 2026
…lt ein TRIGGER. Aktionsarm ist S. Reiner Lese-Pass, kein Boot, kein Code. Beide Quellen je Zeile. == PFLICHT-VORFRAGE BEANTWORTET: es ist ein UNGEDECKTER TRANSPORT == Das haengende all_reduce laeuft auf `grp = self.tp_cpu_group` (scheduler.py:5964) -- eine GLOO-CPU-Gruppe. Die sgl-project#650/sgl-project#653-Klassen (Bar1CollectiveAborted / HostCollectiveAborted / Bar1CollectiveStalled) liegen ausschliesslich in `distributed/device_communicators/barlink_*.py` plus den zwei debug_utils-Lesern: sie decken den DEVICE/BAR1-Pfad. Also NICHT "Deckung da, Schwellwert/Leser defekt", sondern: der Pfad hat diese Deckung nie gehabt. Auf dem gloo-Pfad gibt es weder Deadline noch Peer-Positions-Bericht. == CENSUS-FAEHIGKEIT: IDENTITAET IST DA, und mehr == `_history.append((family, nbytes))` (:246) -- jede Eintragung traegt die KOLLEKTIV-FAMILIE, nicht nur Zaehlstaende. `Divergence(family, counts)` (:459) vergleicht bereits NACH FAMILIE. Und der Modulkopf beschreibt zwei Berichtspfade: 1. `compare_across_ranks` -- Detektor, laeuft auf gloo `tp_cpu_group`, replizierte Kadenz, feuert waehrend die Raenge noch gesund sind. 2. `format_local_census` -- die WEDGE-FESTE Haelfte: NIMMT KEIN KOLLEKTIV, jeder Rang kippt seine EIGENEN Zaehler ins Log, die drei Dumps werden hinterher gediffed. Die Identitaet fehlt also NICHT. Das ist nicht die Luecke. == DIE LUECKE, praezise == Der Modulkopf sagt woertlich, der Detektor laufe auf dem CPU-Pfad, "NEVER the device/BAR1 path, SO A WEDGED DEVICE GROUP DOES NOT DISABLE THE INSTRUMENT THAT IS SUPPOSED TO EXPLAIN IT". Genau diese Immunitaet greift bei uns NICHT: unser Wedge IST auf der CPU-Gruppe. Der Detektor braucht ein `all_gather_object` auf ebendieser Gruppe -- er kann eine Uneinigkeit der Gruppe, die er selbst benutzt, nicht melden. Die Schutz-Annahme ist um genau einen Transport verschoben. Und die wedge-feste Haelfte (`format_local_census`, kollektiv-frei) EXISTIERT fuer diesen Fall -- aber sie wird nie AUSGELOEST, weil auf dem gloo-Pfad nichts abbricht: kein Deadline, kein Abort, kein Ausloeser. DAMIT: die Luecke ist ein TRIGGER, keine Faehigkeit. == AKTIONSARM-DESIGN, S == Diskriminator: NICHT "Follower wartet lange" (Normalzustand) und NICHT "PP0 macht keinen Fortschritt" (PP0 wartet selbst) -- sondern eine BEGRENZTE Wartezeit am gloo-Kollektiv, nach deren Ablauf jeder Rang `format_local_census()` ruft (kollektiv-frei, schon gebaut) und die Gruppe per CRASH/STOP stirbt. Die Beider-Seiten-Diagnose faellt dabei GRATIS an: jeder Rang druckt seine eigene Familien-Historie, und der bestehende Hinweistext sagt bereits, was damit zu tun ist -- "Diff this line against the peers' to find where the sequences diverged". DEADLINE-BEGRUENDUNG gegen die zwei Lehren: * sgl-project#630 (wait() treiben, nie is_completed() pollen): der bounded wait muss ueber das async work-handle mit Timeout laufen, NICHT ueber eine Poll-Schleife. * sgl-project#615 (Build-Fenster-Dehnung): die Deadline darf nicht in ein legitimes JIT-Cold-Build-Fenster feuern (bis SGLANG_BARLINK_BUILD_WINDOW_CAP_S=900s). Sie muss dasselbe Fenster konsultieren, das barlink schon fuehrt -- sonst ersetzt sie einen Wedge durch einen Fehlalarm beim ersten Boot. KEIN timeout-und-weiter: Ablauf = Tod der Gruppe, per Rang-Gesetz. == VERDIKT: S == Wiederverwendet wird durchweg Vorhandenes (Census-Familien-Historie, kollektiv-freier Dump, barlink-Build-Fenster). Neu ist nur der bounded wait plus sein Ausloeser. Kein neues Kollektiv auf dem Pfad -- das Gesetz bleibt gewahrt, weil die Diagnose-Haelfte ausdruecklich kollektiv-frei ist. OFFEN und ehrlich: die Deadline-ZAHL ist nicht herleitbar aus dem, was ich gemessen habe -- sie braucht die legitime Maximal-Wartezeit unter Last, und die kenne ich nur als "DE-06 dauerte 95,6 s im gesunden Lauf". Das ist die eine Groesse, die vor dem Bau gemessen gehoert. BELEG-STUFE: DESK-BEWIESEN (Transport-Zuordnung, Deckungs-Scope, Census- Faehigkeit -- alle am Pin gelesen, Modulkopf woertlich zitiert).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
detokenizer_managerachieves very low performance when 1) streaming is enabled, 2) model forward is very swift.Changed the decoding process into incremental detokenization in detokenizer_manager, and it is not compatible with mutable output_ids (when jump forward).