Skip to content

Detokenize incrementally when streaming - #653

Merged
hnyls2002 merged 4 commits into
mainfrom
fix-streaming-detokenizer
Jul 19, 2024
Merged

hnyls2002 merged 4 commits into
mainfrom
fix-streaming-detokenizer

Conversation

@hnyls2002

@hnyls2002 hnyls2002 commented Jul 19, 2024 •

Copy link
Copy Markdown
Collaborator

The detokenizer_manager achieves very low performance when 1) streaming is enabled, 2) model forward is very swift.

Changed the decoding process into incremental detokenization in detokenizer_manager, and it is not compatible with mutable output_ids (when jump forward).

@hnyls2002
hnyls2002 merged commit a9ef49c into main Jul 19, 2024
@hnyls2002
hnyls2002 deleted the fix-streaming-detokenizer branch July 19, 2024 00:57
@zhyncs

zhyncs commented Jul 19, 2024 •

Copy link
Copy Markdown
Contributor

@hnyls2002 Nice work! cc @merrymercy @Ying1123 @yzh119

model: Llama 3 8B Instruct
hardware: A100 80G

# server
python3 -m sglang.launch_server --model /root/Meta-Llama-3-8B-Instruct --trust-remote-code --port 23333 --disable-radix-cach --stream-interval 1

# client
python3 benchmark_serving.py --backend openai --host 127.0.0.1 --port 23333 --dataset /root/ShareGPT_V3_unfiltered_cleaned_split.json --model /root/Meta-Llama-3-8B-Instruct --tokenizer /root/Meta-Llama-3-8B-Instruct --num-prompts 1000 --request-rate 128 --trust-remote-code

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 8, 2026
…e, never __trap

The last live instance of the sgl-project#583 context-killing mechanism: wait_ge in
the host-staged transport still called __trap() on spin expiry, which
destroys the CUDA context and surfaces as sticky 'unspecified launch
failure' at whatever unrelated call site runs next. Unreachable in every
production configuration (host is explicitly-selected, no-fallback,
bar1 pinned) — but a landmine for any host-transport measurement boot.

Ported the device transport's sgl-project#583 fix honestly, not minimally:
- wait_ge returns bool and writes a per-site code into the abort word on
  expiry (put reuse-guard=1, reduce peer-wait=2, copyout wait=3); the
  BARLINK_HOST_TRAP macro and its hipify plumbing are deleted.
- Block-uniform exit: shared abortS written by the spinning thread 0,
  __syncthreads(), all threads return together before any payload loop,
  fence, or last_block barrier — verified at all three sites. Terminal-
  abort contract documented: cross-block skew and stale blk_ctr are
  acceptable residue of a process the host is about to kill by raise.
- The transport's THREE sequence-counter families (collective, per-peer
  send, per-peer recv) are unified into one zero-initialized backing
  tensor with 2-word rows — the spec's naive grow-to-2 would have made
  the p2p kernels' abort store corrupt a NEIGHBOR's sequence number, and
  the unified shape makes check_aborted a single read over every abort
  word the transport can produce.
- HostCollectiveAborted mirrors DeviceCollectiveAborted (rank/world/
  code/where, garbage-results contract) and appends the sgl-project#650 peer
  statement. check_aborted is a plain in-line read (documented choice:
  this is a measurement vehicle, not the bs=1 hot path) wired twice:
  the duck-typed _after_transport seam (sgl-project#431) for eager collectives,
  and abort-gate registration for replay-only collectives — host is
  capturable, so a graph-resident collective never reaches the seam
  (the same gap sgl-project#583 closed for the device transport).
- The load-bearing reachability claim is pinned: 'host' is in the
  no-fallback set via the real parallel_state lookup, and the dispatch
  seam is proven transport-stable mid-run — an explicit
  SGLANG_BARLINK_TRANSPORT=host boot is the only way this code runs.

Falsifiers, red-first (executed against the pre-fix source inside the
test): old wait_ge provably trapped and returned void, old seq tensor
had no abort-word room; fixed source has zero traps, bool returns, all
three consuming sites, distinct codes. nvcc compile smoke rc=0 (sm_86).
GPU-gated on-card falsifier written and documented (put-kernel forced
timeout must set the word, leave the context ALIVE, and raise named)
— skips at the desk by design.

Counting basis: 35 passed + 1 GPU-skip + 19 subtests in the new file;
full battery with abort/stub-drift/sentinel regression suites 81 passed
+ 1 skipped + 19 subtests (97.9 s, my rerun); ruff and imports clean.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 8, 2026
…port wait_ge

wait_ge in the host transport writes a per-site abort code and returns
instead of trapping the context; block-uniform exit via shared flag +
barrier at all three sites; HostCollectiveAborted mirrors the structured
device/bar1 shape and carries the sgl-project#650 peer statement. Also closes the
capturable-transport abort-gate wiring gap for host, and unifies the
abort/seq words into one zero-initialized backing tensor (the spec's
grow-to-2 layout would have corrupted p2p slice neighbors).
Falsifiers red-first against the pre-fix source; 35 passed + 1 GPU-skip
+ 19 subtests new; 81 passed full battery.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 9, 2026
…keeps cycling

Boot 10 showed the arm delivery finally correct -- all three ranks armed
identically, zero control messages -- and it still wedged:

  rank0:   on_round -> bounded_collective        (flip reduction)
  rank1/2: _pull_raw_reqs -> point_to_point_pyobj (chain recv)

Ranks 1 and 2 DID arm and DID reach on_round, but were not parked yet, so
they returned and came back to the chain recv, which is an UNBOUNDED
wait. Rank 0's park predicate went true first; it entered the blocking
reduction -- and entering it stops rank 0 forwarding the very batch its
peers are blocked on. Nobody can join, and rank 0 cannot time out because
the park deadline is evaluated before entry, never from inside the wait.

The existing safety argument ("a parked rank owes no pipeline send")
covers hidden-state sends. It does NOT cover the request-chain forward,
which a parked rank still owes its downstream stage every iteration.

An unbounded wait in a coordination path is also against this fork's
bounded-wait canon (sgl-project#610/sgl-project#615/sgl-project#653). Bound it: a stage with nothing to
receive returns empty, re-reaches its own on_round -- by then parked --
and joins. Idle coordination then needs no chain message at all, which is
what lets the policy stay message-free.

THE INVARIANT, and the reason this is not a one-line timeout: a posted
irecv is a standing promise to the sender, which will deliver into that
exact tensor. It can never be abandoned on a deadline. The Work and its
tensor therefore live in chain_recv_state and are re-polled across calls;
a second irecv is never posted while one is outstanding. Only the SIZE
header is polled -- once it lands the sender is committed to the payload,
so the body keeps the ordinary blocking wait.

Gated on chain_recv_poll_s, wired to 50 ms only when the policy is on.
Every other boot keeps the upstream unbounded path byte-identical.

Tests: 317 passed in the sgl-project#631 flip family, CPU-only. New: a bounded recv
returns empty rather than blocking and never waits an incomplete Work;
three polls of one pending receive post exactly ONE irecv (the
desynchronisation falsifier); a landed size==0 clears the state; poll 0.0
keeps the upstream path.

Also recorded: the first VALID sustained load of this task -- 4 min
agent-like mixed traffic (1 long-prefill worker at 8192/32768 + 3 decode
workers, bs<=4) against the manual-policy server: 52 requests, 52 ok,
0 aborted, post-idle 32768-tok probe 4383.7 tok/s. Earlier load attempts
measured wedged servers and were no measurement at all.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 9, 2026
Restores the arm as a chain message and fixes the ORDER that made it
deadlock, rather than removing the message. The message-free variant was
built and measured (boot 10): every rank reached an identical verdict and
armed itself with zero messages -- and it wedged solid. arms=3,
cutovers=0, 0 % GPU, /generate dead at 45 s.

THE MECHANISM, which inverts the earlier reading: THE ARM MESSAGE IS THE
WAKEUP. Ranks 1..n-1 spend idle time BLOCKED in the chain recv, and
forwarding the arm is the only thing that makes them runnable. That is
why the manual POST /phase_flip has always worked (~1.2 s cutovers,
reproduced all session) and why removing the channel removed the
synchronisation. The hazard was never the channel; it was the order.

THE INVARIANT (named in code): no rank may enter the flip's BLOCKING
consensus reduction while it still owes, or has uncommitted, chain sends.
A rank acts on an arm only in the pass AFTER _pp_commit_comm_work has
committed the forward carrying it downstream. Intermediate stages obey
the same rule on receipt, so the wake propagates strictly ahead of every
rank's block; the last rank owes no forward and its deferral is a
one-pass no-op. With this ordering the automatic path is byte-for-byte
the manual one -- same request, same chain, same wake, same
drain-park-meet -- and only the originator differs (policy on rank 0 vs
an HTTP caller). That equivalence with a mechanism proven on metal is the
design's strongest argument.

Two alternatives stay rejected, both measured: a SYNCHRONOUS forward
deadlocks worse (a peer need not be in the chain recv at all; found
blocked in the hidden-states exchange while this rank blocked in send),
and BOUNDING the chain recv decouples a 1:1 send/consume contract so the
consumer can skip a consume, unmatched sends accumulate and the SENDERS
block (reverted in 3c0115fd6d; its posted-irecv-is-a-promise test is kept
as the artifact).

(c) BOUNDED JOIN, unconditional backstop: entry to the consensus
reduction is now a bounded wait that abandons from INSIDE, disarms, logs
loudly and returns to cycling, so the policy may re-arm. Every other flip
deadline is evaluated BEFORE entry, so a rank that got in and found no
peers previously waited for ever -- an unbounded wait in a coordination
path, against sgl-project#610/sgl-project#615/sgl-project#653. The consensus channel now forwards a
per-caller timeout_s to bounded_collective. Default 45 s, generous by
intent: a peer draining a long prefill is normal and must not trip it.
This is a wedge breaker, not a latency control.

Tests: 317 passed in the sgl-project#631 flip family, CPU-only. Can-fail proven
against the boot-10 specimen: reintroducing arm-in-same-pass turns all
three delivery-before-block tests red with "acted on the arm in the pass
that forwarded it". Also pinned: the origin guard (a downstream stage
receiving a populated list injects nothing -- the 1/2/3-arm self-kill),
the last-rank no-op deferral, that a human RPC is never deferred, that
abandonment never raises into the event loop, and that the join deadline
actually reaches the channel.

Operational acceptance under load NOT yet claimed; the boot carrying this
is running.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 27, 2026
…stead of clobbering it

The re-issue this ticket asked for already existed. A cutover retracts every
resident; the retracted request is re-queued through _add_request_to_queue,
which calls _prefetch_kvcache, which re-runs prefetch_from_storage -- a fresh
radix walk under the CURRENT binding generation, allocating a fresh host span
from the currently bound pool. Building a second requeue path would have been
redundant machinery on top of a working one.

What did not exist was a guard on the registration. prefetch_from_storage ended
with an unconditional `self.ongoing_prefetch[req_id] = ...`, so when the
post-cutover registration arrived while the pre-cutover one was still ongoing,
the incumbent was overwritten outright: its host_indices were lost to every
owner (nothing else holds them) and its anchor_lock_params lock ref was never
decremented, leaving that node PROTECTED for good. The next cutover's
drop_prefix_tree_returning_rows then orphans exactly those device rows -- one
request's allocation per cutover, which is the sgl-project#938 forensic shape, and it needs
no in-flight write-through to explain it.

RETIRE, DO NOT REAP. Freeing the displaced span at the registration site is not
available: whether it is safe to release is answered by can_terminate_prefetch,
which checks pool_transfers_done and is COLLECTIVE, and
HiCacheController.terminate_prefetch is only mark_terminate() -- a flag, not a
join -- so the transfer thread may still be writing into that span (sgl-project#913 IMA
family). Running that collective there is not available either: whether a
clobber happens is RANK-LOCAL, and the site already sits inside the sgl-project#580
participation region. So the record is retired -- terminated, moved out of the
req_id slot, nothing freed -- and drain_retired_prefetch reaps it from the
per-round site, where collectives are legal.

That also DEFINES the ordering instead of leaving it to timing: the re-issue
owns the slot, so a late completion resolves through check_prefetch_progress to
the new record and cannot reach the displaced one, which is reachable only from
the reap. Neither insert nor free happens on a retired record anywhere else.

UNIFORMITY IS ENFORCED, NOT INFERRED. It is tempting to argue the retired list
is rank-uniform (retirement follows a registration that only happens under the
sgl-project#580 positive consensus), but this module refuses arguments of that shape
elsewhere -- _prefetch_done_for RAISES rather than assume a request is in the
replicated set -- and a wrong inference here hangs three ranks. Both halves are
enforced instead: candidates are taken in sorted req_id order (canonical, never
insertion order), and the ranks AGREE on the candidate before anyone reaps it,
via one all_reduce contributing [d, -d] under MIN so min(d) and max(d) arrive in
a single collective. Reaping proceeds only on min == max != 0. A rank that has
not retired it yet contributes 0, agreement fails, and the record waits a round.
A latecomer delays a reap; it never wedges one. The collective is UNCONDITIONAL
-- gating it on a non-empty local list would be the rank-local participation
predicate this whole design exists to avoid. Empty contributes zeros; the common
case is one two-element all_reduce per round. Not micro-optimised into an
existing round collective: correctness first, measure before optimising.

The re-fetch budget is a REPORT, not a gate (sgl-project#939 RE-FETCH BUDGET SPENT ...
recomputing). Refusing a re-issue would be a rank-local decision taken inside
the participation region, which is the one thing this path may not do; the line
exists so a cutover cadence faster than a fetch completes is named instead of
surfacing only as an unexplained cached=0.

Red-first, single process (13 behavioural + wiring pins): retire does not free,
the span returns to its MINTING pool and never to the bound one, the drain
empties the list with no further event from the request at all (otherwise this
trades a clobber for unbounded growth), a completion cannot touch a retired
record, and the registration retires before it assigns. Mutants killed: retire
frees immediately (5 red), the clobber restored (1 red).

MULTI-RANK, and I had this wrong first. I recorded "the agreement only bites
multi-rank, which a CPU test can't exercise" -- false, and the user caught it:
three gloo processes on localhost exercise it fine, as sgl-project#630/sgl-project#650/sgl-project#653/sgl-project#899
already do here. Only a SINGLE-PROCESS test cannot. The new gloo falsifier runs
the real drain and the real _all_reduce_attn_groups over three real ranks with
DIVERGENT retired lists. With the agreement every rank reports rounds [0, 1] --
nobody reaps while rank 2 is behind, everybody reaps once it catches up.

Removing it does NOT hang, which is worse and is why this needed measuring
rather than predicting: gloo pairs ranks 0/1's round-1 all_reduce with rank 2's
round-2 one -- same group, same shape, nothing to distinguish them -- so no rank
blocks, every rank reports success, and the ranks silently reap in different
rounds. That is the sgl-project#580 signature, not a deadlock, and the next collective
carrying a differently-shaped payload is the one that corrupts. The test asserts
on that divergence (and accepts a timeout too, since another backend may block).
Verified to kill the production mutant only once BOTH agreement gates are
removed -- the second, `digest != agreed_min`, independently covers the
all-but-one-rank case, so a partial mutation leaves it alive.

Also fixed here, and it was mine: test_prefetch_participation_desync_580's cache
stand-in drives prefetch_from_storage as a real bound method and had no
_retire_ongoing_prefetch, so the new call broke 4 of its tests. Carried as the
REAL bound method rather than a stub -- production instances always have it, so
a defensive getattr would have hidden a genuine missing-method bug.

Tests: test/registered/unit/mem_cache/ plus the gloo module -- 2875 passed, 3
failed (test_acceptance_emitters_758 RefillTiming x2, test_kv_row_ownership_822
TestCallSiteAdapters), all three identical by name and count on the pristine
file at HEAD, i.e. pre-existing. ruff clean on both new modules and unchanged on
the two production files (unified_radix_cache 0/0, scheduler 103/103
pre-existing); black clean.

The gloo module spawns real ranks, so it belongs in the narrow RANKS lane.
gate_partition.tsv is generated and sha256-checked, so it is not hand-edited
here; gate_partition_build.py has to be re-run for the classification to land.

Stage-A interaction, so a reader does not misdiagnose it: a retired-but-unreaped
record still holds its lock ref, so sgl-project#938 PROTECTED RESIDUE ORPHANED can show it
TRANSIENTLY. Healthy is a counter that RETURNS across reap rounds; monotone
growth is still a leak.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 30, 2026
…lt ein TRIGGER. Aktionsarm ist S.

Reiner Lese-Pass, kein Boot, kein Code. Beide Quellen je Zeile.

== PFLICHT-VORFRAGE BEANTWORTET: es ist ein UNGEDECKTER TRANSPORT ==
Das haengende all_reduce laeuft auf `grp = self.tp_cpu_group`
(scheduler.py:5964) -- eine GLOO-CPU-Gruppe. Die sgl-project#650/sgl-project#653-Klassen
(Bar1CollectiveAborted / HostCollectiveAborted / Bar1CollectiveStalled)
liegen ausschliesslich in `distributed/device_communicators/barlink_*.py`
plus den zwei debug_utils-Lesern: sie decken den DEVICE/BAR1-Pfad.
Also NICHT "Deckung da, Schwellwert/Leser defekt", sondern: der Pfad hat
diese Deckung nie gehabt. Auf dem gloo-Pfad gibt es weder Deadline noch
Peer-Positions-Bericht.

== CENSUS-FAEHIGKEIT: IDENTITAET IST DA, und mehr ==
`_history.append((family, nbytes))` (:246) -- jede Eintragung traegt die
KOLLEKTIV-FAMILIE, nicht nur Zaehlstaende. `Divergence(family, counts)`
(:459) vergleicht bereits NACH FAMILIE. Und der Modulkopf beschreibt zwei
Berichtspfade:
  1. `compare_across_ranks` -- Detektor, laeuft auf gloo `tp_cpu_group`,
     replizierte Kadenz, feuert waehrend die Raenge noch gesund sind.
  2. `format_local_census` -- die WEDGE-FESTE Haelfte: NIMMT KEIN KOLLEKTIV,
     jeder Rang kippt seine EIGENEN Zaehler ins Log, die drei Dumps werden
     hinterher gediffed.
Die Identitaet fehlt also NICHT. Das ist nicht die Luecke.

== DIE LUECKE, praezise ==
Der Modulkopf sagt woertlich, der Detektor laufe auf dem CPU-Pfad,
"NEVER the device/BAR1 path, SO A WEDGED DEVICE GROUP DOES NOT DISABLE THE
INSTRUMENT THAT IS SUPPOSED TO EXPLAIN IT".
Genau diese Immunitaet greift bei uns NICHT: unser Wedge IST auf der
CPU-Gruppe. Der Detektor braucht ein `all_gather_object` auf ebendieser
Gruppe -- er kann eine Uneinigkeit der Gruppe, die er selbst benutzt, nicht
melden. Die Schutz-Annahme ist um genau einen Transport verschoben.
Und die wedge-feste Haelfte (`format_local_census`, kollektiv-frei) EXISTIERT
fuer diesen Fall -- aber sie wird nie AUSGELOEST, weil auf dem gloo-Pfad
nichts abbricht: kein Deadline, kein Abort, kein Ausloeser.
DAMIT: die Luecke ist ein TRIGGER, keine Faehigkeit.

== AKTIONSARM-DESIGN, S ==
Diskriminator: NICHT "Follower wartet lange" (Normalzustand) und NICHT "PP0
macht keinen Fortschritt" (PP0 wartet selbst) -- sondern eine BEGRENZTE
Wartezeit am gloo-Kollektiv, nach deren Ablauf jeder Rang
`format_local_census()` ruft (kollektiv-frei, schon gebaut) und die Gruppe
per CRASH/STOP stirbt. Die Beider-Seiten-Diagnose faellt dabei GRATIS an:
jeder Rang druckt seine eigene Familien-Historie, und der bestehende
Hinweistext sagt bereits, was damit zu tun ist -- "Diff this line against
the peers' to find where the sequences diverged".
DEADLINE-BEGRUENDUNG gegen die zwei Lehren:
  * sgl-project#630 (wait() treiben, nie is_completed() pollen): der bounded wait muss
    ueber das async work-handle mit Timeout laufen, NICHT ueber eine
    Poll-Schleife.
  * sgl-project#615 (Build-Fenster-Dehnung): die Deadline darf nicht in ein legitimes
    JIT-Cold-Build-Fenster feuern (bis SGLANG_BARLINK_BUILD_WINDOW_CAP_S=900s).
    Sie muss dasselbe Fenster konsultieren, das barlink schon fuehrt --
    sonst ersetzt sie einen Wedge durch einen Fehlalarm beim ersten Boot.
KEIN timeout-und-weiter: Ablauf = Tod der Gruppe, per Rang-Gesetz.

== VERDIKT: S ==
Wiederverwendet wird durchweg Vorhandenes (Census-Familien-Historie,
kollektiv-freier Dump, barlink-Build-Fenster). Neu ist nur der bounded wait
plus sein Ausloeser. Kein neues Kollektiv auf dem Pfad -- das Gesetz bleibt
gewahrt, weil die Diagnose-Haelfte ausdruecklich kollektiv-frei ist.
OFFEN und ehrlich: die Deadline-ZAHL ist nicht herleitbar aus dem, was ich
gemessen habe -- sie braucht die legitime Maximal-Wartezeit unter Last, und
die kenne ich nur als "DE-06 dauerte 95,6 s im gesunden Lauf". Das ist die
eine Groesse, die vor dem Bau gemessen gehoert.

BELEG-STUFE: DESK-BEWIESEN (Transport-Zuordnung, Deckungs-Scope, Census-
Faehigkeit -- alle am Pin gelesen, Modulkopf woertlich zitiert).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants