Repository navigation
Conversation
Ying1123
force-pushed
the
bench
branch
6 times, most recently
from
May 26, 2024 19:07
1e3bfdb to
dfc4c0b
Compare
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
EdwardXuy
pushed a commit
to shun8686/sglang
that referenced
this pull request
May 9, 2026
Submit performance test code
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…th collective sgl-project#517 named five BAR1 broadcasts on the HOST path per NEXTN decode round. Verified against the current tree, that inventory is now THREE, not five: the three verify-result syncs were fused into one by pack_accept_payload (#616c). The remaining two are the draft-pick sync in _draft_extend_for_decode, which capture_safe_tp_broadcast still issues one collective PER tensor from its loop. This fuses that pair into one buffer. The picks have mixed dtypes (topk_index integer, topk_p float32 at the SAME shape), so unlike the int32 accept payload the fusion is byte-level. Contiguity is required rather than coerced: reshape(-1) on a non-contiguous tensor returns a copy, so an in-place unpack through it would silently drop the broadcast. fuse is opt-in per call site and defaults OFF. _broadcast_draft_picks is also called from inside the draft CUDA-graph capture (draft_forward), and enabling fusion globally would change what a captured region allocates -- a capture change. Only the site that is both per-decode-round and outside any captured region opts in; the prefill, catchup and in-loop sites stay byte-identical. Not claimed here: a ms/round figure. sgl-project#476's 6.64 pp Seam A cost was measured with the pre-sgl-project#517 guard, which synchronised once per collective; sgl-project#517 removed that sync, so the per-collective price must be re-measured on metal rather than carried over. The A/B is window-gated. Tests: test_draft_pick_fusion_524.py, 7 passed (red first: the three symbols did not exist). Regression over test/registered/unit/debug -k "616 or spec_util or draft_pick or eagle or broadcast": 351 passed, 4 failed. Those 4 are pre-existing and unrelated -- verified by reverting both source files and re-running: test_barlink_bar1_abort_poll_616f.py fails identically on the clean tree (SimpleNamespace missing _round_dev, a sgl-project#624-class stub drift). ruff check clean on both changed files (the one F401 in eagle_worker_v2.py is pre-existing on HEAD); ruff format applied.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.