Skip to content

Add benchmark scripts - #476

Merged
Ying1123 merged 2 commits into
mainfrom
bench
May 26, 2024
Merged

Ying1123 merged 2 commits into
mainfrom
bench

Conversation

@Ying1123

Copy link
Copy Markdown
Contributor

No description provided.

@Ying1123
Ying1123 force-pushed the bench branch 6 times, most recently from 1e3bfdb to dfc4c0b Compare May 26, 2024 19:07
@Ying1123
Ying1123 merged commit 947bda7 into main May 26, 2024
@Ying1123
Ying1123 deleted the bench branch May 26, 2024 19:09
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
EdwardXuy pushed a commit to shun8686/sglang that referenced this pull request May 9, 2026
efschu added a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…th collective

sgl-project#517 named five BAR1 broadcasts on the HOST path per NEXTN decode round.
Verified against the current tree, that inventory is now THREE, not five:
the three verify-result syncs were fused into one by pack_accept_payload
(#616c). The remaining two are the draft-pick sync in
_draft_extend_for_decode, which capture_safe_tp_broadcast still issues one
collective PER tensor from its loop.

This fuses that pair into one buffer. The picks have mixed dtypes
(topk_index integer, topk_p float32 at the SAME shape), so unlike the
int32 accept payload the fusion is byte-level. Contiguity is required
rather than coerced: reshape(-1) on a non-contiguous tensor returns a
copy, so an in-place unpack through it would silently drop the broadcast.

fuse is opt-in per call site and defaults OFF. _broadcast_draft_picks is
also called from inside the draft CUDA-graph capture (draft_forward), and
enabling fusion globally would change what a captured region allocates --
a capture change. Only the site that is both per-decode-round and outside
any captured region opts in; the prefill, catchup and in-loop sites stay
byte-identical.

Not claimed here: a ms/round figure. sgl-project#476's 6.64 pp Seam A cost was
measured with the pre-sgl-project#517 guard, which synchronised once per collective;
sgl-project#517 removed that sync, so the per-collective price must be re-measured on
metal rather than carried over. The A/B is window-gated.

Tests: test_draft_pick_fusion_524.py, 7 passed (red first: the three
symbols did not exist). Regression over test/registered/unit/debug
-k "616 or spec_util or draft_pick or eagle or broadcast": 351 passed,
4 failed. Those 4 are pre-existing and unrelated -- verified by reverting
both source files and re-running: test_barlink_bar1_abort_poll_616f.py
fails identically on the clean tree (SimpleNamespace missing _round_dev,
a sgl-project#624-class stub drift). ruff check clean on both changed files (the one
F401 in eagle_worker_v2.py is pre-existing on HEAD); ruff format applied.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants