Skip to content

Improve tensor parallel performance - #625

Merged
merrymercy merged 13 commits into
mainfrom
bench
Jul 15, 2024
Merged

merrymercy merged 13 commits into
mainfrom
bench

Conversation

@Ying1123

Copy link
Copy Markdown
Contributor

No description provided.

@merrymercy
merrymercy merged commit 6a2941f into main Jul 15, 2024
@merrymercy
merrymercy deleted the bench branch July 15, 2024 14:10
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
Co-authored-by: Mingyi <wisclmy0611@gmail.com>
Jiminator added a commit to Jiminator/sglang that referenced this pull request May 28, 2026
Adds an investigation report for the nightly-test-perf-4-gpu-b200 failure of
test_a_gsm8k under TestDeepseekV3FP4CuteDSLMoE. The regression is traced to
commit 044649c (PR sgl-project#22669) which under-sizes the CuteDslMoEWrapper buffer
for the EP=1, TP=4, no-DP-attention DeepSeek V3 FP4 configuration.

The report includes:
- pass/fail boundary (run sgl-project#624 pass -> sgl-project#625 fail on main)
- bisection across the 50 commits in 7f154ba..c4a7d12
- root-cause diff in flashinfer_cutedsl.py:ensure_cutedsl_wrapper
- direct repro at offending and parent SHAs (GPU 4 only)
- a four-line recommended patch and short-term workarounds

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
efschu added a commit to efschu/htsglang that referenced this pull request Aug 7, 2026
… route pricing, PP-vs-TP prefill measurement
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 7, 2026
…contract, path B verdict

Seven commits from the sgl-project#625 Route A line, all desk-proven, none needing cards.

#631a guard 1 (57e13eb): a PD pair whose arms hold different weights is
refused, on model_identity_hash only with include_parallel_vectors=False so a
PP prefill and a TP+DCP decode on the same weights still pair. Surfaced and
fixed a latent TypeError: revision and quantization were the only identity
parts not str()-coerced, harmless while every caller ran late with a resolved
ServerArgs, reachable from the registration path.

sgl-project#636 (4f486ab): the four preconditions of the DCP token-sharded handover
become ONE boot-time contract instead of four runtime deaths on first
transfer. Placed after _reject_unsupported_draft_kv_dcp because dcp_size is
not resolved until _handle_uneven_tp (server_args.py:10319, called at :5957,
after the PD hook at :5916) -- a gate at the hook reads the unresolved value
and passes silently on the exact production configuration it guards. The
ordering is pinned by a test, not by a comment.

#631b contract (214433e) and spec (ec3a440): versioned canonical
full-head draft-KV layout with a geometry gate whose bound is the rejected
alternative's own cost rather than a constant in MB, plus the L1-L11 lift
criterion for the #631a refusal.

#631b composition verdict (66da8b3): --draft-kv-layout and
disaggregation_mode were never considered together. Under the default
'replicated' layout the draft pool is sized to the full global context while
the decode arm addresses every registered buffer with compact owner-rule rows
-- two coordinate systems, silent wrong-slot drift. That makes 'dcp' the only
layout on which the existing shared-index transfer is correct at all, not
merely the cheaper one.

sgl-project#642 (66ca5ee, a79fc3c): guards for that composition, with a hazard
test that stays green when the guard is removed so the arms distinguish a
broken guard from a broken hazard, and an identity-hash invariant pinned by
behaviour after the source-text pin was shown to miss the likely refactor.

The #631a refusal stays in force: the wiring slice needs cards and a
half-wired draft path that boots and silently decodes without speculation is
the failure this line exists to remove.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants