Repository navigation
Conversation
|
Here are some information need to be clarify to continue this work:
|
a8cd545 to
ddccab6
Compare
|
948bd95 to
affd9cb
Compare
affd9cb to
e6ff953
Compare
e6ff953 to
affd9cb
Compare
Port apinge/sglang PR sgl-project#8 (a6f5d30) using the previously tested v0.5.21 adaptation. Route PTPC-FP8 and unquantized BF16 linears through AITER hipb_mm behind the existing swizzle flags. Track weight layouts per Parameter and retain fallback dispatch for weights that fail the swizzle shape guard. Handle v0.5.21 prequantized three-element tuples and leading dimensions, and support BF16 caller-owned output buffers. (cherry picked from commit 0bcf65b) Signed-off-by: TianHao65 <283868507+TianHao65@users.noreply.github.com> Signed-off-by: TianHao65 <Hao.Tian@amd.com>
affd9cb to
f5ac45b
Compare
Signed-off-by: TianHao65 <283868507+TianHao65@users.noreply.github.com>
| return unfused_matches[0] if all(unfused_matches) else None | ||
|
|
||
|
|
||
| class AiterHipblaslt: |
There was a problem hiding this comment.
I wonder if we need cherry pick this PR as is? I see unconditionally call hipblaslt will trigger the decode perf regression.
Can we consider choose aiter.tgemm instead
Example: sammysun0711@ca9c322
Current this example only cover FP8 linear in LLM case, should add BF16 linear in VIT part for compute bound case.
For decode case memory bound case, we can either
- Fallback to ck bpreshuffle ck gemm for decode case, tune the ck gemm in aiter branch
- Choose skinny gemm implementation from feat(lm_head):Triton skinny GEMV for the bf16 lm_head on gfx942 #46 or [ROCm] Add decode swizzled GEMV and safe caller-owned outputs #76
I create a follow up issue: #88
We may not reach so high con (con64/128) to see the per gain. Please use standard benchmark scripts (con=1) to report before/after performance on target machine with PTL on |
Performance changes are
The remaining summary metrics and the unchanged GSM8K reference/minimum are retained below. Acceptance length applies to speculative cases.
The observed tradeoffs are:
|
What
Port PR #8 hipBLASLt linears to
qwen4_v0.5.21and fix DFlash startup with packed QKV weights. Based onc0a27e48, which includes merged CI PR #65.f5ac45b7dahipb_mmfor PTPC-FP8 and BF16 linears, with per-weight layout guards and v0.5.21 API adaptation4183a98ec5Enable with:
The two linear flags default off; hipBLASLt output is BF16.
Data
Historical port measurements (2026-10-20):
e00930c54→0bcf65b8f, Qwen3.5-397B-A17B-PTPC-FP8, TP8, 8× MI308X, ROCm 7.2 /lmsysorg/sglang:v0.5.21-rocm720-mi30x. AITER attention, Triton GDN, FP8 KV, radix disabled; 512 input / 256 output, seed 42. One startup per side, discarded warmups, six formal rounds of ≥60s, 7,168 successful requests. Cross-start noise was not estimated. These comparisons were not rerun for the current DFlash fix.Benchmarking and Profiling
Historical TP0 profiles from the same baseline/port pair, collected separately at concurrency 1. The four screenshots use complete original traces and native Perfetto selections. CK FP8 preshuffle calls changed 240 → 0, hipBLASLt FP8 0 → 240, and hipBLASLt BF16 0 → 210.
These are sampled single-call durations. The Decode pair increased 15.478 → 18.318 µs (+18.35%), including a BF16 regression; this is not an ETE comparison.
Prefill
Before:
Open prefill before in Perfetto.
After:
Open prefill after in Perfetto.
Decode
Before:
Open decode before in Perfetto.
After:
Open decode after in Perfetto.
Complete original traces, hashes and capture manifest.
Validation
Local validation of
4183a98ec5(2026-11-04): MI308X, Torch 2.12 / ROCm 7.14 / Triton 3.8, AITER05420f166. Five CPU regression tests, 21 CI driver tests, six real packed-weight numerical cases, and two changed-input CUDA Graph replays passed (maximum relative L2 0.001850 against dense FP32).All six normal CI configurations passed: each completed 128 performance requests and 200 GSM8K examples, with zero terminal request errors. The 35B/TP1/DFlash result was reused from the same repaired source; the other five were run once.
The CI workload and gates were unchanged: performance concurrency 1, ten warmups, target input/output 12000/350 for Qwen3.8 and 8000/500 for Qwen3.5; GSM8K permits a maximum 3% relative drop. Test services were stopped and GPU memory release verified.
Daily pressure, full 1319-example GSM8K, and a controlled before/after comparison of the fix remain unmeasured. Packed fallback computes full QKV, so its performance cost remains to be measured.