Skip to content

[Benchmark] [AMD] Add Qwen3.5 DFlash acceptance verification - #82

Draft
TianHao65 wants to merge 1 commit into
apinge:qwen4_v0.5.21from
TianHao65:qwen35-dflash-acceptance-issue49
Draft

TianHao65 wants to merge 1 commit into
apinge:qwen4_v0.5.21from
TianHao65:qwen35-dflash-acceptance-issue49

Conversation

@TianHao65

@TianHao65 TianHao65 commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

What

Refs #49. Add five DFlash workload drivers, response-counter recording, dataset downloads and test entry points. Rebased onto apinge:qwen4_v0.5.21 at c0a27e481010cd2f285fe426eed35860ae1920a5. CI PR65 is merged into this target; all separately cherry-picked CI and stacked-CI compatibility commits were removed. Head 531f5ea32a607c22fc74639e86c69f85de3b3df2 is one commit /18 files directly above the merged target. The README development baseline is updated accordingly.

Data

Historical measurements on source 720f8ad566, 2026-11-03: Qwen3.5-397B PTPC-FP8 + DFlash, 8×MI308X/TP8, draft16, FP8 KV/page64, radix disabled, AITER full attention and Triton linear/draft attention. Greedy sampling, thinking disabled; one concurrency-sized warmup batch excluded.

Dataset Measured requests Concurrency Output limit Weighted accept length Draft acceptance rate
GSM8K 136 64 4096 7.1286 40.9339%
MATH-500 96 32 2048 7.9902 46.6560%
HumanEval 132 32 1024 11.5025 70.2728%
MBPP 96 32 1024 4.3861 22.9269%
MT-Bench (first turn) 48 32 1024 4.1267 20.8970%

Weighted accept length = total completion tokens / total verify steps, including bonus tokens. Acceptance rate uses matched/proposed draft counters, excluding bonus tokens. All 28 length-truncated responses remain included.

Benchmarking and Profiling

All five frozen datasets, raw measured responses, summaries and configuration/hashes. GSM8K reuses the complete first run on measured source 720f8ad566; the other four ran subsequently with the same weights/backends/sampling. Two server starts; no baseline speedup comparison.

Validation

508 measured responses have complete counters and pass independent recomputation. Owned servers stopped; eight GPUs released. Current checks:8 counter/HTTP fixture tests (10 workload/batch subcases), all five native CLI help commands,21 CI driver tests,17 evaluation shell syntax checks, Python syntax and clean diff. No new GPU evaluation was run after rebasing. Single-request probes produced malformed output on this stack; C1 numerical correctness remains unresolved. These measurements cover the listed client concurrencies32/64. Retain Draft.

@TianHao65
TianHao65 force-pushed the qwen35-dflash-acceptance-issue49 branch from 232877c to e9abb36 Compare November 3, 2026 03:15
@TianHao65
TianHao65 changed the base branch from qwen4_v0.5.21 to ci/qwen4-unified November 3, 2026 03:16
@TianHao65
TianHao65 force-pushed the qwen35-dflash-acceptance-issue49 branch from e9abb36 to 396e349 Compare November 3, 2026 03:25
@TianHao65
TianHao65 changed the base branch from ci/qwen4-unified to qwen4_v0.5.21 November 3, 2026 03:25
@TianHao65
TianHao65 force-pushed the qwen35-dflash-acceptance-issue49 branch from 396e349 to 720f8ad Compare November 3, 2026 03:34
Add the packaged DFlash workload drivers, complete acceptance recording, dataset validation, launch and benchmark entry points, and usage documentation on qwen4_v0.5.21. Use the unified CI already merged into the target branch.

Signed-off-by: TianHao65 <283868507+TianHao65@users.noreply.github.com>
@TianHao65
TianHao65 force-pushed the qwen35-dflash-acceptance-issue49 branch from 720f8ad to 531f5ea Compare November 3, 2026 07:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant