Skip to content

[CI] Add Unified Qwen3.5 & Qwen3.8 CI - #65

Merged
sammysun0711 merged 22 commits into
apinge:qwen4_v0.5.21from
sammysun0711:ci/qwen4-unified
Nov 3, 2026
Merged

sammysun0711 merged 22 commits into
apinge:qwen4_v0.5.21from
sammysun0711:ci/qwen4-unified

Conversation

@sammysun0711

@sammysun0711 sammysun0711 commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

This PR aim to address #48 to build unified CI infra for Qwen3.5 & Qwen3.8 on MI308X.

Modifications

  • Add six independently retryable Actions jobs for the configurations. Overall Qwen4 CI Gate check requires every selected case to pass.
  • Reuse the existing launch, benchmark, and GSM8K scripts through one local runner.
  • Add --output-dir and opt-in --strict completion accounting to the shared evaluator so direct run_gsm8k.sh runs also produce auditable results.
  • The default runs performance then accuracy; --case, --accuracy-only, and --smoke select smaller workloads.

Please check detailed CI usage in: evaluation/Qwen4_Evaluation_Guide.md, section Qwen 3.5 & Qwen 3.8 CI Overview.

Accuracy Tests

Full runs require 200 requested/scored/completed examples, zero request errors, and score >= reference * 0.97: at most a 3% relative accuracy drop. Smoke accuracy is diagnostic.

Local MI308X accuracy-only validation passed all six gates, with 200 completed/scored examples and zero request errors per case:

Case Correct / 200 Accuracy Reference
Qwen3.5 35B TP4 base 196 98.0% 97.0%
Qwen3.5 35B TP1 DFlash 195 97.5% 97.0%
Qwen3.5 397B TP8 base 197 98.5% 99.0%
Qwen3.5 397B TP8 DFlash 197 98.5% 98.5%
Qwen3.8 Flash TP2 base 196 98.0% 98.0%
Qwen3.8 Flash TP2 MTP 197 98.5% 98.5%

Validation used AITER 05420f166, evaluator fc6ddbbf7, and pyhip edde17b08.

Speed Tests and Profiling

Default performance: 10 warmups + 128 random requests at concurrency 1, with 8K input / 500 output tokens for Qwen3.5 and 12K / 350 for Qwen3.8. Summaries report QPS, TTFT, TPOT, output tokens/s, and speculative acceptance length. Performance thresholds are advisory; incomplete requests fail the case.

The 35B DFlash benchmark path passed a two-request smoke run. The six-case validation used --accuracy-only; the full 128-request performance sweep and profiling were not rerun for the final implementation.

Validation

python3 -m pytest scripts/ci/test_qwen4_ci.py -q: 21 passed.

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ❌ Run #41105847195
Latest PR Test (Extra): ❌ Blocked -- run-ci is required first.
Latest PR Test (AMD ROCm 10): ❌ Run #41105846930

@sammysun0711
sammysun0711 marked this pull request as draft October 25, 2026 06:06
@sammysun0711
sammysun0711 marked this pull request as ready for review October 26, 2026 01:29
@sammysun0711
sammysun0711 requested a balanced review from Copilot October 26, 2026 01:30

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

5 open issues
What changed in this PR

Adds a unified MI308X CI harness for Qwen3.5/Qwen3.8 evaluation with a single image build, standardized artifact contracts, and accuracy/performance reporting.

Changes:

  • Introduces a new GitHub Actions workflow to authorize, run, and report Qwen4 MI308X CI (with optional rerun/reuse support).
  • Adds a Python CI driver (qwen4_ci.py) plus a comprehensive pytest suite validating lifecycle, reporting, and contracts.
  • Updates evaluation + GSM8K scripts and run_eval/sampler to support strict accounting and stable output locations.
File Description
scripts/​ci/​qwen4_ci.py New CI orchestrator: plan resolution, Docker build/run, artifact validation, reporting, baselines/gates
scripts/​ci/​test_qwen4_ci.py New pytest coverage for harness behavior, error handling, reuse, and report contracts
python/​sglang/​test/​simple_eval_common.py Adds thread-safe request accounting used by strict GSM8K runs
python/​sglang/​test/​run_eval.py Adds --output-dir + --strict GSM8K completeness enforcement and report placement changes
evaluation/​run_qwen38_pure_text_bench.sh Makes bench invocation reproducible and writes structured perf output
evaluation/​run_qwen35_pure_text_bench.sh Same as above for Qwen3.5 bench
evaluation/​run_gsm8k.sh New strict GSM8K runner wrapper with explicit args + output directory
evaluation/​launch_qwen38_flash_next_fp8_mi308x_pure_tp_4_or_8_or_2.sh Adds MTP toggle and uses SERVER_PYTHON module invocation
evaluation/​launch_qwen3.5-397B-fp8_tp8_disable_prefix_cache.sh Standardizes env overrides, logging, and robustness flags
evaluation/​launch_qwen3.5-397B-fp8_tp8_dflash_with_prefix_cache.sh Same standardization for DFlash case
evaluation/​launch_qwen3.5-35B-fp8_tp4_disable_prefix_cache.sh Same standardization for 35B base
evaluation/​launch_qwen3.5-35B-fp8_tp1_dflash_with_prefix_cache_mfs0.7_bs32.sh Same standardization for 35B DFlash
evaluation/​Qwen4_Evaluation_Guide.md Minor guide fixes (typo + missing cd)
.github/​workflows/​qwen4-ci.yml New workflow wiring (CPU validation, authorization gate, GPU run, reporting/status)

Review effort: Lite

Comment thread python/sglang/test/run_eval.py Resolved
Comment thread python/sglang/test/run_eval.py Resolved
Comment thread scripts/ci/qwen4_ci.py Resolved
Comment thread scripts/ci/qwen4_ci.py Resolved
Comment thread scripts/ci/qwen4_ci.py Resolved
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
- Reuse evaluation scripts while preserving runtime pins and accuracy gates.
- Run six independently retryable cases with full console logs and artifacts.
- Remove redundant wrappers and tests, and document local runs.

Signed-off-by: Xiake Sun <xiake.sun@amd.com>
@sammysun0711
sammysun0711 requested a balanced review from Copilot October 30, 2026 15:35

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread python/sglang/test/run_eval.py Resolved
Comment thread scripts/ci/qwen4_ci.py Resolved
Comment thread scripts/ci/qwen4_ci.py Resolved
Comment thread evaluation/Qwen4_Evaluation_Guide.md Resolved
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Oct 30, 2026
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Comment thread scripts/ci/qwen4_ci.py
EVALUATION = ROOT / "evaluation"
CLIENT_SHA = "fc6ddbbf7aac80167ea04cc00bf200af8adf2b85"
PRESSURE_BENCHMARK = "benchmark_example-20261013-363695ea/concurrency"
GSM8K_MAX_RELATIVE_DROP = 0.03

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it too big ?

@sammysun0711 sammysun0711 Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previously we use GSM8K_MAX_RELATIVE_DROP = 0.02 for accuracy gate: https://github.com/apinge/sglang/actions/runs/41310495689/job/124859754260, the model's long reasoning cause some task return with empty answer, that cause the GSM8K drop slightly from 97% to 95.0%, slightly exceed 2% (95.06%) accuracy gate. Increase to 3% to avoid sporadically failure.
See summary: https://github.com/apinge/sglang/actions/runs/41310495689

@sammysun0711
sammysun0711 merged commit c0a27e4 into apinge:qwen4_v0.5.21 Nov 3, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation run-qwen4-ci

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants