Fix Mistral GSM8K chat eval by JustinTong0323 · Pull Request #27757 · sgl-project/sglang

JustinTong0323 · 2026-06-10T03:35:48Z

Summary

Move the nightly text-model GSM8K eval from the hand-rolled 5-shot chat-API scorer to sgl-eval (zero-shot chat, \boxed{} answer, math_verify grading), fill in real thresholds for every nightly model, and fix the CI install-order bug that blocked the sgl-eval path.

Root cause

nightly-test-text-accuracy-2-gpu-h100 was red on Mistral-family models (e.g. mistralai/Mistral-7B-Instruct-v0.3 0.414 vs 0.47 threshold). Reproduced on a devbox: the model generates correctly with finish_reason=stop and no runaway — it is not a serving regression. The old scorer wraps a completion-style 5-shot Question:/Answer: prompt in Mistral's [INST] chat template, which breaks the few-shot context for Mistral-family models; last-number extraction then occasionally grabs the wrong number when the response continues into another Q/A block.

What this PR does

Switch the nightly eval to sgl-eval (api="sgl_eval" in test_text_models_gsm8k_eval.py). sgl-eval is a black-box subprocess running zero-shot chat with \boxed{} formatting and math_verify grading, which avoids the few-shot/chat-template interaction.
Make sgl-eval opt-in: only the nightly correctness eval sets api='sgl_eval'. Every other gsm8k caller keeps the hand-rolled 5-shot completion scorer unchanged.
Fill real thresholds for all 15 nightly models, baselined on H100 2-GPU over the full 1319-example split (measured_score − 0.05).
Fix the antlr4 install-order bug (below).

The CI install-order bug

sgl-eval's latex2sympy2_extended requires antlr4-python3-runtime 4.9.3 / 4.11 / 4.13.2 and raises ImportError: Unsupported ANTLR version 4.7.2 on 4.7.x. The antlr4-python3-runtime==4.9.3 pin in python/pyproject.toml [test] was not enough — CI still loaded 4.7.2:

install_sglang        # [dev]→[test] → sgl-eval → latex2sympy2_extended (needs antlr4 4.9.3) ✓
install_extra_deps    # uv pip install -e lmms-eval/ → lmms-eval v0.5 → latex2sympy2 1.9.1
                      #   hard-pins antlr4==4.7.2, clobbering 4.9.3 ✗
install_test_tools    # does not restore antlr4

The clobber comes from latex2sympy2==1.9.1 (a transitive dep of lmms-eval, distinct from latex2sympy2_extended), which hard-pins antlr4-python3-runtime==4.7.2.

Fix: in scripts/ci/cuda/ci_install_dependency.sh, right after the lmms-eval install, force it back:

$PIP_CMD install "antlr4-python3-runtime==4.9.3" --force-reinstall --no-deps $PIP_INSTALL_SUFFIX

Thresholds (H100 2-GPU, full 1319 split, sgl-eval zero-shot)

model	measured	threshold
meta-llama/Llama-3.1-8B-Instruct	0.81	0.77
Qwen/Qwen3-8B	0.81	0.76
Qwen/Qwen3-4B	0.82	0.77
meta-llama/Llama-3.1-70B-Instruct	0.95	0.90
mistralai/Mixtral-8x7B-Instruct-v0.1	0.44	0.39
Qwen/Qwen2-57B-A14B-Instruct	0.51	0.46
neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8	0.82	0.77
neuralmagic/Mistral-7B-Instruct-v0.3-FP8	0.28	0.23
neuralmagic/DeepSeek-Coder-V2-Lite-Instruct-FP8	0.85	0.80
zai-org/GLM-4.5-Air-FP8	0.77	0.73
neuralmagic/gemma-2-2b-it-FP8	0.07	0.02
neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8	0.94	0.89
neuralmagic/Mixtral-8x7B-Instruct-v0.1-FP8	0.40	0.35
neuralmagic/Qwen2-72B-Instruct-FP8	0.88	0.83
neuralmagic/Qwen2-57B-A14B-Instruct-FP8	0.45	0.40

Low scorers (gemma-2-2b-it-FP8 0.07, Mistral-7B-v0.3-FP8 0.28) are genuine — verified on a devbox that the model emits \boxed{} correctly and the grader extracts the answer (no format issue, no no_answer). GSM8K is a numeric answer task, so these reflect weak zero-shot reasoning on small/older-instruct models, not an eval bug.

Validation

/rerun-test test/registered/eval/test_text_models_gsm8k_eval.py on 2-gpu-h100 — all 15 models pass: run 28645576771 ✅
Prior rerun 28642161462 failed with the antlr4 clobber on all models — fixed here.
pre-commit run --files and py_compile on changed files.

CI States

Latest PR Test (Base): ❌ Run #29058072699
Latest PR Test (Extra): ❌ Run #29058072562

# Conflicts: # test/registered/unit/server_args/test_server_args.py

The FA3 backend-disable (server_args guard + is_fa3_default_disabled_architecture + TestDefaultAttentionBackend) is orthogonal to the GSM8K nightly regression: bisect attributes the break to sgl-project#21931 (mgsm_en -> gsm8k eval migration), not a backend change, and the PR body premise (Mistral already on FlashInfer) is false on main, which defaults Mistral/Mixtral to fa3 on Hopper. The chat-mode prompt + stop sequences + extract_answer is the actual fix. - remove is_fa3_default_disabled_architecture, the server_args guard, and the TestDefaultAttentionBackend unit tests - hoist the idx-invariant _chat_prompt_style out of the per-sample loop - collapse the duplicated chat-branch scaffolding in _build_prompt - lower FP8 Mixtral threshold to match bf16 (shared-threshold invariant)

github-actions · 2026-06-19T00:24:07Z

Results for /rerun-test test/registered/unit/bench/test_simple_eval_gsm8k.py test/registered/eval/test_text_models_gsm8k_eval.py:

🚀 ubuntu-latest (1 test): ✅ View workflow run

cd test/ && python3 registered/unit/bench/test_simple_eval_gsm8k.py

🚀 2-gpu-h100 (1 test): ✅ View workflow run

cd test/ && python3 registered/eval/test_text_models_gsm8k_eval.py

JustinTong0323 · 2026-07-03T06:11:46Z

/rerun-test test/registered/eval/test_text_models_gsm8k_eval.py

github-actions · 2026-07-03T06:12:11Z

Results for /rerun-test test/registered/eval/test_text_models_gsm8k_eval.py:

🚀 2-gpu-h100 (1 test): ❌ View workflow run

cd test/ && python3 registered/eval/test_text_models_gsm8k_eval.py

lmms-eval v0.5 pins antlr4-python3-runtime==4.7.2, which overwrites the 4.9.3 installed via sglang[test] (sgl-eval's latex2sympy2_extended raises ImportError on 4.7.x). Re-pin 4.9.3 right after the lmms-eval install so the nightly sgl-eval GSM8K path can import.

JustinTong0323 · 2026-07-03T06:40:16Z

/rerun-test test/registered/eval/test_text_models_gsm8k_eval.py

github-actions · 2026-07-03T06:40:41Z

Results for /rerun-test test/registered/eval/test_text_models_gsm8k_eval.py:

🚀 2-gpu-h100 (1 test): ✅ View workflow run

cd test/ && python3 registered/eval/test_text_models_gsm8k_eval.py

Baselined on H100 2-GPU over the full 1319-example GSM8K test split using sgl-eval (zero-shot chat, \boxed{} answer, math_verify grading). Thresholds are measured_score - 0.05. Replaces the 0.0 placeholders that only checked 'sgl-eval did not crash'.

JustinTong0323 · 2026-07-03T07:30:08Z

/rerun-test test/registered/eval/test_text_models_gsm8k_eval.py

github-actions · 2026-07-03T07:30:31Z

Results for /rerun-test test/registered/eval/test_text_models_gsm8k_eval.py:

🚀 2-gpu-h100 (1 test): ✅ View workflow run

cd test/ && python3 registered/eval/test_text_models_gsm8k_eval.py

JamesBrianD · 2026-07-07T07:04:49Z

-        from sglang.test.simple_eval_gsm8k import GSM8KEval
+        if getattr(args, "api", None) == "sgl_eval":
+            # Only the nightly correctness eval opts into sgl-eval (zero-shot
+            # chat, \boxed{}, math_verify). Every other gsm8k caller — spec
+            # decoding perf/accuracy, disaggregation, quant, model e2e — uses
+            # the 5-shot completion last-number scorer and relies on
+            # max_tokens/throughput behavior sgl-eval cannot provide.
+            return _run_sgl_eval("gsm8k", args)
+        from sglang.test.simple_eval_mixed_prefix_gsm8k import GSM8KEval


Are there plans to have sgl-eval support these two features and ultimately unify the evaluation technology stack?

Yes, we are planing to migrate all evals gradually.

alisonshao · 2026-07-09T02:47:10Z

/rerun-test test_mimo_v2_flash.py

github-actions · 2026-07-09T02:47:29Z

Results for /rerun-test test_mimo_v2_flash.py:

🚀 8-gpu-h200 (1 test): ✅ View workflow run

cd test/ && python3 registered/models_e2e/test_mimo_v2_flash.py

alisonshao · 2026-07-09T02:51:04Z

https://github.com/sgl-project/sglang/actions/runs/28645557096?pr=27757

# Conflicts: # python/sglang/test/run_eval.py # python/sglang/test/simple_eval_common.py

JustinTong0323 and others added 3 commits June 9, 2026 12:28

Fix Mistral FA3 default attention backend

53306a8

Merge branch 'main' into fix/mistral-fa3-default

d0fa031

Fix GSM8K chat eval for Mistral models

52e0ea5

github-actions Bot added the run-ci label Jun 10, 2026

JustinTong0323 added 4 commits June 10, 2026 11:49

Fix GSM8K eval unit test setup

49291be

Refine GSM8K chat prompt for Mistral

73e1865

Tune GSM8K prompt routing

743011c

Fix import order in simple_eval_gsm8k

46dcdef

amd-bot mentioned this pull request Jun 10, 2026

[CI Monitor] Daily Report - 2026-06-10 bingxche/sglang-ci-bot#99

Open

JustinTong0323 added 4 commits June 11, 2026 04:22

Scope GSM8K chat stop sequences to Mistral models

302bfd2

Limit GSM8K chat prompt wrapper to models that need it

ebe7e91

Retrigger CI

a43f518

Merge remote-tracking branch 'upstream/main' into fix/mistral-gsm8k-eval

c0f44ce

# Conflicts: # test/registered/unit/server_args/test_server_args.py

amd-bot mentioned this pull request Jun 18, 2026

[CI Monitor] Daily Report - 2026-06-18 bingxche/sglang-ci-bot#108

Open

amd-bot mentioned this pull request Jun 19, 2026

[CI Monitor] Daily Report - 2026-06-19 bingxche/sglang-ci-bot#109

Open

sgl-project deleted a comment from github-actions Bot Jun 19, 2026

sgl-project deleted a comment from gemini-code-assist Bot Jun 19, 2026

sgl-project deleted a comment from github-actions Bot Jun 19, 2026

This was referenced Jul 1, 2026

[CI Monitor] Daily Report - 2026-07-01 bingxche/sglang-ci-bot#121

Open

[CI Monitor] Daily Report - 2026-07-02 bingxche/sglang-ci-bot#122

Open

[CI Monitor] Daily Report - 2026-07-03 bingxche/sglang-ci-bot#123

Open

JamesBrianD reviewed Jul 7, 2026

View reviewed changes

amd-bot mentioned this pull request Jul 8, 2026

[CI Monitor] Daily Report - 2026-07-08 bingxche/sglang-ci-bot#129

Open

Merge remote-tracking branch 'origin/main' into pr-27757

f850857

# Conflicts: # python/sglang/test/run_eval.py # python/sglang/test/simple_eval_common.py

alisonshao added the high priority label Jul 9, 2026

Merge branch 'main' into fix/mistral-gsm8k-eval

30ebf1d

alisonshao enabled auto-merge (squash) July 10, 2026 04:08

alisonshao disabled auto-merge July 10, 2026 04:08

alisonshao merged commit b76dd0b into sgl-project:main Jul 10, 2026
147 of 188 checks passed

Uh oh!

Conversation

JustinTong0323 commented Jun 10, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Root cause

What this PR does

The CI install-order bug

Thresholds (H100 2-GPU, full 1319 split, sgl-eval zero-shot)

Validation

CI States

Uh oh!

github-actions Bot commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JustinTong0323 commented Jul 3, 2026

Uh oh!

github-actions Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JustinTong0323 commented Jul 3, 2026

Uh oh!

github-actions Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JustinTong0323 commented Jul 3, 2026

Uh oh!

github-actions Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JamesBrianD Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

JustinTong0323 Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

alisonshao commented Jul 9, 2026

Uh oh!

github-actions Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alisonshao commented Jul 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

JustinTong0323 commented Jun 10, 2026 •

edited by github-actions Bot

Loading

github-actions Bot commented Jun 19, 2026 •

edited

Loading

github-actions Bot commented Jul 3, 2026 •

edited

Loading

github-actions Bot commented Jul 3, 2026 •

edited

Loading

github-actions Bot commented Jul 3, 2026 •

edited

Loading

github-actions Bot commented Jul 9, 2026 •

edited

Loading