[sunway513]

Summary

Removes the Meta-Llama-3-8B-Instruct entry from .github/benchmark/models_accuracy.json. The model is no longer part of what ATOM needs to track, and its accuracy job has been failing on main on every recent run.

The pull-request accuracy set goes from 14 models to 13. Nightly goes from 41 to 40.

Why now

Every ATOM Test run on main in recent history is red, and Meta-Llama-3-8B-Instruct is a failing job in each one:

• 412f5bfe5 — 19/21 accuracy jobs pass; failures are Meta-Llama-3-8B-Instruct and gpt-oss-120b (2 GPUs)
• f95ef3ec3 — same two
• e4e473f8c — 16/21; Meta-Llama-3-8B-Instruct among the failures

Dropping this test does not by itself turn main green. gpt-oss-120b (2 GPUs) is failing independently and is not touched here.

Knock-on changes

atomesh-accuracy-validation.yaml. Its local atomesh_levels whitelist listed Meta-Llama-3-8B-Instruct as its only "pr" model, and it filters against models_accuracy.json. With the entry gone the key would match nothing while still reading as intentional, so it is removed and the effect is written into the file:

atomesh standalone now validates zero models on a pull request. Simulated against the new config:

atomesh  event=pull_request  level=pr       -> 0 models
atomesh  event=push          level=main     -> 1 model
atomesh  event=schedule      level=nightly  -> 3 models

Promoting a replacement to "pr" changes what every atomesh PR pays for in runner time, which is a resourcing call for the job owner, not something to decide inside this change. Flagging it here rather than papering over it.

Deliberately not changed

The offline inference smoke test in atom-test.yaml keeps using this model. It is not an accuracy test. It asserts that the offline entrypoint completes, and it guards the path the nightly Docker release smoke-tests after building the image. Its existing comment records a bug that killed the offline output thread, reached main, and surfaced a day later as a 60-minute nightly timeout. One small model exercises that path as well as thirteen do. Only its SMOKE_EXTRA_ARGS comment is updated, because it pointed at the accuracy entry that this PR removes.

docker-release.yaml's golden-output check also still uses this model, for the same reason.

If the intent is to remove this model from ATOM automation entirely, those two need a replacement smoke model chosen first. Say the word and I will follow up separately.

Verification

• models_accuracy.json parses; 41 entries to 40; no Meta-Llama-3-8B entry remains.

• Both edited workflow files parse as YAML.

• Model selection simulated against the real filter logic for all three event types:

  atom-test  event=pull_request  level=pr       -> 13/40 models
  atom-test  event=push          level=main     -> 20/40 models
  atom-test  event=schedule      level=nightly  -> 40/40 models
  

• The JSON edit is textual, so the diff is 12 deleted lines and nothing else. Re-serialising the file would have silently rewritten 0.90 to 0.9 in three unrelated entries.

[github-actions]

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

• ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

• ✅ Run after the PR is approved and Pre Checkin passes
• ✅ Run immediately when an approval review is submitted
• ✅ Can be requested before approval with labels

Label	Tests
ci:full	Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom	Run native ATOM model accuracy tests
ci:vllm	Run ATOM vLLM OOT model accuracy tests
ci:sglang	Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 2135 --add-label <label>

[Copilot]

🟢 Approval recommended

The changes are scoped to CI configuration cleanup and workflow commentary, with only a minor documentation clarification suggested.

Pull request overview

Removes the Meta-Llama-3-8B-Instruct model from ATOM’s CI accuracy model set and adjusts CI workflows that referenced that accuracy entry, while explicitly keeping the model for offline smoke coverage.

Changes:

• Removed Meta-Llama-3-8B-Instruct from .github/benchmark/models_accuracy.json (accuracy matrix no longer tracks it).
• Updated atomesh-accuracy-validation.yaml to drop the now-nonexistent "pr" whitelist entry and document that atomesh PRs currently validate zero models.
• Updated atom-test.yaml comments to decouple the offline smoke test settings from the accuracy matrix entry.

File summaries

File	Description
.github/workflows/atomesh-accuracy-validation.yaml	Removes the atomesh PR whitelist entry for Meta-Llama-3-8B-Instruct and documents the resulting PR behavior.
.github/workflows/atom-test.yaml	Updates offline smoke-test comments to clarify it is independent of the accuracy matrix.
.github/benchmark/models_accuracy.json	Drops the Meta-Llama-3-8B-Instruct accuracy entry from the tracked model list.

Review details

• Files reviewed: 3/3 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.