[sergereview]

Original CI failure

• Failure group: 2 integration tests for model minimaxfailing withoutput_mismatch (tensor values differ (2))
• tests/models/minimax/test_modeling_minimax.py::MiniMaxIntegrationTest::test_small_model_logits [multi-gpu] (output_mismatch, seen 6/7)

CI traceback — tests/models/minimax/test_modeling_minimax.py::MiniMaxIntegrationTest::test_small_model_logits

(line 247)  AssertionError: Tensor-likes are not close!

Where to watch it:

• test_small_model_logits — Test dashboard

Relates to #48514

MiniMaxIntegrationTest::test_small_model_logits fails on the daily GPU runner because the selected ("cuda", 8) expectation drifts from the actual bfloat16 logits by ~0.005 absolute (max ~0.12 relative on a near-zero value), exceeding atol=rtol=1e-3. The divergence is consistent with normal bfloat16 rounding variation, not a model regression. Update the existing CUDA 8 expected slice to the reproduced values; the (None, None) fallback is left unchanged so other hardware keeps passing.

Key differences vs. old ("cuda", 8): positions (0,2) -0.3203 → -0.3184, (1,0) -0.1201 → -0.1177, (1,1) 0.4375 → 0.4355, (1,2) 0.2402 → 0.2393, (2,2) -0.0396 → -0.0349; all within ~0.005 absolute.

---

⚠️ Not verified — this patch changed the expectation

This patch changes only expected values in test files — the assertions were rewritten, not the code under test.

The GPU run below is not evidence that this change is correct: the tests were re-run after the patch rewrote what they assert, so they pass by construction. Judge the new value on its merits.

• Passes with this patch: https://github.com/huggingface/transformers/actions/runs/33815339776
• Failed before the fix (base commit): https://github.com/huggingface/transformers/actions/runs/33813912169

Possibly related

Existing issues/PRs mentioning test_small_model_logits (keyword match — not verified to share a root cause):

• #48171 — Fix stale expected values in integration tests (cuda sm_86 / Aug04 regressions) (PR, closed, updated 2026-08-21)
• #47284 — Fix failing tests for mimo_v2_flash (PR, closed, updated 2026-07-24)
• #43324 — minimax_m2: fix failed test case for XPU (PR, closed, updated 2026-04-13)
• #30793 — Add torch compile for mixtral (PR, closed, updated 2024-07-15)
• #30127 — Fix SDPA sliding window compatibility (PR, closed, updated 2024-04-17)

---

This change was produced automatically by serge from a CI failure report. The patch was generated by an LLM and applied by serge; review before merging.

serge v0.1.0 · model: moonshotai/Kimi-K2.7-Code · 26 LLM turns · 25 tool calls · 27.8s · 468025 in / 4635 out tokens

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: minimax

[github-actions]

---

CI recap

Dashboard: View test results in Grafana
Latest run: 33815602481
Result: success | Grafana metrics are not available yet.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ydshieh]

Root cause investigation (via SSH runner on A10G, CUDA 8.6, PyTorch 2.13.0+cu130)

Root commit confirmed by git bisect

PR #41580 — "Refactor weight loading" (ArthurZucker, 6f6095e0cf, Nov 13 2025) is the first bad commit. Verified by checking out the parent vs the commit directly on the GPU runner — the values change between them.

Actual model output at each commit

Parent c4cfc2e023 (last passing commit — matches OLD expected values):

logits[0,:3,:3] = logits[1,:3,:3]:
[[ 1.03125,    -0.515625,   -0.3203125  ],
 [-0.1201171,   0.4375,      0.240234375],
 [ 1.21875,    -0.58984375, -0.03955078 ]]

Arthur's commit 6f6095e0cf (first bad — breaks old expected values):

logits[0,:3,:3] = logits[1,:3,:3]:
[[ 1.03125,    -0.51171875, -0.322265625],
 [-0.1171875,   0.43359375,  0.2392578  ],
 [ 1.21875,    -0.59375,    -0.0361328  ]]

Latest main 9d13f8cf6e (current — matches Serge's proposed values exactly):

logits[0,:3,:3] = logits[1,:3,:3]:
[[ 1.0312, -0.5156, -0.3184],
 [-0.1177,  0.4355,  0.2393],
 [ 1.2188, -0.5898, -0.0349]]

Note

The values at Arthur's commit and at latest main differ noticeably (e.g. -0.51172 vs -0.5156 at position [0,1]), meaning further commits after #41580 also shifted the outputs. The proposed expected values in this PR reflect the current main, which is correct.

Serge's values are an exact match with current main output ✅ — safe to merge.

[ydshieh]

Investigation findings

After digging into the root cause, here is what happened:

Root commit: PR #41580 (ArthurZucker, 6f6095e0cf, Nov 13 2025 — "Refactor weight loading") broke the test. Confirmed via git bisect: parent commit c4cfc2e023 passes, 6f6095e0cf fails — reproducible on current A10G runner.

Why the values changed: Arthur's PR did two things simultaneously:

1. Refactored the general weight loading infrastructure (core_model_loading.py)
2. Changed MiniMax MoE expert weight layout: from per-expert individual tensors (block_sparse_moe.experts.{i}.w1/w2/w3) to fused batched tensors (mlp.experts.gate_up_proj/down_proj across all experts)

The forward pass is mathematically equivalent, but a batched fused matmul across all experts accumulates floating-point rounding in a different order than sequential per-expert matmuls on GPU → tiny bfloat16-level logit differences. Not broken logic, just a different computation path.

Further drift after Arthur's commit: The values on current main differ slightly from Arthur's commit values too — this is normal and expected (PyTorch version upgrades, TF32 attribute changes, cuBLAS kernel selection, etc.). In any case, we have no single passing day after Arthur's commit (first_fail: 2025-11-14, fail_count: 266 single-GPU / 270 multi-GPU as of today), so there is no stable baseline to compare against.

Conclusion: Updating the expected logit values to match current main is the correct and sufficient fix. This PR is good to merge.

[ydshieh]

Serge, this time you did a good job 🎉 but I'm still better!