Skip to content

[serge] Fix 2 integration tests for model mamba2 failing with output_mismatch (other (2)) - #49945

Open
sergereview[bot] wants to merge 1 commit into
mainfrom
serge/fix/itf-c70c93706083-a83a52d0
Open

sergereview[bot] wants to merge 1 commit into
mainfrom
serge/fix/itf-c70c93706083-a83a52d0

Conversation

@sergereview

@sergereview sergereview Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

🤗 Transformers CI

Original CI failure

  • Failure group: 2 integration tests for model mamba2failing withoutput_mismatch (other (2))
  • tests/models/mamba2/test_modeling_mamba2.py::Mamba2IntegrationTest::test_simple_generate [single-gpu] (output_mismatch, seen 21/21)
CI traceback — tests/models/mamba2/test_modeling_mamba2.py::Mamba2IntegrationTest::test_simple_generate
(line 468)  AssertionError: '<s>[[81 chars]rld!" program written in C++:\n\n```cpp\n#include <iostream>\n' != '<s>[[81 chars]rld!" program in C++:\n\n```cpp\n#include <iostream>\n\n'

Where to watch it:

The A10G (compute capability 8.6) produces the "written in C++" variant, identical to the existing ("xpu", 3) entry. Without a ("cuda", 8) key, Expectations falls back to ("cuda", 7) (score 1, the only cuda key), which has the "in C++" variant — a stale expectation.

Adds a ("cuda", 8) key with the value the runner actually produces. The difference from ("cuda", 7) is a single word ("written") in the generated text — normal hardware-level variation for a language model, not a regression.

Relates to #49943


⚠️ Not verified — this patch changed the expectation

This patch changes only expected values in test files — the assertions were rewritten, not the code under test.

The GPU run below is not evidence that this change is correct: the tests were re-run after the patch rewrote what they assert, so they pass by construction. Judge the new value on its merits.

Possibly related

Existing issues/PRs mentioning test_simple_generate (keyword match — not verified to share a root cause):

  • #46773 — Fix padding mask skipped for batch size 1 in linear-attention models (PR, closed, updated 2026-10-06)
  • #48978 — Contract the mamba2 chunk scan with einsum instead of broadcast-then-sum (PR, closed, updated 2026-09-21)
  • #47630 — 🚨 [Kernels] Refactor all linear attn models & native kernels fallback (PR, closed, updated 2026-08-05)
  • #44950 — 🚨 [Cache] Native mamba & hybrid cache (PR, closed, updated 2026-03-31)
  • #43590 — Remove many output_attentions and other traced outputs on 100+ models (PR, closed, updated 2026-03-12)

This change was produced automatically by serge from a CI failure report. The patch was generated by an LLM and applied by serge; review before merging.

serge v0.1.0 · model: Qwen/Qwen3.8-27B:deepinfra · 14 LLM turns · 13 tool calls · 391.8s · 288606 in / 6105 out tokens

@sergereview
sergereview Bot marked this pull request as ready for review October 24, 2026 00:20
@github-actions
github-actions Bot requested a review from ydshieh October 24, 2026 00:20
@github-actions

Copy link
Copy Markdown
Contributor

[For maintainers] Suggested jobs to run (before merge)

run-slow: mamba2

@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant