Repository navigation
[Gemma4] Fix stale expected values in integration tests - #48233
Merged
Merged
Conversation
…model_multiimage
Update expected outputs for `("cuda", 8)` (sm_86 / A10G) in two integration
tests that have been failing since the initial Gemma4 integration landed:
- `test_model_with_image`: cow beach image generates "ocean in the background
under a clear" rather than "ocean and a blue sky in the background"
- `test_model_multiimage`: Australia street scene generates "Roadway" rather
than "Traffic Sign" as the first bullet point
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Consolidate the three cuda sub-keys `(8,0)`, `(8,6)`, `(9,0)` into a single
`("cuda", 8)` key. All three produce the same output on current hardware
(torch 2.13 / CUDA 13), so the per-minor-version split is no longer needed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Collaborator
Author
|
run-slow: gemma4 |
Contributor
|
This comment contains models: ["models/gemma4"] |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
…dow_1_eager The test is flaky across different A10G runner instances: each runner produces a consistent output within itself, but two valid greedy-decode outputs exist: "That sounds lovely! ..." "That sounds like a very pleasant place! ..." The updated value matches the output observed consistently across several commits of this test's history on the runner used to update it. Should be monitored over the next few days to see if it flips again. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Collaborator
Author
|
run-slow: gemma4 |
Contributor
|
This comment contains models: ["models/gemma4"] |
…ache The full model (~4 GiB bfloat16) OOMs on 22.3 GiB GPUs when torch.export adds ~4+ GiB tracing overhead. Run on CPU instead to avoid the OOM, and reduce max_cache_len (1024→19) and max_new_tokens (20→3) to keep runtime acceptable (~45s). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Revert the cuda-8 expected value to the original 'That sounds lovely!' (which sdpa consistently produces), and override only for the eager attention variant on CUDA — where a different output has been observed on some A10G runner instances. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
CI ResultsCommit Info
Model CI Report❌ 1 new failed tests from this PR 😭
|
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Collaborator
Author
|
run-slow: gemma4 |
Contributor
|
This comment contains models: ["models/gemma4"] |
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
|
[For maintainers] Suggested jobs to run (before merge) run-slow: gemma4 |
Contributor
CI recapDashboard: View test results in Grafana |
sbucaille
pushed a commit
to sbucaille/transformers
that referenced
this pull request
Sep 16, 2026
…48233) * [Gemma4] Fix stale expected values in test_model_with_image and test_model_multiimage Update expected outputs for `("cuda", 8)` (sm_86 / A10G) in two integration tests that have been failing since the initial Gemma4 integration landed: - `test_model_with_image`: cow beach image generates "ocean in the background under a clear" rather than "ocean and a blue sky in the background" - `test_model_multiimage`: Australia street scene generates "Roadway" rather than "Traffic Sign" as the first bullet point Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [Gemma4] Fix stale expected values in test_model_with_image_batch Consolidate the three cuda sub-keys `(8,0)`, `(8,6)`, `(9,0)` into a single `("cuda", 8)` key. All three produce the same output on current hardware (torch 2.13 / CUDA 13), so the per-minor-version split is no longer needed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [Gemma4] Update expected value for test_generation_beyond_sliding_window_1_eager The test is flaky across different A10G runner instances: each runner produces a consistent output within itself, but two valid greedy-decode outputs exist: "That sounds lovely! ..." "That sounds like a very pleasant place! ..." The updated value matches the output observed consistently across several commits of this test's history on the runner used to update it. Should be monitored over the next few days to see if it flips again. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [tests] Fix test_export_text_only for gemma4: run on CPU with small cache The full model (~4 GiB bfloat16) OOMs on 22.3 GiB GPUs when torch.export adds ~4+ GiB tracing overhead. Run on CPU instead to avoid the OOM, and reduce max_cache_len (1024→19) and max_new_tokens (20→3) to keep runtime acceptable (~45s). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Add comment explaining CPU + small cache choice in test_export_text_only Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [Gemma4] Fix sliding window eager expected value without breaking sdpa Revert the cuda-8 expected value to the original 'That sounds lovely!' (which sdpa consistently produces), and override only for the eager attention variant on CUDA — where a different output has been observed on some A10G runner instances. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Condense flakiness comment, reference PR huggingface#48233 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * format * fix * [Gemma4] Revert test_generation_beyond_sliding_window to main version Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes stale expected outputs and a failing export test in
Gemma4IntegrationTest. No model logic is changed.Changes
test_model_with_imageandtest_model_multiimageBoth tests were enabled by Cyril on 2026-04-08 and passed once on 2026-04-09. Cyril made several Gemma4 changes on 2026-04-09 which very likely altered the model outputs. Since then the expected values have been stale.
Fix: Update
("cuda", 8)expected strings to match current A10G (sm_86) output.test_model_with_image: cow beach image →"ocean** in the background under a **clear"(was"ocean and a blue sky** in the background")test_model_multiimage: Australia street scene →"Roadway: There is an"(was"Traffic Sign: The most prominent")test_model_with_image_batchThe test started failing at
a374d990(2026-04-29 CI run). Investigation shows the model output on current hardware (torch 2.13 / CUDA 13) is identical across31469146(last passing),a374d990(first failing), and today — the actual output has always been the "breakdown" format:The most likely cause is
93bed93e75changingimage_transforms.py(patch extraction:image[:, ...]→image[..., ...]), which may have shifted the output on the then-current torch version — though we cannot reproduce this without the original torch version. On torch 2.13, all three sub-keys(8,0),(8,6),(9,0)produce the same result, so they are consolidated into a single("cuda", 8)key.test_generation_beyond_sliding_window_1_eager[NOT FIX IN THIS PR - WIP]The flakiness appeared right after the switch to torch 2.13 + CUDA 13.0 on 2026-08-03. Before that, the test was consistently passing — confirmed by downgrading to torch 2.11+cu126 on the same runner that fails with torch 2.13, which immediately restores the
"That sounds lovely!"output and makes the test pass.The new torch/CUDA version appears to be sensitive enough to produce different greedy-decode outputs for the same input. Two outputs have been observed on different days:
"That sounds lovely! It seems like you're really enjoying the place you'""That sounds like a very pleasant place! It seems like you're really enjoying"On 2026-08-23, running git bisect across all commits of this test's history on a single SSH runner gave identical output on every commit.
No fix is applied in this PR. The test is left at its main-branch state while the root cause is still being investigated.
test_model_text_only(no code change needed)Already fixed by PR #48171 (
d56c55bf). No change needed here.test_export_text_only(new fix in this PR)Gemma4IntegrationTestwas re-enabled on 2026-04-08, and this test has been OOM since then. The full E2B model (~4 GiB bfloat16) plustorch.exporttracing overhead (~4 GiB) exceeds the 22.3 GiB GPU memory available in CI.Fix: Run on CPU instead of GPU — avoids the OOM entirely. Reduce
max_cache_len(1024→19, covering the ~16-token prompt + 3 new tokens) andmax_new_tokens(20→3) to keep CPU runtime acceptable (~45s in practice). Export correctness is still verified end-to-end: the exported program must produce identical output to eager generation.Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com