Skip to content

[serge] Fix 3 integration tests for model deepseek_vl_hybrid failing with other (other (3)) - #49277

Closed
sergereview[bot] wants to merge 1 commit into
mainfrom
serge/fix/itf-8687102be459-3bbf44ec
Closed

sergereview[bot] wants to merge 1 commit into
mainfrom
serge/fix/itf-8687102be459-3bbf44ec

Conversation

@sergereview

@sergereview sergereview Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

CPU CI GPU run-slow

Original CI failure

  • Failure group: 3 integration tests for model deepseek_vl_hybridfailing withother (other (3))
  • tests/models/deepseek_vl_hybrid/test_modeling_deepseek_vl_hybrid.py::DeepseekVLHybridIntegrationTest::test_model_text_generation [multi-gpu] (other, seen 21/21)
CI traceback — tests/models/deepseek_vl_hybrid/test_modeling_deepseek_vl_hybrid.py::DeepseekVLHybridIntegrationTest::test_model_text_generation
(line 67)  RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!

Where to watch it:

The three DeepseekVLHybridIntegrationTest text-generation tests crash on multi-gpu with Expected all tensors to be on the same device (cuda:0/cuda:1) and on single-gpu with You can't move a model that has some modules offloaded to cpu or disk / OOM.

Root cause: each test loads the 7B model with device_map="auto" then immediately calls model.to(torch_device). The device map installs accelerate hooks that keep moving submodule inputs to their mapped device; .to() moves weights but does not remove the hooks, so tensors keep crossing devices.

Fix: drop the redundant model.to(torch_device) (inputs already go to model.device), mirroring the merged, GPU-verified, maintainer-approved sibling fix for deepseek_vl (#48536). Applied to all three integration tests plus test_model_text_generation_batched (byte-identical pattern).

Relates to #49275


✅ Verified on GPU

serge ran the targeted @slow test(s) on a GPU runner and opened this PR only after they passed with this patch. Each targeted test was run 5× on both the pre-patch and patched trees to rule out flakiness — the result held on every run.

Possibly related

Existing issues/PRs mentioning test_model_text_generation (keyword match — not verified to share a root cause):

  • #48171 — Fix stale expected values in integration tests (cuda sm_86 / Aug04 regressions) (PR, closed, updated 2026-08-21)
  • #44739 — fix series of failed test case for janus model (PR, closed, updated 2026-04-01)
  • #44179 — Models with incorrect tokenizer_class in tokenization_config.json tha… (PR, closed, updated 2026-02-23)
  • #43187 — 🚨Fix floating-point precision in JanusImageProcessor resize (PR, closed, updated 2026-02-02)
  • #41977 — Add Phi3.5 Vision Model (PR, open, updated 2025-12-18)

This change was produced automatically by serge from a CI failure report. The patch was generated by an LLM and applied by serge; review before merging.

serge v0.1.0 · model: Qwen/Qwen3.8-27B:novita · 22 LLM turns · 26 tool calls · 1172.5s · 797097 in / 76194 out tokens

@sergereview
sergereview Bot marked this pull request as ready for review October 2, 2026 23:30
@github-actions
github-actions Bot requested a review from ydshieh October 2, 2026 23:31
@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

[For maintainers] Suggested jobs to run (before merge)

run-slow: deepseek_vl_hybrid

@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

CI recap

Dashboard: View test results in Grafana
Latest run: 37077936827:1
Result: success | Jobs: 5 | Tests: 354 | Failures: 0 | Duration: 38s

@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@sergereview

sergereview Bot commented Oct 11, 2026

Copy link
Copy Markdown
Contributor Author

Closing as superseded by #49335, which targets the same test(s):

  • tests/models/deepseek_vl_hybrid/test_modeling_deepseek_vl_hybrid.py::DeepseekVLHybridIntegrationTest::test_model_text_generation

Closed by serge at the request of the nightly CI triage.

@sergereview sergereview Bot closed this Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant