Repository navigation
[BMG] sparse mul hangs > 10 min for complex128/float64 (sub-issue of #4947) #5928
Description
Sub-issue of #4947. These cases hang past the 10-minute timeout on BMG (B60) and take down the pytest-xdist worker. The hangs are dtype-specific: some float64 variants finish quickly while complex128 hangs.
Cases
| # | Case |
|---|---|
| 1 | test_sparse_xpu.py::TestSparseAnyXPU::test_binary_operation_mul_SparseCSR_xpu_complex128 |
| 2 | test_sparse_xpu.py::TestSparseAnyXPU::test_binary_operation_mul_SparseCSR_xpu_float64 |
| 3 | test_sparse_xpu.py::TestSparseAnyXPU::test_binary_operation_mul_SparseCSC_xpu_complex128 |
| 4 | test_sparse_xpu.py::TestSparseAnyXPU::test_binary_operation_mul_SparseCSC_xpu_float64 |
| 5 | test_sparse_xpu.py::TestSparseAnyXPU::test_binary_operation_mul_SparseBSR_xpu_complex128 |
| 6 | test_sparse_xpu.py::TestSparseAnyXPU::test_binary_operation_mul_SparseBSC_xpu_complex128 |
| 7 | test_sparse_xpu.py::TestSparseAnyXPU::test_binary_operation_mul_SparseBSC_xpu_float64 |
Reproducer
cd pytorch/third_party/torch-xpu-ops/test/xpu
export PYTORCH_TEST_WITH_SLOW=1
ZE_AFFINITY_MASK=0 timeout 660 pytest -v <file>::<class>::<test> # hangs > 10 minFull logs and environment are in #4947.
Activity
ZhaoqiongZ commentedon Oct 29, 2026
@torchxpubot fix
torchxpubot commentedon Oct 29, 2026
Automated fix session for this issue. Stages are appended below as they complete.
Issue Triage
Sub-items handled by this run (numbered as in the issue body):
test_binary_operation_mul_SparseCSR_xpu_complex128(TestSparseAnyXPU)test_binary_operation_mul_SparseCSR_xpu_float64test_binary_operation_mul_SparseCSC_xpu_complex128test_binary_operation_mul_SparseCSC_xpu_float64test_binary_operation_mul_SparseBSR_xpu_complex128test_binary_operation_mul_SparseBSC_xpu_complex128test_binary_operation_mul_SparseBSC_xpu_float64All seven are dtype/layout variants of the same parametrized test and all seven are
already listed in
test/xpu/skip_list_common.py, so this is handled as ahomogeneous skip-list batch: sub-item 1 takes any fix and the remaining six are
re-checked against it rather than fixed separately.
Automated by issue-triage.
Reproduce
Checked against the latest XPU nightly wheel
2.16.0.dev20261029+xpu(
torchvision 0.30.0.dev20261029,torchaudio 2.11.0.dev20261029) on thisBMG runner --
Intel(R) Arc(TM) Pro B60 Graphics, 8 devices, driver1.17.39395+16-- i.e. the same hardware class as the original report in#4947. torch-xpu-ops tree at
0eb9b32e6(main).None of the seven cases hangs any more. Each finishes in about 7 s against a
660 s cap; the original report had them killed at 660 s.
test_binary_operation_mul_SparseCSR_xpu_complex1281 passed in 7.56stest_binary_operation_mul_SparseCSR_xpu_float641 passed in 7.59stest_binary_operation_mul_SparseCSC_xpu_complex1281 passed in 7.32stest_binary_operation_mul_SparseCSC_xpu_float641 passed in 7.15stest_binary_operation_mul_SparseBSR_xpu_complex1281 passed in 7.10stest_binary_operation_mul_SparseBSC_xpu_complex1281 passed in 7.39stest_binary_operation_mul_SparseBSC_xpu_float641 passed in 7.25sA wider sweep in a single process with no
ZE_AFFINITY_MASKpin also came backclean: all 130
test_binary_operation_mul_Sparse*variants ofTestSparseAnyXPUpassed in 32 s.One detour worth recording.
test/xpu/test_sparse_xpu.pydoes not collectagainst current
pytorch/pytorchmain at all: its module-level@onlyCUDA-retarget block still referencesTestSparse.test_cuda_emptyandTestSparse.test_same_gpu, which the device-agnostic refactor inpytorch/pytorch#198121(2026-09-29) renamed and dropped, so the module raisesAttributeError: type object 'TestSparse' has no attribute 'test_cuda_empty'before any test runs. That is a separate, pre-existing breakage in this file --
unrelated to the hang -- and it is why the runs above used the last pytorch
commit before that rename (
32b144349df), where the test body oftest_binary_operationis byte-identical to main's. As a cross-check that theresult is not an artefact of that older tree, the same test body was also
replayed standalone against the nightly wheel's own OpInfo machinery, with no
pytorch source tree involved; all seven layout/dtype combinations passed in
under 2 s.
Automated by fix-reproduce.
Batch fan-out results
Base:
torch-xpu-ops@0eb9b32e6(main), reproduce base nightly2.16.0.dev20261029+xpu, BMGIntel(R) Arc(TM) Pro B60 Graphics.test_binary_operation_mul_SparseCSR_xpu_complex128test_binary_operation_mul_SparseCSR_xpu_float64test_binary_operation_mul_SparseCSC_xpu_complex128test_binary_operation_mul_SparseCSC_xpu_float64test_binary_operation_mul_SparseBSR_xpu_complex128test_binary_operation_mul_SparseBSC_xpu_complex128test_binary_operation_mul_SparseBSC_xpu_float64cap, so the skip entries added for this hang are obsolete. Nothing was
auto-removed; removing a skip is its own change and wants its own verify
plus PR.
Follow-ups
test/xpu/skip_list_common.py, the"test_sparse_xpu.py"tuple (lines 270-276 at0eb9b32e6) holds exactlythese seven
test_binary_operation_mul_*entries; they can go. Leave thefour neighbouring entries from the same comment block
(
test_index_select_*,test_to_sparse_*) alone -- those are the othersub-issues of [release/2.14] 20 UT cases hang (>10 min timeout) on BMG: sparse / linalg / stft / multi_head_attention #4947 and were not checked by this run.
test/xpu/test_sparse_xpu.pyno longer collects againstpytorch/pytorchmain, because its
@onlyCUDA-retarget block referencesTestSparse.test_cuda_emptyandTestSparse.test_same_gpu, both removed bypytorch/pytorch#198121. Worth its own issue if one is not open already --[release/2.15] test_sparse_xpu.py fail to collect on BMG #5620 covers the same symptom but on
release/2.15andWindows.
Automated by issue-handler.
Summary
Outcome: SKIPPED(no_longer_reproduces)
All seven cases are layout/dtype parametrizations of one test,
TestSparseAny.test_binary_operationwithop=mul, and none of them hangs anymore: on this BMG B60 runner against nightly
2.16.0.dev20261029+xpueach onepasses in roughly 7 s against the issue's own 660 s cap, and a wider sweep of
all 130
test_binary_operation_mul_Sparse*variants passed in 32 s. Theoriginal report in #4947 was taken on
torch 2.14.0.dev, sothe hang looks to have been resolved somewhere between 2.14 and 2.16; no root
cause was pursued and no code was changed, because there is no longer a failure
to fix. What is left is a cleanup: the seven skip entries this hang earned in
test/xpu/skip_list_common.pyare now stale.0eb9b32e6d437edcda4588c16c2daee8934cd891machine-readable outcome is in
agent_space_xpu/batch_summary.json.Two caveats on the evidence. First,
test/xpu/test_sparse_xpu.pycannot becollected against current pytorch main at all (see follow-up 2 above), so the
seven node ids were run against the last pytorch commit before that rename,
32b144349df, where the body oftest_binary_operationis byte-identical tomain's; the same test body was additionally replayed standalone against the
nightly wheel's own OpInfo machinery with no pytorch source tree, and passed
there too. Second, the original failure surfaced under
pytest-xdistwitheight workers, whereas these runs were single-process -- matching the
per-case re-run that the issue itself specifies as the reproducer.
Generated by fix job.
Review
Copy into a new comment, keep ONE verdict, add a note. Two verdicts in
one comment are dropped as ambiguous.
On a batch run add
item=<pytorch issue>to say which sub-item theverdict is about, in the bold text and in the marker -- the marker
is invisible, so the bold line is all a reader sees. One comment per
sub-item, since a bare verdict on a batch does not say which test it
means and is dropped.
run=<id>only when judging a run other than thenewest, in both places too.
A CI DISABLED sub-item,
item=from the fan-out table:Another sub-item of the same batch, judged on an earlier run:
An
[xpu-alignment]issue that should not have been filed -- one issueper difference, so no
item=:daisyden commentedon Nov 9, 2026
label-issue: #5928
Root cause: inconclusive; hang with no traceback. Compressed mul (
mul_sparse_csr->mul_out_sparse_csr,aten/src/ATen/native/sparse/SparseCsrTensorMath.cpp:208-224,254-278) round-tripsthrough COO (
to_sparse, COOmul_out_sparse_xpuatsrc/ATen/native/sparse/xpu/SparseTensorMath.cpp:53,to_sparse_csr); the hanging step is not pinned.Analyzed case: issue as a whole (the 7 cases are in a markdown table, so no test_cases entry was parsed)
typeBugissue_typeempty; body reports a hang past the 10-min timeout that takes down the xdist worker.testtest: uttest: utlabel; reproducer runspytest -v <file>::<class>::<test>under test/xpu.modulemodule: sparsetest_binary_operation_mul_Sparse{CSR,CSC,BSR,BSC}, compressed sparse mul path (SparseCsrTensorMath.cpp:254).priorityHighhwhw: BMGhw: BMGlabel; title tag[BMG].dtypedtype: complex..._xpu_complex128(CSR/CSC/BSR/BSC).dtypedtype: float64..._xpu_float64(CSR/CSC/BSC).functionalityTest cases (7):
Applied to this issue. Generated by a local
label-issuerun.xuhancn commentedon Nov 11, 2026
Re-verified on the latest nightly
2.16.0.dev20261110+xpu(pytorchdf21fc2d) on B60 (ZE_AFFINITY_MASK=0, 660 s cap): all seventest_binary_operation_mul_*_{complex128,float64}cases pass in < 1 s each (7 passed in 2.43 s). NOT_REPRODUCED. Note: thetest/xpu/test_sparse_xpu.pyshim no longer collects against this nightly — upstream #193175 enabled XPU forTestSparseCSR/TestSparseAnyintest/test_sparse.py, so the shim'sdtypesIfXPUre-definitions now raisedtypes redefinition for xpu; the XPU variants run from upstreamtest_sparse.pydirectly.