[upstream_ut] XPU backward kernels missing input-validation error checks (replication_pad2d, grid_sample)

🐛 Describe the bug

Running upstream test/test_nn.py on XPU, two "error-checking" cases fail because
the XPU backward kernels do not raise the validation error that CPU/CUDA raise.
Both PASS on CPU.

test_Pad_backward_channel_mismatch_xpu (test/test_nn.py:8791)

with self.assertRaisesRegex(RuntimeError, "grad_output channel unexpected"):
AssertionError: RuntimeError not raised

The XPU replication_pad2d_backward kernel lacks the channel-dim bounds check that
CPU/CUDA got from pytorch/pytorch#189463; it silently returns a wrong-shaped result.
Related: #4565.

test_grid_sample_backward_error_checking_xpu (test/test_nn.py:7132)

with self.assertRaisesRegex(ValueError, "expected grad_output to have sizes"):
AssertionError: ValueError not raised

The XPU grid_sampler backward does not validate grad_output sizes.

Note: neither case is caught by torch-xpu-ops CI because the vendored
test/xpu/test_nn_xpu.py is a stale copy that does not contain these cases.

Versions

main

@torchxpubot

@torchxpubot fix

Automated fix session for this issue. Progress is reported here, stage by stage; follow the workflow run for live logs.

Issue Triage

Field	Value
Issue type	batch-bug (heterogeneous)
Reproduction missing	no
Scope	torch-xpu-ops
Dependencies	(none)
Handling	agent-fixable

Sub-items handled by this run:

test_Pad_backward_channel_mismatch_xpu (test/test_nn.py) -- XPU replication_pad2d_backward missing the channel-dim bounds check.
test_grid_sample_backward_error_checking_xpu (test/test_nn.py) -- XPU grid_sampler backward missing grad_output size validation.

The two sub-items sit in unrelated kernels, so each takes its own fix branch; neither is expected to cover the other.

Automated by issue-triage.

Reproduce

Checked against the latest nightly wheel 2.16.0.dev20261028+xpu on BMG (Intel(R) Arc(TM) Pro B60 Graphics), driving the two aten backward ops directly, exactly as the upstream tests do.

Sub-item	Verdict	Observed
1. `test_Pad_backward_channel_mismatch_xpu`	REPRODUCED	three of the six pad backward ops disagree with CPU/CUDA -- see breakdown below
2. `test_grid_sample_backward_error_checking_xpu`	NOT_REPRODUCED	both `grid_sampler_2d_backward` and `grid_sampler_3d_backward` already raise `ValueError: grid_sampler(): expected grad_output to have sizes ...`

Sub-item 1 breakdown. The test asserts every {reflection,replication}_pad{1,2,3}d_backward raises RuntimeError, "grad_output channel unexpected":

Op	Result
`reflection_pad1d_backward`	ok -- `grad_output channel unexpected. Expected: 2, Got: 0`
`reflection_pad2d_backward`	no error raised at all
`reflection_pad3d_backward`	ok
`replication_pad1d_backward`	ok
`replication_pad2d_backward`	wrong message -- `gradOutput channel unexpected` (camelCase, regex does not match)
`replication_pad3d_backward`	wrong message -- `grad_output width unexpected` (mislabeled; it is the channel check)

The missing reflection_pad2d_backward check is not merely cosmetic: running it leaves the device wedged, and the next XPU kernel launch in the same process fails with UR_RESULT_ERROR_OUT_OF_RESOURCES -- i.e. the out-of-bounds read the check is meant to prevent.

Sub-item 2 was fixed after this issue was filed, by #5584, which added check_grid_sampler_{2d,3d}_backward to the XPU kernels.

Automated by fix-reproduce.

Root Cause

Analyzed at torch-xpu-ops@73a6552c, pytorch a183c26b (the sha the 2.16.0.dev20261028+xpu wheel was built from).

Upstream splits the pad backward ops into two groups. reflection_pad{1,3}d_backward and replication_pad1d_backward are structured, so their TORCH_META_FUNC runs at::native::padding::pad_backward_shape_check for every backend -- XPU included, for free. reflection_pad2d_backward, replication_pad2d_backward and replication_pad3d_backward are not structured: each backend's own function has to call that helper itself. CPU does so in aten/src/ATen/native/ReflectionPad.cpp:84 and aten/src/ATen/native/ReplicationPadding.cpp:68,84; CUDA does so in aten/src/ATen/native/cuda/ReflectionPad.cu:601 and aten/src/ATen/native/cuda/ReplicationPadding.cu:241,296. XPU is exactly the backend that never adopted the shared helper, and hand-rolls its own checks instead -- which is why all three of its non-structured backward ops disagree with CPU/CUDA, and the three structured ones agree:

src/ATen/native/xpu/sycl/ReflectionPadKernels.cpp:819-832 validates only grad_output's width and height. With no channel check, a grad_output of (2, 0, 6, 8) against an input of (2, 2, 4, 4) passes, a non-empty grad_input is allocated from the input shape, and the kernel then indexes grad_output with the input's channel counter -- the out-of-bounds read that wedges the device (UR_RESULT_ERROR_OUT_OF_RESOURCES on the next launch). This is the segfault pytorch/pytorch#142834 described.
src/ATen/native/xpu/sycl/ReplicationPaddingKernels.cpp:555-560 does have a channel check (added by [XPU] Validate replication_pad2d_backward channel dimension #4758) but spells the message gradOutput channel unexpected, while every other backend and the meta kernel say grad_output channel unexpected.
src/ATen/native/xpu/sycl/ReplicationPaddingKernels.cpp:694-699 checks the channel dim but labels the message grad_output width unexpected -- a copy-paste slip, so the text names the wrong dimension.

So the fix is the one upstream already settled on: delete the hand-rolled, drift-prone copies in those three XPU functions and call at::native::padding::pad_backward_shape_check(grad_output, input, padding, dim), the same single source of truth CPU, CUDA and _meta_registrations.py share. That closes the missing check, corrects both messages, and stops the messages from drifting again.

Sub-item 2 needs no change: #5584 already wired check_grid_sampler_{2d,3d}_backward into src/ATen/native/xpu/sycl/GridSampler.cpp:686,1831.

Verdict: IMPLEMENTING -- missing/incorrect use of a shared ATen validation helper in three XPU kernels.
Target repo: torch-xpu-ops
Domain: xpu-kernel

Automated by fix-root-cause.

Implement Result

--- a/src/ATen/native/xpu/sycl/ReflectionPadKernels.cpp
+++ b/src/ATen/native/xpu/sycl/ReflectionPadKernels.cpp
@@ -813,23 +813,8 @@ void reflection_pad2d_backward_kernel(
   int64_t input_h = input.size(dim_h);
   int64_t input_w = input.size(dim_w);
 
-  int64_t output_h = input_h + pad_t + pad_b;
-  int64_t output_w = input_w + pad_l + pad_r;
-
-  TORCH_CHECK(
-      output_w == grad_output_.size(dim_w),
-      "grad_output width "
-      "unexpected. Expected: ",
-      output_w,
-      ", Got: ",
-      grad_output_.size(dim_w));
-  TORCH_CHECK(
-      output_h == grad_output_.size(dim_h),
-      "grad_output height "
-      "unexpected. Expected: ",
-      output_h,
-      ", Got: ",
-      grad_output_.size(dim_h));
+  at::native::padding::pad_backward_shape_check(
+      grad_output_, input, padding, /*dim=*/2);
 
   Tensor grad_output = grad_output_.contiguous();
--- a/src/ATen/native/xpu/sycl/ReplicationPaddingKernels.cpp
+++ b/src/ATen/native/xpu/sycl/ReplicationPaddingKernels.cpp
@@ -19,6 +19,7 @@ DISABLE_RETURN_TYPE_WARNING_BEGIN
 #include <ATen/native/IndexingUtils.h>
+#include <ATen/native/Padding.h>
 #include <ATen/native/xpu/sycl/Atomics.h>
@@ -526,51 +527,12 @@ void replication_pad2d_backward_kernel(
   TORCH_CHECK(padding.size() == 4, "padding Size is expected to be 4");
 
+  at::native::padding::pad_backward_shape_check(
+      grad_output, input, padding, /*dim=*/2);
+
   const auto padL = padding[0];
-  const auto padR = padding[1];
   const auto padT = padding[2];
-  const auto padB = padding[3];
-  ... 40 lines of hand-rolled channel/width/height checks removed ...
-  TORCH_CHECK(
-      ichannel == grad_output.size(dimc),
-      "gradOutput channel unexpected. Expected: ",
+  const int numInputDims = input.dim();
@@ -632,12 +594,7 @@ static inline void shapeAndGradOutputCheck3d(
-    int64_t pad_left,
-    int64_t pad_right,
-    int64_t pad_top,
-    int64_t pad_bottom,
-    int64_t pad_front,
-    int64_t pad_back) {
+    IntArrayRef padding) {
@@ -689,30 +643,8 @@ static inline void shapeAndGradOutputCheck3d(
-  TORCH_CHECK(
-      num_planes == grad_output.size(plane_dim),
-      "grad_output width unexpected. Expected: ",
-      ... width / height / depth checks removed ...
+  at::native::padding::pad_backward_shape_check(
+      grad_output, input, padding, /*dim=*/3);
 }

test/regressions/test_replication_pad_channel_mismatch.py asserted the old gradOutput channel unexpected text, so it moves to the shared message and is widened from replication 2d only to all six {reflection,replication}_pad{1,2,3}d_backward ops, mirroring the upstream test.

Three files changed, 56 insertions, 124 deletions. lintrunner reports no issues.

Automated by fix-implement.

Verify

Source build: pytorch a183c26b (the sha the 2.16.0.dev20261028+xpu nightly was built from) with third_party/xpu.txt pinned to the fix branch, TORCH_XPU_ARCH_LIST=bmg,pvc, on Intel Arc Pro B60 (BMG).

Upstream test/test_nn.py calls instantiate_device_type_tests(TestNNDeviceType, globals(), allow_mps=True) -- no allow_xpu=True -- so the _xpu variants of these two cases do not exist in the upstream file as checked out. The XPU variants were generated with a wrapper that imports TestNNDeviceType under XPUPatchForImport and re-instantiates it only_for="xpu", i.e. exactly what the vendored test/xpu/test_nn_xpu.py does, so the node ids match the ones the issue reports.

Per-op before/after on the one failing sub-item (each row run in its own process, since the pre-fix out-of-bounds read wedges the device for the rest of the process):

Op	Before	After
`reflection_pad1d_backward`	pass	pass
`reflection_pad2d_backward`	fail -- no error raised	pass
`reflection_pad3d_backward`	pass	pass
`replication_pad1d_backward`	pass	pass
`replication_pad2d_backward`	fail -- `gradOutput channel unexpected`	pass
`replication_pad3d_backward`	fail -- `grad_output width unexpected`	pass

Test	Result
`TestNNDeviceTypeXPU::test_Pad_backward_channel_mismatch_xpu`	PASSED
`TestNNDeviceTypeXPU::test_grid_sample_backward_error_checking_xpu`	PASSED
`test/regressions/test_replication_pad_channel_mismatch.py`	8 passed

Regression sweep over everything that goes through the touched validation paths -- no failures:

Suite	Result
upstream `test_nn.py -k "Pad or pad"` (XPU)	80 passed, 5 skipped
`test/xpu/test_nn_xpu.py -k "Pad or pad"`	174 passed, 10 skipped
`test/xpu/test_ops_xpu.py` (replication/reflection/functional pad)	122 passed
`test/xpu/test_ops_gradients_xpu.py` (same ops)	20 passed, 30 skipped

lintrunner -m 73a6552c reports no issues, so there was nothing for a lint pass to amend.

Verdict: PASSED

Automated by fix-verify.

Batch fan-out results

Base: torch-xpu-ops@73a6552c, pytorch a183c26b (reproduce base: nightly 2.16.0.dev20261028+xpu)

Sub-item	Outcome	Branch / Reason
1. `test_Pad_backward_channel_mismatch_xpu`	FIXED	`agent/fix-issue-5508-1-test_pad_backward_channel_mismatch`
2. `test_grid_sample_backward_error_checking_xpu`	ALREADY_FIXED	no longer reproduces; #5584 added `check_grid_sampler_{2d,3d}_backward`

FIXED: 1 sub-item -- one branch, exported as a patch artifact for a human to review and open as a PR.
ALREADY_FIXED: 1 sub-item -- the reported bug is resolved, no action needed.

Automated by issue-handler.

Summary

Outcome: IMPLEMENTING(fix_verified) -- 1 of the 2 sub-items needed a fix; the other no longer reproduces.

Both sub-items trace to one thing: XPU is the only backend that never adopted ATen's shared padding::pad_backward_shape_check. Upstream makes reflection_pad{1,3}d_backward and replication_pad1d_backward structured so their TORCH_META_FUNC runs that helper for every backend, while reflection_pad2d_backward, replication_pad2d_backward and replication_pad3d_backward are not structured and each backend must call it itself -- which is exactly why XPU's three non-structured ops had drifted (one with no channel check at all, two with wrong message text) and its three structured ones were fine. The fix calls the helper in those three functions and deletes the hand-rolled copies, so all four backends and the meta kernel share one source of truth. Verification on a source build shows both upstream tests passing and 376 further pad tests green, with the pre-fix reflection_pad2d_backward out-of-bounds read -- which used to wedge the device -- now rejected cleanly.

The second sub-item, test_grid_sample_backward_error_checking_xpu, was fixed after this issue was filed by #5584 and needs no change.

Branch: agent/fix-issue-5508-1-test_pad_backward_channel_mismatch
Base: 73a6552c3c122614d2651de2a0c6feb8f81ae613
Changed files: src/ATen/native/xpu/sycl/ReflectionPadKernels.cpp, src/ATen/native/xpu/sycl/ReplicationPaddingKernels.cpp, test/regressions/test_replication_pad_channel_mismatch.py
Follow-up (not done here): the vendored test/xpu/test_nn_xpu.py is a stale copy of upstream test_nn.py and still lacks both of these cases, so CI would not have caught either one. This fix adds the equivalent coverage under test/regressions/, but refreshing the vendored copy is its own task.
Nothing was pushed. The patch is exported as a workflow artifact for a human to review and apply.

Generated by fix job.

Review

Copy into a new comment, keep ONE verdict, add a note. Two verdicts in
one comment are dropped as ambiguous.

**strong-accepted** <!-- review: strong-accepted -->
**weak-accepted** <!-- review: weak-accepted -->
**weak-rejected** <!-- review: weak-rejected -->
**strong-rejected** <!-- review: strong-rejected -->
**issue-drop** <!-- review: issue-drop -->

notes:

On a batch run add item=<pytorch issue> to say which sub-item the
verdict is about, in the bold text and in the marker -- the marker
is invisible, so the bold line is all a reader sees. One comment per
sub-item, since a bare verdict on a batch does not say which test it
means and is dropped. run=<id> only when judging a run other than the
newest, in both places too.

A CI DISABLED sub-item, item= from the fan-out table:

**strong-accepted item=197334** <!-- review: strong-accepted item=197334 -->
notes: same TensorAssign overload as upstream CUDA, fp8 dispatch list matches

Another sub-item of the same batch, judged on an earlier run:

**weak-rejected run=35171979678 item=197335** <!-- review: weak-rejected run=35171979678 item=197335 -->
notes: green only because the test is skipped, the kernel still lacks fp8

An [xpu-alignment] issue that should not have been filed -- one issue
per difference, so no item=:

**issue-drop** <!-- review: issue-drop -->
notes: accepted XPU/CUDA numerics difference, nothing to repair

label-issue: #5508

Root cause: XPU pad backward kernels do not call padding::pad_backward_shape_check (Padding.h:127; CUDA calls it at ReplicationPadding.cu:241).
reflection_pad2d bwd only checks H/W (sycl/ReflectionPadKernels.cpp:819-832), and the replication_pad2d message "gradOutput..." (ReplicationPaddingKernels.cpp:557) fails the test regex.

axis	value	reason
`type`	`Bug`	`issue_type` empty; body reports `AssertionError: RuntimeError not raised`.
`test`	`test: ut`	Existing human label `test: ut`; cases are in test/test_nn.py (test_nn.py:8404).
`module`	`module: ops`	Validation is missing in the native ATen padding / grid_sampler backward kernels in torch-xpu-ops.
`priority`	`Medium`	2 UT cases; the missing check allows an OOB grad_output read, so this is more than an error-message difference.
symptom	`functionality`	Expected RuntimeError/ValueError is not raised; the backward silently accepts bad shapes.

Applied to this issue. Generated by a local label-issue run.

Navigation Menu

Repository navigation

[upstream_ut] XPU backward kernels missing input-validation error checks (replication_pad2d, grid_sample) #5508

🐛 Describe the bug

Versions

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[upstream_ut] XPU backward kernels missing input-validation error checks (replication_pad2d, grid_sample) #5508

Description

🐛 Describe the bug

Versions

Activity

ZhaoqiongZ commented on Oct 28, 2026

torchxpubot commented on Oct 28, 2026

daisyden commented on Nov 10, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions