[xpu-alignment] [26-11-09] XPU Tensor.renorm divides by norm + 1e-7, so renormalised rows land below max_norm (33% short at max_norm=1e-7) #6128

Summary

Tensor.renorm / Tensor.renorm_ on XPU scale a too-long row by max_norm / (norm + 1e-7) instead of max_norm / norm, so the renormalised row never reaches max_norm, which is what the docstring promises. The 1e-7 is absolute, so the error is about 2e-8 relative at max_norm near 1 but grows without bound as max_norm shrinks: about 0.05% at 1e-4, 4.8% at 1e-6 and 33% at 1e-7. The guard is also unnecessary: the scaling branch only runs when norm > max_norm >= 0, so norm > 0 and the division is always safe.

Upstream source

• F.embedding and nn.Embedding with max_norm renormalise to max_norm/(1+1e-7/norm) instead of max_norm pytorch/pytorch#204119 (open, filed 2026-11-09) reports the same +1e-7 for F.embedding(max_norm=...), nn.Embedding, embedding_bag and Tensor.renorm/renorm_ on CPU.
• [nn] Preserve exact max norms during renormalization pytorch/pytorch#204132 ("[nn] Preserve exact max norms during renormalization", open, not landed) is the upstream fix. It divides by norm directly in aten/src/ATen/native/cpu/RenormKernel.cpp, aten/src/ATen/native/cuda/RenormKernel.cu, aten/src/ATen/native/mps/kernels/RenormKernel.metal, aten/src/ATen/native/Embedding.cpp and aten/src/ATen/native/cuda/Embedding.cu, and adds device-generic regressions. It does not touch intel/torch-xpu-ops.
• Found by an XPU alignment scan of the window [2026-11-09T00:00:00Z, 2026-11-10T00:00:00Z) over pytorch/pytorch, frozen default-branch head a68f117948121cdff578ff7813c5c906ff529b57.

Observed XPU behavior

Validated on XPU with torch 2.16.0.dev20261109+xpu, float64:

renorm_row0_norm 0.9999999800000003    # x.renorm(2, 0, 1.0) on [[3.,4.],[.6,.8],[0.,0.]]

0.9999999800000003 is exactly 1.0 * 5.0 / (5.0 + 1e-7), the value upstream reports, so the XPU renorm scale factor carries the same absolute guard.

Scope of what was measured, stated explicitly:

• Confirmed: Tensor.renorm in float64.
• Not reproduced: F.embedding(..., max_norm=...) in float64 returned exactly 1.0 (scale 0.20000000000000004) with zero relative shortfall at max_norm 1e-4, 1e-6 and 1e-7. That is because the XPU embedding renorm uses a dtype-relative epsilon rather than upstream's absolute 1e-7 (see below), and numeric_limits<double>::epsilon() is about 2.2e-16.
• Not measured: the float32 embedding path. There accscalar_t is float and numeric_limits<float>::epsilon() is 1.19e-7, the same order as upstream's 1e-7, so a comparable shortfall is expected but was not exercised by this reproducer. It should be checked as part of the fix.

Target path and root cause

Two separate XPU copies of the guard:

// src/ATen/native/xpu/sycl/RenormKernel.cpp:22-24  (Tensor.renorm / renorm_)
    const auto eps = static_cast<scalar_t>(1e-7);
    const auto one = static_cast<scalar_t>(1.0);
    return (norm > maxnorm_elm) ? maxnorm_elm / (norm + eps) : one;

// src/ATen/native/xpu/sycl/Embedding.cpp:146-148  (embedding / embedding_bag renorm)
  if (smem_[0] > max_norm) {
    auto factor = static_cast<scalar_t>(
        max_norm / (smem_[0] + std::numeric_limits<accscalar_t>::epsilon()));

RenormKernel.cpp is a faithful port of upstream's CUDA kernel at the scanned head (maxnorm_s / (norm + eps), eps = 1e-7). Embedding.cpp already deviates: upstream CUDA uses an absolute max_norm / (sdata[0] + 1e-7), XPU uses numeric_limits<accscalar_t>::epsilon(). After #204132 lands, upstream CPU/CUDA/MPS divide by norm exactly and both XPU copies are left adding an epsilon.

Target-path evidence: the weight, index, embedding output and renorm output are all xpu:0, and torch.xpu.synchronize() is called before each value is read.

Reproducer

import torch

x = torch.tensor([[3.0, 4.0], [0.6, 0.8], [0.0, 0.0]], dtype=torch.float64, device="xpu")
rn = x.renorm(2, 0, 1.0)
torch.xpu.synchronize()
print("row 0 norm after renorm:", float(rn[0].norm()))  # 0.9999999800000003, expected 1.0
print("1.0 * 5 / (5 + 1e-7)   :", 1.0 * 5.0 / (5.0 + 1e-7))

# float32, the dtype the upstream regression test uses
small = torch.tensor([[2e-6, 0.0]], device="xpu").renorm(2, 0, 1e-6)
torch.xpu.synchronize()
print("small max_norm:", float(small.norm()), "expected 1e-06")

Environment

torch 2.16.0.dev20261109+xpu
python 3.13.13
device Intel(R) Arc(TM) Pro B60 Graphics

Ownership

renorm_scale_factor_stub is registered by this repository (src/ATen/native/xpu/sycl/RenormKernel.cpp) and the embedding renorm kernel lives here too, so neither is reachable from the upstream fix. #204132 changes only CPU, CUDA and MPS; the XPU change is independent work in this repository and does not depend on #204132 landing.

Stage of the upstream fix: proposed, not landed (#204132 is open against main as of this scan, and its own test plan states that no local rebuild or device run was performed and that CI is required). Its approach, drop the guard and divide by norm, justified by norm > max_norm >= 0, applies unchanged to both XPU copies.

Related, not covering this finding:

• [xpu-alignment] embedding_renorm_ derives the row width from stride(0) and faults on a zero-width embedding #5152 (closed, fixed by Fix embedding_renorm_ row width to use size(1) not stride(0) #5163) was a different embedding_renorm_ defect in the same file: the row width came from stride(0) instead of size(1). It did not touch the epsilon, and closing it did not fix the guard.
• Align the semantic of accumulate type in the renorm operator with stock PyTorch #823 and Align the semantic of accumulate type in the renorm operator with stock PyTorch #822 (merged, 2024) aligned the renorm accumulate type with upstream, not the epsilon.

No open intel/torch-xpu-ops tracker was found for the renorm epsilon (searched renorm, max_norm, renorm 1e-7 epsilon norm, and PRs touching renorm).

Regression test

test/xpu/test_torch_xpu.py runs the upstream device-generic test_torch.py suite with no skips (test/xpu/skip_list_common.py:259 maps it to None). #204132 adds TestTorchDeviceType.test_renorm_exact_maxnorm, which asserts that torch.tensor([[2e-6, 0.0]], device=device).renorm(2, 0, 1e-6).norm() equals 1e-6 within atol=1e-8; in float32 the current XPU kernel returns about 9.524e-7, a 4.8e-8 miss, so that test will fail on XPU once the PR lands. #204132 also adds nn/test_embedding.py coverage for the functional, module and embedding_bag entry points on accelerator devices, which will exercise the Embedding.cpp copy. A fix here should additionally add a float32 F.embedding(max_norm=...) case, since the float64 path masks the dtype-relative epsilon.