[AI][BUG][DCU] Track FlagGems remainder errors for infinite divisors #658

Issue Type

• Bug Report

AI Agent Information

• Agent: Codex coding agent.
• Model: Exact identifier not exposed/recorded; not inferred.
• Session Context: User-authorized separate operator reports following a three-way CPU/DAS/FlagGems audit of correctness-test exports, with paired Torch-FL integration tracking.

Summary

Integration tracking for the primary FlagGems operator report: flagos-ai/FlagGems#7755.
aten.remainder.Tensor on the Hygon FlagGems backend returns NaN for cases such as remainder(2,+Inf), remainder(0,+Inf), and remainder(-2,-Inf), where CPU and DAS return the finite dividend. This changes exceptional-value masks and can propagate NaNs downstream.

This tracker is for affected Torch-FL routing/compatibility and acceptance of an upstream fix. It does not allege a second independent Torch-FL kernel defect or request changing the whole initialization file.

Environment (for bug reports)

• Platform: Hygon DCU, Torch-FL PrivateUse1 (flagos), eager execution; one accelerator.
• Device properties report BW, gfx936:sramecc+:xnack-, 80 CUs, warp size 64, 68702699520 bytes memory. This is a device-API report, not a separate PCI-ID audit.
• OS: Linux 5.10.134-17.1.3.sga8.x86_64, glibc 2.28.
• Python: 3.10.20, GCC 14.3.0.
• PyTorch frontend: 2.10.0+cpu, source 449b1768410104d3ed79d3bcfe4ba1d65c7f22c0; effective HIP metadata 6.3.26113. The frontend suffix does not mean the tested tensors executed on CPU.
• Torch-FL: 2.10.0+das.dtk2604.glibc228.g6d84d12b, source 6d84d12b715c307e5b660fc33e0f51e660602f6a.
• FlagGems: 5.4.0rc2.post1+ge7b4a865f, source e7b4a865fce6d85861ee91a6aca56564ef9acf7d.
• FlagTree: 0.6.0+hcu3.6.glibc228.compat1.git8a1dda94; exposed Triton version 3.6.0.
• Loaded hycu driver-module version: 6.3.31-V1.5.0a. A native HIP shared-library path/PCI query was not captured; no stronger driver identity is inferred.
• Tested release: r5-20260928-tfl6d84d12b-fge7b4a865. Prebuilt deployed environment; build commands were not run or recovered in this reproduction. No package install or shared source/configuration modification.

Build config: prebuilt deployment; the original build command is unavailable and was not guessed. No rebuild/install in this reproduction.

Runtime config and scope:
The report was prompted by a 2026-10-21 test export, but is supported by new deterministic three-way controls, not by the CSV verdict alone. On 2026-10-26, a full supplemental diagnostic ran 53 cases per route; a separate reduced-reproducer job then executed the scripts below in three fresh processes (CPU, DAS, FlagGems). The reduced job completed successfully in 133 seconds; success means the harness matched the expected good controls and expected FlagGems failure, not that FlagGems passed correctness. Actual ATen calls and Python implementation entries were witnessed, source/configuration hashes were retained and loaded source hashes stayed unchanged. No performance or fix claim is made.

A preceding reduced job failed because its witness filter omitted the Hygon backend override; that failed result was retained, the diagnostic filter alone was corrected in a separate revision, and the independent successful run is used here. These are synthetic public inputs, not a replay of the unavailable original Lilith harness/inputs. Current FlagGems mainline was inspected at 415ea38c63141382b0051eac5ef0fa22cbbbec69; current mainline and proposed fixes were not hardware-tested.

Save the code as repro.py. Run each route in a fresh process in the pinned environment:

ROUTE=cpu python repro.py
ROUTE=das python repro.py
ROUTE=flaggems python repro.py

The script sets the relevant FLAGOS_OP_* overrides before importing Torch-FL (cuda is the DAS-boxing route selector here; it is not an NVIDIA device claim). No private file/configuration is needed by the reproducer. The observed route was independently checked with ATen tracing and actual FlagGems entry calls; CPU and DAS controls had no FlagGems operator-entry calls. Targeted overrides were used rather than changing the installed default configuration.

Reproduction

OPS = ['remainder__Tensor']
# Standalone route-controlled correctness reproduction.
import os
route = os.environ.get("ROUTE", "flaggems")
assert route in ("cpu", "das", "flaggems")
if route != "cpu":
    for op in OPS:
        os.environ["FLAGOS_OP_" + op] = "cuda" if route == "das" else "flaggems"
    import torch_fl
import torch
if route != "cpu":
    torch.cuda.set_device(0)
device = "cpu" if route == "cpu" else "flagos:0"
base = torch.tensor([-float("inf"), -2., -0., 0., 2., float("inf"), float("nan"), 1e-45, -1e-45, 1e30, -1e30])
base = base.repeat(1490)[:16384].reshape(128, 128)
den = torch.tensor([-2., 2., -float("inf"), float("inf"), 0., float("nan"), 0.5]).repeat(2341)[:16384].reshape(128, 128)
ref = torch.remainder(base, den)
out = torch.remainder(base.to(device), den.to(device)).cpu()
for i in (2, 3, 23, 59):
    print(route, i, base.flatten()[i].item(), den.flatten()[i].item(), ref.flatten()[i].item(), out.flatten()[i].item())
torch.testing.assert_close(out, ref, rtol=0, atol=0, equal_nan=True)

Expected vs Actual Behavior

CPU/DAS return 2 for remainder(2,+Inf), +0 for remainder(+0,+Inf), -2 for remainder(-2,-Inf), and -0 for remainder(-0,-Inf). FlagGems returns NaN in these cases. The full deterministic special-value matrix has 3406 mismatches in the FP32 supplemental test; the public reduced run also reproduces the exceptional-value failures.

Actual route and diagnostics

Target FlagGems entry witness(es): {"flag_gems/runtime/backend/_hygon/ops/div.py:remainder": 1}. Target ATen overloads are present in the captured dispatcher trace. Route controls and implementation-entry witnesses were checked independently; a configuration setting alone was not treated as proof.

Observed stdout:

cpu 2 -0.0 -inf -0.0 -0.0
cpu 3 0.0 inf 0.0 0.0
cpu 23 -2.0 -inf -2.0 -2.0
cpu 59 2.0 inf 2.0 2.0
das 2 -0.0 -inf -0.0 -0.0
das 3 0.0 inf 0.0 0.0
das 23 -2.0 -inf -2.0 -2.0
das 59 2.0 inf 2.0 2.0
flaggems 2 -0.0 -inf -0.0 nan
flaggems 3 0.0 inf 0.0 nan
flaggems 23 -2.0 -inf -2.0 nan
flaggems 59 2.0 inf 2.0 nan

Full caught failure traceback from the instrumented reduced run (only absolute environment/repro roots redacted; diagnostic frames retained):

Traceback (most recent call last):
  File "<repro>/repro_runner.py", line 65, in <module>
    with contextlib.redirect_stdout(output),torch.inference_mode(),Trace():runpy.run_path(str(R/'repros'/name),run_name='__main__')
  File "<python>/runpy.py", line 289, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "<python>/runpy.py", line 96, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "<python>/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "<repro>/repros/03-remainder-inf.py", line 21, in <module>
    torch.testing.assert_close(out, ref, rtol=0, atol=0, equal_nan=True)
  File "<python>/site-packages/torch/testing/_comparison.py", line 1600, in assert_close
    raise error_metas[0].to_error(msg)
AssertionError: Tensor-likes are not equal!

Mismatched elements: 3406 / 16384 (20.8%)
Greatest absolute difference: nan at index (0, 2)
Greatest relative difference: nan at index (0, 2)

Loaded source SHA256 (stable before/after run):

{
  "flag_gems/ops/remainder.py": "d629f7c1c7880966c3c0ce2be43dd92779826e59a1fffac1ec92d8fccc13968d",
  "flag_gems/runtime/backend/_hygon/ops/div.py": "708457acfae99a9e2922afae86bb75bad3232e62c28510278f8399229d440343",
  "triton/language/core.py": "b8c33dc0501719f2bf88c15242e3e5157faf2256fd887fa160489c1d5984f167"
}

Root Cause Analysis

Eager torch API -> exact ATen overload -> PrivateUse1/flagos Torch-FL route -> actual FlagGems entry -> backend execution (or round's pre-launch guard) -> output copied to CPU. CPU and DAS controls pass. No compile/fusion or performance claim.

The executed entry was flag_gems/runtime/backend/_hygon/ops/div.py:remainder, not the generic remainder.py entry. It dispatches tensor/tensor inputs to rem_tt, then _remainder: r = x % y; c1 = r != 0; c2 = (x < 0) ^ (y < 0); return tl.where(c1 & c2, r + y, r). This path has no explicit treatment for finite dividend/infinite divisor. The tested same-sign examples already fail without needing the sign-correction branch to explain them, so attributing everything to r+y would be unsupported. The evidence localizes the operator/backend arithmetic path; the precise remainder lowering still needs kernel/IR inspection before blaming a compiler or runtime. Primary ownership is the FlagGems Hygon override, with FlagTree involvement only if isolated lowering proves necessary.

Proposed Solution

Handle finite/Inf and signed-zero semantics explicitly in the Hygon override (and audit the corresponding generic arithmetic pattern), matching PyTorch remainder rather than only ordinary finite modulo. Isolate and inspect the % lowering before assigning a compiler defect. Do not use nan_to_num as a semantic fix.

Track and validate the primary FlagGems fix in the deployed pair. If a temporary boxing route is chosen, keep it per-op, preserve semantics, document it as a workaround rather than a root-cause fix, and do not silently change the shared environment.

Verification Plan

• Unit test: Exercise all sign combinations for finite dividend and +/-Inf divisor, signed zero, NaN/Inf inputs, zero divisor and normal finite cases; compare masks, finite values and signed zero separately against CPU/DAS. Extend scalar/tensor, broadcast, dtype and layout variants after the tensor/tensor reproducer.
• Integration test: retain fresh CPU/DAS/FlagGems processes, exact overload and actual implementation-entry witnesses; verify selected deployment and route, not just environment-variable strings.
• Manual verification: rerun the self-contained code on the candidate fixed pair, record hashes/versions, then separately assess rollout. No fix is claimed validated by filing.

Context & Investigation

The 2026-10-21 CSV only initiated triage. On 2026-10-26, a 53-case-per-route supplemental diagnostic and an independent reduced-script run on the pinned R5 pair supplied the evidence. Loaded source, backend overrides, route configuration, masks and CPU/DAS controls were inspected. The initial reduced harness omitted Hygon backend witnesses and failed; a separate corrected diagnostic revision passed all expected outcome/route checks. No source or shared configuration changes. Open/closed issues and PRs were searched again; no same-root Torch-FL tracker was found. Existing topk issue #173 is a different operator/root and was not reopened.

Related Code Locations

• https://github.com/flagos-ai/FlagGems/blob/e7b4a865fce6d85861ee91a6aca56564ef9acf7d/src/flag_gems/runtime/backend/_hygon/ops/div.py#L340-L376
• https://github.com/flagos-ai/FlagGems/blob/e7b4a865fce6d85861ee91a6aca56564ef9acf7d/src/flag_gems/ops/remainder.py#L1-L62
• Primary operator report: [Bug][Hygon/DCU] remainder returns NaN for finite dividends with infinite divisors FlagGems#7755

Checklist - AI Agents MUST Complete All

• I have provided complete environment information (unavailable original build command and native PCI/HIP-library identity are explicitly disclosed).
• I have included a minimal, self-contained reproducer.
• I have included full error output with traceback (absolute private roots redacted only).
• I have analyzed the root cause, separating confirmed entry/source semantics from unvalidated lowering hypotheses.
• I have proposed a specific solution with implementation approach.
• I have identified affected code locations with line numbers.
• I have described how to verify the fix.
• I have checked for duplicate issues.
• All text is in English.
• Code follows the existing Torch-FL/FlagGems eager usage conventions.