Activity

1. wzx226226

   added

   bugSomething isn't workingSomething isn't working

   on Sep 2, 2026
2. lvyufeng commented on Sep 15, 2026

   

   lvyufeng

   on Sep 15, 2026

   Collaborator

   More actions

   标题？
3. wzx226226 commented on Sep 15, 2026

   

   wzx226226

   on Sep 15, 2026 via email

   Author

   More actions

   ？怎么了 万彰雄 ***@***.***
4. lvyufeng

   changed the title [-][BUG][/-] [+][Perf] aten::add.Tensor via FlagGems is slower than DAS boxing on DCU[/+] on Sep 26, 2026
5. lvyufeng commented on Sep 28, 2026

   

   lvyufeng

   on Sep 28, 2026

   Collaborator

   More actions

   Re-measured on current main. The regression still reproduces when the FlagGems route is forced, but the shipped DCU config no longer takes that route, so this is no longer reachable in a stock install. Details and the numbers below.

   What changed since the report

   Two things in the reproduction commands no longer hold on main:

   1. FLAGOS_USE_FLAGGEMS does not select anything on DCU any more. The per-op override is now FLAGOS_OP_<op>=<backend> with dots doubled (torch_fl/_env.py, LoadBackendConfig in csrc/aten/common.cc), e.g. FLAGOS_OP_add__Tensor=flaggems. The issue's two commands (FLAGOS_USE_FLAGGEMS unset vs set) now run the same route, so the A/B has to be expressed as an explicit override.

   2. add.Tensor is pinned to cuda in backends_dcu.conf. PR fix: repair DCU SDPA backend selection and pointwise routing for Qwen-Image #345 moved eleven pointwise overloads to boxing as part of the Qwen-Image-2512 work; add.Tensor is one of them (torch_fl/configs/backends_dcu.conf:769). The pin is for an independent correctness reason — with a bf16 tensor beside a Python float, ATen's wrapped number puts an f64 into FlagGems' pointwise kernel and the HCU backend aborts in TruncFOpConversion::createDestOps (FlagGems #6212) — not because of this perf issue. The effect on this issue is that the FlagGems arm is now reachable only by overriding that one entry, and a user running the installed conf gets boxing.

   Method

   The two arms are therefore

   arm	selector
   boxing	FLAGOS_OP_add__Tensor=cuda (what the shipped conf routes)
   flaggems	FLAGOS_OP_add__Tensor=flaggems (what #252 measured)

   The four cases, warmup, repetition count, synchronization and input shapes are the issue's, unchanged. Two deliberate deviations from the issue's script:

   • One process per (case, route). The route is resolved from the conf when torch_fl loads it, so the two arms cannot share an interpreter; the issue says this. A case's reading also moves by more than an order of magnitude when it shares an interpreter with the cases before it, so the per-case process boundary is what makes the four numbers comparable rather than a nicety.
   • Inputs are built on the host and moved to flagos:0, not created on the device. A device torch.randn is itself routed through the conf, so it is a different measurement under each arm.

   The route each arm actually took was confirmed with FLAGOS_LOG=dispatch, not inferred from the selector:

   [flagos dispatch] add.Tensor -> flagos_python
   

   with the executing frame at flag_gems/ops/add.py:107 in add — i.e. the FlagGems Python/Triton path the issue is about, on the arm that is supposed to reach it.

   Result

   Two consecutive runs, flagos:0, warmup 30 / repeat 200, synchronized batch average:

   case	boxing µs/call (run 1 / run 2)	FlagGems µs/call (run 1 / run 2)	ratio (run 1 / run 2)
   residual_101x3x320_noncontig	14.663 / 14.684	272.696 / 273.239	18.60x / 18.61x
   attention_60x101x16_noncontig	14.474 / 13.995	275.136 / 269.080	19.01x / 19.23x
   ffn_101x3x1280_scalar	15.357 / 15.673	203.016 / 204.989	13.22x / 13.08x
   bias_3x101x33	13.487 / 14.016	265.998 / 263.096	19.72x / 18.77x

   All four cases produced finite outputs on both arms, and both arms exit 0.

   The regression is unchanged in shape: three of four cases reproduce at 18.6x–19.7x against the report's 18.93x–19.97x, and the ordering is the same apart from the scalar case. Two differences worth recording rather than smoothing over:

   • Absolute latency is lower on both arms here than in the report (boxing 13.5–15.7 µs vs 15.4–18.1 µs; FlagGems 203–275 µs vs 292–337 µs). Different node and a different FlagGems build; the ratio is the quantity that carries over, not the absolutes.
   • The scalar case moved from the report's worst ratio (18.56x, 336.5 µs) to the best here (13.1x, 204 µs). That is a real attenuation on the current FlagGems build for that one case, not scatter — it reproduces across both runs to within 0.15x. It is still 6.5x the 2x threshold the issue proposes.

   Environment

   • Hygon DCU / BW1000, gfx936, DTK 26.04, HIP 6.3.26113
   • torch 2.10.0+cpu (DCU boxing install), flag_gems 5.4.0rc2.post1+g437ba3938
   • Torch-FL tree = e95bea7 (fix: make gen_vendor_confs.py the sole writer of the vendor confs #461) + the [BUG][DCU][Distributed] Build and package _flagos_nccl against libtorch_hip and RCCL #366 wheel-bridge commit; torch_fl/configs/backends_dcu.conf and every DCU source are byte-identical between e95bea7 and current main 6b699995 (that commit is MUSA-only), so the configuration measured here is main's
   • Device flagos:0, 8 devices visible, no HIP_VISIBLE_DEVICES pinning

   Provenance gap, stated rather than glossed: this run is in a container whose hostname is 98c077ef0619, not on the report's a02r4n07 / a02r2n06. The A/B is same-node, same-device, same-process-boundary discipline within this run, so the ratio is comparable; the absolute numbers are not a reproduction on the report's hardware. If a same-node comparison against a02r4n07 matters, the harness below runs as-is there.

   Where the residual lives

   Unchanged from the report's own investigation: the device kernel is a few microseconds and the rest is host-side — pointwise_dynamic.prepare_args, the generated wrapper, config/dtype/stride metadata and launch preparation, per call, with no signature-keyed caching of the prepared metadata. Nothing in Torch-FL changed that, and the fix directions the issue lists (cache op.meta() by layout signature, cache launch descriptors, a C++ fast path for common binary elementwise ops) are all FlagGems-side. That work is tracked upstream by FlagGems #6570 ([Performance][DCU] FlagGems pointwise_dynamic adds high steady-state latency to aten::add.Tensor, the same report, open) and by the generic root cause in FlagGems #4812 (pointwise_dynamic/libentry per-call CPU dispatch overhead dominates small-tensor latency).

   Disposition

   Closing this: the shipped DCU configuration routes add.Tensor to boxing for an independent correctness reason, so no installed DCU user reaches the slow path and the "Expected Behavior" section's framing ("or the backend should provide a general mechanism that avoids paying the full Python/Triton dispatch cost") is now the whole of the remaining ask — and it belongs to FlagGems, where it is filed. Please reopen if the cuda pin is ever lifted for add.Tensor, or if a DCU conf is shipped that routes it to FlagGems again; the tables above are the baseline to compare against. The 2x regression threshold the issue proposes is not implemented as a test — a threshold test whose arms both need a DCU runner and whose expected ratio is currently 19x would be red on arrival, and this repo measures routes on demand rather than in CI (tests/manual/flaggems_overload_survey.py).

   Harness used for the two runs

   tests/manual/dcu_add_tensor_ab.py in style (one subprocess per case/route, like the existing dcu_pointwise_replay.py):

   import argparse
   import json
   import os
   import subprocess
   import sys
   from pathlib import Path
   
   CHILD = r"""
   import json
   import sys
   import time
   
   import torch_fl  # noqa: F401  - must precede `import torch`
   import torch
   
   name = sys.argv[1]
   device = torch.device(sys.argv[2])
   warmup = int(sys.argv[3])
   repeat = int(sys.argv[4])
   
   # Built on the host and moved, rather than created on the device: a device
   # `torch.randn` is itself routed through the conf (to flag_gems.ops.randn under
   # the FlagGems arm), which is a different measurement.
   CASES = {
       "residual_101x3x320_noncontig": lambda: (
           torch.randn(3, 101, 320).to(device).transpose(0, 1),
           torch.randn(101, 3, 320).to(device),
       ),
       "attention_60x101x16_noncontig": lambda: (
           torch.randn(101, 60, 16).to(device).transpose(0, 1),
           torch.randn(60, 101, 16).to(device),
       ),
       "ffn_101x3x1280_scalar": lambda: (
           torch.randn(101, 3, 1280).to(device),
           1.0,
       ),
       "bias_3x101x33": lambda: (
           torch.randn(3, 101, 33).to(device),
           torch.randn(33).to(device),
       ),
   }
   
   left, right = CASES[name]()
   
   for _ in range(warmup):
       output = torch.ops.aten.add.Tensor(left, right, alpha=1)
   torch_fl.flagos.synchronize(device)
   
   started = time.perf_counter()
   for _ in range(repeat):
       output = torch.ops.aten.add.Tensor(left, right, alpha=1)
   torch_fl.flagos.synchronize(device)
   elapsed_us = (time.perf_counter() - started) * 1e6 / repeat
   
   assert torch.isfinite(output).all().item()
   print("RESULT " + json.dumps(elapsed_us))
   """
   
   CASES = (
       "residual_101x3x320_noncontig",
       "attention_60x101x16_noncontig",
       "ffn_101x3x1280_scalar",
       "bias_3x101x33",
   )
   
   ROUTES = {"boxing": "cuda", "flaggems": "flaggems"}
   
   
   def run_once(route, case, device, warmup, repeat, timeout):
       env = dict(os.environ)
       env["FLAGOS_OP_add__Tensor"] = ROUTES[route]
       child = Path("/tmp") / f"dcu_add_tensor_ab_{route}_{case}.py"
       child.write_text(CHILD, encoding="utf-8")
       proc = subprocess.run(
           [sys.executable, str(child), case, device, str(warmup), str(repeat)],
           capture_output=True,
           text=True,
           env=env,
           timeout=timeout,
           check=False,
       )
       out = proc.stdout + proc.stderr
       for line in out.splitlines():
           if line.startswith("RESULT "):
               return json.loads(line[len("RESULT ") :])
       raise SystemExit(f"{route}/{case} produced no measurement (exit {proc.returncode}):\n{out}")
   
   
   def main():
       parser = argparse.ArgumentParser()
       parser.add_argument("--device", default="flagos:0")
       parser.add_argument("--warmup", type=int, default=30)
       parser.add_argument("--repeat", type=int, default=200)
       parser.add_argument("--timeout", type=int, default=900)
       args = parser.parse_args()
   
       print(f"device: {args.device}   warmup: {args.warmup}   repeat: {args.repeat}")
       print(f"{'case':32s} {'boxing us/call':>14s} {'flaggems us/call':>16s} {'ratio':>7s}")
       for case in CASES:
           boxing = run_once("boxing", case, args.device, args.warmup, args.repeat, args.timeout)
           flaggems = run_once("flaggems", case, args.device, args.warmup, args.repeat, args.timeout)
           print(f"{case:32s} {boxing:14.3f} {flaggems:16.3f} {flaggems / boxing:6.2f}x")
       return 0
   
   
   if __name__ == "__main__":
       sys.exit(main())

   The one environment note for running it on the shared install: keep /usr/local/lib and /usr/local/lib64 off LD_LIBRARY_PATH so the checkout's libtorch_fl is the one that loads, and set FLAGOS_ACCELERATOR=dcu plus the DTK roots.
6. lvyufeng

   closed this as not plannedon Sep 28, 2026