[AI][Performance][DCU] R5 host-dispatch patch silently skipped with pinned FlagGems: missing _descriptor_cache_key #586

Issue Type

• Performance Issue

AI Agent Information

• Agent: Codex.
• Model: The exact model identifier is not exposed by this session.
• Session Context: The user authorized a read-only, pinned-R5 diagnostic and this compatibility report, alongside additional cross-operator evidence on FlagGems #4812. No shared package modification, source patch, rebuild, or performance-fix claim is involved.

Summary

The published R5 Torch-FL/FlagGems combination contains the #352 host-dispatch installer, but its LibEntry.run/key optimization is silently skipped. With both eligibility gates true, the installer accesses libentry._descriptor_cache_key; the pinned FlagGems module has no such attribute. An actual hardware diagnostic on 2026-10-19 captured the resulting AttributeError inside _patch_flaggems_pointwise_dispatch. Its broad exception handler suppresses the error, so it returns normally and leaves the original methods installed. Subsequent real native_layer_norm calls use stock LibEntry.run/key.

Scope: this is a deployed/pinned-dependency compatibility and observability report. It does not claim that latest Torch-FL plus latest FlagGems fails, that every part of #352 is inactive, or that mathematical correctness or HIP runtime is broken. Current FlagGems master does provide the helper; the tested R5 pin does not.

Environment

• Linux; Python 3.10.20; one allocated Hygon DCU. The successful compiled layer-norm code targets gfx936, HIP backend, warp size 64.
• Official PyTorch frontend 2.10.0+cpu, with Torch-FL's decoupled DCU integration; runtime reports HIP 6.3.26113, DTK 26.04.
• FlagTree 0.6.0+hcu3.6.glibc228.compat1.git8a1dda94 / Triton 3.6.0.
• Release r5-20260928-tfl6d84d12b-fge7b4a865.
• Torch-FL 2.10.0+das.dtk2604.glibc228.g6d84d12b, commit 6d84d12b715c307e5b660fc33e0f51e660602f6a.
• FlagGems 5.4.0rc2.post1+ge7b4a865f, commit e7b4a865fce6d85861ee91a6aca56564ef9acf7d.
• The two installed Python source hashes exactly match the files retrieved from those immutable GitHub commits:
  • torch_fl/flagos/__init__.py: SHA256 d4819e8922def7ef77073b5bf8bf158c3e030bea21986426a98bc589169b1e37.
  • flag_gems/utils/libentry.py: SHA256 4c22919c3ec3a4a0ae652326fe36496e0a5db83c7a378a57a369438c5d6054ba.

Build config: existing published prebuilt R5 artifacts were used; no build was performed for this report. The original compiler invocation is not reconstructed. Manifest commits and runtime source hashes were checked, not inferred solely from version strings.

Runtime config: source the appropriate pinned R5 activation entry, use its shipped backends_dcu.conf (native_layer_norm = flaggems), keep its DTK/HIP loader setup, and run in a separate process on one allocated DCU. ACCELERATOR=dcu, GEMS_VENDOR=hygon, FLAGTREE_BACKEND=hcu, FLAGOS_USE_FLAGTREE=1; FLAGOS_USE_FLAGGEMS and FLAGOS_USE_FLAGGEMS_CPP were unset. The selected config hash was 0dd915f8b1377125ae0b2700c4a444489bed80f4156ef9907fa7c80fd9da6198. No route changed. Activation/data paths are deliberately not required by the public reproducer.

Reproduction

The following is a minimal reduction of the diagnostic. It needs the pinned dependency combination above, no model or private dataset. torch_fl is imported before torch. The full validation additionally checked source identities, CPU-reference numerics, device/alias/metadata invariants, and post-install call witnesses. The output below is from that full hardware validation; the reduced listing was syntax-checked, not separately benchmarked or hardware-run.

import importlib
import sys
import torch_fl  # Must precede torch in this deployment.
import torch

lib = importlib.import_module("flag_gems.utils.libentry")
dev = importlib.import_module("torch_fl.flagos")
torch.manual_seed(1731)
x = torch.randn(3, 101, 320).to("flagos:0").transpose(0, 1)
with torch.inference_mode():
    output = torch.ops.aten.native_layer_norm.default(x, [320], None, None, 1e-5)
torch.flagos.synchronize()
print("gates:", torch_fl._build_accelerator(), torch_fl._conf_routes_to_flaggems())
print("helper:", hasattr(lib, "_descriptor_cache_key"))
install = dev._patch_flaggems_pointwise_dispatch
before = (lib.LibEntry.run, lib.LibEntry.key)
events = []
def trace(frame, event, arg):
    if frame.f_code is install.__code__:
        if event == "exception":
            events.append((arg[0].__name__, str(arg[1]), frame.f_lineno))
        return trace
    return None
try:
    sys.settrace(trace)
    install()
finally:
    sys.settrace(None)
print("caught inside installer:", events)
print("same methods:", before == (lib.LibEntry.run, lib.LibEntry.key))
print("fast markers:", [bool(getattr(f, "_torch_fl_fast", False)) for f in before])

This invokes the existing optional installer in an isolated process; it does not implement a replacement, inject a missing helper, or alter shared files. Trace hooks are restricted to that installer frame and removed in finally.

Expected vs Actual Behavior

Expected: the supported/published dependency combination either installs a semantically compatible optimization, or explicitly reports that this component is unsupported/skipped and why. A normal initialization return must not be mistaken for proof that the launch fast path is active.

Actual captured result:

build_accelerator = dcu
config_routes_to_flaggems = True
helper_available = False

Caught inside _patch_flaggems_pointwise_dispatch:
  torch_fl/flagos/__init__.py:1916
  if not getattr(libentry._descriptor_cache_key, "_torch_fl_hoisted", False):
AttributeError: module 'flag_gems.utils.libentry' has no attribute '_descriptor_cache_key'

installer_returned_normally = True
methods_same_objects = True
run: LibEntry.run, libentry.py:1632, _torch_fl_fast=False (before and after)
key: LibEntry.key, libentry.py:1601, _torch_fl_fast=False (before and after)
post-install stock run calls = 2
post-install stock key calls = 2
trace/profile hooks cleared = True

This is a caught exception event, not an uncaught process-crash traceback. The diagnostic and scheduler job completed with exit status 0, after positively checking these observations.

Root Cause Analysis / Bottom-layer Execution

The verified operation chain is eager aten::native_layer_norm.default on a noncontiguous flagos input -> Torch-FL PrivateUse1/config route -> FlagGems native_layer_norm -> layer_norm -> persistent Triton/FlagTree kernel -> HIP. There is no torch.compile/Inductor graph or cross-op fusion in this reproduction.

The installer resolves the descriptor helper before rebinding either LibEntry.key or LibEntry.run. The pinned FlagGems file normalizes descriptor-like arguments inside the original run implementation instead of exporting _descriptor_cache_key. Attribute lookup therefore fails; the installer's outer except Exception: pass hides the failure. Both gates were measured true, the exception site was actually captured, method objects were unchanged, and subsequent target calls entered the original methods. This localizes the failure to Torch-FL's optional FlagGems patch/dependency contract, not a slow device normalization kernel.

The seeded workload has FP32 shape [101,3,320], stride [320,32320,1]; the full diagnostic used affine weight/bias [320] and eps=1e-5. All three output leaves matched independent CPU ATen within rtol=atol=1e-4 on both checked post-install calls. Output devices, fresh/nonalias storage, input values/metadata immutability and prior-output stability passed. The minimal listing omits affine only to keep the installer reproduction short; it does not provide the historical performance measurements.

Performance Context (not a patch A/B)

A separate 2026-10-16, same-R5 route comparison on a BW/gfx936 device measured this captured norm shape:

Default: 5 independent processes x 100 primary samples; boxing: 3 later independent processes x 100. Correctness passed. The private control config differed only in native_layer_norm = cuda; shared files were unchanged. These are not paired AB/BA confirmations, device-kernel timings, or a measurement of the missing patch's potential speedup. The 2026-10-19 diagnostic above performed no formal performance measurement.

Proposed Solution

1. Check required capabilities/dependency support before installing any component. Either declare/enforce an appropriate supported FlagGems version or provide a reviewed adapter for the old ABI; do not silently assume this helper exists.
2. Make optional performance setup observable: component installed/skipped status and a specific reason, at least an actionable once-per-process warning for unsupported published combinations. Keep unsupported configurations safe rather than turning an optional performance failure into a correctness failure.
3. Test actual callers and post-warmup target hits, not only enable() returning or all-positional mocks. Preserve keyword constexpr/options, autotuner/heuristic wrappers, descriptor/specialization rules, alignment and current-stream semantics. Do not solve this by merely widening a kwargs allowlist or copying a complete initialization file over another version.
4. Do not cache tensor results, retain input pointers, or reuse stale streams as a way to remove metadata cost.

Verification Plan

• Compatibility/unit test for a LibEntry module without _descriptor_cache_key: explicit supported behavior or skip reason, no silent partial installation.
• Test a supported helper-present module, repeated initialization/idempotence and unsupported-call stock fallback.
• Integration matrix covering the published R5 dependency pin and a supported newer pair; use real generated pointwise kwargs and autotuned reduction launches.
• Check target main-kernel fast-path hits separately from incidental copy/helper kernels; preserve output device, numerics, alias, input metadata and non-default-stream semantics.
• Only after proving the patch is active: unprofiled same-route A/B for eq.Scalar, masked_fill.Scalar, native_layer_norm, plus small add/sum controls. Report host/API and device time separately.

Context & Investigation

Inspected the shipped module files and exact immutable commit versions, runtime method identities and flags, configuration gates, caught exception event, real post-install call witnesses and correctness invariants. Rechecked current upstream on 2026-10-19: Torch-FL 530d54f49163962a618a6c6849d2de9c69b96bb6 still accesses this helper; FlagGems f89952b8afd1fdbf44e99265ec7835c3aa2bb532 does define it. That source inspection does not certify latest/latest hardware behavior and is not evidence that this old pair was upgraded.

Related common-overhead reports: flagos-ai/FlagGems#4812 and flagos-ai/FlagGems#6570; R5 integration umbrella #582; original optimization #352. #359 is merged, but widening its platform gate does not supply the missing API in the tested old FlagGems pin. No same-root compatibility issue was found in the checked open/closed issue searches.

Related Code Locations

• Pinned installer and direct helper access.
• Pinned original LibEntry key/run.
• Current Torch-FL installer dependency.

Checklist

• Complete environment, pin and build/runtime provenance supplied; no compiler invocation invented.
• Self-contained reduced reproducer plus observed hardware diagnostic result supplied.
• Actual caught exception context supplied; not fabricated as an uncaught traceback.
• Root cause, owner boundary, proposed solution and regression plan supplied.
• Affected files/lines and immutable source references supplied.
• Duplicate issues and current upstream checked.
• All text is English; code follows ordinary Python import/indentation conventions.