Skip to content

[AI][BUG][Kunlunxin] libxpytorch.so cannot bind torch 2.10: c10_cuda_check_implementation ABI signature break (…ib vs …jb), first flagos op hangs forever #573

Open
Open
[AI][BUG][Kunlunxin] libxpytorch.so cannot bind torch 2.10: c10_cuda_check_implementation ABI signature break (…ib vs …jb), first flagos op hangs forever#573
Labels
P1Priority 1 - highai-generatedbugSomething isn't working

Description

@lvyufeng
lvyufeng
opened on Oct 16, 2026

Issue Type

  • Bug Report

AI Agent Information

  • Agent: Claude Code CLI
  • Model: Claude (1M context)
  • Session Context: Working the goal "fix all bert-related issues so every chip fully passes bert, including nvidia". 8 of 10 fleet chips are now measured at 151/0 on a compliant stack. Kunlunxin and TsingMicro are the remaining two. This issue files the Kunlunxin half — a vendor plugin / torch ABI break — with exact symbols from both sides of the boundary.

Summary

Kunlunxin's XPU plugin libxpytorch.so cannot bind to this torch build: it imports _ZN3c104cuda29c10_cuda_check_implementationEiPKcS2_ib, while the installed torch/lib/libc10_cuda.so defines only _ZN3c104cuda29c10_cuda_check_implementationEiPKcS2_jb. The differing final parameter (ib = unsigned int, bool vs jb) is a torch 2.9 → 2.10 C++ signature change, so this is an ABI break inside libc10_cuda.so rather than a missing symbol — which is exactly why the dynamic loader emits only a warning instead of failing hard. The plugin then does not bind, and the first flagos operation spins forever (100% CPU, never returns). No route change can pass bert on this chip until the plugin and torch agree.

Environment

Click to expand environment details
  • Platform: Kunlunxin XPU (P800-96G)
  • Host: bm-dhrj-bj-zone8-d-p800-96g-1-13, kernel 5.4.119-19-0009.56
  • Container: lvyufeng-torch-fl
  • Python: 3.10 (env torch-fl-hf-test)
  • PyTorch: 2.10.0+cu129 (vendor)
  • torch_fl: c2ded4b (upstream main), editable at kx-c2ded4b
  • torch_xmlir plugin: <env>/lib/python3.10/site-packages/torch_xmlir/libxpytorch.so
  • 8 devices visible once the env is conda-activated

Runtime config:

# Must conda-activate: activate.d sets a vendor-first LD_LIBRARY_PATH
docker exec lvyufeng-torch-fl bash -lc \
  'source /public-flash/lvyufeng/miniconda3/etc/profile.d/conda.sh; \
   conda activate torch-fl-hf-test; python ...'

Reproduction

E=/public-flash/lvyufeng/miniconda3/envs/torch-fl-hf-test

# 1. What libc10_cuda.so DEFINES
nm -D --defined-only $E/lib/python3.10/site-packages/torch/lib/libc10_cuda.so \
  | grep c10_cuda_check_implementation

# 2. What the vendor plugin WANTS
nm -D --undefined-only $E/lib/python3.10/site-packages/torch_xmlir/libxpytorch.so \
  | grep c10_cuda_check_implementation

# 3. The symptom: a plain flagos op never returns
docker exec lvyufeng-torch-fl bash -lc \
  'source /public-flash/lvyufeng/miniconda3/etc/profile.d/conda.sh; \
   conda activate torch-fl-hf-test; \
   timeout 60 python -c "import torch, torch_fl; \
print(torch.cuda.is_available(), torch.cuda.device_count()); \
x = torch.randn(4, 4, device=\"flagos\"); print(x.sum().item())"'

Expected vs Actual Behavior

Expected: torch.cuda.is_available() → True, 8 devices, and the flagos op returns.

Actual — the ABI break, exact symbols from both sides:

$ nm -D --defined-only .../torch/lib/libc10_cuda.so | grep c10_cuda_check_implementation
00000000000440c0 T _ZN3c104cuda29c10_cuda_check_implementationEiPKcS2_jb

$ nm -D --undefined-only .../torch_xmlir/libxpytorch.so | grep c10_cuda_check_implementation
                 U _ZN3c104cuda29c10_cuda_check_implementationEiPKcS2_ib

Actual — the runtime symptom:

WARNING: import hook error: .../torch_xmlir/libxpytorch.so: undefined symbol:
  _ZN3c104cuda29c10_cuda_check_implementationEiPKcS2_ib

torch.cuda.is_available() is True and device_count is 8 — the plugin's import emitted only a warning — but the first flagos op spins at 100% CPU and timeout kills it (exit 124). No bert count is reachable.

Root Cause Analysis

Mechanism: a C++ signature change in c10 across torch minor versions.

Demangling the two names:

  • Definition (jb): c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, unsigned int, bool)
  • Reference (ib): c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool)

torch 2.9 exported the …ib form (4th parameter int); this build is torch 2.10, which widened that parameter to unsigned int and therefore exports …jb. libxpytorch.so was built against the 2.9 form. Because C++ mangled names encode the full signature, the two are different symbols, so libxpytorch.so's reference is unresolved.

Why it only warns instead of failing to load: the symbol name differs, not the library. dlopen of libxpytorch.so does not fail outright in this configuration — the import hook reports the unresolved symbol as a warning — so torch and torch_fl import successfully and the failure is deferred. The deferred failure is a hang, not an exception, because the vendor runtime spins on a path that never completes. This is why the chip reads as "imports fine, then hangs" and why the symptom is easy to misattribute to a torch_fl routing defect.

Why it is not route-dependent (important for triage): the spin is op-shape-dependent, not xdnn/vendor-route-dependent. x@y via the vendor route passes, and a flagos→cpu copy passes, while both x*y via the vendor route and a non-vendor aten.add hang. So it is not "the vendor op path is broken"; it is that the plugin's device management is unusable once it fails to bind.

Why this is not a torch_fl defect: the unresolved symbol is inside torch/lib/libc10_cuda.so (torch's own library) and the reference is inside a vendor .so (libxpytorch.so). torch_fl's route configuration operates above both. No conf/codegen change moves either symbol.

Proposed Solution

Two viable directions, both infrastructure/vendor-side:

  1. Align the plugin with torch 2.10 — rebuild libxpytorch.so against this torch build so it references the …jb form. This is the forward-looking fix.
  2. Pin the plugin to its matching torch minor — install the torch 2.9-era build the plugin was compiled against, and pin it, so the …ib reference resolves.

A partial workaround worth stating explicitly because it is not sufficient: shipping the ABI mismatch as a hard error instead of a warning would convert the hang into a legible failure, but it would not make bert pass. It is a diagnosability improvement, at best a follow-up.

No change to torch_fl is proposed or needed.

Verification Plan

  • nm -D --defined-only .../torch/lib/libc10_cuda.so | grep c10_cuda_check_implementation and the same for libxpytorch.so now show the same mangled name
  • The WARNING: import hook error ... undefined symbol line is absent at import
  • torch.randn(4, 4, device="flagos") (and torch.ones(4, 4, device="flagos") * 2) returns instead of timing out
  • The non-vendor aten.add case that currently hangs returns
  • Suite-level: run the bert suite in lvyufeng-torch-fl, env torch-fl-hf-test, tree c2ded4b, and report the = N passed / M failed = line

Context & Investigation

  • Started from "import torch / import torch_fl fail", which turned out to be a measurement artifact: calling $E/bin/python without conda activate skips activate.d, which sets a vendor-first LD_LIBRARY_PATH (/usr/local/xpu/lib:/usr/local/xcudart/lib:<torch_xmlir>:$CONDA_PREFIX/lib). Without it the XPU import hook falls back to re-executing torch/__init__.py and torch/library.py raises RuntimeError: Only a single TORCH_LIBRARY can be used to register the namespace triton. That double-triton error is a missing-LD_LIBRARY_PATH artifact, not a torch defect — with conda activate both imports succeed. This issue is about what remains after that.
  • Checked whether the plugin loads at all: it does — ops reach the vendor runtime — so this is not a "plugin missing" problem.
  • Tested whether the hang is route-specific: it is not. x@y (vendor) and a flagos→cpu copy both pass; x*y (vendor) and a non-vendor aten.add both hang. Shape-dependent, not route-dependent. That refuted the earlier working theory that the xdnn/vendor route was the culprit.
  • Extracted the unresolved symbol name from the import hook warning and compared it against the definition in torch's own libc10_cuda.so with nm on both files — this is what converted a vague "unresolved symbol" into a pinned ABI signature break.
  • Ruled out the py311 env as a red herring: torch-fl-xpu-py311 imports cleanly but carries no flagtree/triton and reports devs 0, so its apparent health is not evidence.

Superseded reading (retracted): an earlier note attributed the residual to "index.Tensor needs a dispatch key" / a normal_ route defect. That is not supported: the trigger point varies with op shape and the ABI break above explains the hang. Do not re-file the index/dispatch theory without new evidence.

Related Code Locations

No torch_fl source location is implicated. The failing boundary is:

  • torch/lib/libc10_cuda.so — defines _ZN3c104cuda29c10_cuda_check_implementationEiPKcS2_jb
  • torch_xmlir/libxpytorch.so — references _ZN3c104cuda29c10_cuda_check_implementationEiPKcS2_ib
  • XPU import hook (xpytorch_import_hook → torch_xray_import_hook) — reports the unresolved symbol as a warning

Related Issues

Checklist - AI Agents MUST Complete All

  • I have provided complete environment information
  • I have included a minimal, self-contained reproducer
  • I have included full error output with traceback
  • I have analyzed the root cause (not just symptoms)
  • I have proposed a specific solution with implementation approach
  • I have identified affected code locations with line numbers
  • I have described how to verify the fix
  • I have checked for duplicate issues
  • All text is in English (required per CLAUDE.md)
  • Code follows project conventions (checked existing code style)

Activity

  1. lvyufeng
    added
    bugSomething isn't working
    P1Priority 1 - high
    ai-generated
    on Oct 16, 2026

  2. lvyufeng commented on Oct 17, 2026

    @lvyufeng
    lvyufeng
    on Oct 17, 2026
    CollaboratorAuthor

    Correction: this is a fleet-environment provisioning fault, not a chip-level ABI break

    Investigated further on 2026-10-17. The ABI mismatch in the original report is real, but its
    framing is misleading: kunlunxin's own CI lane passes, on a different stack than the one
    this issue was measured on.

    Evidence 1 — the kunlunxin CI lane is green

    On run 38856759463, job
    116830306874 (Platform pipeline (kunlunxin) / Build and test (Kunlun P800 XPU)):

    conclusion: success   11m13s
  Run Kunlun P800 XPU integration tests: success

    That step is not a no-op. .github/integration-manifest.yml defines the kunlunxin
    vendor_ops group as:

    python -m pytest tests/integration/ops/ \
  -m "(anyplatform or main_ops) and not flaggems and not flaggems_python and not flaggems_cpp" \
  --ignore=tests/integration/ops/test_flaggems_device_name.py -v -s --tb=short

    Evidence 2 — CI runs a DIFFERENT stack than the host this issue was measured on

    .github/configs/kunlunxin.yml + .github/scripts/set_env_kunlunxin.sh state the contract:

    setting value
    ci_image flagos-runtime-kunlunxin-xre5.37.1:2.2.0
    environment_check.torch_version_base 2.10.0
    environment_check.cuda_version null
    environment_check.flagos_devices at_least: 1
    environment_check.triton_backend xpu (flagtree, not stock triton)

    The hook is explicit: kunlunxin is a native PrivateUse1 backend — the port's runtime
    under csrc/runtime/accelerator/kunlunxin/ calls the vendor libxpurt runtime, kernels
    come from the vendor xdnn library, "There is no CUDA boxing", and "a stock CPU
    PyTorch wheel is enough     to build and run the backend."

    Evidence 3 — the host violates that contract in every env

    Host bm-dhrj-bj-zone8-d-p800-96g-1-13, container lvyufeng-torch-fl, image
    ...kunlunxin-xre5.37.1:**2.1.2** (not CI's 2.2.0):

    torch-fl-hf-test   : torch 2.10.0+cu129  version.cuda=12.9  libtorch_cuda.so PRESENT
                     -> ImportError: libbkcl.so: cannot open shared object file
torch-fl-xpu-py310 : -> ImportError: libbkcl.so: cannot open shared object file
torch-xpu-py310    : vendor binding load fails
torch-fl-xpu-py311 : torch 2.10.0+cu129  (triton has no __version__)
/tmp/xpu210v       : torch 2.10.0+cu129  -> torch_xmlir undefined symbol
                     _ZN2at4_ops5triu_10redispatchEN3c1014DispatchKeySetERNS_6TensorEl

    Every candidate pairs a +cu129 vendor torch with a torch_xmlir plugin built against
    a different torch ABI. No env satisfies cuda_version: null. The …ib vs …jb symbol in
    the original report is one instance of that same mismatch; the /tmp/xpu210v venv shows a
    second, different one (triu_).

    What the symbol mismatch actually is

    _ZN3c104cuda29c10_cuda_check_implementationEiPKcS2_ib (referenced) vs …jb (defined) is a
    torch 2.9 → 2.10 signature widening — between two vendor components (a CUDA torch and a
    torch_xmlir plugin), inside a stack that kunlunxin's CI configuration says must not contain
    a CUDA torch at all.

    Corrected statement

    • Not supported by the evidence: "kunlunxin cannot run; the first flagos op hangs forever;
      no code change can help."
    • Supported: kunlunxin passes its CI integration suite on the native PrivateUse1 +
      CPU-torch stack. The fleet host is provisioned with a vendor-CUDA stack whose torch_xmlir
      plugin has an ABI break against its torch. That is an environment/provisioning problem.

    What remains genuinely unknown

    CI runs tests/integration/ops/, not the HF bert suite. Whether kunlunxin's bert suite
    passes on a correctly-provisioned fleet env is not yet measured — that gap is real.

    Suggested next step

    Provision the fleet host to the CI contract (the :2.2.0 image, or a stock CPU-torch venv
    with flagtree-xpu and no torch_xmlir), then run the bert suite. If it passes, this issue
    should be reclassified as a provisioning bug in the fleet host, not a kunlunxin port defect.

    I am not changing any code for this; the port itself is not implicated by any of the above.

  3. lvyufeng commented on Oct 17, 2026

    @lvyufeng
    lvyufeng
    on Oct 17, 2026
    CollaboratorAuthor

    Closing the gap the correction left open: bert is still not measured, and the blocker is now located in the vendor runtime

    The correction above ends at the right question — "Whether kunlunxin's bert suite
    passes on a correctly-provisioned fleet env is not yet measured"     — and proposes
    provisioning to the CI contract, then running bert. That provisioning was done. This
    comment reports what it produced, and it refines the attribution rather than
    overturning the correction.

    The env gate was cleared, and the port does run

    Built exactly as the correction prescribes — a venv with torch==2.10.0+cpu
    (torch.version.cuda is None) and no torch_xmlir, then a wheel built with
    FLAGOS_ACCELERATOR=kunlunxin XDNN_ROOT=… FLAGOS_BUILD_FLAGGEMS_CPP=0 against that
    CPU torch. Linking against CPU torch is what keeps libtorch_cuda.so out of the
    wheel's NEEDED list; the oracle agrees:

    readelf -d …/torch_fl/lib/libtorch_fl.so | grep -c libtorch_cuda   ->  0
wheel md5 ddfd7b58d679e5138ff1924a6a029842

    So the ABI mismatch described in the original report is gone on this stack, as the
    correction predicted. And the port works: the vendor_ops group from
    .github/integration-manifest.yml passes —

    668 passed, 255 skipped, 743 deselected, 3 xpassed in 29.12s

    But bert still cannot be measured, for a second and different reason

    The ops group is narrow. It exercises mm / bmm / randn, which the conf routes
    to kunlunxin and which survive. The bert forward opens with nn.Embedding, and that
    hangs. Reproduced on the same wheel, same md5, same box:

    ARM A (as-shipped, 57 routes intact): HANG
  Timeout (0:01:00)!
    File …/torch/nn/functional.py, line 2567 in embedding
    File …/torch_fl/__init__.py, line 1501 in __torch_function__
    File …/transformers/models/bert/modeling_bert.py, line 99 in forward
    File /tmp/klx_min.py, line 14 in <module>

    Never reaches FORWARD_OK. No pytest summary line was ever produced by any run — every
    attempt was terminated by the hang, never by pytest. bert is NOT MEASURED on
    kunlunxin    , and that is the honest ceiling here.

    There is no de-routed control arm — the conf and the registration are in lockstep

    The obvious discriminator ("does bert finish if the vendor route is bypassed?") cannot
    be constructed on this wheel    , and that is itself a finding:

    • none in the conf raises, rather than falling back:
      embedding: routed to 'none' … but the op is registered on PrivateUse1 -- regenerate the vendor conf so registration and routing agree (dispatcher.h:306-312; the kNone path
      is only reachable when codegen skipped m.impl()).
    • The degrade arms (dispatcher.h:281-304) all bottom out at cuda_fn_ / flaggems_fn_ /
      flaggems_cpp_fn_; on this wheel cuda_fn_ is empty (no boxing kernel) and flag_gems
      is not importable at all, so they cannot yield cpu_fallback either.
    • Root cause, in the tree: csrc/aten/register.cc:556-566 (the USE_KUNLUNXIN arm) —
      kunlunxin_register.inc claims exactly the ops the conf routes to kunlunxin. Counted on
      the wheel-owning tree: 58 m.impl claims vs 57 = kunlunxin routes, with embedding
      and sum.dim_IntList among the claims. Conf and registration are deliberately in lockstep,
      so every value that de-routes a claimed op is a disagreement → raise. Reaching
      cpu_fallback would require rebuilding with a reduced .inc — a different md5,
      outside this measurement.
    • FLAGOS_FORCE_BACKEND offers no lever: _env.py:413-420 documents only
      flaggems / vendor / tileops. (An earlier probe of mine claimed =flaggems reproduced
      the spin; that arm was invalid — the dispatch log shows
      FLAGOS_FORCE_BACKEND=flaggems: 0 ops repinned, 2037 left as configured and
      embedding -> kunlunxin. FlagGems implements none of these ops here, so nothing moved.
      Retracted.)

    Therefore the A/B separation is NOT established, and this comment does not claim it.

    Where the hang actually is

    Per-thread /proc/<pid>/task sampling of a live spinner, twice, 3s apart:

    R-threads: exactly 1 (of 66)
  utime 1325 -> 1645 = 320 ticks / 300 ticks-per-3s = ~107% of one core, wchan=0
the other 65: state=S wchan=futex_wait_queue_me utime=10   (parked, no work)

    A thread blocked on a mutex someone else holds looks like those 65 — futex wait, ~0% CPU.
    The spinner does the opposite: a full core in the low-level lock fast path
    (lll_mutex_unlock_optimized, frame #0) with no other thread holding or progressing that
    lock    . So this is a single-thread userspace self-loop in the vendor stack, not a futex
    deadlock and not thread starvation — no device or neighbour action can clear it.

    Stack, verified against the shipped binaries:

    #12 xpu_wait () from /usr/local/xpu/so/libxpurt.so.2          <- nm -D: defined=1
#3-#11 ?? () from /usr/local/xcudart/lib/libxpucuda.so.1

    Two corrections to the shape of that stack, both material:

    • libxpurt does NOT link libxpucuda. Its DT_NEEDED is only
      librt/libpthread/libdl/libstdc++/libm/libgcc/libc; libxpucuda arrives via a dlopen
      string (?libxpucuda.so.1). So the edge is a runtime-dlopened call, not a static one.
    • libxpucuda.so.1 is STRIPPED (zero .symtab) — which is exactly why Uni wrapper #3-[Bug] torch_fl and torch_npu conflict: duplicate PrivateUse1 backend fallback registration on Ascend platform #11 render as
      ??. They are resolved-but-stripped, not unknown code. It is a CUDA-compat shim, exporting
      cuCtxSynchronize, cuStreamSynchronize, cuStreamWaitEvent, cuStreamWaitValue32/64;
      xpu_wait forwards into a cuStreamSynchronize reimplementation and the lock lives there.
      kunlunxin_common.h:117-118 confirms the sync is xpu_wait(nullptr) draining the stream.

    Device health at the same moment: no hw exception, ECC 0, temps 36-44C. (An earlier
    "XPU 0 at 100% util" reading was a live sibling spinner's stream, not a hardware fault.)

    Corrected statement

    • Not supported: "kunlunxin's port is broken" and "routing can be fixed to make bert
      pass". Neither is what the evidence shows.
    • Supported: the port is sound and its ops group passes; the env gate cleared with a
      CPU-torch build; and bert is blocked by a vendor-runtime defect — a self-loop inside
      libxpucuda.so.1's stream-synchronisation lock, reached from libxpurt's xpu_wait, on an
      op that is properly registered and routed. Resolving it needs a vendor SDK fix, not a
      torch_fl change.
    • Still unknown: whether bert passes once that vendor lock is fixed. That is the
      remaining measurement, and it is gated on the vendor, not on this repo.

    Closing this comment's scope as a vendor-runtime blocker. If the vendor xpucuda
    xpu_wait sync is fixed, rerun tests/models/bert/test_modeling_bert.py on the same
    CPU-torch stack above and this issue can be revisited with a real tally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1Priority 1 - highai-generatedbugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions