[BUG][DCU][Distributed] Build and package _flagos_nccl against libtorch_hip and RCCL #366

Issue Type

• Bug Report
• Feature Request
• Operator Implementation
• Platform Support
• Performance Issue
• Documentation

AI Agent Information

• Agent: OpenAI Codex
• Model: GPT-5.6
• Session Context: Investigation of the missing native RCCL communication bridge in a released Torch-FL DCU environment, followed by an isolated source build and two-rank hardware validation.

Summary

Torch-FL contains _nccl_ext/nccl_backend.cpp, a thin pybind factory that constructs c10d::ProcessGroupNCCL from an externally supplied vendor libtorch, but the extension is not part of the main build and the standalone build.py is NVIDIA-specific. In the tested DCU release, _flagos_nccl*.so and nccl_backend.cpp were absent from the installed package, so the native Hygon fallback could not be used. An isolated diagnostic build using GCC 11.4, DTK headers, libc10_hip.so, libtorch_hip.so, and RCCL succeeded and completed a two-rank DCU AllReduce, demonstrating that the missing piece is Torch-FL build/package integration rather than a missing DAS/RCCL implementation.

Environment

python torch_fl/comm/_nccl_ext/build.py

Compile definitions:
  USE_C10D_NCCL
  C10_CUDA_NO_CMAKE_CONFIGURE_FILE

Include roots:
  <DTK>/cuda/cuda-12/include
  <DTK>/include
  <DTK>/include/rccl

Libraries:
  c10_hip
  torch_hip
  rccl

export GEMS_VENDOR=hygon
export FLAGOS_USE_FLAGGEMS=0

Reproduction

1. Inspect the installed release

python - <<'PY'
from pathlib import Path
import torch_fl

root = Path(torch_fl.__file__).resolve().parent
ext = root / "comm" / "_nccl_ext"
print("extension_dir:", ext)
print("shared_objects:", list(ext.glob("_flagos_nccl*.so")))
print("source_present:", (ext / "nccl_backend.cpp").exists())
PY

Observed in the tested r3 release:

shared_objects: []
source_present: False

2. Inspect or run the current standalone builder

The current builder selects only the NVIDIA assets and libraries:

libraries=["c10_cuda", "torch_cuda", "nccl"]

On the tested DCU environment, the unmodified build first failed with:

PyTorch requires GCC 9 or later
fatal error: cuda.h: No such file or directory

Even after selecting a supported compiler, the builder has no DCU branch for c10_hip, torch_hip, or rccl.

Expected vs Actual Behavior

Expected:

A supported DCU Torch-FL build/release should provide a usable native communication fallback:

ProcessGroupFlagOS
  -> _flagos_nccl
  -> DAS libtorch_hip ProcessGroupNCCL
  -> RCCL

The installed wheel should either contain the compiled extension or have a documented, release-tested build step that produces and packages it.

Actual:

• The standalone builder is CUDA/NVIDIA-specific.
• The main setup.py builds only torch_fl._C; _flagos_nccl is not an extension target.
• The tested DCU release did not contain _flagos_nccl*.so.
• The release also omitted nccl_backend.cpp, so its installed build.py could not rebuild the extension in place.
• ProcessGroupFlagOS therefore cannot use the native RCCL fallback when FlagCX is unavailable.

Root Cause Analysis

There are three build/package gaps:

1. Vendor selection gap: _nccl_ext/build.py hard-codes NVIDIA discovery and links c10_cuda, torch_cuda, and nccl.
2. Main build gap: the primary setup.py extension list contains torch_fl._C but no _flagos_nccl target.
3. Release artifact gap: the DCU package bundles lib_dcu/*.so*, including DAS libtorch_hip.so, but does not build/install the thin _flagos_nccl factory that exposes ProcessGroupNCCL to the CPU-only Python torch bindings.

The C++ factory itself did not require a functional rewrite for the pinned environment. With only build/link adaptation, it produced a 208,720-byte extension that imported successfully and drove the existing DAS ProcessGroupNCCL/RCCL implementation.

Verified execution path

flagos tensor
  -> ProcessGroupFlagOS
  -> torch_fl._C._flagos_to_cuda_view (zero-copy alias)
  -> _flagos_nccl.make_nccl_backend
  -> libtorch_hip.so::c10d::ProcessGroupNCCL
  -> librccl.so.1
  -> 2-DCU collective

Verified result:

RANK_PASS rank=1 local_rank=1 value=3.0
RANK_PASS rank=0 local_rank=0 value=3.0
FLAGOS_NCCL_EXT_ALLREDUCE_PASS

Proposed Solution

Implement this in two reviewable phases.

Phase 1: standalone DCU build support

• Add a Hygon/DCU branch to _nccl_ext/build.py.
• Detect DTK/HIP/RCCL include and library paths from explicit environment/configuration rather than private absolute paths.
• Link c10_hip, torch_hip, and rccl while preserving the existing NVIDIA path.
• Require or clearly diagnose a compiler compatible with the installed PyTorch headers (GCC >= 9 for the tested PyTorch 2.10 environment).
• Use relocatable RPATH entries where possible, including the installed torch_fl/lib_dcu location.

Phase 2: main build and wheel integration

• Build _flagos_nccl as part of the supported Torch-FL package build.
• Install it under torch_fl/comm/_nccl_ext/.
• Ensure wheel staging includes the extension and required metadata/source according to maintainer policy.
• Add wheel-install validation rather than validating only from a source checkout.

Maintainer decision requested

Which integration should be used for Phase 2?

1. Invoke/extend the standalone setuptools builder from packaging;
2. Add another Python extension target in setup.py;
3. Add a CMake target and install it into torch_fl/comm/_nccl_ext.

Also, should every DCU wheel package this native RCCL bridge, or should it remain optional when a separately validated FlagCX backend is guaranteed?

This issue should remain open after Phase 1. It is complete only when an installed Torch-FL wheel can load the extension without a source checkout and pass a two-rank DCU collective smoke test.

Verification Plan

• Build test: DCU configuration compiles _flagos_nccl*.so with GCC 11.4.
• Loader test: import succeeds after torch_fl/torch load the matching runtime libraries.
• Factory test: make_nccl_backend is exported and callable.
• Hardware prototype: two-rank, two-DCU AllReduce returns 3.0 on both ranks.
• Unit test: vendor-specific build configuration selects the expected libraries without requiring hardware.
• CUDA regression: existing NVIDIA build configuration remains unchanged and passes its build/import tests.
• Wheel test: install the built wheel into a clean environment and confirm _flagos_nccl*.so is present.
• Linkage test: validate readelf/ldd -r with the supported runtime activation.
• Release smoke: installed-wheel two-rank DCU AllReduce.
• Follow-up coverage: broadcast, all_gather, reduce_scatter, async Work, DDP, HSDP, subgroup/DeviceMesh.

Context & Investigation

• Searched open and closed Torch-FL Issues and PRs for _flagos_nccl, make_nccl_backend, _nccl_ext DCU, libtorch_hip RCCL, and ProcessGroupFlagOS DCU; no same-root Issue or active PR was found. PR Migrate CUDA backend to schema codegen + native distributed (flagos) backend #19 introduced the extension/native backend design; PRs fix: FlagGems factory device index (all backends) + DCU comm vendor routing #36 and fix(comm): run collectives on their operand's device #57 are related DCU routing/device fixes but do not build or package this extension.
• Inspected current main at 7298bfbd14c604f260bd84052dd1d8b2dd7ddc82 on 2026-09-21.
• Inspected the current standalone builder, main setup.py, package data, and the r3 installed artifact inventory.
• Compared the unmodified build with an isolated DCU-specific diagnostic build.
• Verified the built extension import and two-rank hardware execution without modifying the shared r3 release.

Related Code Locations

• torch_fl/comm/_nccl_ext/build.py:42-90 — NVIDIA-specific discovery and link libraries.
• torch_fl/comm/_nccl_ext/nccl_backend.cpp — thin factory; validated unchanged against the pinned DCU runtime.
• setup.py:764-783 — wheel staging for vendor shared libraries.
• setup.py:852-880 — main extension list and package data.
• torch_fl/comm/process_group.py:416-453 — Hygon native fallback consumer.
• Related history: Migrate CUDA backend to schema codegen + native distributed (flagos) backend #19, fix: FlagGems factory device index (all backends) + DCU comm vendor routing #36, fix(comm): run collectives on their operand's device #57.

Explicitly Not Included

• init_process_group(device_id=torch.device("flagos:*")) eager-connect/HIPGuard behavior.
• FlagCX DU adaptor integration.
• Multi-node support or performance claims.
• Modifications to DAS libtorch_hip.so, RCCL, or PyTorch core.

Checklist - AI Agents MUST Complete All

• I have provided complete environment information
• I have included a complete reproduction/inspection procedure
• I have included the observed build errors and hardware result
• I have analyzed the root cause (not just symptoms)
• I have proposed a specific phased solution
• I have identified affected code locations with line numbers
• I have described how to verify the fix
• I have checked for duplicate issues and PRs
• All text is in English
• Code follows project conventions checked against current source