[xccl] WA: preload libccl.so.1 so the oneCCL legacy plugin can load by Chao1Han · Pull Request #5648 · intel/torch-xpu-ops

Chao1Han · 2026-10-09T07:16:14Z

libccl.so.2 dlopen()s its legacy plugin libccl_legacy.so, which lists libccl.so.1 in DT_NEEDED but ships without RPATH/RUNPATH. A dlopen'd object does not inherit the search path of whoever pulled it in, so in a flat PyPI/venv layout libccl.so.1 is looked up only in /etc/ld.so.cache, /lib and /usr/lib. It is not found there and oneCCL fails with "oneCCL could not be initialized! Could not load any plugin.", taking down every XCCL process group creation unless the user sets LD_LIBRARY_PATH by hand.

dlopen("libccl.so.1", RTLD_NOW | RTLD_GLOBAL) from this library resolves it through our own RPATH -- the same one that already locates libccl.so.2 -- and leaves it in the global scope, so the plugin binds against the already-loaded object instead of searching the filesystem.

Reproduced outside torch with a stub mirroring the real layout (NEEDED libccl.so.2 + RUNPATH to the oneCCL dir, oneCCL not on LD_LIBRARY_PATH):

without preload: dlopen libccl_legacy.so -> libccl.so.1: cannot open
shared object file
with preload : dlopen libccl.so.1 -> OK
dlopen libccl_legacy.so -> OK

libccl.so.2 dlopen()s its legacy plugin libccl_legacy.so, which lists libccl.so.1 in DT_NEEDED but ships without RPATH/RUNPATH. A dlopen'd object does not inherit the search path of whoever pulled it in, so in a flat PyPI/venv layout libccl.so.1 is looked up only in /etc/ld.so.cache, /lib and /usr/lib. It is not found there and oneCCL fails with "oneCCL could not be initialized! Could not load any plugin.", taking down every XCCL process group creation unless the user sets LD_LIBRARY_PATH by hand. dlopen("libccl.so.1", RTLD_NOW | RTLD_GLOBAL) from this library resolves it through our own RPATH -- the same one that already locates libccl.so.2 -- and leaves it in the global scope, so the plugin binds against the already-loaded object instead of searching the filesystem. Reproduced outside torch with a stub mirroring the real layout (NEEDED libccl.so.2 + RUNPATH to the oneCCL dir, oneCCL not on LD_LIBRARY_PATH): without preload: dlopen libccl_legacy.so -> libccl.so.1: cannot open shared object file with preload : dlopen libccl.so.1 -> OK dlopen libccl_legacy.so -> OK

chuanqi129 · 2026-10-09T16:46:41Z

Hi @Chao1Han could yo please help to remove the WA in distributed test workflow also? for example here https://github.com/intel/torch-xpu-ops/blob/main/.github/actions/linux-uttest/action.yml#L174

Chao1Han · 2026-10-10T02:42:47Z

distributed tests passed, seems add libccl.so.1 safely.

chuanqi129 · 2026-10-10T12:45:46Z

@torchxpubot merge -f

torchxpubot · 2026-10-10T12:46:06Z

PR has been successfully merged by @chuanqi129 (force merge).

chuanqi129 · 2026-10-10T12:47:12Z

@torchxpubot cherry-pick --onto release/2.15 -c critical

torchxpubot · 2026-10-10T12:47:41Z

Cherry picking #5648

The cherry pick PR is at #5666

The following tracker issue is updated:

[v.2.15.0] Release Tracker #5645 (comment)

…5648) libccl.so.2 dlopen()s its legacy plugin libccl_legacy.so, which lists libccl.so.1 in DT_NEEDED but ships without RPATH/RUNPATH. A dlopen'd object does not inherit the search path of whoever pulled it in, so in a flat PyPI/venv layout libccl.so.1 is looked up only in /etc/ld.so.cache, /lib and /usr/lib. It is not found there and oneCCL fails with "oneCCL could not be initialized! Could not load any plugin.", taking down every XCCL process group creation unless the user sets LD_LIBRARY_PATH by hand. dlopen("libccl.so.1", RTLD_NOW | RTLD_GLOBAL) from this library resolves it through our own RPATH -- the same one that already locates libccl.so.2 -- and leaves it in the global scope, so the plugin binds against the already-loaded object instead of searching the filesystem. Reproduced outside torch with a stub mirroring the real layout (NEEDED libccl.so.2 + RUNPATH to the oneCCL dir, oneCCL not on LD_LIBRARY_PATH): without preload: dlopen libccl_legacy.so -> libccl.so.1: cannot open shared object file with preload : dlopen libccl.so.1 -> OK dlopen libccl_legacy.so -> OK (cherry picked from commit d8bc81a)

…CL legacy plugin can load (#5666) libccl.so.2 dlopen()s its legacy plugin libccl_legacy.so, which lists libccl.so.1 in DT_NEEDED but ships without RPATH/RUNPATH. A dlopen'd object does not inherit the search path of whoever pulled it in, so in a flat PyPI/venv layout libccl.so.1 is looked up only in /etc/ld.so.cache, /lib and /usr/lib. It is not found there and oneCCL fails with "oneCCL could not be initialized! Could not load any plugin.", taking down every XCCL process group creation unless the user sets LD_LIBRARY_PATH by hand. dlopen("libccl.so.1", RTLD_NOW | RTLD_GLOBAL) from this library resolves it through our own RPATH -- the same one that already locates libccl.so.2 -- and leaves it in the global scope, so the plugin binds against the already-loaded object instead of searching the filesystem. Reproduced outside torch with a stub mirroring the real layout (NEEDED libccl.so.2 + RUNPATH to the oneCCL dir, oneCCL not on LD_LIBRARY_PATH): without preload: dlopen libccl_legacy.so -> libccl.so.1: cannot open shared object file with preload : dlopen libccl.so.1 -> OK dlopen libccl_legacy.so -> OK Cherry-pick of #5648 to `release/2.15`. Cherry-picked from commit d8bc81a. Co-authored-by: Chao Han <chao1.han@intel.com>

…#5670) #5648 preloaded libccl.so.1 by soname, assuming libtorch_xpu.so's RPATH would locate it the same way it locates libccl.so.2. It does not, so the workaround never took effect and PYTORCHDGQ-10126 still reproduces. LD_DEBUG on a PyPI/venv install shows only three RPATHs ever being used -- numpy's extension module, torch/_C.cpython-*.so and libtorch_global_deps.so -- all objects that were dlopen'd directly. libtorch_xpu.so is pulled in as an indirect dependency, so glibc resolves our dlopen() against those, never against ours. libccl.so.2 is found only because it is a DT_NEEDED of libtorch_global_deps.so: find library=libccl.so.2 ... (RPATH from file .../libtorch_global_deps.so) find library=libccl.so.1 ... search cache=/etc/ld.so.cache ... (system search path) Ask the loader for the resolved path of libccl.so.2 and open libccl.so.1 from the same directory instead, which does not depend on any search path.

… already loaded libccl.so.2 (#5676) #5648 preloaded libccl.so.1 by soname, assuming libtorch_xpu.so's RPATH would locate it the same way it locates libccl.so.2. It does not, so the workaround never took effect and PYTORCHDGQ-10126 still reproduces. LD_DEBUG on a PyPI/venv install shows only three RPATHs ever being used -- numpy's extension module, torch/_C.cpython-*.so and libtorch_global_deps.so -- all objects that were dlopen'd directly. libtorch_xpu.so is pulled in as an indirect dependency, so glibc resolves our dlopen() against those, never against ours. libccl.so.2 is found only because it is a DT_NEEDED of libtorch_global_deps.so: find library=libccl.so.2 ... (RPATH from file .../libtorch_global_deps.so) find library=libccl.so.1 ... search cache=/etc/ld.so.cache ... (system search path) Ask the loader for the resolved path of libccl.so.2 and open libccl.so.1 from the same directory instead, which does not depend on any search path. Cherry-pick of #5670 to `release/2.15`. Cherry-picked from commit d44bbbc. Co-authored-by: Chao Han <chao1.han@intel.com>

…ntel#5648) libccl.so.2 dlopen()s its legacy plugin libccl_legacy.so, which lists libccl.so.1 in DT_NEEDED but ships without RPATH/RUNPATH. A dlopen'd object does not inherit the search path of whoever pulled it in, so in a flat PyPI/venv layout libccl.so.1 is looked up only in /etc/ld.so.cache, /lib and /usr/lib. It is not found there and oneCCL fails with "oneCCL could not be initialized! Could not load any plugin.", taking down every XCCL process group creation unless the user sets LD_LIBRARY_PATH by hand. dlopen("libccl.so.1", RTLD_NOW | RTLD_GLOBAL) from this library resolves it through our own RPATH -- the same one that already locates libccl.so.2 -- and leaves it in the global scope, so the plugin binds against the already-loaded object instead of searching the filesystem. Reproduced outside torch with a stub mirroring the real layout (NEEDED libccl.so.2 + RUNPATH to the oneCCL dir, oneCCL not on LD_LIBRARY_PATH): without preload: dlopen libccl_legacy.so -> libccl.so.1: cannot open shared object file with preload : dlopen libccl.so.1 -> OK dlopen libccl_legacy.so -> OK

…intel#5670) intel#5648 preloaded libccl.so.1 by soname, assuming libtorch_xpu.so's RPATH would locate it the same way it locates libccl.so.2. It does not, so the workaround never took effect and PYTORCHDGQ-10126 still reproduces. LD_DEBUG on a PyPI/venv install shows only three RPATHs ever being used -- numpy's extension module, torch/_C.cpython-*.so and libtorch_global_deps.so -- all objects that were dlopen'd directly. libtorch_xpu.so is pulled in as an indirect dependency, so glibc resolves our dlopen() against those, never against ours. libccl.so.2 is found only because it is a DT_NEEDED of libtorch_global_deps.so: find library=libccl.so.2 ... (RPATH from file .../libtorch_global_deps.so) find library=libccl.so.1 ... search cache=/etc/ld.so.cache ... (system search path) Ask the loader for the resolved path of libccl.so.2 and open libccl.so.1 from the same directory instead, which does not depend on any search path.

tsocha approved these changes Oct 9, 2026

View reviewed changes

Chao1Han added 2 commits October 10, 2026 00:53

also remove ci wa

7d3e518

Merge branch 'main' into xccl/preload-libccl-so-1

9e187f1

chuanqi129 approved these changes Oct 10, 2026

View reviewed changes

torchxpubot merged commit d8bc81a into main Oct 10, 2026
26 of 29 checks passed

torchxpubot deleted the xccl/preload-libccl-so-1 branch October 10, 2026 12:46

torchxpubot mentioned this pull request Oct 10, 2026

[Cherry-pick release/2.15] [xccl] WA: preload libccl.so.1 so the oneCCL legacy plugin can load #5666

Merged

torchxpubot mentioned this pull request Oct 10, 2026

[v.2.15.0] Release Tracker #5645

Open

Chao1Han mentioned this pull request Oct 12, 2026

[xccl] WA: resolve libccl.so.1 next to the already loaded libccl.so.2 #5670

Merged

This was referenced Oct 12, 2026

[Cherry-pick release/2.15] [xccl] WA: resolve libccl.so.1 next to the already loaded libccl.so.2 #5676

Merged

[CI] Add bmg distributed test in CI and nightly wheel weekly #5147

Open

Navigation Menu

Repository navigation

[xccl] WA: preload libccl.so.1 so the oneCCL legacy plugin can load - #5648

[xccl] WA: preload libccl.so.1 so the oneCCL legacy plugin can load#5648
torchxpubot merged 3 commits into
mainfrom
xccl/preload-libccl-so-1

Chao1Han commented Oct 9, 2026

Uh oh!

chuanqi129 commented Oct 9, 2026

Uh oh!

Chao1Han commented Oct 10, 2026

Uh oh!

chuanqi129 commented Oct 10, 2026

Uh oh!

Uh oh!

torchxpubot commented Oct 10, 2026

Uh oh!

chuanqi129 commented Oct 10, 2026

Uh oh!

torchxpubot commented Oct 10, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

Chao1Han commented Oct 9, 2026

Uh oh!

chuanqi129 commented Oct 9, 2026

Uh oh!

Chao1Han commented Oct 10, 2026

Uh oh!

chuanqi129 commented Oct 10, 2026

Uh oh!

Uh oh!

torchxpubot commented Oct 10, 2026

Uh oh!

chuanqi129 commented Oct 10, 2026

Uh oh!

torchxpubot commented Oct 10, 2026

Cherry picking #5648

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants