Skip to content

[Bug] custom_all_reduce.cu:397 hipIpcGetMemHandle fails when caller uses PYTORCH_HIP_ALLOC_CONF=expandable_segments:True #4174

Closed

Description

@Rohan138

[Bug] custom_all_reduce.cu:397 hipIpcGetMemHandle [HIP error](invalid argument) when caller uses PYTORCH_HIP_ALLOC_CONF=expandable_segments:True

Summary

aiter.get_meta_buffer_ipc_handle (backed by hipIpcGetMemHandle at
aiter_meta/csrc/kernels/custom_all_reduce.cu:397)
fails with [HIP error](invalid argument) when the input buffer was
allocated by PyTorch's caching allocator with
PYTORCH_HIP_ALLOC_CONF=expandable_segments:True. This env var switches
PyTorch to hipMallocAsync / stream-ordered memory pool allocations, and
pointers into those pools are not compatible with the classic
hipIpcGetMemHandle API on ROCm 7.2.x.

Downstream: any vLLM TP≥2 configuration that opts into
expandable_segments:True (which vLLM does in several code paths to
mitigate HBM fragmentation on big-model tensor-parallel serving) fails
at CustomAllreduce init.

Environment

  • GPU: AMD Instinct MI300X (gfx942)
  • ROCm 7.2.53211
  • PyTorch 2.11.0+gitd0c8b1f
  • amd-aiter 0.1.16.post3 (also reproduced on 0.1.13.post1 and
    0.1.16.post2 — the code path is unchanged across those versions)
  • Container: vllm/vllm-openai-rocm:nightly (image digest
    sha256:b678b9cd2cc402…)

Minimal repro (single process, no distributed)

# aiter_ipc_repro.py
import os, torch, aiter
torch.cuda.set_device(0)
buf = torch.empty(1 << 30, dtype=torch.uint8, device="cuda")
handle = torch.empty(64, dtype=torch.uint8)
aiter.get_meta_buffer_ipc_handle(buf.data_ptr(), handle.data_ptr())
print("OK")
$ python aiter_ipc_repro.py
OK

$ PYTORCH_HIP_ALLOC_CONF=expandable_segments:True python aiter_ipc_repro.py
[AITER] /app/aiter/aiter_meta/csrc/kernels/custom_all_reduce.cu:397 fail to
call hipIpcGetMemHandle((hipIpcMemHandle_t*)out_handle_ptr, (void*)inp_ptr)
---> [HIP error](invalid argument)

Where it manifests in vLLM

At CustomAllreduce.__init__ in
aiter/dist/device_communicators/custom_all_reduce.py, the "input" meta
buffer is allocated with torch.empty(...) on the current CUDA stream.
Under expandable_segments:True that pointer lives in an hipMallocAsync
pool. The subsequent get_meta_buffer_ipc_handle call fails and takes
down all TP workers before model load completes.

We first hit this on GLM-5.1-FP8 / DeepSeek-V3.2 TP=8 startup, where
vLLM's OOM-hardening path sets expandable_segments:True and then
initializes AITER CustomAllreduce.

What we ruled out

  • Not a per-buffer alignment issue: buffer size / dtype don't matter (1 MiB,
    1 GiB, uint8, int32 — all fail with expandable_segments:True).
  • Not fixed by switching only the "input" buffer to uncached=True in
    aiter's Python wrapper. That eliminates the first call site, but a
    second IPC call in graph-buffer registration
    (custom_all_reduce.cuh::register_graph_buffers) then fails identically
    on user-supplied tensors that also came from the pool.
  • Not resolved by the recent consolidation of vLLM's Custom Allreduce onto
    AITER's path ([ROCm][AITER] Directly Implement AITER Custom All-reduce in CudaCommunicator vllm-project/vllm#46065, merged). That PR moved callers
    onto AITER without touching the pool-pointer / IPC compatibility.

Suggested fix directions

The classic hipIpcGetMemHandle API is documented to work on pointers
returned by hipMalloc. Pointers from hipMallocAsync / stream-ordered
pools need hipMemPoolExportToShareableHandle +
hipMemPoolImportFromShareableHandle on the receiver, with the
pool-native IPC flow.

Two plausible fixes on the AITER side:

  1. Detect and fall back. In CustomAllreduce.__init__, check the
    allocation origin of the input tensor (e.g. via
    hipPointerGetAttributes or HIP_POINTER_ATTRIBUTE_MEMPOOL_HANDLE).
    If it comes from a pool, hipMalloc a fresh IPC-compatible buffer
    and copy into it — or refuse-to-attach with a clear diagnostic so
    callers can --disable-custom-all-reduce explicitly.
  2. Support pool-based IPC. Use hipMemPoolExportToShareableHandle /
    hipMemPoolImportFromShareableHandle when the pointer originates
    from a pool. Cleaner but requires reworking the receiver side too.

Direction 1 is the smaller change and unblocks vLLM immediately. It also
lets expandable_segments:True remain enabled at the caller (which
helps memory-tight big-model TP configs stay within budget).

Workarounds today

  • --disable-custom-all-reduce at vLLM serve time.
  • Unset PYTORCH_HIP_ALLOC_CONF=expandable_segments:True — but this
    re-introduces fragmentation-induced OOMs at big-model TP=8 startup.

Repro artifact

Self-contained script attached: aiter_ipc_issue_repro.py (runs both
pass and fail cases, no distributed init required).

Activity

  1. tantara commented on Jul 30, 2026

    @tantara

    Hi @Rohan138 , can you please share how you fixed it?

  2. jamesETsmith commented on Aug 4, 2026

    @jamesETsmith
    Contributor

    Reproduced on current AITER

    I reproduced this on a newer stack, so the issue is still present:

    • AITER: c76aca0af15015572849f8938002780e66fe317d
    • PyTorch: 2.12.0+rocm7.14.0
    • HIP runtime: 7.14.60850
    • GPU: AMD Instinct MI355X (gfx950)

    Results

    Allocation/export path Result
    Default PyTorch allocation + AITER hipIpcGetMemHandle Pass
    expandable_segments:True allocation + AITER hipIpcGetMemHandle Fails with invalid argument
    expandable_segments:True allocation + PyTorch _share_cuda_() Pass
    AITER uncached allocation + hipIpcGetMemHandle Pass in both allocator modes

    The failing case on current AITER is:

    [AITER] /aiter/csrc/kernels/custom_all_reduce.cu:402 fail to call hipIpcGetMemHandle((hipIpcMemHandle_t*)out_handle_ptr, (void*)inp_ptr) ---> [HIP error](invalid argument)
    

    Next steps

    Can we reopen this issue?

  3. added a commit that references this issue on Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions