Skip to content

[Core][Distributed] improve p2p access check - #4992

Merged
youkaichao merged 17 commits into
vllm-project:mainfrom
youkaichao:p2p_check
May 29, 2024
Merged

youkaichao merged 17 commits into
vllm-project:mainfrom
youkaichao:p2p_check

Conversation

@youkaichao

@youkaichao youkaichao commented May 22, 2024 •

Copy link
Copy Markdown
Member

Done

@youkaichao

Copy link
Copy Markdown
Member Author

Previously, we use the following check for actual p2p access in case cuda driver is broken:

# code partly borrowed from
# https://github.com/turboderp/exllamav2/blob/1c67f97f3d2a968605a9c31ab791a05c85bb7879/exllamav2/compat.py#L10
# License: MIT
def _can_actually_p2p(idx_a, idx_b):
    dev_i = f"cuda:{idx_a}"
    dev_j = f"cuda:{idx_b}"
    a = torch.randn(5, device=dev_i) + 123.0
    b = a.to(dev_j)
    c = b.to(dev_i)
    return torch.all(a == c).cpu().item()

However, pytorch somehow fixes the bug, and it will always return True, no matter whether p2p is available:

import torch
torch.cuda.can_device_access_peer(0, 1) # False
_can_actually_p2p(0, 1) # True

This is reported in #4770 (comment) .

@youkaichao

Copy link
Copy Markdown
Member Author

cc @hanzhi713

@youkaichao youkaichao changed the title [WIP][Core][Distributed] improve p2p access check [Core][Distributed] improve p2p access check May 22, 2024
@youkaichao
youkaichao requested a review from WoosukKwon May 23, 2024 06:57
@youkaichao

Copy link
Copy Markdown
Member Author

@WoosukKwon ready for review

@WoosukKwon WoosukKwon self-assigned this May 28, 2024

@WoosukKwon WoosukKwon left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks for the PR! Left some minor comments.

@youkaichao

Copy link
Copy Markdown
Member Author

@WoosukKwon thanks for the very detailed review!

@youkaichao

Copy link
Copy Markdown
Member Author

Since we still don't have ci machines with p2p capability, I tested this PR locally.

cc @simon-mo for nvlink machines.

@youkaichao
youkaichao enabled auto-merge (squash) May 29, 2024 07:21
@youkaichao
youkaichao merged commit 594392d into vllm-project:main May 29, 2024
chengzhi-lu pushed a commit to chengzhi-lu/vllm-infersche that referenced this pull request May 29, 2024
@youkaichao
youkaichao deleted the p2p_check branch May 29, 2024 15:16
dtrifiro pushed a commit to opendatahub-io/vllm that referenced this pull request May 31, 2024
robertgshaw2-redhat pushed a commit to neuralmagic/nm-vllm that referenced this pull request Jun 8, 2024
joerunde pushed a commit to joerunde/vllm that referenced this pull request Jun 17, 2024
robertgshaw2-redhat pushed a commit to neuralmagic/nm-vllm that referenced this pull request Jul 14, 2024
michael-gpu pushed a commit to practical-inference/vllm-ampere-extended that referenced this pull request Oct 20, 2026
shen-shanshan added a commit to shen-shanshan/vllm that referenced this pull request Nov 13, 2026
…28 without offloading

InferenceX vllm-project#5036 now also runs DP2 at concurrency 128, carried over from the
closed vllm-project#4992, with no KV offloading: UMBP did not help there in local runs,
since the GPU KV cache holds the working set. Add that point to the
mi355x_agentx_dp2_umbp preset with `kv_offload: off`; 256 and 384 keep UMBP.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: shen-shanshan <467638484@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants