Skip to content

[TransferEngine] Fix RDMA GID auto-discovery for IPv6 and reduce spur… #1597

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
alogfans merged 3 commits into from
Mar 10, 2026
Merged

[TransferEngine] Fix RDMA GID auto-discovery for IPv6 and reduce spur… #1597

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
bf17c03
[TransferEngine] Fix RDMA GID auto-discovery for IPv6 and reduce spur…
stmatengss Mar 2, 2026
de5404c
update
stmatengss Mar 9, 2026
1be319d
update
stmatengss Mar 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 8 additions & 9 deletions mooncake-transfer-engine/src/transport/rdma_transport/rdma_context.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -447,13 +447,11 @@ GidNetworkState RdmaContext::findBestGidIndex(const std::string &device_name,

for (i = 0; i < port_attr.gid_tbl_len; i++) {
if (ibv_query_gid_ex(context, port, i, &gid_entry, 0)) {
PLOG(ERROR) << "Failed to query GID " << i << " on " << device_name
<< "/" << port;
continue; // if gid is invalid ibv_query_gid_ex() will return !0
// Reached end of valid GID indices
break;
}

if ((ipv6_addr_v4mapped((struct in6_addr *)gid_entry.gid.raw) &&
gid_entry.gid_type == IBV_GID_TYPE_ROCE_V2) ||
if (gid_entry.gid_type == IBV_GID_TYPE_ROCE_V2 ||
Copy link
Copy Markdown
Collaborator

@alogfans alogfans Mar 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We originally add this restriction to avoid finding loopback GID if the network only supports IPv4. So it's better to find V4 GID first, the fallback to V6.

Copy link
Copy Markdown
Collaborator Author

@stmatengss stmatengss Mar 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK

gid_entry.gid_type == IBV_GID_TYPE_IB) {
// Check if this GID has an associated network device
if (hasNetworkDevice(device_name, port, i)) {
Expand Down Expand Up @@ -584,16 +582,17 @@ int RdmaContext::openRdmaDevice(const std::string &device_name, uint8_t port,
}
} else {
// Also check network state for user-specified GID
if (!hasNetworkDevice(device_name, port, gid_index)) {
bool has_ndev = hasNetworkDevice(device_name, port, gid_index);
if (!has_ndev) {
LOG(WARNING) << "User-specified GID index " << gid_index
<< " on " << device_name << "/" << port
<< " has no associated network device, "
<< "may not be optimal for RDMA operations";
goto cleanup_context_and_devices;
}
LOG(INFO) << "Using user-specified GID index: " << gid_index
<< " on " << device_name << "/" << port
<< " (with network device)";
<< " on " << device_name << "/" << port << " ("
<< (has_ndev ? "with" : "without")
<< " network device)";
}

// Continue with GID validation
Expand Down
Loading