Skip to content

UCT/IB: disable indirect atomic registration for direct nic - #11111

Merged
gleon99 merged 1 commit into
openucx:masterfrom
roiedanino:cuda-mkey-fail
Jan 22, 2026
Merged

gleon99 merged 1 commit into
openucx:masterfrom
roiedanino:cuda-mkey-fail

Conversation

@roiedanino

Copy link
Copy Markdown
Contributor

What?

Added a flag to identify direct-NIC memh, and disabled indirect atomic mkey registration

Why?

Currently, there is no support for indirect atomics over direct-NIC; registration fails, causing major latency overhead.

Signed-off-by: Roie Danino <rdanino@nvidia.com>
@roiedanino roiedanino self-assigned this Jan 12, 2026
@yosefe

yosefe commented Jan 12, 2026

Copy link
Copy Markdown
Member

Why the registration fails? Can we identify it in a more direct way, instead of tying it to direct-NIC?

@roiedanino

Copy link
Copy Markdown
Contributor Author

Why the registration fails? Can we identify it in a more direct way, instead of tying it to direct-NIC?

When trying to create the indirect atomic mkeys the syndrome is BAD_PARAM | 0x45D3A4 | check indirect mkey, not atomic support, which, to my understanding, means indirect atomics are not supported in direct-nic mrs.

Disabling either direct-nic or indirect atomics solves the issue, and it only happens with cuda memory.

@yosefe

yosefe commented Jan 13, 2026

Copy link
Copy Markdown
Member

Why the registration fails? Can we identify it in a more direct way, instead of tying it to direct-NIC?

When trying to create the indirect atomic mkeys the syndrome is BAD_PARAM | 0x45D3A4 | check indirect mkey, not atomic support, which, to my understanding, means indirect atomics are not supported in direct-nic mrs.

Disabling either direct-nic or indirect atomics solves the issue, and it only happens with cuda memory.

Let's check if have better solution , such as using the "normal" PF for atomics.

@gleon99

gleon99 commented Jan 13, 2026

Copy link
Copy Markdown
Contributor

I'm checking. Hold this one for now.

@gleon99

gleon99 commented Jan 20, 2026

Copy link
Copy Markdown
Contributor

Pending check by @Artemy-Mellanox

@gleon99
gleon99 merged commit c92698e into openucx:master Jan 22, 2026
147 checks passed
roiedanino added a commit to roiedanino/ucx that referenced this pull request Jan 26, 2026
…11111)

Signed-off-by: Roie Danino <rdanino@nvidia.com>
(cherry picked from commit c92698e)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants