Repository navigation
UCP/RMA: Emulate SGL PUT with contiguous PUTs on Vera Rubin - #12283
Conversation
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 Starting review — findings will be posted here when done. |
|
pls add a test for the new protocol: |
|
🤖 Starting review — findings will be posted here when done. |
|
Coverage note: |
tvegas1
left a comment
There was a problem hiding this comment.
can we have some perf numbers vs the previous put/sgl/am/bcopy fallback?
- typical SGL element count and sizes in the target workload
- bw/latency of
am/bcopyvsput/sgl/contigfor a few counts and sizes, especially small elements UCX_PROTO_INFO=usedoutput
|
🤖 CI Triage Agent — TL;DR: The ASAN "roce on worker 2" job failed with a single test failure, Full analysisSummary: Root cause: For the IB component, Implicated commit: none for the PR under test; the test/feature area originates from #9308 "UCT/IB: Added support for device memory allocation (memic/rdma)". The failure is environmental, not caused by commit [REDACTED:Hex High Entropy String]. File: test/gtest/uct/test_md.cc:479-561 ( Suggested fix:
Related: PR #12283 (build under test), PR #9308 (introduced IB device-memory alloc + this test coverage)
|
|
🤖 Starting review — findings will be posted here when done. |
# Conflicts: # test/gtest/ucp/test_ucp_rma.cc
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: The aarch64 build failed to compile Full analysisSummary: Root cause: Compiler output: With Implicated commit: 2d46ac2 (merge of master into File: Suggested fix: Update the two stale calls to the renamed helper: UCS_TEST_P(test_ucp_rma_sgl_contig, put) {
test_sgl({64, 256, UCS_KBYTE, 4 * UCS_KBYTE, 512});
}
UCS_TEST_P(test_ucp_rma_sgl_contig, put_with_callback) {
test_sgl(10, UCS_KBYTE, true, true);
}Verify the argument lists still match the new Related: PR #12283 (this branch), PR #12277 (the rename that introduced the conflict).
|
|
🤖 Starting review — findings will be posted here when done. |
|
Residual coverage gap: |
|
🤖 CI Triage Agent — TL;DR: Full analysisSummary: Azure job "Tests BlueField on worker 0" failed in the new Root cause: A hang, not a slow test. Timeline from the log: last application output is This configuration is brand new: the gtest task with Implicated commit: 75371a0 — "UCT/IB/RC/VERBS: Support QPs without SRQ (#12066)", Guangguan Wang (2026-11-11). Not the PR under test (f8284bb). File: Suggested fix:
Related: PR #12283 (#12283) — job under test; PR #12066 "UCT/IB/RC/VERBS: Support QPs without SRQ" — suspected source of the hang. No existing issue found for this signature; worth opening one against #12066. |
|
Thanks @ovidiusm for catching 2 protocol selection issues in this PR. I've pushed the fixes. |
|
🤖 Starting review — findings will be posted here when done. |
|
can we drop
why is |
|
🤖 CI Triage Agent — TL;DR: The ASAN roce gtest shard did not fail an assertion — it hung: Full analysisSummary: Root cause: Timeline from the Azure log proves a hang, not slowness:
So both workers were being progressed, yet a single UCP request never completed. The test configuration is The PR under test ( Implicated commit: unknown — not the PR commit (6dcc03b); the hung path is pre-existing master code. Plausible bisect candidates touching this exact scenario: File: Suggested fix: Re-run the ASAN roce job to confirm it is not reproducible on this branch (the diff cannot reach the tag/ Related: PR #12283 ( |
Why
On Vera Rubin, localized CUDA memory SGL PUT falls back to
put/sgl/am/bcopy, which is slow.What
Add
put/sgl/contig. It splits an SGL PUT into one contiguous PUT per element.It is only picked when no native SGL protocol is available.
How
put/rndvor staging.put/sgl/offloadandput/sgl/offload_swsetfallback_class, so this protocol is not picked when they work.