Skip to content

TEST/UCP: Add operation variant to RMA SGL tests - #12277

Merged
michal-shalev merged 8 commits into
openucx:masterfrom
michal-shalev:gtest-ucp-rma-sgl-per-op
Nov 11, 2026
Merged

michal-shalev merged 8 commits into
openucx:masterfrom
michal-shalev:gtest-ucp-rma-sgl-per-op

Conversation

@michal-shalev

Copy link
Copy Markdown
Contributor

What?

Make the RMA SGL gtests take the operation from a test variant instead of a parameter.
Only PUT is enabled.

Why?

Prep for GET SGL support, which will add the GET variant.
No change in coverage.

@michal-shalev michal-shalev self-assigned this Nov 8, 2026
@michal-shalev
michal-shalev requested a review from a team as a code owner November 8, 2026 13:13
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread test/gtest/ucp/test_ucp_rma.cc Resolved
Comment thread test/gtest/ucp/test_ucp_rma.cc Resolved
Comment thread test/gtest/ucp/test_ucp_rma.cc Resolved
Comment thread test/gtest/ucp/test_ucp_rma.cc Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gap worth noting: with only the put variant instantiated, CI will not exercise the get branches at all, and the put_emulation* tests still force PROTOS=put/sgl/am/* while calling the now-generic test_sgl(), so they will need a get-specific protocol list when the get variant is turned on.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread test/gtest/ucp/test_ucp_rma.cc Resolved
Comment thread test/gtest/ucp/test_ucp_rma.cc Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

test/gtest/ucp/test_ucp_rma.cc

Residual gap (already raised previously, no new thread needed): the get variant is not instantiated (UCS_BIT(SGL_OP_PUT)), so all op() == SGL_OP_GET branches — including the "get/sgl/offload" name lookup and the UCT_IFACE_FLAG_GET_ZCOPY check — are untested, and UCP_SGL_EMULATION_PROTOS is still put-only.

test/gtest/ucp/test_ucp_rma.cc

Noted but minor: the /* Only the emulation protocol can complete the put in-place ... */ comment inside offload_proto_selected() is now slightly stale for the generic function, and init_sgl_ctx_mixed_mem_types() still memsets src/dst directly rather than via data_src()/data_dst() (harmless — only used by invalid-param tests that never verify data).

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-nvidia-pr-review

Copy link
Copy Markdown

test/gtest/ucp/test_ucp_rma.cc

minor: in this class src/dst means local/remote (ctx.src, src_mem_type()), while data_src/data_dst mean the data direction, so for get they return the opposite buffers. Can we add a one-line comment here to make the two meanings explicit?

test/gtest/ucp/test_ucp_rma.cc

Residual coverage gap (previously raised): get_test_variants() still instantiates only UCS_BIT(SGL_OP_PUT), so every op() == SGL_OP_GET path — the UCT_IFACE_FLAG_GET_ZCOPY check, the "get/sgl/offload" name lookup (that protocol does not exist yet; only put/sgl/offload and put/sgl/offload_sw are defined in src/ucp/rma/put_offload.c), and the swapped data_src/data_dst branches — is unexercised by CI, and UCP_SGL_EMULATION_PROTOS is still PROTOS=put/sgl/am/*.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread test/gtest/ucp/test_ucp_rma.cc Resolved
Comment thread test/gtest/ucp/test_ucp_rma.cc Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-nvidia-pr-review

Copy link
Copy Markdown

get_test_variants() still instantiates only UCS_BIT(SGL_OP_PUT), so all op() == SGL_OP_GET paths (the UCT_IFACE_FLAG_GET_ZCOPY skip check, the "get/sgl/offload" name lookup for a protocol that does not exist yet, and the swapped data_src/data_dst branches) stay unexercised by CI.

UCP_SGL_EMULATION_PROTOS is still PROTOS=put/sgl/am/* and the put_emulation* tests now call the generic test_sgl(), so they will need a get-specific protocol list when the get variant is enabled.

init_sgl_ctx_mixed_mem_types() still fills src/dst directly instead of via data_src()/data_dst() (harmless: only used by invalid-param tests that never verify data).

The src/dst (local/remote) vs data_src/data_dst (data direction) naming overlap is still undocumented.

@michal-shalev
michal-shalev requested a review from tvegas1 November 8, 2026 15:05
@svc-ucx

svc-ucx commented Nov 8, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (AddressSanitizer roce on worker 0) · commit 8ee49adf

TL;DR: The only failing test is dc_mlx5/uct_atomic_key_reg_rdma_mem_type.fadd64/0, a device-memory (MEMIC / UCS_MEMORY_TYPE_RDMA) atomic test that is unrelated to this gtest-only PR; it is a known flaky failure when MEMIC allocation keeps returning UCS_ERR_NO_MEMORY after the retry loop, so the fix is to skip (not fail) the test on persistent RDMA-memory allocation failure and re-run the job.

Full analysis

Summary: make test in the AddressSanitizer/roce gtest job exited with code 2 because 1 of 8960 tests failed: [ FAILED ] dc_mlx5/uct_atomic_key_reg_rdma_mem_type.fadd64/0, where GetParam() = dc_mlx5/mlx5_0:1 (build 143602, job "roce on worker 0").

Root cause: The failing test is the only one in the run that allocates a UCS_MEMORY_TYPE_RDMA buffer, i.e. on-device MEMIC memory:
mapped_buffer recvbuf(sizeof(uint64_t), receiver(), 0UL, UCS_MEMORY_TYPE_RDMA, UCT_MD_MEM_ACCESS_ALL, 10); (test/gtest/uct/test_atomic_key_reg_rdma_mem_type.cc:33). That path goes to uct_test::entity::mem_alloc(), which retries uct_mem_alloc() up to 10 times while the status is UCS_ERR_NO_MEMORY and then does a hard ASSERT_UCS_OK(status) (test/gtest/uct/uct_test.cc:1005-1035). MEMIC is a tiny, device-global resource shared by all processes on the HCA, and this CI machine runs several gtest workers in parallel ("worker 0"), so the allocation can stay exhausted across all 10 retries and the test fails. The whole history of this test file is exactly this flake being papered over with retries (6c7715d → d7d57e1 "Limit MEMIC allocation to prevent out-of-memory test failures" → [REDACTED:Hex High Entropy String] "Retry when it cannot allocate MEMIC memory" → 69718f6 "Add retries and rand sleep when it cannot allocate MEMIC memory"). Note: the per-test error text itself is earlier in the 2.3M-ms console log than the retrievable tail, so the NO_MEMORY/retry lines could not be quoted directly; everything else in the log is OK/SKIP, there is no ASAN report, no timeout and no hang (log shows continuous activity until teardown). PR #12277 ("TEST/UCP: Add operation variant to RMA SGL tests") only touches UCP RMA gtest code and cannot affect a UCT dc_mlx5 device-memory atomic test.

Implicated commit: Not the PR commit (8ee49ad). The fragile retry-then-assert logic comes from 69718f6 / [REDACTED:Hex High Entropy String] (Michal Shalev), on top of 6c7715d (Roie Danino, added the test).

File: test/gtest/uct/test_atomic_key_reg_rdma_mem_type.cc:33 (and test/gtest/uct/uct_test.cc:1035 — ASSERT_UCS_OK(status) after the retry loop)

Suggested fix: Re-trigger the job — this failure is environmental and not caused by the PR. To stop the recurring flake, make RDMA/MEMIC allocation failure a skip instead of a hard failure: add an optional "allow failure" mode to uct_test::entity::mem_alloc() (or let the caller pre-allocate with uct_md_mem_alloc under scoped_log_handler) and in uct_atomic_key_reg_rdma_mem_type do UCS_TEST_SKIP_R("no MEMIC memory available") when the status is still UCS_ERR_NO_MEMORY after the retries, rather than ASSERT_UCS_OK. Optionally also guard the test so only one gtest worker at a time exercises UCS_MEMORY_TYPE_RDMA.

Related: #10475 (increase retries + random sleep on MEMIC allocation failure), #12277 (the PR under test — unrelated to the failure)

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id dc27d5b8-01d7-44ed-92a3-02df440d62b8 in the triage console for the audit trail.

Comment thread test/gtest/ucp/test_ucp_rma.cc Outdated
Comment on lines 1325 to 1326
/* Parameter validation tests do not depend on the memory type, so they
* run with a single memory type for both source and destination */

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: the comment still sits above get_test_variants(), which now only adds the put/get op variant; it describes get_memtype_variants(), so could you move it one function up?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

* run with a single memory type for both source and destination */
static void get_test_variants(std::vector<ucp_test_variant>& variants) {
add_variant_memtypes(variants, get_base_variants, SGL_MEM_TYPES_MASK);
static const char *op_names[] = {"put", "get"};

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe it worth removing "get" from here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why? This PR is preparation for GET SGL support

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but you have only UCS_BIT(SGL_OP_PUT), so "get" will be never reached, but you can ignore

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@michal-shalev
michal-shalev enabled auto-merge (squash) November 9, 2026 18:57
@svc-nvidia-pr-review

Copy link
Copy Markdown

test/gtest/ucp/test_ucp_rma.cc

minor: these positional variant indexes are easy to break — adding the op dimension silently moved the dst memory type from index 2 to 3 (get_variant_value(3) in dst_mem_type()). Consider adding named constants for the variant indexes (op, src/dst memtype).

test/gtest/ucp/test_ucp_rma.cc

The get variant is not instantiated (UCS_BIT(SGL_OP_PUT) only), so every op() == SGL_OP_GET branch is dead code in CI. (Same concern as an existing open thread.)

test/gtest/ucp/test_ucp_rma.cc

The "get/sgl/offload" protocol name lookup refers to a protocol that does not exist. All existing get protocols are get/zcopy, get/bcopy, get/am/bcopy — none have an offload segment (unlike put/offload/zcopy), so this guessed name is likely wrong. (Same concern as an existing open thread.)

test/gtest/ucp/test_ucp_rma.cc

UCP_SGL_EMULATION_PROTOS is still PROTOS=put/sgl/am/* while the put_emulation* tests now call the generic test_sgl(), and they keep the put_ prefix that every other test dropped.

test/gtest/ucp/test_ucp_rma.cc

init_sgl_ctx_mixed_mem_types() still fills src/dst directly instead of going through the new accessors (harmless, since those ctxs are only used by invalid-param tests).

test/gtest/ucp/test_ucp_rma.cc

The naming overlap between src/dst (local/remote) and data_src/data_dst (data direction) is undocumented and confusing.

test/gtest/ucp/test_ucp_rma.cc

Residual coverage gap: nothing in this change is exercised for get, so the swapped-buffer logic, the UCT_IFACE_FLAG_GET_ZCOPY skip, and the get protocol-name match are only compile-checked; correctness rests entirely on the unchanged put variants.

@michal-shalev
michal-shalev merged commit 4c8063b into openucx:master Nov 11, 2026
167 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants