Skip to content

UCT/IB/RC/VERBS: Support QPs without SRQ - #12066

Merged
tvegas1 merged 33 commits into
openucx:masterfrom
GuangguanWang:pr4-rc-verbs-no-srq
Nov 11, 2026
Merged

tvegas1 merged 33 commits into
openucx:masterfrom
GuangguanWang:pr4-rc-verbs-no-srq

Conversation

@GuangguanWang

@GuangguanWang GuangguanWang commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

What?

Allow rc_verbs to run on devices without SRQ support by creating each QP
with its own receive queue. RC_VERBS_SRQ_ENABLE (y|try|n, default try)
selects whether to use an SRQ.

Why?

rc_verbs currently rejects devices whose max_srq is zero, so it cannot
run on devices which only support per-QP receive queues.

How?

  • Preparation: the RC QP creation helpers accept the receive queue length.
    Other transports continue passing zero. uct_ib_device_has_srq() checks
    device support. The RC flow-control window limit is supplied explicitly
    by rc_verbs and the mlx5 common interface constructor.
  • Selection: SRQ_ENABLE=try uses an SRQ when supported and otherwise falls
    back to per-QP receive queues. n always selects per-QP receive queues.
    y fails with UCS_ERR_UNSUPPORTED when SRQ support is absent.
  • Per-QP receive: RC_VERBS_RX_MAX_WR sets the maximum receive depth
    without SRQ (default 128). Values outside
    1..min(max_qp_wr, UINT16_MAX) are rejected.
    QPs initially prepost up to min(2 * batch, RX_MAX_WR) receive WRs to
    save memory on idle QPs. Receive completions refill the queue in batches
    until fewer than one batch of slots remain or receive buffers are
    unavailable. Preposting and completion-driven refill share the same
    batch-posting helper. Initial posting may be partial when buffers are
    unavailable.
  • Batch and flow control: without SRQ, the refill batch is
    max(min(RX_MAX_BATCH, RX_MAX_WR / 4), 1), independent of RX_QUEUE_LEN.
    The FC window is capped at
    max(floor((RX_MAX_WR - batch + 1) / (1 + FC_SOFT_THRESH)), 1)
    to reserve receive capacity for sends outstanding across soft grants.
    The soft threshold is clamped at zero for this calculation and validated
    afterwards by the existing RC FC threshold check.
    The default RX settings remain 128/32768, giving an FC window of 75
    with the default batch size and soft threshold.
    Partial preposting does not reduce CQ reservations or the FC limit;
    a fresh QP's first burst may still encounter RNR. RNR is also possible
    when flow control is disabled.
    With SRQ, the batch size remains as initialized by the IB base interface.
  • RX progress: the first pass processes active messages and releases
    descriptors for REM_ABORT_ERR and WR_FLUSH_ERR completions. Other receive
    errors remain fatal. Without SRQ, progress then retries empty QPs before
    a second pass performs per-QP receive accounting and completion-driven
    refill. Flushed WRs are not reposted.
  • Refill retry: an empty QP cannot generate another completion to retry
    posting, so posting failure on an empty QP sets an interface flag.
    Retry scans empty QPs, stops when the buffer pool cannot serve a QP,
    and rotates the endpoint list to resume there. Cancelled or failed
    endpoints are not refilled.
  • RX CQ: RC_VERBS_RX_CQ_LEN controls its length without SRQ (default
    32768). It is capped by the device's max_cqe; inf uses that device limit.
    A CQ smaller than RX_MAX_WR is rejected. Each QP reserves RX_MAX_WR
    entries, including QPs pending destruction. Endpoint creation fails
    with UCS_ERR_EXCEEDS_LIMIT when insufficient capacity remains.
    Capacity is returned after QP draining and destruction.
    Diagnostics report the remaining reservation capacity rather than
    the provider-rounded CQ size. MAX_NUM_EPS no longer determines CQ
    allocation; without SRQ, max_num_eps is reported as
    min(MAX_NUM_EPS, RX_CQ_LEN / RX_MAX_WR).
  • QP draining: LAST_WQE is registered for all rc_verbs QPs. Without SRQ,
    uct_rc_iface_qp_cleanup_ready() signals cleanup when all posted receive
    WRs have been accounted, or immediately when none remain at destruction.
    This reuses the existing async cleanup path, including device-fatal
    handling. Hardware LAST_WQE events on QPs without SRQ are ignored.
    Receive accounting for destroyed endpoints is handled outside the inline
    RX path.

Tests

  • test_rc_no_srq covers initial posting, completion-driven refill,
    batch and FC limits, buffer exhaustion and recovery, empty-QP retry
    priority, cancellation, failure and endpoint destruction.
  • rx_batch_limits covers a zero configured batch and small RX_QUEUE_LEN
    values. rx_prepost_grows and rx_prepost_grows_batch_one check
    two-batch initial posting and completion-driven refill for batch sizes
    16 and 1, while an idle QP retains its initial depth.
  • fc_window checks window sizing with different receive depths and
    soft thresholds. fc_soft_thresh_invalid checks rejection of invalid
    thresholds, including negative values and infinity. Receive
    configuration checks also verify that fc_wnd_size + fc_soft_thresh - 1
    does not exceed RX_MAX_WR - batch + 1.
  • failed_rx_release_desc changes a real completion to WR_FLUSH_ERR,
    preserves its QP number, and checks descriptor release, receive-slot
    accounting and absence of reposting.
  • rx_cq_num_eps checks the reported max_num_eps, CQ capacity for four
    QPs, rejection of a fifth endpoint, and capacity recovery after draining
    and destruction. Additional cases cover invalid CQ lengths, finite and
    inf device-limit truncation, and RX_MAX_WR boundaries.
  • LAST_WQE tests cover draining after destruction, already-drained QPs,
    and endpoint destruction from an AM callback before receive accounting.
  • test_rc_srq_select covers try/y selection, try fallback, strict y
    failure without SRQ support, and ignoring the no-SRQ CQ setting with SRQ.
  • Hardware LAST_WQE tests skip QPs without SRQ. Software event tests cover
    dispatch before and after subscribing, duplicate dispatch, and
    unregistering a pending callback.
  • CI reruns a UCT/UCP subset, including AM payload and descriptor tests,
    with UCX_RC_VERBS_SRQ_ENABLE=n, UCX_RC_VERBS_RX_CQ_LEN=262144 and
    UCX_RC_VERBS_RX_MAX_WR=64. Dedicated no-SRQ and SRQ-selection suites
    remain covered by the regular gtest run.
  • RX_MAX_WR is included in the fuzzy environment-variable matching test.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-ucx

svc-ucx commented Oct 9, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests roce on worker 3) · commit 597e636b

TL;DR: rc_mlx5/uct_fence_test.and64/0 hung for exactly 900 s (the gtest watchdog limit) with zero output and was SIGABRT'd — the worker thread is stuck forever in m_entity.flush() waiting for an atomic-fetch completion that never arrives, pointing at this PR series' changes to the shared RC base QP-cleanup / CQ-credit accounting (srq_disable plumbing). This is a hang, not a slow test; do not raise the timeout.

Full analysis

Summary: Build 138607 ("roce on worker 3") failed when gtest's watchdog aborted the run: rc_mlx5/uct_fence_test.and64/0 <rc_mlx5/mlx5_0:1> deadlocked, make: *** [Makefile:4925: test] Aborted (core dumped).

Root cause: Timestamp analysis makes this unambiguous — the test starts at 18:04:35.5630249 and the next log line is at 18:19:35.5854547: a single 900.02-second gap with no intervening output, which is the entire remaining runtime of the job. 900 s is exactly watchdog_timeout = 900. (test/gtest/common/test_helpers.cc:31), so the watchdog fired (ADD_FAILURE() << strerror(ret) << " - abort testing" at test_helpers.cc:57, printing "Connection timed out - abort testing") and pthread_kill'd the watched thread → Caught signal 6 (Aborted).

The backtrace names the hung operation precisely:

  • frame 5: uct_fence_test::worker::join() at test_fence.cc:80 — pthread_join
  • frame 6: uct_fence_test::run_workers<(uct_atomic_op)1>() at test_fence.cc:157

So the main thread is blocked joining a worker that never finished. The worker's only unbounded blocking construct is the per-iteration loop atomic64_post → uct_ep_fence → atomic64_fetch → m_entity.flush() (test_fence.cc:125-129). flush() spins until outstanding ops complete, so the worker is parked waiting on an atomic completion/CQ credit that never returns. No error, no CQE dump, no async event — a silent resource deadlock, not a wire error. Note also a corroborating earlier anomaly in the same run: dcx/test_ucp_wireup_errh_peer.stress_connect_force_disconnect/3 took 33657 ms while its sibling /7 took 669 ms — a 50× stall consistent with intermittent RC TX-resource starvation.

The branch under test (pr4-rc-verbs-no-srq, PR #12066) is the 4th patch of a series that rewrites shared RC base code, not just the verbs TL — git history confirms src/uct/ib/rc/base/rc_iface.c, rc_iface.h and rc_ep.c were all last touched by this series (4fd8bcd2 RC init attrs → 9338869b receive queue length → [REDACTED:Hex High Entropy String] ep_list cleanup → 87649b7d QPs without SRQ, all Guangguan Wang). The most plausible deadlock mechanism introduced there is in uct_rc_ep_cleanup_qp (src/uct/ib/rc/base/rc_ep.c:128-143): when srq_disable is set, the LAST_WQE async-event waiter is not registered and the context is left inert (cb = NULL; cbq = NULL; cb_id = UCS_CALLBACKQ_ID_NULL), relying on the transport to call uct_rc_iface_qp_cleanup_progress() itself. If that self-trigger is missed on any path, the cleanup ctx stays on qp_gc_list forever and its cq_credits are never returned through uct_rc_iface_add_cq_credits() (rc_iface.c:772-775), so iface->tx.cq_available is permanently under-credited, uct_rc_iface_have_tx_cqe_avail() stays false, every post returns UCS_ERR_NO_RESOURCE, and flush() never completes — exactly the observed silent hang.

Implicated commit: 87649b7d "UCT/IB/RC/VERBS: Support QPs without SRQ" (Guangguan Wang), in the series with 4fd8bcd2 and 9338869b — these are the only recent changes to the shared RC base files; HEAD of the PR is [REDACTED:Hex High Entropy String].

File: test/gtest/uct/test_fence.cc:129 (hang point, m_entity.flush(), joined at :80 from :157); root cause in src/uct/ib/rc/base/rc_ep.c:131-143 and src/uct/ib/rc/base/rc_iface.c:763-775.

Suggested fix:

  1. Don't treat this as a flake or bump watchdog_timeout — the 900 s gap is a deadlock. Reproduce with GTEST_FILTER='rc_mlx5/uct_fence_test.and64*' --gtest_repeat=50 and attach gdb to dump iface->tx.cq_available, iface->config.tx_cq_len, txqp->available, and whether iface->qp_gc_list is non-empty — a non-empty qp_gc_list with depleted cq_available confirms the leaked-CQ-credit path above.
  2. In uct_rc_ep_cleanup_qp, make the srq_disable branch non-silent: either schedule uct_rc_iface_qp_cleanup_progress on the worker progress queue unconditionally (so credits are always reclaimed), or add a ucs_assert/ucs_warn that the transport has registered a self-trigger, so a missed trigger fails loudly instead of hanging.
  3. Verify init_attr->srq_disable is explicitly initialized on every RC-derived transport. rc_mlx5 relies on uct_rc_iface_init_attr_t init_attr = {} (rc_mlx5_iface.c:1001) to zero it; add an explicit assignment plus an assert in UCS_CLASS_INIT_FUNC(uct_rc_iface_t) (rc_iface.c:596) so a future transport can't silently inherit the no-SRQ cleanup path.
  4. Since the failure is on rc_mlx5 — a transport this PR nominally shouldn't change — bisect the series (4fd8bcd2, 9338869b, [REDACTED:Hex High Entropy String], 87649b7d) to pin which patch regresses the mlx5 atomic/fence path, and consider gating the shared-base refactor behind the verbs TL until that's resolved.

Related: PR #12066 (this build), and the rest of the series: #12061 ("UCT/IB/RC: Preparation for QPs without an SRQ"), #11933, #11938 (gtest coverage for rc_verbs QPs without SRQ), #12060.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 9525ed7c-cace-4f6f-b133-5816afc7145a in the triage console for the audit trail.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-ucx

svc-ucx commented Oct 10, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests roce on worker 1) · commit 653e4b53

TL;DR: The roce on worker 1 gtest run aborted in shm_ib_ipc/test_ucp_atomic64.fetch because the mlx5_0 UMR QP self-loopback never completed (ibv_poll_cq(UMR CQ, registration) timed out after 30 s), which failed ucp_rkey_pack and then left a referenced rcache region behind, tripping the region->refcount == 0 assertion at ucp_cleanup. This is the known RoCE/LAG UMR-QP address-handle bug (issue #11709 / PR #11780), not a regression from PR #12066.

Full analysis

Summary: gtest shm_ib_ipc/test_ucp_atomic64.fetch failed with Error: Input/output error at test/gtest/ucp/test_ucp_memheap.cc:68, followed by Assertion 'region->refcount == 0' failed in ucs_mem_region_destroy_internal() → make: *** [test] Aborted (core dumped).

Root cause: Two-stage failure, both in the UMR (exported-mkey) path on a RoCE device:

  1. uct_ib_mlx5_devx_umr_post_sync() posted the UMR BIND_MW ("registration") WR on the internal self-connected UMR RC QP and spun for the full 30-second timeout without a completion (mlx5_0: ibv_poll_cq(UMR CQ, registration, UMR mkey ... index 0x3183) timed out), then the follow-up invalidate failed with ibv_post_send(UMR QP, invalidation, ...) returned 12: Bad address (QP already in error state). The UMR QP is brought to RTR in uct_ib_mlx5_devx_umr_modify_qp() using an AH built for IB semantics: ah_attr.dlid = port_attr->lid (always 0 on RoCE), is_global = 1 with only grh.dgid filled — grh.hop_limit, grh.sgid_index and traffic_class are left zero. With hop_limit 0 the RoCEv2 loopback packets are never delivered, so the UMR completion never arrives. This is exactly the failure mode tracked in issue UMR QP INIT2RTR fails on LAG-bonded devices, causing segfault on memory registration #11709 and still-open PR UCT/IB/MLX5: Fix UMR QP INIT2RTR failure on LAG-bonded devices #11780.
  2. The timeout propagates as UCS_ERR_IO_ERROR out of ucp_rkey_pack (gtest ASSERT_UCS_OK at test_ucp_memheap.cc:68), but the partially-registered memh is not rolled back — ucs_rcache: destroying inuse region ... ref 1 md[2]=mlx5_0 ... — so ucs_rcache_purge() at ucp_cleanup hits the refcount == 0 assertion and aborts the whole gtest binary (killing the remaining test suites in the shard).

Implicated commit: unknown — the failing code is the pre-existing UMR export path (uct_ib_mlx5_devx_umr_modify_qp / uct_ib_mlx5_devx_umr_post_sync). PR #12066 (a0c9e002/0f666eda, Guangguan Wang, rc_verbs QPs without SRQ) touches only rc_verbs TX/RX queues and is not on this code path; the branch name pr4-rc-verbs-no-srq is incidental.

File: src/uct/ib/mlx5/dv/ib_mlx5dv_md.c:1218-1240 (RTR transition, missing RoCE GRH fields) and src/uct/ib/mlx5/dv/ib_mlx5dv_md.c:1447-1454 (30 s poll timeout); secondary: src/ucs/memory/rcache.c:394 reached via src/ucp/core/ucp_mm.c:1860.

Suggested fix:

  1. Fix the UMR QP AH for RoCE — set qp_attr.ah_attr.grh.hop_limit (e.g. 64, or UCT_IB_ROCE_UDP_...-consistent default), grh.sgid_index = UCT_IB_DEVICE_DEFAULT_GID_INDEX, grh.traffic_class (RoCEv2 DSCP), and skip/ignore dlid when the port link layer is Ethernet (IBV_LINK_LAYER_ETHERNET). Adopting/merging PR UCT/IB/MLX5: Fix UMR QP INIT2RTR failure on LAG-bonded devices #11780 is the direct path.
  2. Make the error path safe: when uct_ib_mlx5_devx_umr_mkey_bind() fails, the caller must fully unwind the registration so the rcache region's refcount drops to 0 (and prefer reporting the failure rather than leaving the region in use) — otherwise any UMR failure turns a single test failure into a core dump that aborts the entire shard.
  3. Short term, to unblock PR CI: re-run the roce on worker 1 job; this is an environment-dependent (RoCE device mlx5_0 on swx-rain03) latent bug, not a PR regression.

Related: #11709, #11780, #11695, PR under test: #12066

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-ucx

svc-ucx commented Oct 20, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR · commit e19a08e5

TL;DR: The roce perftest jobs hung — ucx_perftest over rc_verbs,rc_x produced its last output at 19:05:40 and then went completely silent for 3h17m until the agent was killed. The hang points at this PR's new ucs_list_del(&self->list) in the uct_rc_ep_t cleanup path (rc_ep.c:188), which deletes the EP from iface->ep_list without re-initializing the node, corrupting the list for subsequent traversal/teardown.

Full analysis

Summary: All three roce workers failed; ucx_perftest (UCP stream latency test, UCX_TLS=rc_verbs,rc_x, UCX_NET_DEVICES=mlx5_0:1) hung indefinitely during/after the test and the Azure agent was force-shut-down, terminating orphaned mpirun and two ucx_perftest processes.

Root cause: This is a hang, not a timeout. Timeline evidence from the roce on worker 1 log:

  • 19:05:39.61 — mpirun ... ucx_perftest -b test_types_short_ucp -b msg_pow2_short -w 1 launched
  • 19:05:40.36 — last application output: the ucp_contig_stream_data_lat/1048576 stage header plus two benign UCX_TCP_PORT_RANGE unused-env warnings
  • 19:05:40 → 22:22:31 — 3 hours 17 minutes of total silence (the entire remaining runtime)
  • 22:22:31 — ##[error]The agent has received a shutdown signal, then Terminate orphan process: pid (3945526) (ucx_perftest) / pid (3945529) (ucx_perftest)

The exit/cancel is a symptom; both ucx_perftest ranks were still alive and wedged. The test never printed a single result row, so it stalled at the very start of the ucp_contig_stream_data_lat stage on an RC transport — exactly the code path this PR modifies.

The PR under test (#12059, "UCT/IB/RC: Remove an endpoint from ep_list in uct_rc_ep_t cleanup") adds ucs_list_del(&self->list) to the base-class destructor:

  • EPs are linked in at rc_ep.c:164 (ucs_list_add_head(&iface->ep_list, &self->list)) during UCS_CLASS_INIT_FUNC(uct_rc_ep_t).
  • The new ucs_list_del(&self->list) at rc_ep.c:188 unlinks it but leaves self->list pointing at its old neighbours — it is not reset with ucs_list_head_init(&self->list).

Two concrete hazards follow, both consistent with a silent infinite spin:

  1. Double removal / stale node. Base-class cleanup runs after derived (rc_verbs/rc_mlx5) EP cleanup. If any other path already unlinks the EP, the second ucs_list_del on a stale node writes through dangling prev/next pointers and splices iface->ep_list into a cycle. Any later ucs_list_for_each over ep_list then never terminates.
  2. Deferred QP teardown. uct_rc_ep_cleanup_qp (rc_ep.c:110-134) does not complete synchronously — it registers an IBV_EVENT_QP_LAST_WQE_REACHED async-event wait and defers to uct_rc_iface_qp_cleanup_progress via the worker progress queue. Removing the EP from ep_list at destroy time means a still-in-flight EP is no longer reachable for flush/progress/error handling, so a pending flush or credit-return can wait forever for a completion that is never driven. Note uct_rc_iface_t cleanup only warns ("some eps were not destroyed", rc_iface.c:795-797) — it does not recover.

Implicated commit: [REDACTED:Hex High Entropy String] — "UCT/IB/RC: Remove an endpoint from ep_list in uct_rc_ep_t cleanup", Guangguan Wang (the sole recent change to rc_ep.c introducing ep_list removal; branch pr1-rc-eplist-cleanup, PR #12066 built from #12059)

File: src/uct/ib/rc/base/rc_ep.c:188 (removal) — paired with insertion at src/uct/ib/rc/base/rc_ep.c:164

Suggested fix:

  1. Make the unlink idempotent and safe: initialize the node in UCS_CLASS_INIT_FUNC and reset it after deletion, so a stale/second ucs_list_del cannot corrupt the list:
    /* in cleanup, rc_ep.c:188 */
    ucs_list_del(&self->list);
    ucs_list_head_init(&self->list);   /* make re-deletion harmless */
    and audit rc_verbs_ep.c / rc_mlx5_ep.c (and the GGA/GDAKI derivatives) for any pre-existing ucs_list_del(&ep->list) that would now run twice.
  2. Verify ordering against the deferred QP cleanup: because uct_rc_ep_cleanup_qp completes asynchronously via qp_gc_list + the progress queue, confirm nothing still needs to reach the EP through ep_list after the destructor. If it does, keep the EP linked until uct_rc_iface_qp_cleanup_progress fires rather than unlinking in the base destructor.
  3. Reproduce locally with UCX_TLS=rc_verbs,rc_x ucx_perftest -b test_types_short_ucp -b msg_pow2_short -w 1 and attach gdb/pstack to a wedged rank to capture the spin — a loop inside an ep_list traversal would confirm hypothesis (1), while a blocked flush/completion wait would confirm (2).
  4. Add a CI guard: the job burned 3h17m on a dead process. Give ucx_perftest invocations in contrib/test_jenkins.sh a timeout wrapper so a hang fails fast with a usable stack instead of an agent shutdown.

Related: PR #12059 (the change being tested); PR #12066 (this build). No existing issue found for this hang signature.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id d72f24a0-1561-46f8-b726-dc876c1fa137 in the triage console for the audit trail.

@GuangguanWang
GuangguanWang marked this pull request as ready for review October 20, 2026 07:48

@tvegas1 tvegas1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need to fix conflict with master.

Comment thread src/uct/ib/rc/base/rc_iface.h Resolved
Comment thread src/uct/ib/base/ib_device.h Resolved
Comment thread src/uct/ib/base/ib_iface.h Resolved
Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c Resolved
Comment thread src/uct/ib/rc/base/rc_ep.c Resolved
Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c Resolved
Comment thread src/uct/ib/rc/base/rc_ep.c Resolved
Comment thread src/uct/ib/rc/verbs/rc_verbs_ep.c Resolved
Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c Resolved
Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread contrib/test_jenkins.sh Resolved
Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c Resolved
Comment thread src/uct/ib/rc/verbs/rc_verbs_impl.h Resolved
Comment thread src/uct/ib/rc/verbs/rc_verbs.h Resolved
Comment thread src/uct/ib/rc/verbs/rc_verbs_impl.h Outdated
{
uct_rc_verbs_iface_qp_cleanup_ctx_t *cleanup_ctx;

ucs_list_for_each(cleanup_ctx, &iface->super.qp_gc_list, super.list) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this scans the whole gc list for every flush CQE, so destroying N endpoints costs O(N * rx_max_wr) list steps. can we reach the cleanup ctx directly, e.g. keep it addressable by qp_num until it is drained?

Comment thread src/uct/ib/base/ib_device.c Resolved
Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

PR-level (Size): this PR adds well over 500 lines (≈1020 in test/gtest/uct/ib/test_rc.cc alone, plus ~90 in test_ib_event.cc and ~250 across rc_verbs_*). REVIEW.md sets 500 added lines as a hard limit. The uct_ib_device_async_event_fire() + test_ib_event.cc software-LAST_WQE part, and the test_rc_no_srq suite, are separable from the core no-SRQ RX path and would each make a reviewable PR.

Residual coverage gap: all new test_rc_no_srq/test_rc_srq_select cases are excluded from the new no-SRQ CI job's filter, so the no-SRQ path with SRQ_ENABLE=n is exercised either by the dedicated fixture (default job) or by the broad filter (new job), but never both in one configuration. Worth confirming the new job actually ran green on hardware before merge.

Notes on things I checked and found OK:

  • rx_cq_available accounting is symmetric: decremented only after a successful uct_rc_verbs_ep_t init, returned in uct_rc_verbs_iface_qp_cleanup() after ibv_destroy_qp(), so flush CQEs of a destroyed QP are still covered by the CQ.
  • cleanup_ctx->rx_remaining = rx_max_wr - rx_available matches the outstanding recv WRs, including the partial-prepost case; the err_remove_qp path correctly ucs_list_del()s the pending-list link that post_recv_common() may have added.
  • repost_pending() keeps a short-served ep at the head and cannot double-insert, since post_recv_common() only adds when the link is empty.
  • config.srq_enabled is assigned before ops->init_rx() runs, and rc_mlx5 sets init_attr->srq_enabled = 1 for rc_mlx5/dc_mlx5/gdaki, so no iface reaches init_rx with a stale value.

@svc-ucx

svc-ucx commented Oct 29, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Coverity coverity devel on coverity_rh7) · commit b4874598

TL;DR: The Coverity stage failed with 3 new NEGATIVE_RETURNS defects in test/gtest/uct/ib/test_rc.cc — the new helper first_waiting_ep_index() can return -1, and Coverity doesn't treat the following ASSERT_NE(-1, ...) as terminating, so the value flows into EP-index lookups; replace the gtest ASSERT guard with a UCS_TEST_ABORT (throwing) check so the negative path is provably dead.

Full analysis

Summary: Coverity analysis job coverity devel on coverity_rh7 for PR #12066 ended with ##[error]Coverity found 3 issues (exit 3) — three NEGATIVE_RETURNS findings at test/gtest/uct/ib/test_rc.cc:1190, :1196 and :1228.

Root cause: Commit b487459 ("GTEST/UCT/IB: Extend the test coverage of QPs without SRQ") added the helper int first_waiting_ep_index() which explicitly returns -1 when no EP is waiting (test_rc.cc:783). The new test bodies store that value and guard it only with ASSERT_NE(-1, first_ep) / ASSERT_NE(-1, short_ep). Coverity's gtest model does not understand that a failing ASSERT_* returns from the test body — the report explicitly shows "Condition gtest_ar.operator bool(), taking true branch … Falling through to end of if statement" — so the -1 path remains live and the index is then passed to sinks that cannot accept negatives:

  • line 1190 → rx_post_recv_waits(m_e2, first_ep) → verbs_ep() → rc_ep() → entity::ep() → std::vector::at(index) (uct_test.cc:1227)
  • line 1196 → rx_available(m_e2, short_ep) → same chain
  • line 1228 → m_e2->destroy_ep(first_ep) → operator[](index) (uct_test.cc:1299)

Note prev_available[short_ep] at line 1196 would also be an out-of-bounds read on that path, so this is a legitimate hardening request, not pure noise. The build itself compiled cleanly (752/752 TUs); only the defect gate failed.

Implicated commit: [REDACTED:Hex High Entropy String] — Guangguan Wang, "GTEST/UCT/IB: Extend the test coverage of QPs without SRQ"

File: test/gtest/uct/ib/test_rc.cc:772-784 (helper returning -1); defect sites at :1183-1196 and :1224-1228

Suggested fix: Make the negative path unreachable in a way Coverity can model, by using the throwing UCS_TEST_ABORT instead of ASSERT_NE:

first_ep = first_waiting_ep_index();
if (first_ep < 0) {
    UCS_TEST_ABORT("no endpoint is waiting for receive buffers");
}

(and likewise for short_ep at :1194 and first_ep at :1224). UCS_TEST_ABORT throws, so Coverity sees the call as non-returning and the -1 value never reaches vector::at/operator[].

Alternative: keep one assertion helper, e.g. have first_waiting_ep_index() itself abort when nothing is queued and return an unsigned/size_t, with a separate bool any_waiting_ep() predicate for the tests that need to check for absence (EXPECT_FALSE(rx_post_recv_pending(...))). Avoid suppressing via a Coverity annotation — the prev_available[short_ep] indexing makes the guard worth having for real.

Related: PR #12066 (branch pr4-rc-verbs-no-srq); preceding commits in the same series: [REDACTED:Hex High Entropy String], db6e83a, [REDACTED:Hex High Entropy String], b07e85e. No matching existing issue found.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 38341d74-b273-43f8-9d46-4eb7f4d177e8 in the triage console for the audit trail.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread contrib/test_jenkins.sh Resolved
Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gap (unchanged from the previous round): every new test_rc_no_srq/test_rc_srq_select case is excluded by the new job's negative filter, so no single configuration runs both the dedicated no-SRQ fixtures and the broad suite with SRQ_ENABLE=n. Please confirm the new job ran green on real hardware (especially given the inf RX CQ sizing finding) before merge.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/ib/rc/verbs/rc_verbs_impl.h Outdated
uct_rc_verbs_iface_poll_rx(uct_rc_verbs_iface_t *iface)
{
unsigned i, num_wcs = iface->super.super.config.rx_max_poll;
uint8_t srq_enabled = iface->super.config.srq_enabled;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we select the mode at setup time instead of branching per CQE? super.progress is already a function pointer, so two wrappers passing srq_enabled as a constant to an ALWAYS_INLINE poll_rx() (like uct_rc_mlx5_iface_progress_cyclic/_ll/_tm do with flags) would keep the SRQ RX path free of the new conditionals - docs/OptimizationStyle.md: "Prefer setup-time selection over hot-path conditionals".

Comment thread src/uct/ib/rc/verbs/rc_verbs.h Resolved
Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c Resolved
@svc-ucx

svc-ucx commented Oct 29, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (AddressSanitizer roce on worker 0) · commit 908db0a8

TL;DR: The new run_rc_verbs_nosrq_gtest task (added by this PR) hung: rc/test_ucp_wireup_1sided.one_sided_wireup_rndv/5 <rc_v/tag,unified> made no progress for ~15 min until the gtest watchdog SIGTERM'd it, because with UCX_RC_VERBS_SRQ_ENABLE=n the per-QP receive queue stopped being replenished, so the rendezvous RTS never reached the peer (RNR forever) and neither the send nor the recv request ever completed.

Full analysis

Summary: AddressSanitizer/roce worker 0 → ==== Running rc_verbs gtest with SRQ disabled ==== → rc/test_ucp_wireup_1sided.one_sided_wireup_rndv/5 stalls permanently; ucp_test.cc:310 request ... did not complete on time, test_ucp_wireup.cc:344/346 show m_recv_data[0] == 0 and 0 matching elements, then watchdog "Connection timed out - abort testing" → signal 6, make: *** [Makefile:4971: test] Aborted (core dumped).

Root cause: A hang, not slowness — the log has one 11-minute gap with zero output (12:53:23 [ RUN ] ...one_sided_wireup_rndv/5 → 13:04:40 request-timeout error), then a second silent gap to the 900 s watchdog kill at 13:08:22. No data arrived at all (recv buffer all zeros), there is no ASAN report and no UCX error, which is the signature of a receive-side RNR stall in the new no-SRQ path, exercised only by this PR's new job (UCX_RC_VERBS_SRQ_ENABLE=n UCX_RC_VERBS_MAX_NUM_EPS=inf, contrib/test_jenkins.sh:1277-1294).

The no-SRQ design relies on an invariant that the new reposting logic does not guarantee:

  • uct_rc_verbs_iface_iface_init sets the FC window to fc_max_wnd_size = rx_max_wr - rx_batch_limit (112 of 128 WRs) — safe only if every QP always keeps ≥ rx_max_wr - batch receive WRs posted (rc_verbs_iface.c:336-349).
  • Replenishment is driven exclusively by receive completions: uct_rc_verbs_iface_rx_complete_no_srq() → uct_rc_verbs_iface_post_recv_common(..., fill=0), which posts nothing unless ep->rx_available >= rx_max_batch (rc_verbs_impl.h:43-56, 144-161). An empty RQ produces no CQE, so once a QP drains, nothing re-arms it.
  • The only rescue path, rx_repost_pending_eps (added in the PR's top commit), is lossy: uct_rc_verbs_iface_repost_pending() removes the endpoint from the retry list after a single successful batch, and also removes it whenever ep->rx_available < batch — without posting the remainder (fill=0) (rc_verbs_iface.c:539-560). A partial ibv_post_recv (descriptor pool momentarily short — much likelier under ASAN, where this job also creates a ~4M-entry RX CQ because MAX_NUM_EPS=inf ⇒ rx_cq_len = max_cqe) therefore leaves the QP below the FC window permanently, and the peer's next AM (the rndv RTS) sits in RNR retry forever with rnr_retry=7 (infinite), which produces exactly this silent, unrecoverable hang.

Implicated commit: 5ad8c71 "UCT/IB/RC: Queue endpoints pending receive reposting" (Guangguan Wang), part of the series 6d7565b / 79057a2 / 849e2b3 / 0572bd0 / 0e7c11a in PR #12066.

File: src/uct/ib/rc/verbs/rc_verbs_iface.c:539-560 (and src/uct/ib/rc/verbs/rc_verbs_impl.h:39-70, 144-161; FC-window assumption at rc_verbs_iface.c:336-349)

Suggested fix:

  1. Make the retry queue sticky: in uct_rc_verbs_iface_repost_pending(), keep the endpoint queued until its receive queue is actually refilled (e.g. ep->rx_available == 0, or posted WRs ≥ iface->super.config.fc_wnd_size), instead of dequeuing after one full batch.
  2. Never drop an endpoint that still has unposted slots: call post_recv_common(iface, ep, /*fill=*/1) for sub-batch remainders rather than list_deling it when rx_available < batch.
  3. Track posted RWRs per endpoint and add a ucs_warn/assert when a connected QP's RQ reaches 0 or falls below fc_wnd_size — this turns the current silent 15-minute hang into an actionable diagnostic.
  4. Also consider replenishing from the descriptor-release path (release_desc) and capping rx_cq_len when MAX_NUM_EPS=inf, so a 256 MB RX CQ per iface isn't created in the ASAN job.
  5. To confirm quickly, re-run the failing filter with UCX_LOG_LEVEL=debug UCX_RC_VERBS_RX_MAX_WR=512 UCX_IB_RX_MAX_BATCH=4 — if it stops hanging, the receive-replenishment threshold is confirmed as the culprit.

Related: PR #12066 (this PR, "UCT/IB/RC/VERBS: Support QPs without SRQ"), earlier attempt #11933, and #12060 ("Release the descriptor of a failed receive completion") which touched the same no-SRQ receive-completion path.

Comment thread src/uct/ib/rc/base/rc_iface.c Resolved
Comment thread src/uct/ib/rc/base/rc_iface.c Resolved
Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c Resolved
Comment thread src/uct/ib/rc/verbs/rc_verbs_impl.h Resolved
Comment thread src/uct/ib/rc/verbs/rc_verbs_ep.c Resolved
Comment thread contrib/test_jenkins.sh Outdated
echo "==== Running rc_verbs gtest with SRQ disabled ===="
$TIMEOUT env \
UCX_RC_VERBS_SRQ_ENABLE=n \
UCX_RC_VERBS_MAX_NUM_EPS=inf \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CI OOM looks self-inflicted: run_rc_verbs_nosrq_gtest sets UCX_RC_VERBS_MAX_NUM_EPS=inf, overriding the RC default of 256, so every rc_verbs iface created by the gtests gets an RX CQ of ~max_cqe entries (≈4M, ~256MB pinned on ConnectX). Why is inf needed? If some tests create more than 256 eps, could we use a finite value (e.g. 1024, as test_ucp_wireup does with UCX_RC_MAX_NUM_EPS)? Also, since inf silently maps to the largest CQ the device supports, maybe cap it or document the memory cost in the RX_MAX_WR help text.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replaced in de689c3 by MAX_NUM_EPS=4096 with RX_MAX_WR=48, i.e. an RX
CQ of at most 196608 entries. I used 4096 rather than 1024 to leave headroom
for stress_connect2, which connects up to 1000 endpoints while the RX CQ
capacity of destroyed endpoints returns only after their QPs drain.

I left the RX_MAX_WR help as is, since it already says that
MAX_NUM_EPS=inf uses the largest RX CQ the device supports.

Comment thread src/uct/ib/rc/verbs/rc_verbs.h Resolved
Comment thread src/uct/ib/rc/verbs/rc_verbs.h Resolved
Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c Outdated
return uct_rc_iface_init_rx(rc_iface, config, &iface->srq);
}

iface->srq = NULL;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

might bwe able to moving =NULL inside uct_rc_iface_init_rx()?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I kept it in rc_verbs: uct_rc_iface_init_rx() always creates the SRQ and is
shared with rc_mlx5, and rc_verbs does not call it without an SRQ, so the
assignment there would not run on this path.



/* Force rc_verbs to create QPs without an SRQ. */
class test_rc_no_srq : public test_rc {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New tests could be trimmed:

  • rx_cq_no_space overlaps rx_cq_num_eps (both check EXCEEDS_LIMIT and the quota return after drain), and it sets rx_cq_available directly
  • am_and_qp_drain and flush_cancel are covered by last_wqe_after_rx_drain / last_wqe_already_drained, and the reposting part by uct_p2p_am_test in the new no-SRQ CI job
  • fc_wnd_size and rx_max_batch_limit could be a single config test
  • in test_rc_srq_select, try_uses_srq duplicates yes_uses_srq, and "RC_VERBS_SRQ_ENABLE=try" is the default
  • no_rx_bufs_waiting_order/no_rx_bufs_waiting_ep_destroy and their helpers test the queue order rather than behavior; no_rx_bufs_many_eps already checks that every QP recovers (and they would go away with the repost suggestion above)
  • the hand-written deadline loops could use wait_for_cond()/wait_for_value()

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 576f311, with two differences:

  • try_uses_srq and yes_uses_srq are merged into one uses_srq test
    rather than dropping try: setting it explicitly prevents an SRQ_ENABLE
    environment setting from overriding it, and lets devices without SRQ skip
    instead of failing iface creation; the test then reopens with y.
  • The waiting tests are kept, adapted to the iface flag. They check what
    no_rx_bufs_many_eps does not: empty QPs are served before partially
    filled ones, and destroying the QP at the retry position, or all empty
    QPs, leaves no stale retry.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

* enabled, which is not done by opening an interface. */
EXPECT_EQ(UCS_OK, try_rx_max_wr("1"));

snprintf(str, sizeof(str), "%d",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[redacted: contained suspected secret] this opens an iface with RX_MAX_WR=max_qp_wr while MAX_NUM_EPS is the default 256, so rx_cq_len = min(256, max_cqe/[REDACTED]) * [REDACTED] ≈ 4M CQEs (~260MB of pinned memory on mlx5) — slow and likely to fail on the memlock limit. pls add "MAX_NUM_EPS=1" to the test config; the fixture still works because rx_cq_len is max'ed with IB_RX_QUEUE_LEN.

Rationale: the comment above the check only reasons about receive descriptors ("receive WRs are posted when progress is enabled"), but the RX CQ is sized and allocated at uct_iface_open() time regardless of progress.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, with RX_MAX_WR=max_qp_wr and the default 256 eps the RX CQ is ~127 * 32K ≈ 4M entries (~260MB pinned) just to check the config; MAX_NUM_EPS=1 limits it to a single receive queue.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 2e7070e. With the separate CQ setting, the upper-bound test sets
RX_CQ_LEN to the tested receive depth instead of using MAX_NUM_EPS=1.

Comment thread src/uct/ib/base/ib_device.c Resolved
Comment thread src/uct/ib/rc/base/rc_iface.c Resolved
Comment thread src/uct/ib/rc/verbs/rc_verbs_impl.h Outdated
uct_rc_verbs_iface_poll_rx(uct_rc_verbs_iface_t *iface)
{
unsigned i, num_wcs = iface->super.super.config.rx_max_poll;
uint8_t srq_disabled = (iface->srq == NULL);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

soft suggestion: maybe select a dedicated progress function for the no-SRQ mode (like rc_mlx5 does for srq topology) and pass the flag as a compile-time argument to uct_rc_verbs_iface_poll_rx()? it would keep the existing SRQ path free of the new per-CQE branches.

@svc-nvidia-pr-review

Copy link
Copy Markdown

PR size (>500 added lines, ~900 in test_rc.cc alone): already raised as a PR-level comment in an earlier round; it still applies to this revision, so it should get a +1 reaction on the existing thread rather than a new comment.

MAX_NUM_EPS=inf sizing the RX CQ to max_cqe: raised in an earlier round and now documented in the RX_MAX_WR help text — not re-raised.

Residual coverage gap (unchanged from earlier rounds): the new no-SRQ job excludes test_rc_no_srq/test_rc_srq_select, so no single configuration runs both the dedicated no-SRQ fixtures and the broad suite with SRQ_ENABLE=n. Worth confirming the new job ran green on real hardware before merge, especially the RNR-driven rnr_fc_disabled test and the large-CQ paths.

Comment thread src/uct/ib/rc/base/rc_iface.c Resolved
* enabled, which is not done by opening an interface. */
EXPECT_EQ(UCS_OK, try_rx_max_wr("1"));

snprintf(str, sizeof(str), "%d",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, with RX_MAX_WR=max_qp_wr and the default 256 eps the RX CQ is ~127 * 32K ≈ 4M entries (~260MB pinned) just to check the config; MAX_NUM_EPS=1 limits it to a single receive queue.

Comment thread src/uct/ib/rc/verbs/rc_verbs.h Outdated
size_t max_inline;
size_t max_send_sge;
unsigned tx_max_wr;
uint16_t rx_max_wr; /* receive WRs per QP without SRQ,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would it make sense to move the new uint16_t to unsigned like above for tx_max_wr?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 5222798.

Comment thread src/uct/ib/rc/verbs/rc_verbs_ep.c Resolved
Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c Resolved
Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c Outdated
ucs_max(rx_max_wr / 4, 1));
/* Room for the receive WRs of all endpoints; the 64-bit endpoint
* count is capped by the largest CQ before multiplying. */
num_eps = ucs_min(ib_config->super.max_num_eps,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about a dedicated RC_VERBS_RX_CQ_LEN for the no-SRQ case, instead of deriving the RX CQ size from MAX_NUM_EPS?

Today MAX_NUM_EPS is only a hint reported to UCP; with this change it sizes the CQ and becomes a hard endpoint limit, and RX_QUEUE_LEN ends up only as a CQ minimum. Something like:

  {"RX_MAX_WR", "128",
   "Number of receive WRs posted to each QP when SRQ_ENABLE=n or the device\n"
   "does not support SRQ.",
   ucs_offsetof(uct_rc_verbs_iface_config_t, rx_max_wr), UCS_CONFIG_TYPE_UINT},

  {"RX_CQ_LEN", "32768",
   "Length of the receive completion queue when SRQ_ENABLE=n or the device\n"
   "does not support SRQ. Limits the number of endpoints to RX_CQ_LEN/RX_MAX_WR.\n"
   "\"inf\" uses the maximum CQ length reported by the device.",
   ucs_offsetof(uct_rc_verbs_iface_config_t, rx_cq_len), UCS_CONFIG_TYPE_ULUNITS},
rx_cq_len = ucs_min(config->rx_cq_len, IBV_DEV_ATTR(&ib_md->dev, max_cqe));
if (rx_cq_len < rx_max_wr) {
    /* error: RX_CQ_LEN must be >= RX_MAX_WR */
}
/* endpoint limit = rx_cq_len / rx_max_wr */

The defaults RX_MAX_WR=128 and RX_CQ_LEN=32768 keep today's 256-endpoint limit and ~4 MB CQ. It keeps MAX_NUM_EPS a hint, gives one explicit knob for the CQ memory (~64 B per CQE, so 128K entries ≈ 8-16 MB covers 1024 endpoints), and the endpoint creation error could point users to RC_VERBS_RX_CQ_LEN.

As a follow-up (separate PR), it would be nice to remove the hard endpoint limit, either by:

  • growing the RX CQ with ibv_resize_cq() when a new endpoint does not fit, or
  • allowing more QPs than RX_CQ_LEN/RX_MAX_WR while never preposting more than RX_CQ_LEN receives in total across all QPs (QPs beyond the budget would get fewer receives and rely on RNR retry/FC).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 2e7070e. Removing the hard endpoint limit remain follow-up work for a separate PR.

Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c Resolved
Comment thread src/uct/ib/rc/verbs/rc_verbs_impl.h Resolved
Comment thread src/uct/ib/rc/verbs/rc_verbs_impl.h Outdated
}
UCT_IB_IFACE_VERBS_COMPLETION_FATAL("receive", &iface->super.super, i, wc);
uct_rc_verbs_iface_rx_error(iface, &wc[i], desc);
} else {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we remove the else() block/indent as error seems fatal to improve readability

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The error helper is not always fatal: REM_ABORT_ERR and WR_FLUSH_ERR
release the descriptor and return. These completions must skip AM handling
but still reach the receive accounting below, so I kept the else block.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair point, thinking more, can we rework the code like below? The no-SRQ accounting to out:, like the SRQ refill which would simplify the function. This removes the previously requested helper funtion.

        if (ucs_unlikely(wc[i].status != IBV_WC_SUCCESS)) {
            if ((wc[i].status != IBV_WC_REM_ABORT_ERR) &&
                (wc[i].status != IBV_WC_WR_FLUSH_ERR)) {
                UCT_IB_IFACE_VERBS_COMPLETION_FATAL("receive",
                                                    &iface->super.super, i,
                                                    wc);
            }

            /* A failed receive never delivers an active message. */
            ucs_mpool_put_inline(desc);
            continue;
        }

        /* hdr, valgrind, log, handle_am */
    }
    ...

out:
    if (<srq is enabled>) {
        iface->super.rx.srq.available += num_wcs;
        uct_rc_verbs_iface_post_recv_common(iface, NULL, 0);
    } else {
        /* Give empty QPs returned buffers before refilling active QPs. */
        if (ucs_unlikely(iface->rx_repost_needed)) {
            uct_rc_verbs_iface_repost_recv(iface);
        }

        for (i = 0; i < num_wcs; i++) {
            /* Do not repost flushed WRs */
            uct_rc_verbs_iface_rx_complete_no_srq(iface, wc[i].qp_num,
                                                  wc[i].status !=
                                                  IBV_WC_WR_FLUSH_ERR);
        }
    }

This would:

  • bring the loop back to master's shape, without the helper and the else
  • keep all the no-SRQ receive handling in one place, so the entry check goes away
  • remove the per-CQE branch from the SRQ path, which also covers the bot's comments about that
  • fix the completion[0] index from the bot's comment

At out:, before the accounting loop, repost_recv() sees the same rx_available values as at entry today, so the set of QPs it serves doesn't change; it can also use buffers released by AM callbacks during this poll.

An ep destroyed from an AM callback is still accounted correctly: its rx_remaining includes the not yet accounted CQEs of this poll, and they reach gc_drain_cqe() in the second pass.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 853d427.

Rename uct_rc_verbs_iface_srq_prepost_recvs() to
uct_rc_verbs_iface_prepost_recvs_srq() and
uct_rc_verbs_iface_repost_empty_qps() to
uct_rc_verbs_iface_repost_empty_qps_no_srq(), and assert that the no-SRQ
receive helpers run without an SRQ.

Document the receive prepost and refill sizes without an SRQ, and why
the SRQ prepost on progress enable is a no-op without it.

Drop uct_rc_iface_config_check_fc_thresh() and clamp the soft threshold
in the no-SRQ FC window computation instead; uct_rc_init_fc_thresh() still
rejects invalid values.

Fix assignment alignment in the no-SRQ tests.

Signed-off-by: Guangguan Wang <guangguan.wang@linux.alibaba.com>
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@@ -284,15 +405,23 @@ static UCS_CLASS_INIT_FUNC(uct_rc_verbs_iface_t, uct_md_h tl_md,
&uct_rc_verbs_iface_ops, tl_md, worker, params,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since completion-driven refill fills the queue up to rx_max_wr, every QP that receives even one message pins its full receive queue (~1MB with the default 8256 seg size), so 256 eps hold ~270MB vs. a shared 4095-buffer SRQ today. can we lower the default or derive it from IB_RX_QUEUE_LEN?

(uct_rc_verbs_ep_prepost_recvs() preposts only 2 * rx_max_batch "to save memory on idle QPs", but the first receive completion calls uct_rc_verbs_ep_post_recv_batches_no_srq(iface, ep, iface->config.rx_max_wr), which keeps posting while rx_available >= batch. IB_RX_MAX_BUFS is -1 by default, so nothing bounds this.)

Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gap: contrib/ibmock only mocks EFA (UD/SRD), so the no-SRQ rc_verbs path — including the hardware-vs-software LAST_WQE split and rnr_fc_disabled, which depends on real RNR retry — runs only on real hardware. Worth confirming run_rc_verbs_nosrq_gtest was green on hardware before merge.

Residual coverage gap: no test covers a mixed peer configuration (one side SRQ, the other per-QP RQ) where the two FC windows differ, even though uct_rc_iface_t still assumes "number of recv buffers is the same on all peers".

Residual coverage gap: test_rc_flow_control runs in the new job with RX_MAX_WR=64 while overriding fc_wnd_size to 512; this relies on rnr_retry = 7 meaning infinite RNR retries, which is untested with a lower RC_RNR_RETRY_COUNT.

Comment on lines +350 to +352
rx_max_wr = 0;
rx_cq_len = ib_config->rx.queue_len;
init_attr.fc_max_wnd_size = ib_config->rx.queue_len;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pls always set rx_batch_limit, so it can be assigned unconditionally after the super init (same value as the base iface computes with SRQ):

Suggested change
rx_max_wr = 0;
rx_cq_len = ib_config->rx.queue_len;
init_attr.fc_max_wnd_size = ib_config->rx.queue_len;
rx_max_wr = 0;
rx_cq_len = ib_config->rx.queue_len;
rx_batch_limit = ucs_min(ib_config->rx.max_batch,
ib_config->rx.queue_len / 4);
init_attr.fc_max_wnd_size = ib_config->rx.queue_len;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 56b2364.

Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c Outdated
Comment on lines +420 to +423
/* A QP must be able to reach the repost threshold. */
if (!srq_enabled) {
self->super.super.config.rx_max_batch = rx_batch_limit;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

then the condition is not needed:

Suggested change
/* A QP must be able to reach the repost threshold. */
if (!srq_enabled) {
self->super.super.config.rx_max_batch = rx_batch_limit;
}
/* Without SRQ, a QP must be able to reach the repost threshold. */
self->super.super.config.rx_max_batch = rx_batch_limit;

assignment can also be moved above.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 56b2364.

&config->super.super, &init_attr);

self->config.rx_max_wr = rx_max_wr;
self->rx_cq_available = srq_enabled ? 0 : rx_cq_len;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, slight change from previous comment, pls report min(MAX_NUM_EPS, rx_cq_len / rx_max_wr) as max_num_eps without SRQ; otherwise UCP may select rc_verbs and then fail ep creation with EXCEEDS_LIMIT when RX_CQ_LEN is lowered.

Comment on lines +53 to +56
{"RX_MAX_WR", "128",
"Maximum number of receive WRs per QP when SRQ_ENABLE=n or the device\n"
"does not support SRQ.",
ucs_offsetof(uct_rc_verbs_iface_config_t, rx_max_wr), UCS_CONFIG_TYPE_UINT},

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe mention the memory cost here, since each active QP keeps its queue full:

Suggested change
{"RX_MAX_WR", "128",
"Maximum number of receive WRs per QP when SRQ_ENABLE=n or the device\n"
"does not support SRQ.",
ucs_offsetof(uct_rc_verbs_iface_config_t, rx_max_wr), UCS_CONFIG_TYPE_UINT},
{"RX_MAX_WR", "128",
"Maximum number of receive WRs per QP when SRQ_ENABLE=n or the device\n"
"does not support SRQ. Each QP which receives messages keeps up to\n"
"RX_MAX_WR receive buffers of SEG_SIZE bytes posted.",
ucs_offsetof(uct_rc_verbs_iface_config_t, rx_max_wr), UCS_CONFIG_TYPE_UINT},

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 56b2364.

Without an SRQ, report max_num_eps as the minimum of MAX_NUM_EPS and
RX_CQ_LEN / RX_MAX_WR, since each endpoint reserves RX_MAX_WR RX CQ
entries. Check the reported limit in rx_cq_num_eps.

Set the receive batch size in both SRQ modes and assign it
unconditionally after the base interface initialization.

Document the receive buffer memory of RX_MAX_WR.

Signed-off-by: Guangguan Wang <guangguan.wang@linux.alibaba.com>
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@@ -419,7 +418,12 @@ static void uct_ib_async_event_handler(int fd, ucs_event_set_types_t events,
case IBV_EVENT_SQ_DRAINED:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why assert and not ignore the event? if some provider does report LAST_WQE for a QP without an SRQ, a debug build aborts here, and a release build would schedule the rc_verbs cleanup before all receive WRs drained, leaking their descriptors.

uct_rc_verbs_iface_qp_cleanup_ctx_t *cleanup_ctx =
ucs_derived_of(rc_cleanup_ctx, uct_rc_verbs_iface_qp_cleanup_ctx_t);
uct_rc_verbs_iface_t *iface = ucs_derived_of(rc_cleanup_ctx->iface,
uct_rc_verbs_iface_t);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: rx_batch_limit/max_num_eps in the srq_enabled branch are exactly what uct_ib_iface_t/uct_base_iface_t already compute from the same config, so the overrides below are no-ops with SRQ; can we set rx_max_batch/max_num_eps only in the no-SRQ branch?

ucs_offsetof(uct_rc_verbs_iface_config_t, rx_max_wr), UCS_CONFIG_TYPE_UINT},

{"RX_CQ_LEN", "32768",
"Length of the receive completion queue when SRQ_ENABLE=n or the device\n"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New RX_MAX_WR config entry: with the defaults (128 WRs per QP, RC_MAX_NUM_EPS=256, SEG_SIZE=8256) an iface that refills every QP ends up holding ~32K receive buffers (~270MB), against ~34MB for the 4095-entry SRQ. can we scale the per-QP count with the endpoint count, or at least state the cost in the help text?

@svc-nvidia-pr-review

Copy link
Copy Markdown

Already raised in earlier rounds, still present in this revision (support the existing threads with +1, do not open new ones): PR size far above the 500 added-line limit; the software-LAST_WQE part (uct_ib_device_async_event_dispatch() export + test_ib_event.cc) is still separable.

Already raised in earlier rounds, still present: SRQ_ENABLE=try does not fall back when ibv_create_srq() itself fails (only when max_srq == 0).

Already raised in earlier rounds, still present: ucs_assert(init_attr->fc_max_wnd_size > 0) silently yields fc_wnd_size = 0 with assertions disabled.

Already raised in earlier rounds, still present: reposting on IBV_WC_REM_ABORT_ERR calls ibv_post_recv() on a QP already moved to ERROR, where a non-zero return is ucs_fatal().

Already raised in earlier rounds, still present: run_rc_verbs_nosrq_gtest excludes test_rc_no_srq/test_rc_srq_select; do_distributed_task's second argument is unused, so the 4→5 edits are cosmetic.

Already raised in earlier rounds, still present: UCX_RC_VERBS_RX_CQ_LEN=262144 in the new CI job implies a ~16MB RX CQ per iface.

Residual coverage gap: the new job's filter does not include test_ib_event*, so the has_srq() skip conditions added in test_ib_event.cc are not exercised by any CI job; they only matter for a manual run with UCX_RC_VERBS_SRQ_ENABLE=n.

Residual coverage gap: contrib/ibmock has no async-event path and mocks EFA only, so the hardware-vs-software LAST_WQE split and rnr_fc_disabled (real RNR retry) run on real hardware only. Worth confirming run_rc_verbs_nosrq_gtest was green on hardware before merge.

Residual coverage gap: no test covers a mixed peer configuration (one side SRQ, the other per-QP RQ) where the two FC windows differ, although uct_rc_iface_t still assumes "number of recv buffers is the same on all peers".

@svc-ucx

svc-ucx commented Nov 9, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests new on worker 3) · commit 56b23646

TL;DR: The build died on a UCS rcache assertion (rcache.c:405, !(region->flags & UCS_RCACHE_REGION_FLAG_PGTABLE)) during ucp_mem_unmap in tcp/test_ucp_rma.put_nonblocking/2 <...​/user_memh> — an rcache region's refcount hit 0 while it was still in the page table, i.e. the page-table reference was lost. This is a master-side regression in the registration-cache refcount/merge flow, unrelated to this PR's rc_verbs/SRQ change.

Full analysis

Summary: gtest aborted (core dumped, make: *** [Makefile:5014: test] Aborted) on tcp/test_ucp_rma.put_nonblocking/2 <tcp,cuda_copy,rocm_copy/user_memh> with an assertion failure inside ucs_mem_region_destroy_internal().

Root cause: From the backtrace:
test_ucp_memheap.cc:120 (ucp_mem_unmap) → ucp_mem_unmap() (ucp_mm.c:1298) → ucp_memh_cleanup() (ucp_mm.c:473) → ucp_memh_put_rcache() (ucp_mm.c:444) → ucs_rcache_region_put_unsafe() (rcache.inl:128) → ucs_mem_region_destroy_internal() (rcache.c:405).
ucs_rcache_region_put_unsafe() decrements the region refcount and, when it reaches 0, destroys the region immediately. The assert proves the region was still inserted in the page table at that moment: a registered, page-table-resident region must have refcount ≥ 2 ("Page-table + user", set at rcache.c:1108), so the user's single put should have left refcount 1. The page-table reference had therefore already been dropped without the UCS_RCACHE_REGION_FLAG_PGTABLE flag being cleared (that flag is only cleared in ucs_rcache_region_invalidate_internal(), rcache.c:501-510). Two code-level weaknesses make this reachable: (a) ucs_rcache_region_put_unsafe() decrements region->refcount non-atomically and without pgt_lock, whereas ucs_rcache_region_hold()/ucs_rcache_region_put_internal() use atomics — a lost update yields an off-by-one refcount; (b) the newly added adjacent-merge flow (ucs_rcache_check_adj_regions() / ucs_rcache_check_overlap_one(), rcache.c:756-891) releases page-table references for neighbour regions in a path that was not previously exercised. Nothing in the failure touches rc_verbs/SRQ — the test ran over tcp and crashed in generic UCP/UCS memory-registration code, so this is a master regression/flake, not a PR defect. (The 1-minute log gaps at 20:30:48/20:31:46 are the UCS fault handler resolving the backtrace, not a hang.)

Implicated commit: e536ef03 — "UCS/RCACHE: Adds capability for merging adjacent rcache regions (#11958)", william-gallagher-nv, 2026-11-04 (the only recent commit touching both src/ucs/memory/rcache.c and src/ucp/core/ucp_mm.c, merged 5 days before this build). The PR commit under test (56b2364, rc-verbs-no-srq) is not implicated.

File: src/ucs/memory/rcache.c:405 (assert) / src/ucs/memory/rcache.inl:121-131 (ucs_rcache_region_put_unsafe) / src/ucp/core/ucp_mm.c:444

Suggested fix:

  1. Make ucs_rcache_region_put_unsafe() consistent with the rest of the refcounting: use ucs_atomic_fsub32() instead of --region->refcount, and add ucs_assertv(region->refcount > 1 || !(region->flags & UCS_RCACHE_REGION_FLAG_PGTABLE), ...) before destroying, so the broken invariant is caught at the put site (with region/rcache name) instead of later in destroy_internal.
  2. Audit the adjacent-merge path from UCS/RCACHE: Adds capability for merging adjacent rcache regions #11958: in ucs_rcache_check_adj_regions()/ucs_rcache_check_overlap_one() verify that every neighbour region added to region_list is actually in the page table, that it cannot be added twice (it reuses region->tmp_list, also used for the GC list), and that exactly one page-table reference is released per region via ucs_rcache_region_invalidate_internal().
  3. Re-run to confirm this is not PR-related: ./gtest --gtest_filter='*test_ucp_rma.put_nonblocking*user_memh*' on master at e536ef03 and at its parent, with UCX_RCACHE_MERGE_ADJACENT=n vs =y (default is n, rcache.c:117). If it reproduces only with merge-adjacent enabled, gate/disable that flag in CI until fixed; if it reproduces on master with the default, open a blocking issue against UCS/RCACHE: Adds capability for merging adjacent rcache regions #11958 and retrigger this PR's CI.

Related: #11958 (implicated feature), PR under test #12066. No existing issue found for this assertion signature — worth filing one.

@tvegas1 tvegas1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

last round

Comment thread src/uct/ib/base/ib_device.c Resolved
Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c Resolved
Ignore a hardware LAST_WQE event on a QP without an SRQ instead of
asserting, since rc_verbs starts the cleanup of such a QP itself after
its receive completions are drained. Share the QP number assignment with
the other QP events.

Print the reported endpoint limit in the rc_verbs no-SRQ debug message,
since it is also capped by MAX_NUM_EPS.

Signed-off-by: Guangguan Wang <guangguan.wang@linux.alibaba.com>
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@@ -92,31 +152,49 @@ uct_rc_verbs_iface_poll_rx_common(uct_rc_verbs_iface_t *iface)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docs/OptimizationStyle.md asks to prefer setup-time selection over hot-path conditionals — can we add a uct_rc_verbs_iface_progress_no_srq() and select it in iface init through self->super.progress (like rc_mlx5 does with its progress_ll/progress_cyclic variants), instead of testing iface->srq on every RX poll? That would also let uct_rc_verbs_post_recv_common() lose its ep != NULL branches.

@@ -429,15 +573,50 @@ unsigned uct_rc_verbs_iface_post_recv_always(uct_rc_verbs_iface_t *iface, unsign
return 0;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: for srq_enabled these two assignments just restore what the base classes already computed — uct_ib_iface_t sets rx_max_batch to the same ucs_min(rx.max_batch, rx.queue_len / 4) and uct_base_iface_t sets max_num_eps from the config. Can we apply them only in the no-SRQ case and drop rx_batch_limit/max_num_eps from the SRQ branch?

@@ -50,6 +50,28 @@ static ucs_config_field_t uct_rc_verbs_iface_config_table[] = {
"a minimum between this value and the TX queue length. -1 means no limit.",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UCS_CONFIG_TYPE_ULUNITS also accepts auto, which arrives here as UCS_ULUNITS_AUTO and gets clamped to max_cqe exactly like inf — pls reject it or document it, otherwise RC_VERBS_RX_CQ_LEN=auto quietly allocates the largest CQ the device allows.

@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gap: test_rc_srq_select covers max_srq == 0 by patching the cached device attribute, but there is still no coverage for a device that reports max_srq != 0 while ibv_create_srq() fails — which is the case the earlier "try should fall back" finding is about. Everything else (CI filter excluding test_rc_no_srq/test_rc_srq_select, ibmock having no async-event path, rnr_fc_disabled needing real RNR retry, mixed SRQ/no-SRQ FC windows) is unchanged from previous rounds.

@tvegas1
tvegas1 enabled auto-merge (squash) November 10, 2026 13:43
@tvegas1
tvegas1 merged commit 75371a0 into openucx:master Nov 11, 2026
167 checks passed
@tvegas1

tvegas1 commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

@GuangguanWang, is the no-srq support complete with this PR? Do you have more related fixes?

@GuangguanWang

GuangguanWang commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor Author

@GuangguanWang, is the no-srq support complete with this PR? Do you have more related fixes?

@tvegas1 So far I haven't found any confirmed no-SRQ related issue.
Some CI failures are currently referenced to this PR, and I'm checking whether they are really related. I'll follow up with a fix if any of them turns out to be caused by this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants