Repository navigation
UCT/IB/RC/VERBS: Support QPs without SRQ - #12066
Conversation
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: Full analysisSummary: Build 138607 ("roce on worker 3") failed when gtest's watchdog aborted the run: Root cause: Timestamp analysis makes this unambiguous — the test starts at The backtrace names the hung operation precisely:
So the main thread is blocked joining a worker that never finished. The worker's only unbounded blocking construct is the per-iteration loop The branch under test ( Implicated commit: File: Suggested fix:
Related: PR #12066 (this build), and the rest of the series: #12061 ("UCT/IB/RC: Preparation for QPs without an SRQ"), #11933, #11938 (gtest coverage for rc_verbs QPs without SRQ), #12060.
|
597e636 to
653e4b5
Compare
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: gtest Root cause: Two-stage failure, both in the UMR (exported-mkey) path on a RoCE device:
Implicated commit: unknown — the failing code is the pre-existing UMR export path ( File: Suggested fix:
|
653e4b5 to
f641628
Compare
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: All three Root cause: This is a hang, not a timeout. Timeline evidence from the
The exit/cancel is a symptom; both The PR under test (#12059, "UCT/IB/RC: Remove an endpoint from ep_list in uct_rc_ep_t cleanup") adds
Two concrete hazards follow, both consistent with a silent infinite spin:
Implicated commit: File: Suggested fix:
Related: PR #12059 (the change being tested); PR #12066 (this build). No existing issue found for this hang signature.
|
tvegas1
left a comment
There was a problem hiding this comment.
need to fix conflict with master.
f641628 to
b487459
Compare
|
🤖 Starting review — findings will be posted here when done. |
| { | ||
| uct_rc_verbs_iface_qp_cleanup_ctx_t *cleanup_ctx; | ||
|
|
||
| ucs_list_for_each(cleanup_ctx, &iface->super.qp_gc_list, super.list) { |
There was a problem hiding this comment.
this scans the whole gc list for every flush CQE, so destroying N endpoints costs O(N * rx_max_wr) list steps. can we reach the cleanup ctx directly, e.g. keep it addressable by qp_num until it is drained?
|
PR-level (Size): this PR adds well over 500 lines (≈1020 in Residual coverage gap: all new Notes on things I checked and found OK:
|
|
🤖 CI Triage Agent — TL;DR: The Coverity stage failed with 3 new NEGATIVE_RETURNS defects in Full analysisSummary: Coverity analysis job Root cause: Commit b487459 ("GTEST/UCT/IB: Extend the test coverage of QPs without SRQ") added the helper
Note Implicated commit: [REDACTED:Hex High Entropy String] — Guangguan Wang, "GTEST/UCT/IB: Extend the test coverage of QPs without SRQ" File: test/gtest/uct/ib/test_rc.cc:772-784 (helper returning -1); defect sites at :1183-1196 and :1224-1228 Suggested fix: Make the negative path unreachable in a way Coverity can model, by using the throwing first_ep = first_waiting_ep_index();
if (first_ep < 0) {
UCS_TEST_ABORT("no endpoint is waiting for receive buffers");
}(and likewise for Alternative: keep one assertion helper, e.g. have Related: PR #12066 (branch
|
b487459 to
6b908d7
Compare
|
🤖 Starting review — findings will be posted here when done. |
6b908d7 to
908db0a
Compare
|
Residual coverage gap (unchanged from the previous round): every new |
|
🤖 Starting review — findings will be posted here when done. |
| uct_rc_verbs_iface_poll_rx(uct_rc_verbs_iface_t *iface) | ||
| { | ||
| unsigned i, num_wcs = iface->super.super.config.rx_max_poll; | ||
| uint8_t srq_enabled = iface->super.config.srq_enabled; |
There was a problem hiding this comment.
can we select the mode at setup time instead of branching per CQE? super.progress is already a function pointer, so two wrappers passing srq_enabled as a constant to an ALWAYS_INLINE poll_rx() (like uct_rc_mlx5_iface_progress_cyclic/_ll/_tm do with flags) would keep the SRQ RX path free of the new conditionals - docs/OptimizationStyle.md: "Prefer setup-time selection over hot-path conditionals".
|
🤖 CI Triage Agent — TL;DR: The new Full analysisSummary: AddressSanitizer/roce worker 0 → Root cause: A hang, not slowness — the log has one 11-minute gap with zero output (12:53:23 The no-SRQ design relies on an invariant that the new reposting logic does not guarantee:
Implicated commit: 5ad8c71 "UCT/IB/RC: Queue endpoints pending receive reposting" (Guangguan Wang), part of the series 6d7565b / 79057a2 / 849e2b3 / 0572bd0 / 0e7c11a in PR #12066. File: src/uct/ib/rc/verbs/rc_verbs_iface.c:539-560 (and src/uct/ib/rc/verbs/rc_verbs_impl.h:39-70, 144-161; FC-window assumption at rc_verbs_iface.c:336-349) Suggested fix:
Related: PR #12066 (this PR, "UCT/IB/RC/VERBS: Support QPs without SRQ"), earlier attempt #11933, and #12060 ("Release the descriptor of a failed receive completion") which touched the same no-SRQ receive-completion path. |
| echo "==== Running rc_verbs gtest with SRQ disabled ====" | ||
| $TIMEOUT env \ | ||
| UCX_RC_VERBS_SRQ_ENABLE=n \ | ||
| UCX_RC_VERBS_MAX_NUM_EPS=inf \ |
There was a problem hiding this comment.
The CI OOM looks self-inflicted: run_rc_verbs_nosrq_gtest sets UCX_RC_VERBS_MAX_NUM_EPS=inf, overriding the RC default of 256, so every rc_verbs iface created by the gtests gets an RX CQ of ~max_cqe entries (≈4M, ~256MB pinned on ConnectX). Why is inf needed? If some tests create more than 256 eps, could we use a finite value (e.g. 1024, as test_ucp_wireup does with UCX_RC_MAX_NUM_EPS)? Also, since inf silently maps to the largest CQ the device supports, maybe cap it or document the memory cost in the RX_MAX_WR help text.
There was a problem hiding this comment.
Replaced in de689c3 by MAX_NUM_EPS=4096 with RX_MAX_WR=48, i.e. an RX
CQ of at most 196608 entries. I used 4096 rather than 1024 to leave headroom
for stress_connect2, which connects up to 1000 endpoints while the RX CQ
capacity of destroyed endpoints returns only after their QPs drain.
I left the RX_MAX_WR help as is, since it already says that
MAX_NUM_EPS=inf uses the largest RX CQ the device supports.
| return uct_rc_iface_init_rx(rc_iface, config, &iface->srq); | ||
| } | ||
|
|
||
| iface->srq = NULL; |
There was a problem hiding this comment.
might bwe able to moving =NULL inside uct_rc_iface_init_rx()?
There was a problem hiding this comment.
I kept it in rc_verbs: uct_rc_iface_init_rx() always creates the SRQ and is
shared with rc_mlx5, and rc_verbs does not call it without an SRQ, so the
assignment there would not run on this path.
|
|
||
|
|
||
| /* Force rc_verbs to create QPs without an SRQ. */ | ||
| class test_rc_no_srq : public test_rc { |
There was a problem hiding this comment.
New tests could be trimmed:
rx_cq_no_spaceoverlapsrx_cq_num_eps(both checkEXCEEDS_LIMITand the quota return after drain), and it setsrx_cq_availabledirectlyam_and_qp_drainandflush_cancelare covered bylast_wqe_after_rx_drain/last_wqe_already_drained, and the reposting part byuct_p2p_am_testin the new no-SRQ CI jobfc_wnd_sizeandrx_max_batch_limitcould be a single config test- in
test_rc_srq_select,try_uses_srqduplicatesyes_uses_srq, and"RC_VERBS_SRQ_ENABLE=try"is the default no_rx_bufs_waiting_order/no_rx_bufs_waiting_ep_destroyand their helpers test the queue order rather than behavior;no_rx_bufs_many_epsalready checks that every QP recovers (and they would go away with the repost suggestion above)- the hand-written deadline loops could use
wait_for_cond()/wait_for_value()
There was a problem hiding this comment.
Done in 576f311, with two differences:
try_uses_srqandyes_uses_srqare merged into oneuses_srqtest
rather than droppingtry: setting it explicitly prevents an SRQ_ENABLE
environment setting from overriding it, and lets devices without SRQ skip
instead of failing iface creation; the test then reopens withy.- The waiting tests are kept, adapted to the iface flag. They check what
no_rx_bufs_many_epsdoes not: empty QPs are served before partially
filled ones, and destroying the QP at the retry position, or all empty
QPs, leaves no stale retry.
|
🤖 Starting review — findings will be posted here when done. |
| * enabled, which is not done by opening an interface. */ | ||
| EXPECT_EQ(UCS_OK, try_rx_max_wr("1")); | ||
|
|
||
| snprintf(str, sizeof(str), "%d", |
There was a problem hiding this comment.
[redacted: contained suspected secret] this opens an iface with RX_MAX_WR=max_qp_wr while MAX_NUM_EPS is the default 256, so rx_cq_len = min(256, max_cqe/[REDACTED]) * [REDACTED] ≈ 4M CQEs (~260MB of pinned memory on mlx5) — slow and likely to fail on the memlock limit. pls add "MAX_NUM_EPS=1" to the test config; the fixture still works because rx_cq_len is max'ed with IB_RX_QUEUE_LEN.
Rationale: the comment above the check only reasons about receive descriptors ("receive WRs are posted when progress is enabled"), but the RX CQ is sized and allocated at uct_iface_open() time regardless of progress.
There was a problem hiding this comment.
+1, with RX_MAX_WR=max_qp_wr and the default 256 eps the RX CQ is ~127 * 32K ≈ 4M entries (~260MB pinned) just to check the config; MAX_NUM_EPS=1 limits it to a single receive queue.
There was a problem hiding this comment.
Done in 2e7070e. With the separate CQ setting, the upper-bound test sets
RX_CQ_LEN to the tested receive depth instead of using MAX_NUM_EPS=1.
| uct_rc_verbs_iface_poll_rx(uct_rc_verbs_iface_t *iface) | ||
| { | ||
| unsigned i, num_wcs = iface->super.super.config.rx_max_poll; | ||
| uint8_t srq_disabled = (iface->srq == NULL); |
There was a problem hiding this comment.
soft suggestion: maybe select a dedicated progress function for the no-SRQ mode (like rc_mlx5 does for srq topology) and pass the flag as a compile-time argument to uct_rc_verbs_iface_poll_rx()? it would keep the existing SRQ path free of the new per-CQE branches.
|
PR size (>500 added lines, ~900 in
Residual coverage gap (unchanged from earlier rounds): the new no-SRQ job excludes |
| * enabled, which is not done by opening an interface. */ | ||
| EXPECT_EQ(UCS_OK, try_rx_max_wr("1")); | ||
|
|
||
| snprintf(str, sizeof(str), "%d", |
There was a problem hiding this comment.
+1, with RX_MAX_WR=max_qp_wr and the default 256 eps the RX CQ is ~127 * 32K ≈ 4M entries (~260MB pinned) just to check the config; MAX_NUM_EPS=1 limits it to a single receive queue.
| size_t max_inline; | ||
| size_t max_send_sge; | ||
| unsigned tx_max_wr; | ||
| uint16_t rx_max_wr; /* receive WRs per QP without SRQ, |
There was a problem hiding this comment.
would it make sense to move the new uint16_t to unsigned like above for tx_max_wr?
| ucs_max(rx_max_wr / 4, 1)); | ||
| /* Room for the receive WRs of all endpoints; the 64-bit endpoint | ||
| * count is capped by the largest CQ before multiplying. */ | ||
| num_eps = ucs_min(ib_config->super.max_num_eps, |
There was a problem hiding this comment.
What do you think about a dedicated RC_VERBS_RX_CQ_LEN for the no-SRQ case, instead of deriving the RX CQ size from MAX_NUM_EPS?
Today MAX_NUM_EPS is only a hint reported to UCP; with this change it sizes the CQ and becomes a hard endpoint limit, and RX_QUEUE_LEN ends up only as a CQ minimum. Something like:
{"RX_MAX_WR", "128",
"Number of receive WRs posted to each QP when SRQ_ENABLE=n or the device\n"
"does not support SRQ.",
ucs_offsetof(uct_rc_verbs_iface_config_t, rx_max_wr), UCS_CONFIG_TYPE_UINT},
{"RX_CQ_LEN", "32768",
"Length of the receive completion queue when SRQ_ENABLE=n or the device\n"
"does not support SRQ. Limits the number of endpoints to RX_CQ_LEN/RX_MAX_WR.\n"
"\"inf\" uses the maximum CQ length reported by the device.",
ucs_offsetof(uct_rc_verbs_iface_config_t, rx_cq_len), UCS_CONFIG_TYPE_ULUNITS},rx_cq_len = ucs_min(config->rx_cq_len, IBV_DEV_ATTR(&ib_md->dev, max_cqe));
if (rx_cq_len < rx_max_wr) {
/* error: RX_CQ_LEN must be >= RX_MAX_WR */
}
/* endpoint limit = rx_cq_len / rx_max_wr */The defaults RX_MAX_WR=128 and RX_CQ_LEN=32768 keep today's 256-endpoint limit and ~4 MB CQ. It keeps MAX_NUM_EPS a hint, gives one explicit knob for the CQ memory (~64 B per CQE, so 128K entries ≈ 8-16 MB covers 1024 endpoints), and the endpoint creation error could point users to RC_VERBS_RX_CQ_LEN.
As a follow-up (separate PR), it would be nice to remove the hard endpoint limit, either by:
- growing the RX CQ with
ibv_resize_cq()when a new endpoint does not fit, or - allowing more QPs than
RX_CQ_LEN/RX_MAX_WRwhile never preposting more thanRX_CQ_LENreceives in total across all QPs (QPs beyond the budget would get fewer receives and rely on RNR retry/FC).
There was a problem hiding this comment.
Done in 2e7070e. Removing the hard endpoint limit remain follow-up work for a separate PR.
| } | ||
| UCT_IB_IFACE_VERBS_COMPLETION_FATAL("receive", &iface->super.super, i, wc); | ||
| uct_rc_verbs_iface_rx_error(iface, &wc[i], desc); | ||
| } else { |
There was a problem hiding this comment.
can we remove the else() block/indent as error seems fatal to improve readability
There was a problem hiding this comment.
The error helper is not always fatal: REM_ABORT_ERR and WR_FLUSH_ERR
release the descriptor and return. These completions must skip AM handling
but still reach the receive accounting below, so I kept the else block.
There was a problem hiding this comment.
Fair point, thinking more, can we rework the code like below? The no-SRQ accounting to out:, like the SRQ refill which would simplify the function. This removes the previously requested helper funtion.
if (ucs_unlikely(wc[i].status != IBV_WC_SUCCESS)) {
if ((wc[i].status != IBV_WC_REM_ABORT_ERR) &&
(wc[i].status != IBV_WC_WR_FLUSH_ERR)) {
UCT_IB_IFACE_VERBS_COMPLETION_FATAL("receive",
&iface->super.super, i,
wc);
}
/* A failed receive never delivers an active message. */
ucs_mpool_put_inline(desc);
continue;
}
/* hdr, valgrind, log, handle_am */
}
...
out:
if (<srq is enabled>) {
iface->super.rx.srq.available += num_wcs;
uct_rc_verbs_iface_post_recv_common(iface, NULL, 0);
} else {
/* Give empty QPs returned buffers before refilling active QPs. */
if (ucs_unlikely(iface->rx_repost_needed)) {
uct_rc_verbs_iface_repost_recv(iface);
}
for (i = 0; i < num_wcs; i++) {
/* Do not repost flushed WRs */
uct_rc_verbs_iface_rx_complete_no_srq(iface, wc[i].qp_num,
wc[i].status !=
IBV_WC_WR_FLUSH_ERR);
}
}This would:
- bring the loop back to master's shape, without the helper and the
else - keep all the no-SRQ receive handling in one place, so the entry check goes away
- remove the per-CQE branch from the SRQ path, which also covers the bot's comments about that
- fix the
completion[0]index from the bot's comment
At out:, before the accounting loop, repost_recv() sees the same rx_available values as at entry today, so the set of QPs it serves doesn't change; it can also use buffers released by AM callbacks during this poll.
An ep destroyed from an AM callback is still accounted correctly: its rx_remaining includes the not yet accounted CQEs of this poll, and they reach gc_drain_cqe() in the second pass.
Rename uct_rc_verbs_iface_srq_prepost_recvs() to uct_rc_verbs_iface_prepost_recvs_srq() and uct_rc_verbs_iface_repost_empty_qps() to uct_rc_verbs_iface_repost_empty_qps_no_srq(), and assert that the no-SRQ receive helpers run without an SRQ. Document the receive prepost and refill sizes without an SRQ, and why the SRQ prepost on progress enable is a no-op without it. Drop uct_rc_iface_config_check_fc_thresh() and clamp the soft threshold in the no-SRQ FC window computation instead; uct_rc_init_fc_thresh() still rejects invalid values. Fix assignment alignment in the no-SRQ tests. Signed-off-by: Guangguan Wang <guangguan.wang@linux.alibaba.com>
6b75d7b to
86a943a
Compare
|
🤖 Starting review — findings will be posted here when done. |
| @@ -284,15 +405,23 @@ static UCS_CLASS_INIT_FUNC(uct_rc_verbs_iface_t, uct_md_h tl_md, | |||
| &uct_rc_verbs_iface_ops, tl_md, worker, params, | |||
There was a problem hiding this comment.
since completion-driven refill fills the queue up to rx_max_wr, every QP that receives even one message pins its full receive queue (~1MB with the default 8256 seg size), so 256 eps hold ~270MB vs. a shared 4095-buffer SRQ today. can we lower the default or derive it from IB_RX_QUEUE_LEN?
(uct_rc_verbs_ep_prepost_recvs() preposts only 2 * rx_max_batch "to save memory on idle QPs", but the first receive completion calls uct_rc_verbs_ep_post_recv_batches_no_srq(iface, ep, iface->config.rx_max_wr), which keeps posting while rx_available >= batch. IB_RX_MAX_BUFS is -1 by default, so nothing bounds this.)
|
Residual coverage gap: Residual coverage gap: no test covers a mixed peer configuration (one side SRQ, the other per-QP RQ) where the two FC windows differ, even though Residual coverage gap: |
| rx_max_wr = 0; | ||
| rx_cq_len = ib_config->rx.queue_len; | ||
| init_attr.fc_max_wnd_size = ib_config->rx.queue_len; |
There was a problem hiding this comment.
pls always set rx_batch_limit, so it can be assigned unconditionally after the super init (same value as the base iface computes with SRQ):
| rx_max_wr = 0; | |
| rx_cq_len = ib_config->rx.queue_len; | |
| init_attr.fc_max_wnd_size = ib_config->rx.queue_len; | |
| rx_max_wr = 0; | |
| rx_cq_len = ib_config->rx.queue_len; | |
| rx_batch_limit = ucs_min(ib_config->rx.max_batch, | |
| ib_config->rx.queue_len / 4); | |
| init_attr.fc_max_wnd_size = ib_config->rx.queue_len; |
| /* A QP must be able to reach the repost threshold. */ | ||
| if (!srq_enabled) { | ||
| self->super.super.config.rx_max_batch = rx_batch_limit; | ||
| } |
There was a problem hiding this comment.
then the condition is not needed:
| /* A QP must be able to reach the repost threshold. */ | |
| if (!srq_enabled) { | |
| self->super.super.config.rx_max_batch = rx_batch_limit; | |
| } | |
| /* Without SRQ, a QP must be able to reach the repost threshold. */ | |
| self->super.super.config.rx_max_batch = rx_batch_limit; |
assignment can also be moved above.
| &config->super.super, &init_attr); | ||
|
|
||
| self->config.rx_max_wr = rx_max_wr; | ||
| self->rx_cq_available = srq_enabled ? 0 : rx_cq_len; |
There was a problem hiding this comment.
+1, slight change from previous comment, pls report min(MAX_NUM_EPS, rx_cq_len / rx_max_wr) as max_num_eps without SRQ; otherwise UCP may select rc_verbs and then fail ep creation with EXCEEDS_LIMIT when RX_CQ_LEN is lowered.
| {"RX_MAX_WR", "128", | ||
| "Maximum number of receive WRs per QP when SRQ_ENABLE=n or the device\n" | ||
| "does not support SRQ.", | ||
| ucs_offsetof(uct_rc_verbs_iface_config_t, rx_max_wr), UCS_CONFIG_TYPE_UINT}, |
There was a problem hiding this comment.
maybe mention the memory cost here, since each active QP keeps its queue full:
| {"RX_MAX_WR", "128", | |
| "Maximum number of receive WRs per QP when SRQ_ENABLE=n or the device\n" | |
| "does not support SRQ.", | |
| ucs_offsetof(uct_rc_verbs_iface_config_t, rx_max_wr), UCS_CONFIG_TYPE_UINT}, | |
| {"RX_MAX_WR", "128", | |
| "Maximum number of receive WRs per QP when SRQ_ENABLE=n or the device\n" | |
| "does not support SRQ. Each QP which receives messages keeps up to\n" | |
| "RX_MAX_WR receive buffers of SEG_SIZE bytes posted.", | |
| ucs_offsetof(uct_rc_verbs_iface_config_t, rx_max_wr), UCS_CONFIG_TYPE_UINT}, |
Without an SRQ, report max_num_eps as the minimum of MAX_NUM_EPS and RX_CQ_LEN / RX_MAX_WR, since each endpoint reserves RX_MAX_WR RX CQ entries. Check the reported limit in rx_cq_num_eps. Set the receive batch size in both SRQ modes and assign it unconditionally after the base interface initialization. Document the receive buffer memory of RX_MAX_WR. Signed-off-by: Guangguan Wang <guangguan.wang@linux.alibaba.com>
|
🤖 Starting review — findings will be posted here when done. |
| @@ -419,7 +418,12 @@ static void uct_ib_async_event_handler(int fd, ucs_event_set_types_t events, | |||
| case IBV_EVENT_SQ_DRAINED: | |||
There was a problem hiding this comment.
why assert and not ignore the event? if some provider does report LAST_WQE for a QP without an SRQ, a debug build aborts here, and a release build would schedule the rc_verbs cleanup before all receive WRs drained, leaking their descriptors.
| uct_rc_verbs_iface_qp_cleanup_ctx_t *cleanup_ctx = | ||
| ucs_derived_of(rc_cleanup_ctx, uct_rc_verbs_iface_qp_cleanup_ctx_t); | ||
| uct_rc_verbs_iface_t *iface = ucs_derived_of(rc_cleanup_ctx->iface, | ||
| uct_rc_verbs_iface_t); |
There was a problem hiding this comment.
minor: rx_batch_limit/max_num_eps in the srq_enabled branch are exactly what uct_ib_iface_t/uct_base_iface_t already compute from the same config, so the overrides below are no-ops with SRQ; can we set rx_max_batch/max_num_eps only in the no-SRQ branch?
| ucs_offsetof(uct_rc_verbs_iface_config_t, rx_max_wr), UCS_CONFIG_TYPE_UINT}, | ||
|
|
||
| {"RX_CQ_LEN", "32768", | ||
| "Length of the receive completion queue when SRQ_ENABLE=n or the device\n" |
There was a problem hiding this comment.
New RX_MAX_WR config entry: with the defaults (128 WRs per QP, RC_MAX_NUM_EPS=256, SEG_SIZE=8256) an iface that refills every QP ends up holding ~32K receive buffers (~270MB), against ~34MB for the 4095-entry SRQ. can we scale the per-QP count with the endpoint count, or at least state the cost in the help text?
|
Already raised in earlier rounds, still present in this revision (support the existing threads with Already raised in earlier rounds, still present: Already raised in earlier rounds, still present: Already raised in earlier rounds, still present: reposting on Already raised in earlier rounds, still present: Already raised in earlier rounds, still present: Residual coverage gap: the new job's filter does not include Residual coverage gap: Residual coverage gap: no test covers a mixed peer configuration (one side SRQ, the other per-QP RQ) where the two FC windows differ, although |
|
🤖 CI Triage Agent — TL;DR: The build died on a UCS rcache assertion ( Full analysisSummary: gtest aborted (core dumped, Root cause: From the backtrace: Implicated commit: File: Suggested fix:
Related: #11958 (implicated feature), PR under test #12066. No existing issue found for this assertion signature — worth filing one. |
Ignore a hardware LAST_WQE event on a QP without an SRQ instead of asserting, since rc_verbs starts the cleanup of such a QP itself after its receive completions are drained. Share the QP number assignment with the other QP events. Print the reported endpoint limit in the rc_verbs no-SRQ debug message, since it is also capped by MAX_NUM_EPS. Signed-off-by: Guangguan Wang <guangguan.wang@linux.alibaba.com>
|
🤖 Starting review — findings will be posted here when done. |
| @@ -92,31 +152,49 @@ uct_rc_verbs_iface_poll_rx_common(uct_rc_verbs_iface_t *iface) | |||
|
|
|||
There was a problem hiding this comment.
docs/OptimizationStyle.md asks to prefer setup-time selection over hot-path conditionals — can we add a uct_rc_verbs_iface_progress_no_srq() and select it in iface init through self->super.progress (like rc_mlx5 does with its progress_ll/progress_cyclic variants), instead of testing iface->srq on every RX poll? That would also let uct_rc_verbs_post_recv_common() lose its ep != NULL branches.
| @@ -429,15 +573,50 @@ unsigned uct_rc_verbs_iface_post_recv_always(uct_rc_verbs_iface_t *iface, unsign | |||
| return 0; | |||
There was a problem hiding this comment.
minor: for srq_enabled these two assignments just restore what the base classes already computed — uct_ib_iface_t sets rx_max_batch to the same ucs_min(rx.max_batch, rx.queue_len / 4) and uct_base_iface_t sets max_num_eps from the config. Can we apply them only in the no-SRQ case and drop rx_batch_limit/max_num_eps from the SRQ branch?
| @@ -50,6 +50,28 @@ static ucs_config_field_t uct_rc_verbs_iface_config_table[] = { | |||
| "a minimum between this value and the TX queue length. -1 means no limit.", | |||
There was a problem hiding this comment.
UCS_CONFIG_TYPE_ULUNITS also accepts auto, which arrives here as UCS_ULUNITS_AUTO and gets clamped to max_cqe exactly like inf — pls reject it or document it, otherwise RC_VERBS_RX_CQ_LEN=auto quietly allocates the largest CQ the device allows.
|
Residual coverage gap: |
|
@GuangguanWang, is the no-srq support complete with this PR? Do you have more related fixes? |
@tvegas1 So far I haven't found any confirmed no-SRQ related issue. |
What?
Allow rc_verbs to run on devices without SRQ support by creating each QP
with its own receive queue. RC_VERBS_SRQ_ENABLE (y|try|n, default try)
selects whether to use an SRQ.
Why?
rc_verbs currently rejects devices whose max_srq is zero, so it cannot
run on devices which only support per-QP receive queues.
How?
Other transports continue passing zero. uct_ib_device_has_srq() checks
device support. The RC flow-control window limit is supplied explicitly
by rc_verbs and the mlx5 common interface constructor.
back to per-QP receive queues. n always selects per-QP receive queues.
y fails with UCS_ERR_UNSUPPORTED when SRQ support is absent.
without SRQ (default 128). Values outside
1..min(max_qp_wr, UINT16_MAX) are rejected.
QPs initially prepost up to min(2 * batch, RX_MAX_WR) receive WRs to
save memory on idle QPs. Receive completions refill the queue in batches
until fewer than one batch of slots remain or receive buffers are
unavailable. Preposting and completion-driven refill share the same
batch-posting helper. Initial posting may be partial when buffers are
unavailable.
max(min(RX_MAX_BATCH, RX_MAX_WR / 4), 1), independent of RX_QUEUE_LEN.
The FC window is capped at
max(floor((RX_MAX_WR - batch + 1) / (1 + FC_SOFT_THRESH)), 1)
to reserve receive capacity for sends outstanding across soft grants.
The soft threshold is clamped at zero for this calculation and validated
afterwards by the existing RC FC threshold check.
The default RX settings remain 128/32768, giving an FC window of 75
with the default batch size and soft threshold.
Partial preposting does not reduce CQ reservations or the FC limit;
a fresh QP's first burst may still encounter RNR. RNR is also possible
when flow control is disabled.
With SRQ, the batch size remains as initialized by the IB base interface.
descriptors for REM_ABORT_ERR and WR_FLUSH_ERR completions. Other receive
errors remain fatal. Without SRQ, progress then retries empty QPs before
a second pass performs per-QP receive accounting and completion-driven
refill. Flushed WRs are not reposted.
posting, so posting failure on an empty QP sets an interface flag.
Retry scans empty QPs, stops when the buffer pool cannot serve a QP,
and rotates the endpoint list to resume there. Cancelled or failed
endpoints are not refilled.
32768). It is capped by the device's max_cqe; inf uses that device limit.
A CQ smaller than RX_MAX_WR is rejected. Each QP reserves RX_MAX_WR
entries, including QPs pending destruction. Endpoint creation fails
with UCS_ERR_EXCEEDS_LIMIT when insufficient capacity remains.
Capacity is returned after QP draining and destruction.
Diagnostics report the remaining reservation capacity rather than
the provider-rounded CQ size. MAX_NUM_EPS no longer determines CQ
allocation; without SRQ, max_num_eps is reported as
min(MAX_NUM_EPS, RX_CQ_LEN / RX_MAX_WR).
uct_rc_iface_qp_cleanup_ready() signals cleanup when all posted receive
WRs have been accounted, or immediately when none remain at destruction.
This reuses the existing async cleanup path, including device-fatal
handling. Hardware LAST_WQE events on QPs without SRQ are ignored.
Receive accounting for destroyed endpoints is handled outside the inline
RX path.
Tests
batch and FC limits, buffer exhaustion and recovery, empty-QP retry
priority, cancellation, failure and endpoint destruction.
values. rx_prepost_grows and rx_prepost_grows_batch_one check
two-batch initial posting and completion-driven refill for batch sizes
16 and 1, while an idle QP retains its initial depth.
soft thresholds. fc_soft_thresh_invalid checks rejection of invalid
thresholds, including negative values and infinity. Receive
configuration checks also verify that fc_wnd_size + fc_soft_thresh - 1
does not exceed RX_MAX_WR - batch + 1.
preserves its QP number, and checks descriptor release, receive-slot
accounting and absence of reposting.
QPs, rejection of a fifth endpoint, and capacity recovery after draining
and destruction. Additional cases cover invalid CQ lengths, finite and
inf device-limit truncation, and RX_MAX_WR boundaries.
and endpoint destruction from an AM callback before receive accounting.
failure without SRQ support, and ignoring the no-SRQ CQ setting with SRQ.
dispatch before and after subscribing, duplicate dispatch, and
unregistering a pending callback.
with UCX_RC_VERBS_SRQ_ENABLE=n, UCX_RC_VERBS_RX_CQ_LEN=262144 and
UCX_RC_VERBS_RX_MAX_WR=64. Dedicated no-SRQ and SRQ-selection suites
remain covered by the regular gtest run.