Skip to content

UCT/API: Add UCP-UCT APIs for endpoint path bandwidth monitoring - #12173

Merged
gleon99 merged 10 commits into
openucx:masterfrom
ZhenlongMa:dlb_api
Nov 10, 2026
Merged

gleon99 merged 10 commits into
openucx:masterfrom
ZhenlongMa:dlb_api

Conversation

@ZhenlongMa

@ZhenlongMa ZhenlongMa commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

What?

Add the APIs required for endpoint-specific path bandwidth monitoring.

Why?

Device-level speed information does not describe the performance of each
endpoint path. Consumers need endpoint-specific notification and query APIs
to obtain the affected path's bandwidth.

How?

  • Add UCT_EP_EVENT_PATH_SPEED_CHANGE and an endpoint-aware callback delivered
    in serialized worker-progress context.
  • Extend uct_ep_query() with UCT_EP_ATTR_FIELD_BANDWIDTH to return the latest
    available cached estimate in bytes per second, without blocking or requesting
    an update.
  • Define callback lifetimes, bandwidth-field semantics and query error behavior.
  • Report unsupported endpoint-event registration through UCS_ERR_UNSUPPORTED.

This PR defines the API contract only. Transport implementation and event
handling are outside its scope.

@ZhenlongMa
ZhenlongMa marked this pull request as ready for review October 26, 2026 15:13
@ZhenlongMa
ZhenlongMa marked this pull request as draft October 26, 2026 15:13
@ZhenlongMa
ZhenlongMa marked this pull request as ready for review October 26, 2026 15:15
@ZhenlongMa ZhenlongMa changed the title UCP/CORE: Add APIs for endpoint path bandwidth monitoring UCT/API: Add APIs for endpoint path bandwidth monitoring Oct 26, 2026
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@evgeny-leksikov evgeny-leksikov left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pls del everything except uct/api from this PR

Comment thread src/uct/ib/mlx5/rc/rc_mlx5_ext.h Resolved
Comment thread src/uct/ib/mlx5/rc/rc_mlx5_ext.h Resolved
Comment thread src/uct/ib/mlx5/rc/rc_mlx5_ext.h Resolved
Comment thread src/uct/base/uct_iface.c Resolved
Comment thread src/uct/api/v2/uct_v2.h Resolved
Comment thread src/uct/ib/mlx5/rc/rc_mlx5_ext.h Resolved
Comment thread src/uct/api/v2/uct_v2.h Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

This PR only adds API surface: uct_iface_handle_ep_event() has no caller, UCT_EVENT_PATH_SPEED_CHANGE/UCT_IFACE_FLAG_EVENT_PATH_SPEED_CHANGE are not set by any transport (and UCP's async handler ignores them), and UCT_EP_ATTR_FIELD_BANDWIDTH/UCT_IFACE_FLAG_V2_EP_QUERY_BW are not filled by any ep_query. Pls split this into per-feature PRs where each adds the API together with its transport implementation, UCP user, and a gtest (e.g. test_rc.cc already covers uct_ep_query with UCT_EP_ATTR_FIELD_TX_TOKEN).

Residual coverage gaps: no gtest exercises uct_iface_set_async_ep_event_handler (including its UCS_ERR_INVALID_PARAM path), the new event bits, or UCT_EP_ATTR_FIELD_BANDWIDTH; the mlx5 provider path cannot be covered at all without the implementation (and would need real mlx5 HW or contrib/ibmock/).

Signed-off-by: Zhenlong Ma <zhenlongm@nvidia.com>
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/api/uct.h Resolved
Comment thread src/uct/api/v2/uct_v2.h Outdated
* @ingroup UCT_RESOURCE
* @brief Endpoint-aware asynchronous event callback.
*
* The transport invokes this callback synchronously from its asynchronous

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UCT_EVENT_SPEED_CHANGE is not delivered from async context today — uct_ib_iface_port_speed_change_progress() defers it to the worker progress queue, and UCP still takes UCS_ASYNC_BLOCK around it. Pls document the context the user can actually rely on, since that decides whether the callback needs the async lock.

Comment thread src/uct/api/v2/uct_v2.h Outdated
* for the duration of the callback.
* @param [in] events Event flags from @ref uct_iface_event_types.
*/
typedef void (*uct_async_ep_event_cb_t)(void *arg, uct_ep_h ep,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UCT already registers an async event handler via uct_iface_params_t::async_event_cb/async_event_arg. Why not add async_ep_event_cb/arg fields there instead of a second setter, so a transport has one handler slot rather than two with per-event-bit routing? This currently forces two handler slots and a per-event-bit routing decision in every transport and in UCP.

Comment thread src/uct/api/v2/uct_v2.h Outdated
* Latest completed endpoint bandwidth sample, in bytes per second.
* Valid when @ref UCT_EP_ATTR_FIELD_BANDWIDTH is set in @ref field_mask
* and the interface advertises @ref UCT_IFACE_FLAG_V2_EP_QUERY_BW.
* A transport may return @ref UCS_ERR_NO_ELEM before the first sample or

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uct_ep_query returns a single status for all requested fields, so this makes a combined query (e.g. BANDWIDTH | TX_TOKEN) fail entirely when the sample is not ready. "may return" also leaves no portable handling. Pls make it definite — which status is returned, and whether the other requested fields are still filled.

Comment thread src/uct/api/v2/uct_v2.h Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gaps (unchanged from the prior review): no gtest exercises uct_iface_set_async_ep_event_handler (including the UCS_ERR_INVALID_PARAM path), the new event bits, or UCT_EP_ATTR_FIELD_BANDWIDTH. Additionally, the async-vs-progress callback context question cannot be settled from this diff because the provider implementation is not included. The diff is API-surface only (two public headers); the missing implementation/UCP user/tests and the request to split the PR still apply.

@svc-ucx

svc-ucx commented Oct 27, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests roce on worker 0) · commit 01663d2f

TL;DR: The build didn't fail on an assertion — the gtest ud_verbs/test_uct_perf.envelope/1 <ud_verbs/mlx5_1:1> test hung with zero output for exactly the 900 s gtest watchdog window and was killed (main thread stuck in pthread_join at test_perf.cc:300), i.e. one of the two UD perf threads never returned from ucx_perf_run. The PR's commit only adds API declarations to uct.h/uct_v2.h with no transport-side implementation, so this is a UD/RoCE data-path hang on mlx5_1:1, not a regression introduced by the diff — re-run and, if reproducible, debug the UD connection on mlx5_1.

Full analysis

Summary: Tests roce on worker 0 aborted because the UCT perf gtest ud_verbs/test_uct_perf.envelope/1 <ud_verbs/mlx5_1:1> hung and tripped the 15-minute gtest watchdog (test_helpers.cc:57 "Connection timed out - abort testing", SIGABRT, make: *** [Makefile:4948: test] Aborted (core dumped)).

Root cause: Hang, not slowness. Log timestamps show the last application line [ RUN ] ud_verbs/test_uct_perf.envelope/1 <ud_verbs/mlx5_1:1> at 02:59:40.4466 and the next line at 03:14:40.4466 — a single 900.000 s gap with no intervening output, exactly watchdog_timeout = 900. (test/gtest/common/test_helpers.cc:31). The captured backtrace shows the watched (main) thread parked in __pthread_clockjoin_ex → test_perf::run_multi_threaded() at test/gtest/common/test_perf.cc:300, so one of the two perf0/perf1 threads never completed ucx_perf_run() for the very first sub-test ("am short latency", 100k iters, which normally takes well under a second). That means the two-thread UD run over ud_verbs/mlx5_1:1 never made progress — either the UD CREQ/CREP handshake never completed or sends/receives were being dropped, leaving the threads spinning in rte_comm::pop()/ucx_perf_run forever. Nothing in the PR's commit can produce this: 01663d2f touches only src/uct/api/uct.h and src/uct/api/v2/uct_v2.h (new UCT_EVENT_PATH_SPEED_CHANGE, UCT_IFACE_FLAG_EVENT_PATH_SPEED_CHANGE, UCT_IFACE_FLAG_V2_EP_QUERY_BW, uct_ep_attr::bandwidth, uct_perf_attr_t::path_bandwidth, uct_iface_set_async_ep_event_handler declaration) — declarations and struct/enum additions with no transport implementation on the UD send/receive or connection path.

Implicated commit: Not the PR head. 01663d2f (Zhenlong Ma, "UCT/API: Add endpoint path bandwidth monitoring interfaces") is API-header-only. Candidate areas for the UD hang, if it reproduces, are the recent UD/IB connection-path changes already in master: f53da7f0 (Thomas Vegas, "UCT/IB: Add refcounted, TTL-based address handle cache"), 0ee470a8 / 8a38ae21 (Ilia Yastrebov, UD dest_id generation ID / early CEP removal).

File: test/gtest/common/test_perf.cc:300 (hang observed here); test case test/gtest/uct/test_uct_perf.cc:242; watchdog at test/gtest/common/test_helpers.cc:31,57

Suggested fix:

  1. Re-trigger the roce on worker 0 job — on current evidence this is an environment/transport flake unrelated to the PR diff.
  2. If it reproduces on the same agent (swx-rain03, device mlx5_1:1), check that RoCE port mlx5_1:1 is fully usable for UD (link/GID table/ibv_devinfo, ucx_info -d on that device) and re-run just --gtest_filter=ud_verbs/test_uct_perf.* with UCX_LOG_LEVEL=debug to see whether the UD CREQ/CREP handshake completes.
  3. CI-diagnosability improvement (worth a small separate PR): when the watchdog fires it only SIGTERMs the watched thread, which is blocked in pthread_join, so the actually-hung perf thread's stack is never captured. Making the watchdog dump backtraces of all threads (e.g. ucs_debug_print_backtrace for every thread / enable core-dump of all threads) would turn this class of failure from "unknown hang" into an actionable stack.

Related: PR #12173 (this PR), companion PRs #12123 and #12162 (same feature series). No existing issue found for a test_uct_perf.envelope hang.

@evgeny-leksikov evgeny-leksikov left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • doc needs to be polished as well.

Comment thread src/uct/api/v2/uct_v2.h Resolved
Comment thread src/uct/api/uct.h Resolved
Comment thread src/uct/api/uct.h Resolved
Comment thread src/uct/api/v2/uct_v2.h Resolved
@ZhenlongMa
ZhenlongMa marked this pull request as draft October 27, 2026 11:54
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/api/v2/uct_v2.h Resolved
Comment thread src/uct/api/v2/uct_v2.h Outdated
* @return UCS_OK on success, or UCS_ERR_INVALID_PARAM if @a params does not
* contain a callback field.
*/
ucs_status_t uct_iface_set_async_ep_event_handler(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we already have uct_iface_params_t::async_event_cb/async_event_arg (bits UCT_IFACE_PARAM_FIELD_ASYNC_EVENT_{ARG,CB}) for async iface events; why not add the ep-aware callback there as another UCT_IFACE_PARAM_FIELD_* instead of a separate setter with a second callback type and its own lifetime/replacement semantics?

Comment thread src/uct/api/v2/uct_v2.h Resolved
Comment thread src/uct/api/v2/uct_v2.h Resolved
Comment thread src/uct/api/v2/uct_v2.h Resolved
Signed-off-by: Zhenlong Ma <zhenlongm@nvidia.com>
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/api/v2/uct_v2.h Resolved
Comment thread src/uct/api/v2/uct_v2.h Outdated
* @ingroup UCT_RESOURCE
* @brief Register a callback for endpoint-specific asynchronous events.
*
* This routine replaces the callback currently registered on @a iface. Pass a

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pls clarify the relation to uct_iface_params_t::async_event_cb: is this an additional, independent handler, and which of the two gets UCT_EVENT_PATH_SPEED_CHANGE? As written, "the callback currently registered on iface" can be read as replacing async_event_cb.

Comment thread src/uct/api/v2/uct_v2.h Outdated
* @brief Register a callback for endpoint-specific asynchronous events.
*
* This routine replaces the callback currently registered on @a iface. Pass a
* NULL callback to unregister it. The callback field must be present in

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all other UCT params structs treat an absent field as the default value (UCT_IFACE_PARAM_VALUE/UCS_PARAM_VALUE), so making the CB bit mandatory is inconsistent. Since a NULL cb already means "unregister", can a missing ..._FIELD_CB just default to NULL instead of failing?

Comment thread src/uct/api/uct.h Resolved
Comment thread src/uct/api/v2/uct_v2.h Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gaps: still no gtest for uct_iface_set_async_ep_event_handler (including the UCS_ERR_INVALID_PARAM path), the new event bits, or UCT_EP_ATTR_FIELD_BANDWIDTH; and the callback-context contract ("invoked synchronously from its asynchronous event context") cannot be verified here because no transport implementation is included — note that the existing UCT_EVENT_SPEED_CHANGE is in fact delivered from a progress callback (uct_ib_iface_port_speed_change_progress), so the documented context should be re-checked when the provider code lands.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/api/v2/uct_v2.h Resolved
Comment thread src/uct/api/v2/uct_v2.h Outdated
* @return UCS_OK on success, or UCS_ERR_INVALID_PARAM if @a params does not
* contain a callback field.
*/
ucs_status_t uct_iface_set_async_ep_event_handler(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how does the user know that a transport supports endpoint events? There is no event capability flag and the documented return values are only UCS_OK/UCS_ERR_INVALID_PARAM - pls document UCS_ERR_UNSUPPORTED (or add an iface cap flag) for transports that don't implement it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 for "pls document UCS_ERR_UNSUPPORTED (or add an iface cap flag) for transports that don't implement it."

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in d06f9a2: registration documents UCS_ERR_UNSUPPORTED for interfaces that do not support endpoint events, without adding a capability flag.

Comment thread src/uct/api/v2/uct_v2.h Resolved
Comment thread src/uct/api/v2/uct_v2.h Outdated
*
* @param [in] arg User argument registered with
* @ref uct_iface_set_async_ep_event_handler.
* @param [in] ep Endpoint associated with the event. It is valid only

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pls document how ep is kept alive: if the callback runs from the transport async thread, the user thread may be inside uct_ep_destroy() at the same time, so it is not clear what the user must hold (async context blocked?) to use ep safely.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this comment is right, need to check if transport does support sync/async callbacks and thread safety

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not addressed

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in d06f9a2: the API supports only serialized worker-progress delivery, with endpoint destruction and handler changes using the same serialization. The names no longer imply async-thread support.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The header only says “worker progress context.” Pls document what the caller must serialize with progress to keep ep alive, and whether uct_ep_query() may be called from this callback.

Comment thread src/uct/api/uct.h Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

The declared uct_iface_set_async_ep_event_handler() has no definition anywhere in src/uct (uct_iface.c has no such wrapper, and uct_iface_internal_ops_t has no matching op), so this revision still ships a public symbol with no implementation — the previously raised "declared public API with no implementation / split the PR" concern remains unresolved (should be +1'd on the existing thread rather than opened as a new one).

Residual coverage gap (unchanged): no gtest exercises uct_iface_set_async_ep_event_handler (including the UCS_ERR_INVALID_PARAM path), the new event bit, or UCT_EP_ATTR_FIELD_BANDWIDTH.

ep_query providers that are plain ucs_empty_function (mm_iface.c:608, gga_mlx5.c:738) return UCS_OK without touching ep_attr, so a UCT_EP_ATTR_FIELD_BANDWIDTH query there would leave bandwidth uninitialized — worth covering once the implementation lands.

@ZhenlongMa ZhenlongMa changed the title UCT/API: Add APIs for endpoint path bandwidth monitoring UCT/API: Add UCP-UCT APIs for endpoint path bandwidth monitoring Oct 29, 2026
@ZhenlongMa
ZhenlongMa marked this pull request as ready for review October 29, 2026 08:03
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/api/v2/uct_v2.h Resolved
Comment thread src/uct/api/v2/uct_v2.h Resolved
Comment thread src/uct/api/v2/uct_v2.h Outdated
* @retval UCS_ERR_INVALID_PARAM A required parameter field is missing.
* @retval UCS_ERR_UNSUPPORTED The interface does not support endpoint events.
*/
ucs_status_t uct_iface_set_ep_event_handler(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: docs/CodeStyle.md says "Indent function arguments on column"; the rest of this header wraps as ucs_status_t on its own line with arguments aligned (see uct_iface_query_v2). Fits in 80 columns:

Suggested change
ucs_status_t uct_iface_set_ep_event_handler(
ucs_status_t
uct_iface_set_ep_event_handler(uct_iface_h iface,
const uct_iface_set_ep_event_params_t *params);

@svc-nvidia-pr-review

Copy link
Copy Markdown

Public symbol uct_iface_set_ep_event_handler has no definition and no uct_iface_internal_ops_t entry, no UCP consumer, and the request to split per feature is unchanged in this revision (+1 to the existing thread).

UCT_EP_ATTR_FIELD_BANDWIDTH has no base-level handling: uct_ep_query (uct_iface.c:830) dispatches straight to the transport, most providers are ucs_empty_function_return_unsupported, and mm_iface.c:608 / gga_mlx5.c:738 are plain ucs_empty_function returning UCS_OK without touching ep_attr. The newly documented "defaults to the uct_iface_estimate_perf value when unsupported" therefore cannot hold until base or transport code lands (same concern previously filed as the UCS_ERR_UNSUPPORTED contract).

Residual coverage gap: no gtest exercises uct_iface_set_ep_event_handler (including the UCS_ERR_INVALID_PARAM path for a missing CB field), UCT_EP_EVENT_PATH_SPEED_CHANGE, or UCT_EP_ATTR_FIELD_BANDWIDTH; test/gtest/uct/ib/test_rc.cc:1893 is the existing precedent for covering a uct_ep_query field. Nothing is testable in this revision because the diff is header-only, and the mlx5 path would need real HW or contrib/ibmock/.

@svc-ucx

svc-ucx commented Nov 4, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (AddressSanitizer gpu on worker 0) · commit 4c139d37

TL;DR: The ASan GPU gtest shard aborted because ib/test_ucp_rma_rndv_cuda_async.get_nonblocking/0 hung — the released ucp_get_nbx rendezvous requests never completed, so two consecutive flush_worker waits timed out (~11 min of total log silence each) and the watchdog raised "Connection timed out - abort testing"; this is a rendezvous-GET hang on async-allocated CUDA memory, not something PR #12173 can cause (its commits only touch UCT API headers/docs).

Full analysis

Summary: gtest shard 1/4 of the AddressSanitizer GPU job aborted (SIGABRT, make: *** [Makefile:4992: test] Aborted (core dumped), Bash exit 2) after ib/test_ucp_rma_rndv_cuda_async.get_nonblocking/0 <ib,cuda_copy,rocm_copy/flush_worker> hung.

Root cause: A true hang, not a slow test. Timeline from the log: test starts 16:36:40, then total silence until 16:47:55 where ucp_test.cc:310 UCX ERROR request 0x62e000021160 did not complete on time is printed (11m15s gap), followed by test_ucp_memheap.cc:115: Failure / data validation failed (a consequence of the unfinished transfer, not corruption), then another 11-minute gap to 16:59:10 with a second request ... did not complete on time, and finally test_helpers.cc:57: Connection timed out - abort testing → Caught signal 6. The backtrace shows the main thread parked in the gtest progress/wait loop (__sched_yield), i.e. the UCP flush request never completed.

The hanging operation is a forced RMA rendezvous GET from host into async (stream-ordered) CUDA memory over IB with GPUDirect RDMA enabled: test_ucp_rma_rndv_cuda_async sets UCX_IB_GPU_DIRECT_RDMA=y and allocates CUDA buffers with mem_buffer::alloc_mode::ASYNC (test/gtest/ucp/test_ucp_rma.cc:724-747), while its base class forces PROTOS=put/rndv,get/rndv,rndv/* (test_ucp_rma.cc:660). The last INFO progress line is host->cuda 5x128 5x197 5x1371 5x7776 5x28038, so it stalled on the ~28 KB host→cuda iteration. get_nbi releases the request and relies on flush_worker(), so a single lost/never-posted rendezvous completion deadlocks the flush.

This is unrelated to the PR under test: on branch dlb_api, commit 4c139d37 ("UCT/API: Simplify endpoint bandwidth and event documentation", Roie Danino) and the preceding branch commits ([REDACTED:Hex High Entropy String], [REDACTED:Hex High Entropy String], 89c9dbe4, 76ca2c3b, 3eaea825) touch only src/uct/api/uct.h and src/uct/api/v2/uct_v2.h — pure enum/struct/callback declarations and documentation (UCT_EP_EVENT_PATH_SPEED_CHANGE, uct_iface_set_ep_event_params_t) with no transport or UCP implementation, so they cannot affect the rndv data path. The newest src/ucp commit on this branch is from master (Oct 19).

Implicated commit: Not the PR commit 4c139d37 (API-header-only). Most likely pre-existing master regression in the rendezvous/GET repost path — prime candidates: [REDACTED:Hex High Entropy String] "UCP/PROTO: Repost GET and zcopy on the selected lane" (#12091), 75b4226e (#12047), [REDACTED:Hex High Entropy String] "UCP/RNDV: Use local memory detection for RTR_REQ responder" (#12041), af8174b8 "UCP/UCT/CUDA: Avoid gdr_copy for non gdr capable memory" (#12039).

File: test/gtest/ucp/test_ucp_rma.cc:724-769 (test_ucp_rma_rndv_cuda_async); failure reported at test/gtest/ucp/test_ucp_memheap.cc:115, timeout at test/gtest/ucp/ucp_test.cc:300.

Suggested fix:

  1. Re-trigger this job — the PR's diff cannot produce this hang, so do not block on it as a PR regression.
  2. Reproduce/triage on master with --gtest_filter=*test_ucp_rma_rndv_cuda_async*get_nonblocking*, UCX_LOG_LEVEL=debug, UCX_PROTO_INFO=used, and capture the core dump to see which lane/protocol instance is stuck.
  3. Note the existing in-tree hint at test/gtest/ucp/test_ucp_rma.cc:532-534: CI exports UCX_PROTO_REQUEST_RESET=y, and rndv/put/mtype (used for host-staged GPU rendezvous) does not implement request reset, which restarts pending requests. Try reproducing with UCX_PROTO_REQUEST_RESET=n; if that cures it, apply the same modify_config("PROTO_REQUEST_RESET", "n") workaround to test_ucp_rma_rndv_cuda_async (or implement reset for rndv/put/mtype) and open a tracking issue.
  4. Do not raise the gtest timeout — the log shows ~11 minutes of zero output, so the request is deadlocked, not slow.

Related: PR #12173 (build under test, API-only). Candidate culprits to check: #12091, #12047, #12041, #12039.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 55df14fc-4e23-49cd-b1bc-7b74b473791c in the triage console for the audit trail.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/api/v2/uct_v2.h Resolved
Comment thread src/uct/api/v2/uct_v2.h Resolved
Comment thread src/uct/api/v2/uct_v2.h Resolved
Comment thread src/uct/api/v2/uct_v2.h Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

+1 to the existing thread: uct_iface_set_ep_event_handler is a public declaration with no definition, no uct_iface_internal_ops_t entry, no UCP consumer, and the request to split per feature — unchanged in this revision.

+1 to the existing thread: UCT_EP_ATTR_FIELD_BANDWIDTH has no base-level handling, so the documented "defaults to the uct_iface_estimate_perf value when unsupported" cannot hold (most ep_query are ucs_empty_function_return_unsupported; mm_iface.c:608 and gga_mlx5.c:738 are plain ucs_empty_function returning UCS_OK without touching ep_attr).

Residual coverage gaps: no gtest exercises uct_iface_set_ep_event_handler (including the UCS_ERR_INVALID_PARAM path for a missing CB field), UCT_EP_EVENT_PATH_SPEED_CHANGE, or UCT_EP_ATTR_FIELD_BANDWIDTH; test/gtest/uct/ib/test_rc.cc is the precedent for covering a uct_ep_query field (UCT_EP_ATTR_FIELD_TX_TOKEN). Nothing is testable in this revision because the diff is header-only, and the mlx5 path would need real HW or contrib/ibmock/.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/api/v2/uct_v2.h Resolved
Comment thread src/uct/api/v2/uct_v2.h
* @retval UCS_ERR_INVALID_PARAM A required parameter field is missing.
* @retval UCS_ERR_UNSUPPORTED The interface does not support endpoint events.
*/
ucs_status_t

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we pass this callback through uct_iface_params_t (like async_event_cb /
UCT_IFACE_PARAM_FIELD_ASYNC_EVENT_CB) instead of adding a new setter plus
params struct and field-mask enum? it would reuse the existing iface event
registration pattern and shrink the new API surface.

@svc-nvidia-pr-review

Copy link
Copy Markdown

uct_iface_set_ep_event_handler is a public symbol with no definition, no uct_iface_internal_ops_t entry, and no UCP consumer; the request to split the change per feature is also unchanged in this revision. (+1 to the existing thread.)

UCT_EP_ATTR_FIELD_BANDWIDTH has no base-level handling, so the documented "defaults to the uct_iface_estimate_perf value when unsupported" cannot hold yet (mm_iface.c:608, gga_mlx5.c:738 return UCS_OK without filling ep_attr). (+1 to the existing thread.)

Residual coverage gap: no gtest exercises uct_iface_set_ep_event_handler (including the UCS_ERR_INVALID_PARAM path for a missing CB field), UCT_EP_EVENT_PATH_SPEED_CHANGE, or UCT_EP_ATTR_FIELD_BANDWIDTH; test/gtest/uct/ib/test_rc.cc is the precedent for covering a uct_ep_query field (UCT_EP_ATTR_FIELD_TX_TOKEN).

Comment thread src/uct/api/v2/uct_v2.h
* @ingroup UCT_RESOURCE
* @brief Register a callback for endpoint-specific events.
*
* This routine sets the endpoint-specific event handler on @a iface.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pls define replacement/unregistration semantics here: must the caller serialize this call with worker progress, and after removing the handler, can any queued callback still use the old arg? The caller needs to know when it can free that argument.

gleon99
gleon99 previously approved these changes Nov 8, 2026
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/api/v2/uct_v2.h
enum uct_ep_event_types {
/** Endpoint path speed changed. The updated bandwidth estimate is available
* through @ref uct_ep_query. */
UCT_EP_EVENT_PATH_SPEED_CHANGE = UCS_BIT(0)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this bit space overlaps uct_iface_event_types (UCS_BIT(0) is UCT_EVENT_SEND_COMP), and both are passed as plain unsigned event flags. why not extend uct_iface_event_types with the next free bit instead of a second enum, given UCT_EVENT_SPEED_CHANGE already exists there?

@svc-nvidia-pr-review

Copy link
Copy Markdown

src/uct/api/v2/uct_v2.h

+1 on the existing thread: public symbol uct_iface_set_ep_event_handler has no definition, no uct_iface_internal_ops_t op, and no UCP user. Please split per feature (API + transport impl + UCP user + gtest).

src/uct/api/v2/uct_v2.h

+1 on the existing thread: UCT_EP_ATTR_FIELD_BANDWIDTH has no base-level handling. uct_ep_query (uct_iface.c:830) dispatches straight to the transport; most ep_query implementations are ucs_empty_function_return_unsupported, and mm_iface.c:608 / gga_mlx5.c:738 are plain ucs_empty_function returning UCS_OK without touching ep_attr, so the newly documented "defaults to the uct_iface_estimate_perf value when unsupported" cannot hold (cf. uct_iface_base_query_v2's token_mask rejection as the precedent for a base-level mask check).

src/uct/api/v2/uct_v2.h

+1 on the existing thread about callback-context wording — the current "worker progress context" matches uct_ib_iface_port_speed_change_progress (ib_iface.c:1744), so this looks settled.

Coverage gap: no gtest exercises uct_iface_set_ep_event_handler (including the UCS_ERR_INVALID_PARAM path when UCT_IFACE_SET_EP_EVENT_PARAM_FIELD_CB is missing), UCT_EP_EVENT_PATH_SPEED_CHANGE, or UCT_EP_ATTR_FIELD_BANDWIDTH. test/gtest/uct/ib/test_rc.cc is the precedent for covering a uct_ep_query field (UCT_EP_ATTR_FIELD_TX_TOKEN), and test/gtest/ucp/test_ucp_proto_mock.cc (test_ucp_proto_mock_rcx_speed_change) is the precedent for the speed-change flow.

@svc-ucx

svc-ucx commented Nov 9, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests roce on worker 0) · commit 7958256e

TL;DR: The gtest shard aborted because the first test in it, ud/test_ucp_am_id.am_id_overflow/0 <ud_v>, hung forever in teardown — ucp_worker_flush never completed on the ud_v (UD over RoCE) lane, so close_all_eps() spun until the gtest global deadline aborted the process. PR #12173 only changes UCT API headers/docs, so this is a UD/RoCE wireup-or-flush hang on the agent, not a regression from this PR — re-trigger, and if it reproduces investigate the recent UD ep changes.

Full analysis

Summary: roce on worker 0 → make -C test/gtest test (shard 1 of 4) aborted with SIGABRT: "Connection timed out - abort testing" while tearing down ud/test_ucp_am_id.am_id_overflow/0 <ud_v>.

Root cause: Timestamp analysis shows a hang, not slowness:

  • 09:00:55.90 [ RUN ] ud/test_ucp_am_id.am_id_overflow/0 <ud_v>
  • 09:00:56.03 last application output (invalid AM id 4294901761 ..., the expected API validation errors)
  • 3-minute silence → 09:03:56 ucp_test.cc:310 UCX ERROR request 0x5584... did not complete on time — this is the flush_worker() call inside ucp_test::disconnect() (ucp_test.cc:223) whose flush request never completed.
  • 12-minute silence → 09:15:56 test_helpers.cc:57: Failure / Connection timed out - abort testing, with the backtrace showing the thread stuck in ucp_test_base::entity::close_all_eps() → ucp_test::progress() → sched_yield() (ucp_test.cc:937, :183).

So the UD endpoint over the RoCE device never drained: neither worker flush nor EP close made progress for 15 minutes. This was the very first test executed in the shard, i.e. the first ud_v connection on that machine (swx-rain04 / agent-08), which points at UD wireup/ACK/retransmit not completing (device/GID reachability or a UD connection-state bug) rather than at anything the test did.

The PR cannot explain it: every commit on branch dlb_api (01663d2f, 3eaea825, 76ca2c3b, 89c9dbe4, [REDACTED:Hex High Entropy String], [REDACTED:Hex High Entropy String], 4c139d37, e67e6ada, 11a8c1df, 7958256e) touches only src/uct/api/uct.h and src/uct/api/v2/uct_v2.h — new enums/typedefs (UCT_EP_EVENT_PATH_SPEED_CHANGE, uct_iface_set_ep_event_params_t) and documentation, with no implementation anywhere under src/uct/ib/... or src/ucp/.... The HEAD commit is literally "UCT/API: Refine endpoint event and bandwidth documentation".

Implicated commit: None from this PR. Pre-existing/environmental. If it reproduces, the UD candidates currently in master are 183a66db "UCT: Keep-alive for private UD EPs" (Ilia Yastrebov), 0ee470a8 "UCT/UD: Add generation ID to dest_id to make it unique (#11800)" (Ilia Yastrebov) and f53da7f0 "UCT/IB: Add refcounted, TTL-based address handle cache (#11845)" (Thomas Vegas) — all touching src/uct/ib/ud/base/ud_ep.c.

File: test/gtest/ucp/ucp_test.cc:937 (hang detected in close_all_eps; flush timeout reported at test/gtest/ucp/ucp_test.cc:300) — underlying component src/uct/ib/ud/base/ud_ep.c / src/uct/ib/ud/base/ud_iface.c

Suggested fix:

  1. Re-run the roce on worker 0 job to confirm; this single-test, first-test-in-shard UD stall is characteristic of a bad RoCE device state on swx-rain04/agent-08. Check the agent's RoCE port state/GID table (ibv_devinfo, port ACTIVE, RoCE v2 GID present) before blaming the code.
  2. Do not raise the gtest timeout — the 3-min and 12-min silent gaps prove a hang, not slow progress.
  3. If it reproduces on other agents, debug the UD flush path: rerun with UCX_LOG_LEVEL=debug UCX_TLS=ud_v --gtest_filter=*test_ucp_am_id.am_id_overflow* and inspect UD ACK/retransmit state (uct_ud_ep_t::tx.acked_psn vs psn) to see whether a pending/unacked packet is blocking uct_ud_ep_flush; the private-UD-EP keep-alive and dest_id generation-ID changes above are the most recent behavioural changes in that path.
  4. Separately, consider making close_all_eps() bail out (force-close) once the flush request has already timed out, so one stuck EP wastes 3 minutes instead of 15 and the shard reports a normal test failure rather than a core dump.

Related: PR #12173 (this build). No existing issue found for this UD flush hang signature — worth filing one with this log if it recurs.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 72e69a78-2655-43a7-8d6e-4abff18d13ce in the triage console for the audit trail.

@gleon99
gleon99 enabled auto-merge (squash) November 9, 2026 10:20
@roiedanino roiedanino self-assigned this Nov 9, 2026
@gleon99
gleon99 disabled auto-merge November 10, 2026 07:53
@gleon99
gleon99 merged commit 74b413a into openucx:master Nov 10, 2026
165 of 167 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants