Repository navigation
UCT/API: Add UCP-UCT APIs for endpoint path bandwidth monitoring - #12173
Conversation
|
🤖 Starting review — findings will be posted here when done. |
evgeny-leksikov
left a comment
There was a problem hiding this comment.
pls del everything except uct/api from this PR
|
This PR only adds API surface: Residual coverage gaps: no gtest exercises |
Signed-off-by: Zhenlong Ma <zhenlongm@nvidia.com>
|
🤖 Starting review — findings will be posted here when done. |
| * @ingroup UCT_RESOURCE | ||
| * @brief Endpoint-aware asynchronous event callback. | ||
| * | ||
| * The transport invokes this callback synchronously from its asynchronous |
There was a problem hiding this comment.
UCT_EVENT_SPEED_CHANGE is not delivered from async context today — uct_ib_iface_port_speed_change_progress() defers it to the worker progress queue, and UCP still takes UCS_ASYNC_BLOCK around it. Pls document the context the user can actually rely on, since that decides whether the callback needs the async lock.
| * for the duration of the callback. | ||
| * @param [in] events Event flags from @ref uct_iface_event_types. | ||
| */ | ||
| typedef void (*uct_async_ep_event_cb_t)(void *arg, uct_ep_h ep, |
There was a problem hiding this comment.
UCT already registers an async event handler via uct_iface_params_t::async_event_cb/async_event_arg. Why not add async_ep_event_cb/arg fields there instead of a second setter, so a transport has one handler slot rather than two with per-event-bit routing? This currently forces two handler slots and a per-event-bit routing decision in every transport and in UCP.
| * Latest completed endpoint bandwidth sample, in bytes per second. | ||
| * Valid when @ref UCT_EP_ATTR_FIELD_BANDWIDTH is set in @ref field_mask | ||
| * and the interface advertises @ref UCT_IFACE_FLAG_V2_EP_QUERY_BW. | ||
| * A transport may return @ref UCS_ERR_NO_ELEM before the first sample or |
There was a problem hiding this comment.
uct_ep_query returns a single status for all requested fields, so this makes a combined query (e.g. BANDWIDTH | TX_TOKEN) fail entirely when the sample is not ready. "may return" also leaves no portable handling. Pls make it definite — which status is returned, and whether the other requested fields are still filled.
|
Residual coverage gaps (unchanged from the prior review): no gtest exercises |
|
🤖 CI Triage Agent — TL;DR: The build didn't fail on an assertion — the gtest Full analysisSummary: Root cause: Hang, not slowness. Log timestamps show the last application line Implicated commit: Not the PR head. File: Suggested fix:
Related: PR #12173 (this PR), companion PRs #12123 and #12162 (same feature series). No existing issue found for a |
evgeny-leksikov
left a comment
There was a problem hiding this comment.
- doc needs to be polished as well.
|
🤖 Starting review — findings will be posted here when done. |
| * @return UCS_OK on success, or UCS_ERR_INVALID_PARAM if @a params does not | ||
| * contain a callback field. | ||
| */ | ||
| ucs_status_t uct_iface_set_async_ep_event_handler( |
There was a problem hiding this comment.
we already have uct_iface_params_t::async_event_cb/async_event_arg (bits UCT_IFACE_PARAM_FIELD_ASYNC_EVENT_{ARG,CB}) for async iface events; why not add the ep-aware callback there as another UCT_IFACE_PARAM_FIELD_* instead of a separate setter with a second callback type and its own lifetime/replacement semantics?
Signed-off-by: Zhenlong Ma <zhenlongm@nvidia.com>
|
🤖 Starting review — findings will be posted here when done. |
| * @ingroup UCT_RESOURCE | ||
| * @brief Register a callback for endpoint-specific asynchronous events. | ||
| * | ||
| * This routine replaces the callback currently registered on @a iface. Pass a |
There was a problem hiding this comment.
pls clarify the relation to uct_iface_params_t::async_event_cb: is this an additional, independent handler, and which of the two gets UCT_EVENT_PATH_SPEED_CHANGE? As written, "the callback currently registered on iface" can be read as replacing async_event_cb.
| * @brief Register a callback for endpoint-specific asynchronous events. | ||
| * | ||
| * This routine replaces the callback currently registered on @a iface. Pass a | ||
| * NULL callback to unregister it. The callback field must be present in |
There was a problem hiding this comment.
all other UCT params structs treat an absent field as the default value (UCT_IFACE_PARAM_VALUE/UCS_PARAM_VALUE), so making the CB bit mandatory is inconsistent. Since a NULL cb already means "unregister", can a missing ..._FIELD_CB just default to NULL instead of failing?
|
Residual coverage gaps: still no gtest for |
|
🤖 Starting review — findings will be posted here when done. |
| * @return UCS_OK on success, or UCS_ERR_INVALID_PARAM if @a params does not | ||
| * contain a callback field. | ||
| */ | ||
| ucs_status_t uct_iface_set_async_ep_event_handler( |
There was a problem hiding this comment.
how does the user know that a transport supports endpoint events? There is no event capability flag and the documented return values are only UCS_OK/UCS_ERR_INVALID_PARAM - pls document UCS_ERR_UNSUPPORTED (or add an iface cap flag) for transports that don't implement it.
There was a problem hiding this comment.
👍 for "pls document UCS_ERR_UNSUPPORTED (or add an iface cap flag) for transports that don't implement it."
There was a problem hiding this comment.
Updated in d06f9a2: registration documents UCS_ERR_UNSUPPORTED for interfaces that do not support endpoint events, without adding a capability flag.
| * | ||
| * @param [in] arg User argument registered with | ||
| * @ref uct_iface_set_async_ep_event_handler. | ||
| * @param [in] ep Endpoint associated with the event. It is valid only |
There was a problem hiding this comment.
pls document how ep is kept alive: if the callback runs from the transport async thread, the user thread may be inside uct_ep_destroy() at the same time, so it is not clear what the user must hold (async context blocked?) to use ep safely.
There was a problem hiding this comment.
this comment is right, need to check if transport does support sync/async callbacks and thread safety
There was a problem hiding this comment.
Updated in d06f9a2: the API supports only serialized worker-progress delivery, with endpoint destruction and handler changes using the same serialization. The names no longer imply async-thread support.
There was a problem hiding this comment.
The header only says “worker progress context.” Pls document what the caller must serialize with progress to keep ep alive, and whether uct_ep_query() may be called from this callback.
|
The declared Residual coverage gap (unchanged): no gtest exercises
|
|
🤖 Starting review — findings will be posted here when done. |
| * @retval UCS_ERR_INVALID_PARAM A required parameter field is missing. | ||
| * @retval UCS_ERR_UNSUPPORTED The interface does not support endpoint events. | ||
| */ | ||
| ucs_status_t uct_iface_set_ep_event_handler( |
There was a problem hiding this comment.
minor: docs/CodeStyle.md says "Indent function arguments on column"; the rest of this header wraps as ucs_status_t on its own line with arguments aligned (see uct_iface_query_v2). Fits in 80 columns:
| ucs_status_t uct_iface_set_ep_event_handler( | |
| ucs_status_t | |
| uct_iface_set_ep_event_handler(uct_iface_h iface, | |
| const uct_iface_set_ep_event_params_t *params); |
|
Public symbol
Residual coverage gap: no gtest exercises |
|
🤖 CI Triage Agent — TL;DR: The ASan GPU gtest shard aborted because Full analysisSummary: gtest shard 1/4 of the AddressSanitizer GPU job aborted (SIGABRT, Root cause: A true hang, not a slow test. Timeline from the log: test starts The hanging operation is a forced RMA rendezvous GET from host into async (stream-ordered) CUDA memory over IB with GPUDirect RDMA enabled: This is unrelated to the PR under test: on branch Implicated commit: Not the PR commit File: test/gtest/ucp/test_ucp_rma.cc:724-769 ( Suggested fix:
Related: PR #12173 (build under test, API-only). Candidate culprits to check: #12091, #12047, #12041, #12039.
|
|
🤖 Starting review — findings will be posted here when done. |
|
+1 to the existing thread: +1 to the existing thread: Residual coverage gaps: no gtest exercises |
|
🤖 Starting review — findings will be posted here when done. |
| * @retval UCS_ERR_INVALID_PARAM A required parameter field is missing. | ||
| * @retval UCS_ERR_UNSUPPORTED The interface does not support endpoint events. | ||
| */ | ||
| ucs_status_t |
There was a problem hiding this comment.
can we pass this callback through uct_iface_params_t (like async_event_cb /
UCT_IFACE_PARAM_FIELD_ASYNC_EVENT_CB) instead of adding a new setter plus
params struct and field-mask enum? it would reuse the existing iface event
registration pattern and shrink the new API surface.
|
Residual coverage gap: no gtest exercises |
| * @ingroup UCT_RESOURCE | ||
| * @brief Register a callback for endpoint-specific events. | ||
| * | ||
| * This routine sets the endpoint-specific event handler on @a iface. |
There was a problem hiding this comment.
Pls define replacement/unregistration semantics here: must the caller serialize this call with worker progress, and after removing the handler, can any queued callback still use the old arg? The caller needs to know when it can free that argument.
|
🤖 Starting review — findings will be posted here when done. |
| enum uct_ep_event_types { | ||
| /** Endpoint path speed changed. The updated bandwidth estimate is available | ||
| * through @ref uct_ep_query. */ | ||
| UCT_EP_EVENT_PATH_SPEED_CHANGE = UCS_BIT(0) |
There was a problem hiding this comment.
this bit space overlaps uct_iface_event_types (UCS_BIT(0) is UCT_EVENT_SEND_COMP), and both are passed as plain unsigned event flags. why not extend uct_iface_event_types with the next free bit instead of a second enum, given UCT_EVENT_SPEED_CHANGE already exists there?
|
+1 on the existing thread: public symbol
+1 on the existing thread:
+1 on the existing thread about callback-context wording — the current "worker progress context" matches Coverage gap: no gtest exercises |
|
🤖 CI Triage Agent — TL;DR: The gtest shard aborted because the first test in it, Full analysisSummary: Root cause: Timestamp analysis shows a hang, not slowness:
So the UD endpoint over the RoCE device never drained: neither worker flush nor EP close made progress for 15 minutes. This was the very first test executed in the shard, i.e. the first The PR cannot explain it: every commit on branch Implicated commit: None from this PR. Pre-existing/environmental. If it reproduces, the UD candidates currently in master are File: test/gtest/ucp/ucp_test.cc:937 (hang detected in Suggested fix:
Related: PR #12173 (this build). No existing issue found for this UD flush hang signature — worth filing one with this log if it recurs.
|
What?
Add the APIs required for endpoint-specific path bandwidth monitoring.
Why?
Device-level speed information does not describe the performance of each
endpoint path. Consumers need endpoint-specific notification and query APIs
to obtain the affected path's bandwidth.
How?
UCT_EP_EVENT_PATH_SPEED_CHANGEand an endpoint-aware callback deliveredin serialized worker-progress context.
uct_ep_query()withUCT_EP_ATTR_FIELD_BANDWIDTHto return the latestavailable cached estimate in bytes per second, without blocking or requesting
an update.
UCS_ERR_UNSUPPORTED.This PR defines the API contract only. Transport implementation and event
handling are outside its scope.