Repository navigation
UCT/IB: Add configurable TTL knob for the AH cache - #11847
Conversation
Add UCX_IB_AH_CACHE_TTL (0 or inf) to bypass or keep the address handle cache. All non-SRD AH creation call sites (UD, RC DEVX QP connect, DC pure grant, compact AV probe) honor the setting, SRD always uses the cache directly since it needs a stable AH pointer for is_connected().
|
🤖 Starting review — findings will be posted here when done. |
|
minor: The AH_CACHE_TTL option is typed as TIME_UNITS and named TTL, but only 0/inf are legal and any real duration is rejected at MD open. That is a surprising UX (a "TTL" that accepts no time value). Config naming/UX is never a blocker, so this is just a note — consider clarifying in the doc string that no intermediate values are supported yet (the doc already says "No other values are allowed", which is adequate). Coverage note: No test covers the UCX_IB_AH_CACHE_TTL=0 path through SRD / dc-mlx5 FC-grant / rc-devx connect AH release (i.e., that the uncached AHs are actually destroyed and not leaked). This relies on existing transport tests running under a non-default config; if CI does not run a job with IB_AH_CACHE_TTL=0, leak/lifetime regressions in those release sites would go uncaught. Worth confirming a CI job or valgrind run exercises the disabled-cache path across RC/DC/UD/SRD, or adding one. |
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 Review complete — no issues found. |
| typedef struct { | ||
| uct_ud_ep_t super; | ||
| uct_ud_verbs_ep_peer_address_t peer_address; | ||
| struct ibv_ah *ah; |
There was a problem hiding this comment.
it increases ep size, maybe we can fix just mlx5 transports in this Pr?
There was a problem hiding this comment.
i'd rather not, if we leave ud_verbs always on the cache, this could be an issue if it ends-up being also used. seems it can be auxiliary, and also it simpler to understand that option affects all relevant transports.
There was a problem hiding this comment.
removed ud_verbs from the pr
| int uct_ib_iface_prepare_rx_wrs(uct_ib_iface_t *iface, ucs_mpool_t *mp, | ||
| uct_ib_recv_wr_t *wrs, unsigned n); | ||
|
|
||
| /* Honors UCX_IB_AH_CACHE_TTL=0 to bypass the AH cache */ |
There was a problem hiding this comment.
This comment seems out of place
| ucs_offsetof(uct_ib_md_config_t, ext.direct_nic), UCS_CONFIG_TYPE_BOOL}, | ||
|
|
||
| {"AH_CACHE_TTL", "inf", | ||
| "Address handle (AH) cache: 0 disables it, inf keeps it enabled. No\n" |
There was a problem hiding this comment.
Maybe disables it when possible? SRD still has the cache
There was a problem hiding this comment.
maybe move it to uct_ib_mlx5_iface_config_table?
or anyway we will have to add others later?
There was a problem hiding this comment.
i would leave it there as the plan to generalize to ud_v and srd afaiu
| return uct_ib_device_create_ah_cached(uct_ib_iface_device(iface), ah_attr, | ||
| uct_ib_device_t *dev = uct_ib_iface_device(iface); | ||
|
|
||
| /* ah_cache_ttl=0 (UCX_IB_AH_CACHE_TTL) bypasses the cache */ |
There was a problem hiding this comment.
This comment is not useful
| return UCS_OK; | ||
| } | ||
|
|
||
| static void uct_ud_verbs_ep_destroy_ah(uct_ud_verbs_ep_t *ep) |
There was a problem hiding this comment.
Consider naming uct_ud_verbs_ep_release_ah
Rename uct_ud_verbs_ep_destroy_ah() to uct_ud_verbs_ep_release_ah() to match uct_ib_iface_release_ah(), and simplify its early-return into a single if-block.
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 Review complete — no issues found. |
|
🤖 Starting review — findings will be posted here when done. |
|
AH_CACHE_TTL is typed as TIME_UNITS but only accepts 0/inf (config UX note, not a blocker; the doc string already states the restriction). Coverage of the IB_AH_CACHE_TTL=0 destroy path for SRD and rc-devx connect is not exercised by a dedicated test (dc-mlx5 FC-grant and generic create/release are covered by the new gtests). This relies on existing transport tests running under a ttl=0 CI job. Residual coverage gap worth confirming (not a blocker): whether any CI job runs the RC/DC/UD/SRD suites with UCX_IB_AH_CACHE_TTL=0 (or under valgrind) so that leaks/lifetime regressions in the uncached release sites are actually caught. |
UD verbs no longer honors UCX_IB_AH_CACHE_TTL: it now creates its AH via uct_ib_device_create_ah_cached() directly, matching SRD's existing always-cached behavior instead of depending on the TTL knob. This reverts UD verbs' is_connected()/peer-address logic and the ep_resolve_peer_address hook back to the pre-existing pointer-identity scheme (via uct_ib_iface_is_connected()), since a cache-owned AH is stable again and no longer needs a separate dlid/gid comparison. Also shortens the uct_ib_device_create_ah_cached() doc comment and consolidates the duplicated "always cached" call-site comments into it.
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 Review complete — no issues found. |
brminich
left a comment
There was a problem hiding this comment.
minor comment regarding config
| ucs_offsetof(uct_ib_md_config_t, ext.direct_nic), UCS_CONFIG_TYPE_BOOL}, | ||
|
|
||
| {"AH_CACHE_TTL", "inf", | ||
| "Address handle (AH) cache: 0 disables it, inf keeps it enabled. No\n" |
There was a problem hiding this comment.
maybe move it to uct_ib_mlx5_iface_config_table?
or anyway we will have to add others later?
|
🤖 CI Triage Agent — TL;DR: The gtest suite on the GPU worker completed but 2 of 10223 tests failed — Full analysisSummary: Root cause: The failure signature is highly selective and |
UCT/IB: Add configurable TTL knob for the AH cache (#11847)
What?
Add
UCX_IB_AH_CACHE_TTL:0creates AHs uncached,infkeeps them cached (no eviction).Why?
The AH cache speeds up connection setup, but its underlying L2 resolution can become invalid (RoCE), leaving a stale cached entry.
How?
Users of the TTL configuration (cached or uncached):
Users of the always-cached path:
Added the configuration knob and tests.