Skip to content

UCT/IB: Add configurable TTL knob for the AH cache - #11847

Merged
tvegas1 merged 5 commits into
openucx:masterfrom
tvegas1:ah_cache_tll
Aug 29, 2026
Merged

tvegas1 merged 5 commits into
openucx:masterfrom
tvegas1:ah_cache_tll

Conversation

@tvegas1

@tvegas1 tvegas1 commented Aug 27, 2026 •

Copy link
Copy Markdown
Contributor

What?

Add UCX_IB_AH_CACHE_TTL: 0 creates AHs uncached, inf keeps them cached (no eviction).

Why?

The AH cache speeds up connection setup, but its underlying L2 resolution can become invalid (RoCE), leaving a stale cached entry.

How?

Users of the TTL configuration (cached or uncached):

  • RC mlx5, UD mlx5, DC (connect and grant)

Users of the always-cached path:

  • UD Verbs, SRD (still compare raw AH pointers)

Added the configuration knob and tests.

Add UCX_IB_AH_CACHE_TTL (0 or inf) to bypass or keep the address handle
cache. All non-SRD AH creation call sites (UD, RC DEVX QP connect, DC
pure grant, compact AV probe) honor the setting, SRD always uses the
cache directly since it needs a stable AH pointer for is_connected().
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/ib/mlx5/dc/dc_mlx5_ep.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

src/uct/ib/base/ib_md.c

minor: The AH_CACHE_TTL option is typed as TIME_UNITS and named TTL, but only 0/inf are legal and any real duration is rejected at MD open. That is a surprising UX (a "TTL" that accepts no time value). Config naming/UX is never a blocker, so this is just a note — consider clarifying in the doc string that no intermediate values are supported yet (the doc already says "No other values are allowed", which is adequate).

Coverage note: No test covers the UCX_IB_AH_CACHE_TTL=0 path through SRD / dc-mlx5 FC-grant / rc-devx connect AH release (i.e., that the uncached AHs are actually destroyed and not leaked). This relies on existing transport tests running under a non-default config; if CI does not run a job with IB_AH_CACHE_TTL=0, leak/lifetime regressions in those release sites would go uncaught. Worth confirming a CI job or valgrind run exercises the disabled-cache path across RC/DC/UD/SRD, or adding one.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Review complete — no issues found.

Comment thread src/uct/ib/ud/verbs/ud_verbs.h Outdated
typedef struct {
uct_ud_ep_t super;
uct_ud_verbs_ep_peer_address_t peer_address;
struct ibv_ah *ah;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it increases ep size, maybe we can fix just mlx5 transports in this Pr?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'd rather not, if we leave ud_verbs always on the cache, this could be an issue if it ends-up being also used. seems it can be auxiliary, and also it simpler to understand that option affects all relevant transports.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed ud_verbs from the pr

@tomerg-nvidia tomerg-nvidia left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor comments

Comment thread src/uct/ib/base/ib_iface.h Outdated
int uct_ib_iface_prepare_rx_wrs(uct_ib_iface_t *iface, ucs_mpool_t *mp,
uct_ib_recv_wr_t *wrs, unsigned n);

/* Honors UCX_IB_AH_CACHE_TTL=0 to bypass the AH cache */

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment seems out of place

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed

Comment thread src/uct/ib/base/ib_md.c
ucs_offsetof(uct_ib_md_config_t, ext.direct_nic), UCS_CONFIG_TYPE_BOOL},

{"AH_CACHE_TTL", "inf",
"Address handle (AH) cache: 0 disables it, inf keeps it enabled. No\n"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe disables it when possible? SRD still has the cache

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe move it to uct_ib_mlx5_iface_config_table?
or anyway we will have to add others later?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i would leave it there as the plan to generalize to ud_v and srd afaiu

Comment thread src/uct/ib/base/ib_iface.c Outdated
return uct_ib_device_create_ah_cached(uct_ib_iface_device(iface), ah_attr,
uct_ib_device_t *dev = uct_ib_iface_device(iface);

/* ah_cache_ttl=0 (UCX_IB_AH_CACHE_TTL) bypasses the cache */

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment is not useful

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove

Comment thread src/uct/ib/base/ib_md.c Resolved
Comment thread src/uct/ib/ud/verbs/ud_verbs.c Outdated
return UCS_OK;
}

static void uct_ud_verbs_ep_destroy_ah(uct_ud_verbs_ep_t *ep)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider naming uct_ud_verbs_ep_release_ah

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed function

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

renamed

Rename uct_ud_verbs_ep_destroy_ah() to uct_ud_verbs_ep_release_ah()
to match uct_ib_iface_release_ah(), and simplify its early-return
into a single if-block.
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Review complete — no issues found.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-nvidia-pr-review

Copy link
Copy Markdown

AH_CACHE_TTL is typed as TIME_UNITS but only accepts 0/inf (config UX note, not a blocker; the doc string already states the restriction).

Coverage of the IB_AH_CACHE_TTL=0 destroy path for SRD and rc-devx connect is not exercised by a dedicated test (dc-mlx5 FC-grant and generic create/release are covered by the new gtests). This relies on existing transport tests running under a ttl=0 CI job.

Residual coverage gap worth confirming (not a blocker): whether any CI job runs the RC/DC/UD/SRD suites with UCX_IB_AH_CACHE_TTL=0 (or under valgrind) so that leaks/lifetime regressions in the uncached release sites are actually caught.

UD verbs no longer honors UCX_IB_AH_CACHE_TTL: it now creates its AH via
uct_ib_device_create_ah_cached() directly, matching SRD's existing
always-cached behavior instead of depending on the TTL knob. This
reverts UD verbs' is_connected()/peer-address logic and the
ep_resolve_peer_address hook back to the pre-existing pointer-identity
scheme (via uct_ib_iface_is_connected()), since a cache-owned AH is
stable again and no longer needs a separate dlid/gid comparison.

Also shortens the uct_ib_device_create_ah_cached() doc comment and
consolidates the duplicated "always cached" call-site comments into it.
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Review complete — no issues found.

@brminich brminich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor comment regarding config

Comment thread src/uct/ib/base/ib_md.c
ucs_offsetof(uct_ib_md_config_t, ext.direct_nic), UCS_CONFIG_TYPE_BOOL},

{"AH_CACHE_TTL", "inf",
"Address handle (AH) cache: 0 disables it, inf keeps it enabled. No\n"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe move it to uct_ib_mlx5_iface_config_table?
or anyway we will have to add others later?

@tvegas1
tvegas1 enabled auto-merge (squash) August 28, 2026 16:39
@svc-ucx

svc-ucx commented Aug 28, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests gpu on worker 3) · commit 09f4c1a3

TL;DR: The gtest suite on the GPU worker completed but 2 of 10223 tests failed — rc_gda/test_ucp_device_xfer.put/6 and /10, i.e. exactly the two host destination + warp level + without_req variants — a completion/visibility gap in the device-put path that only shows up when the destination is host memory (no implicit CUDA sync in the verification path); the AH-cache commits in this PR do not touch the GDA path, so this is most likely a pre-existing race rather than a regression from #11847.

Full analysis

Summary: make test in build-test/test/gtest exited 1 because rc_gda/test_ucp_device_xfer.put/6 <rc,rc_gda,cuda_copy,rocm_copy/host/warp/nodelay_without_req> and .../put/10 <.../host/warp/lazy_without_req> failed; everything else passed (10201 passed, 20 skipped, no timeouts, no signals).

Root cause: The failure signature is highly selective and

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants