Skip to content

Device API V2 - #11123

Merged
yosefe merged 13 commits into
openucx:masterfrom
ofirfarjun7:topic/device-api-v2
Jan 24, 2026
Merged

yosefe merged 13 commits into
openucx:masterfrom
ofirfarjun7:topic/device-api-v2

Conversation

@ofirfarjun7

@ofirfarjun7 ofirfarjun7 commented Jan 19, 2026 •

Copy link
Copy Markdown
Contributor

What?

Device API V2 following new design

Why?

Gives more flexibility to the App while maintaining less GPU handles

How?

In this API the mem list elements is split into two lists, one for local and one for remote.

Comment on lines +279 to +293
const void *address;
const uct_device_mem_element_t *uct_elem;
const uct_device_local_mem_list_elem_t *src_uct_elem;
uint64_t remote_address;
uct_device_completion_t *comp;
uct_device_ep_t *device_ep;
ucs_status_t status;

status = ucp_device_prepare_send(src_mem_list_h, src_mem_list_index,
dst_mem_list_h, dst_mem_list_index,
address, remote_address, req, device_ep,
src_uct_elem, uct_elem, comp);
if (status != UCS_OK) {
return status;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's better to flatten the function instead of calling ucp_device_prepare_send which also calls ucp_device_prepare_send_remote. In the following code it reduces the latency in ~300 nsec.

Suggested change
const void *address;
const uct_device_mem_element_t *uct_elem;
const uct_device_local_mem_list_elem_t *src_uct_elem;
uint64_t remote_address;
uct_device_completion_t *comp;
uct_device_ep_t *device_ep;
ucs_status_t status;
status = ucp_device_prepare_send(src_mem_list_h, src_mem_list_index,
dst_mem_list_h, dst_mem_list_index,
address, remote_address, req, device_ep,
src_uct_elem, uct_elem, comp);
if (status != UCS_OK) {
return status;
}
/* Validate parameters upfront */
if ((src_mem_list_h->version != UCP_DEVICE_MEM_LIST_VERSION_V1) ||
(src_mem_list_index >= src_mem_list_h->mem_list_length) ||
(dst_mem_list_h->version != UCP_DEVICE_MEM_LIST_VERSION_V1) ||
(dst_mem_list_index >= dst_mem_list_h->mem_list_length)) {
return UCS_ERR_INVALID_PARAM;
}
/* Calculate element sizes - compiler should constant-fold these */
const size_t local_elem_size = ucs_offsetof(uct_device_local_mem_list_elem_t,
uct_mem_element) +
sizeof(uct_tl_device_mem_element_t);
const size_t remote_elem_size = ucs_offsetof(uct_device_remote_mem_list_elem_t,
uct_mem_element) +
sizeof(uct_tl_device_mem_element_t);
/* Get remote element - contains device_ep, remote_addr, and rkey */
const auto *dst_elem = (const uct_device_remote_mem_list_elem_t*)
UCS_PTR_BYTE_OFFSET(dst_mem_list_h->mem_elements,
dst_mem_list_index * remote_elem_size);
/* Get local element - contains local_addr and lkey */
const auto *src_elem = (const uct_device_local_mem_list_elem_t*)
UCS_PTR_BYTE_OFFSET(src_mem_list_h->mem_elements,
src_mem_list_index * local_elem_size);
/* Initialize completion inline */
uct_device_completion_t *comp = nullptr;
if (req != nullptr) {
comp = &req->comp;
req->device_ep = dst_elem->device_ep;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought prepare send was inline anyways, is it also happening with optimizations on?

Comment thread src/ucp/api/device/ucp_device_impl.h Resolved
Comment thread src/ucp/core/ucp_device.c Resolved
Comment thread src/ucp/api/device/ucp_device_types.h Resolved
Comment thread src/ucp/api/device/ucp_device_types.h Resolved
Comment thread src/ucp/api/device/ucp_device_impl.h Resolved
Comment thread src/ucp/api/device/ucp_device_impl.h Resolved
Comment thread src/uct/ib/mlx5/gdaki/gdaki.c Resolved
Comment thread src/uct/api/v2/uct_v2.h Resolved
Comment thread src/ucp/api/device/ucp_host.h Resolved
Comment thread src/ucp/api/device/ucp_host.h Resolved
Comment thread src/ucp/api/device/ucp_device_impl.h Resolved
Comment thread src/ucp/core/ucp_device.c Resolved
Comment thread src/ucp/core/ucp_device.c Resolved
Comment thread src/ucp/core/ucp_device.c Resolved
Comment thread src/ucp/api/device/ucp_device_impl.h Resolved
Comment thread src/ucp/api/device/ucp_device_impl.h Resolved
Comment thread src/uct/api/device/uct_device_impl.h Resolved
Comment thread src/uct/api/device/uct_device_impl.h Resolved
Comment thread src/uct/api/device/uct_device_impl.h Resolved
Comment thread src/ucp/api/device/ucp_device_impl.h Resolved
Comment thread src/ucp/api/device/ucp_host.h Resolved
Comment thread src/ucp/api/device/ucp_host.h Resolved
Comment thread src/ucp/api/device/ucp_host.h Resolved
Comment thread src/ucp/api/device/ucp_host.h Resolved
Comment thread src/ucp/core/ucp_device.c Resolved
Comment thread src/ucp/core/ucp_device.c Resolved
Comment thread src/ucp/core/ucp_device.c Resolved
Comment thread src/uct/api/device/uct_device_types.h Resolved
Comment thread src/uct/api/device/uct_device_types.h Resolved
Comment thread src/ucp/api/device/ucp_device_impl.h Outdated
Comment on lines +118 to +129
if ((dst_mem_list_h->version != UCP_DEVICE_MEM_LIST_VERSION_V1) ||
(dst_mem_list_index >= dst_mem_list_h->mem_list_length)) {
return UCS_ERR_INVALID_PARAM;
}

const uct_device_remote_mem_list_elem_t *dst_mem_element =
(uct_device_remote_mem_list_elem_t*)
UCS_PTR_BYTE_OFFSET(dst_mem_list_h->mem_elements,
dst_mem_list_index * elem_size);
remote_address = dst_mem_element->remote_addr;
device_ep = dst_mem_element->device_ep;
uct_elem = &dst_mem_element->uct_mem_element;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nextpr: use __ldg for mem handle read access

Comment thread src/ucp/api/device/ucp_device_impl.h Outdated
Comment on lines +152 to +155
if ((src_mem_list_h->version != UCP_DEVICE_MEM_LIST_VERSION_V1) ||
(src_mem_list_index >= src_mem_list_h->mem_list_length)) {
return UCS_ERR_INVALID_PARAM;
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nextpr: let's see if we can optimize some of these validations

Comment thread src/ucp/api/device/ucp_device_impl.h Outdated
template<ucs_device_level_t level = UCS_DEVICE_LEVEL_THREAD>
UCS_F_DEVICE ucs_status_t ucp_device_counter_inc(
ucp_device_remote_mem_list_handle_h dst_mem_list_h,
unsigned dst_mem_list_index, uint64_t inc_value, size_t remote_offset,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uint64_t inc_value should be 1st param, to keep it consistent that source comes before destination

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why you consider inc_value as "source"?
In all API first param is always the handle.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the firs param is usually the source handle
here inc_value is the source because this is the data send from local to remote side

Comment thread src/ucp/api/device/ucp_device_types.h Outdated
* Protocol index computed by host handle management functions when
* creating handle.
*/
uint8_t proto_idx;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems unused

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it is unused. inheritance from API V1.

Comment thread src/ucp/api/device/ucp_device_types.h Resolved
Comment thread src/ucp/core/ucp_device.c Outdated
return status;
}

handle_size = sizeof(handle) + uct_elem_size * params->num_elements;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use ( )

Comment thread src/ucp/core/ucp_device.c Outdated
Comment on lines +1098 to +1100
if (handle == NULL) {
return;
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why needed?

Comment thread src/uct/api/uct_def.h Outdated
Comment on lines +111 to +113
typedef int uct_worker_cb_id_t;
typedef void *uct_conn_request_h;
typedef struct uct_device_ep *uct_device_ep_h;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

align on column

Comment thread src/uct/ib/mlx5/gdaki/gdaki.c Outdated
Comment on lines +656 to +657
if (invalid) {
ucs_error("Invalid parameters: memh=%p, rkey=%lu\n", memh, rkey);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why need this error?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This func expect to get memh, rkey or both.

Comment thread test/gtest/ucp/test_ucp_device.cc Resolved
Comment thread test/gtest/ucp/test_ucp_device.cc Resolved
Comment thread test/gtest/ucp/test_ucp_device.cc Resolved
Comment thread test/gtest/ucp/test_ucp_device.cc Resolved
Comment thread src/ucp/api/device/ucp_device_impl.h Resolved

@tvegas1 tvegas1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cosmetic review

Comment thread src/ucp/api/device/ucp_device_impl.h Resolved
Comment thread src/ucp/api/device/ucp_device_impl.h Resolved
Comment on lines +279 to +293
const void *address;
const uct_device_mem_element_t *uct_elem;
const uct_device_local_mem_list_elem_t *src_uct_elem;
uint64_t remote_address;
uct_device_completion_t *comp;
uct_device_ep_t *device_ep;
ucs_status_t status;

status = ucp_device_prepare_send(src_mem_list_h, src_mem_list_index,
dst_mem_list_h, dst_mem_list_index,
address, remote_address, req, device_ep,
src_uct_elem, uct_elem, comp);
if (status != UCS_OK) {
return status;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought prepare send was inline anyways, is it also happening with optimizations on?

*/
uint32_t
ucp_device_get_mem_list_length(const ucp_device_mem_list_handle_h handle);
ucp_device_get_mem_list_length(const void* handle);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

enhance doxygen to detail that param can either be local or remote mem list handle?

Comment thread src/uct/api/device/uct_device_impl.h Resolved
Comment thread src/ucp/core/ucp_device.c Resolved
Comment thread src/ucp/core/ucp_device.c Resolved
Comment thread src/ucp/core/ucp_device.c Resolved
Comment thread src/ucp/core/ucp_device.c Resolved
Comment thread src/ucp/core/ucp_device.c Resolved
rakhmets
rakhmets previously approved these changes Jan 22, 2026
Comment thread src/ucp/api/device/ucp_device_impl.h Resolved
Comment thread src/ucp/api/device/ucp_device_impl.h Resolved
Comment on lines +276 to +278
unsigned src_mem_list_index, size_t src_offset,
ucp_device_remote_mem_list_handle_h dst_mem_list_h,
unsigned dst_mem_list_index, size_t dst_offset, size_t length,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Q: Why do we need usigned type? It looks like they can be size_t.

Comment thread src/ucp/core/ucp_device.c Resolved
Comment thread src/ucp/core/ucp_device.c Resolved
rakhmets
rakhmets previously approved these changes Jan 23, 2026
Comment on lines +10 to +12
#ifdef HAVE_CONFIG_H
#include "config.h"
#endif

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using the macro in public headers breaks compilation of external projects.
Perhaps we cannot implement parameters check in this way.

Comment thread src/ucp/api/device/ucp_host.h Resolved
#define ENABLE_PARAMS_CHECK 0
#endif

#define UCP_DEVICE_MEM_LIST_PARAMS_CHECK(_mem_list_h, _mem_list_index) \

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nextpr: could be inline function that return ok in release mode


UCP_DEVICE_MEM_LIST_PARAMS_CHECK(dst_mem_list_h, dst_mem_list_index);

const uct_device_remote_mem_list_elem_t *dst_mem_element =

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nextpr: auto

Comment thread src/ucp/api/device/ucp_device_impl.h Resolved
Comment thread src/ucp/api/device/ucp_device_types.h Resolved
/**
* Number of entries in the memory descriptors array @a elems.
*/
uint32_t mem_list_length;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nextpr: length

*/
uint32_t
ucp_device_get_mem_list_length(const ucp_device_mem_list_handle_h handle);
ucp_device_get_mem_list_length(const void* handle);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nextpr:

Suggested change
ucp_device_get_mem_list_length(const void* handle);
ucp_device_get_mem_list_length(const void* mem_list_h);

* @param [in] handle Created handle to release.
*/
void ucp_device_mem_list_release(ucp_device_mem_list_handle_h handle);
void ucp_device_mem_list_release(void *handle);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nextpr:

Suggested change
void ucp_device_mem_list_release(void *handle);
void ucp_device_mem_list_release(void *mem_list_h);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Require API change. You can decide if to merge it or not:
#11145

Comment thread src/ucp/api/ucp_def.h
Comment on lines +879 to +882
typedef struct ucp_device_local_mem_list_handle
*ucp_device_local_mem_list_handle_h;
typedef struct ucp_device_remote_mem_list_handle
*ucp_device_remote_mem_list_handle_h;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
typedef struct ucp_device_local_mem_list_handle
*ucp_device_local_mem_list_handle_h;
typedef struct ucp_device_remote_mem_list_handle
*ucp_device_remote_mem_list_handle_h;
typedef struct ucp_device_local_mem_list *ucp_device_local_mem_list_h;
typedef struct ucp_device_remote_mem_list *ucp_device_remote_mem_list_h;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Require API change. You can decide if to merge it or not:
#11145

uct_rc_gdaki_device_mem_element_t gdaki_mem_element;
uct_cuda_ipc_device_mem_element_t cuda_ipc_mem_element;
};

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nextpr: 2 space lines in H file

{
rte_comm c0to1, c1to0;
ucx_perf_params_t params;
ucx_perf_params_t params{};

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nextpr: why need to initialize? can we fix test_params_init to initialize all fields?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not related to this PR but Coverity for some reason start complaining about it.
I want to quickly fix it to pass CI.

@yosefe
yosefe merged commit f3c91fa into openucx:master Jan 24, 2026
147 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants