Skip to content

UCM/BISTRO: Wait for syscall completion before patching - #11088

Closed
shasson5 wants to merge 10 commits into
openucx:masterfrom
shasson5:hook_race
Closed

shasson5 wants to merge 10 commits into
openucx:masterfrom
shasson5:hook_race

Conversation

@shasson5

@shasson5 shasson5 commented Jan 1, 2026 •

Copy link
Copy Markdown
Contributor

What?

Wait for syscall completion before patching

Why?

Reduce race condition if a syscall is in progress while trying to patch a function

solves https://redmine.mellanox.com/issues/4739519

@shasson5 shasson5 added the Bugfix label Jan 1, 2026
@shasson5
shasson5 requested review from gleon99 and yosefe January 1, 2026 11:14
@gleon99
gleon99 requested a review from amastbaum January 4, 2026 09:53
@gleon99

gleon99 commented Jan 4, 2026

Copy link
Copy Markdown
Contributor

@amastbaum please review

Comment thread src/ucm/mmap/install.c Resolved
Comment thread src/ucm/bistro/bistro_x86_64.h Resolved
Comment thread src/ucs/sys/sys.c Resolved
Comment thread src/ucs/sys/sys.c Resolved
Comment thread src/ucm/bistro/bistro.c Resolved
Comment thread src/ucm/bistro/bistro_rv64.h Resolved
Comment thread src/ucm/bistro/bistro_x86_64.h Resolved
Comment thread test/gtest/ucm/malloc_hook.cc Resolved
Comment thread src/ucm/util/sys.c Outdated

fp = fopen(syscall_file, "r");
if (fp == NULL) {
ucm_error("failed to open %s", syscall_file);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
ucm_error("failed to open %s", syscall_file);
ucm_error("failed to open %s: %m", syscall_file);

Comment thread src/ucm/util/sys.c Outdated
fclose(fp);

if (res == NULL) {
ucm_error("failed to read %s", syscall_file);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
ucm_error("failed to read %s", syscall_file);
ucm_error("failed to read %s: %m", syscall_file);

Comment thread src/ucm/util/sys.c
}
}

out:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: out label is optional, break would suffice

Comment thread src/ucm/bistro/bistro.c Resolved
Comment thread test/gtest/ucm/malloc_hook.cc Resolved
@shasson5
shasson5 requested review from iyastreb and tvegas1 January 7, 2026 10:03

@tvegas1 tvegas1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor comments

Comment thread src/ucm/util/sys.h Resolved
Comment thread src/ucm/util/sys.c Resolved
Comment thread src/ucm/util/sys.c Resolved
Comment thread src/ucm/bistro/bistro.c Outdated
break;
}

ucm_bistro_wait_sec(0.5);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe make it shorter time? or use grace_duration ?

Comment thread src/ucm/util/sys.c Outdated
snprintf(syscall_file, sizeof(syscall_file), "%s/%s/syscall", task_dir,
entry->d_name);

fp = fopen(syscall_file, "r");

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. we cannot call any functions that may allocate memory to avoid deadlock with malloc(), and in general better avoid any comples C library calls
  2. IMO better ust procfs to check instruction pointer of other threads (it's possible) and wait until it's outside the patched function code

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems procfs doesn't give instruction pointer if process is "not blocked"

@shasson5
shasson5 requested review from tvegas1 and yosefe January 8, 2026 10:17
Comment thread src/ucm/util/sys.c
0) {
for (bpos = 0; bpos < nread;) {
entry = (struct linux_dirent*)(dir_buf + bpos);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe first make sure it is DT_DIR for future proof?

Comment thread src/ucm/util/sys.c
(curr_syscall == syscall_num)) {
found = 1;
break;
if (read_bytes < 0) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also return error if read_bytes == sizeof(line)-1 as it might have been truncated?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems redundant IMO (and also sscanf would fail in that case)

Comment thread src/ucm/util/sys.c Outdated
fd = open(syscall_file, O_RDONLY);
if (fd < 0) {
ucm_error("failed to open %s: %m", syscall_file);
break;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we can't read some thread data, I think it's normal because thread might have been terminated. We should not break, but continue?

Comment thread src/ucm/util/sys.c Outdated

if (read_bytes < 0) {
ucm_error("failed to read %s: %m", syscall_file);
break;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here

Comment thread src/ucm/util/sys.c Outdated
for (bpos = 0; bpos < nread;) {
entry = (struct linux_dirent64*)(dir_buf + bpos);

if ((entry->d_name[0] == '.') || (entry->d_type != DT_DIR)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

d_type may be DT_UNKNOWN on some systems, maybe we skip this condition

Comment thread src/ucm/util/sys.c Outdated
if ((sscanf(line, "%d", &curr_syscall) == 1) &&
(curr_syscall == syscall_num)) {
found = 1;
break;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this breaks only the inner loop?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants