Repository navigation
Conversation
|
🤖 Starting review — findings will be posted here when done. |
|
Residual coverage gap: the new |
|
🤖 CI Triage Agent — TL;DR: The gtest binary aborted on a UCS rcache assertion — Full analysisSummary: Azure job "Tests new on worker 1" (build 144486) failed in the gtest step: Root cause: From the backtrace: The rcache ownership contract (see Two concrete suspect sites in the code I read:
Note the failing branch ( Implicated commit: File: Suggested fix:
Related: #11958 (merge adjacent rcache regions — implicated), #12267 (would enable merging by default — hold until this is fixed), #12104 (rcache explicit registration, same area). |
william-gallagher-nv
left a comment
There was a problem hiding this comment.
@shasson5 Looks good. A couple of minor points that you can address if you think appropriate. I also wanted to confirm the tests pass when running on Blossom.
| if (status == UCS_OK) { | ||
| wait_for_bits(&m_state, TEST_STATE_CLIENT_GOT_SERVER_UNAVAILABLE, | ||
| 300); | ||
| EXPECT_TRUE(m_state & TEST_STATE_CLIENT_GOT_SERVER_UNAVAILABLE); | ||
| } else { | ||
| EXPECT_EQ(UCS_ERR_UNREACHABLE, status); | ||
| } |
There was a problem hiding this comment.
Just curious, why rework the validation here?
| /* TCP_SYNCNT may still leave SYN retransmission timeouts in the | ||
| * minutes range on some systems. Bound TCP_USER_TIMEOUT as well when | ||
| * the transport supports it. */ | ||
| modify_config("TCP_CM_USER_TIMEOUT", "1s", IGNORE_IF_NOT_EXIST); |
There was a problem hiding this comment.
The RDMACM tests take 10 seconds to fail. One possibility is to set RDMA_CM_TIMEOUT to 1 seconds like the TCP case. Feel free to ignore as out of scope.
What?
Combine #12213 and #12216: match unreachable destinations to the source address family, handle immediate route errors, and bound worker batches while checking all affinity CPUs.
Why?
Prevent address-family mismatches and descriptor exhaustion in Blossom runs.
HPCINFRA-5026, HPCINFRA-5031.