Repository navigation
AZP/TESTS: Run all test applications under a timeout - #12337
Alexey-Rivkin wants to merge 1 commit into
Conversation
|
🤖 Starting review — findings will be posted here when done. |
e0e50e3 to
843d129
Compare
|
🤖 Starting review — findings will be posted here when done. |
|
20m is uniform for every app; the longest runs (
|
843d129 to
63ac347
Compare
|
🤖 Starting review — findings will be posted here when done. |
The limit is per process, not per batch. In the 36 passing |
|
Residual coverage gap: validation is limited to this PR's CI run — nothing here proves the 20m budget is enough for the longest batches, and the hang path ( |
Only the main gtest runs have a timeout. The example and perftest applications, ucx_info, the MPI runs and the small test programs run unbounded, so when one of them hangs the job sits idle until the 6-hour Azure job timeout, holds the agent for all that time, and the log does not show which application hung. Limit each of these applications to 20 minutes under Azure, far above their normal run time of up to 3 minutes, and run the remaining gtest invocations under the existing gtest timeout. Background applications are waited for with "timeout tail --pid" instead of being started under timeout(1), so the pid the script signals stays the application itself. A timed-out background application gets SIGABRT before SIGKILL, so the UCX error handler prints the backtrace of the hang, and the script exits with an error even where the exit code of the application is ignored. Signed-off-by: Alexey Rivkin <arivkin@nvidia.com>
63ac347 to
c5fa342
Compare
|
🤖 Starting review — findings will be posted here when done. |
Both were checked by hand. Budget: the longest single process in the 36 passing |
|
🤖 CI Triage Agent — TL;DR: The ASAN/RoCE job was killed by the gtest watchdog after Full analysisSummary: Root cause: Evidence from the stage log:
Implicated commit: File: Suggested fix:
Related: #12324 (RNR retry workaround for this exact task), #12066 and #12061 (no-SRQ support), #12337 (PR under test, CI-only).
|
What?
Run every test application in
contrib/test_jenkins.shunder a timeout.Why?
Only the main gtest runs have a timeout. When an example application,
ucx_perftest,ucx_infoortest_ucx_tls.pyhangs, the job sits idle until the 6-hour Azure job timeout and holds the agent all that time. Recent BlueField cases: 144735 (uct_hello_world), 144700 (io_demo), 143124 (ucp_client_server), 143306 (ucx_info).This does not fix the hangs, it makes them fail in 20 minutes with the application named in the log.
How?
APP_TIMEOUT:timeout -k 1m 20munder Azure, empty for interactive runs. The slowest of these commands in a passing run takes under 3 minutes.wait_app: waits for a background application withtimeout tail --pid, so the pid the script signals stays the application itself. On timeout it sends SIGABRT to get a UCX backtrace, then SIGKILL, and fails the script. Used inrun_client_server_app,run_loopback_appand for the perftest daemons.ucx_info,ucx_perftest,mpirunand the small test programs run under$APP_TIMEOUTdirectly.test_arch) run under the existing$TIMEOUT.