Skip to content

AZP/TESTS: Run all test applications under a timeout - #12337

Open
Alexey-Rivkin wants to merge 1 commit into
openucx:masterfrom
Alexey-Rivkin:ci-test-app-timeout
Open

Alexey-Rivkin wants to merge 1 commit into
openucx:masterfrom
Alexey-Rivkin:ci-test-app-timeout

Conversation

@Alexey-Rivkin

@Alexey-Rivkin Alexey-Rivkin commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

What?

Run every test application in contrib/test_jenkins.sh under a timeout.

Why?

Only the main gtest runs have a timeout. When an example application, ucx_perftest, ucx_info or test_ucx_tls.py hangs, the job sits idle until the 6-hour Azure job timeout and holds the agent all that time. Recent BlueField cases: 144735 (uct_hello_world), 144700 (io_demo), 143124 (ucp_client_server), 143306 (ucx_info).

This does not fix the hangs, it makes them fail in 20 minutes with the application named in the log.

How?

  • APP_TIMEOUT: timeout -k 1m 20m under Azure, empty for interactive runs. The slowest of these commands in a passing run takes under 3 minutes.
  • wait_app: waits for a background application with timeout tail --pid, so the pid the script signals stays the application itself. On timeout it sends SIGABRT to get a UCX backtrace, then SIGKILL, and fails the script. Used in run_client_server_app, run_loopback_app and for the perftest daemons.
  • ucx_info, ucx_perftest, mpirun and the small test programs run under $APP_TIMEOUT directly.
  • The gtest runs that had no timeout (memtrack, watchdog, test_arch) run under the existing $TIMEOUT.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread contrib/test_jenkins.sh Resolved
Comment thread contrib/test_jenkins.sh Resolved
Comment thread contrib/test_jenkins.sh Resolved
Comment thread contrib/test_jenkins.sh Resolved
@Alexey-Rivkin Alexey-Rivkin changed the title AZP/TESTS: Add timeout to non-gtest test applications AZP/TESTS: Run all test applications under a timeout Nov 15, 2026
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread contrib/test_jenkins.sh Resolved
Comment thread contrib/test_jenkins.sh Resolved
Comment thread contrib/test_jenkins.sh Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

20m is uniform for every app; the longest runs (run_io_demo batches, per-device MPI ucx_perftest batches) are the ones to watch — if any legitimately exceeds it, this turns hangs into flaky failures. CI results on this PR are the only validation available for that.

expect_failure now reports "Expected error not found" when the command times out (exit 124), which is a slightly misleading message, but it still fails the step correctly.

$TIMEOUT (200m) added in run_gtest_watchdog_test can never fire, since the test itself asserts a runtime under 300s; harmless, so not worth a comment.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@Alexey-Rivkin

Copy link
Copy Markdown
Contributor Author

20m is uniform for every app; the longest runs (run_io_demo batches, per-device MPI ucx_perftest batches) are the ones to watch — if any legitimately exceeds it, this turns hangs into flaky failures.

The limit is per process, not per batch. In the 36 passing test_jenkins.sh logs of build 143352 the longest single run is an mpirun at 73 s, and the longest client/server app is 16 s.

Comment thread contrib/test_jenkins.sh Resolved
Comment thread contrib/test_jenkins.sh Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gap: validation is limited to this PR's CI run — nothing here proves the 20m budget is enough for the longest batches, and the hang path (kill -ABRT → backtrace → kill -9) is not exercised by any job unless something actually hangs.

Only the main gtest runs have a timeout. The example and perftest
applications, ucx_info, the MPI runs and the small test programs run
unbounded, so when one of them hangs the job sits idle until the 6-hour
Azure job timeout, holds the agent for all that time, and the log does
not show which application hung.

Limit each of these applications to 20 minutes under Azure, far above
their normal run time of up to 3 minutes, and run the remaining gtest
invocations under the existing gtest timeout. Background applications
are waited for with "timeout tail --pid" instead of being started under
timeout(1), so the pid the script signals stays the application itself.
A timed-out background application gets SIGABRT before SIGKILL, so the
UCX error handler prints the backtrace of the hang, and the script
exits with an error even where the exit code of the application is
ignored.

Signed-off-by: Alexey Rivkin <arivkin@nvidia.com>
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@Alexey-Rivkin

Copy link
Copy Markdown
Contributor Author

Residual coverage gap: validation is limited to this PR's CI run — nothing here proves the 20m budget is enough for the longest batches, and the hang path (kill -ABRT → backtrace → kill -9) is not exercised by any job unless something actually hangs.

Both were checked by hand. Budget: the longest single process in the 36 passing test_jenkins.sh logs of build 143352 is an mpirun at 73 s. Hang path: wait_app was run against a ucx_perftest server left waiting for a client (backtrace printed, exit 134) and against a process that ignores SIGABRT (killed after 10 s), and the script exited with 1 in both cases.

Comment thread contrib/test_jenkins.sh Resolved
Comment thread contrib/test_jenkins.sh Resolved
@Alexey-Rivkin
Alexey-Rivkin marked this pull request as ready for review November 17, 2026 10:22
@Alexey-Rivkin
Alexey-Rivkin requested a review from a team as a code owner November 17, 2026 10:22
@svc-ucx

svc-ucx commented Nov 17, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (AddressSanitizer roce on worker 0) · commit c5fa342e

TL;DR: The ASAN/RoCE job was killed by the gtest watchdog after rc_verbs/test_uct_peer_failure_multiple.test/0 hung for 15 minutes in the new "SRQ disabled" rc_verbs task — AMs to freshly created receivers were never delivered (m_am_count==0 four times, 200 s apart), i.e. receiver QPs ended up with no posted receive WRs and the sender retried RNR forever (rnr_retry=7 = infinite). Fix in the no-SRQ receive path (prepost/repost the full RX_MAX_WR and guarantee empty-QP refill) and use a finite RNR retry so this fails fast instead of hanging; it is not a timeout that needs raising, and it is unrelated to this PR's CI-timeout change.

Full analysis

Summary: run_rc_verbs_nosrq_gtest (task 4/5) aborted: rc_verbs/test_uct_peer_failure_multiple.test/0 <rc_verbs/mlx5_0:1> hung, the gtest watchdog raised Connection timed out - abort testing and SIGABRT'd the process (make: *** [Makefile:5015: test] Aborted (core dumped)).

Root cause: Evidence from the stage log:

  • The task runs with UCX_RC_VERBS_SRQ_ENABLE=n UCX_RC_VERBS_RX_CQ_LEN=262144 UCX_RC_VERBS_RX_MAX_WR=64.
  • Test /1 (mlx5_1:1) passed in 27 s; test /0 started 12:59:37 and produced four identical failures at test/gtest/uct/test_peer_failure.cc:194 (m_am_count[index] is 0, expected 1) at 13:02:57, 13:06:18, 13:09:38, 13:12:59 — i.e. gaps of ~200 s each, which is the wait_for_flag deadline, not progress. At 13:14:36 (≈899 s after test start) the watchdog with watchdog_timeout = 900. (test/gtest/common/test_helpers.cc:31) timed out and did ADD_FAILURE() << strerror(ETIMEDOUT) << " - abort testing" + pthread_kill (line 57); the backtrace shows the watched thread inside libuct_ib (the UCT progress loop). So this is a hang, not a slow test.
  • The failures are inside test_uct_peer_failure::new_receiver() → send_recv_am(): a single AM to a brand-new receiver QP was posted successfully but never delivered. In the no-SRQ path each QP only preposts two batches (2 * rx_max_batch = 32 of RX_MAX_WR=64, rc_verbs_ep.c:701-708) and depends on RX completions / the rx_repost_needed scan (rc_verbs_iface.c:594-618) to refill. test_uct_peer_failure_multiple creates tx_queue_len receiver ifaces, each reserving a 262144-entry RX CQ plus per-QP buffers — under ASAN this starves the RX descriptor pool, leaving QPs with zero posted receives. Because RNR_RETRY_COUNT defaults to 7 = infinite (src/uct/ib/rc/base/rc_iface.c:52, applied as qp_attr.rnr_retry at line 964), the sender retries RNR indefinitely instead of failing, so the AM never arrives and the test hangs until the watchdog kills it.
  • A maintainer workaround for exactly this is already in flight: PR AZP/TESTS: Limit RNR retries in the no-SRQ gtest task [test only, do not merge] #12324 "AZP/TESTS: Limit RNR retries in the no-SRQ gtest task".

Implicated commit: 75371a03 — "UCT/IB/RC/VERBS: Support QPs without SRQ (#12066)", Guangguan Wang (added both the no-SRQ receive path and the run_rc_verbs_nosrq_gtest CI task). The PR head commit [REDACTED:Hex High Entropy String] (Alexey Rivkin, "AZP/TESTS: Run all test applications under a timeout") is unrelated to this failure.

File: src/uct/ib/rc/verbs/rc_verbs_ep.c:691-711 (prepost of only 2 batches) and src/uct/ib/rc/verbs/rc_verbs_iface.c:594-618 (uct_rc_verbs_iface_repost_empty_qps_no_srq); symptom at test/gtest/uct/test_peer_failure.cc:194; CI task at contrib/test_jenkins.sh:1317-1335.

Suggested fix:

  1. In the no-SRQ path, prepost the full rx_max_wr (or at least fc_max_wnd_size) per QP instead of 2 batches, and make the empty-QP refill robust: if uct_rc_verbs_post_recv_common() posts nothing, keep the QP queued for retry from a periodic/async progress callback rather than only from RX CQ polls, and fail EP creation loudly if a QP can end up with zero posted receives.
  2. Set a finite rnr_retry (or report UCS_ERR_ENDPOINT_TIMEOUT on RNR exhaustion) when SRQ is disabled, so a starved RQ surfaces as an endpoint error in seconds instead of an infinite RNR stall.
  3. Short term, un-break CI by landing AZP/TESTS: Limit RNR retries in the no-SRQ gtest task [test only, do not merge] #12324 (limit RNR retries in the no-SRQ task) and/or reducing UCX_RC_VERBS_RX_CQ_LEN / capping the number of receivers in run_rc_verbs_nosrq_gtest so the RX descriptor pool is not exhausted under ASAN. Do not raise the 200 m step timeout or the 900 s gtest watchdog — the process was hung, not slow.

Related: #12324 (RNR retry workaround for this exact task), #12066 and #12061 (no-SRQ support), #12337 (PR under test, CI-only).

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id c20a5126-4872-490f-8cfe-efe387a6ce19 in the triage console for the audit trail.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants