Skip to content

fix(vllm_model): opt-in time cap on connection retries to a dead model server - #5181

Merged
wprazuch merged 5 commits into
mainfrom
wprazuch/dead-policy-server-fails-run
Nov 9, 2026
Merged

wprazuch merged 5 commits into
mainfrom
wprazuch/dead-policy-server-fails-run

Conversation

@wprazuch

@wprazuch wprazuch commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

What goes wrong

vllm_model on a static base_url retries a refused connection without a time limit. When the policy engine dies mid-run, every in-flight model call hangs until something above it gives up, and an evaluation can stall for a long time before anything reports the dead server.

What this changes

An opt-in endpoint_connection_retry_timeout_s on vllm_model (connection_retry_timeout_s on NeMoGymAsyncOpenAI) caps the time one model call spends retrying connection errors. Past the cap, the original connection error is raised. The default null keeps today's unlimited retries, so training can still ride out an engine restart.

  • The window opens at the call's first connection error and closes when the server answers. It bounds only an unbroken run of connection failures, so a reachable server that answers 5xx is never reported as "connection refused".
  • While the window is open, each reconnect, including the wait for a pool slot, is bounded by the time left (ClientTimeout.connect). Expiry is checked before another attempt, and the backoff is cut to the time left. A connected attempt is never cut short.
  • The new argument is keyword-only on request(), _traced_request() and _request_with_retries(), so positional callers keep their meaning.

This supersedes #4853 (same option).

An earlier revision also made verifiers_agent fail /run on a policy-call failure. That is dropped: since #5679, main already records such a rollout as a typed, masked model_transport_error / model_request_error row that the outcome policy acts on, and the raise would have replaced it with an untyped agent_run_error. If a dead server should stop a run, that belongs in the outcome policy for model_transport_error.

Not changed

  • verifiers_agent matching APITimeoutError as a dead server also fires on a slow but live server. That is existing behaviour on main and is left for a separate change.
  • Each model call gets its own window, so a caller that retries the failed call (for example the OpenAI SDK's max_retries in an agent's policy client) waits up to one window per retry. The docs say so, and say to set the cap above the engine's startup time.

Tests

  • tests/unit_tests/client/test_http_client.py: a refused port fails within the window (with and without tracing) and retries forever without one. A reconnect that hangs fails at the deadline with the saved connection error. A slow answer after a reconnect still succeeds. A spent window raises before connecting again. New: an answer closes the window, so a later retry after the window would have run out connects instead of raising the earlier "refused" (fails without the fix). Positional arguments keep their meaning.
  • tests/unit_tests/client/test_openai_client.py: a 503 after a connection error keeps its full Retry-After wait and the call succeeds; an unopened window keeps the full status wait; no window by default.
  • responses_api_models/vllm_model/tests/test_app.py: the option reaches every client, including rebuilt endpoint clients, and a dead engine fails a real call within the window.

Results: tests/unit_tests/client/test_http_client.py + test_openai_client.py 259 passed; vllm_model 507 passed; the full tests/unit_tests run shows no new failures compared with main in the same environment (the six test_super35_batch_launcher prebuilt-environment cases are skipped on a fresh main checkout, and test_short_http_reply_is_a_package_store_error passes when run alone). pre-commit clean on the changed files.

Rollout evidence: N/A. The option is off by default, and with it unset no request path changes. The only behaviour change, the verifiers_agent raise, has been removed.

@nemo-automation-bot
nemo-automation-bot Bot requested review from a team October 28, 2026 11:38
@svcnvidia-nemo-ci svcnvidia-nemo-ci added bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer labels Oct 28, 2026

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP WITH CARE — two RISKs and two NOTEs; no BLOCKERs.

Completed static review of all nine changed files and their directly affected paths. Tests were not executed in this review.

RISK — Representative model-rollout evidence is missing

Could you add real smoke-rollout evidence for the new default in VerifiersAgent.responses? The supplied validation lists unit tests only, while AGENTS.md requires representative model rollouts and inspection of agent/verifier behavior for behavior-changing agent work.

Evidence gap: The new test fixture replaces run_group and supplies an error dictionary directly. It verifies the handler's reaction, but not whether the installed environment exports an actual mid-rollout policy failure in the shape _model_server_failure consumes. The external verifiers implementation is not mounted, so that production boundary remains unverified—not a demonstrated implementation defect.

Correction and preservation: Record the environment, model, configuration, commands and inspected results in the PR validation. No implementation patch is justified by this evidence gap alone. Healthy rewards and trajectories must remain valid; model-output and tool errors must remain scored; the opt-out must preserve the previous scoring behavior.

Acceptance — proposed, not executed: Run a healthy representative rollout and inspect its completion, reward and trajectory. Interrupt the policy service during another real rollout and let the configured retries/timeouts finish: agent /run should return 500 rather than a successful scored result. Record the actual error chain. Retain the existing preservation assertions: opt-out returns 200 with reward 0.0, and model-output errors, tool connection errors and clean rollouts return 200.

NOTE — Suggestion: document the new failure-handling controls

The new opt-in retry deadline is only exposed as a null key in the exemplar YAML; the vLLM guide and verifiers-agent README do not explain these controls. Operators following those guides cannot discover how to enable bounded retries or restore scoring after the new default /run failure behavior. This is a documentation gap, not a runtime defect.

Could you add a short operational section to those two documents? Include a working configuration example for endpoint_connection_retry_timeout_s, its positive-value requirement, and the fail_rollout_on_model_server_error default/opt-out. Preserve these distinctions explicitly: the window starts at the first connection error, status retries share it, connected generations are not cut short, and null adds no time cap but does not remove existing attempt caps. Model-output/tool errors remain scored; the flag changes policy-server failure handling only.

Static acceptance: The guides contain usable configuration examples and match the client contract and agent flag contract, without implying a total generation deadline or unlimited retries when a finite attempt cap applies.


Inline comments (escaped v1):

  • nemo_gym/openai_utils.py line 1845: RISK — Bound status-retry backoff by the opened connection window.

    Failure: Set connection_retry_timeout_s=1. Let the first connection attempt fail, then reconnect and immediately receive 503 with Retry-After: 60. The new window is running, but this branch sleeps the full 60 seconds before the transport checks expiry. An opted-in call therefore remains pending roughly a minute beyond its one-second deadline even though no generation is in progress. The PR creates one shared window, but this existing status-wait branch does not consult it; the transport notices only at the next attempt. This affects calls that encounter both a connection failure and a retryable status.

    Correction: Could you replace this await sleep(wait) with the following, so retry-only waits cannot extend an opened window?

    window = request_kwargs.get("_connection_retry_window")
    remaining = window.remaining() if window is not None else None
    await sleep(wait if remaining is None else max(0.0, min(wait, remaining)))

    The next transport entry raises the saved connection error without connecting again. This is a source-reasoned suggestion; please verify it before adoption.

    Preservation: Keep ordinary Retry-After/exponential waits unchanged when there is no window or no connection error has opened it. Retain one window across status retries and a fresh one for the next logical call. Keep terminal-status/no-resend branches' no-sleep behavior. Do not impose a whole-request deadline: an already connected slow generation must still finish.

    Acceptance — proposed, not executed: Add a public-client regression using a connector that first raises a connection error, then reaches a server returning 503 with Retry-After: 60. With a one-second window, assert the saved error is raised near one second and no post-deadline network attempt occurs. Also assert that 503 → 200 without a preceding connection error honors the full ordinary wait, and that no-window behavior is unchanged. Retain the shared-window/fresh-next-call assertions and the slow connected-response success case.

  • tests/unit_tests/test_server_utils.py line 2251: NOTE — Suggestion: actually enable the traced timeout case.

    Coverage failure: tracing_enabled=True changes only the span-group gate. request() also requires is_trace_exporter_active(), whose initial state is false. An isolated run therefore exercises the untraced transport twice and cannot catch lost window forwarding through _traced_request. Current forwarding is correct by inspection; this is not a demonstrated runtime bug.

    Correction: Following the existing retry-test setup, add immediately after this monkeypatch:

    monkeypatch.setattr(nemo_gym.server_utils, "is_trace_exporter_active", lambda: tracing_enabled)
    monkeypatch.setattr("nemo_gym.telemetry.spans.client_span", lambda _: nullcontext(None))
    monkeypatch.setattr("nemo_gym.telemetry.contrib.inject_trace_context", lambda headers: None)

    This selects the intended branch without requiring optional telemetry dependencies. Please verify the suggestion before adopting it.

    Preservation and acceptance: Keep both parameter values and the real refused-port assertion: each must raise ClientConnectorError with 1.0 <= elapsed < 1.8. Static acceptance is that the true case satisfies both dispatch gates and traverses _traced_request, while the false case still selects _request_with_retries directly. Proposed checks only; tests were not executed here.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

@svcnvidia-nemo-ci svcnvidia-nemo-ci added the area:agent Agent harnesses and Responses API agent behavior label Oct 28, 2026
@nemo-automation-bot
nemo-automation-bot Bot requested a review from a team October 29, 2026 11:20
@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

@wprazuch

Copy link
Copy Markdown
Contributor Author

/review reset=true

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP WITH CARE — one RISK and one NOTE; no BLOCKERs.

Completed static review of all 11 changed files and their directly affected retry, client-construction, rollout and scoring paths. Tests were not executed.

The earlier status-backoff, traced-test and documentation findings are addressed: the status loop clips waits to an opened window; the refused-port test enables both tracing gates and verifies dispatch; and the vLLM guide and agent README document the controls.

RISK — Representative model-rollout evidence is still missing

Could you provide the real smoke-rollout evidence requested in review 5682476798? The supplied validation still lists unit tests only, while AGENTS.md requires representative model rollouts and inspection of agent/verifier behavior for behavior-changing agent work.

Evidence gap and affected behavior: The new default rejects a rollout before scoring based on its exported error chain. The test fixture replaces run_group and supplies that error directly. It verifies the handler's reaction, but not whether an actual mid-rollout policy failure reaches it in that shape. The external verifiers implementation is not mounted, so this remains an unverified production boundary—not a demonstrated implementation defect.

Correction and preservation: Record the representative environment, model, resolved configuration, commands and inspected results in the PR validation. No implementation patch is justified by this evidence gap alone. Preserve healthy rewards and trajectories, scoring of model-output/tool errors, and the flag's previous-behavior opt-out.

Acceptance — proposed, not executed: Run a healthy representative rollout and inspect its completion, reward and trajectory. During another real rollout, interrupt the policy service and allow configured retries/timeouts to finish; agent /run should return 500 rather than a successful scored result. Record the actual exported error chain. Retain the existing preservation assertions: opt-out returns 200 with reward 0.0, while model-output errors, tool connection errors and clean rollouts return 200.


Inline comments (escaped v1):

  • nemo_gym/server_utils.py line 944: NOTE — Suggestion: clip the generic-error backoff to the opened connection window too.

    Failure: The new window can still be exceeded by almost 0.5 seconds through this branch. For example, use a 0.6-second window: the first ClientOSError opens it, the connection-error branch sleeps 0.5 seconds, and the reconnect raises a ConnectionTimeoutError while budget remains. The expiry check then falls through, and this unconditional sleep delays the saved connection error until approximately one second. No generation is running during that extra wait. This is a bounded deadline-accuracy issue, not an unbounded hang.

    Correction: Could you replace this sleep in _request_with_retries with:

    await asyncio.sleep(_connection_backoff_s(window))

    The invariant is that retry-only backoff cannot extend an already opened window. Also update the neighboring diagnostic's hard-coded “Sleeping 0.5s” wording to describe the bounded backoff. This is a source-reasoned suggestion; please verify it before adoption.

    Preservation: The existing helper still returns 0.5 when no window exists or no connection error has opened it. Keep attempt-cap and no-resend exits before sleeping. Do not alter total/read timeouts or interrupt connected responses; retain the slow-generation case, which must return status 200 and body ok despite outliving the window.

    Acceptance — proposed, not executed: Add a deterministic regression with a controlled monotonic clock and sleep stub that advances it. Give the mock client a real ClientTimeout and make requests raise ClientOSError, then ConnectionTimeoutError before a 0.6-second window expires. Assert sleeps of 0.5 and approximately 0.1 seconds, the saved ClientOSError is raised, and no third transport request starts. Retain the no-window recovery assertion: success after exactly one 0.5-second sleep.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

Bound connection retries to a dead model server by a wall-clock window, cap status-retry and generic-error backoff at that window, and make verifiers_agent /run return 500 when the policy server failed the rollout.

Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

@AdamRajfer AdamRajfer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The client side looks good to me: the cap is opt-in, keyword-only, and the status retries share the window. I ran tests/unit_tests/client/test_http_client.py and test_openai_client.py on the PR head (258 passed). My concern is the verifiers_agent change, which I think collides with what #5679 put on main today.

  1. responses_api_agents/verifiers_agent/app.py:914. On current main, run() already handles a policy call that ends a rollout. _ModelFailureObservingClient records it as model_transport_error (no reply or timeout) or model_request_error plus the status (5xx). The row is masked, the agent-stage outcome policy picks the action (retry by default), and the failure report stays on the row. The new raise sits in responses(), which run() awaits before any of that. I ran the PR head with the flag off and on, for a timeout, a refused connection and a 503:

    • Flag off: 200, mask_sample: true, failure_kind model_transport_error / model_request_error, failure_outcome_action: retry.
    • Flag on, the default: 500 "Internal Server Error", no failure header.

    So this swaps a typed, masked, policy-driven row for an untyped agent_run_error, and that has consequences:

    • A dead server is now matched by count_failure_classes_as_zero=[agent_run_error], which benchmarks/hle_diamond/README.md recommends. The masked row could never be counted as a model zero.
    • A configured by_status_code: {503: count_as_zero | fail} is ignored by default, because the raise comes before the policy is read.
    • The health and coverage views lose the cause (no kind, no event chain).
    • With route_failures_to_sidecar off (the default), the first such /run ends the whole run and is not an attempt, instead of leaving one masked row that recovery reruns.

    The PR description ("scores the aborted rollout like a finished one") matches the pre-#5679 behaviour. test_the_old_behaviour_is_one_flag_away only asserts 200 and reward 0.0, not mask_sample, so it doesn't show that main already masks this. I'd drop the raise and the flag. If a dead server must stop the run, it should be the policy action on model_transport_error, with a test that asserts the row is typed and masked. This is a regression risk.

  2. app.py:100. APITimeoutError is also what a live, slow server produces (read timeout, 600s by default, on a long generation or a queue). Matching on the type name can't tell that apart from a dead server, and the docs and field description call it one. main already retries it as a transport failure, so this is an existing defect rather than a regression. But a model that is slow every time ends up retried and then absent, instead of reported as its own result, and the PR makes that unconditional with a 500.

  3. nemo_gym/client/http_client.py:716 and :761. The window opens at the first connection error and never closes. After it expires, any later loop iteration raises the stale connection error. That includes the iteration after a reconnect, where the server answered with a 5xx and the call is only waiting to retry it. A reachable server then surfaces as "connection refused". Low risk, since vllm_model sets no rate-limit wait, but the docs say the cap bounds connection setup only.

  4. vllm_model.yaml and the policy client. The shipped value is null, so a static base_url still retries without limit unless the eval config sets it. The openai SDK in the policy client also retries the resulting 500 (client_max_retries defaults to 2), and each retry opens a fresh window, so one verifiers call can wait about three windows. With a cap set, a server still loading at startup now fails calls at the cap instead of being waited out. I didn't test that.

Not covered by the tests: a hung (not refused) host, which goes through the generic exception branch and isn't recorded in the window, and endpoint_file rebinding combined with the cap (the 8-retry limit dominates there).

Checked: the verifiers_agent suite on the PR head shows 71 passed and 3 failures, which fail the same way on main (httpx stream assertion in this environment). Not checked: the new vllm_model tests (the run timed out here), a real vLLM restart, multi-node.

@wprazuch
wprazuch enabled auto-merge November 9, 2026 16:06
…server-fails-run

Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
…swer

main (#5679) already records a policy-call failure in verifiers_agent as a
typed, masked model_transport_error/model_request_error row that the outcome
policy acts on. The raise replaced that with an untyped agent_run_error, so it
and its flag are removed.

The connection retry window now closes when the server answers, so it bounds
only an unbroken run of connection failures. A reachable server that answers
5xx after a reconnect is no longer reported as the earlier connection error,
and the status-wait clip that relied on a still-open window is removed.

Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
@wprazuch wprazuch changed the title fix: fail rollouts whose policy server died instead of scoring them fix(vllm_model): opt-in time cap on connection retries to a dead model server Nov 9, 2026
@AdamRajfer
AdamRajfer self-requested a review November 9, 2026 17:28
@wprazuch

wprazuch commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 8584037

@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42696402151

@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42696594397

@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42699050488

Merged via the queue into main with commit 716eaef Nov 9, 2026
69 checks passed
@wprazuch
wprazuch deleted the wprazuch/dead-policy-server-fails-run branch November 9, 2026 21:20
oyilmaz-nvidia added a commit that referenced this pull request Nov 11, 2026
A call between Gym servers retried connection errors and other failures
without a limit, and the shared aiohttp sessions had no timeout at all, so
a peer that stayed unreachable could stall a rollout indefinitely.

- Give calls with no caller-owned bound an internal retry budget
  (global_aiohttp_internal_retry_budget_seconds, default 600 s, 0 keeps the
  old unlimited behaviour). It is a connection retry window plus an elapsed
  cap on retries of other errors. Calls with _max_num_tries,
  _max_connection_retries or their own window are unchanged.
- Bound one TCP handshake with sock_connect
  (global_aiohttp_sock_connect_timeout_seconds, default 60 s). The wait for a
  connection-pool slot, total time and read time stay unlimited, so requests
  queued behind a busy pool are not cut short.

Model connection retries stay unbounded by default (see #5181), a POST is
still resent after an ambiguous disconnect, and the internal=True rate-limit
retries of NeMoGymAsyncOpenAI are left to #5890; each is a separate decision.

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>
oyilmaz-nvidia added a commit that referenced this pull request Nov 11, 2026
A call between Gym servers retried connection errors and other failures
without a limit, and the shared aiohttp sessions had no timeout at all, so
a peer that stayed unreachable could stall a rollout indefinitely.

- Give calls with no caller-owned bound an internal retry budget
  (global_aiohttp_internal_retry_budget_seconds, default 600 s, 0 keeps the
  old unlimited behaviour). It is a connection retry window plus an elapsed
  cap on retries of other errors. Calls with _max_num_tries,
  _max_connection_retries or their own window are unchanged.
- Bound one TCP handshake with sock_connect
  (global_aiohttp_sock_connect_timeout_seconds, default 60 s). The wait for a
  connection-pool slot, total time and read time stay unlimited, so requests
  queued behind a busy pool are not cut short.
- NeMoGymAsyncOpenAI(internal=True) now spends max_rate_limit_wait_seconds on
  every retryable rate-limit status instead of extending max tries forever;
  a budget of 0 keeps the unlimited behaviour.

Model connection retries stay unbounded by default (see #5181), and a POST
is still resent after an ambiguous disconnect; both are separate decisions.

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>
MichielMAnalytics pushed a commit to MichielMAnalytics/Gym that referenced this pull request Nov 12, 2026
…VIDIA-NeMo#6013)

* fix(client): bound retries between Gym servers and the TCP handshake

A call between Gym servers retried connection errors and other failures
without a limit, and the shared aiohttp sessions had no timeout at all, so
a peer that stayed unreachable could stall a rollout indefinitely.

- Give calls with no caller-owned bound an internal retry budget
  (global_aiohttp_internal_retry_budget_seconds, default 600 s, 0 keeps the
  old unlimited behaviour). It is a connection retry window plus an elapsed
  cap on retries of other errors. Calls with _max_num_tries,
  _max_connection_retries or their own window are unchanged.
- Bound one TCP handshake with sock_connect
  (global_aiohttp_sock_connect_timeout_seconds, default 60 s). The wait for a
  connection-pool slot, total time and read time stay unlimited, so requests
  queued behind a busy pool are not cut short.
- NeMoGymAsyncOpenAI(internal=True) now spends max_rate_limit_wait_seconds on
  every retryable rate-limit status instead of extending max tries forever;
  a budget of 0 keeps the unlimited behaviour.

Model connection retries stay unbounded by default (see NVIDIA-NeMo#5181), and a POST
is still resent after an ambiguous disconnect; both are separate decisions.

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

* fix(client): resend a connect timeout like a refused connection, cover and document the budget

Address review of NVIDIA-NeMo#6013.

- A handshake or pool-wait timeout (ConnectionTimeoutError) means the request
  was never sent, like a refused connection. Handle it with the connection
  errors in _request_with_retries, so a call that must not resend a delivered
  request (checkpointed /run, the collector's _max_num_tries with
  _retry_other_errors=False) still retries it within its bound. A timeout
  under an open retry window keeps the window's original connection error.
- Add is_unsent_connection_failure and use it for the collector's connect
  grace loop and its delivery="not_sent" classification, which only
  recognised ClientConnectorError.
- Tests: handshake timeouts are resent under each replay policy and stop at
  the attempt limit, read timeouts are not resent, the collector classifies a
  handshake timeout as not sent (mocked, and with a real full accept queue),
  and a caller's limit replaces the internal budget instead of adding to it
  (a short budget that connection errors and timeouts would spend).
- Document the two settings, the 0 opt-out and the clocks in the
  configuration reference and the request() docstring.

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

* fix(single_agent_turn): treat a connect timeout like a refused connection in the model-request policy

_model_request_not_sent decides whether a failed model request may follow the
configured outcome action (it never connected) or must fail (it may have been
delivered). It recognised only ClientConnectorError. With the handshake
timeout on the shared client, a connect that hangs now surfaces as
ConnectionTimeoutError instead of the OS's ClientConnectorError, so such a
request was classified as possibly delivered. Both types report an incomplete
upload, and neither connected, so accept both.

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

---------

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

This branch was previously deployed

1 inactive deployment
public — 85840372 Deployed Nov 9, 2026 by copy-pr-bot[bot] via release / finalize / notify #11158
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:agent Agent harnesses and Responses API agent behavior bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants