Repository navigation
fix(vllm_model): opt-in time cap on connection retries to a dead model server - #5181
Conversation
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP WITH CARE — two RISKs and two NOTEs; no BLOCKERs.
Completed static review of all nine changed files and their directly affected paths. Tests were not executed in this review.
RISK — Representative model-rollout evidence is missing
Could you add real smoke-rollout evidence for the new default in VerifiersAgent.responses? The supplied validation lists unit tests only, while AGENTS.md requires representative model rollouts and inspection of agent/verifier behavior for behavior-changing agent work.
Evidence gap: The new test fixture replaces run_group and supplies an error dictionary directly. It verifies the handler's reaction, but not whether the installed environment exports an actual mid-rollout policy failure in the shape _model_server_failure consumes. The external verifiers implementation is not mounted, so that production boundary remains unverified—not a demonstrated implementation defect.
Correction and preservation: Record the environment, model, configuration, commands and inspected results in the PR validation. No implementation patch is justified by this evidence gap alone. Healthy rewards and trajectories must remain valid; model-output and tool errors must remain scored; the opt-out must preserve the previous scoring behavior.
Acceptance — proposed, not executed: Run a healthy representative rollout and inspect its completion, reward and trajectory. Interrupt the policy service during another real rollout and let the configured retries/timeouts finish: agent /run should return 500 rather than a successful scored result. Record the actual error chain. Retain the existing preservation assertions: opt-out returns 200 with reward 0.0, and model-output errors, tool connection errors and clean rollouts return 200.
NOTE — Suggestion: document the new failure-handling controls
The new opt-in retry deadline is only exposed as a null key in the exemplar YAML; the vLLM guide and verifiers-agent README do not explain these controls. Operators following those guides cannot discover how to enable bounded retries or restore scoring after the new default /run failure behavior. This is a documentation gap, not a runtime defect.
Could you add a short operational section to those two documents? Include a working configuration example for endpoint_connection_retry_timeout_s, its positive-value requirement, and the fail_rollout_on_model_server_error default/opt-out. Preserve these distinctions explicitly: the window starts at the first connection error, status retries share it, connected generations are not cut short, and null adds no time cap but does not remove existing attempt caps. Model-output/tool errors remain scored; the flag changes policy-server failure handling only.
Static acceptance: The guides contain usable configuration examples and match the client contract and agent flag contract, without implying a total generation deadline or unlimited retries when a finite attempt cap applies.
Inline comments (escaped v1):
-
nemo_gym/openai_utils.pyline 1845: RISK — Bound status-retry backoff by the opened connection window.Failure: Set
connection_retry_timeout_s=1. Let the first connection attempt fail, then reconnect and immediately receive503withRetry-After: 60. The new window is running, but this branch sleeps the full 60 seconds before the transport checks expiry. An opted-in call therefore remains pending roughly a minute beyond its one-second deadline even though no generation is in progress. The PR creates one shared window, but this existing status-wait branch does not consult it; the transport notices only at the next attempt. This affects calls that encounter both a connection failure and a retryable status.Correction: Could you replace this
await sleep(wait)with the following, so retry-only waits cannot extend an opened window?window = request_kwargs.get("_connection_retry_window") remaining = window.remaining() if window is not None else None await sleep(wait if remaining is None else max(0.0, min(wait, remaining)))
The next transport entry raises the saved connection error without connecting again. This is a source-reasoned suggestion; please verify it before adoption.
Preservation: Keep ordinary Retry-After/exponential waits unchanged when there is no window or no connection error has opened it. Retain one window across status retries and a fresh one for the next logical call. Keep terminal-status/no-resend branches' no-sleep behavior. Do not impose a whole-request deadline: an already connected slow generation must still finish.
Acceptance — proposed, not executed: Add a public-client regression using a connector that first raises a connection error, then reaches a server returning
503withRetry-After: 60. With a one-second window, assert the saved error is raised near one second and no post-deadline network attempt occurs. Also assert that503 → 200without a preceding connection error honors the full ordinary wait, and that no-window behavior is unchanged. Retain the shared-window/fresh-next-call assertions and the slow connected-response success case. -
tests/unit_tests/test_server_utils.pyline 2251: NOTE — Suggestion: actually enable the traced timeout case.Coverage failure:
tracing_enabled=Truechanges only the span-group gate. request() also requiresis_trace_exporter_active(), whose initial state is false. An isolated run therefore exercises the untraced transport twice and cannot catch lost window forwarding through_traced_request. Current forwarding is correct by inspection; this is not a demonstrated runtime bug.Correction: Following the existing retry-test setup, add immediately after this monkeypatch:
monkeypatch.setattr(nemo_gym.server_utils, "is_trace_exporter_active", lambda: tracing_enabled) monkeypatch.setattr("nemo_gym.telemetry.spans.client_span", lambda _: nullcontext(None)) monkeypatch.setattr("nemo_gym.telemetry.contrib.inject_trace_context", lambda headers: None)
This selects the intended branch without requiring optional telemetry dependencies. Please verify the suggestion before adopting it.
Preservation and acceptance: Keep both parameter values and the real refused-port assertion: each must raise
ClientConnectorErrorwith1.0 <= elapsed < 1.8. Static acceptance is that the true case satisfies both dispatch gates and traverses_traced_request, while the false case still selects_request_with_retriesdirectly. Proposed checks only; tests were not executed here.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
|
🌿 Preview your docs: https://nvidia-preview-pr-5181-85840372c024.docs.buildwithfern.com/nemo/gym |
|
/review reset=true |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP WITH CARE — one RISK and one NOTE; no BLOCKERs.
Completed static review of all 11 changed files and their directly affected retry, client-construction, rollout and scoring paths. Tests were not executed.
The earlier status-backoff, traced-test and documentation findings are addressed: the status loop clips waits to an opened window; the refused-port test enables both tracing gates and verifies dispatch; and the vLLM guide and agent README document the controls.
RISK — Representative model-rollout evidence is still missing
Could you provide the real smoke-rollout evidence requested in review 5682476798? The supplied validation still lists unit tests only, while AGENTS.md requires representative model rollouts and inspection of agent/verifier behavior for behavior-changing agent work.
Evidence gap and affected behavior: The new default rejects a rollout before scoring based on its exported error chain. The test fixture replaces run_group and supplies that error directly. It verifies the handler's reaction, but not whether an actual mid-rollout policy failure reaches it in that shape. The external verifiers implementation is not mounted, so this remains an unverified production boundary—not a demonstrated implementation defect.
Correction and preservation: Record the representative environment, model, resolved configuration, commands and inspected results in the PR validation. No implementation patch is justified by this evidence gap alone. Preserve healthy rewards and trajectories, scoring of model-output/tool errors, and the flag's previous-behavior opt-out.
Acceptance — proposed, not executed: Run a healthy representative rollout and inspect its completion, reward and trajectory. During another real rollout, interrupt the policy service and allow configured retries/timeouts to finish; agent /run should return 500 rather than a successful scored result. Record the actual exported error chain. Retain the existing preservation assertions: opt-out returns 200 with reward 0.0, while model-output errors, tool connection errors and clean rollouts return 200.
Inline comments (escaped v1):
-
nemo_gym/server_utils.pyline 944: NOTE — Suggestion: clip the generic-error backoff to the opened connection window too.Failure: The new window can still be exceeded by almost 0.5 seconds through this branch. For example, use a 0.6-second window: the first
ClientOSErroropens it, the connection-error branch sleeps 0.5 seconds, and the reconnect raises aConnectionTimeoutErrorwhile budget remains. The expiry check then falls through, and this unconditional sleep delays the saved connection error until approximately one second. No generation is running during that extra wait. This is a bounded deadline-accuracy issue, not an unbounded hang.Correction: Could you replace this sleep in
_request_with_retrieswith:await asyncio.sleep(_connection_backoff_s(window))
The invariant is that retry-only backoff cannot extend an already opened window. Also update the neighboring diagnostic's hard-coded “Sleeping 0.5s” wording to describe the bounded backoff. This is a source-reasoned suggestion; please verify it before adoption.
Preservation: The existing helper still returns 0.5 when no window exists or no connection error has opened it. Keep attempt-cap and no-resend exits before sleeping. Do not alter total/read timeouts or interrupt connected responses; retain the slow-generation case, which must return status 200 and body
okdespite outliving the window.Acceptance — proposed, not executed: Add a deterministic regression with a controlled monotonic clock and sleep stub that advances it. Give the mock client a real
ClientTimeoutand make requests raiseClientOSError, thenConnectionTimeoutErrorbefore a 0.6-second window expires. Assert sleeps of 0.5 and approximately 0.1 seconds, the savedClientOSErroris raised, and no third transport request starts. Retain the no-window recovery assertion: success after exactly one 0.5-second sleep.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
Bound connection retries to a dead model server by a wall-clock window, cap status-retry and generic-error backoff at that window, and make verifiers_agent /run return 500 when the policy server failed the rollout. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
471b433 to
e868d48
Compare
AdamRajfer
left a comment
There was a problem hiding this comment.
The client side looks good to me: the cap is opt-in, keyword-only, and the status retries share the window. I ran tests/unit_tests/client/test_http_client.py and test_openai_client.py on the PR head (258 passed). My concern is the verifiers_agent change, which I think collides with what #5679 put on main today.
-
responses_api_agents/verifiers_agent/app.py:914. On current main,run()already handles a policy call that ends a rollout._ModelFailureObservingClientrecords it asmodel_transport_error(no reply or timeout) ormodel_request_errorplus the status (5xx). The row is masked, the agent-stage outcome policy picks the action (retry by default), and the failure report stays on the row. The new raise sits inresponses(), whichrun()awaits before any of that. I ran the PR head with the flag off and on, for a timeout, a refused connection and a 503:- Flag off: 200,
mask_sample: true,failure_kindmodel_transport_error/model_request_error,failure_outcome_action: retry. - Flag on, the default: 500 "Internal Server Error", no failure header.
So this swaps a typed, masked, policy-driven row for an untyped
agent_run_error, and that has consequences:- A dead server is now matched by
count_failure_classes_as_zero=[agent_run_error], whichbenchmarks/hle_diamond/README.mdrecommends. The masked row could never be counted as a model zero. - A configured
by_status_code: {503: count_as_zero | fail}is ignored by default, because the raise comes before the policy is read. - The health and coverage views lose the cause (no kind, no event chain).
- With
route_failures_to_sidecaroff (the default), the first such/runends the whole run and is not an attempt, instead of leaving one masked row that recovery reruns.
The PR description ("scores the aborted rollout like a finished one") matches the pre-#5679 behaviour.
test_the_old_behaviour_is_one_flag_awayonly asserts 200 and reward 0.0, notmask_sample, so it doesn't show that main already masks this. I'd drop the raise and the flag. If a dead server must stop the run, it should be the policy action onmodel_transport_error, with a test that asserts the row is typed and masked. This is a regression risk. - Flag off: 200,
-
app.py:100.APITimeoutErroris also what a live, slow server produces (read timeout, 600s by default, on a long generation or a queue). Matching on the type name can't tell that apart from a dead server, and the docs and field description call it one. main already retries it as a transport failure, so this is an existing defect rather than a regression. But a model that is slow every time ends up retried and then absent, instead of reported as its own result, and the PR makes that unconditional with a 500. -
nemo_gym/client/http_client.py:716and:761. The window opens at the first connection error and never closes. After it expires, any later loop iteration raises the stale connection error. That includes the iteration after a reconnect, where the server answered with a 5xx and the call is only waiting to retry it. A reachable server then surfaces as "connection refused". Low risk, sincevllm_modelsets no rate-limit wait, but the docs say the cap bounds connection setup only. -
vllm_model.yamland the policy client. The shipped value isnull, so a staticbase_urlstill retries without limit unless the eval config sets it. The openai SDK in the policy client also retries the resulting 500 (client_max_retriesdefaults to 2), and each retry opens a fresh window, so one verifiers call can wait about three windows. With a cap set, a server still loading at startup now fails calls at the cap instead of being waited out. I didn't test that.
Not covered by the tests: a hung (not refused) host, which goes through the generic exception branch and isn't recorded in the window, and endpoint_file rebinding combined with the cap (the 8-retry limit dominates there).
Checked: the verifiers_agent suite on the PR head shows 71 passed and 3 failures, which fail the same way on main (httpx stream assertion in this environment). Not checked: the new vllm_model tests (the run timed out here), a real vLLM restart, multi-node.
…server-fails-run Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
…swer main (#5679) already records a policy-call failure in verifiers_agent as a typed, masked model_transport_error/model_request_error row that the outcome policy acts on. The raise replaced that with an untyped agent_run_error, so it and its flag are removed. The connection retry window now closes when the server answers, so it bounds only an unbroken run of connection failures. A reachable server that answers 5xx after a reconnect is no longer reported as the earlier connection error, and the status-wait clip that relied on a still-open window is removed. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
|
/ok to test 8584037 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42696402151 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42696594397 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42699050488 |
A call between Gym servers retried connection errors and other failures without a limit, and the shared aiohttp sessions had no timeout at all, so a peer that stayed unreachable could stall a rollout indefinitely. - Give calls with no caller-owned bound an internal retry budget (global_aiohttp_internal_retry_budget_seconds, default 600 s, 0 keeps the old unlimited behaviour). It is a connection retry window plus an elapsed cap on retries of other errors. Calls with _max_num_tries, _max_connection_retries or their own window are unchanged. - Bound one TCP handshake with sock_connect (global_aiohttp_sock_connect_timeout_seconds, default 60 s). The wait for a connection-pool slot, total time and read time stay unlimited, so requests queued behind a busy pool are not cut short. Model connection retries stay unbounded by default (see #5181), a POST is still resent after an ambiguous disconnect, and the internal=True rate-limit retries of NeMoGymAsyncOpenAI are left to #5890; each is a separate decision. Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>
A call between Gym servers retried connection errors and other failures without a limit, and the shared aiohttp sessions had no timeout at all, so a peer that stayed unreachable could stall a rollout indefinitely. - Give calls with no caller-owned bound an internal retry budget (global_aiohttp_internal_retry_budget_seconds, default 600 s, 0 keeps the old unlimited behaviour). It is a connection retry window plus an elapsed cap on retries of other errors. Calls with _max_num_tries, _max_connection_retries or their own window are unchanged. - Bound one TCP handshake with sock_connect (global_aiohttp_sock_connect_timeout_seconds, default 60 s). The wait for a connection-pool slot, total time and read time stay unlimited, so requests queued behind a busy pool are not cut short. - NeMoGymAsyncOpenAI(internal=True) now spends max_rate_limit_wait_seconds on every retryable rate-limit status instead of extending max tries forever; a budget of 0 keeps the unlimited behaviour. Model connection retries stay unbounded by default (see #5181), and a POST is still resent after an ambiguous disconnect; both are separate decisions. Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>
…VIDIA-NeMo#6013) * fix(client): bound retries between Gym servers and the TCP handshake A call between Gym servers retried connection errors and other failures without a limit, and the shared aiohttp sessions had no timeout at all, so a peer that stayed unreachable could stall a rollout indefinitely. - Give calls with no caller-owned bound an internal retry budget (global_aiohttp_internal_retry_budget_seconds, default 600 s, 0 keeps the old unlimited behaviour). It is a connection retry window plus an elapsed cap on retries of other errors. Calls with _max_num_tries, _max_connection_retries or their own window are unchanged. - Bound one TCP handshake with sock_connect (global_aiohttp_sock_connect_timeout_seconds, default 60 s). The wait for a connection-pool slot, total time and read time stay unlimited, so requests queued behind a busy pool are not cut short. - NeMoGymAsyncOpenAI(internal=True) now spends max_rate_limit_wait_seconds on every retryable rate-limit status instead of extending max tries forever; a budget of 0 keeps the unlimited behaviour. Model connection retries stay unbounded by default (see NVIDIA-NeMo#5181), and a POST is still resent after an ambiguous disconnect; both are separate decisions. Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * fix(client): resend a connect timeout like a refused connection, cover and document the budget Address review of NVIDIA-NeMo#6013. - A handshake or pool-wait timeout (ConnectionTimeoutError) means the request was never sent, like a refused connection. Handle it with the connection errors in _request_with_retries, so a call that must not resend a delivered request (checkpointed /run, the collector's _max_num_tries with _retry_other_errors=False) still retries it within its bound. A timeout under an open retry window keeps the window's original connection error. - Add is_unsent_connection_failure and use it for the collector's connect grace loop and its delivery="not_sent" classification, which only recognised ClientConnectorError. - Tests: handshake timeouts are resent under each replay policy and stop at the attempt limit, read timeouts are not resent, the collector classifies a handshake timeout as not sent (mocked, and with a real full accept queue), and a caller's limit replaces the internal budget instead of adding to it (a short budget that connection errors and timeouts would spend). - Document the two settings, the 0 opt-out and the clocks in the configuration reference and the request() docstring. Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * fix(single_agent_turn): treat a connect timeout like a refused connection in the model-request policy _model_request_not_sent decides whether a failed model request may follow the configured outcome action (it never connected) or must fail (it may have been delivered). It recognised only ClientConnectorError. With the handshake timeout on the shared client, a connect that hangs now surfaces as ConnectionTimeoutError instead of the OS's ClientConnectorError, so such a request was classified as possibly delivered. Both types report an incomplete upload, and neither connected, so accept both. Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> --------- Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>
What goes wrong
vllm_modelon a staticbase_urlretries a refused connection without a time limit. When the policy engine dies mid-run, every in-flight model call hangs until something above it gives up, and an evaluation can stall for a long time before anything reports the dead server.What this changes
An opt-in
endpoint_connection_retry_timeout_sonvllm_model(connection_retry_timeout_sonNeMoGymAsyncOpenAI) caps the time one model call spends retrying connection errors. Past the cap, the original connection error is raised. The defaultnullkeeps today's unlimited retries, so training can still ride out an engine restart.ClientTimeout.connect). Expiry is checked before another attempt, and the backoff is cut to the time left. A connected attempt is never cut short.request(),_traced_request()and_request_with_retries(), so positional callers keep their meaning.This supersedes #4853 (same option).
An earlier revision also made
verifiers_agentfail/runon a policy-call failure. That is dropped: since #5679,mainalready records such a rollout as a typed, maskedmodel_transport_error/model_request_errorrow that the outcome policy acts on, and the raise would have replaced it with an untypedagent_run_error. If a dead server should stop a run, that belongs in the outcome policy formodel_transport_error.Not changed
verifiers_agentmatchingAPITimeoutErroras a dead server also fires on a slow but live server. That is existing behaviour onmainand is left for a separate change.max_retriesin an agent's policy client) waits up to one window per retry. The docs say so, and say to set the cap above the engine's startup time.Tests
tests/unit_tests/client/test_http_client.py: a refused port fails within the window (with and without tracing) and retries forever without one. A reconnect that hangs fails at the deadline with the saved connection error. A slow answer after a reconnect still succeeds. A spent window raises before connecting again. New: an answer closes the window, so a later retry after the window would have run out connects instead of raising the earlier "refused" (fails without the fix). Positional arguments keep their meaning.tests/unit_tests/client/test_openai_client.py: a 503 after a connection error keeps its fullRetry-Afterwait and the call succeeds; an unopened window keeps the full status wait; no window by default.responses_api_models/vllm_model/tests/test_app.py: the option reaches every client, including rebuilt endpoint clients, and a dead engine fails a real call within the window.Results:
tests/unit_tests/client/test_http_client.py+test_openai_client.py259 passed;vllm_model507 passed; the fulltests/unit_testsrun shows no new failures compared withmainin the same environment (the sixtest_super35_batch_launcherprebuilt-environment cases are skipped on a freshmaincheckout, andtest_short_http_reply_is_a_package_store_errorpasses when run alone).pre-commitclean on the changed files.Rollout evidence: N/A. The option is off by default, and with it unset no request path changes. The only behaviour change, the
verifiers_agentraise, has been removed.