Repository navigation
fix(model): bound connection retries whenever route-around is on - #4365
Conversation
`VLLMModel._post_init` applied `endpoint_connection_retries` to the engine clients only when `endpoint_file` was configured. With `route_around_failing_endpoints` enabled over a static `base_url` list, the clients were built with `max_connection_retries=None`, and `_request_with_retries` in `nemo_gym/server_utils.py` then retries a refused connection forever instead of raising. `_call_endpoint` records a connection error against an endpoint only when the awaited call raises, so a dead endpoint never reached `endpoint_failure_threshold` and its sessions stayed pinned to it for the whole run. The clients now receive the bound when either `endpoint_file` or `route_around_failing_endpoints` is set. The config comment states the new condition and why an unbounded client defeats the route-around. A new test builds the model over a two-URL static list with route-around on and no endpoint file and checks that every client carries the configured bound, and that the same list without either setting keeps unbounded retries. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: alexandery <alexandery@nvidia.com>
linj-glitch
left a comment
There was a problem hiding this comment.
The configured retry bound now applies to static endpoint lists with route-around enabled, allowing connection failures to reach the existing endpoint health accounting. Default static clients retain their prior behavior. No blocking findings; no dependency on #3914 or #4262.
Validation: 406 vLLM model tests and 54 shared HTTP retry tests passed; full pre-commit passed. A real local HTTP probe observed exactly three refused-connection attempts per failing call, then routed the same session to a live endpoint after the failure threshold. Restoring the old initializer condition in memory made the new regression test fail. This validates the affected transport path without model/GPU compute.
Shared model-server maintainer review and trusted runtime CI remain part of the handoff. Reviewed with Codex, including an independent source/history pass.
|
/ok to test 7f09aad |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/38312189502 |
What does this PR do?
VLLMModel._post_initbuilds oneNeMoGymAsyncOpenAIclient perbase_urland passesendpoint_connection_retriesas the client'smax_connection_retriesonly whenendpoint_fileis configured. Onmainthe condition readsself.config.endpoint_connection_retries if self.config.endpoint_file else None(responses_api_models/vllm_model/app.py:466-468onmain). A deployment that enablesroute_around_failing_endpointsover a staticbase_urllist has no endpoint file, so its clients are built withmax_connection_retries=None.Nonemeans retry forever.NeMoGymAsyncOpenAIforwards the value as_max_connection_retrieson every request (nemo_gym/openai_utils.py:1378-1384andnemo_gym/openai_utils.py:1468), and in_request_with_retriestheexcept ClientOSErrorbranch, which a refused connection reaches asaiohttp.ClientConnectorError, re-raises only when_max_connection_retriesis set (nemo_gym/server_utils.py:638-654). Otherwise it sleeps 0.5 s and loops.The endpoint health accounting that #3979 added depends on the call raising.
_call_endpointawaits the engine call and counts aClientConnectionErroragainst the endpoint only in itsexceptbranch (responses_api_models/vllm_model/app.py:1835-1853), and_note_endpoint_failuremoves sessions off the endpoint onceendpoint_failure_thresholdconsecutive calls have failed (responses_api_models/vllm_model/app.py:1855-1873). With an unbounded client the call never returns from the retry loop, the endpoint never records a failure, and every session pinned to a dead endpoint hangs on it for the rest of the run. The feature cannot react to the failure mode it was built for.The change:
responses_api_models/vllm_model/app.py:469-473: the clients receiveendpoint_connection_retrieswhenendpoint_fileorroute_around_failing_endpointsis set. A static list without either setting still getsNone, and the endpoint-file rebind path already passed the bound unconditionally (responses_api_models/vllm_model/app.py:1756).responses_api_models/vllm_model/app.py:338-342: the comment onendpoint_connection_retriesstates the condition and why an unbounded client defeats the route-around.responses_api_models/vllm_model/tests/test_app.py:6116-6135: a new test builds the model over a two-URL static list withroute_around_failing_endpoints=True, no endpoint file andendpoint_connection_retries=3, and checks that both clients carrymax_connection_retries == 3. The same list without either setting yieldsNonefor both clients. The existing assertion atresponses_api_models/vllm_model/tests/test_app.py:6114already pinned the single-URL default; only its comment changed, because "static clients retry forever" now holds only without route-around.Issue
route_around_failing_endpoints. This PR fixes its static-URL configuration.ServerDisconnectedErrorandClientOSErrorbranches of_request_with_retriesnever compare their counter againstMAX_NUM_TRIES, so a permanently unreachable server loops forever. That unbounded loop is what hides a dead endpoint here. This PR does not change the loop; it makesvllm_modelpass the per-client bound that the loop does honor._max_connection_retriescap stop internal callers in the genericexcept Exceptionbranch of the same loop. Thevllm_modelclients are external callers (internal=False), so this PR does not depend on it; both concern the same cap._resolve_clientin the same file. This PR does not touch those lines, so the two compose.Validation
pytest -q -p no:cacheprovider -o addopts= responses_api_models/vllm_model/tests/test_app.pywith the repository root onPYTHONPATH(Python 3.13.14, pytest 9.1.1, aiohttp 3.14.3): 254 passed, 0 failed. The same command onmainat f128d43 gives 253 passed; the difference is the new test.mainversion ofresponses_api_models/vllm_model/app.pywith[None, None] == [3, 3].pytest -q -p no:cacheprovider -o addopts= responses_api_models/vllm_model/tests/(the whole directory, three test files): 406 passed, 0 failed.pytest tests/unit_tests, withtest_collect_downstream_junit.pyexcluded for a missing local package): 7752 passed, 56 skipped, 17 failed, 5 errors; the 22 failures are the same 22 as onmainin this environment (telemetry, opensandbox, and local-vLLM tests that need packages this machine lacks). Each otherresponses_api_models/<server>/testsdirectory, run as its own session, matchesmain.ruff checkandruff format --checkon both changed files: clean.🤖 Generated with Claude Code