Skip to content

fix(model): bound connection retries whenever route-around is on - #4365

Merged
yaoyu-33 merged 1 commit into
mainfrom
alexandery/vllm-model-bound-retries-under-route-around
Oct 12, 2026
Merged

yaoyu-33 merged 1 commit into
mainfrom
alexandery/vllm-model-bound-retries-under-route-around

Conversation

@alexandery-nvidia

@alexandery-nvidia alexandery-nvidia commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

VLLMModel._post_init builds one NeMoGymAsyncOpenAI client per base_url and passes endpoint_connection_retries as the client's max_connection_retries only when endpoint_file is configured. On main the condition reads self.config.endpoint_connection_retries if self.config.endpoint_file else None (responses_api_models/vllm_model/app.py:466-468 on main). A deployment that enables route_around_failing_endpoints over a static base_url list has no endpoint file, so its clients are built with max_connection_retries=None.

None means retry forever. NeMoGymAsyncOpenAI forwards the value as _max_connection_retries on every request (nemo_gym/openai_utils.py:1378-1384 and nemo_gym/openai_utils.py:1468), and in _request_with_retries the except ClientOSError branch, which a refused connection reaches as aiohttp.ClientConnectorError, re-raises only when _max_connection_retries is set (nemo_gym/server_utils.py:638-654). Otherwise it sleeps 0.5 s and loops.

The endpoint health accounting that #3979 added depends on the call raising. _call_endpoint awaits the engine call and counts a ClientConnectionError against the endpoint only in its except branch (responses_api_models/vllm_model/app.py:1835-1853), and _note_endpoint_failure moves sessions off the endpoint once endpoint_failure_threshold consecutive calls have failed (responses_api_models/vllm_model/app.py:1855-1873). With an unbounded client the call never returns from the retry loop, the endpoint never records a failure, and every session pinned to a dead endpoint hangs on it for the rest of the run. The feature cannot react to the failure mode it was built for.

The change:

  • responses_api_models/vllm_model/app.py:469-473: the clients receive endpoint_connection_retries when endpoint_file or route_around_failing_endpoints is set. A static list without either setting still gets None, and the endpoint-file rebind path already passed the bound unconditionally (responses_api_models/vllm_model/app.py:1756).
  • responses_api_models/vllm_model/app.py:338-342: the comment on endpoint_connection_retries states the condition and why an unbounded client defeats the route-around.
  • responses_api_models/vllm_model/tests/test_app.py:6116-6135: a new test builds the model over a two-URL static list with route_around_failing_endpoints=True, no endpoint file and endpoint_connection_retries=3, and checks that both clients carry max_connection_retries == 3. The same list without either setting yields None for both clients. The existing assertion at responses_api_models/vllm_model/tests/test_app.py:6114 already pinned the single-URL default; only its comment changed, because "static clients retry forever" now holds only without route-around.

Issue

Validation

  • pytest -q -p no:cacheprovider -o addopts= responses_api_models/vllm_model/tests/test_app.py with the repository root on PYTHONPATH (Python 3.13.14, pytest 9.1.1, aiohttp 3.14.3): 254 passed, 0 failed. The same command on main at f128d43 gives 253 passed; the difference is the new test.
  • The new test fails against the main version of responses_api_models/vllm_model/app.py with [None, None] == [3, 3].
  • pytest -q -p no:cacheprovider -o addopts= responses_api_models/vllm_model/tests/ (the whole directory, three test files): 406 passed, 0 failed.
  • Unit suite (pytest tests/unit_tests, with test_collect_downstream_junit.py excluded for a missing local package): 7752 passed, 56 skipped, 17 failed, 5 errors; the 22 failures are the same 22 as on main in this environment (telemetry, opensandbox, and local-vLLM tests that need packages this machine lacks). Each other responses_api_models/<server>/tests directory, run as its own session, matches main.
  • ruff check and ruff format --check on both changed files: clean.

🤖 Generated with Claude Code

`VLLMModel._post_init` applied `endpoint_connection_retries` to the engine
clients only when `endpoint_file` was configured. With
`route_around_failing_endpoints` enabled over a static `base_url` list, the
clients were built with `max_connection_retries=None`, and
`_request_with_retries` in `nemo_gym/server_utils.py` then retries a refused
connection forever instead of raising. `_call_endpoint` records a connection
error against an endpoint only when the awaited call raises, so a dead
endpoint never reached `endpoint_failure_threshold` and its sessions stayed
pinned to it for the whole run.

The clients now receive the bound when either `endpoint_file` or
`route_around_failing_endpoints` is set. The config comment states the new
condition and why an unbounded client defeats the route-around. A new test
builds the model over a two-URL static list with route-around on and no
endpoint file and checks that every client carries the configured bound,
and that the same list without either setting keeps unbounded retries.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: alexandery <alexandery@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 10, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@alexandery-nvidia
alexandery-nvidia marked this pull request as ready for review October 10, 2026 08:12
@yaoyu-33 yaoyu-33 added area:model Model servers, inference providers, and model adapters bug Something isn't working complexity:low Localized change in one scope with a small, straightforward review surface needs-review PR is ready for code review and waiting on a reviewer labels Oct 10, 2026
@github-actions github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Oct 12, 2026

@linj-glitch linj-glitch left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The configured retry bound now applies to static endpoint lists with route-around enabled, allowing connection failures to reach the existing endpoint health accounting. Default static clients retain their prior behavior. No blocking findings; no dependency on #3914 or #4262.

Validation: 406 vLLM model tests and 54 shared HTTP retry tests passed; full pre-commit passed. A real local HTTP probe observed exactly three refused-connection attempts per failing call, then routed the same session to a live endpoint after the failure threshold. Restoring the old initializer condition in memory made the new regression test fail. This validates the affected transport path without model/GPU compute.

Shared model-server maintainer review and trusted runtime CI remain part of the handoff. Reviewed with Codex, including an independent source/history pass.

@linj-glitch
linj-glitch requested a review from ananthsub October 12, 2026 20:18
@linj-glitch

Copy link
Copy Markdown
Collaborator

/ok to test 7f09aad

@nemo-automation-bot
nemo-automation-bot Bot requested review from a team October 12, 2026 20:19
@github-actions github-actions Bot removed the sla:triage-overdue Review assignment is over the one-business-day SLA label Oct 12, 2026
@yaoyu-33
yaoyu-33 added this pull request to the merge queue Oct 12, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/38312189502

Merged via the queue into main with commit 7f16ec5 Oct 12, 2026
56 of 58 checks passed
@yaoyu-33
yaoyu-33 deleted the alexandery/vllm-model-bound-retries-under-route-around branch October 12, 2026 21:58

This branch was previously deployed

1 inactive deployment
public — 7f09aada Deployed Oct 12, 2026 by copy-pr-bot[bot] via release / finalize / notify #5088
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:model Model servers, inference providers, and model adapters bug Something isn't working complexity:low Localized change in one scope with a small, straightforward review surface needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants