Repository navigation
feat: judge endpoint resiliency - #2383
Conversation
e197cd6 to
4a76616
Compare
|
/ok to test 4a76616 |
|
/claude review |
|
SHIP WITH CARE — sound feature, well-tested; one operability tail-risk and a config-convention gap, both non-blocking. What this doesAdds endpoint rebinding for vLLM model servers whose backend moves hosts (shared HPC serving jobs): Correctness — looks right
Non-blocking findings (inline)
No async-hang, data-corruption, or public-API concerns. Test coverage is good. |
| ) | ||
|
|
||
| def _resolve_client(self, request: Request) -> NeMoGymAsyncOpenAI: | ||
| self._maybe_rebind_endpoint() |
There was a problem hiding this comment.
NOTE (operability): _maybe_rebind_endpoint() runs a blocking os.stat() (and occasionally open()/read()) on the event loop for every _resolve_client() call — i.e. once per inference request. The feature's target is shared HPC serving jobs, so endpoint_file will typically live on a network FS (Lustre is explicitly called out as a gotcha in CLAUDE.md). A degraded/slow stat there blocks the single-threaded event loop for all concurrent requests on this model server.
BLAST RADIUS: throughput stall under high concurrency when the shared FS is slow — the exact conditions this feature is deployed in. Inference latency normally dominates a sub-ms stat, so this is a tail-risk, not a common case.
FIX: throttle the filesystem check to at most once per N seconds (track last-checked monotonic()), so bursts of requests reuse the cached mtime instead of each issuing a syscall. The grace-period semantics are unaffected since rebinds are rare.
There was a problem hiding this comment.
You're absolutely right about this! Fixed exactly how you suggested.
|
/ok to test 73e0be9 |
|
/claude review |
|
SHIP — solid, well-tested, backward-compatible. This adds endpoint-file rebinding to
One non-blocking NOTE inline about blocking file I/O ( No async-HTTP, verifier/scoring, or dependency-hygiene concerns. |
| return | ||
| self._endpoint_last_check_at = now | ||
| try: | ||
| mtime = os.stat(self.config.endpoint_file).st_mtime |
There was a problem hiding this comment.
NOTE — blocking file I/O on the async event loop. _maybe_rebind_endpoint() is a sync function called from _resolve_client(), which runs inside the async responses()/chat_completions() handlers. os.stat() (and the subsequent open().read()) block the single event loop, stalling all concurrent sessions on this process, not just the caller.
BLAST RADIUS: bounded in practice — the check is throttled to once per endpoint_check_interval_s (10s) and the file read only fires on an mtime change. On a local/tmpfs endpoint file this is microseconds and harmless. But this feature explicitly targets HPC shared serving jobs, and if endpoint_file lives on a networked FS (NFS/Lustre — which CLAUDE.md flags this project runs on), a metadata stall in os.stat freezes every in-flight model request on the process for the duration.
FIX (optional/defense-in-depth): if the endpoint file may live on a cluster FS, run the stat/read via asyncio.to_thread(...) (which requires making the rebind path async) or a thread executor. Given the 10s throttle, author's call — not a blocker.
There was a problem hiding this comment.
I agree with your analysis that this is a well-bounded problem.
Your better solution that you suggest would involve a larger refactor and is not suited for this PR.
73e0be9 to
456ac8c
Compare
456ac8c to
76f4cb2
Compare
76f4cb2 to
127a6ca
Compare
127a6ca to
48244c4
Compare
Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com>
Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com>
|
/ok to test 4f62ecb |
48244c4 to
4f62ecb
Compare
## What does this PR do? Follow-up to NVIDIA-NeMo#3632, split out as requested in NVIDIA-NeMo#3632 (comment). In `_request_with_retries()`, the generic `except Exception` branch handles every error except `ServerDisconnectedError` and `ClientOSError`, which have their own branches. This includes aiohttp timeouts. When the caller does not pass `_max_num_tries`, this branch counted tries only for external callers. So an internal caller (`_internal=True`) that set `_max_connection_retries` to 2 or more never reached the cap and retried forever. NVIDIA-NeMo#3415 already counts tries for callers that pass `_max_num_tries`. This PR fixes the remaining path, where `_max_num_tries` is not passed. - Count tries for internal callers too, so `_max_connection_retries` stops them. - Keep everything else the same: - internal callers with no cap still retry with no limit and print no retry message, as NVIDIA-NeMo#90 intended; - external callers still stop at `MAX_NUM_TRIES`; - callers that pass `_max_num_tries` are unchanged. - Update the comment that said internal calls are not counted. It keeps NVIDIA-NeMo#90's reason for not limiting internal calls. - Add three tests: - the cap stops internal callers, with tracing off and on; - internal callers with no cap retry silently past `MAX_NUM_TRIES`; - external callers stop at `MAX_NUM_TRIES` and print one retry message per try. - The cap test, `test_request_retry_cap_bounds_internal_callers`, replaces `test_retry_cap_bounds_internal_callers`, which was removed from NVIDIA-NeMo#3632. - Make the retry tests fail instead of hang when the loop never stops: - The test helper `_mock_global_client` (added in NVIDIA-NeMo#2383) now takes the error to raise. - Its fake `request()` calls `pytest.fail()` once it is called more often than any test needs. `pytest.fail()` raises a `BaseException`, which the loop's `except Exception` does not catch. So the test fails at once, even if the loop never sleeps. - Two NVIDIA-NeMo#3415 tests now use this helper instead of their own mocks: `test_explicit_transport_attempt_cap_disables_request_retry` and `test_request_caps_generic_errors_at_the_lower_attempt_limit`. Before, if the generic branch ignored `_max_num_tries`, the first test hung, sleeping a real 0.5 s per retry. No caller in this repo sets both `_internal=True` and `_max_connection_retries` today: - The internal callers pass no cap: `ServerClient`, the simple agent's direct tool calls to Resources servers (`tool_access`), the Terminus 2 sandboxed agent's model client (`NeMoGymAsyncOpenAI(internal=True)`), the OpenHands capture route, and the LiteLLM version check. - The callers with a cap are not internal: the remote agent and sandbox transport use 1, Tavily search uses 0, and vLLM with `endpoint_file` uses a default of 8. - The OpenAI model server and the Gymnasium agent pass `_max_num_tries` only when `upstream_max_num_tries` or `model_server_transport_max_attempts` is set. Neither passes `_max_connection_retries`, so neither changes. Related: - NVIDIA-NeMo#3553 added the cap check to this branch without counting internal tries. - NVIDIA-NeMo#3415 added `_max_num_tries` and counts tries on that path. - NVIDIA-NeMo#3159 tracks bounding every retry by default, including internal callers with no cap. This PR only makes an explicit cap work for them. - NVIDIA-NeMo#2216 tracks the `ServerDisconnectedError` and `ClientOSError` branches, which this PR does not change. - Open PR NVIDIA-NeMo#2935 adds a timeout branch that copies the old counting, so internal callers there still do not count tries. Whichever PR merges later must count tries for every caller. ## Validation All runs used an activated venv from `uv sync --extra dev --extra sandbox --extra openshell --extra telemetry --extra mlflow --extra wandb` (the extras in `full-test-suite.yml`). Every installed package matches `uv.lock` (192 packages, including aiohttp 3.14.4, nemo-lens 0.2.0+b0f977d, and pytest 9.0.3). - `pytest tests/unit_tests/test_server_utils.py responses_api_agents/remote_agent/tests/test_app.py tests/unit_tests/telemetry/`: 465 passed. - With the old code restored, `test_request_retry_cap_bounds_internal_callers` fails in under a second with `Failed: retry loop did not stop` instead of hanging. It fails the same way when the retry sleep is also removed. - Each change below makes at least one test fail. Each run of `test_server_utils.py` took about 3 s, and none hung: - the cap is ignored; - internal callers are limited by `MAX_NUM_TRIES`; - internal callers print; - external tries are double-counted; - external callers stop printing the retry message; - the `MAX_NUM_TRIES` raise is removed; - the retry sleep after a generic error is skipped; - the cap check ignores `_max_num_tries` attempts; - the generic branch ignores `_max_num_tries`; - the traced path drops `_internal` or `_max_connection_retries`. - `pytest -n 16 tests/unit_tests responses_api_agents/remote_agent/tests`: 8089 passed, 5 skipped. The per-test results match `main` at a3cf403 (8085 passed, 5 skipped). The 4 extra results are the new tests, with the cap test run twice (tracing off and on). - The same focused tests pass on a local merge with the latest `main` (NVIDIA-NeMo#4300), which changes no package versions. - `pre-commit run --all-files`: passed. ## Rollouts N/A. This changes a retry counter in the shared HTTP helper. No caller in this repo changes behavior, and no model, agent, environment, or verifier logic is involved. ## Compatibility and benchmark impact None for callers in this repo. A caller that sets `_internal=True` and `_max_connection_retries` to 2 or more, without `_max_num_tries`, now stops after that many tries on errors that reach the generic `except Exception` branch, including timeouts, instead of retrying forever. This includes code outside this repo that uses `NeMoGymAsyncOpenAI(internal=True, max_connection_retries=N)`. `ServerDisconnectedError` and `ClientOSError` keep their own count, as before. ## Checklist - [x] I have read the [contributing guidelines](https://github.com/NVIDIA-NeMo/Gym/blob/main/CONTRIBUTING.md). - [x] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [x] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). - [x] Documentation added or updated, or N/A with justification. N/A: internal helper behavior with no user-facing docs. - [x] Pre-commit checks pass locally (`pre-commit run --all-files`) (so CI lint/format checks pass). - [x] New source files include the required Apache-2.0 SPDX header. N/A: no new files. - [x] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). --------- Signed-off-by: Abhishree <athittenaman@nvidia.com>
Judge models are commonly hosted independently of other jobs. This allows for their reuse.
If a judge model must be rehosted, we cannot allow this to bring down the entire training cluster.
Commonly, training infrastructure have job duration limits. If a judge model meets such a limit in the middle of a training run, it will change addresses, and it needs to communicate this to the rest of the Gym setup.
This PR adds two optional arguments:
max_connection_retriesprevents infinite retries (which cause hangs)endpoint_fileallows the judge model to communicate its address via a local filesystem; this flag comes with associated helper configs.