Repository navigation
fix(cli): bound Gym server startup wait - #3735
Conversation
|
🌿 Preview your docs: https://nvidia-preview-ananthsub-server-spinup-timeout.docs.buildwithfern.com/nemo/gym Here are the markdown pages you've updated: |
Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
Shut down every started server when RunHelper.start() fails or is interrupted, instead of only on the two readiness paths that called shutdown() inline. Make shutdown() safe before the head server starts and on repeated calls. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
4286672 to
1bd0319
Compare
yuhezhang-ai
left a comment
There was a problem hiding this comment.
Please address the readiness deadline; the other three comments are non-blocking suggestions. Validation: 265 tests passed, with focused HTTP and process cleanup probes.
Cap each server readiness probe and the poll sleep at the time left before server_spinup_timeout_seconds, and skip probes once it passes, so many stalled servers cannot stretch the wait past the deadline. Validate the setting before anything starts: accept numeric strings from environment interpolation, fall back to the default for null, and raise ConfigError for anything else. Name the override and zero opt-out in the timeout error, and document that the budget covers dependency install and local model loading. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
yuhezhang-ai
left a comment
There was a problem hiding this comment.
The four earlier findings are addressed. All 356 focused tests passed, and the stalled-server reproduction now respects the 1-second limit. Approved, with one non-blocking documentation wording request below.
Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
yuhezhang-ai
left a comment
There was a problem hiding this comment.
The documentation wording is corrected. Confirmed this is the only change since the reviewed revision; approval renewed.
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/36606883896 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/36608982967 |
Add a configurable `server_spinup_timeout_seconds` bound to `gym env start` server readiness polling. The default is 600 seconds; `0` preserves the previous unbounded wait for debugging. Each readiness probe and poll sleep is capped at the time left, so the wait ends at the deadline no matter how many servers are stalled. When the deadline expires, Gym raises an error that names the instances that never became ready and explains how to raise or disable the limit. The value is validated before anything starts: numeric strings from environment interpolation are accepted, and malformed values raise `ConfigError`. Move startup cleanup to one boundary in `RunHelper.start()`. Any exception or interrupt raised before readiness now shuts down every server started so far and then re-raises the original exception. Before this change, only two failure paths cleaned up, and each called `shutdown()` inline: the model-endpoint timeout, and this PR's new timeout. The boundary also covers a server that exits during startup, Ctrl-C during startup, and a failure partway through spawning servers. Each of these used to orphan the servers that were already running. `RunHelper.shutdown()` now tolerates being called before the head server starts and being called more than once. NeMo-RL relies on repeated calls: when startup fails, it calls `shutdown()` again on the same `RunHelper`. Part of #3160. This PR covers the component-readiness deadline and the startup-unwind criteria for the stages `start()` owns. It does not bound head-server or model-endpoint readiness. It also does not add process-group ownership of descendants or a single total shutdown deadline. A server process can stay alive without ever becoming ready. For example, a Uvicorn supervisor can keep replacing workers that fail while the application is being constructed. The launcher only treated process exit as failure, so it waited forever. On scheduled infrastructure this can hold the allocation until its external walltime. All `RunHelper` callers invoke `start()` outside their own `try` block. The spawned servers have no process group and no atexit handler, and the head-server thread is a daemon. So when startup failed on any path that did not call `shutdown()` explicitly, the orchestrator exited and the already-started servers were reparented to init and kept holding their ports and GPUs. - `pytest tests/unit_tests/test_cli.py tests/unit_tests/test_cli_endpoint_readiness.py tests/unit_tests/test_global_config.py tests/unit_tests/test_server_utils.py -q`: 356 passed. - Two real stalled HTTP servers with `server_spinup_timeout_seconds=1`: raises in 1.00s. Before readiness probes were bounded by the deadline, this took 10.01s. - Real `gym env start` with `++server_spinup_timeout_seconds='ten minutes'`: exits 1 with a `ConfigError` message before Ray or any server starts. - Scoped pre-commit over all changed files: passed. - Ran a real `gym env start` with two local servers. `crasher` exits 8 seconds into startup, and `hanger` stays alive but never becomes ready. - Before the boundary: `RuntimeError: Process crasher finished unexpectedly!`, and `hanger` was left running with parent PID 1. - After: the same error reaches the caller, `shutdown()` runs, and `hanger` exits. - Sent `SIGINT` only to the `gym` process while it waited on `hanger`. Before the boundary, `hanger` was left running with parent PID 1. After, `shutdown()` runs and `hanger` exits. N/A. This changes CLI startup failure handling and does not alter model requests, trajectories, verification, or rewards. - Default server startup wait is now bounded at 600 seconds. - Long-starting deployments can raise `server_spinup_timeout_seconds`. - Setting `server_spinup_timeout_seconds: 0` keeps the previous unbounded behavior. - `RunHelper.wait_for_model_endpoints()` no longer calls `shutdown()` itself. Its only caller is `start()`, which now performs the cleanup. --------- Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> (cherry picked from commit 329683e) Conflict in tests/unit_tests/test_cli.py resolved for this branch: kept the 13 tests #3735 adds, with its 3 wait_for_spinup timeout tests in a new TestRunHelperServerSpinupTimeout class because TestRunHelperServerReadiness does not exist here. Dropped the 4 tests from other main commits that shared the conflict hunks (head-ready ordering, /health while starting, and two memory-profiler tests), since they exercise code this branch does not have. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What changed
Add a configurable
server_spinup_timeout_secondsbound togym env startserver readiness polling. The default is 600 seconds;0preserves the previous unbounded wait for debugging. Each readiness probe and poll sleep is capped at the time left, so the wait ends at the deadline no matter how many servers are stalled. When the deadline expires, Gym raises an error that names the instances that never became ready and explains how to raise or disable the limit. The value is validated before anything starts: numeric strings from environment interpolation are accepted, and malformed values raiseConfigError.Move startup cleanup to one boundary in
RunHelper.start(). Any exception or interrupt raised before readiness now shuts down every server started so far and then re-raises the original exception. Before this change, only two failure paths cleaned up, and each calledshutdown()inline: the model-endpoint timeout, and this PR's new timeout. The boundary also covers a server that exits during startup, Ctrl-C during startup, and a failure partway through spawning servers. Each of these used to orphan the servers that were already running.RunHelper.shutdown()now tolerates being called before the head server starts and being called more than once. NeMo-RL relies on repeated calls: when startup fails, it callsshutdown()again on the sameRunHelper.Part of #3160. This PR covers the component-readiness deadline and the startup-unwind criteria for the stages
start()owns. It does not bound head-server or model-endpoint readiness. It also does not add process-group ownership of descendants or a single total shutdown deadline.Why
A server process can stay alive without ever becoming ready. For example, a Uvicorn supervisor can keep replacing workers that fail while the application is being constructed. The launcher only treated process exit as failure, so it waited forever. On scheduled infrastructure this can hold the allocation until its external walltime.
All
RunHelpercallers invokestart()outside their owntryblock. The spawned servers have no process group and no atexit handler, and the head-server thread is a daemon. So when startup failed on any path that did not callshutdown()explicitly, the orchestrator exited and the already-started servers were reparented to init and kept holding their ports and GPUs.Validation
pytest tests/unit_tests/test_cli.py tests/unit_tests/test_cli_endpoint_readiness.py tests/unit_tests/test_global_config.py tests/unit_tests/test_server_utils.py -q: 356 passed.server_spinup_timeout_seconds=1: raises in 1.00s. Before readiness probes were bounded by the deadline, this took 10.01s.gym env startwith++server_spinup_timeout_seconds='ten minutes': exits 1 with aConfigErrormessage before Ray or any server starts.gym env startwith two local servers.crasherexits 8 seconds into startup, andhangerstays alive but never becomes ready.RuntimeError: Process crasher finished unexpectedly!, andhangerwas left running with parent PID 1.shutdown()runs, andhangerexits.SIGINTonly to thegymprocess while it waited onhanger. Before the boundary,hangerwas left running with parent PID 1. After,shutdown()runs andhangerexits.Rollout evidence
N/A. This changes CLI startup failure handling and does not alter model requests, trajectories, verification, or rewards.
Compatibility
server_spinup_timeout_seconds.server_spinup_timeout_seconds: 0keeps the previous unbounded behavior.RunHelper.wait_for_model_endpoints()no longer callsshutdown()itself. Its only caller isstart(), which now performs the cleanup.