Skip to content

fix(cli): bound Gym server startup wait - #3735

Merged
ananthsub merged 4 commits into
NVIDIA-NeMo:mainfrom
ananthsub:ananthsub/server-spinup-timeout
Sep 29, 2026
Merged

ananthsub merged 4 commits into
NVIDIA-NeMo:mainfrom
ananthsub:ananthsub/server-spinup-timeout

Conversation

@ananthsub

@ananthsub ananthsub commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

What changed

Add a configurable server_spinup_timeout_seconds bound to gym env start server readiness polling. The default is 600 seconds; 0 preserves the previous unbounded wait for debugging. Each readiness probe and poll sleep is capped at the time left, so the wait ends at the deadline no matter how many servers are stalled. When the deadline expires, Gym raises an error that names the instances that never became ready and explains how to raise or disable the limit. The value is validated before anything starts: numeric strings from environment interpolation are accepted, and malformed values raise ConfigError.

Move startup cleanup to one boundary in RunHelper.start(). Any exception or interrupt raised before readiness now shuts down every server started so far and then re-raises the original exception. Before this change, only two failure paths cleaned up, and each called shutdown() inline: the model-endpoint timeout, and this PR's new timeout. The boundary also covers a server that exits during startup, Ctrl-C during startup, and a failure partway through spawning servers. Each of these used to orphan the servers that were already running.

RunHelper.shutdown() now tolerates being called before the head server starts and being called more than once. NeMo-RL relies on repeated calls: when startup fails, it calls shutdown() again on the same RunHelper.

Part of #3160. This PR covers the component-readiness deadline and the startup-unwind criteria for the stages start() owns. It does not bound head-server or model-endpoint readiness. It also does not add process-group ownership of descendants or a single total shutdown deadline.

Why

A server process can stay alive without ever becoming ready. For example, a Uvicorn supervisor can keep replacing workers that fail while the application is being constructed. The launcher only treated process exit as failure, so it waited forever. On scheduled infrastructure this can hold the allocation until its external walltime.

All RunHelper callers invoke start() outside their own try block. The spawned servers have no process group and no atexit handler, and the head-server thread is a daemon. So when startup failed on any path that did not call shutdown() explicitly, the orchestrator exited and the already-started servers were reparented to init and kept holding their ports and GPUs.

Validation

  • pytest tests/unit_tests/test_cli.py tests/unit_tests/test_cli_endpoint_readiness.py tests/unit_tests/test_global_config.py tests/unit_tests/test_server_utils.py -q: 356 passed.
  • Two real stalled HTTP servers with server_spinup_timeout_seconds=1: raises in 1.00s. Before readiness probes were bounded by the deadline, this took 10.01s.
  • Real gym env start with ++server_spinup_timeout_seconds='ten minutes': exits 1 with a ConfigError message before Ray or any server starts.
  • Scoped pre-commit over all changed files: passed.
  • Ran a real gym env start with two local servers. crasher exits 8 seconds into startup, and hanger stays alive but never becomes ready.
    • Before the boundary: RuntimeError: Process crasher finished unexpectedly!, and hanger was left running with parent PID 1.
    • After: the same error reaches the caller, shutdown() runs, and hanger exits.
  • Sent SIGINT only to the gym process while it waited on hanger. Before the boundary, hanger was left running with parent PID 1. After, shutdown() runs and hanger exits.

Rollout evidence

N/A. This changes CLI startup failure handling and does not alter model requests, trajectories, verification, or rewards.

Compatibility

  • Default server startup wait is now bounded at 600 seconds.
  • Long-starting deployments can raise server_spinup_timeout_seconds.
  • Setting server_spinup_timeout_seconds: 0 keeps the previous unbounded behavior.
  • RunHelper.wait_for_model_endpoints() no longer calls shutdown() itself. Its only caller is start(), which now performs the cleanup.

@ananthsub ananthsub added bug Something isn't working area:cli Command-line interfaces and user-facing command behavior needs-review PR is ready for code review and waiting on a reviewer labels Sep 28, 2026
@nemo-automation-bot
nemo-automation-bot Bot requested review from a team September 28, 2026 19:46
@github-actions

Copy link
Copy Markdown
Contributor

🌿 Preview your docs: https://nvidia-preview-ananthsub-server-spinup-timeout.docs.buildwithfern.com/nemo/gym

Here are the markdown pages you've updated:

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
Shut down every started server when RunHelper.start() fails or is
interrupted, instead of only on the two readiness paths that called
shutdown() inline. Make shutdown() safe before the head server starts
and on repeated calls.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
@ananthsub
ananthsub force-pushed the ananthsub/server-spinup-timeout branch from 4286672 to 1bd0319 Compare September 28, 2026 22:31
@yaoyu-33 yaoyu-33 added the complexity:low Localized change in one scope with a small, straightforward review surface label Sep 28, 2026

@yuhezhang-ai yuhezhang-ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address the readiness deadline; the other three comments are non-blocking suggestions. Validation: 265 tests passed, with focused HTTP and process cleanup probes.

Comment thread nemo_gym/cli/env.py Resolved
Comment thread nemo_gym/cli/env.py Resolved
Comment thread nemo_gym/cli/env.py Resolved
Comment thread tests/unit_tests/test_cli.py Resolved
Cap each server readiness probe and the poll sleep at the time left
before server_spinup_timeout_seconds, and skip probes once it passes,
so many stalled servers cannot stretch the wait past the deadline.

Validate the setting before anything starts: accept numeric strings from
environment interpolation, fall back to the default for null, and raise
ConfigError for anything else. Name the override and zero opt-out in the
timeout error, and document that the budget covers dependency install
and local model loading.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
yuhezhang-ai
yuhezhang-ai previously approved these changes Sep 29, 2026

@yuhezhang-ai yuhezhang-ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The four earlier findings are addressed. All 356 focused tests passed, and the stalled-server reproduction now respects the 1-second limit. Approved, with one non-blocking documentation wording request below.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>

@yuhezhang-ai yuhezhang-ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The documentation wording is corrected. Confirmed this is the only change since the reviewed revision; approval renewed.

@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/36606883896

@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/36608982967

Merged via the queue into NVIDIA-NeMo:main with commit 329683e Sep 29, 2026
42 checks passed
@ananthsub
ananthsub deleted the ananthsub/server-spinup-timeout branch September 29, 2026 18:14
trias702 pushed a commit that referenced this pull request Oct 30, 2026
Add a configurable `server_spinup_timeout_seconds` bound to `gym env
start` server readiness polling. The default is 600 seconds; `0`
preserves the previous unbounded wait for debugging. Each readiness
probe and poll sleep is capped at the time left, so the wait ends at the
deadline no matter how many servers are stalled. When the deadline
expires, Gym raises an error that names the instances that never became
ready and explains how to raise or disable the limit. The value is
validated before anything starts: numeric strings from environment
interpolation are accepted, and malformed values raise `ConfigError`.

Move startup cleanup to one boundary in `RunHelper.start()`. Any
exception or interrupt raised before readiness now shuts down every
server started so far and then re-raises the original exception. Before
this change, only two failure paths cleaned up, and each called
`shutdown()` inline: the model-endpoint timeout, and this PR's new
timeout. The boundary also covers a server that exits during startup,
Ctrl-C during startup, and a failure partway through spawning servers.
Each of these used to orphan the servers that were already running.

`RunHelper.shutdown()` now tolerates being called before the head server
starts and being called more than once. NeMo-RL relies on repeated
calls: when startup fails, it calls `shutdown()` again on the same
`RunHelper`.

Part of #3160. This PR covers the component-readiness deadline and the
startup-unwind criteria for the stages `start()` owns. It does not bound
head-server or model-endpoint readiness. It also does not add
process-group ownership of descendants or a single total shutdown
deadline.

A server process can stay alive without ever becoming ready. For
example, a Uvicorn supervisor can keep replacing workers that fail while
the application is being constructed. The launcher only treated process
exit as failure, so it waited forever. On scheduled infrastructure this
can hold the allocation until its external walltime.

All `RunHelper` callers invoke `start()` outside their own `try` block.
The spawned servers have no process group and no atexit handler, and the
head-server thread is a daemon. So when startup failed on any path that
did not call `shutdown()` explicitly, the orchestrator exited and the
already-started servers were reparented to init and kept holding their
ports and GPUs.

- `pytest tests/unit_tests/test_cli.py
tests/unit_tests/test_cli_endpoint_readiness.py
tests/unit_tests/test_global_config.py
tests/unit_tests/test_server_utils.py -q`: 356 passed.
- Two real stalled HTTP servers with `server_spinup_timeout_seconds=1`:
raises in 1.00s. Before readiness probes were bounded by the deadline,
this took 10.01s.
- Real `gym env start` with `++server_spinup_timeout_seconds='ten
minutes'`: exits 1 with a `ConfigError` message before Ray or any server
starts.
- Scoped pre-commit over all changed files: passed.
- Ran a real `gym env start` with two local servers. `crasher` exits 8
seconds into startup, and `hanger` stays alive but never becomes ready.
- Before the boundary: `RuntimeError: Process crasher finished
unexpectedly!`, and `hanger` was left running with parent PID 1.
- After: the same error reaches the caller, `shutdown()` runs, and
`hanger` exits.
- Sent `SIGINT` only to the `gym` process while it waited on `hanger`.
Before the boundary, `hanger` was left running with parent PID 1. After,
`shutdown()` runs and `hanger` exits.

N/A. This changes CLI startup failure handling and does not alter model
requests, trajectories, verification, or rewards.

- Default server startup wait is now bounded at 600 seconds.
- Long-starting deployments can raise `server_spinup_timeout_seconds`.
- Setting `server_spinup_timeout_seconds: 0` keeps the previous
unbounded behavior.
- `RunHelper.wait_for_model_endpoints()` no longer calls `shutdown()`
itself. Its only caller is `start()`, which now performs the cleanup.

---------

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
(cherry picked from commit 329683e)

Conflict in tests/unit_tests/test_cli.py resolved for this branch: kept the 13 tests #3735
adds, with its 3 wait_for_spinup timeout tests in a new TestRunHelperServerSpinupTimeout class
because TestRunHelperServerReadiness does not exist here. Dropped the 4 tests from other main
commits that shared the conflict hunks (head-ready ordering, /health while starting, and two
memory-profiler tests), since they exercise code this branch does not have.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was previously deployed

1 inactive deployment
public — f0e7b47f Deployed Sep 29, 2026 by copy-pr-bot[bot] via release / finalize / notify #3950
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:cli Command-line interfaces and user-facing command behavior bug Something isn't working complexity:low Localized change in one scope with a small, straightforward review surface needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants