Skip to content

Comp coding fixes; lots of misc infra items - #90

Merged
bxyu-nvidia merged 113 commits into
mainfrom
bxyu/dev-20250923
Sep 27, 2025
Merged

bxyu-nvidia merged 113 commits into
mainfrom
bxyu/dev-20250923

Conversation

@bxyu-nvidia

Copy link
Copy Markdown
Contributor

No description provided.

Signed-off-by: Brian Yu <bxyu@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 23, 2025

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
@bxyu-nvidia bxyu-nvidia changed the title bxyu/dev-20250923 Profiling exposes /stats endpoint Sep 24, 2025
@bxyu-nvidia bxyu-nvidia changed the title Profiling exposes /stats endpoint Profiling exposes /stats endpoint; comp coding fixes Sep 24, 2025
@bxyu-nvidia
bxyu-nvidia marked this pull request as ready for review September 24, 2025 00:55
Signed-off-by: Brian Yu <bxyu@nvidia.com>
@bxyu-nvidia
bxyu-nvidia marked this pull request as draft September 24, 2025 03:23
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 26, 2025

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
@bxyu-nvidia
bxyu-nvidia marked this pull request as ready for review September 26, 2025 23:38
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
@bxyu-nvidia
bxyu-nvidia merged commit 8e41fc2 into main Sep 27, 2025
@bxyu-nvidia
bxyu-nvidia deleted the bxyu/dev-20250923 branch September 27, 2025 01:19
abhibha-nvidia pushed a commit that referenced this pull request Sep 28, 2025
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Khushi Bhardwaj <kbhardwaj@nvidia.com>
Co-authored-by: Khushi Bhardwaj <kbhardwaj@nvidia.com>
Signed-off-by: Abhibha Gupta <abhibhag@nvidia.com>
abhibha-nvidia pushed a commit that referenced this pull request Sep 29, 2025
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Khushi Bhardwaj <kbhardwaj@nvidia.com>
Co-authored-by: Khushi Bhardwaj <kbhardwaj@nvidia.com>
Signed-off-by: Abhibha Gupta <abhibhag@nvidia.com>
sidnarayanan added a commit that referenced this pull request Oct 29, 2025
* Instantiate one httpx async client per unique connection / base url (#75)

Signed-off-by: Brian Yu <bxyu@nvidia.com>

* handle model calling failures, e.g. max token limits

* Swap async http backend from httpx to aiohttp; various server infra improvements (#77)

Signed-off-by: Brian Yu <bxyu@nvidia.com>

* Remove unnecessary GHA CI and add uv config to enable dependency scanning (#66)

Remove unnecessary Github Action CI and add uv config to enable
dependency scanning

* This project's current CI doesn't need to build and test through a
docker image. So, deleting the unnecssary CI Dockerfile and Github
Actions template
* Adding `managed = true` under `[uv.tool]` to allow for repo dependency
scanning

---------

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* VLLMModel fix whitespace stripping and unwarranted spaces (#70)

Signed-off-by: Brian Yu <bxyu@nvidia.com>

* gracefully handle vllm failures

* check in metrics

* Fix aggregation rounding in ng_prepare_data (#76)

This implements a simple rounding rule for `AvgMinMax` floats in order
to keep example_metrics consistent.
For background, the addition of median and std dev did not assign a
ceiling for decimal places, so trivial value differences such as `1.2 !=
1.200002` caused ValueErrors.

---------

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

* working on returning a list of transitions

* Add profiling; improve rollout collection usability and efficiency; add uvicorn logging filtering (#79)

Signed-off-by: Brian Yu <bxyu@nvidia.com>

* break if over token limit

* log total length when response call fails

* bbh

* update_agent_state

* EnvStateMessage support

* support collapsing env states during rollout

* fix import?

* bbh data prep

* set default factory

* Delete .github/ISSUE_TEMPLATE directory (#87)

From now on, any Github repo under the Nvidia-NeMo org will use the
default Issues system / behavior
(unless overwritten i.e. in this case the individual repo would have a
.github/ISSUE_TEMPLATE folder in the repo itself, effectively overriding
the default behavior coming from the NVIDIA-NeMo/.github repo [which
establishes the default system]).

Default behavior for Issues creation now is :
- There is a Bug Report issue
- There is a Feature Request issue
- Blank issues are NOT allowed to be created

* wip

* drop secrets

* Add support for `num_repeats` (#99)

# Add `num_repeats` hyperparameter for dataset repetition

## Summary
Adds optional `num_repeats` parameter to `DatasetConfig` that allows
repeating each dataset sample during training data processing and
preparation.

## Changes
- **Config**: Added `num_repeats: Optional[int] = Field(default=None,
ge=1)` to `DatasetConfig`
- **Processing**: Modified `_iter_dataset_lines()` to repeat each line
`num_repeats` times (defaults to 1)
- **Integration**: Updated data validation, metrics aggregation, and
preparation workflows to handle repeated samples
- **Documentation**: Updated README with usage examples

## Usage
```yaml
datasets:
  - name: train
    type: train
    jsonl_fpath: data/train.jsonl
    num_repeats: 3  # Each sample appears 3 times during processing
```

## Testing
Added comprehensive unit tests covering:
- Configuration validation (accepts positive integers, rejects invalid
values)
- Data iteration with different repeat values
- Metrics aggregation with repeated samples
- Data preparation workflow integration

All existing functionality preserved with backward compatibility
(defaults to 1 repetition).

---------

Signed-off-by: Mahan Fathi <mfathi@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Co-authored-by: Brian Yu <bxyu@nvidia.com>

* Comp coding fixes; lots of misc infra items (#90)

Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Khushi Bhardwaj <kbhardwaj@nvidia.com>
Co-authored-by: Khushi Bhardwaj <kbhardwaj@nvidia.com>

* chore: Update cherry-pick workflow to use v0.63.0 (#108)

* Make Workbench stateful and sign commits  (#110)

Signed-off-by: Abhibha Gupta <abhibhag@nvidia.com>

* better max_seq_len check, more error handling

* Clean deprecated Comp coding (#106)

Signed-off-by: Brian Yu <bxyu@nvidia.com>

* logging etc

* await json()

* bunch of fixes for EnvStateMessage hiding and Qwen3

* log max seq len too

* set max_steps in config

* update no-tool-call message

* actually collapse

* only collapse env state messages after  successful transitions

* allow postprocessing empty message

* Bxyu/misc infra 20251001 (#116)

Signed-off-by: Brian Yu <bxyu@nvidia.com>

* Resource Server Organization (#80)

This PR adds a pre-commit hook that scans the resource_server
directories and populates/updates the designated table in the readme
with the pertinent information.

Addresses item 1 from #81

---------

Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Co-authored-by: Brian Yu <bxyu@nvidia.com>

* Add metrics conflict error FAQ to Readme (#93)

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

* Azure OpenAI model support (#112)

Signed-off-by: slikhite-1 <slikhite@nvidia.com>
Co-authored-by: slikhite-1 <slikhite@nvidia.com>

* Use python env for precommit hook; alter files trigger (#125)

This change uses python instead of bash system language to get around
the `python: command not found` error when using vs code source control
gui.

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

* retry seed session if it fails

* Update issue templates (#152)

Signed-off-by: Brian Yu <bxyu@nvidia.com>

* Add back Nemo Framework templates (#153)

Signed-off-by: Brian Yu <bxyu@nvidia.com>

* Fix Workbench invalid function name (#167)

Signed-off-by: Brian Yu <bxyu@nvidia.com>

* VLLMModel enable reasoning parsing (#129)

Signed-off-by: Brian Yu <bxyu@nvidia.com>

* raise reset response for status

* Add Attributions for Third Party Softwares (#154)

Add Attributions for Third Party Softwares.

---------

Signed-off-by: banghuaz <banghuaz@nvidia.com>

* Fix infinite OpenAI endpoint query; misc improvements (#171)

Signed-off-by: Brian Yu <bxyu@nvidia.com>

* set domain; pass cookies

* domain should be coding

* proper gsm8k train/val split

* clean up bbh stuff

---------

Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Signed-off-by: Mahan Fathi <mfathi@nvidia.com>
Signed-off-by: Khushi Bhardwaj <kbhardwaj@nvidia.com>
Signed-off-by: Abhibha Gupta <abhibhag@nvidia.com>
Signed-off-by: slikhite-1 <slikhite@nvidia.com>
Signed-off-by: banghuaz <banghuaz@nvidia.com>
Co-authored-by: bxyu-nvidia <bxyu@nvidia.com>
Co-authored-by: Charlie Truong <chtruong@nvidia.com>
Co-authored-by: fsiino-nvidia <fsiino@nvidia.com>
Co-authored-by: Pablo Garay <palenq@gmail.com>
Co-authored-by: Mahan <25934206+MahanFathi@users.noreply.github.com>
Co-authored-by: Khushi Bhardwaj <kbhardwaj@nvidia.com>
Co-authored-by: abhibha-nvidia <abhibhag@nvidia.com>
Co-authored-by: slikhite-1 <slikhite@nvidia.com>
Co-authored-by: banghuaz-nvidia <banghuaz@nvidia.com>
sidnarayanan added a commit that referenced this pull request Jan 7, 2026
* Instantiate one httpx async client per unique connection / base url (#75)

Signed-off-by: Brian Yu <bxyu@nvidia.com>

* handle model calling failures, e.g. max token limits

* Swap async http backend from httpx to aiohttp; various server infra improvements (#77)

Signed-off-by: Brian Yu <bxyu@nvidia.com>

* Remove unnecessary GHA CI and add uv config to enable dependency scanning (#66)

Remove unnecessary Github Action CI and add uv config to enable
dependency scanning

* This project's current CI doesn't need to build and test through a
docker image. So, deleting the unnecssary CI Dockerfile and Github
Actions template
* Adding `managed = true` under `[uv.tool]` to allow for repo dependency
scanning

---------

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* VLLMModel fix whitespace stripping and unwarranted spaces (#70)

Signed-off-by: Brian Yu <bxyu@nvidia.com>

* gracefully handle vllm failures

* check in metrics

* Fix aggregation rounding in ng_prepare_data (#76)

This implements a simple rounding rule for `AvgMinMax` floats in order
to keep example_metrics consistent.
For background, the addition of median and std dev did not assign a
ceiling for decimal places, so trivial value differences such as `1.2 !=
1.200002` caused ValueErrors.

---------

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

* working on returning a list of transitions

* Add profiling; improve rollout collection usability and efficiency; add uvicorn logging filtering (#79)

Signed-off-by: Brian Yu <bxyu@nvidia.com>

* break if over token limit

* log total length when response call fails

* bbh

* update_agent_state

* EnvStateMessage support

* support collapsing env states during rollout

* fix import?

* bbh data prep

* set default factory

* Delete .github/ISSUE_TEMPLATE directory (#87)

From now on, any Github repo under the Nvidia-NeMo org will use the
default Issues system / behavior
(unless overwritten i.e. in this case the individual repo would have a
.github/ISSUE_TEMPLATE folder in the repo itself, effectively overriding
the default behavior coming from the NVIDIA-NeMo/.github repo [which
establishes the default system]).

Default behavior for Issues creation now is :
- There is a Bug Report issue
- There is a Feature Request issue
- Blank issues are NOT allowed to be created

* wip

* drop secrets

* Add support for `num_repeats` (#99)

# Add `num_repeats` hyperparameter for dataset repetition

## Summary
Adds optional `num_repeats` parameter to `DatasetConfig` that allows
repeating each dataset sample during training data processing and
preparation.

## Changes
- **Config**: Added `num_repeats: Optional[int] = Field(default=None,
ge=1)` to `DatasetConfig`
- **Processing**: Modified `_iter_dataset_lines()` to repeat each line
`num_repeats` times (defaults to 1)
- **Integration**: Updated data validation, metrics aggregation, and
preparation workflows to handle repeated samples
- **Documentation**: Updated README with usage examples

## Usage
```yaml
datasets:
  - name: train
    type: train
    jsonl_fpath: data/train.jsonl
    num_repeats: 3  # Each sample appears 3 times during processing
```

## Testing
Added comprehensive unit tests covering:
- Configuration validation (accepts positive integers, rejects invalid
values)
- Data iteration with different repeat values
- Metrics aggregation with repeated samples
- Data preparation workflow integration

All existing functionality preserved with backward compatibility
(defaults to 1 repetition).

---------

Signed-off-by: Mahan Fathi <mfathi@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Co-authored-by: Brian Yu <bxyu@nvidia.com>

* Comp coding fixes; lots of misc infra items (#90)

Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Khushi Bhardwaj <kbhardwaj@nvidia.com>
Co-authored-by: Khushi Bhardwaj <kbhardwaj@nvidia.com>

* chore: Update cherry-pick workflow to use v0.63.0 (#108)

* Make Workbench stateful and sign commits  (#110)

Signed-off-by: Abhibha Gupta <abhibhag@nvidia.com>

* better max_seq_len check, more error handling

* Clean deprecated Comp coding (#106)

Signed-off-by: Brian Yu <bxyu@nvidia.com>

* logging etc

* await json()

* bunch of fixes for EnvStateMessage hiding and Qwen3

* log max seq len too

* set max_steps in config

* update no-tool-call message

* actually collapse

* only collapse env state messages after  successful transitions

* allow postprocessing empty message

* Bxyu/misc infra 20251001 (#116)

Signed-off-by: Brian Yu <bxyu@nvidia.com>

* Resource Server Organization (#80)

This PR adds a pre-commit hook that scans the resource_server
directories and populates/updates the designated table in the readme
with the pertinent information.

Addresses item 1 from #81

---------

Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Co-authored-by: Brian Yu <bxyu@nvidia.com>

* Add metrics conflict error FAQ to Readme (#93)

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

* Azure OpenAI model support (#112)

Signed-off-by: slikhite-1 <slikhite@nvidia.com>
Co-authored-by: slikhite-1 <slikhite@nvidia.com>

* Use python env for precommit hook; alter files trigger (#125)

This change uses python instead of bash system language to get around
the `python: command not found` error when using vs code source control
gui.

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

* retry seed session if it fails

* Update issue templates (#152)

Signed-off-by: Brian Yu <bxyu@nvidia.com>

* Add back Nemo Framework templates (#153)

Signed-off-by: Brian Yu <bxyu@nvidia.com>

* Fix Workbench invalid function name (#167)

Signed-off-by: Brian Yu <bxyu@nvidia.com>

* VLLMModel enable reasoning parsing (#129)

Signed-off-by: Brian Yu <bxyu@nvidia.com>

* raise reset response for status

* Add Attributions for Third Party Softwares (#154)

Add Attributions for Third Party Softwares.

---------

Signed-off-by: banghuaz <banghuaz@nvidia.com>

* Fix infinite OpenAI endpoint query; misc improvements (#171)

Signed-off-by: Brian Yu <bxyu@nvidia.com>

* set domain; pass cookies

* domain should be coding

* proper gsm8k train/val split

* clean up bbh stuff

---------

Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Signed-off-by: Mahan Fathi <mfathi@nvidia.com>
Signed-off-by: Khushi Bhardwaj <kbhardwaj@nvidia.com>
Signed-off-by: Abhibha Gupta <abhibhag@nvidia.com>
Signed-off-by: slikhite-1 <slikhite@nvidia.com>
Signed-off-by: banghuaz <banghuaz@nvidia.com>
Co-authored-by: bxyu-nvidia <bxyu@nvidia.com>
Co-authored-by: Charlie Truong <chtruong@nvidia.com>
Co-authored-by: fsiino-nvidia <fsiino@nvidia.com>
Co-authored-by: Pablo Garay <palenq@gmail.com>
Co-authored-by: Mahan <25934206+MahanFathi@users.noreply.github.com>
Co-authored-by: Khushi Bhardwaj <kbhardwaj@nvidia.com>
Co-authored-by: abhibha-nvidia <abhibhag@nvidia.com>
Co-authored-by: slikhite-1 <slikhite@nvidia.com>
Co-authored-by: banghuaz-nvidia <banghuaz@nvidia.com>

Signed-off-by: Siddharth Narayanan <sidnarayanan@users.noreply.github.com>
abubakaria56 pushed a commit to abubakaria56/Gym that referenced this pull request Mar 2, 2026
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Khushi Bhardwaj <kbhardwaj@nvidia.com>
Co-authored-by: Khushi Bhardwaj <kbhardwaj@nvidia.com>
abubakaria56 pushed a commit to abubakaria56/Gym that referenced this pull request Mar 2, 2026
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Khushi Bhardwaj <kbhardwaj@nvidia.com>
Co-authored-by: Khushi Bhardwaj <kbhardwaj@nvidia.com>
khazic pushed a commit to khazic/Gym-lao that referenced this pull request Oct 13, 2026
## What does this PR do?

Follow-up to NVIDIA-NeMo#3632, split out as requested in
NVIDIA-NeMo#3632 (comment).

In `_request_with_retries()`, the generic `except Exception` branch
handles every error except `ServerDisconnectedError` and
`ClientOSError`, which have their own branches. This includes aiohttp
timeouts. When the caller does not pass `_max_num_tries`, this branch
counted tries only for external callers. So an internal caller
(`_internal=True`) that set `_max_connection_retries` to 2 or more never
reached the cap and retried forever.

NVIDIA-NeMo#3415 already counts tries for callers that pass `_max_num_tries`. This
PR fixes the remaining path, where `_max_num_tries` is not passed.

- Count tries for internal callers too, so `_max_connection_retries`
stops them.
- Keep everything else the same:
- internal callers with no cap still retry with no limit and print no
retry message, as NVIDIA-NeMo#90 intended;
  - external callers still stop at `MAX_NUM_TRIES`;
  - callers that pass `_max_num_tries` are unchanged.
- Update the comment that said internal calls are not counted. It keeps
NVIDIA-NeMo#90's reason for not limiting internal calls.
- Add three tests:
  - the cap stops internal callers, with tracing off and on;
  - internal callers with no cap retry silently past `MAX_NUM_TRIES`;
- external callers stop at `MAX_NUM_TRIES` and print one retry message
per try.
- The cap test, `test_request_retry_cap_bounds_internal_callers`,
replaces `test_retry_cap_bounds_internal_callers`, which was removed
from NVIDIA-NeMo#3632.
- Make the retry tests fail instead of hang when the loop never stops:
- The test helper `_mock_global_client` (added in NVIDIA-NeMo#2383) now takes the
error to raise.
- Its fake `request()` calls `pytest.fail()` once it is called more
often than any test needs. `pytest.fail()` raises a `BaseException`,
which the loop's `except Exception` does not catch. So the test fails at
once, even if the loop never sleeps.
- Two NVIDIA-NeMo#3415 tests now use this helper instead of their own mocks:
`test_explicit_transport_attempt_cap_disables_request_retry` and
`test_request_caps_generic_errors_at_the_lower_attempt_limit`. Before,
if the generic branch ignored `_max_num_tries`, the first test hung,
sleeping a real 0.5 s per retry.

No caller in this repo sets both `_internal=True` and
`_max_connection_retries` today:
- The internal callers pass no cap: `ServerClient`, the simple agent's
direct tool calls to Resources servers (`tool_access`), the Terminus 2
sandboxed agent's model client (`NeMoGymAsyncOpenAI(internal=True)`),
the OpenHands capture route, and the LiteLLM version check.
- The callers with a cap are not internal: the remote agent and sandbox
transport use 1, Tavily search uses 0, and vLLM with `endpoint_file`
uses a default of 8.
- The OpenAI model server and the Gymnasium agent pass `_max_num_tries`
only when `upstream_max_num_tries` or
`model_server_transport_max_attempts` is set. Neither passes
`_max_connection_retries`, so neither changes.

Related:
- NVIDIA-NeMo#3553 added the cap check to this branch without counting internal
tries.
- NVIDIA-NeMo#3415 added `_max_num_tries` and counts tries on that path.
- NVIDIA-NeMo#3159 tracks bounding every retry by default, including internal
callers with no cap. This PR only makes an explicit cap work for them.
- NVIDIA-NeMo#2216 tracks the `ServerDisconnectedError` and `ClientOSError`
branches, which this PR does not change.
- Open PR NVIDIA-NeMo#2935 adds a timeout branch that copies the old counting, so
internal callers there still do not count tries. Whichever PR merges
later must count tries for every caller.

## Validation

All runs used an activated venv from `uv sync --extra dev --extra
sandbox --extra openshell --extra telemetry --extra mlflow --extra
wandb` (the extras in `full-test-suite.yml`). Every installed package
matches `uv.lock` (192 packages, including aiohttp 3.14.4, nemo-lens
0.2.0+b0f977d, and pytest 9.0.3).

- `pytest tests/unit_tests/test_server_utils.py
responses_api_agents/remote_agent/tests/test_app.py
tests/unit_tests/telemetry/`: 465 passed.
- With the old code restored,
`test_request_retry_cap_bounds_internal_callers` fails in under a second
with `Failed: retry loop did not stop` instead of hanging. It fails the
same way when the retry sleep is also removed.
- Each change below makes at least one test fail. Each run of
`test_server_utils.py` took about 3 s, and none hung:
  - the cap is ignored;
  - internal callers are limited by `MAX_NUM_TRIES`;
  - internal callers print;
  - external tries are double-counted;
  - external callers stop printing the retry message;
  - the `MAX_NUM_TRIES` raise is removed;
  - the retry sleep after a generic error is skipped;
  - the cap check ignores `_max_num_tries` attempts;
  - the generic branch ignores `_max_num_tries`;
  - the traced path drops `_internal` or `_max_connection_retries`.
- `pytest -n 16 tests/unit_tests
responses_api_agents/remote_agent/tests`: 8089 passed, 5 skipped. The
per-test results match `main` at a3cf403 (8085 passed, 5 skipped). The
4 extra results are the new tests, with the cap test run twice (tracing
off and on).
- The same focused tests pass on a local merge with the latest `main`
(NVIDIA-NeMo#4300), which changes no package versions.
- `pre-commit run --all-files`: passed.

## Rollouts

N/A. This changes a retry counter in the shared HTTP helper. No caller
in this repo changes behavior, and no model, agent, environment, or
verifier logic is involved.

## Compatibility and benchmark impact

None for callers in this repo. A caller that sets `_internal=True` and
`_max_connection_retries` to 2 or more, without `_max_num_tries`, now
stops after that many tries on errors that reach the generic `except
Exception` branch, including timeouts, instead of retrying forever. This
includes code outside this repo that uses
`NeMoGymAsyncOpenAI(internal=True, max_connection_retries=N)`.
`ServerDisconnectedError` and `ClientOSError` keep their own count, as
before.

## Checklist

- [x] I have read the [contributing
guidelines](https://github.com/NVIDIA-NeMo/Gym/blob/main/CONTRIBUTING.md).
- [x] The change is focused; unrelated "drive-by" edits are tracked as
separate issues/PRs.
- [x] Tests added or updated and pass locally, or N/A for docs-only /
non-code changes (so CI unit/server checks pass when applicable).
- [x] Documentation added or updated, or N/A with justification. N/A:
internal helper behavior with no user-facing docs.
- [x] Pre-commit checks pass locally (`pre-commit run --all-files`) (so
CI lint/format checks pass).
- [x] New source files include the required Apache-2.0 SPDX header. N/A:
no new files.
- [x] All commits have DCO sign-off (`git commit -s`) (so the DCO check
passes).

---------

Signed-off-by: Abhishree <athittenaman@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants