Skip to content

fix: bound model request retries - #2527

Draft
pplattner wants to merge 4 commits into
NVIDIA-NeMo:mainfrom
elisam0:fix/bounded-model-retries-upstream
Draft

pplattner wants to merge 4 commits into
NVIDIA-NeMo:mainfrom
elisam0:fix/bounded-model-retries-upstream

Conversation

@pplattner

@pplattner pplattner commented Aug 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • cap model-request retries at 120 seconds
  • use exponential backoff and honor Retry-After delay-seconds and HTTP-date values

Context

During a five-shard benchmark evaluation against a hosted endpoint, the model endpoint began returning HTTP 429 responses. Each shard ran with concurrency 10, allowing roughly 50 concurrent agent slots for this evaluation while other hosted-model evaluations were also active.

The retry loop started with MAX_NUM_TRIES = 3, but increased max_num_tries for every rate-limit response. Because both the attempt counter and the maximum increased together, a request could never exhaust its retry budget while the endpoint continued returning 429. Every failed request was then retried after a fixed 0.5-second sleep.

The affected evaluation recorded approximately 6.17 million retry attempts over about 22.5 hours. Individual logical requests reached more than 10,000 attempts, with the longest observed chain reaching 10,895. This produced a sustained lower-bound average of roughly 76 retry requests per second, in addition to normal model calls. Although the workload was a legitimate benchmark, this retry amplification made its traffic resemble a DDoS to the hosted endpoint.

This change adds a hard 120-second wall-clock retry budget so rate-limit responses cannot extend retries indefinitely. It also replaces the fixed delay with exponential backoff and honors server-provided Retry-After guidance in both delay-seconds and HTTP-date formats.

Tests

  • ruff format --check nemo_gym/openai_utils.py nemo_gym/server_utils.py tests/unit_tests/test_openai_utils.py
  • ruff check nemo_gym/openai_utils.py nemo_gym/server_utils.py tests/unit_tests/test_openai_utils.py
  • pytest tests/unit_tests/test_openai_utils.py -q — 128 passed

@copy-pr-bot

copy-pr-bot Bot commented Aug 13, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: Patrick Plattner <pplattner@nvidia.com>
@pplattner
pplattner force-pushed the fix/bounded-model-retries-upstream branch from 8a4b7b5 to 009d2fe Compare August 13, 2026 09:40
@github-actions github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Aug 14, 2026
Bounded backoff (exponential, Retry-After-aware, capped at MAX_SECONDS_TRIES)
now only applies when the request isn't marked _internal.

_internal already reaches this function today (NeMoGymAsyncOpenAI.internal ->
_request()'s kwargs -> here).

Signed-off-by: Riccardo Roveri <rroveri@nvidia.com>
# Conflicts:
#	tests/unit_tests/test_openai_utils.py
@ananthsub

Copy link
Copy Markdown
Contributor

This remains the provider-response implementation candidate for #2845. Before landing, it should consume the single logical request budget defined by #3159, preserve the external-vs-internal ownership added by #3079, honor Retry-After within the remaining budget, and treat terminal input errors separately. A transport retry or changing sequence of status codes must not reset accounting.

@svcnvidia-nemo-ci svcnvidia-nemo-ci added area:model Model servers, inference providers, and model adapters bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer labels Oct 20, 2026
khazic pushed a commit to khazic/Gym-lao that referenced this pull request Nov 12, 2026
* fix(model): bound internal provider-response retries

An internal NeMoGymAsyncOpenAI client raised its attempt cap on every
429/502/503/504/520 answer, so a saturated provider was retried forever
(NVIDIA-NeMo#2845; the failure NVIDIA-NeMo#2527 reported). Internal clients now wait out those
statuses on the same elapsed budget (max_rate_limit_wait_seconds) that
external clients use for 429/503, and fall back to max_http_attempts when
the budget is 0. One try counter and one wait budget cover a changing
status sequence.

- A propagated context_length_exceeded answer is terminal whatever its
  status; input, auth and permission statuses were already not retried.
- An error body is cut to a 64 KiB prefix before a retry decision and on
  the final error; the final status is kept.
- The model servers pass internal=False explicitly. The internal flag stays.
- Tests: unit tests with a fake sleep (mixed statuses, Retry-After bounds,
  cancellation, concurrent callers) and real-HTTP tests against a stub
  provider. The old test that pinned unbounded retries is replaced.

Moving retry-vs-fail decisions into the shared outcome policy is a
follow-up.

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

* fix(model): classify the whole error body before keeping its prefix

The error body was cut to 64 KiB before the permanent-quota, invalid-key and
context-overflow checks parsed it. A valid JSON reply longer than that became
incomplete JSON, so its error code was lost: a spent-quota 429 was retried
for the whole wait budget, an invalid-key 401/403 no longer tripped the
client, and an oversized context_length_exceeded reply was retried.

The complete body is now classified first; only its first 64 KiB is kept for
diagnostics and the final error. Truncation no longer changes retry
classification.

Also qualify the documentation: HTTP-status retries are bounded, while
connection-error retries have separate limits and can still be unlimited, and
the internal wait-out statuses are named exactly (429, 502, 503, 504, 520).

Tests: oversized valid JSON with the code before or after the long text and
inside the detail.error wrapper, for internal and external clients, over a
fake provider and a real HTTP stub.

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

* fix(model): count elapsed time and keep terminal codes in the retry budget

The provider-retry budget summed only scheduled sleeps, so response time and
body reads never spent it. It now counts time since the call started, for
internal and external clients.

An oversized JSON error body was cut to a prefix, which a Gym model server
then forwarded as incomplete JSON; the next client could not read its terminal
code and retried. The kept body is now rebuilt as valid JSON of at most 64 KiB
with its code and type intact. Small and non-JSON bodies are unchanged.

Tests: fake-clock latency and body-read budget tests, a real-HTTP slow
provider test, the forwarded-error chain through the model server's handler,
and a request ceiling in the replaced unit test.

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

---------

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>
@yaoyu-33
yaoyu-33 marked this pull request as draft November 18, 2026 02:15
@yaoyu-33

Copy link
Copy Markdown
Contributor

We’re doing some cleanup of the open PR list. This PR has had no activity for over a month, so I’ve moved it back to draft.

If this change is still needed, please resolve any merge conflicts and mark the PR ready for review again when it’s ready to merge. Thanks!

@github-actions github-actions Bot removed the sla:triage-overdue Review assignment is over the one-business-day SLA label Nov 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:model Model servers, inference providers, and model adapters bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants