Repository navigation
Conversation
Signed-off-by: Patrick Plattner <pplattner@nvidia.com>
pplattner
force-pushed
the
fix/bounded-model-retries-upstream
branch
from
August 13, 2026 09:40
8a4b7b5 to
009d2fe
Compare
8 tasks
Bounded backoff (exponential, Retry-After-aware, capped at MAX_SECONDS_TRIES) now only applies when the request isn't marked _internal. _internal already reaches this function today (NeMoGymAsyncOpenAI.internal -> _request()'s kwargs -> here). Signed-off-by: Riccardo Roveri <rroveri@nvidia.com>
# Conflicts: # tests/unit_tests/test_openai_utils.py
# Conflicts: # tests/unit_tests/test_openai_utils.py
This was referenced Sep 9, 2026
Contributor
|
This remains the provider-response implementation candidate for #2845. Before landing, it should consume the single logical request budget defined by #3159, preserve the external-vs-internal ownership added by #3079, honor Retry-After within the remaining budget, and treat terminal input errors separately. A transport retry or changing sequence of status codes must not reset accounting. |
This was referenced Oct 22, 2026
khazic
pushed a commit
to khazic/Gym-lao
that referenced
this pull request
Nov 12, 2026
* fix(model): bound internal provider-response retries An internal NeMoGymAsyncOpenAI client raised its attempt cap on every 429/502/503/504/520 answer, so a saturated provider was retried forever (NVIDIA-NeMo#2845; the failure NVIDIA-NeMo#2527 reported). Internal clients now wait out those statuses on the same elapsed budget (max_rate_limit_wait_seconds) that external clients use for 429/503, and fall back to max_http_attempts when the budget is 0. One try counter and one wait budget cover a changing status sequence. - A propagated context_length_exceeded answer is terminal whatever its status; input, auth and permission statuses were already not retried. - An error body is cut to a 64 KiB prefix before a retry decision and on the final error; the final status is kept. - The model servers pass internal=False explicitly. The internal flag stays. - Tests: unit tests with a fake sleep (mixed statuses, Retry-After bounds, cancellation, concurrent callers) and real-HTTP tests against a stub provider. The old test that pinned unbounded retries is replaced. Moving retry-vs-fail decisions into the shared outcome policy is a follow-up. Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * fix(model): classify the whole error body before keeping its prefix The error body was cut to 64 KiB before the permanent-quota, invalid-key and context-overflow checks parsed it. A valid JSON reply longer than that became incomplete JSON, so its error code was lost: a spent-quota 429 was retried for the whole wait budget, an invalid-key 401/403 no longer tripped the client, and an oversized context_length_exceeded reply was retried. The complete body is now classified first; only its first 64 KiB is kept for diagnostics and the final error. Truncation no longer changes retry classification. Also qualify the documentation: HTTP-status retries are bounded, while connection-error retries have separate limits and can still be unlimited, and the internal wait-out statuses are named exactly (429, 502, 503, 504, 520). Tests: oversized valid JSON with the code before or after the long text and inside the detail.error wrapper, for internal and external clients, over a fake provider and a real HTTP stub. Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * fix(model): count elapsed time and keep terminal codes in the retry budget The provider-retry budget summed only scheduled sleeps, so response time and body reads never spent it. It now counts time since the call started, for internal and external clients. An oversized JSON error body was cut to a prefix, which a Gym model server then forwarded as incomplete JSON; the next client could not read its terminal code and retried. The kept body is now rebuilt as valid JSON of at most 64 KiB with its code and type intact. Small and non-JSON bodies are unchanged. Tests: fake-clock latency and body-read budget tests, a real-HTTP slow provider test, the forwarded-error chain through the model server's handler, and a request ceiling in the replaced unit test. Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> --------- Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>
yaoyu-33
marked this pull request as draft
November 18, 2026 02:15
Contributor
|
We’re doing some cleanup of the open PR list. This PR has had no activity for over a month, so I’ve moved it back to draft. If this change is still needed, please resolve any merge conflicts and mark the PR ready for review again when it’s ready to merge. Thanks! |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Retry-Afterdelay-seconds and HTTP-date valuesContext
During a five-shard benchmark evaluation against a hosted endpoint, the model endpoint began returning HTTP 429 responses. Each shard ran with concurrency 10, allowing roughly 50 concurrent agent slots for this evaluation while other hosted-model evaluations were also active.
The retry loop started with
MAX_NUM_TRIES = 3, but increasedmax_num_triesfor every rate-limit response. Because both the attempt counter and the maximum increased together, a request could never exhaust its retry budget while the endpoint continued returning 429. Every failed request was then retried after a fixed 0.5-second sleep.The affected evaluation recorded approximately 6.17 million retry attempts over about 22.5 hours. Individual logical requests reached more than 10,000 attempts, with the longest observed chain reaching 10,895. This produced a sustained lower-bound average of roughly 76 retry requests per second, in addition to normal model calls. Although the workload was a legitimate benchmark, this retry amplification made its traffic resemble a DDoS to the hosted endpoint.
This change adds a hard 120-second wall-clock retry budget so rate-limit responses cannot extend retries indefinitely. It also replaces the fixed delay with exponential backoff and honors server-provided
Retry-Afterguidance in both delay-seconds and HTTP-date formats.Tests
ruff format --check nemo_gym/openai_utils.py nemo_gym/server_utils.py tests/unit_tests/test_openai_utils.pyruff check nemo_gym/openai_utils.py nemo_gym/server_utils.py tests/unit_tests/test_openai_utils.pypytest tests/unit_tests/test_openai_utils.py -q— 128 passed