Repository navigation
fix(openai): jittered backoff, Retry-After and a default provider timeout - #4786
Merged
AdamRajfer merged 4 commits intoOct 20, 2026
Merged
Conversation
Contributor
|
🌿 Preview your docs: https://nvidia-preview-pr-4786-e6ae1f9a8815.docs.buildwithfern.com/nemo/gym |
…eout Retries of 429, 502, 503, 504 and 520 waited a fixed 0.5 s and gave up after max_http_attempts (3) tries, so an overloaded provider was hit again within a second and a burst of 429s became failures. A provider call through openai_model also had no timeout, so a provider that never answered held its rollout until the walltime. - Retries wait with exponential backoff and full jitter (1 s doubling, capped at 60 s), or for the answer's Retry-After (seconds or an HTTP date) within the same cap. A Retry-After of zero or in the past uses the backoff. - A 429 or 503 is retried until max_rate_limit_wait_seconds (default 600) of waiting is spent; other retryable statuses keep max_http_attempts tries. Spent-quota and invalid-key answers still stop at once. Calls between NeMo Gym's own servers keep retrying without limit, with the new waits. - The vllm_model clients set the wait budget to 0 and keep 3 tries, so a job still notices a down model server quickly. - openai_model's upstream_request_timeout_seconds defaults to 1800 s per attempt; null keeps the old unlimited wait. - GDPval's judge proxies set max_http_attempts to 36 instead of 1800, which keeps their window at about 15 minutes on average with the new waits (at most about 30). - The model_retry log line shows each wait. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
AdamRajfer
force-pushed
the
fix/openai-backoff-with-jitter
branch
from
October 20, 2026 14:29
b289630 to
f59c005
Compare
Contributor
Author
|
/review |
svcnvidia-nemo-ci
left a comment
Contributor
There was a problem hiding this comment.
SHIP WITH CARE — 2 RISK findings in retry-limit handling.
Completed the changed-file review and traced the affected provider, vLLM, and GDPval retry paths. No BLOCKER found; the reliability risks below warrant an explicit merge decision.
Static review only: tests were not executed. Suggested code is source-reasoned and must be verified before adoption.
Review generated by codex
…ponent Two findings from the automated review: - A busy provider (429/503) past max_rate_limit_wait_seconds stopped extending the attempt count but could still use attempts that max_http_attempts allowed, so a client with 36 attempts and a 2 s budget slept 35 s. A positive budget now ends the request once the next wait would pass it. A budget of 0 keeps 429 and 503 on the max_http_attempts count, also for a zero-length jittered wait. - The full-jitter window computed 2 ** (tries - 1) before the 60 s cap, which overflows a float after 1024 tries on the unlimited internal retry path. The exponent is capped at 6 (64 s, past the cap). The GDPval judge proxies set max_rate_limit_wait_seconds: 0, so they keep their documented 36-attempt window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
Review feedback: a 10-minute default for max_rate_limit_wait_seconds could interrupt ongoing work while a shared inference endpoint is badly backed up. The default is now 3600 s, so 429 and 503 answers are waited out for up to an hour before the request fails. Callers that want a shorter window set it themselves; vllm_model clients keep 0 and the GDPval judge proxies keep 0 (count-only, 36 attempts). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
Contributor
Author
|
/ok to test 2ce62fd |
bxyu-nvidia
approved these changes
Oct 20, 2026
AdamRajfer
enabled auto-merge
October 20, 2026 17:58
Contributor
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/39474988729 |
7 tasks done
8 tasks
This was referenced Nov 9, 2026
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Retries of 429, 502, 503, 504 and 520 waited a fixed 0.5 s and gave up after
max_http_attempts(3) tries, so an overloaded provider was hit again within a second and a burst of 429s became failures. A provider call throughopenai_modelalso had no timeout, so a provider that never answered held its rollout until the walltime.Changes:
Retry-After(seconds or an HTTP date) within the same cap. ARetry-Afterof zero or in the past uses the backoff.max_rate_limit_wait_seconds(default 3600, one hour) of waiting is spent; other retryable statuses keepmax_http_attemptstries. Spent-quota and invalid-key answers still stop at once. Calls between NeMo Gym's own servers keep retrying without limit, with the new waits.vllm_modelclients set the wait budget to 0 and keep 3 tries, so a job still notices a down model server quickly.openai_model'supstream_request_timeout_secondsnow defaults to 1800 s per attempt, so a stuck provider call fails and can be retried;nullkeeps the old unlimited wait.max_http_attemptsto 36 instead of 1800, which keeps their window at about 15 minutes on average with the new waits (at most about 30).model_retrylog line shows each wait, and the OpenAI model server docs describe the new field and default.Behaviour to note: a single provider attempt through
openai_modelthat runs longer than 30 minutes now times out unlessupstream_request_timeout_secondsis raised or set tonull.Tested with unit tests on mocked time: jittered waits,
Retry-Afteras seconds, as a date and as zero, the 429 and 503 wait budget, permanent errors, and the client defaults inopenai_modelandvllm_model; the GDPval judge panel tests.test_openai_utils.py, theopenai_model,vllm_modeland GDPval judge panel suites pass (766 tests).🤖 Generated with Claude Code