Skip to content

fix(openai): jittered backoff, Retry-After and a default provider timeout - #4786

Merged
AdamRajfer merged 4 commits into
NVIDIA-NeMo:mainfrom
AdamRajfer:fix/openai-backoff-with-jitter
Oct 20, 2026
Merged

AdamRajfer merged 4 commits into
NVIDIA-NeMo:mainfrom
AdamRajfer:fix/openai-backoff-with-jitter

Conversation

@AdamRajfer

@AdamRajfer AdamRajfer commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Retries of 429, 502, 503, 504 and 520 waited a fixed 0.5 s and gave up after max_http_attempts (3) tries, so an overloaded provider was hit again within a second and a burst of 429s became failures. A provider call through openai_model also had no timeout, so a provider that never answered held its rollout until the walltime.

Changes:

  • Retries wait with exponential backoff and full jitter (1 s doubling, capped at 60 s), or for the answer's Retry-After (seconds or an HTTP date) within the same cap. A Retry-After of zero or in the past uses the backoff.
  • A 429 or 503 is retried until max_rate_limit_wait_seconds (default 3600, one hour) of waiting is spent; other retryable statuses keep max_http_attempts tries. Spent-quota and invalid-key answers still stop at once. Calls between NeMo Gym's own servers keep retrying without limit, with the new waits.
  • The vllm_model clients set the wait budget to 0 and keep 3 tries, so a job still notices a down model server quickly.
  • openai_model's upstream_request_timeout_seconds now defaults to 1800 s per attempt, so a stuck provider call fails and can be retried; null keeps the old unlimited wait.
  • GDPval's judge proxies set max_http_attempts to 36 instead of 1800, which keeps their window at about 15 minutes on average with the new waits (at most about 30).
  • The model_retry log line shows each wait, and the OpenAI model server docs describe the new field and default.

Behaviour to note: a single provider attempt through openai_model that runs longer than 30 minutes now times out unless upstream_request_timeout_seconds is raised or set to null.

Tested with unit tests on mocked time: jittered waits, Retry-After as seconds, as a date and as zero, the 429 and 503 wait budget, permanent errors, and the client defaults in openai_model and vllm_model; the GDPval judge panel tests. test_openai_utils.py, the openai_model, vllm_model and GDPval judge panel suites pass (766 tests).

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

…eout

Retries of 429, 502, 503, 504 and 520 waited a fixed 0.5 s and gave up
after max_http_attempts (3) tries, so an overloaded provider was hit again
within a second and a burst of 429s became failures. A provider call
through openai_model also had no timeout, so a provider that never
answered held its rollout until the walltime.

- Retries wait with exponential backoff and full jitter (1 s doubling,
  capped at 60 s), or for the answer's Retry-After (seconds or an HTTP
  date) within the same cap. A Retry-After of zero or in the past uses
  the backoff.
- A 429 or 503 is retried until max_rate_limit_wait_seconds (default 600)
  of waiting is spent; other retryable statuses keep max_http_attempts
  tries. Spent-quota and invalid-key answers still stop at once. Calls
  between NeMo Gym's own servers keep retrying without limit, with the
  new waits.
- The vllm_model clients set the wait budget to 0 and keep 3 tries, so a
  job still notices a down model server quickly.
- openai_model's upstream_request_timeout_seconds defaults to 1800 s per
  attempt; null keeps the old unlimited wait.
- GDPval's judge proxies set max_http_attempts to 36 instead of 1800,
  which keeps their window at about 15 minutes on average with the new
  waits (at most about 30).
- The model_retry log line shows each wait.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
@AdamRajfer

Copy link
Copy Markdown
Contributor Author

/review

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP WITH CARE — 2 RISK findings in retry-limit handling.

Completed the changed-file review and traced the affected provider, vLLM, and GDPval retry paths. No BLOCKER found; the reliability risks below warrant an explicit merge decision.

Static review only: tests were not executed. Suggested code is source-reasoned and must be verified before adoption.

Review generated by codex

Comment thread nemo_gym/openai_utils.py Resolved
Comment thread nemo_gym/openai_utils.py Resolved
@yaoyu-33 yaoyu-33 added area:core Shared APIs, servers, telemetry, health, and registries bug Something isn't working complexity:high Cross-component or shared-contract change with a large review surface needs-review PR is ready for code review and waiting on a reviewer labels Oct 20, 2026
…ponent

Two findings from the automated review:

- A busy provider (429/503) past max_rate_limit_wait_seconds stopped
  extending the attempt count but could still use attempts that
  max_http_attempts allowed, so a client with 36 attempts and a 2 s
  budget slept 35 s. A positive budget now ends the request once the
  next wait would pass it. A budget of 0 keeps 429 and 503 on the
  max_http_attempts count, also for a zero-length jittered wait.
- The full-jitter window computed 2 ** (tries - 1) before the 60 s cap,
  which overflows a float after 1024 tries on the unlimited internal
  retry path. The exponent is capped at 6 (64 s, past the cap).

The GDPval judge proxies set max_rate_limit_wait_seconds: 0, so they
keep their documented 36-attempt window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
Review feedback: a 10-minute default for max_rate_limit_wait_seconds
could interrupt ongoing work while a shared inference endpoint is badly
backed up. The default is now 3600 s, so 429 and 503 answers are waited
out for up to an hour before the request fails. Callers that want a
shorter window set it themselves; vllm_model clients keep 0 and the
GDPval judge proxies keep 0 (count-only, 36 attempts).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
@AdamRajfer

Copy link
Copy Markdown
Contributor Author

/ok to test 2ce62fd

@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/39474988729

Merged via the queue into NVIDIA-NeMo:main with commit 7128f82 Oct 20, 2026
48 of 53 checks passed
@AdamRajfer
AdamRajfer deleted the fix/openai-backoff-with-jitter branch October 20, 2026 18:45

This branch was previously deployed

1 inactive deployment
public — e6ae1f9a Deployed Oct 20, 2026 by copy-pr-bot[bot] via release / finalize / notify #6040
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:core Shared APIs, servers, telemetry, health, and registries bug Something isn't working complexity:high Cross-component or shared-contract change with a large review surface needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants