Skip to content

fix(core): retry transient gateway authentication failures - #3998

Open
sylvesterkaczmarek wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
sylvesterkaczmarek:fix/transient-gateway-auth-retries
Open

sylvesterkaczmarek wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
sylvesterkaczmarek:fix/transient-gateway-auth-retries

Conversation

@sylvesterkaczmarek

@sylvesterkaczmarek sylvesterkaczmarek commented Oct 4, 2026 •

Copy link
Copy Markdown

Summary

Fixes #3945.

Retry transient inference-gateway authentication database failures that are returned as HTTP 401/403, while preserving fail-fast behaviour for genuinely invalid API keys and tokens. These outages can otherwise leave a model client permanently disabled after one failed request.

Change

  • Ported the fix onto current upstream main following the client restructuring. The implementation now lives in nemo_gym/client/openai_client.py rather than the deleted legacy nemo_gym/openai_utils.py.
  • Recognise documented database-failure messages, even with an authentication_error type. An explicit invalid-key code or message takes priority.
  • Reuse the already-read response body, the existing bounded retry/backoff path, and the same request payload. Preserve the final status and diagnostic body on exhaustion, and allow subsequent calls on the same client.
  • Retain the current attempt-level failure observations and existing behaviour for unrelated 401/403 responses.
  • Update the model-server configuration docs and regressions in tests/unit_tests/client/test_openai_client.py.

Validation

  • Mac arm64, CPython 3.13, project-required uv>=0.13.4 (using uv 0.13.9).
  • Client test module: 126 passed, zero failed, including 401/403 recovery, exact attempt limits, permanent-key precedence, single body read, and same-client reuse.
  • ruff check, ruff format --check on both modified Python files, compileall and change-scoped git diff --check: passed.
  • Signed and DCO-signed merge commit cb853bf67, with current upstream main integrated. GitHub reports the PR mergeable.
  • NVIDIA-hosted workflows still require the repository's normal vetting/approval process.

No API, configuration, model-generation or benchmark changes.

@copy-pr-bot

copy-pr-bot Bot commented Oct 4, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@nemo-automation-bot nemo-automation-bot Bot added the community-request Issue reported or requested by someone from the community label Oct 4, 2026
@sylvesterkaczmarek
sylvesterkaczmarek force-pushed the fix/transient-gateway-auth-retries branch from 58c34ec to ae101f2 Compare October 4, 2026 17:47
@yaoyu-33 yaoyu-33 added area:core Shared APIs, servers, telemetry, health, and registries bug Something isn't working complexity:low Localized change in one scope with a small, straightforward review surface needs-review PR is ready for code review and waiting on a reviewer labels Oct 4, 2026
@github-actions github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Oct 5, 2026
Recognize gateway database failures returned as HTTP 401 or 403 and use
the existing bounded retry path without permanently disabling the client.
Preserve specific invalid-key fail-fast behavior and terminal error bodies.

Add real HTTP recovery and exhaustion tests, compatibility controls, and
clarify the max_http_attempts documentation.

Fixes NVIDIA-NeMo#3945.

Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
@sylvesterkaczmarek
sylvesterkaczmarek force-pushed the fix/transient-gateway-auth-retries branch from ae101f2 to a7654a9 Compare October 6, 2026 11:35
@sylvesterkaczmarek

Copy link
Copy Markdown
Author

Refreshed onto current main and resolved the newer docs/test overlap while preserving the transient gateway-auth retry coverage. Ruff passes and tests/unit_tests/test_openai_utils.py passes 285/285. The PR is mergeable; workflows are waiting for NVIDIA vetting/approval.

@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Oct 8, 2026
@yaoyu-33 yaoyu-33 removed the needs-review PR is ready for code review and waiting on a reviewer label Oct 8, 2026
@yaoyu-33
yaoyu-33 marked this pull request as draft November 18, 2026 01:57
@yaoyu-33

Copy link
Copy Markdown
Contributor

We’re doing some cleanup of the open PR list. This PR has had no activity for over a month, so I’ve moved it back to draft.

If this change is still needed, please resolve any merge conflicts and mark the PR ready for review again when it’s ready to merge. Thanks!

@github-actions github-actions Bot removed the sla:triage-overdue Review assignment is over the one-business-day SLA label Nov 18, 2026
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
@sylvesterkaczmarek
sylvesterkaczmarek marked this pull request as ready for review November 18, 2026 02:35
@sylvesterkaczmarek

Copy link
Copy Markdown
Author

Thanks for the note and for keeping the PR list current. I merged the newer upstream client restructuring, carried this fix into the active HTTP client, and reran its test module (126 passing). The PR is now marked ready for review again, with the updated scope and validation in the description. I appreciate the maintenance work; happy to address any follow-up. NVIDIA workflow vetting may still be needed.

@svcnvidia-nemo-ci svcnvidia-nemo-ci added needs-review PR is ready for code review and waiting on a reviewer and removed waiting-on-maintainers Waiting on maintainers to respond labels Nov 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:core Shared APIs, servers, telemetry, health, and registries bug Something isn't working community-request Issue reported or requested by someone from the community complexity:low Localized change in one scope with a small, straightforward review surface needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(openai-utils): transient gateway 401 with type authentication_error permanently trips the client

3 participants