Repository navigation
fix(core): retry transient gateway authentication failures - #3998
sylvesterkaczmarek wants to merge 2 commits into
Conversation
58c34ec to
ae101f2
Compare
Recognize gateway database failures returned as HTTP 401 or 403 and use the existing bounded retry path without permanently disabling the client. Preserve specific invalid-key fail-fast behavior and terminal error bodies. Add real HTTP recovery and exhaustion tests, compatibility controls, and clarify the max_http_attempts documentation. Fixes NVIDIA-NeMo#3945. Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
ae101f2 to
a7654a9
Compare
|
Refreshed onto current main and resolved the newer docs/test overlap while preserving the transient gateway-auth retry coverage. Ruff passes and tests/unit_tests/test_openai_utils.py passes 285/285. The PR is mergeable; workflows are waiting for NVIDIA vetting/approval. |
|
We’re doing some cleanup of the open PR list. This PR has had no activity for over a month, so I’ve moved it back to draft. If this change is still needed, please resolve any merge conflicts and mark the PR ready for review again when it’s ready to merge. Thanks! |
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
|
Thanks for the note and for keeping the PR list current. I merged the newer upstream client restructuring, carried this fix into the active HTTP client, and reran its test module (126 passing). The PR is now marked ready for review again, with the updated scope and validation in the description. I appreciate the maintenance work; happy to address any follow-up. NVIDIA workflow vetting may still be needed. |
Summary
Fixes #3945.
Retry transient inference-gateway authentication database failures that are returned as HTTP 401/403, while preserving fail-fast behaviour for genuinely invalid API keys and tokens. These outages can otherwise leave a model client permanently disabled after one failed request.
Change
mainfollowing the client restructuring. The implementation now lives innemo_gym/client/openai_client.pyrather than the deleted legacynemo_gym/openai_utils.py.authentication_errortype. An explicit invalid-key code or message takes priority.tests/unit_tests/client/test_openai_client.py.Validation
uv>=0.13.4(using uv 0.13.9).ruff check,ruff format --checkon both modified Python files,compilealland change-scopedgit diff --check: passed.cb853bf67, with current upstreammainintegrated. GitHub reports the PR mergeable.No API, configuration, model-generation or benchmark changes.