Skip to content

fix(openai-utils): transient gateway 401 with type authentication_error permanently trips the client #3945

Open

Description

@ushnish-de

Summary

Since #3571, NeMoGymAsyncOpenAI._request_with_retry permanently trips the client on any 401/403 whose body has error.code / error.type set to authentication_error. Some inference gateways return this shape for a transient backend failure that has nothing to do with the API key:

{
  "error": {
    "message": "Authentication Error, Error querying the database: FATAL: remaining connection slots are reserved for roles with the SUPERUSER attribute",
    "type": "authentication_error",
    "code": "401"
  }
}

_parsed_error_codes returns ['401', 'authentication_error'], so _error_body_is_permanent_auth returns True. After that, every later call on that client raises PermanentEndpointError without reaching the endpoint, even though the gateway recovers within seconds.

Impact

In a long gym eval run, a short gateway outage silently disables the judge or the policy model client for the rest of the job:

  • Later rollouts end with no answer or a score of 0.
  • The run does not stop or fail loudly, so the results look valid but are not.
  • Example: in a 135-rollout finance-agent eval, 65 of 95 completed rollouts ended early after the gateway returned this 401. A clean rerun of the same config, with no gateway errors, scored normally.

Repro

from nemo_gym.openai_utils import _error_body_is_permanent_auth

body = (
    b'{"error":{"message":"Authentication Error, Error querying the database: '
    b'FATAL: remaining connection slots are reserved for roles with the SUPERUSER attribute",'
    b'"type":"authentication_error","code":"401"}}'
)
assert _error_body_is_permanent_auth(body)  # passes today; should be False

Expected behavior

Only genuine invalid-key responses should trip the client permanently. Transient backend failures returned as 401 should be retried with the normal backoff, the same way 429/5xx responses are.

Suggested fix

  • Don't treat the authentication_error type alone as permanent, since it is too broad. Alternatively, exclude bodies containing backend-failure text such as error querying the database, connection slots or too many clients.
  • Retry those 401s with the normal backoff instead of returning them once.
  • Add the body above to the classifier test parameters as a non-permanent case.

Related

Activity

  1. added 2 commits that reference this issue on Oct 4, 2026
    ae101f2
    a7654a9
  2. zyzhou5 commented on Oct 14, 2026

    @zyzhou5
    Contributor

    A fix is in progress in #3998.

  3. added
    trackingTracks a coordinated body of work across multiple issues or pull requests.
    on Oct 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    RollouttrackingTracks a coordinated body of work across multiple issues or pull requests.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions