Repository navigation
Description
Summary
Since #3571, NeMoGymAsyncOpenAI._request_with_retry permanently trips the client on any 401/403 whose body has error.code / error.type set to authentication_error. Some inference gateways return this shape for a transient backend failure that has nothing to do with the API key:
{
"error": {
"message": "Authentication Error, Error querying the database: FATAL: remaining connection slots are reserved for roles with the SUPERUSER attribute",
"type": "authentication_error",
"code": "401"
}
}_parsed_error_codes returns ['401', 'authentication_error'], so _error_body_is_permanent_auth returns True. After that, every later call on that client raises PermanentEndpointError without reaching the endpoint, even though the gateway recovers within seconds.
Impact
In a long gym eval run, a short gateway outage silently disables the judge or the policy model client for the rest of the job:
- Later rollouts end with no answer or a score of 0.
- The run does not stop or fail loudly, so the results look valid but are not.
- Example: in a 135-rollout finance-agent eval, 65 of 95 completed rollouts ended early after the gateway returned this 401. A clean rerun of the same config, with no gateway errors, scored normally.
Repro
from nemo_gym.openai_utils import _error_body_is_permanent_auth
body = (
b'{"error":{"message":"Authentication Error, Error querying the database: '
b'FATAL: remaining connection slots are reserved for roles with the SUPERUSER attribute",'
b'"type":"authentication_error","code":"401"}}'
)
assert _error_body_is_permanent_auth(body) # passes today; should be FalseExpected behavior
Only genuine invalid-key responses should trip the client permanently. Transient backend failures returned as 401 should be retried with the normal backoff, the same way 429/5xx responses are.
Suggested fix
- Don't treat the
authentication_errortype alone as permanent, since it is too broad. Alternatively, exclude bodies containing backend-failure text such aserror querying the database,connection slotsortoo many clients. - Retry those 401s with the normal backoff instead of returning them once.
- Add the body above to the classifier test parameters as a non-permanent case.
Related
- fix(openai-utils): fail fast on spent-quota 429 and invalid-key 401/403 #3571 (introduced the permanent auth trip)
- [VDR][v0.6.0] ERR-ec1e818f · An upstream 401 from the model endpoint reaches the CLI as a bare 500 Internal Server Error #2682 / [VDR][v0.6.0] Error legibility at the model boundary #2681 (readability of upstream errors at the model boundary)
Activity
fix(core): retry transient gateway authentication failures
fix(core): retry transient gateway authentication failures
zyzhou5 commentedon Oct 14, 2026
A fix is in progress in #3998.