Repository navigation
Description
Real defects in shipped code — hangs and swallowed errors that affect every user at scale. These are not documentation gaps.
Findings: 4 (1 critical, 2 high, 1 medium)
Code paths: nemo_gym/server_utils.py, nemo_gym/rollout_collection.py
Source: DX review Gym-1-20260730170035 · target ref e446e4f4 · 2026-07-30
Tracker: #2214
Sub-issues (critical + high)
Filed individually as sub-issues of this one:
- 🔴
ERR-8508619d— Connection-level retries are unbounded and silent for the first ~50 seconds - 🟠
QS-22b494eb—gym eval run --outputdoes not create the parent directory, breaking the PyPI install path - 🟠
QS-efab2b5c— Model-provider error is fully swallowed; user sees only an opaque 500 plus a raw traceback
Additional findings in this workstream
Tracked here rather than as separate sub-issues (below the critical/high bar for individual issues, but in scope for this workstream).
- 🟡
ERR-10403e82— Bareexcept:in the ASGI middleware swallows CancelledError and KeyboardInterrupt into a generic 500
Detail for the findings above
🟡 ERR-10403e82 — Bare except: in the ASGI middleware swallows CancelledError and KeyboardInterrupt into a generic 500
Severity: medium
Path: nemo_gym/server_utils.py:601
Found by: error-quality-auditor
Status: new in this review
exception_handling_middleware has a bare except: following except Exception as e:. Everything reachable there is a non-Exception BaseException — principally asyncio.CancelledError, plus KeyboardInterrupt and SystemExit. Converting a cancellation into a JSON 500 defeats client-disconnect propagation and task cancellation, and the client receives the fully context-free string 'An unknown error occurred' with the traceback only printed to server stdout.
Suggested fix: Re-raise asyncio.CancelledError and KeyboardInterrupt/SystemExit rather than converting them to responses. If a catch-all is still wanted, narrow it to except BaseException with an explicit re-raise guard, and include a correlation id in the 500 body so the response can be tied back to the logged traceback.
Evidence
server_utils.py:594 `except Exception as e:` ... :601 `except:` -> JSONResponse(content="An unknown error occurred", status_code=500)
Filed from an automated first-time-developer DX review. Severity reflects impact on a new user: 🔴 critical = blocks usage entirely · 🟠 high = significant friction, workaround required · 🟡 medium = causes confusion · 🔵 low = cosmetic.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsDev WIP
- StatusShow more project fieldsDev Todo
Activity
gym eval run --outputdoes not create the parent directory, breaking the PyPI install path #2217ananthsub commentedon Sep 9, 2026
The client-disconnect half landed in #3064 and issue #23 is now closed. The remaining ERR-10403e82 work—re-raising CancelledError/KeyboardInterrupt/SystemExit through exception middleware and composing cancellation with bounded cleanup—is owned by #3160. The unbounded transport finding remains #2216 and maps to #3159.