Skip to content

[VDR][v0.5.0] Runtime reliability #2215

Open
1 of 3 issues completed
Open

Description

@anwithk

Real defects in shipped code — hangs and swallowed errors that affect every user at scale. These are not documentation gaps.

Findings: 4 (1 critical, 2 high, 1 medium)
Code paths: nemo_gym/server_utils.py, nemo_gym/rollout_collection.py
Source: DX review Gym-1-20260730170035 · target ref e446e4f4 · 2026-07-30
Tracker: #2214

Sub-issues (critical + high)

Filed individually as sub-issues of this one:

  • 🔴 ERR-8508619d — Connection-level retries are unbounded and silent for the first ~50 seconds
  • 🟠 QS-22b494eb — gym eval run --output does not create the parent directory, breaking the PyPI install path
  • 🟠 QS-efab2b5c — Model-provider error is fully swallowed; user sees only an opaque 500 plus a raw traceback

Additional findings in this workstream

Tracked here rather than as separate sub-issues (below the critical/high bar for individual issues, but in scope for this workstream).

  • 🟡 ERR-10403e82 — Bare except: in the ASGI middleware swallows CancelledError and KeyboardInterrupt into a generic 500
Detail for the findings above

🟡 ERR-10403e82 — Bare except: in the ASGI middleware swallows CancelledError and KeyboardInterrupt into a generic 500

Severity: medium
Path: nemo_gym/server_utils.py:601
Found by: error-quality-auditor
Status: new in this review

exception_handling_middleware has a bare except: following except Exception as e:. Everything reachable there is a non-Exception BaseException — principally asyncio.CancelledError, plus KeyboardInterrupt and SystemExit. Converting a cancellation into a JSON 500 defeats client-disconnect propagation and task cancellation, and the client receives the fully context-free string 'An unknown error occurred' with the traceback only printed to server stdout.

Suggested fix: Re-raise asyncio.CancelledError and KeyboardInterrupt/SystemExit rather than converting them to responses. If a catch-all is still wanted, narrow it to except BaseException with an explicit re-raise guard, and include a correlation id in the 500 body so the response can be tied back to the logged traceback.

Evidence
server_utils.py:594 `except Exception as e:` ... :601 `except:` -> JSONResponse(content="An unknown error occurred", status_code=500)

Filed from an automated first-time-developer DX review. Severity reflects impact on a new user: 🔴 critical = blocks usage entirely · 🟠 high = significant friction, workaround required · 🟡 medium = causes confusion · 🔵 low = cosmetic.

Activity

  1. added
    vdrFinding from a DX / first-time-developer review (VDR)
    on Jul 30, 2026
  2. moved this from Label Sync to Dev WIP in NeMo Gym 0.5.0 - Aug 5on Aug 5, 2026
  3. ananthsub commented on Sep 9, 2026

    @ananthsub
    Contributor

    The client-disconnect half landed in #3064 and issue #23 is now closed. The remaining ERR-10403e82 work—re-raising CancelledError/KeyboardInterrupt/SystemExit through exception middleware and composing cancellation with bounded cleanup—is owned by #3160. The unbounded transport finding remains #2216 and maps to #3159.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    RolloutvdrFinding from a DX / first-time-developer review (VDR)

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions