Skip to content

fix(health): do not flag a context overflow after an answer, and flag a reasoning-only runaway - #4921

Merged
AdamRajfer merged 19 commits into
NVIDIA-NeMo:mainfrom
AdamRajfer:eval/health-context-overflow
Oct 26, 2026
Merged

AdamRajfer merged 19 commits into
NVIDIA-NeMo:mainfrom
AdamRajfer:eval/health-context-overflow

Conversation

@AdamRajfer

@AdamRajfer AdamRajfer commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

A model that answers and then exhausts its context should not be reported as an infrastructure failure. A request that never fit, or a generation that spends its output budget on reasoning without usable output, should remain visible in health checks.

flowchart TD
    A[Captured model calls] --> B{Recorded context overflow?}
    B -->|Yes| C{Earlier answer in same conversation?}
    C -->|Yes| D[Keep score; omit infrastructure finding]
    C -->|No| E[Report failed model call]
    B -->|No| F[Apply other failure checks]
    A --> G{Output limit reached without usable output?}
    G -->|Yes| H[Report runaway generation]
    G -->|No| I[No runaway finding]
Loading

Conversation matching uses captured model, request and session identity. Health checks use the recorded overflow category, not arbitrary error text. An output budget larger than the engine's reported window remains an invalid-parameter failure even after an answer.

The hollow-turn check exempts only a turn whose own overflow is accounted for by the other checks. Empty retries still flag. Answers, refusals and tool calls count as usable output; reasoning alone does not. Incomplete captures remain unobserved where evidence is required.

Regression tests cover Responses, Chat Completions and Messages, vLLM budget precedence and both overflow propagation modes. The full unit comparison found no new failures. Scores and retry behavior are unchanged.

Fixes #4984.
Fixes #4980.

rollout_ended_on_failed_model_call flagged every rollout whose last
model call failed, including one the upstream rejected because the
context was full. That zero is the model's own result and is scored
like any other answer, so counting the rollout as unhealthy inflated
the unhealthy share of agentic runs that fill their context.

The check now leaves out a last call that the upstream rejected with a
400 recorded as context_length_exceeded, the same rule the token checks
already use. It covers both shapes the capture holds: a propagated 400,
and a local empty "length" answer when overflows are not propagated.
The overflow is read from the recorded category, never inferred from a
400 or from error text, and every other failed last call still flags.
A chain whose calls all failed is now ordered by time like any other,
so it flags only when its last call is not an overflow; without
timestamps it flags unless every call was an overflow.

run.stats.model_call_errors.ended_on_error follows the check, because
it counts the same rollouts. The overflow call is still counted as a
failed model call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
@AdamRajfer AdamRajfer self-assigned this Oct 22, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added area:core Shared APIs, servers, telemetry, health, and registries bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer labels Oct 22, 2026
@nemo-automation-bot
nemo-automation-bot Bot requested review from a team October 22, 2026 14:11
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

AdamRajfer and others added 2 commits October 23, 2026 01:17
…ot hollow

When overflows are not propagated, the model server answers a turn that
overflowed the context with an empty local reply. The turn has no message
and no tool call, so agent_turn_hollow still flagged the rollout after
rollout_ended_on_failed_model_call stopped flagging it. The check now
skips a turn whose own model call the upstream rejected for a full
context, by the same rule. Without captured calls, every empty turn is
still flagged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
agent_turn_hollow collected the overflow calls by their call id, but a
turn can point at its call by the model and response id instead, as
Terminal-Bench 4's turns do. Such a turn never matched its overflow
call, so its empty turn was still flagged. The check now matches a call
by every form a turn reference can take, and the test covers both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
@AdamRajfer

Copy link
Copy Markdown
Contributor Author

/review

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HOLD — The overflow exemptions can hide non-overflow failures and genuinely hollow turns.

Two RISK findings and one NOTE below. Reviewed all three changed files and the directly affected capture, binding, and reporting paths. Impact is limited to health findings, counts, and verdicts; no score changes were identified.

Static review only: tests were inspected, not executed. Acceptance checks below are proposed regressions.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

Comment thread nemo_gym/health/checks.py Resolved
Comment thread nemo_gym/health/checks.py Resolved
Comment thread nemo_gym/health/checks.py Resolved
An empty turn was excused when any call matched by any turn's reference
was a context overflow. Two cases slipped through:
- an overflow followed by a successful empty retry in the same turn: the
  retry is how the turn ended, so the turn is hollow;
- a turn whose own reference does not resolve, such as a stale response
  id next to a call id: another turn's resolution of the same call says
  nothing about this turn.

The check now resolves each turn's own references, with the resolutions
the binding step already computes, and lets the last of the turn's calls
decide. When the calls cannot be ordered, every one must be an overflow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
Two overflow calls share one model and response id. The turn that names
the call by its id is excused; the turn that names the shared response
id matches both calls and stays hollow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
@AdamRajfer

Copy link
Copy Markdown
Contributor Author

/review reset=true

@AdamRajfer

Copy link
Copy Markdown
Contributor Author

/ok to test e549a58

@AdamRajfer

Copy link
Copy Markdown
Contributor Author

/review reset=true

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP — No material reliability concerns found in the reviewed changes.

The failure reported in prior comment 4366315406 is fixed: the runaway observation gate now uses the same length-refusal predicate as the check. The regression cases cover both refusal categories, alone and beside a reasoning-only clipped call. Ordinary generated calls without saved responses remain unobserved.

Completed a fresh broad review of all seven changed files and traced the directly affected adapter, capture, session, binding, response-shape, and health-accounting paths. The reference-specific, retry-order, unordered-chain, and incomplete-capture safeguards remain intact. Static review only: tests were inspected, not executed.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

oyilmaz-nvidia
oyilmaz-nvidia previously approved these changes Oct 26, 2026
@oyilmaz-nvidia

Copy link
Copy Markdown
Contributor

/ok to merge 74fee0c

@oyilmaz-nvidia
oyilmaz-nvidia added this pull request to the merge queue Oct 26, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Oct 26, 2026
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40377194891

This branch was previously deployed

1 inactive deployment
public — 14ae4dde Deployed Oct 26, 2026 by copy-pr-bot[bot] via release / finalize / notify #7362
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:core Shared APIs, servers, telemetry, health, and registries bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer sla:review-overdue Review response is over the one-business-day SLA

Projects

None yet

3 participants