Repository navigation
fix(health): do not flag a context overflow after an answer, and flag a reasoning-only runaway - #4921
Conversation
rollout_ended_on_failed_model_call flagged every rollout whose last model call failed, including one the upstream rejected because the context was full. That zero is the model's own result and is scored like any other answer, so counting the rollout as unhealthy inflated the unhealthy share of agentic runs that fill their context. The check now leaves out a last call that the upstream rejected with a 400 recorded as context_length_exceeded, the same rule the token checks already use. It covers both shapes the capture holds: a propagated 400, and a local empty "length" answer when overflows are not propagated. The overflow is read from the recorded category, never inferred from a 400 or from error text, and every other failed last call still flags. A chain whose calls all failed is now ordered by time like any other, so it flags only when its last call is not an overflow; without timestamps it flags unless every call was an overflow. run.stats.model_call_errors.ended_on_error follows the check, because it counts the same rollouts. The overflow call is still counted as a failed model call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
|
🌿 Preview your docs: https://nvidia-preview-pr-4921-14ae4ddee6bd.docs.buildwithfern.com/nemo/gym |
…ot hollow When overflows are not propagated, the model server answers a turn that overflowed the context with an empty local reply. The turn has no message and no tool call, so agent_turn_hollow still flagged the rollout after rollout_ended_on_failed_model_call stopped flagging it. The check now skips a turn whose own model call the upstream rejected for a full context, by the same rule. Without captured calls, every empty turn is still flagged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
agent_turn_hollow collected the overflow calls by their call id, but a turn can point at its call by the model and response id instead, as Terminal-Bench 4's turns do. Such a turn never matched its overflow call, so its empty turn was still flagged. The check now matches a call by every form a turn reference can take, and the test covers both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
|
/review |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
HOLD — The overflow exemptions can hide non-overflow failures and genuinely hollow turns.
Two RISK findings and one NOTE below. Reviewed all three changed files and the directly affected capture, binding, and reporting paths. Impact is limited to health findings, counts, and verdicts; no score changes were identified.
Static review only: tests were inspected, not executed. Acceptance checks below are proposed regressions.
Note: formal approval is not available for this request, so this review is posted as a comment only.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
An empty turn was excused when any call matched by any turn's reference was a context overflow. Two cases slipped through: - an overflow followed by a successful empty retry in the same turn: the retry is how the turn ended, so the turn is hollow; - a turn whose own reference does not resolve, such as a stale response id next to a call id: another turn's resolution of the same call says nothing about this turn. The check now resolves each turn's own references, with the resolutions the binding step already computes, and lets the last of the turn's calls decide. When the calls cannot be ordered, every one must be an overflow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
Two overflow calls share one model and response id. The turn that names the call by its id is excused; the turn that names the shared response id matches both calls and stays hollow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
|
/review reset=true |
|
/ok to test e549a58 |
|
/review reset=true |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP — No material reliability concerns found in the reviewed changes.
The failure reported in prior comment 4366315406 is fixed: the runaway observation gate now uses the same length-refusal predicate as the check. The regression cases cover both refusal categories, alone and beside a reasoning-only clipped call. Ordinary generated calls without saved responses remain unobserved.
Completed a fresh broad review of all seven changed files and traced the directly affected adapter, capture, session, binding, response-shape, and health-accounting paths. The reference-specific, retry-order, unordered-chain, and incomplete-capture safeguards remain intact. Static review only: tests were inspected, not executed.
Note: formal approval is not available for this request, so this review is posted as a comment only.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
|
/ok to merge 74fee0c |
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40377194891 |
A model that answers and then exhausts its context should not be reported as an infrastructure failure. A request that never fit, or a generation that spends its output budget on reasoning without usable output, should remain visible in health checks.
flowchart TD A[Captured model calls] --> B{Recorded context overflow?} B -->|Yes| C{Earlier answer in same conversation?} C -->|Yes| D[Keep score; omit infrastructure finding] C -->|No| E[Report failed model call] B -->|No| F[Apply other failure checks] A --> G{Output limit reached without usable output?} G -->|Yes| H[Report runaway generation] G -->|No| I[No runaway finding]Conversation matching uses captured model, request and session identity. Health checks use the recorded overflow category, not arbitrary error text. An output budget larger than the engine's reported window remains an invalid-parameter failure even after an answer.
The hollow-turn check exempts only a turn whose own overflow is accounted for by the other checks. Empty retries still flag. Answers, refusals and tool calls count as usable output; reasoning alone does not. Incomplete captures remain unobserved where evidence is required.
Regression tests cover Responses, Chat Completions and Messages, vLLM budget precedence and both overflow propagation modes. The full unit comparison found no new failures. Scores and retry behavior are unchanged.
Fixes #4984.
Fixes #4980.