Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
721f30a
fix(health): do not flag a rollout that ended on a context overflow
AdamRajfer Oct 22, 2026
2e71c8f
fix(health): an empty turn from the model's own context overflow is n…
AdamRajfer Oct 22, 2026
00f1ee5
fix(health): match an overflow call by the reference its turn uses
AdamRajfer Oct 22, 2026
247f0f7
fix(health): excuse an empty turn only by its own last call's overflow
AdamRajfer Oct 23, 2026
c111485
test(health): an ambiguous response reference excuses no empty turn
AdamRajfer Oct 23, 2026
71c72c1
Merge main into the PR branch
AdamRajfer Oct 23, 2026
a281b6c
fix(vllm_model): label an invalid output budget apart from a context …
AdamRajfer Oct 23, 2026
6fea91d
fix(health): excuse an overflow only once the model has answered in t…
AdamRajfer Oct 23, 2026
914ab32
fix(health): excuse an overflow only after an answer in the same conv…
AdamRajfer Oct 23, 2026
2df9023
docs(health): say what a missing reference resolves to in the call bi…
AdamRajfer Oct 23, 2026
e0fc1ca
fix(health): two captured sessions that open alike are not one conver…
AdamRajfer Oct 23, 2026
49a906a
fix(health): flag a length-limited call that gave the agent nothing t…
AdamRajfer Oct 24, 2026
3d4bf6c
fix(health): keep an empty overflow turn hollow when no other check a…
AdamRajfer Oct 24, 2026
b9cc04f
fix(vllm_model): name an output budget larger than the whole window a…
AdamRajfer Oct 24, 2026
a0af975
fix(vllm_model): read the context window from vLLM 0.27's refusal wor…
AdamRajfer Oct 24, 2026
3dd9063
fix(health): skip the token checks for any upstream length refusal
AdamRajfer Oct 24, 2026
e549a58
fix(health): exempt any length refusal from the runaway observation gate
AdamRajfer Oct 24, 2026
74fee0c
Merge branch 'main' into eval/health-context-overflow
oyilmaz-nvidia Oct 26, 2026
14ae4dd
Merge main into eval/health-context-overflow
AdamRajfer Oct 26, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions fern/versions/latest/pages/evaluation/rollout-health.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -98,9 +98,9 @@ Check IDs identify both the evidence being evaluated and the condition detected.
| `record_unreadable` | Rollout | A non-empty JSONL line is not a JSON object, or its `ng_trajectory` cannot be parsed as a `TrajectoryRecord`. |
| `rollout_duplicate_identity` | Rollout | More than one input record has the same `_ng_task_index` and `_ng_rollout_index`. Every duplicated record receives the finding. |
| `rollout_missing_agent_turns` | Rollout | Agent turns are observable, but no turn contains any of the following: (1) non-empty answer or reasoning content, (2) a tool call, or (3) a model-call reference. |
| `rollout_ended_on_failed_model_call` | Rollout | A model's own last call in the rollout has an HTTP error status, an error category, or response status `failed`, `error`, or `cancelled`. Read from the captured calls directly rather than through a reference, because a failed call carries no response ID to be referenced by. Judged per model and in `started_at` order, so a call to a different model landing later does not mask the failure, and a failure the client retried successfully produces no finding. One finding per model. When a chain mixes failures and successes and carries no timestamps it cannot be ordered, and no finding is produced. |
| `rollout_ended_on_failed_model_call` | Rollout | A model's own last call in the rollout has an HTTP error status, an error category, or response status `failed`, `error`, or `cancelled`. A last call the upstream rejected because the context was full (an upstream `400` recorded with `error_category` or `local_response_reason` `context_length_exceeded`) produces no finding: the rollout ended on the model's own context overflow, which is scored like any other answer. Read from the captured calls directly rather than through a reference, because a failed call carries no response ID to be referenced by. Judged per model and in `started_at` order, so a call to a different model landing later does not mask the failure, and a failure the client retried successfully produces no finding. One finding per model. When a chain mixes failures and successes and carries no timestamps it cannot be ordered, and no finding is produced. |
| `rollout_token_count_mismatch` | Rollout | Both top-level rollout token totals and a complete set of invocation- or turn-owned policy-model-call token counts are available, but the prompt-token total or completion-token total differs from the corresponding call-level sum. |
| `agent_turn_hollow` | Agent turn | An observable `TrajectoryTurn` does not contain any of the following: (1) non-empty answer content, (2) non-empty reasoning content, or (3) a tool call. |
| `agent_turn_hollow` | Agent turn | An observable `TrajectoryTurn` does not contain any of the following: (1) non-empty answer content, (2) non-empty reasoning content, or (3) a tool call. A turn whose own model call the upstream rejected because the context was full is not hollow, by the same rule as `rollout_ended_on_failed_model_call`. |
| `trajectory_capture_mismatch` | Trajectory and model calls | An explicit turn- or invocation-to-model-call reference resolves to zero or multiple model calls, or `ng_trajectory.gaps` records an unmatched, ambiguous, or conflicting reference. |
| `model_call_missing_token_counts` | Policy-model call | On an exactly bound policy-model call, `token_stats.prompt_tokens` or `token_stats.completion_tokens` is omitted or set to `null`. A value of zero counts as present. A failed call that produced no generation, such as a cancelled call or an HTTP error reply, is skipped: it has no tokens to account for. |
| `model_call_runaway_generation` | Policy-model call | An exactly bound policy-model call ended with `finish_reason: length` and its saved response contains no answer or reasoning content. |
Expand Down Expand Up @@ -276,6 +276,8 @@ IDs whose count is zero. `run.artifacts.coverage` reports how often each check w
`run.artifacts.captures` counts rollout trajectories that contain model-call evidence; health does not count or reopen raw
capture files. Raw statistics under `run.stats` describe all model calls stored in `ng_trajectory`, including unreferenced
auxiliary calls. Checks with a policy-model subject evaluate only exactly bound calls.
`model_call_errors.ended_on_error` counts rollouts that `rollout_ended_on_failed_model_call` would flag, so a
rollout that ended on a context-length overflow is not in it.
`model_call_errors.usage_unknown` counts failed calls that returned no finished response and carry no token counts. The
token checks skip these calls, so any tokens the server spent on them before a cancellation or timeout are counted only here,
as a number of calls.
Expand Down
50 changes: 39 additions & 11 deletions nemo_gym/health/checks.py
Original file line number Diff line number Diff line change
Expand Up @@ -340,6 +340,17 @@ def _is_failed_attempt(call: dict[str, Any]) -> bool:
)


def _call_reference_keys(call: dict[str, Any]) -> set[str]:
"""Every ``_call_ref_key`` that a turn can use to point at this call: its call id, or its model and response id."""
keys = set()
if call.get("model_call_id"):
keys.add(f"call:{call['model_call_id']}")
model_ref = call.get("model_ref")
if isinstance(model_ref, dict) and call.get("response_id"):
keys.add(f"response:{model_ref.get('type')}:{model_ref.get('name')}:{call['response_id']}")
return keys


def _call_identity(call: dict[str, Any]) -> str | None:
if call.get("model_call_id"):
return f"call:{call['model_call_id']}"
Expand Down Expand Up @@ -514,7 +525,18 @@ def _rollout_missing_agent_turns(trajectory: dict[str, Any], subject: dict[str,
]


def _agent_turn_hollow(trajectory: dict[str, Any], subject: dict[str, int | str]) -> list[Finding]:
def _agent_turn_hollow(
trajectory: dict[str, Any], bindings: _CallBindings, subject: dict[str, int | str]
) -> list[Finding]:
# An empty turn whose own model call the upstream rejected for a full context is the model filling its
# context, as in `rollout_ended_on_failed_model_call`. The calls are optional evidence here: without
# captured calls, every empty turn is still flagged.
overflow_calls = {
key
for call in bindings.matched_calls
if _is_context_overflow_rejection(call)
for key in _call_reference_keys(call)
Comment thread
AdamRajfer marked this conversation as resolved.
Resolved
}
return [
Finding(
check="agent_turn_hollow",
Expand All @@ -523,7 +545,7 @@ def _agent_turn_hollow(trajectory: dict[str, Any], subject: dict[str, int | str]
detail={"reason": "agent turn has no message or tool calls"},
)
for step in _agent_steps(trajectory)
if not step.has_message and not step.has_tool_calls
if not step.has_message and not step.has_tool_calls and not overflow_calls.intersection(step.model_call_refs)
Comment thread
AdamRajfer marked this conversation as resolved.
Resolved
]


Expand Down Expand Up @@ -638,19 +660,21 @@ def _chain_ended_on_failure(chain: Sequence[dict[str, Any]]) -> bool:
the position in the merged trajectory list, so a producer-supplied retry
can precede the attempt it retried. Say nothing rather than call a
recovered rollout unhealthy.

A last call the upstream rejected because the context was full is the
model's own end, scored like any other answer, so it is not a failure here.
An unordered chain of only failures ended on one of them, so it is flagged
unless every one was such an overflow.
"""
if not chain:
return False
failed = [call for call in chain if _is_failed(call)]
if not failed:
return False
if len(failed) == len(chain):
# Nothing to order: whichever went last, it failed.
return True
if any(call.get("started_at") is None for call in chain):
return False
ordered = sorted(chain, key=lambda call: (call.get("started_at") or 0.0, call.get("call_index") or 0))
return _is_failed(ordered[-1])
if all(call.get("started_at") is not None for call in chain):
last = sorted(chain, key=lambda call: (call.get("started_at") or 0.0, call.get("call_index") or 0))[-1]
return _is_failed(last) and not _is_context_overflow_rejection(last)
Comment thread
AdamRajfer marked this conversation as resolved.
Resolved
Comment thread
AdamRajfer marked this conversation as resolved.
Resolved
return len(failed) == len(chain) and not all(_is_context_overflow_rejection(call) for call in chain)


def _ended_on_failed_call(calls: Sequence[dict[str, Any]]) -> bool:
Expand All @@ -670,7 +694,9 @@ def _rollout_ended_on_failed_model_call(trajectory: dict[str, Any], subject: dic
Judged per model and in time order -- see `_chain_ended_on_failure`. Only
the chain's last call counts, so a failure the client retried successfully
stays healthy; the signal is that the rollout ENDED on a failure, which is
what makes its reward indistinguishable from a genuine zero.
what makes its reward indistinguishable from a genuine zero. A last call
the upstream rejected as a context-length overflow is not flagged: that
zero is the model's own result.
"""
findings = []
for key, chain in _model_chains(_normalized_trajectory_calls(trajectory)).items():
Expand Down Expand Up @@ -750,7 +776,9 @@ def _model_call_runaway_generation(bindings: _CallBindings, subject: dict[str, i
"rollout_ended_on_failed_model_call": lambda record, trajectory, bindings, subject: (
_rollout_ended_on_failed_model_call(trajectory, subject)
),
"agent_turn_hollow": lambda record, trajectory, bindings, subject: _agent_turn_hollow(trajectory, subject),
"agent_turn_hollow": lambda record, trajectory, bindings, subject: _agent_turn_hollow(
trajectory, bindings, subject
),
"model_call_zero_completion_tokens": lambda record, trajectory, bindings, subject: (
_model_call_zero_completion_tokens(bindings, subject)
),
Expand Down
141 changes: 141 additions & 0 deletions tests/unit_tests/test_rollout_health.py
Original file line number Diff line number Diff line change
Expand Up @@ -1429,6 +1429,147 @@ def test_a_rollout_ending_on_a_failed_attempt_is_unhealthy_for_that_reason_only(
assert {finding.check for finding in digest.findings} == {"rollout_ended_on_failed_model_call"}


# The two shapes a context-length overflow takes in the capture.
CONTEXT_OVERFLOWS = {
# vLLM answers locally with an empty "length" completion when overflow errors are not propagated.
"answered_locally": {
"status_code": 200,
"upstream_status_code": 400,
"upstream_attempted": True,
"error_category": "context_length_exceeded",
"local_response_reason": "context_length_exceeded",
"response_source": "local",
"response_status": None,
"finish_reason": "length",
},
# Propagated to the client as the upstream's own 400.
"propagated": {
"status_code": 400,
"upstream_status_code": 400,
"upstream_attempted": True,
"error_category": "context_length_exceeded",
"response_source": "upstream",
"response": {"error": {"code": "context_length_exceeded"}},
**_NO_GENERATION,
},
}


_POLICY_REF = {"type": "responses_api_models", "name": "policy_model"}
# A turn points at its call by the call id, or, as Terminal-Bench 4's turns do, by the model and response id.
TURN_CALL_REFERENCES = {
"by_call_id": {"model_call_id": "c1"},
"by_response_id": {"model_ref": _POLICY_REF, "response_id": "r1"},
}


@pytest.mark.parametrize("reference", TURN_CALL_REFERENCES.values(), ids=TURN_CALL_REFERENCES.keys())
@pytest.mark.parametrize("overflow", CONTEXT_OVERFLOWS.values(), ids=CONTEXT_OVERFLOWS.keys())
def test_an_empty_turn_from_a_context_overflow_is_not_hollow(tmp_path, overflow, reference):
"""The empty turn is the model filling its context, so the rollout is not unhealthy."""
record = _record(0, 0, answer=None, refs=[reference], include_response_output=False)
overflow_call = _call(
**({"tokens_in": None, "tokens_out": 0, "response": {}, "model_ref": _POLICY_REF} | overflow)
)
[digest] = run_health_checks(_write_fixture(tmp_path, [(record, [overflow_call])]), workers=1).rollouts

assert "agent_turn_hollow" not in {finding.check for finding in digest.findings}
assert digest.verdict != "unhealthy"


def test_an_empty_turn_from_any_other_call_is_still_hollow(tmp_path):
record = _record(0, 0, answer=None, include_response_output=False)
empty_call = _call(tokens_out=0, response={})
[digest] = run_health_checks(_write_fixture(tmp_path, [(record, [empty_call])]), workers=1).rollouts

assert "agent_turn_hollow" in {finding.check for finding in digest.findings}


@pytest.mark.parametrize("overflow", CONTEXT_OVERFLOWS.values(), ids=CONTEXT_OVERFLOWS.keys())
def test_a_rollout_ending_on_a_context_overflow_is_not_flagged(tmp_path, overflow):
"""The model filled its context: a zero it earned, not an unhealthy rollout."""
succeeded = _call(model_call_id="first", started_at=1.0)
last = _call(model_call_id="last", started_at=2.0, tokens_in=None, tokens_out=0, **overflow)
rollout_path = _write_fixture(tmp_path, [(_owned_attempts(succeeded, last), [succeeded, last])])

result = run_health_checks(rollout_path, workers=1)
[digest] = result.rollouts

assert "rollout_ended_on_failed_model_call" not in {finding.check for finding in digest.findings}
assert result.summary["run"]["stats"]["model_call_errors"]["ended_on_error"] == 0
# The call is still a failed call in the error counts.
assert digest.model_call_errors == 1


@pytest.mark.parametrize("overflow", CONTEXT_OVERFLOWS.values(), ids=CONTEXT_OVERFLOWS.keys())
def test_a_chain_of_only_context_overflows_is_not_flagged(tmp_path, overflow):
calls = [
_call(call_index=index, model_call_id=None, response_id=None, tokens_out=0, **overflow) for index in range(2)
]
rollout_path = _write_fixture(tmp_path, [(_unreferenced_failure_record([]), calls)])

[digest] = run_health_checks(rollout_path, workers=1).rollouts

assert "rollout_ended_on_failed_model_call" not in {finding.check for finding in digest.findings}


@pytest.mark.parametrize("failure", FAILED_ATTEMPTS.values(), ids=FAILED_ATTEMPTS.keys())
def test_a_failure_after_a_context_overflow_still_flags(tmp_path, failure):
"""Only the last call decides: an overflow earlier in the chain hides nothing that came after it."""
overflow = _call(model_call_id="first", started_at=1.0, tokens_out=0, **CONTEXT_OVERFLOWS["propagated"])
failed = _call(model_call_id="last", started_at=2.0, tokens_in=None, tokens_out=None, **failure)
rollout_path = _write_fixture(tmp_path, [(_owned_attempts(overflow, failed), [overflow, failed])])

[digest] = run_health_checks(rollout_path, workers=1).rollouts

assert "rollout_ended_on_failed_model_call" in {finding.check for finding in digest.findings}


@pytest.mark.parametrize("failure", FAILED_ATTEMPTS.values(), ids=FAILED_ATTEMPTS.keys())
def test_a_context_overflow_after_failed_calls_is_not_flagged(tmp_path, failure):
"""Every call failed, but in time order the rollout ended on the model's own overflow."""
failed = _call(model_call_id="first", started_at=1.0, tokens_in=None, tokens_out=None, **failure)
overflow = _call(model_call_id="last", started_at=2.0, tokens_out=0, **CONTEXT_OVERFLOWS["propagated"])
rollout_path = _write_fixture(tmp_path, [(_owned_attempts(failed, overflow), [failed, overflow])])

[digest] = run_health_checks(rollout_path, workers=1).rollouts

assert "rollout_ended_on_failed_model_call" not in {finding.check for finding in digest.findings}


def test_an_unordered_chain_of_an_overflow_and_a_transport_failure_still_flags(tmp_path: Path) -> None:
calls = [
_call(call_index=0, model_call_id=None, response_id=None, tokens_out=0, **CONTEXT_OVERFLOWS["propagated"]),
_call(call_index=1, model_call_id=None, response_id=None, **FAILED_ATTEMPTS["cancelled"]),
]
rollout_path = _write_fixture(tmp_path, [(_unreferenced_failure_record([]), calls)])

[digest] = run_health_checks(rollout_path, workers=1).rollouts

assert "rollout_ended_on_failed_model_call" in {finding.check for finding in digest.findings}


def test_a_400_without_the_overflow_category_still_flags(tmp_path: Path) -> None:
"""The overflow is read from the recorded category, never inferred from a 400 alone."""
succeeded = _call(model_call_id="first", started_at=1.0)
rejected = _call(
model_call_id="last",
started_at=2.0,
status_code=400,
upstream_status_code=400,
upstream_attempted=True,
error_category="client_error",
tokens_in=None,
tokens_out=None,
**_NO_GENERATION,
)
rollout_path = _write_fixture(tmp_path, [(_owned_attempts(succeeded, rejected), [succeeded, rejected])])

[digest] = run_health_checks(rollout_path, workers=1).rollouts

assert "rollout_ended_on_failed_model_call" in {finding.check for finding in digest.findings}


def test_a_body_capture_could_not_parse_still_needs_token_counts(tmp_path: Path) -> None:
"""`capture_parse_error` marks a 2xx call whose body was unreadable: a capture defect, not a failed call."""
unparsed = _call(
Expand Down
Loading