diff --git a/fern/versions/latest/pages/evaluation/rollout-health.mdx b/fern/versions/latest/pages/evaluation/rollout-health.mdx index 5792d9dae31..f8cd90e52f2 100644 --- a/fern/versions/latest/pages/evaluation/rollout-health.mdx +++ b/fern/versions/latest/pages/evaluation/rollout-health.mdx @@ -98,12 +98,12 @@ Check IDs identify both the evidence being evaluated and the condition detected. | `record_unreadable` | Rollout | A non-empty JSONL line is not a JSON object, or its `ng_trajectory` cannot be parsed as a `TrajectoryRecord`. | | `rollout_duplicate_identity` | Rollout | More than one input record has the same `_ng_task_index` and `_ng_rollout_index`. Every duplicated record receives the finding. | | `rollout_missing_agent_turns` | Rollout | Agent turns are observable, but no turn contains any of the following: (1) non-empty answer or reasoning content, (2) a tool call, or (3) a model-call reference. | -| `rollout_ended_on_failed_model_call` | Rollout | A model's own last call in the rollout has an HTTP error status, an error category, or response status `failed`, `error`, or `cancelled`. Read from the captured calls directly rather than through a reference, because a failed call carries no response ID to be referenced by. Judged per model and in `started_at` order, so a call to a different model landing later does not mask the failure, and a failure the client retried successfully produces no finding. One finding per model. When a chain mixes failures and successes and carries no timestamps it cannot be ordered, and no finding is produced. | +| `rollout_ended_on_failed_model_call` | Rollout | A model's own last call in the rollout has an HTTP error status, an error category, or response status `failed`, `error`, or `cancelled`. A last call the upstream rejected because the context was full (an upstream `400` recorded with `error_category` or `local_response_reason` `context_length_exceeded`) produces no finding: the rollout ended on the model's own context overflow, which is scored like any other answer. This holds only once the model has answered earlier in the same conversation: a successful call to the same model whose request opens the same way, with the same `instructions`, `system` prompt, or first item of `messages` or `input` (a plain-text `input` as a whole), and that no captured session tells apart: calls of two different sessions are two conversations, while activations of one harness session are one. A success in another conversation, such as a user simulator served by the same model server, does not count, and a call whose request was not captured belongs to no conversation. An overflow before any such answer means the request never fit the deployment (a prompt or output budget too large for it), and is flagged like any other failed call. Read from the captured calls directly rather than through a reference, because a failed call carries no response ID to be referenced by. Judged per model and in `started_at` order, so a call to a different model landing later does not mask the failure, and a failure the client retried successfully produces no finding. One finding per model. When a chain mixes failures and successes and carries no timestamps it cannot be ordered, and no finding is produced; a chain of only failures is flagged either way. | | `rollout_token_count_mismatch` | Rollout | Both top-level rollout token totals and a complete set of invocation- or turn-owned policy-model-call token counts are available, but the prompt-token total or completion-token total differs from the corresponding call-level sum. | -| `agent_turn_hollow` | Agent turn | An observable `TrajectoryTurn` does not contain any of the following: (1) non-empty answer content, (2) non-empty reasoning content, or (3) a tool call. | +| `agent_turn_hollow` | Agent turn | An observable `TrajectoryTurn` does not contain any of the following: (1) non-empty answer content, (2) non-empty reasoning content, or (3) a tool call. A turn whose own model calls end on one the upstream rejected because the context was full (the last in `started_at` order, or every one when they carry no timestamps) is not hollow when another check accounts for that overflow: an answer from the same model in the same conversation excuses it, and `rollout_ended_on_failed_model_call` flags a request that never fit. When neither applies, as for calls without timestamps that mix in another conversation's answer, the empty turn is still flagged. Each of the turn's model-call references must resolve to exactly one captured call, and the exemption is not applied when a `model_calls_unavailable`, `model_call_capture_incomplete`, `model_call_capture_records_unreadable`, or `model_call_capture_unreadable` gap leaves that check unobserved. | | `trajectory_capture_mismatch` | Trajectory and model calls | An explicit turn- or invocation-to-model-call reference resolves to zero or multiple model calls, or `ng_trajectory.gaps` records an unmatched, ambiguous, or conflicting reference. | | `model_call_missing_token_counts` | Policy-model call | On an exactly bound policy-model call, `token_stats.prompt_tokens` or `token_stats.completion_tokens` is omitted or set to `null`. A value of zero counts as present. A failed call that produced no generation, such as a cancelled call or an HTTP error reply, is skipped: it has no tokens to account for. | -| `model_call_runaway_generation` | Policy-model call | An exactly bound policy-model call ended with `finish_reason: length` and its saved response contains no answer or reasoning content. | +| `model_call_runaway_generation` | Policy-model call | An exactly bound policy-model call ended with `finish_reason: length` and its saved response holds no answer and no tool call. Reasoning alone does not count. | | `model_call_zero_completion_tokens` | Policy-model call | An exactly bound policy-model call reports zero completion tokens. A failed call that produced no generation is skipped. | | `task_consistently_unhealthy` | Task | At least two repeats are computable and every computable repeat is unhealthy. Unobserved repeats do not prevent the flag. | | `task_no_successful_model_calls` | Task | Every repeat has at least one turn-to-model-call reference, every turn reference resolves to exactly one captured call, and none of those turn-bound calls completed successfully. | @@ -165,10 +165,10 @@ evaluate; the gap does not suppress evidence of failed calls or hollow turns. ### Length-Limited Responses -`model_call_runaway_generation` needs one provider-independent definition of an empty saved response. The check recognizes -non-empty text, content, output text, answer, encrypted reasoning content, reasoning, or reasoning summaries in OpenAI Responses, -Chat Completions, and Messages-style responses. If any supported content is present, a length-limited response is not classified -as an empty runaway generation. +`model_call_runaway_generation` needs one provider-independent definition of a response the agent can act on. The check +looks for output text, a message with text or a refusal, or a tool call, in OpenAI Responses, Chat Completions and +Messages-style responses. Reasoning alone, encrypted or summarized, is not something to act on: a call that reasoned until its +output limit without an answer or a tool call is a runaway generation, even when a later call recovers. ## How Health Checks Interpret Existing Observation Gaps @@ -276,6 +276,9 @@ IDs whose count is zero. `run.artifacts.coverage` reports how often each check w `run.artifacts.captures` counts rollout trajectories that contain model-call evidence; health does not count or reopen raw capture files. Raw statistics under `run.stats` describe all model calls stored in `ng_trajectory`, including unreferenced auxiliary calls. Checks with a policy-model subject evaluate only exactly bound calls. +`model_call_errors.ended_on_error` counts rollouts that `rollout_ended_on_failed_model_call` would flag. It leaves out a +rollout whose model-call capture gaps leave that check unobserved, and one that ended on a context-length overflow after +the model answered in the same conversation. `model_call_errors.usage_unknown` counts failed calls that returned no finished response and carry no token counts. The token checks skip these calls, so any tokens the server spent on them before a cancellation or timeout are counted only here, as a number of calls. diff --git a/nemo_gym/health/checks.py b/nemo_gym/health/checks.py index baf328a3b69..4a4306a175f 100644 --- a/nemo_gym/health/checks.py +++ b/nemo_gym/health/checks.py @@ -266,6 +266,12 @@ def _agent_steps(trajectory: dict[str, Any]) -> list[_AgentStep]: def _normalized_trajectory_calls(trajectory: dict[str, Any]) -> list[dict[str, Any]]: + # The session each invocation belongs to: activations of one harness session share it. + sessions = { + invocation["invocation_id"]: invocation.get("harness_session_id") or invocation["invocation_id"] + for invocation in trajectory.get("invocations") or [] + if isinstance(invocation, dict) and invocation.get("invocation_id") + } calls: list[dict[str, Any]] = [] for position, raw in enumerate(trajectory.get("model_calls") or []): metadata = raw.get("response_metadata") or {} @@ -290,6 +296,7 @@ def _normalized_trajectory_calls(trajectory: dict[str, Any]) -> list[dict[str, A "tokens_out": tokens.get("completion_tokens"), "request": raw.get("request"), "response": raw.get("response"), + "session": sessions.get(raw.get("client_session_id"), raw.get("client_session_id")), } ) return calls @@ -310,6 +317,21 @@ def _is_successful(call: dict[str, Any]) -> bool: return not _is_failed(call) and (status is None or (isinstance(status, int) and 200 <= status < 400)) +# The categories vllm_model records for an upstream refusal on length: the conversation outgrew the window, or the +# request's own output budget exceeds all of it. +_LENGTH_REFUSAL_CATEGORIES = frozenset({"context_length_exceeded", "invalid_output_token_parameter"}) + + +def _is_length_refusal(call: dict[str, Any]) -> bool: + """An upstream refusal on length, of either kind: the engine generated nothing, so the token checks have nothing + to read. Only an overflow (`_is_context_overflow_rejection`) can be the model's own end.""" + return ( + call.get("upstream_attempted") is True + and call.get("upstream_status_code") == 400 + and not _LENGTH_REFUSAL_CATEGORIES.isdisjoint({call.get("local_response_reason"), call.get("error_category")}) + ) + + def _is_context_overflow_rejection(call: dict[str, Any]) -> bool: """Exclude only explicit upstream context rejections, never inferred empty generations.""" return ( @@ -448,6 +470,12 @@ def build_bindings(reference_items: tuple[tuple[str, dict[str, Any]], ...]) -> _ matched_calls=tuple(matched_calls), missing_references=tuple(missing_references), duplicated_references=tuple(duplicated_references), + # What each reference resolved to, for the hollow-turn exemption. A missing reference resolves to (), + # so its empty turn stays flagged. + resolutions={ + _call_reference_signature(raw_reference): tuple(resolved[_call_reference_signature(raw_reference)]) + for _, raw_reference in reference_items + }, ) return build_bindings(turn_reference_items), build_bindings(owned_reference_items) @@ -463,19 +491,34 @@ def _call_locator(call: dict[str, Any], fallback: int) -> dict[str, int | str]: return {"call_id": str(call.get("model_call_id") or call.get("response_id") or fallback)} -def _response_has_content(response: Any) -> bool: +def _response_has_usable_output(response: Any) -> bool: + """Whether a saved response holds something the agent can act on: answer text, a refusal or a tool call, in the + Responses, Chat Completions or Messages shape. Reasoning alone, a summary of it included, is not usable.""" if not isinstance(response, dict): return False - if _nonempty(response.get("output_text")) or _nonempty(response.get("content")): + if _nonempty(response.get("output_text")): return True - output = response.get("output") - choices = response.get("choices") - chat_content = any( - _nonempty(choice.get("text")) or _nonempty((choice.get("message") or {}).get("content")) - for choice in choices or [] - if isinstance(choice, dict) - ) - return _nonempty(output) or chat_content + for item in response.get("output") or []: + if not isinstance(item, dict) or item.get("type") == "reasoning": + continue + # Any other output item is a tool call of some kind. + if item.get("type") != "message" or _nonempty(item.get("content")): + return True + content = response.get("content") + if isinstance(content, str): + return bool(content.strip()) + for block in content if isinstance(content, list) else []: + if isinstance(block, dict) and ( + block.get("type") == "tool_use" or (block.get("type") == "text" and _nonempty(block.get("text"))) + ): + return True + for choice in response.get("choices") or []: + if not isinstance(choice, dict): + continue + message = choice.get("message") if isinstance(choice.get("message"), dict) else {} + if _nonempty(choice.get("text")) or _nonempty(message.get("content")) or message.get("tool_calls"): + return True + return False def _usage_tokens(usage: Any) -> tuple[int | None, int | None]: @@ -514,7 +557,15 @@ def _rollout_missing_agent_turns(trajectory: dict[str, Any], subject: dict[str, ] -def _agent_turn_hollow(trajectory: dict[str, Any], subject: dict[str, int | str]) -> list[Finding]: +def _agent_turn_hollow( + trajectory: dict[str, Any], bindings: _CallBindings, subject: dict[str, int | str] +) -> list[Finding]: + # An empty turn that ended on its own model call being rejected for a full context is the model filling its + # context, when another check accounts for that overflow (`_turn_ended_on_context_overflow`). The exemption is not + # applied when an incomplete capture leaves those checks unobserved. The calls are optional evidence here: without + # captured calls, every empty turn is still flagged. + incomplete = _trajectory_has_any_gap(trajectory, _INCOMPLETE_MODEL_CALL_GAPS) + chains = _model_chains(_normalized_trajectory_calls(trajectory)) return [ Finding( check="agent_turn_hollow", @@ -522,11 +573,51 @@ def _agent_turn_hollow(trajectory: dict[str, Any], subject: dict[str, int | str] locator=step.locator, detail={"reason": "agent turn has no message or tool calls"}, ) - for step in _agent_steps(trajectory) - if not step.has_message and not step.has_tool_calls + for step, turn in zip(_agent_steps(trajectory), trajectory.get("turns") or [], strict=True) + if not step.has_message + and not step.has_tool_calls + and (incomplete or not _turn_ended_on_context_overflow(turn, bindings, chains)) ] +def _turn_ended_on_context_overflow( + turn: dict[str, Any], bindings: _CallBindings, chains: dict[Any, list[dict[str, Any]]] +) -> bool: + """Whether a turn's own model calls end on a context-overflow rejection that another check accounts for. + + Each of the turn's references must resolve to exactly one call: a call that another turn's reference + resolved says nothing about this turn. Of the turn's calls, the last in time decides, as in + `_chain_ended_on_failure`, so an overflow followed by an empty retry is still hollow. When the calls + cannot be ordered, every one must be an overflow. + + Each such overflow must be accounted for: by a success of the same model in the same conversation, or by + `rollout_ended_on_failed_model_call` flagging that model's chain. An unordered chain that mixes in another + conversation's success is flagged by neither, so its empty turn stays hollow. + """ + calls = [] + for raw_reference in turn.get("model_calls") or []: + if not isinstance(raw_reference, dict) or _call_ref_key(raw_reference) is None: + continue + matches = bindings.resolutions.get(_call_reference_signature(raw_reference), ()) + if len(matches) != 1: + return False + calls.append(matches[0]) + if not calls: + return False + ends = calls + if all(call.get("started_at") is not None for call in calls): + ends = sorted(calls, key=lambda call: (call.get("started_at") or 0.0, call.get("call_index") or 0))[-1:] + for end in ends: + if not _is_context_overflow_rejection(end): + return False + ref = end.get("model_ref") + chain = chains.get((ref.get("type"), ref.get("name")) if isinstance(ref, dict) else None, []) + answered = any(_is_successful(call) and _same_conversation(call, end) for call in chain) + if not answered and not _chain_ended_on_failure(chain): + return False + return True + + def _model_call_zero_completion_tokens(bindings: _CallBindings, subject: dict[str, int | str]) -> list[Finding]: return [ Finding( @@ -536,7 +627,7 @@ def _model_call_zero_completion_tokens(bindings: _CallBindings, subject: dict[st detail={"completion_tokens": 0}, ) for position, call in enumerate(bindings.matched_calls) - if not _is_context_overflow_rejection(call) and not _is_failed_attempt(call) and call.get("tokens_out") == 0 + if not _is_length_refusal(call) and not _is_failed_attempt(call) and call.get("tokens_out") == 0 ] @@ -555,7 +646,7 @@ def _model_call_missing_token_counts(bindings: _CallBindings, subject: dict[str, }, ) for position, call in enumerate(bindings.matched_calls) - if not _is_context_overflow_rejection(call) + if not _is_length_refusal(call) and not _is_failed_attempt(call) and (call.get("tokens_in") is None or call.get("tokens_out") is None) ] @@ -627,6 +718,32 @@ def _model_chains(calls: Sequence[dict[str, Any]]) -> dict[Any, list[dict[str, A return chains +def _conversation_head(call: dict[str, Any]) -> Any: + """How the conversation a call continues opens: its instructions, system prompt, or first message. + + Calls of one conversation share it; another conversation on the same model server, such as a user simulator's, + opens differently. A call whose request was not captured has none. + """ + request = call.get("request") + if not isinstance(request, dict): + return None + for key in ("instructions", "system"): + if request.get(key): + return request[key] + items = request.get("messages", request.get("input")) + if isinstance(items, list): + return items[0] if items else None + return items or None + + +def _same_conversation(call: dict[str, Any], other: dict[str, Any]) -> bool: + """Whether two calls continue one conversation: their requests open the same way, and no captured session + tells them apart. Two agents can share instructions, so a session only ever separates calls.""" + head = _conversation_head(call) + sessions = {call.get("session"), other.get("session")} - {None} + return head is not None and head == _conversation_head(other) and len(sessions) <= 1 + + def _chain_ended_on_failure(chain: Sequence[dict[str, Any]]) -> bool: """Whether this model's own last call, in time, failed. @@ -638,19 +755,26 @@ def _chain_ended_on_failure(chain: Sequence[dict[str, Any]]) -> bool: the position in the merged trajectory list, so a producer-supplied retry can precede the attempt it retried. Say nothing rather than call a recovered rollout unhealthy. + + A last call the upstream rejected because the context was full is the + model's own end, scored like any other answer, once the model has answered + earlier in the same conversation (`_same_conversation`). A success in another + conversation on the same server, such as a user simulator's or another + session's, says nothing of whether this request fit. Before any answer the request never fit the deployment (a + prompt or output budget too large for it), which is a failure like any + other. An unordered chain of only failures has no answer and ended on one of + them, so it is flagged. """ if not chain: return False failed = [call for call in chain if _is_failed(call)] if not failed: return False - if len(failed) == len(chain): - # Nothing to order: whichever went last, it failed. - return True - if any(call.get("started_at") is None for call in chain): - return False - ordered = sorted(chain, key=lambda call: (call.get("started_at") or 0.0, call.get("call_index") or 0)) - return _is_failed(ordered[-1]) + if all(call.get("started_at") is not None for call in chain): + last = sorted(chain, key=lambda call: (call.get("started_at") or 0.0, call.get("call_index") or 0))[-1] + answered = any(_is_successful(call) and _same_conversation(call, last) for call in chain) + return _is_failed(last) and not (answered and _is_context_overflow_rejection(last)) + return len(failed) == len(chain) def _ended_on_failed_call(calls: Sequence[dict[str, Any]]) -> bool: @@ -670,7 +794,10 @@ def _rollout_ended_on_failed_model_call(trajectory: dict[str, Any], subject: dic Judged per model and in time order -- see `_chain_ended_on_failure`. Only the chain's last call counts, so a failure the client retried successfully stays healthy; the signal is that the rollout ENDED on a failure, which is - what makes its reward indistinguishable from a genuine zero. + what makes its reward indistinguishable from a genuine zero. A last call + the upstream rejected as a context-length overflow is not flagged once the + model has answered earlier in the same conversation: that zero is the + model's own result. """ findings = [] for key, chain in _model_chains(_normalized_trajectory_calls(trajectory)).items(): @@ -728,9 +855,9 @@ def _model_call_runaway_generation(bindings: _CallBindings, subject: dict[str, i detail={"finish_reason": call.get("finish_reason")}, ) for position, call in enumerate(bindings.matched_calls) - if not _is_context_overflow_rejection(call) + if not _is_length_refusal(call) and call.get("finish_reason") in _LENGTH_LIMIT_FINISH_REASONS - and not _response_has_content(call.get("response")) + and not _response_has_usable_output(call.get("response")) ] @@ -750,7 +877,9 @@ def _model_call_runaway_generation(bindings: _CallBindings, subject: dict[str, i "rollout_ended_on_failed_model_call": lambda record, trajectory, bindings, subject: ( _rollout_ended_on_failed_model_call(trajectory, subject) ), - "agent_turn_hollow": lambda record, trajectory, bindings, subject: _agent_turn_hollow(trajectory, subject), + "agent_turn_hollow": lambda record, trajectory, bindings, subject: _agent_turn_hollow( + trajectory, bindings, subject + ), "model_call_zero_completion_tokens": lambda record, trajectory, bindings, subject: ( _model_call_zero_completion_tokens(bindings, subject) ), diff --git a/nemo_gym/health/types.py b/nemo_gym/health/types.py index 7f398b82f97..3d2a0a10891 100644 --- a/nemo_gym/health/types.py +++ b/nemo_gym/health/types.py @@ -6,7 +6,7 @@ from __future__ import annotations from collections.abc import Mapping -from dataclasses import dataclass +from dataclasses import dataclass, field from enum import Enum from pathlib import Path from typing import Any, Literal @@ -216,6 +216,8 @@ class _CallBindings: matched_calls: tuple[dict[str, Any], ...] missing_references: tuple[str, ...] duplicated_references: tuple[tuple[str, int], ...] + # What each reference resolved to, by `_call_reference_signature`: one call, none, or several. + resolutions: Mapping[tuple[str, str, str, str], tuple[dict[str, Any], ...]] = field(default_factory=dict) @property def observed(self) -> bool: diff --git a/nemo_gym/rollout_health.py b/nemo_gym/rollout_health.py index 131df882b09..27bcca72d10 100644 --- a/nemo_gym/rollout_health.py +++ b/nemo_gym/rollout_health.py @@ -31,9 +31,9 @@ _bind_policy_call_views, _canonical_trajectory, _ended_on_failed_call, - _is_context_overflow_rejection, _is_failed, _is_failed_attempt, + _is_length_refusal, _is_successful, _normalized_trajectory_calls, _replay_identity, @@ -215,7 +215,7 @@ def evaluate_record( unobserved.append(spec.id) continue if spec.id == "model_call_runaway_generation" and any( - not _is_context_overflow_rejection(call) + not _is_length_refusal(call) and call.get("finish_reason") in _LENGTH_LIMIT_FINISH_REASONS and call.get("response") is None for call in bindings.matched_calls @@ -266,7 +266,7 @@ def evaluate_record( for call in failed ), errors_by_status=dict(errors_by_status), - ended_on_error=_ended_on_failed_call(calls), + ended_on_error=model_calls_observed and _ended_on_failed_call(calls), duplicated_calls=duplicated, transcript_prompt_tokens=transcript_prompt, transcript_completion_tokens=transcript_completion, diff --git a/responses_api_models/vllm_model/app.py b/responses_api_models/vllm_model/app.py index 58f78146c70..f39424d8b9e 100644 --- a/responses_api_models/vllm_model/app.py +++ b/responses_api_models/vllm_model/app.py @@ -18,6 +18,7 @@ import json import logging import os +import re from copy import deepcopy from dataclasses import dataclass from threading import Lock @@ -136,6 +137,31 @@ def _is_context_length_error(error: ClientResponseError) -> bool: return _names_context_window(message) or any(marker in message for marker in _MAX_TOKENS_PARAMETER_MARKERS) +# How an engine names its context window: "maximum context length is 32768", "max_model_len=32768", and vLLM 0.27's +# "max_model_len=max_total_tokens=32768". +_CONTEXT_WINDOW_SIZE = re.compile( + r"(?:maximum context length|maximum model length|max_model_len|max_total_tokens)\D{0,5}(\d+)" +) + + +def _length_refusal_category( + error: ClientResponseError, outbound: Dict[str, Any], *, completions_api: bool = False +) -> str: + """``invalid_output_token_parameter`` when the engine names its context window and the request's own output budget + exceeds all of it, so that no prompt fits; otherwise ``context_length_exceeded``. + + The budget is the one the engine received: on chat, ``max_completion_tokens`` before ``max_tokens``. A message that + names no window does not prove the budget wrong, so that refusal stays an overflow. + """ + keys = ("max_tokens",) if completions_api else ("max_completion_tokens", "max_tokens") + budget = next((outbound[key] for key in keys if outbound.get(key) is not None), None) + message, _, _ = context_overflow_error_fields(error) + window = _CONTEXT_WINDOW_SIZE.search(message) + if type(budget) is int and window is not None and budget > int(window.group(1)): + return "invalid_output_token_parameter" + return "context_length_exceeded" + + def _mark_propagated_overflow(error: ClientResponseError) -> str | None: """Mark a length refusal the server propagates, and return the envelope code chosen for it. @@ -1270,7 +1296,8 @@ async def chat_completions( """ result_content_str = e.response_content.decode(errors="replace") if _is_context_length_error(e): - execution["error_category"] = "context_length_exceeded" + category = _length_refusal_category(e, body_dict) + execution["error_category"] = category if self.config.propagate_context_overflow_errors: # The refused call generated nothing and gets no token record; the refusal # record resolves the capture intent registered before dispatch. @@ -1288,7 +1315,7 @@ async def chat_completions( completion_tokens_details=CompletionTokensDetails(reasoning_tokens=0), ) res.choices[0].finish_reason = "length" - execution.update(response_source="local", local_response_reason="context_length_exceeded") + execution.update(response_source="local", local_response_reason=category) return res else: _log_unhandled_engine_error(request, e.status, result_content_str, "chat completion") @@ -1630,7 +1657,8 @@ async def _chat_completions_via_completions_api( execution.update(response_source="upstream", upstream_status_code=e.status) result_content_str = e.response_content.decode(errors="replace") if _is_context_length_error(e): - execution["error_category"] = "context_length_exceeded" + category = _length_refusal_category(e, completion_body, completions_api=True) + execution["error_category"] = category if self.config.propagate_context_overflow_errors: # The refused call generated nothing and gets no token record; the refusal # record resolves the capture intent registered before dispatch. @@ -1648,7 +1676,7 @@ async def _chat_completions_via_completions_api( completion_tokens_details=CompletionTokensDetails(reasoning_tokens=0), ) res.choices[0].finish_reason = "length" - execution.update(response_source="local", local_response_reason="context_length_exceeded") + execution.update(response_source="local", local_response_reason=category) return res _log_unhandled_engine_error(request, e.status, result_content_str, "completion") raise diff --git a/responses_api_models/vllm_model/tests/test_app.py b/responses_api_models/vllm_model/tests/test_app.py index 4ea9394d630..32d54a215f9 100644 --- a/responses_api_models/vllm_model/tests/test_app.py +++ b/responses_api_models/vllm_model/tests/test_app.py @@ -5191,6 +5191,102 @@ def _make_completions_backend_model( ) +_WINDOW_32768 = json.dumps( + { + "message": "This model's maximum context length is 32768 tokens. However, you requested 40010 tokens " + "(10 in the messages, 40000 in the completion). Please reduce the length of the messages or completion." + } +) + + +@mark.parametrize("propagate", [False, True], ids=["finish-reason-length", "propagate-error"]) +@mark.parametrize("use_completions_api", [False, True], ids=["chat-completions", "completions"]) +@mark.parametrize( + ("budget", "error_content", "categories"), + [ + ( + {"max_tokens": 256, "max_completion_tokens": 40000}, + _WINDOW_32768, + ("invalid_output_token_parameter", "context_length_exceeded"), + ), + ( + {"max_tokens": 40000, "max_completion_tokens": 256}, + _WINDOW_32768, + ("context_length_exceeded", "invalid_output_token_parameter"), + ), + ({"max_completion_tokens": 40000}, _WINDOW_32768, ("invalid_output_token_parameter",) * 2), + ( + {"max_completion_tokens": 40000}, + '{"message": "max_completion_tokens=40000 cannot be greater than max_model_len=32768."}', + ("invalid_output_token_parameter",) * 2, + ), + ( + {"max_tokens": 40000}, + json.dumps( + { + "message": "max_tokens=40000 cannot be greater than max_model_len=max_total_tokens=32768. " + "Please request fewer output tokens. (parameter=max_tokens, value=40000)" + } + ), + ("invalid_output_token_parameter",) * 2, + ), + ({"max_tokens": 32768}, _WINDOW_32768, ("context_length_exceeded",) * 2), + ({}, _WINDOW_32768, ("context_length_exceeded",) * 2), + ( + {"max_tokens": 40000}, + '{"message": "max_tokens must be at least 1, got -6677."}', + ("context_length_exceeded",) * 2, + ), + ], + ids=[ + "chat-budget-over-window", + "completions-budget-over-window", + "alias-only-over-window", + "max-model-len", + "vllm-0.27-max-total-tokens", + "budget-equal-to-window", + "no-budget", + "no-window-named", + ], +) +async def test_a_budget_larger_than_the_named_window_is_an_invalid_parameter( + monkeypatch: MonkeyPatch, + propagate: bool, + use_completions_api: bool, + budget: dict, + error_content: str, + categories: tuple[str, str], +) -> None: + """No prompt fits a budget larger than the whole window the engine names, so the budget is at fault. The budget + is the one the engine received: chat reads max_completion_tokens first, and completions passes max_tokens.""" + monkeypatch.setattr(nemo_gym.server_utils, "get_global_config_dict", MagicMock(return_value={})) + model = _make_completions_backend_model() if use_completions_api else TestApp()._setup_server(monkeypatch) + model.config.propagate_context_overflow_errors = propagate + error = ClientResponseError(MagicMock(), (), status=400, message="backend request failed") + error.response_content = error_content.encode() + client = MagicMock(spec=NeMoGymAsyncOpenAI) + client.create_chat_completion = AsyncMock(side_effect=error) + client.create_completion = AsyncMock(side_effect=error) + model._clients = [client] + request = MagicMock() + request.session = {SESSION_ID_KEY: "invalid-budget-test"} + request.headers = {} + body = NeMoGymChatCompletionCreateParamsNonStreaming( + messages=[NeMoGymChatCompletionUserMessageParam(role="user", content="hello")], **budget + ) + + if propagate: + with raises(ClientResponseError): + await model.chat_completions(request, body) + else: + assert (await model.chat_completions(request, body)).choices[0].finish_reason == "length" + + category = categories[use_completions_api] + execution = request.state.nemo_gym_model_execution + assert execution["error_category"] == category + assert execution["local_response_reason"] == (None if propagate else category) + + @mark.parametrize("propagate", [False, True], ids=["finish-reason-length", "propagate-error"]) @mark.parametrize("use_completions_api", [False, True], ids=["chat-completions", "completions"]) @mark.parametrize( diff --git a/tests/unit_tests/test_rollout_health.py b/tests/unit_tests/test_rollout_health.py index c6986dfb734..f97c64a07c4 100644 --- a/tests/unit_tests/test_rollout_health.py +++ b/tests/unit_tests/test_rollout_health.py @@ -100,6 +100,7 @@ def _call(**updates) -> dict: def _trajectory_call(call: dict) -> dict: return { "model_call_id": call.get("model_call_id"), + "client_session_id": call.get("client_session_id"), "started_at": call.get("started_at"), "request": call.get("request"), "response": call.get("response"), @@ -1081,8 +1082,8 @@ def test_zero_token_call_is_flagged_and_nonempty_length_response_is_exempt(tmp_p checks = {finding.check for finding in result.rollouts[0].findings} assert "model_call_zero_completion_tokens" in checks assert "model_call_runaway_generation" not in checks - assert health_checks._response_has_content("malformed") is False - assert health_checks._response_has_content({"content": "visible"}) is True + assert health_checks._response_has_usable_output("malformed") is False + assert health_checks._response_has_usable_output({"content": "visible"}) is True @pytest.mark.parametrize( @@ -1141,6 +1142,60 @@ def test_current_capture_length_limit_reasons_trigger_runaway_generation( assert finding.detail == {"finish_reason": expected_reason} +_REASONING_ONLY = { + "responses": {"output": [{"type": "reasoning", "summary": [{"type": "summary_text", "text": "partial"}]}]}, + "chat": {"choices": [{"message": {"content": "", "reasoning_content": "partial"}}]}, + "messages": {"content": [{"type": "thinking", "thinking": "partial"}]}, +} +_ACTIONABLE = { + "responses_function_call": { + "output": [ + {"type": "reasoning", "summary": [{"type": "summary_text", "text": "partial"}]}, + {"type": "function_call", "call_id": "t1", "name": "run", "arguments": "{}"}, + ] + }, + "responses_answer": {"output": [{"type": "message", "content": [{"type": "output_text", "text": "cut off"}]}]}, + "chat_tool_call": {"choices": [{"message": {"content": "", "tool_calls": [{"id": "t1", "type": "function"}]}}]}, + "messages_tool_use": {"content": [{"type": "thinking", "thinking": "partial"}, {"type": "tool_use", "id": "t1"}]}, +} + + +@pytest.mark.parametrize( + ("response", "flagged"), + [ + *((response, True) for response in _REASONING_ONLY.values()), + *((response, False) for response in _ACTIONABLE.values()), + ], + ids=[*_REASONING_ONLY, *_ACTIONABLE], +) +def test_a_length_limited_call_with_only_reasoning_is_a_runaway_generation( + tmp_path: Path, response: dict, flagged: bool +) -> None: + """Reasoning gives the agent nothing to act on; an answer or a tool call does. A later call's recovery does not + erase the clipped one.""" + clipped = _call( + model_call_id="clipped", response_id="r-clipped", started_at=1.0, finish_reason="length", response=response + ) + recovered = _call( + model_call_id="recovered", + response_id="r-recovered", + started_at=2.0, + response={"output": [{"type": "message", "content": [{"type": "output_text", "text": "done"}]}]}, + ) + record = _record( + 0, + 0, + refs=[{"model_call_id": "clipped"}, {"model_call_id": "recovered"}], + usage={"input_tokens": 6, "output_tokens": 4}, + ) + rollout_path = _write_fixture(tmp_path, [(record, [clipped, recovered])]) + + [digest] = run_health_checks(rollout_path, workers=1).rollouts + + flagged_calls = [f.locator["call_id"] for f in digest.findings if f.check == "model_call_runaway_generation"] + assert flagged_calls == (["clipped"] if flagged else []) + + def test_runaway_generation_is_unobserved_without_a_saved_response(tmp_path: Path) -> None: rollout_path = _write_fixture( tmp_path, @@ -1429,6 +1484,504 @@ def test_a_rollout_ending_on_a_failed_attempt_is_unhealthy_for_that_reason_only( assert {finding.check for finding in digest.findings} == {"rollout_ended_on_failed_model_call"} +# The two shapes a context-length overflow takes in the capture. +CONTEXT_OVERFLOWS = { + # vLLM answers locally with an empty "length" completion when overflow errors are not propagated. + "answered_locally": { + "status_code": 200, + "upstream_status_code": 400, + "upstream_attempted": True, + "error_category": "context_length_exceeded", + "local_response_reason": "context_length_exceeded", + "response_source": "local", + "response_status": None, + "finish_reason": "length", + }, + # Propagated to the client as the upstream's own 400. + "propagated": { + "status_code": 400, + "upstream_status_code": 400, + "upstream_attempted": True, + "error_category": "context_length_exceeded", + "response_source": "upstream", + "response": {"error": {"code": "context_length_exceeded"}}, + **_NO_GENERATION, + }, +} + + +# The adapter's category when the request's own output budget exceeds the whole window: no prompt fits. +INVALID_OUTPUT_BUDGETS = { + name: overflow + | {"error_category": "invalid_output_token_parameter"} + | ({"local_response_reason": "invalid_output_token_parameter"} if "local_response_reason" in overflow else {}) + for name, overflow in CONTEXT_OVERFLOWS.items() +} + + +_POLICY_REF = {"type": "responses_api_models", "name": "policy_model"} +# A turn points at its call by the call id, or, as Terminal-Bench 4's turns do, by the model and response id. +TURN_CALL_REFERENCES = { + "by_call_id": {"model_call_id": "c1"}, + "by_response_id": {"model_ref": _POLICY_REF, "response_id": "r1"}, +} + + +@pytest.mark.parametrize("reference", TURN_CALL_REFERENCES.values(), ids=TURN_CALL_REFERENCES.keys()) +@pytest.mark.parametrize("overflow", CONTEXT_OVERFLOWS.values(), ids=CONTEXT_OVERFLOWS.keys()) +def test_an_empty_turn_from_a_context_overflow_is_not_hollow(tmp_path, overflow, reference): + """The model answered, then filled its context: the empty turn is not hollow, and the rollout is not + unhealthy.""" + answered = _call(model_call_id="c0", response_id="r0", started_at=1.0, model_ref=_POLICY_REF) + record = _record( + 0, + 0, + answer=None, + refs=[{"model_call_id": "c0"}, reference], + include_response_output=False, + usage={"input_tokens": 3, "output_tokens": 2}, + ) + overflow_call = _call( + **( + { + "call_index": 1, + "started_at": 2.0, + "tokens_in": None, + "tokens_out": 0, + "response": {}, + "model_ref": _POLICY_REF, + } + | overflow + ) + ) + [digest] = run_health_checks(_write_fixture(tmp_path, [(record, [answered, overflow_call])]), workers=1).rollouts + + assert "agent_turn_hollow" not in {finding.check for finding in digest.findings} + assert digest.verdict != "unhealthy" + + +def test_an_empty_turn_from_any_other_call_is_still_hollow(tmp_path): + record = _record(0, 0, answer=None, include_response_output=False) + empty_call = _call(tokens_out=0, response={}) + [digest] = run_health_checks(_write_fixture(tmp_path, [(record, [empty_call])]), workers=1).rollouts + + assert "agent_turn_hollow" in {finding.check for finding in digest.findings} + + +@pytest.mark.parametrize("overflow", CONTEXT_OVERFLOWS.values(), ids=CONTEXT_OVERFLOWS.keys()) +def test_a_rollout_ending_on_a_context_overflow_is_not_flagged(tmp_path, overflow): + """The model filled its context: a zero it earned, not an unhealthy rollout.""" + succeeded = _call(model_call_id="first", started_at=1.0) + last = _call(model_call_id="last", started_at=2.0, tokens_in=None, tokens_out=0, **overflow) + rollout_path = _write_fixture(tmp_path, [(_owned_attempts(succeeded, last), [succeeded, last])]) + + result = run_health_checks(rollout_path, workers=1) + [digest] = result.rollouts + + assert "rollout_ended_on_failed_model_call" not in {finding.check for finding in digest.findings} + assert result.summary["run"]["stats"]["model_call_errors"]["ended_on_error"] == 0 + # The call is still a failed call in the error counts. + assert digest.model_call_errors == 1 + + +@pytest.mark.parametrize( + "refusal", + [INVALID_OUTPUT_BUDGETS["answered_locally"], CONTEXT_OVERFLOWS["answered_locally"]], + ids=["invalid-budget", "context-overflow"], +) +@pytest.mark.parametrize("with_clipped", [True, False], ids=["beside-a-clipped-call", "alone"]) +def test_a_length_refusal_without_a_saved_body_leaves_runaway_generation_observed( + tmp_path: Path, refusal: dict, with_clipped: bool +) -> None: + """The engine refused before generating anything, so a missing body is no missing evidence, and another call's + runaway generation is still found.""" + calls = [ + _call( + model_call_id="refusal", + response_id="r-refusal", + started_at=1.0, + tokens_in=None, + tokens_out=0, + **(refusal | {"response": None}), + ) + ] + refs = [{"model_call_id": "refusal"}] + if with_clipped: + reasoning = {"type": "reasoning", "summary": [{"type": "summary_text", "text": "partial"}]} + calls.append( + _call( + model_call_id="clipped", + response_id="r-clipped", + started_at=2.0, + finish_reason="length", + response={"output": [reasoning]}, + ) + ) + refs.append({"model_call_id": "clipped"}) + record = _record(0, 0, refs=refs, usage={"input_tokens": 6, "output_tokens": 4}) + rollout_path = _write_fixture(tmp_path, [(record, calls)]) + + [digest] = run_health_checks(rollout_path, workers=1).rollouts + + assert "model_call_runaway_generation" not in digest.unobserved + flagged = [f.locator["call_id"] for f in digest.findings if f.check == "model_call_runaway_generation"] + assert flagged == (["clipped"] if with_clipped else []) + + +@pytest.mark.parametrize("invalid", INVALID_OUTPUT_BUDGETS.values(), ids=INVALID_OUTPUT_BUDGETS.keys()) +def test_a_budget_larger_than_the_window_still_flags_after_an_answer(tmp_path, invalid): + """No prompt fits such a budget, so an earlier answer excuses nothing.""" + succeeded = _call(model_call_id="first", started_at=1.0) + last = _call(model_call_id="last", started_at=2.0, tokens_in=None, tokens_out=0, **invalid) + rollout_path = _write_fixture(tmp_path, [(_owned_attempts(succeeded, last), [succeeded, last])]) + + result = run_health_checks(rollout_path, workers=1) + [digest] = result.rollouts + + checks = {finding.check for finding in digest.findings} + assert "rollout_ended_on_failed_model_call" in checks + assert result.summary["run"]["stats"]["model_call_errors"]["ended_on_error"] == 1 + assert digest.model_call_errors == 1 + # The engine generated nothing, so the token checks have nothing to read, as for an overflow. + assert not checks & { + "model_call_zero_completion_tokens", + "model_call_missing_token_counts", + "model_call_runaway_generation", + } + + +@pytest.mark.parametrize("ordered", [True, False], ids=["ordered", "unordered"]) +@pytest.mark.parametrize("overflow", CONTEXT_OVERFLOWS.values(), ids=CONTEXT_OVERFLOWS.keys()) +def test_an_overflow_before_the_model_ever_answered_still_flags(tmp_path, overflow, ordered): + """With no answer first, the request never fit the deployment: a budget of 0 or one larger than the window, + or a prompt too long for it. That is a broken run, not the model filling its context.""" + calls = [ + _call( + call_index=index, + model_call_id=None, + response_id=None, + tokens_out=0, + **({"started_at": float(index)} if ordered else {}), + **overflow, + ) + for index in range(2) + ] + rollout_path = _write_fixture(tmp_path, [(_unreferenced_failure_record([]), calls)]) + + result = run_health_checks(rollout_path, workers=1) + [digest] = result.rollouts + + assert "rollout_ended_on_failed_model_call" in {finding.check for finding in digest.findings} + assert result.summary["run"]["stats"]["model_call_errors"]["ended_on_error"] == 1 + + +@pytest.mark.parametrize("failure", FAILED_ATTEMPTS.values(), ids=FAILED_ATTEMPTS.keys()) +def test_a_failure_after_a_context_overflow_still_flags(tmp_path, failure): + """Only the last call decides: an overflow earlier in the chain hides nothing that came after it.""" + overflow = _call(model_call_id="first", started_at=1.0, tokens_out=0, **CONTEXT_OVERFLOWS["propagated"]) + failed = _call(model_call_id="last", started_at=2.0, tokens_in=None, tokens_out=None, **failure) + rollout_path = _write_fixture(tmp_path, [(_owned_attempts(overflow, failed), [overflow, failed])]) + + [digest] = run_health_checks(rollout_path, workers=1).rollouts + + assert "rollout_ended_on_failed_model_call" in {finding.check for finding in digest.findings} + + +@pytest.mark.parametrize("failure", FAILED_ATTEMPTS.values(), ids=FAILED_ATTEMPTS.keys()) +def test_a_context_overflow_after_only_failed_calls_still_flags(tmp_path, failure): + """Every call failed, so the model never answered: the overflow says the request never fit.""" + failed = _call(model_call_id="first", started_at=1.0, tokens_in=None, tokens_out=None, **failure) + overflow = _call(model_call_id="last", started_at=2.0, tokens_out=0, **CONTEXT_OVERFLOWS["propagated"]) + rollout_path = _write_fixture(tmp_path, [(_owned_attempts(failed, overflow), [failed, overflow])]) + + [digest] = run_health_checks(rollout_path, workers=1).rollouts + + assert "rollout_ended_on_failed_model_call" in {finding.check for finding in digest.findings} + + +def test_another_models_answer_does_not_excuse_an_overflow(tmp_path): + """Judged per model: a judge's answer says nothing of whether the policy's request fit.""" + judge = _call( + model_call_id="judge", started_at=1.0, model_ref={"type": "responses_api_models", "name": "judge_model"} + ) + overflow = _call( + model_call_id="last", + started_at=2.0, + tokens_in=None, + tokens_out=0, + model_ref=_POLICY_REF, + **CONTEXT_OVERFLOWS["propagated"], + ) + rollout_path = _write_fixture(tmp_path, [(_owned_attempts(judge, overflow), [judge, overflow])]) + + [digest] = run_health_checks(rollout_path, workers=1).rollouts + + assert "rollout_ended_on_failed_model_call" in {finding.check for finding in digest.findings} + + +def _conversation(dialect: str, opening: str, *messages: str) -> dict: + """A chat or Responses request whose conversation opens with `opening`.""" + turns = [{"role": "user", "content": message} for message in messages] + if dialect == "responses": + return {"instructions": opening, "input": turns} + return {"messages": [{"role": "system", "content": opening}, *turns]} + + +@pytest.mark.parametrize("dialect", ["chat", "responses"]) +def test_an_overflow_after_an_answer_in_the_same_conversation_is_not_flagged(tmp_path, dialect): + """A multi-turn agent answered, then its own conversation outgrew the window: its calls open the same way.""" + answered = _call( + model_call_id="first", + started_at=1.0, + model_ref=_POLICY_REF, + request=_conversation(dialect, "You are an agent.", "task"), + ) + overflow = _call( + model_call_id="last", + started_at=2.0, + tokens_in=None, + tokens_out=0, + model_ref=_POLICY_REF, + request=_conversation(dialect, "You are an agent.", "task", "next step"), + **CONTEXT_OVERFLOWS["propagated"], + ) + rollout_path = _write_fixture(tmp_path, [(_owned_attempts(answered, overflow), [answered, overflow])]) + + [digest] = run_health_checks(rollout_path, workers=1).rollouts + + assert "rollout_ended_on_failed_model_call" not in {finding.check for finding in digest.findings} + + +@pytest.mark.parametrize("dialect", ["chat", "responses"]) +@pytest.mark.parametrize("opening", ["You play the customer.", None], ids=["another_conversation", "no_request"]) +def test_an_answer_outside_the_overflowing_conversation_does_not_excuse_it(tmp_path, dialect, opening): + """tau2 can serve its user simulator from the policy's own model server, and the simulator speaks first. Its + answer says nothing of whether the agent's request fit, and neither does an answer whose request was not + captured.""" + answered = _call( + model_call_id="simulator", + started_at=1.0, + model_ref=_POLICY_REF, + request=_conversation(dialect, opening, "hello") if opening else None, + ) + overflow = _call( + model_call_id="agent", + started_at=2.0, + tokens_in=None, + tokens_out=0, + model_ref=_POLICY_REF, + request=_conversation(dialect, "You are an agent.", "task"), + **CONTEXT_OVERFLOWS["propagated"], + ) + rollout_path = _write_fixture(tmp_path, [(_unreferenced_failure_record([]), [answered, overflow])]) + + result = run_health_checks(rollout_path, workers=1) + [digest] = result.rollouts + + assert "rollout_ended_on_failed_model_call" in {finding.check for finding in digest.findings} + assert result.summary["run"]["stats"]["model_call_errors"]["ended_on_error"] == 1 + + +@pytest.mark.parametrize( + "harness_sessions,flagged", + [((None, None), True), (("shared", "shared"), False)], + ids=["two_sessions", "two_activations_of_one_harness_session"], +) +def test_a_session_tells_apart_conversations_that_open_alike(tmp_path, harness_sessions, flagged): + """Two sessions can share an agent's instructions: one session's answer says nothing of whether the other's first + request fit. Activations of one harness session continue one conversation.""" + answered = _call( + model_call_id="first", + started_at=1.0, + model_ref=_POLICY_REF, + client_session_id="s1", + request=_conversation("chat", "You are an agent.", "task A"), + ) + overflow = _call( + model_call_id="second", + started_at=2.0, + tokens_in=None, + tokens_out=0, + model_ref=_POLICY_REF, + client_session_id="s2", + request=_conversation("chat", "You are an agent.", "task B"), + **CONTEXT_OVERFLOWS["propagated"], + ) + record = _unreferenced_failure_record([]) + record["ng_trajectory"]["invocations"] = [ + {"kind": "agent_invocation", "invocation_id": session, "status": "completed"} + | ({"harness_session_id": harness} if harness else {}) + for session, harness in zip(("s1", "s2"), harness_sessions) + ] + rollout_path = _write_fixture(tmp_path, [(record, [answered, overflow])]) + + result = run_health_checks(rollout_path, workers=1) + [digest] = result.rollouts + + assert ("rollout_ended_on_failed_model_call" in {finding.check for finding in digest.findings}) is flagged + assert result.summary["run"]["stats"]["model_call_errors"]["ended_on_error"] == int(flagged) + + +def test_an_empty_overflow_turn_stays_hollow_when_no_other_check_accounts_for_it(tmp_path: Path) -> None: + """Without timestamps, an overflow beside another conversation's answer cannot be ordered, so + `rollout_ended_on_failed_model_call` says nothing: the empty turn is then the only finding left.""" + answered = _call(model_call_id="aux", model_ref=_POLICY_REF, request={"input": "another conversation"}) + overflow = _call( + model_ref=_POLICY_REF, tokens_in=None, tokens_out=0, response={}, **CONTEXT_OVERFLOWS["answered_locally"] + ) + record = _record(0, 0, answer=None, include_response_output=False) + rollout_path = _write_fixture(tmp_path, [(record, [answered, overflow])]) + + [digest] = run_health_checks(rollout_path, workers=1).rollouts + + checks = {finding.check for finding in digest.findings} + assert "agent_turn_hollow" in checks + assert "rollout_ended_on_failed_model_call" not in checks + assert digest.verdict == "unhealthy" + + +def test_an_unordered_chain_of_an_overflow_and_a_transport_failure_still_flags(tmp_path: Path) -> None: + calls = [ + _call(call_index=0, model_call_id=None, response_id=None, tokens_out=0, **CONTEXT_OVERFLOWS["propagated"]), + _call(call_index=1, model_call_id=None, response_id=None, **FAILED_ATTEMPTS["cancelled"]), + ] + rollout_path = _write_fixture(tmp_path, [(_unreferenced_failure_record([]), calls)]) + + [digest] = run_health_checks(rollout_path, workers=1).rollouts + + assert "rollout_ended_on_failed_model_call" in {finding.check for finding in digest.findings} + + +def test_an_empty_retry_after_a_context_overflow_in_the_same_turn_is_still_hollow(tmp_path: Path) -> None: + """Only the turn's last call can excuse it: a successful empty retry after an overflow is hollow.""" + overflow = _call( + model_call_id="overflow", + response_id="r-overflow", + started_at=1.0, + tokens_in=None, + tokens_out=0, + model_ref=_POLICY_REF, + **CONTEXT_OVERFLOWS["propagated"], + ) + retry = _call( + model_call_id="retry", + response_id="r-retry", + call_index=1, + started_at=2.0, + tokens_out=1, + response={}, + model_ref=_POLICY_REF, + ) + record = _record( + 0, + 0, + answer=None, + refs=[{"model_call_id": "overflow"}, {"model_call_id": "retry"}], + include_response_output=False, + usage={"input_tokens": 3, "output_tokens": 1}, + ) + [digest] = run_health_checks(_write_fixture(tmp_path, [(record, [overflow, retry])]), workers=1).rollouts + + checks = {finding.check for finding in digest.findings} + assert "agent_turn_hollow" in checks + assert "rollout_ended_on_failed_model_call" not in checks + assert digest.verdict == "unhealthy" + + +@pytest.mark.parametrize("failure", FAILED_ATTEMPTS.values(), ids=FAILED_ATTEMPTS.keys()) +def test_an_empty_turn_ending_on_a_context_overflow_after_a_failed_call_is_not_hollow(tmp_path, failure): + failed = _call( + model_call_id="failed", response_id="r-failed", started_at=1.0, tokens_in=None, tokens_out=None, **failure + ) + overflow = _call( + model_call_id="overflow", + response_id="r-overflow", + call_index=1, + started_at=2.0, + tokens_in=None, + tokens_out=0, + model_ref=_POLICY_REF, + **CONTEXT_OVERFLOWS["propagated"], + ) + record = _record( + 0, + 0, + answer=None, + refs=[{"model_call_id": "failed"}, {"model_call_id": "overflow"}], + include_response_output=False, + ) + [digest] = run_health_checks(_write_fixture(tmp_path, [(record, [failed, overflow])]), workers=1).rollouts + + assert "agent_turn_hollow" not in {finding.check for finding in digest.findings} + + +def test_an_empty_turn_whose_own_reference_does_not_resolve_is_still_hollow(tmp_path: Path) -> None: + """Turn 2 names the overflow call with a stale response id, so the call that turn 1 resolved excuses only turn 1.""" + overflow = _call( + tokens_in=None, tokens_out=0, response={}, model_ref=_POLICY_REF, **CONTEXT_OVERFLOWS["answered_locally"] + ) + record = _record(0, 0, answer=None, refs=[{"model_call_id": "c1"}], include_response_output=False) + stale_turn = dict(record["ng_trajectory"]["turns"][0]) + stale_turn.update(turn_no=2, timestamp=2.0, model_calls=[{"model_call_id": "c1", "response_id": "stale"}]) + record["ng_trajectory"]["turns"].append(stale_turn) + [digest] = run_health_checks(_write_fixture(tmp_path, [(record, [overflow])]), workers=1).rollouts + + hollow = [finding.locator for finding in digest.findings if finding.check == "agent_turn_hollow"] + assert hollow == [{"turn": 2}] + assert "trajectory_capture_mismatch" in {finding.check for finding in digest.findings} + + +def test_an_ambiguous_response_reference_is_not_excused_but_an_exact_call_reference_is(tmp_path: Path) -> None: + """Two overflow calls share one model and response id: the call-id reference resolves, the response-id one + matches both calls and excuses nothing.""" + shared = {"response_id": "shared", "tokens_in": None, "tokens_out": 0, "response": {}, "model_ref": _POLICY_REF} + first = _call(model_call_id="c1", **shared, **CONTEXT_OVERFLOWS["answered_locally"]) + second = _call(model_call_id="c2", call_index=1, **shared, **CONTEXT_OVERFLOWS["answered_locally"]) + record = _record(0, 0, answer=None, refs=[{"model_call_id": "c1"}], include_response_output=False) + ambiguous_turn = dict(record["ng_trajectory"]["turns"][0]) + ambiguous_turn.update(turn_no=2, timestamp=2.0, model_calls=[{"model_ref": _POLICY_REF, "response_id": "shared"}]) + record["ng_trajectory"]["turns"].append(ambiguous_turn) + [digest] = run_health_checks(_write_fixture(tmp_path, [(record, [first, second])]), workers=1).rollouts + + hollow = [finding.locator for finding in digest.findings if finding.check == "agent_turn_hollow"] + assert hollow == [{"turn": 2}] + + +def test_an_incomplete_capture_excuses_no_empty_turn(tmp_path: Path) -> None: + """An incomplete capture leaves `rollout_ended_on_failed_model_call` unobserved: it cannot flag a request that + never fit, so the empty turn is still hollow.""" + overflow = _call( + tokens_in=None, tokens_out=0, response={}, model_ref=_POLICY_REF, **CONTEXT_OVERFLOWS["answered_locally"] + ) + record = _record(0, 0, answer=None, refs=[{"model_call_id": "c1"}], include_response_output=False) + record["ng_trajectory"]["gaps"] = [{"code": "model_call_capture_incomplete"}] + [digest] = run_health_checks(_write_fixture(tmp_path, [(record, [overflow])]), workers=1).rollouts + + assert "rollout_ended_on_failed_model_call" in digest.unobserved + assert "agent_turn_hollow" in {finding.check for finding in digest.findings} + + +def test_a_400_without_the_overflow_category_still_flags(tmp_path: Path) -> None: + """The overflow is read from the recorded category, never inferred from a 400 alone.""" + succeeded = _call(model_call_id="first", started_at=1.0) + rejected = _call( + model_call_id="last", + started_at=2.0, + status_code=400, + upstream_status_code=400, + upstream_attempted=True, + error_category="client_error", + tokens_in=None, + tokens_out=None, + **_NO_GENERATION, + ) + rollout_path = _write_fixture(tmp_path, [(_owned_attempts(succeeded, rejected), [succeeded, rejected])]) + + [digest] = run_health_checks(rollout_path, workers=1).rollouts + + assert "rollout_ended_on_failed_model_call" in {finding.check for finding in digest.findings} + + def test_a_body_capture_could_not_parse_still_needs_token_counts(tmp_path: Path) -> None: """`capture_parse_error` marks a 2xx call whose body was unreadable: a capture defect, not a failed call.""" unparsed = _call( @@ -1492,6 +2045,24 @@ def test_the_ended_on_error_statistic_and_the_finding_agree(tmp_path: Path) -> N assert "rollout_ended_on_failed_model_call" in {finding.check for finding in digest.findings} +def test_an_incomplete_capture_is_not_counted_as_ending_on_an_error(tmp_path: Path) -> None: + """The check is unobserved on an incomplete capture, so the statistic leaves the rollout out as well.""" + succeeded = _call(model_call_id="first", started_at=1.0) + failed = _call( + model_call_id="last", started_at=2.0, tokens_in=None, tokens_out=None, **FAILED_ATTEMPTS["http_500"] + ) + record = _owned_attempts(succeeded, failed) + record["ng_trajectory"]["gaps"] = [{"code": "model_call_capture_incomplete"}] + + result = run_health_checks(_write_fixture(tmp_path, [(record, [succeeded, failed])]), workers=1) + [digest] = result.rollouts + + assert "rollout_ended_on_failed_model_call" in digest.unobserved + assert "rollout_ended_on_failed_model_call" not in {finding.check for finding in digest.findings} + assert digest.ended_on_error is False + assert result.summary["run"]["stats"]["model_call_errors"]["ended_on_error"] == 0 + + POLICY_REF = {"type": "responses_api_models", "name": "policy_model"} JUDGE_REF = {"type": "responses_api_models", "name": "genrm_model"}