Skip to content
Merged
Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
623d7d7
feat(eval): count rollouts that ran out of time in the aggregate metrics
AdamRajfer Oct 21, 2026
7a4900c
fix(eval): write the time-limit section atomically
AdamRajfer Oct 21, 2026
025cc39
Merge main into eval/time-limit-detection
AdamRajfer Oct 22, 2026
e34f2ca
feat(eval): report timed-out rollouts in the run health summary
AdamRajfer Oct 22, 2026
46ff2ed
docs(eval): document run.stats.timeouts on the rollout health page
AdamRajfer Oct 22, 2026
e321fc3
refactor(eval): rework the timeouts section of the run health summary
AdamRajfer Oct 22, 2026
a440c9f
docs(eval): describe the typed timeout record by what it is
AdamRajfer Oct 22, 2026
40c754a
fix(health): keep the timeouts section safe on malformed rows and ove…
AdamRajfer Oct 23, 2026
0c58077
docs(health): a typed timeout event goes on the row
AdamRajfer Oct 23, 2026
cb1a6f8
Merge main into the PR branch
AdamRajfer Oct 23, 2026
b2b90e8
fix(health): time a failed rollout from its capture file
AdamRajfer Oct 23, 2026
807cbe7
fix(health): one timeout classifier and a safe, linear capture join
AdamRajfer Oct 23, 2026
5ef88a4
fix(health): read gap codes safely, and count unread transcripts and …
AdamRajfer Oct 24, 2026
fae3eec
fix(health): leave tool time unknown when an agent observed no tools,…
AdamRajfer Oct 24, 2026
96f9297
fix(health): leave the time split unknown when part of its evidence w…
AdamRajfer Oct 24, 2026
d10106d
fix(health): count a malformed timeouts value, and leave an unidentif…
AdamRajfer Oct 24, 2026
b4709b6
Merge main into eval/time-limit-detection
AdamRajfer Oct 24, 2026
66cc37d
fix(health): leave tool time unknown when OpenCode drops a tool for i…
AdamRajfer Oct 24, 2026
87b123c
test(health): move the agent-parser timeout tests to the agents' suites
AdamRajfer Oct 24, 2026
fe4e9a4
fix(health): count a Terminus 2 rollout stopped by its run budget as …
AdamRajfer Oct 25, 2026
b1bffd4
fix(anyswe): parse colored R2E pytest summaries
svcnemo-autobot Oct 25, 2026
f194f10
Merge main into eval/time-limit-detection
AdamRajfer Oct 26, 2026
7244137
fix(health): preserve configured policy identity in timeout timing
AdamRajfer Oct 26, 2026
6cdb72e
fix(health): classify episode timeouts consistently in live timing
AdamRajfer Oct 26, 2026
ed7e5db
Merge main into timeout diagnostics and verify shard rechecks
AdamRajfer Oct 26, 2026
9590ba0
Merge main into timeout diagnostics
AdamRajfer Oct 26, 2026
dec7c37
Merge branch 'main' into eval/time-limit-detection
AdamRajfer Oct 27, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
92 changes: 85 additions & 7 deletions fern/versions/latest/pages/evaluation/rollout-health.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -60,13 +60,15 @@ model-call evidence, explicit model-call references, and observation gaps produc
[trajectory capability matrix](/main/reference/trajectory-capabilities) for measured harness conformance and
[model-call capture](/main/model-server/model-call-capture) for collection configuration.

The health checks runner does not reopen raw model-call capture files, inspect `ng_model_call_capture`, or reconstruct turns from an
agent-specific response. A custom rollout driver receives the same behavior as a built-in driver: checks run when it writes a
The integrity checks do not reopen raw model-call capture files, inspect `ng_model_call_capture`, or reconstruct turns from an
agent-specific response. The timeout statistics are separate: their time split reads each row's `ng_model_call_capture`, and
the capture file of a rollout that failed without a row when collection supplies its capture dir or a recheck passes
`--model-call-capture-dir`. A custom rollout driver receives the same behavior as a built-in driver: checks run when it writes a
valid `ng_trajectory`, and checks that need missing evidence are unobserved.

<Note>
This single-source rule makes health results reproducible after capture files move and prevents two observability representations
from disagreeing. Rollouts collected before `ng_trajectory` was introduced can still be parsed, but trajectory-dependent checks
This single-source rule makes the integrity verdicts reproducible after capture files move and prevents two observability
representations from disagreeing. Moving or omitting the capture files can change only `run.stats.timeouts.time_split`. Rollouts collected before `ng_trajectory` was introduced can still be parsed, but trajectory-dependent checks
are unobserved instead of being evaluated from inferred historical formats.
</Note>

Expand Down Expand Up @@ -273,9 +275,9 @@ This abridged example omits the other check-coverage entries and zero-valued iss
`run.issues` counts individual findings, not distinct affected rollouts or tasks. It contains every registered check ID, including
IDs whose count is zero. `run.artifacts.coverage` reports how often each check was evaluated, unobserved, or explicitly ignored.

`run.artifacts.captures` counts rollout trajectories that contain model-call evidence; health does not count or reopen raw
capture files. Raw statistics under `run.stats` describe all model calls stored in `ng_trajectory`, including unreferenced
auxiliary calls. Checks with a policy-model subject evaluate only exactly bound calls.
`run.artifacts.captures` counts rollout trajectories that contain model-call evidence; the integrity checks do not count or
reopen raw capture files. Raw statistics under `run.stats` describe all model calls stored in `ng_trajectory`, including
unreferenced auxiliary calls, except `run.stats.timeouts`, whose time split reads the captured calls themselves. Checks with a policy-model subject evaluate only exactly bound calls.
`model_call_errors.usage_unknown` counts failed calls that returned no finished response and carry no token counts. The
token checks skip these calls, so any tokens the server spent on them before a cancellation or timeout are counted only here,
as a number of calls.
Expand All @@ -290,6 +292,82 @@ as a number of calls.

`expected`, `missing` and `healthy` are null without an inventory. The summary is also written when collection fails or Slurm stops the job with SIGTERM.

### Timeouts

`run.stats.timeouts` counts the rollouts that ran out of time. It reads each rollout's final record: its row in the
rollout file, or the last failure row of a rollout that never landed. `gym eval health-check` recomputes it from the
same files, and from the run's model-call captures when given `--model-call-capture-dir` (see `time_split`). After
`gym eval aggregate` merges shards, recheck the shards: the merged file holds their rows but not their failure rows.
It is a report only and changes no score or verdict.

A rollout counts as timed out on one of two kinds of evidence:

1. `typed`: a timeout record in the row's `timeouts` list. The component that enforces a limit attaches it to the
row; the operation records in `ng_trajectory` have no field for it. Each record has `event_id`, `limit`,
`enforcer`, `evidence` (`internal` or `external`), `configured_s`, `elapsed_s` and `fired_at`. A record listed
twice counts once by its `event_id`. An entry that is not a valid record is not evidence and is counted in
`invalid_events`. A typed record wins over a legacy signal.
2. `legacy`: a field that a Gym component sets today when its own deadline fires. `by_limit` names it:

| Signal | Set by |
|---|---|
| `_ng_failure_class=agent_timeout` | GDPval |
| `_ng_failure_class=timeout_exceeded` | PinchBench |
| `_ng_failure_message=Episode timed out` | an environment server's episode deadline |
| `agent_timed_out` | the Codex agent |
| `agent_timeout_error` | the Harbor agent |
| `eval_timed_out` | SWE-bench Pro's verification |
| `timed_out` | BrowseComp's per-rollout limit (`rollout_timeout_s`) |
| `termination.reason=timeout` | Terminal-Bench's agent termination record |

How long a rollout took never makes it count as timed out. A failed rollout with no evidence is counted in
`failed_without_signal`: it is not known to be a timeout, and not known not to be.

```json
"timeouts": {
"rollouts": 200,
"timed_out": 12,
"by_evidence": {"typed": 8, "legacy": 4},
"by_limit": {"default_episode_timeout_seconds": 8, "_ng_failure_class=timeout_exceeded": 4},
"by_enforcer": {"gym.environment_server": 8},
"events": 8,
"invalid_events": 0,
"failed_without_signal": 3,
"elapsed_s": {"p50": 1790.2, "p95": 1801.4},
"time_split": {
"rollouts_with_captures": 11,
"policy_model": 0.62,
"other_model": 0.04,
"tool": 0.21,
"untracked": 0.13,
"policy_output_tokens_per_s": {"p50": 41.3, "p5": 12.9},
"policy_calls_per_rollout": {"p50": 37.0, "p95": 64.0}
}
}
```

`elapsed_s` is the `/run` wall clock that `gym eval run` stamps on each row as `_ng_elapsed_s`. `time_split` shows
where the time of the timed-out rollouts went. It covers the timed-out rollouts that have both that stamp and a whole
capture of their model calls. A rollout that failed without a row has no capture on it, so the check reads its capture
Comment thread
AdamRajfer marked this conversation as resolved.
Resolved
file from the run's capture dir: collection passes its `model_call_capture_dir`, a recheck needs
`--model-call-capture-dir` and without it warns when the run has failed rollouts and timed-out rollouts left out of the
split, and `gym eval aggregate` does not read capture files yet. A capture with a write still pending, a failed write
or an unreadable record leaves its rollout out. Each share is the time spent in one kind of call over their summed
elapsed time: policy-model calls, other models' calls (a judge or a user simulator) and tool calls. `untracked` is the time in which none of these calls ran, such as setup or verification.
Overlapping calls count once, also across kinds for `untracked`, so the shares can add up to more than the rest when,
for example, a tool calls the policy model. A rollout's busy time is capped at its elapsed time. A share is null when
one of those rollouts does not measure that kind of time: a call without timing or one that ends before it starts,
calls of which one reports only its duration and so cannot be placed against the others, or a trajectory without
tool timing. Collection marks a row whose agent sent no valid trajectory or observations with a
`tool_timing_unavailable` gap, so its tool time is unknown rather than zero; so does an `agent_artifact_unavailable`
gap, which an agent reports when it could not read its own transcript. `policy_output_tokens_per_s` is end to end, so
it includes queue and prefill time, and a call that produced no output counts at zero.

When an exporter is configured, `gym eval run` and `gym eval aggregate` also send this section's numbers as flat keys,
such as `health/timeouts/timed_out` and `health/timeouts/by_evidence/typed`. A null value is not sent. A limit or
enforcer that only an earlier failed attempt named is sent as 0, so a resumed tracking run does not keep its old
count. A failed export is logged and does not stop the run.

A run stopped by SIGTERM gets a provisional summary: the run accounting in `run.artifacts`, with `partial: true`, and no `run.verdicts` or `rollout_verdicts.jsonl`. It evaluates only `run_excessive_failures`, which needs no rollouts, and lists every other check in `run.artifacts.not_evaluated`. Its `healthy` is false when the accounting already has a reason and null otherwise. Run `gym eval health-check` on the run directory for the full summary; it judges the run by the limits the provisional summary recorded. A stopped run exits with status 143, or with status 1 (`EVAL FAILED`) when `require_complete` is on, which is the default under `gym eval submit`.

### `rollout_verdicts.jsonl`
Expand Down
5 changes: 3 additions & 2 deletions nemo_gym/base_responses_api_model.py
Original file line number Diff line number Diff line change
Expand Up @@ -808,9 +808,10 @@ def _validate_rollout_id(rollout_id: str) -> str:
class CaptureStore:
"""Append-only, rollout-keyed JSONL sink for model exchanges."""

def __init__(self, root: str | Path) -> None:
def __init__(self, root: str | Path, *, create: bool = True) -> None:
self._root = Path(root)
self._root.mkdir(parents=True, exist_ok=True)
if create:
self._root.mkdir(parents=True, exist_ok=True)

@property
def root(self) -> Path:
Expand Down
25 changes: 22 additions & 3 deletions nemo_gym/cli/eval.py
Original file line number Diff line number Diff line change
Expand Up @@ -645,9 +645,14 @@ def e2e_rollout_collection(): # pragma: no cover
# A failed or interrupted run reports health too, so its failed and missing rollouts are counted.
try:
if health_check_enabled:
from nemo_gym.rollout_health import capture_dirs_for_health

# A custom driver keeps no landed keys, so its stopped run gets the full check.
_health_check_collected_rollouts(
output_fpath, rollout_collection_config, None if driver_path else rch._landed_keys
output_fpath,
rollout_collection_config,
None if driver_path else rch._landed_keys,
model_call_capture_dirs=capture_dirs_for_health(global_config_dict),
)
finally:
rh.shutdown()
Expand Down Expand Up @@ -683,7 +688,9 @@ def _remove_health_reports(output_fpath: Path) -> None:
(output_fpath.parent / report).unlink(missing_ok=True)


def _health_check_collected_rollouts(output_fpath: Path, config: Any, landed_keys: set | None = None) -> None:
def _health_check_collected_rollouts(
output_fpath: Path, config: Any, landed_keys: set | None = None, *, model_call_capture_dirs: Sequence[Path] = ()
) -> None:
"""Write the quality summary for whatever a run collected; a run that never started collection has none.

After SIGTERM, Slurm kills the job once KillWait (120 s on some clusters) is over, and the full check can take
Expand All @@ -692,6 +699,7 @@ def _health_check_collected_rollouts(output_fpath: Path, config: Any, landed_key
"""
if not output_fpath.exists():
return
from nemo_gym.health.timeouts import export_run_health_timeouts
from nemo_gym.rollout_health import format_health_report, run_health_checks, write_provisional_summary

if _terminated and landed_keys is not None:
Expand All @@ -713,11 +721,13 @@ def _health_check_collected_rollouts(output_fpath: Path, config: Any, landed_key
workers=config.health_check_workers,
ignored_checks=config.health_check_ignored_checks,
limits=config.run_health_limits,
model_call_capture_dirs=model_call_capture_dirs,
)
except Exception:
logger.exception("Rollout health checks failed after collection; rollout artifacts are still available.")
else:
print(format_health_report(health_result))
export_run_health_timeouts(health_result)


@exit_cleanly_on_config_error
Expand Down Expand Up @@ -751,7 +761,14 @@ def collect_rollouts(): # pragma: no cover
shutdown_telemetry()
finally:
if health_check_enabled:
_health_check_collected_rollouts(Path(config.output_jsonl_fpath), config, rch._landed_keys)
from nemo_gym.rollout_health import capture_dirs_for_health

_health_check_collected_rollouts(
Path(config.output_jsonl_fpath),
config,
rch._landed_keys,
model_call_capture_dirs=capture_dirs_for_health(global_config_dict),
)
_exit_if_terminated()


Expand All @@ -774,6 +791,7 @@ def health_check_rollouts(
ignored_checks: Sequence[str] = (),
limits: Any = None,
json_output: bool = False,
model_call_capture_dir: str | Path | None = None,
):
"""Run rollout quality verification for an existing run directory.

Expand All @@ -790,6 +808,7 @@ def health_check_rollouts(
ignored_checks=ignored_checks,
limits=limits,
json_output=json_output,
model_call_capture_dir=model_call_capture_dir,
)


Expand Down
12 changes: 12 additions & 0 deletions nemo_gym/cli/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -804,6 +804,7 @@ def _eval_health_check(args: argparse.Namespace, overrides: list[str]) -> None:
ignored_checks=args.ignore_checks or (),
limits=RunHealthLimits(**given_limits),
json_output=args.json,
model_call_capture_dir=args.model_call_capture_dir,
)
except (FileNotFoundError, ValueError) as exc:
args._parser.error(str(exc))
Expand Down Expand Up @@ -1368,6 +1369,17 @@ def _dev_compare(args: argparse.Namespace, overrides: list[str]) -> None:
)
),
Flag(register=lambda p: p.add_argument("--workers", type=int, help="Number of worker processes.")),
Flag(
register=lambda p: p.add_argument(
"--model-call-capture-dir",
type=Path,
metavar="DIR",
help=(
"The run's model_call_capture_dir. With it, the captured calls of rollouts that failed "
"without a row (an episode deadline, say) count in run.stats.timeouts' time split."
),
)
),
Flag(
register=lambda p: p.add_argument(
"--ignore-checks",
Expand Down
Loading
Loading