Repository navigation
Description
NOTE Up to date requirements are being tracked in the google doc: https://docs.google.com/document/d/1z0wLyl6lNpLhLCqd33EH04RdcIl4UTy08_ROEnNRnUA/edit?tab=t.3xnkyvgfyztw
This issue is outdated.
Standardized trajectory logging requirements
Use cases, pain points, and background
We need standardized trajectory/log data for cost estimation, debugging, and rollout analysis across Gym benchmarks.
This issue is not a request to make ATIF the native Gym format. The request is to define the telemetry Gym should expose. ATIF is useful as a design reference for ordered steps, efficient history reconstruction, observations, metrics, and context boundaries:
Requirements
1. Standard per-model-call stats
PR #1715 already addresses part of this with model-server trajectory capture and StepRecord stats: #1715
The standardized Gym contract should expose, when available:
- task/rollout identity:
task_id,trial_name/ rollout id - turn identity:
turn_no,num_turns,timestamp - response structure: model tool calls, reasoning_content
- token stats: prompt, completion, reasoning, total, cached tokens
2. Reconstructable model input/output history
We need enough data to reconstruct the model-visible request/response pairs for a rollout (also covered by: #1715):
This should preferably be efficient. For multi-turn agents, repeatedly storing the full re-materialized prompt on every turn grows quickly. A delta/append-only representation is preferred.
ATIF has relevant design references:
- complete interaction history: https://github.com/harbor-framework/harbor/blob/main/rfcs/0001-trajectory-format.md#L250-L257
- context boundaries / compaction: https://github.com/harbor-framework/harbor/blob/main/rfcs/0001-trajectory-format.md#L720-L730
Gym can use ATIF or a Gym-native equivalent, but we need consistent reconstruction semantics.
3. Tool-call observations and timing
Gym already has model-visible tool output via function_call_output:
{
"type": "function_call_output",
"call_id": "call_123",
"output": "tool result string",
"status": "completed"
}Schema:
Lines 172 to 182 in ff6f8d2
| class NeMoGymFunctionCallOutput(BaseModel): | |
| """ | |
| We copy openai.types.responses.response_input_param.FunctionCallOutput, originally a TypedDict, as a BaseModel here | |
| so that we can use it in the NeMoGymResponseOutputItem below and be consistent with the other ResponseOutputItem types. | |
| """ | |
| call_id: str | |
| output: str | |
| type: Literal["function_call_output"] = "function_call_output" | |
| id: Optional[str] = None | |
| status: Optional[Literal["in_progress", "completed", "incomplete"]] = None |
We also need logging-only execution metadata:
call_id- model-visible tool output
- status
started_atcompleted_atduration_ms- CPU/sandbox timing stats, when available
- error/timeout metadata
- sandbox/container metadata, when available
Example shape, based on ATIF observations:
{
"observation": {
"results": [
{
"source_call_id": "functions.terminal:0",
"content": "...tool output visible to model...",
"extra": {
"started_at": "2026-06-25T11:37:28.156623",
"completed_at": "2026-06-25T11:37:28.688924",
"duration_ms": 532.301,
"status": "completed"
}
}
]
}
}ATIF references:
- call/observation correlation via
source_call_id: https://github.com/harbor-framework/harbor/blob/main/rfcs/0001-trajectory-format.md#L363-L369 - observation
content/extra: https://github.com/harbor-framework/harbor/blob/main/rfcs/0001-trajectory-format.md#L259-L262 - observation scope: https://github.com/harbor-framework/harbor/blob/main/rfcs/0001-trajectory-format.md#L353-L355
For parallel tool calls, timing cannot be approximated from time between turns. We need independent start/end timestamps from the actual tool execution boundary.
Existing Gym patterns to support
The solution should work for both:
-
Resource-server-backed tools/envs, e.g.
simple_agent, BrowseComp, Finance, CVDP, Gymnasium:Gym/responses_api_agents/simple_agent/app.py
Lines 123 to 162 in ff6f8d2
all_fn_calls: List[NeMoGymResponseFunctionToolCall] = [o for o in output if o.type == "function_call"] all_output_messages: List[NeMoGymResponseOutputMessage] = [ o for o in output if o.type == "message" and o.role == "assistant" ] if not all_fn_calls and all_output_messages: break for output_function_call in all_fn_calls: try: parsed_arguments = json.loads(output_function_call.arguments) except (json.JSONDecodeError, TypeError) as e: # Model produced malformed tool-call arguments. Surface the # error back as a tool response so the rollout can continue # (or terminate with a low reward) instead of crashing the # whole batch on json.loads. tool_response = NeMoGymFunctionCallOutput( type="function_call_output", call_id=output_function_call.call_id, # Use repr(e) so the exception type name is always # included even when str(e) would be empty. output=json.dumps({"error": f"Invalid tool call arguments: {e!r}"}), ) new_outputs.append(tool_response) continue api_response = await self.server_client.post( server_name=self.config.resources_server.name, url_path=f"/{output_function_call.name}", json=parsed_arguments, cookies=resources_server_cookies, ) # We don't raise for status here since it's a valid return for the API to error e.g. if the model outputs an invalid call or something. resources_server_cookies = api_response.cookies tool_response = NeMoGymFunctionCallOutput( type="function_call_output", call_id=output_function_call.call_id, output=(await api_response.content.read()).decode(), ) new_outputs.append(tool_response) Gym/responses_api_agents/browsecomp_agent/app.py
Lines 481 to 514 in ff6f8d2
for output_function_call in all_fn_calls: num_tool_calls += 1 log_event("tool_call_begin", name=output_function_call.name, args=output_function_call.arguments) tool_start = time.monotonic() api_response = await self.server_client.post( server_name=self.config.resources_server.name, url_path=f"/{output_function_call.name}", json=json.loads(output_function_call.arguments), cookies=resources_server_cookies, ) # We don't raise for status here since it's a valid return for the API to error e.g. if the model outputs an invalid call or something. resources_server_cookies = api_response.cookies tool_output = (await api_response.content.read()).decode() tool_dur = time.monotonic() - tool_start tool_total_dur += tool_dur log_event( "tool_call", duration_s=round(tool_dur, 3), name=output_function_call.name, args=output_function_call.arguments, output=tool_output, output_len=len(tool_output), ) if self.config.nudge_steps: turns_left = self.config.max_steps - step tool_output += "\n\n[%d turns remaining out of %d]" % (turns_left, self.config.max_steps) tool_response = NeMoGymFunctionCallOutput( type="function_call_output", call_id=output_function_call.call_id, output=tool_output, ) new_outputs.append(tool_response) Gym/responses_api_agents/finance_agent/app.py
Lines 247 to 290 in ff6f8d2
done = False for output_function_call in all_fn_calls: try: coro = self.server_client.post( server_name=self.config.resources_server.name, url_path=f"/{output_function_call.name}", json=json.loads(output_function_call.arguments), cookies=resources_server_cookies, ) api_response = await asyncio.wait_for(coro, timeout=self.config.tool_call_timeout) resources_server_cookies = api_response.cookies tool_output = (await api_response.content.read()).decode() except asyncio.TimeoutError: logger.warning( "Tool call '%s' timed out after %ss", output_function_call.name, self.config.tool_call_timeout, ) tool_output = json.dumps( { "error": f"Tool call timed out after {self.config.tool_call_timeout}s. " "Please try a different approach or submit your final answer." } ) except Exception as e: logger.error( "Tool call '%s' failed: %s: %s", output_function_call.name, type(e).__name__, e, ) tool_output = json.dumps( { "error": f"Tool call failed: {type(e).__name__}: {e}. " "Please try a different approach or submit your final answer." } ) tool_response = NeMoGymFunctionCallOutput( type="function_call_output", call_id=output_function_call.call_id, output=tool_output, ) new_outputs.append(tool_response) Gym/responses_api_agents/cvdp_agent/app.py
Lines 129 to 144 in ff6f8d2
for output_function_call in all_fn_calls: api_response = await self.server_client.post( server_name=self.config.resources_server.name, url_path=f"/{output_function_call.name}", json=json.loads(output_function_call.arguments), cookies=resources_server_cookies, ) # We don't raise for status here since it's a valid return for the API to error e.g. if the model outputs an invalid call or something. resources_server_cookies = api_response.cookies tool_response = NeMoGymFunctionCallOutput( type="function_call_output", call_id=output_function_call.call_id, output=(await api_response.content.read()).decode(), ) new_outputs.append(tool_response) Gym/responses_api_agents/gymnasium_agent/app.py
Lines 130 to 152 in ff6f8d2
step_resp = await self.server_client.post( server_name=self.config.resources_server.name, url_path="/step", json=body.model_dump() | {"response": model_response.model_dump()}, cookies=env_cookies, ) await raise_for_status(step_resp) step_data = EnvStepResponse.model_validate(await get_response_json(step_resp)) total_reward += step_data.reward env_cookies = step_resp.cookies if step_data.terminated or step_data.truncated: finished = True break for tool_output in (step_data.info or {}).get("tool_outputs", []): new_outputs.append( NeMoGymFunctionCallOutput( type="function_call_output", call_id=tool_output["call_id"], output=tool_output["output"], ) )
-
Custom/sandbox-backed adapters, e.g. Tau2 and PinchBench:
Gym/responses_api_agents/tau2/app.py
Lines 125 to 167 in ff6f8d2
async def run(self, body: Tau2RunRequest) -> Tau2VerifyResponse: body_dict = {name: getattr(body, name) for name in Tau2RunRequest.model_fields} responses_create_params = body_dict.pop("responses_create_params").model_dump(exclude_unset=True) config: TextRunConfig = body_dict["config"] # Need `openai/` provider prefix for LiteLLM config.llm_user = "openai/dummy user model" config.llm_args_user |= { "api_base": f"{get_server_url(self.config.user_model_server.name)}/v1", "api_key": "dummy api key", # pragma: allowlist secret } | self.config.user_llm_args extra_agent_args = {k: v for k, v in responses_create_params.items() if k in ("temperature", "top_p")} if responses_create_params.get("max_output_tokens"): # Convert to chat completions extra_agent_args["max_tokens"] = responses_create_params["max_output_tokens"] # Need `openai/` provider prefix for LiteLLM config.llm_agent = "openai/dummy agent model" config.llm_args_agent = { "api_base": f"{get_server_url(self.config.model_server.name)}/v1", "api_key": "dummy api key", # pragma: allowlist secret } | extra_agent_args config.max_steps = self.config.max_steps result = await run_single_task(**body_dict) messages_to_convert = [] for message in result.messages: if message.role == "user" and message.tool_calls: continue elif message.role == "tool" and message.requestor == "user": continue messages_to_convert.append(message) message_dicts = to_litellm_messages(messages_to_convert) converter = VLLMConverter(return_token_id_information=True) all_items = converter.chat_completions_messages_to_responses_items(message_dicts) input_items_1, output_items = split_responses_input_output_items(all_items) # Tau starts trajectories with an assistant message Gym/responses_api_agents/tau2/app.py
Lines 197 to 215 in ff6f8d2
return Tau2VerifyResponse( **body_dict, responses_create_params=dict( input=body.responses_create_params.input + input_items_1 + input_items_2, model=body.responses_create_params.model or "", parallel_tool_calls=body.responses_create_params.parallel_tool_calls, tool_choice=body.responses_create_params.tool_choice, tools=body.responses_create_params.tools, ), response=dict( id=f"tau2-{body.config.domain}-{body.task.id}", created_at=int(time()), object="response", output=output_items, model=body.responses_create_params.model or "", parallel_tool_calls=body.responses_create_params.parallel_tool_calls, tool_choice=body.responses_create_params.tool_choice, tools=body.responses_create_params.tools, ), - Add PinchBench agentic benchmark integration #1810
- https://github.com/hkumar92/Gym/blob/5886ac8899ac4403fd087c197492bb56eff226a6/responses_api_agents/pinchbench/app.py#L331-L425
- https://github.com/hkumar92/Gym/blob/5886ac8899ac4403fd087c197492bb56eff226a6/responses_api_agents/pinchbench/app.py#L469-L510
Design notes
Suggested direction:
- Reuse/extend PR feat(observability): add per-rollout model-call capture #1715 where possible.
- Keep response/token stats in one standard schema.
- Add a standard place for reconstructable input/output history.
- Add a standard tool observation schema for tool output plus execution metadata.
- Let Gym choose the final schema: ATIF, Responses-plus-telemetry, or Gym-native with exporters.
Out of scope
ATIF also describes broader trajectory capabilities that may be useful later, but they are not required for this issue. This issue is scoped to the minimum logging/telemetry needed above.
Out of scope for now:
- Subagent trajectory trees / delegation graphs. ATIF has fields for subagents and subtrajectory references, but this is not part of the current ask: https://github.com/harbor-framework/harbor/blob/main/rfcs/0001-trajectory-format.md#L293-L296
- Requiring Gym to adopt ATIF as the native format. ATIF is a design reference here; Gym can choose ATIF, Responses-plus-telemetry, or a Gym-native schema with exporters.
- Full env/system-step observability beyond these requirements. ATIF covers richer system/context events such as context compaction, pruning, injection, reset/checkpoint-style events, and observation metadata; those are useful for debugging but not required here: https://github.com/harbor-framework/harbor/blob/main/rfcs/0001-trajectory-format.md#L720-L730, https://github.com/harbor-framework/harbor/blob/main/rfcs/0001-trajectory-format.md#L353-L355
Acceptance Criteria
- Standard schema for per-model-call token stats and response metadata.
- Token stats include prompt, completion, reasoning, total, and cached tokens when available.
- Standard per-turn fields include task identity, rollout identity, turn number, timestamp, question, answer, reasoning content, resolution status, and step count when available.
- Rollout records can reconstruct model-visible input/output history.
- Tool-call observations include output, status, start/end timestamps, and duration.
- Parallel tool calls can be represented with independent timing.
- Works for resource-server-backed and custom/sandbox-backed benchmarks.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsDone
Activity
piojanu commentedon Jul 1, 2026
Thanks, Anna. I want to emphasise a few requirements we should tighten:
Users should retrieve the complete rollout from the recorded trajectory itself, without assembling separate model-server, harness, sandbox, and verifier artifacts. It should include:
Every in-scope event should be recorded with no silent omissions. Incomplete trajectories should fail validation.
The logs should identify:
call_id.The acceptance criteria should define the canonical identifiers and where each is recorded.
We should:
I want to emphasise, enough semantics to reconstruct exactly what the next model call saw after compaction is a P0 requirement. Requirement 2 references ATIF’s context-boundary/compaction convention, but compaction is later listed as out of scope. If compaction can always be recovered from recorded inputs and outputs, we should document and test that guarantee. Otherwise, we need an explicit compaction event or boundary.
Storage efficiency also matters. The [OpenAI Responses API supports state chaining](https://developers.openai.com/api/docs/guides/conversation-state), but Gym does not appear to use it consistently across harnesses. Where stateless harnesses resend the complete conversation for every model call, recording every full request duplicates the growing history. We should require an efficient, fully reconstructable representation. [ATIF’s ordered steps and context-boundary convention](https://github.com/harbor-framework/harbor/blob/main/rfcs/0001-trajectory-format.md) are useful references.
Today, tool observations are captured per benchmark—either through resource servers or reconstructed by custom adapters—and timing would be added in the same places. We should explicitly minimize benchmark-specific work, define which parts are handled by shared Gym infrastructure, and reflect the remaining integration boundary in the acceptance criteria.
Failed, timed-out, cancelled, or retried model and tool calls—and sandbox errors, retries, and health signals—are essential for debugging. Some currently fall under out-of-scope environment/system events. I would include all of these from day one. ATIF-style system events provide a natural representation and illustrate one of the concrete pains ATIF solves.
Subagent trajectories and any remaining environment/system events may stay out of scope for this issue, but they are expected later. We should create or link follow-up issues and record their priority.
The priority is a complete, standardized trajectory contract that can be validated against explicit schemas—not requiring ATIF specifically. ATIF is a useful design reference, especially for external tool-execution telemetry, sandbox lifecycle and health, and verifier rewards, which the OpenAI Responses API does not standardize.
If adopting ATIF is not feasible, extending Gym’s current OpenAI Responses API–based format is fine, provided the resulting trajectory is complete, versioned, and schema-validatable.
When a harness request is converted before inference, it may be useful to label the recorded payload’s dialect or stage so consumers know whether it represents the original harness payload or the normalized model request. I would not treat this as a blocker as long as the payload is preserved.
sephmard commentedon Jul 1, 2026
Related to #1483
ffrujeri commentedon Jul 1, 2026
@piojanu, you mentioned one aspect of ATIF compared to the Responses contract about always dumping the full history of the conversation vs on the delta at each cycle. Could you clarify that better?
piojanu commentedon Jul 2, 2026
@ffrujeri By “delta,” I meant that ATIF represents the complete interaction history as an ordered
steps[]sequence. EachStepObjectrepresents a system prompt, user message, or complete agent turn—its tool calls and observations. Then, ATIF's context-management convention defines how consumers reconstruct the effective context. Including after compaction or other context transformations.In other words, the current Gym solution stores the complete message history in the model request captured at every step. ATIF instead stores only the current user message or agent turn. Consumers reconstruct the context from the ordered preceding steps and context boundaries.
Note: this is not exclusive to ATIF. The OpenAI Responses API supports state chaining through
previous_response_idor Conversations.14 remaining items
[rollout-observability][1/7] Define the rollout observation and corre…
qiaochuz-nv commentedon Aug 4, 2026
PR #1715 — Test Summary
Feature:
feat(observability): add per-rollout model-call captureSUT: NVIDIA-NeMo/Gym @
0b9192c(05eb7df= #1715, the root commit of this shallow clone)Suite:
nemo_llm/test_suite/nemo-gym— 4 testcases, 18 cases, 2 869 lines, ~6.7 min totalBackend: every model call hits a real endpoint — hosted
inference_provider(
nvidia/nvidia/Nemotron-3-Nano-30B-A3B) and a suite-managed real vLLM(
Qwen/Qwen3-0.6B, RTX 5090). No simulated model anywhere.1. What shipped
model_call_capturemodel_call_capture_agentsmodel_call_capture_agent_backendsmodel_call_capture_known_defectsThe first three are regression guards. The fourth asserts positively what the telemetry
should report, so each open defect is one failing assertion; it is registered
expected_to_fail: trueand carries NVBugs 6555630 / 6555639 / 6555643.model_call_capture— middleware and lifecycle contractGiven a prefixed URL, does the middleware classify, record, trim and attach correctly?
model_call_capture_agents— does any shipped harness produce that prefix?Each agent hands the URL to a client stack Gym does not control.
model_call_capture_agent_backends— the same contract across enginesmodel_call_capture_known_defects— the defect backlog, held visible2. Coverage structure
Propagation mechanism — 6/6 groups covered
#1715 touched 14 agents under
responses_api_agents/; 18 propagate the prefix at HEAD(4 were added by later PRs). They reduce to six mechanisms, and every one is now driven
by a real harness:
url_path_for_request+url_path_for_runsimple_agenturl_path_for_runaviary_agentrollout_id_from_run+apply_rollout_prefixclaude_code_agentresolve_model_base_url+rollout_id_from_runcodex_agentbase_url_for_runtau2resolve_model_base_url+url_path_for_runhermes_agentDialect — all three, each through a real harness
Dialects and mechanisms are not independent: each dialect is reachable from exactly one
harness family, so the two axes collapse.
responsessimple_agent,aviary,codexchathermes,tau2messagesclaude_codeDialects were measured, not assumed.
hermesemitschat(its/v1/responsescall isa self-call, not a model call; the model call comes from the third-party
AIAgent),codexemits streamingresponses,tau2emitschat.Other axes
inference_providerand a suite-managed real vLLM, same harness both ways.attachment-reading case. Four of those fields (
reasoning_content,cache_hit,cached_tokens,cache_creation_tokens) have no other assertion holding them in place.where a sum and a passthrough are indistinguishable.
tool_calls: forced deterministically withtool_choice: "required"+max_steps=1, so the structured tool call must survive derivation into the attachment.model_call_capture_no_recordsgap, default-off, and rollout-scopedfresh-run cleanup that spares unrelated capture files.
3. Product defects found
Two confirmed, both inside #1715's telemetry contract, both green upstream by construction.
A — Converted Responses usage fabricates provider-unknown cache and reasoning metrics
NVBug 6555630
InferenceProvider.responses()(responses_api_models/inference_provider/app.py:88-91) andResponsesConverter.chat_completion_to_response()(nemo_gym/responses_converter.py:494,496)both hard-code
cached_tokens=0/reasoning_tokens=0, and both discard the upstreamprompt_tokens_details/completion_tokens_detailsentirely.Measured across 123 captured records:
prompt_tokens_detailscompletion_tokens_detailschatnull×18null×18responses{"cached_tokens": 0}×93{"reasoning_tokens": 0}×93_cache_signal()then reads that synthesized0as a measurement and recordscache_hit: false— a reported miss, not "unknown". Cache-hit rate, the miss denominator,reasoning-token aggregation and cross-dialect comparison are all skewed.
ModelCallRecord's own comment states the rule being violated:"consumers must treat null as 'unknown', not 0."
Upstream
tests/unit_tests/test_responses_converter.py:559,561asserts the fabricated zerosas expected values, so the defect is green there by construction.
nemo_gym/anthropic_converter.py:812is not the same shape: it maps upstreamcache_read_input_tokenswhen present and only hard-codesreasoning_tokens.B — Two independent defects, not one
B1 — the chat→Responses conversion never sets top-level
status. NVBug 6555639The four
status="completed"literals inresponses_converter.pysit on output items, noton the response.
NeMoGymResponseinherits OpenAI'sResponse, whosestatusdefaults toNone, so nothing raises and the object serializes as"status": null— including whenincomplete_detailssays the reply was truncated. 93/93 captured Responses objects carrythe key with a null value.
B2 —
ModelCallRecorddrops Responses termination info. NVBug 6555643build_model_call_record()derivesfinish_reasononly fromstop_reason,choices[0].finish_reasonandincomplete_details.reason— it never readsstatus.Fixing B1 alone does not populate it; upstream pins this with the parametrised case
({"status": "completed"}, None).finish_reasonchat'length'×8,'tool_calls'×9,'stop'×1 — 18/18 populatedmessages'max_tokens'×8,'end_turn'×4 — 12/12 populatedresponses'max_output_tokens'×53,None×40The hole sits exactly at the most common outcome, and only for one dialect. The assertion
accepts either a populated
finish_reasonor a preserved response status, anddeliberately does not demand
completed → "stop": a Responses reply can bestatus: completedwhile its output is afunction_call, and flattening that would destroythe tool-call semantics.
Withdrawn — capture durability
A fifth assertion (capture on disk before the response returns) was drafted and retracted.
The middleware provably orders
_parse_and_recordbefore_flush_deferred_response, and arepeated measurement found the file already on disk at return in 19 of 20 calls. The one
exception was the first call on a freshly started server, lagging 0.56 ms — warmup, not a
defect. The retraction is recorded in the registry note so the brief wait in two cases is not
later misread as papering over a race.
Not a defect — ruled out with evidence
A 500 on the turn carrying a tool result was traced with a recording proxy placed between the
model server and the provider. Gym's upstream request is well-formed OpenAI Chat with
correctly paired
tool_calls[0].id/tool_call_id, and the sametool_choice: "required"succeeds on the first turn. Provider-side.
4. Not covered
stream:true)incomplete/unreadable/records_unreadableno_recordsis exercised-a{n}(rollout_correlation.py:47-50)ToolCallObservation/ parallel tool timingclaude_code_agentemitsng_agent_observations; the resources-server agents named in the requirements emit nonebrowsecomp_agenttavily_api_keyis required and raises at config validation; live web makes reward irreproducibleanyterminal_agentprepare.pyneeds the Harbor CLI, and the committedtask_dirpoints at a Harbor cache path. Docker is not the blockermini_swe_agent_2,stirrup_agent,remote_agent5. Notable decisions
Model names are bound per case, not read from credentials.
config/qa_credentials.yamlnow carries onlybase_url+api_key. Everygym_environment(model=...)names its own model and the command line carries it as aliteral, so a run's own trace records which model produced its assertions — previously the
trace held only an
${oc.env:...}reference and editing one shared value silently retargetedevery case. 28 call sites across 12 files migrated and re-verified.
Assertions target the observability contract, not model output.
Against a real backend, response text, reward, turn count and token counts are backend
results rather than regression constants. The cases assert that every call the harness made
is on disk, that each attachment agrees with its durable record call-for-call, and that the
aggregate equals the sum of the parts.
No assertion was relaxed to accommodate a defect.
The three green testcases pass because they happen not to touch A or B — not because they
were loosened.
model_call_capture.pyeven reproduces one on purpose: it derives its expectedfinish_reasonandtokens_reasoningfrom the raw response with the same logic the productuses, so it mirrors the bug instead of catching it. That is exactly why the RED file exists.
6. Reproduction
Prerequisites:
inference_providercredentials inconfig/qa_credentials.yaml(
base_url+api_keyonly); preinstalled Claude Code 2.1.220 and Codex 0.104.0 CLIs — thecases assert their versions and never auto-install npm packages; a GPU plus the
local_vllm_modelvenv for the vLLM case, which fails with an actionable message rather thanskipping when either is missing.
Minimal defect A/B reproduction without the suite:
[fix] Standardize rollout trajectory records and document producer co…
[rollout-observability][1/7] Define the rollout observation and corre…
[fix] Standardize rollout trajectory records and document producer co…