Skip to content

[epic] standardised trajectories format #1867

Closed
9 of 9 issues completed
Closed

Description

@AWarno

NOTE Up to date requirements are being tracked in the google doc: https://docs.google.com/document/d/1z0wLyl6lNpLhLCqd33EH04RdcIl4UTy08_ROEnNRnUA/edit?tab=t.3xnkyvgfyztw

This issue is outdated.

Standardized trajectory logging requirements

Use cases, pain points, and background

We need standardized trajectory/log data for cost estimation, debugging, and rollout analysis across Gym benchmarks.

This issue is not a request to make ATIF the native Gym format. The request is to define the telemetry Gym should expose. ATIF is useful as a design reference for ordered steps, efficient history reconstruction, observations, metrics, and context boundaries:

Requirements

1. Standard per-model-call stats

PR #1715 already addresses part of this with model-server trajectory capture and StepRecord stats: #1715

The standardized Gym contract should expose, when available:

  • task/rollout identity: task_id, trial_name / rollout id
  • turn identity: turn_no, num_turns, timestamp
  • response structure: model tool calls, reasoning_content
  • token stats: prompt, completion, reasoning, total, cached tokens

2. Reconstructable model input/output history

We need enough data to reconstruct the model-visible request/response pairs for a rollout (also covered by: #1715):

This should preferably be efficient. For multi-turn agents, repeatedly storing the full re-materialized prompt on every turn grows quickly. A delta/append-only representation is preferred.

ATIF has relevant design references:

Gym can use ATIF or a Gym-native equivalent, but we need consistent reconstruction semantics.

3. Tool-call observations and timing

Gym already has model-visible tool output via function_call_output:

{
  "type": "function_call_output",
  "call_id": "call_123",
  "output": "tool result string",
  "status": "completed"
}

Schema:

class NeMoGymFunctionCallOutput(BaseModel):
"""
We copy openai.types.responses.response_input_param.FunctionCallOutput, originally a TypedDict, as a BaseModel here
so that we can use it in the NeMoGymResponseOutputItem below and be consistent with the other ResponseOutputItem types.
"""
call_id: str
output: str
type: Literal["function_call_output"] = "function_call_output"
id: Optional[str] = None
status: Optional[Literal["in_progress", "completed", "incomplete"]] = None

We also need logging-only execution metadata:

  • call_id
  • model-visible tool output
  • status
  • started_at
  • completed_at
  • duration_ms
  • CPU/sandbox timing stats, when available
  • error/timeout metadata
  • sandbox/container metadata, when available

Example shape, based on ATIF observations:

{
  "observation": {
    "results": [
      {
        "source_call_id": "functions.terminal:0",
        "content": "...tool output visible to model...",
        "extra": {
          "started_at": "2026-06-25T11:37:28.156623",
          "completed_at": "2026-06-25T11:37:28.688924",
          "duration_ms": 532.301,
          "status": "completed"
        }
      }
    ]
  }
}

ATIF references:

For parallel tool calls, timing cannot be approximated from time between turns. We need independent start/end timestamps from the actual tool execution boundary.

Existing Gym patterns to support

The solution should work for both:

  • Resource-server-backed tools/envs, e.g. simple_agent, BrowseComp, Finance, CVDP, Gymnasium:

    • all_fn_calls: List[NeMoGymResponseFunctionToolCall] = [o for o in output if o.type == "function_call"]
      all_output_messages: List[NeMoGymResponseOutputMessage] = [
      o for o in output if o.type == "message" and o.role == "assistant"
      ]
      if not all_fn_calls and all_output_messages:
      break
      for output_function_call in all_fn_calls:
      try:
      parsed_arguments = json.loads(output_function_call.arguments)
      except (json.JSONDecodeError, TypeError) as e:
      # Model produced malformed tool-call arguments. Surface the
      # error back as a tool response so the rollout can continue
      # (or terminate with a low reward) instead of crashing the
      # whole batch on json.loads.
      tool_response = NeMoGymFunctionCallOutput(
      type="function_call_output",
      call_id=output_function_call.call_id,
      # Use repr(e) so the exception type name is always
      # included even when str(e) would be empty.
      output=json.dumps({"error": f"Invalid tool call arguments: {e!r}"}),
      )
      new_outputs.append(tool_response)
      continue
      api_response = await self.server_client.post(
      server_name=self.config.resources_server.name,
      url_path=f"/{output_function_call.name}",
      json=parsed_arguments,
      cookies=resources_server_cookies,
      )
      # We don't raise for status here since it's a valid return for the API to error e.g. if the model outputs an invalid call or something.
      resources_server_cookies = api_response.cookies
      tool_response = NeMoGymFunctionCallOutput(
      type="function_call_output",
      call_id=output_function_call.call_id,
      output=(await api_response.content.read()).decode(),
      )
      new_outputs.append(tool_response)
    • for output_function_call in all_fn_calls:
      num_tool_calls += 1
      log_event("tool_call_begin", name=output_function_call.name, args=output_function_call.arguments)
      tool_start = time.monotonic()
      api_response = await self.server_client.post(
      server_name=self.config.resources_server.name,
      url_path=f"/{output_function_call.name}",
      json=json.loads(output_function_call.arguments),
      cookies=resources_server_cookies,
      )
      # We don't raise for status here since it's a valid return for the API to error e.g. if the model outputs an invalid call or something.
      resources_server_cookies = api_response.cookies
      tool_output = (await api_response.content.read()).decode()
      tool_dur = time.monotonic() - tool_start
      tool_total_dur += tool_dur
      log_event(
      "tool_call",
      duration_s=round(tool_dur, 3),
      name=output_function_call.name,
      args=output_function_call.arguments,
      output=tool_output,
      output_len=len(tool_output),
      )
      if self.config.nudge_steps:
      turns_left = self.config.max_steps - step
      tool_output += "\n\n[%d turns remaining out of %d]" % (turns_left, self.config.max_steps)
      tool_response = NeMoGymFunctionCallOutput(
      type="function_call_output",
      call_id=output_function_call.call_id,
      output=tool_output,
      )
      new_outputs.append(tool_response)
    • done = False
      for output_function_call in all_fn_calls:
      try:
      coro = self.server_client.post(
      server_name=self.config.resources_server.name,
      url_path=f"/{output_function_call.name}",
      json=json.loads(output_function_call.arguments),
      cookies=resources_server_cookies,
      )
      api_response = await asyncio.wait_for(coro, timeout=self.config.tool_call_timeout)
      resources_server_cookies = api_response.cookies
      tool_output = (await api_response.content.read()).decode()
      except asyncio.TimeoutError:
      logger.warning(
      "Tool call '%s' timed out after %ss",
      output_function_call.name,
      self.config.tool_call_timeout,
      )
      tool_output = json.dumps(
      {
      "error": f"Tool call timed out after {self.config.tool_call_timeout}s. "
      "Please try a different approach or submit your final answer."
      }
      )
      except Exception as e:
      logger.error(
      "Tool call '%s' failed: %s: %s",
      output_function_call.name,
      type(e).__name__,
      e,
      )
      tool_output = json.dumps(
      {
      "error": f"Tool call failed: {type(e).__name__}: {e}. "
      "Please try a different approach or submit your final answer."
      }
      )
      tool_response = NeMoGymFunctionCallOutput(
      type="function_call_output",
      call_id=output_function_call.call_id,
      output=tool_output,
      )
      new_outputs.append(tool_response)
    • for output_function_call in all_fn_calls:
      api_response = await self.server_client.post(
      server_name=self.config.resources_server.name,
      url_path=f"/{output_function_call.name}",
      json=json.loads(output_function_call.arguments),
      cookies=resources_server_cookies,
      )
      # We don't raise for status here since it's a valid return for the API to error e.g. if the model outputs an invalid call or something.
      resources_server_cookies = api_response.cookies
      tool_response = NeMoGymFunctionCallOutput(
      type="function_call_output",
      call_id=output_function_call.call_id,
      output=(await api_response.content.read()).decode(),
      )
      new_outputs.append(tool_response)
    • step_resp = await self.server_client.post(
      server_name=self.config.resources_server.name,
      url_path="/step",
      json=body.model_dump() | {"response": model_response.model_dump()},
      cookies=env_cookies,
      )
      await raise_for_status(step_resp)
      step_data = EnvStepResponse.model_validate(await get_response_json(step_resp))
      total_reward += step_data.reward
      env_cookies = step_resp.cookies
      if step_data.terminated or step_data.truncated:
      finished = True
      break
      for tool_output in (step_data.info or {}).get("tool_outputs", []):
      new_outputs.append(
      NeMoGymFunctionCallOutput(
      type="function_call_output",
      call_id=tool_output["call_id"],
      output=tool_output["output"],
      )
      )
  • Custom/sandbox-backed adapters, e.g. Tau2 and PinchBench:

    • async def run(self, body: Tau2RunRequest) -> Tau2VerifyResponse:
      body_dict = {name: getattr(body, name) for name in Tau2RunRequest.model_fields}
      responses_create_params = body_dict.pop("responses_create_params").model_dump(exclude_unset=True)
      config: TextRunConfig = body_dict["config"]
      # Need `openai/` provider prefix for LiteLLM
      config.llm_user = "openai/dummy user model"
      config.llm_args_user |= {
      "api_base": f"{get_server_url(self.config.user_model_server.name)}/v1",
      "api_key": "dummy api key", # pragma: allowlist secret
      } | self.config.user_llm_args
      extra_agent_args = {k: v for k, v in responses_create_params.items() if k in ("temperature", "top_p")}
      if responses_create_params.get("max_output_tokens"):
      # Convert to chat completions
      extra_agent_args["max_tokens"] = responses_create_params["max_output_tokens"]
      # Need `openai/` provider prefix for LiteLLM
      config.llm_agent = "openai/dummy agent model"
      config.llm_args_agent = {
      "api_base": f"{get_server_url(self.config.model_server.name)}/v1",
      "api_key": "dummy api key", # pragma: allowlist secret
      } | extra_agent_args
      config.max_steps = self.config.max_steps
      result = await run_single_task(**body_dict)
      messages_to_convert = []
      for message in result.messages:
      if message.role == "user" and message.tool_calls:
      continue
      elif message.role == "tool" and message.requestor == "user":
      continue
      messages_to_convert.append(message)
      message_dicts = to_litellm_messages(messages_to_convert)
      converter = VLLMConverter(return_token_id_information=True)
      all_items = converter.chat_completions_messages_to_responses_items(message_dicts)
      input_items_1, output_items = split_responses_input_output_items(all_items)
      # Tau starts trajectories with an assistant message
    • return Tau2VerifyResponse(
      **body_dict,
      responses_create_params=dict(
      input=body.responses_create_params.input + input_items_1 + input_items_2,
      model=body.responses_create_params.model or "",
      parallel_tool_calls=body.responses_create_params.parallel_tool_calls,
      tool_choice=body.responses_create_params.tool_choice,
      tools=body.responses_create_params.tools,
      ),
      response=dict(
      id=f"tau2-{body.config.domain}-{body.task.id}",
      created_at=int(time()),
      object="response",
      output=output_items,
      model=body.responses_create_params.model or "",
      parallel_tool_calls=body.responses_create_params.parallel_tool_calls,
      tool_choice=body.responses_create_params.tool_choice,
      tools=body.responses_create_params.tools,
      ),
    • Add PinchBench agentic benchmark integration #1810
    • https://github.com/hkumar92/Gym/blob/5886ac8899ac4403fd087c197492bb56eff226a6/responses_api_agents/pinchbench/app.py#L331-L425
    • https://github.com/hkumar92/Gym/blob/5886ac8899ac4403fd087c197492bb56eff226a6/responses_api_agents/pinchbench/app.py#L469-L510

Design notes

Suggested direction:

  • Reuse/extend PR feat(observability): add per-rollout model-call capture #1715 where possible.
  • Keep response/token stats in one standard schema.
  • Add a standard place for reconstructable input/output history.
  • Add a standard tool observation schema for tool output plus execution metadata.
  • Let Gym choose the final schema: ATIF, Responses-plus-telemetry, or Gym-native with exporters.

Out of scope

ATIF also describes broader trajectory capabilities that may be useful later, but they are not required for this issue. This issue is scoped to the minimum logging/telemetry needed above.

Out of scope for now:

Acceptance Criteria

  • Standard schema for per-model-call token stats and response metadata.
  • Token stats include prompt, completion, reasoning, total, and cached tokens when available.
  • Standard per-turn fields include task identity, rollout identity, turn number, timestamp, question, answer, reasoning content, resolution status, and step count when available.
  • Rollout records can reconstruct model-visible input/output history.
  • Tool-call observations include output, status, start/end timestamps, and duration.
  • Parallel tool calls can be represented with independent timing.
  • Works for resource-server-backed and custom/sandbox-backed benchmarks.

Activity

  1. piojanu commented on Jul 1, 2026

    @piojanu

    Thanks, Anna. I want to emphasise a few requirements we should tighten:

    1. Complete recorded trajectory

    Users should retrieve the complete rollout from the recorded trajectory itself, without assembling separate model-server, harness, sandbox, and verifier artifacts. It should include:

    • model inputs, outputs, and statistics,
    • tool calls, observations, timings, and failures,
    • sandbox/system events and health signals,
    • final verifier outcome and reward.

    Every in-scope event should be recorded with no silent omissions. Incomplete trajectories should fail validation.

    1. Canonical identifiers

    The logs should identify:

    • benchmark and version,
    • task/problem,
    • session/run,
    • agent,
    • repeat/trial,
    • turn/model call,
    • tool call through its call_id.

    The acceptance criteria should define the canonical identifiers and where each is recorded.

    1. Mandatory fields and validation

    We should:

    • distinguish mandatory from optional fields,
    • define how unavailable optional fields are represented,
    • version the trajectory schema,
    • validate new benchmarks and harnesses automatically against that schema.
    1. Compaction and efficient history

    I want to emphasise, enough semantics to reconstruct exactly what the next model call saw after compaction is a P0 requirement. Requirement 2 references ATIF’s context-boundary/compaction convention, but compaction is later listed as out of scope. If compaction can always be recovered from recorded inputs and outputs, we should document and test that guarantee. Otherwise, we need an explicit compaction event or boundary.

    Storage efficiency also matters. The [OpenAI Responses API supports state chaining](https://developers.openai.com/api/docs/guides/conversation-state), but Gym does not appear to use it consistently across harnesses. Where stateless harnesses resend the complete conversation for every model call, recording every full request duplicates the growing history. We should require an efficient, fully reconstructable representation. [ATIF’s ordered steps and context-boundary convention](https://github.com/harbor-framework/harbor/blob/main/rfcs/0001-trajectory-format.md) are useful references.

    1. Minimize per-benchmark work

    Today, tool observations are captured per benchmark—either through resource servers or reconstructed by custom adapters—and timing would be added in the same places. We should explicitly minimize benchmark-specific work, define which parts are handled by shared Gym infrastructure, and reflect the remaining integration boundary in the acceptance criteria.

    1. Failures, retries, and system health

    Failed, timed-out, cancelled, or retried model and tool calls—and sandbox errors, retries, and health signals—are essential for debugging. Some currently fall under out-of-scope environment/system events. I would include all of these from day one. ATIF-style system events provide a natural representation and illustrate one of the concrete pains ATIF solves.

    Subagent trajectories and any remaining environment/system events may stay out of scope for this issue, but they are expected later. We should create or link follow-up issues and record their priority.

    1. Format choice

    The priority is a complete, standardized trajectory contract that can be validated against explicit schemas—not requiring ATIF specifically. ATIF is a useful design reference, especially for external tool-execution telemetry, sandbox lifecycle and health, and verifier rewards, which the OpenAI Responses API does not standardize.

    If adopting ATIF is not feasible, extending Gym’s current OpenAI Responses API–based format is fine, provided the resulting trajectory is complete, versioned, and schema-validatable.

    1. Food for thought

    When a harness request is converted before inference, it may be useful to label the recorded payload’s dialect or stage so consumers know whether it represents the original harness payload or the normalized model request. I would not treat this as a blocker as long as the payload is preserved.

  2. sephmard commented on Jul 1, 2026

    @sephmard
    Contributor

    Related to #1483

  3. ffrujeri commented on Jul 1, 2026

    @ffrujeri
    Contributor

    @piojanu, you mentioned one aspect of ATIF compared to the Responses contract about always dumping the full history of the conversation vs on the delta at each cycle. Could you clarify that better?

  4. deleted a comment from linear-code on Jul 2, 2026
  5. deleted a comment from linear-code on Jul 2, 2026
  6. added
    r0.5.0Auto-cherrypick to release branch. Apply before merge; cherrypick happens after merge.
    on Jul 2, 2026
  7. piojanu commented on Jul 2, 2026

    @piojanu

    @ffrujeri By “delta,” I meant that ATIF represents the complete interaction history as an ordered steps[] sequence. Each StepObject represents a system prompt, user message, or complete agent turn—its tool calls and observations. Then, ATIF's context-management convention defines how consumers reconstruct the effective context. Including after compaction or other context transformations.

    In other words, the current Gym solution stores the complete message history in the model request captured at every step. ATIF instead stores only the current user message or agent turn. Consumers reconstruct the context from the ordered preceding steps and context boundaries.

    Note: this is not exclusive to ATIF. The OpenAI Responses API supports state chaining through previous_response_id or Conversations.

  8. 14 remaining items

  9. moved this from Dev WIP to QA Todo in NeMo Gym 0.5.0 - Aug 5on Jul 29, 2026
  10. moved this from QA Todo to QA WIP in NeMo Gym 0.5.0 - Aug 5on Jul 29, 2026
  11. moved this from QA WIP to QA Verified in NeMo Gym 0.5.0 - Aug 5on Aug 4, 2026
  12. qiaochuz-nv commented on Aug 4, 2026

    @qiaochuz-nv
    Contributor

    PR #1715 — Test Summary

    Feature: feat(observability): add per-rollout model-call capture
    SUT: NVIDIA-NeMo/Gym @ 0b9192c (05eb7df = #1715, the root commit of this shallow clone)
    Suite: nemo_llm/test_suite/nemo-gym — 4 testcases, 18 cases, 2 869 lines, ~6.7 min total
    Backend: every model call hits a real endpoint — hosted inference_provider
    (nvidia/nvidia/Nemotron-3-Nano-30B-A3B) and a suite-managed real vLLM
    (Qwen/Qwen3-0.6B, RTX 5090). No simulated model anywhere.

    Scope note: #1715 is the model-server-boundary piece of a 7-part rollout-observability
    epic. #2114 (1/7) and #2153 (2/7) have since landed and modify the same files, so this
    coverage tracks the deployed contract rather than one PR's diff.


    1. What shipped

    testcase tier cases GPU runtime result
    model_call_capture function 7 0 83.6 s 7/7 pass
    model_call_capture_agents e2e 5 0 221.8 s 5/5 pass
    model_call_capture_agent_backends e2e 2 1 52.8 s 2/2 pass
    model_call_capture_known_defects function 4 0 46.1 s 0/4 — deliberately RED

    The first three are regression guards. The fourth asserts positively what the telemetry
    should report, so each open defect is one failing assertion; it is registered
    expected_to_fail: true and carries NVBugs 6555630 / 6555639 / 6555643.

    model_call_capture — middleware and lifecycle contract

    Given a prefixed URL, does the middleware classify, record, trim and attach correctly?

    1. prefixed Responses, Chat, and Messages requests are captured in append order
    2. Responses SSE is reconstructed into a complete durable capture
    3. an unprefixed request after a prefixed request is routed but not captured
    4. eval run attaches one trimmed capture record and exact metrics to every rollout
    5. a tool-calling rollout captures structured tool calls on every model call
    6. rerunning every rollout identity replaces stale captures without deleting an unrelated capture
    7. capture stays off by default even when a capture directory is configured

    model_call_capture_agents — does any shipped harness produce that prefix?

    Each agent hands the URL to a client stack Gym does not control.

    1. hermes propagates the rollout prefix onto Chat Completions
    2. Claude Code propagates the rollout prefix onto Messages and joins its own observations
    3. codex propagates the rollout prefix onto streaming Responses
    4. tau2 keeps every call of a multi-call rollout correlated and aggregated
    5. Claude Code without a Gym model server skips the prefix and reports the absence as a gap

    model_call_capture_agent_backends — the same contract across engines

    1. aviary on the inference provider captures every call in agreement with the attachment
    2. aviary on a real vLLM captures every call it made, in agreement with the attachment

    model_call_capture_known_defects — the defect backlog, held visible

    1. converted Responses usage does not invent cache or reasoning detail the provider never reported
    2. capture records an unreported cache state as unknown rather than as a reported miss
    3. a Responses reply reports its own completion status
    4. the capture record says how a completed Responses call terminated

    2. Coverage structure

    Propagation mechanism — 6/6 groups covered

    #1715 touched 14 agents under responses_api_agents/; 18 propagate the prefix at HEAD
    (4 were added by later PRs). They reduce to six mechanisms, and every one is now driven
    by a real harness:

    mechanism agents in group driven by
    url_path_for_request + url_path_for_run 8 simple_agent
    url_path_for_run 3 aviary_agent
    rollout_id_from_run + apply_rollout_prefix 2 claude_code_agent
    resolve_model_base_url + rollout_id_from_run 2 codex_agent
    base_url_for_run 2 tau2
    resolve_model_base_url + url_path_for_run 1 hermes_agent

    Dialect — all three, each through a real harness

    Dialects and mechanisms are not independent: each dialect is reachable from exactly one
    harness family, so the two axes collapse.

    dialect direct HTTP real harness streaming reconstruction
    responses yes simple_agent, aviary, codex yes — Responses SSE via the Codex CLI
    chat yes hermes, tau2 no — chat SSE (#2130) uncovered
    messages yes claude_code yes — Anthropic SSE via the Claude Code CLI

    Dialects were measured, not assumed. hermes emits chat (its /v1/responses call is
    a self-call, not a model call; the model call comes from the third-party AIAgent),
    codex emits streaming responses, tau2 emits chat.

    Other axes

    • Backends: hosted inference_provider and a suite-managed real vLLM, same harness both ways.
    • Record shape: the 22-field attachment call set is pinned as a whole in every
      attachment-reading case. Four of those fields (reasoning_content, cache_hit,
      cached_tokens, cache_creation_tokens) have no other assertion holding them in place.
    • Multi-call aggregation: asserted at n = 4 via tau2. Every other rollout reaches n = 1,
      where a sum and a passthrough are indistinguishable.
    • Non-empty tool_calls: forced deterministically with tool_choice: "required" +
      max_steps=1, so the structured tool call must survive derivation into the attachment.
    • Plus: the explicit model_call_capture_no_records gap, default-off, and rollout-scoped
      fresh-run cleanup that spares unrelated capture files.

    3. Product defects found

    Two confirmed, both inside #1715's telemetry contract, both green upstream by construction.

    A — Converted Responses usage fabricates provider-unknown cache and reasoning metrics

    NVBug 6555630

    InferenceProvider.responses() (responses_api_models/inference_provider/app.py:88-91) and
    ResponsesConverter.chat_completion_to_response() (nemo_gym/responses_converter.py:494,496)
    both hard-code cached_tokens=0 / reasoning_tokens=0, and both discard the upstream
    prompt_tokens_details / completion_tokens_details entirely.

    Measured across 123 captured records:

    dialect path prompt_tokens_details completion_tokens_details
    chat pass-through — the provider's own numbers null ×18 null ×18
    responses Gym converts chat → responses {"cached_tokens": 0} ×93 {"reasoning_tokens": 0} ×93

    _cache_signal() then reads that synthesized 0 as a measurement and records
    cache_hit: false — a reported miss, not "unknown". Cache-hit rate, the miss denominator,
    reasoning-token aggregation and cross-dialect comparison are all skewed.
    ModelCallRecord's own comment states the rule being violated:
    "consumers must treat null as 'unknown', not 0."

    Upstream tests/unit_tests/test_responses_converter.py:559,561 asserts the fabricated zeros
    as expected values, so the defect is green there by construction.

    nemo_gym/anthropic_converter.py:812 is not the same shape: it maps upstream
    cache_read_input_tokens when present and only hard-codes reasoning_tokens.

    B — Two independent defects, not one

    B1 — the chat→Responses conversion never sets top-level status. NVBug 6555639
    The four status="completed" literals in responses_converter.py sit on output items, not
    on the response. NeMoGymResponse inherits OpenAI's Response, whose status defaults to
    None, so nothing raises and the object serializes as "status": null — including when
    incomplete_details says the reply was truncated. 93/93 captured Responses objects carry
    the key with a null value.

    B2 — ModelCallRecord drops Responses termination info. NVBug 6555643
    build_model_call_record() derives finish_reason only from stop_reason,
    choices[0].finish_reason and incomplete_details.reason — it never reads status.
    Fixing B1 alone does not populate it; upstream pins this with the parametrised case
    ({"status": "completed"}, None).

    dialect derived finish_reason
    chat 'length' ×8, 'tool_calls' ×9, 'stop' ×1 — 18/18 populated
    messages 'max_tokens' ×8, 'end_turn' ×4 — 12/12 populated
    responses 'max_output_tokens' ×53, None ×40

    The hole sits exactly at the most common outcome, and only for one dialect. The assertion
    accepts either a populated finish_reason or a preserved response status, and
    deliberately does not demand completed → "stop": a Responses reply can be
    status: completed while its output is a function_call, and flattening that would destroy
    the tool-call semantics.

    Withdrawn — capture durability

    A fifth assertion (capture on disk before the response returns) was drafted and retracted.
    The middleware provably orders _parse_and_record before _flush_deferred_response, and a
    repeated measurement found the file already on disk at return in 19 of 20 calls. The one
    exception was the first call on a freshly started server, lagging 0.56 ms — warmup, not a
    defect. The retraction is recorded in the registry note so the brief wait in two cases is not
    later misread as papering over a race.

    Not a defect — ruled out with evidence

    A 500 on the turn carrying a tool result was traced with a recording proxy placed between the
    model server and the provider. Gym's upstream request is well-formed OpenAI Chat with
    correctly paired tool_calls[0].id / tool_call_id, and the same tool_choice: "required"
    succeeds on the first turn. Provider-side.


    4. Not covered

    gap reason
    chat SSE (#2130 synthesized stream:true) no shipped harness produces it; a direct-HTTP assertion is cheap and outstanding
    gap codes incomplete / unreadable / records_unreadable only no_records is exercised
    retry attempt suffix -a{n} (rollout_correlation.py:47-50) requires forcing a retry
    ToolCallObservation / parallel tool timing blocked on SUT — only claude_code_agent emits ng_agent_observations; the resources-server agents named in the requirements emit none
    browsecomp_agent blocked — tavily_api_key is required and raises at config validation; live web makes reward irreproducible
    anyterminal_agent blocked — prepare.py needs the Harbor CLI, and the committed task_dir points at a Harbor cache path. Docker is not the blocker
    mini_swe_agent_2, stirrup_agent, remote_agent mechanism-redundant with tau2 / codex / aviary

    5. Notable decisions

    Model names are bound per case, not read from credentials.
    config/qa_credentials.yaml now carries only base_url + api_key. Every
    gym_environment(model=...) names its own model and the command line carries it as a
    literal, so a run's own trace records which model produced its assertions — previously the
    trace held only an ${oc.env:...} reference and editing one shared value silently retargeted
    every case. 28 call sites across 12 files migrated and re-verified.

    Assertions target the observability contract, not model output.
    Against a real backend, response text, reward, turn count and token counts are backend
    results rather than regression constants. The cases assert that every call the harness made
    is on disk, that each attachment agrees with its durable record call-for-call, and that the
    aggregate equals the sum of the parts.

    No assertion was relaxed to accommodate a defect.
    The three green testcases pass because they happen not to touch A or B — not because they
    were loosened. model_call_capture.py even reproduces one on purpose: it derives its expected
    finish_reason and tokens_reasoning from the raw response with the same logic the product
    uses, so it mirrors the bug instead of catching it. That is exactly why the RED file exists.


    6. Reproduction

    export NEMO_GYM_REPO=~/nemo/Gym
    cd nemo_llm/test_suite/nemo-gym
    
    python3 testcases/responses_api_models/model_call_capture.py               # 7/7 pass
    python3 testcases/responses_api_agents/model_call_capture_agents.py        # 5/5 pass
    python3 testcases/responses_api_agents/model_call_capture_agent_backends.py # 2/2 pass, needs a GPU
    python3 testcases/responses_api_models/model_call_capture_known_defects.py # 0/4, exit 1 — expected
    
    # one case at a time
    NEMO_GYM_QA_CASE='tool-calling' python3 testcases/responses_api_models/model_call_capture.py
    # keep the workspace for post-mortem
    NEMO_GYM_KEEP_WS=1 python3 testcases/responses_api_models/model_call_capture_known_defects.py

    Prerequisites: inference_provider credentials in config/qa_credentials.yaml
    (base_url + api_key only); preinstalled Claude Code 2.1.220 and Codex 0.104.0 CLIs — the
    cases assert their versions and never auto-install npm packages; a GPU plus the
    local_vllm_model venv for the vLLM case, which fails with an actionable message rather than
    skipping when either is missing.

    Minimal defect A/B reproduction without the suite:

    gym env start --resources-server mcqa --model-type inference_provider
    # Chat is a pass-through -> the provider's own usage
    curl -s $MODEL_URL/v1/chat/completions -d '{"model":"'$MODEL'","messages":[{"role":"user","content":"hi"}],"max_tokens":16}'
    # the same server, converted -> fabricated zeros and a null status
    curl -s $MODEL_URL/v1/responses       -d '{"model":"'$MODEL'","input":"hi","max_output_tokens":16}'
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions