Skip to content

feat(opencode): capture qualified tool timing - #5507

Merged
lafrancef merged 1 commit into
mainfrom
flafrance/tool-timing-opencode-v2
Nov 10, 2026
Merged

lafrancef merged 1 commit into
mainfrom
flafrance/tool-timing-opencode-v2

Conversation

@lafrancef

@lafrancef lafrancef commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Adds qualified tool timing from OpenCode’s persisted tool parts. Select trajectory_schema_version: "2.0" in collection configuration or run_examples(..., trajectory_schema_version="2.0") to retain attempt IDs, requesting-model links, lifecycle measurements, and observability gaps.

OpenCode’s persisted state.time.start/end values measure lifecycle endpoints. Database storage does not make their boundary transcript; v1 projection retains the existing artifact source label. Runtime collection supplies one wall-clock domain and agent instance ID per local launch or retained sandbox session, allowing tool busy-time unions and stable ownership across session snapshots. Standalone imports without clock-owner evidence remain unknown. Model capture uses Gym model-server timestamps and cannot share the OpenCode clock domain.

Started tools without terminal evidence stay incomplete, including after an enclosing timeout/cancellation. The enclosing outcome remains available to health checks. Distinct persisted parts reusing a provider call ID remain separate attempts rather than being merged, including for v1 consumers; replaying the same part remains idempotent. Failed-tool ATIF export is a pre-existing limitation, also retained by #5510.

The rollout-evidence guide documents current harness mappings and migration toward semantic boundaries wherever instrumentation supports them. Legacy delivery/source labels alone do not prove executor or lifecycle scope. Pi’s legacy upgrade mapping and model-call clock-domain population remain explicit follow-ups.

Direct HTTP /run returns raw v2 producer observations; collection performs the selected v1/v2 projection. A terminal tool status alone does not clear a missing-completion-time diagnostic; end-time evidence is required.

Schema #5660 is merged. This PR follows #5661; merge #5661 first.

@lafrancef

Copy link
Copy Markdown
Contributor Author

/review

@lafrancef lafrancef added area:core Shared APIs, servers, telemetry, health, and registries feature New capabilities, enhancements, or enablement work labels Nov 3, 2026
@copy-pr-bot

copy-pr-bot Bot commented Nov 3, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@lafrancef

Copy link
Copy Markdown
Contributor Author

/review

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HOLD

I found two BLOCKERs in requesting-model attribution and the public v1 projection, plus 15 RISKs affecting snapshot merging, OpenCode identity, enrichment, and downstream consumers.

I completed the changed-file review and traced the affected capture, replay, session, performance, health, and conformance paths. This was a source-only review; I didn't execute tests. The suggested patches and acceptance checks below still need verification.


Inline comments (escaped v1):

  • nemo_gym/trajectory/tool_timing.py line 274: BLOCKER — Restore the model-server constraint when merging replayed tools.

    Failure: Enrich a v2 observation with source_message_id="message" and source_model_ref=A before capture arrives, then enrich it again with capture. The canonical serializer removes input join evidence. This merge restores source_message_id but not source_model_ref, so candidate selection searches every model server. If only B has a matching session/message/native ID, the tool silently links to B despite the producer specifying A. If both exist, a resolvable request becomes ambiguous.

    Correction: Add the missing field to this fill-only-when-unknown tuple:

                "source_message_id",
                "source_model_ref",
    

    The invariant is that replay must not weaken the original requesting-call scope.

    Preservation/acceptance: Proposed regression: perform those two enrichment passes with source model A. Captures from A and B must resolve only to A; capture from B alone must leave the link unset. Repeating enrichment must retain span IDs and resolved links. An observation genuinely lacking model scope must remain ambiguous when both calls match. Retain failed-retry filtering and repeated-native-ID disambiguation.

  • nemo_gym/trajectory/trajectory_format.py line 73: BLOCKER — Project valid tools whose default discriminator was omitted.

    Failure: The new canonical-failure fixture validates as v2, but its tool omits kind, which defaults to "tool_call". Passing that raw trajectory to project_trajectory_v1 takes the non-tool branch of project_observation_v1. It leaves span_id, failure_id, and timings inside an envelope labeled v1, which the v1 decoder rejects. The advertised compatibility API therefore fails for a valid committed wire shape.

    Correction: Entries in tool_calls are tools even without the serialized default:

        result["tool_calls"] = [
            project_observation_v1({"kind": "tool_call", **record})
            for record in result.get("tool_calls", [])
        ]
    

    Preservation/acceptance: Proposed regression: validate the fixture as v2, project its raw dictionary, and validate the result as v1. Expect duration_ms == 10 and no span_id, failure_id, or timings. Explicit-kind tools must retain the same single-boundary projection; invocation, model-call, and turn projection must remain unchanged.

  • nemo_gym/trajectory/trajectory_format.py line 35: RISK — Clear flat timing when several detailed boundaries exist.

    Failure: A valid tool can contain flat timing such as [1, 2] with duration_ms=1000 alongside lifecycle and executor measurements. This projector overwrites flat fields only for one measurement. With two, it removes timings but preserves the preexisting flat interval, presenting an arbitrary boundary to legacy readers. That contradicts the documented rule that multiple boundaries produce no flat interval.

    Correction: Immediately after the single-measurement branch, add:

            elif len(timings) > 1:
                for key in _TOOL_FLAT_FIELDS:
                    record[key] = None

    Preservation/acceptance: Proposed checks: two detailed boundaries plus non-null flat fields must yield null flat endpoints, duration, and source. Zero detailed measurements must preserve a legacy flat interval. One lifecycle measurement must still project its endpoints/duration with timing_source="harness". This keeps legacy evidence without implicitly choosing among detailed boundaries.

  • nemo_gym/trajectory/tool_timing.py line 284: RISK — Retain error details from a compatible completion snapshot.

    Failure: Normalize two observations of the same span: first status="incomplete", error_type=None, then status="failed", error_type="exit_7". The snapshot merge advances the status but never copies error_type. The canonical failed attempt loses supplied failure evidence.

    Correction: After merging the outcome, fill missing error metadata only when the incoming outcome agrees with the retained one:

            if existing.error_type is None and existing.status == tool.status:
                existing.error_type = tool.error_type

    This avoids attaching a rejected terminal snapshot's error to a different retained outcome.

    Preservation/acceptance: Proposed checks: incomplete → failed must produce one failed attempt with error_type="exit_7"; a later null value must not erase it. Completed → contradictory failed must retain completed, emit tool_outcome_conflict, and not adopt the failed error label. Distinct failed/completed retry spans must remain separate, and existing timing conflict checks must remain intact.

  • nemo_gym/trajectory/tool_timing.py line 281: RISK — Don't downgrade an observed incomplete execution to unknown.

    Failure: Normalize same-span observations in this order: explicit status="incomplete", then default status="unknown" from a timing-only snapshot. This branch retains unknown. Because completion-gap generation checks only incomplete, the known unfinished execution also loses its completion gap.

    Correction: Replace the first condition with:

            if existing.status == "unknown" or (
                existing.status == "incomplete" and tool.status != "unknown"
            ):
    

    Keep the assignment and terminal-conflict branch below it. Missing outcome evidence must not erase an explicit unfinished outcome.

    Preservation/acceptance: Proposed transition checks: incomplete → unknown retains incomplete and a span-scoped tool_completion_unavailable gap; unknown → incomplete upgrades; incomplete → completed/failed advances; terminal → unknown remains terminal; contradictory terminal outcomes still emit tool_outcome_conflict.

  • nemo_gym/trajectory/tool_timing.py line 78: RISK — Enforce every supplied requesting-call qualifier.

    Failure: Supply model_call_ref={model_call_id:"m", response_id:"other"} while captured m has response ID "actual" and contains the native tool ID. _matches_reference accepts m solely by ID. A contradictory model-server qualifier is ignored too, unlike the existing shared matcher.

    Correction: Replace this return with:

            return (
                call.model_call_id == reference.model_call_id
                and (reference.model_ref is None or call.response_metadata.model_ref == reference.model_ref)
                and (reference.response_id is None or call.response_metadata.response_id == reference.response_id)
            )
    

    Also preserve these qualifiers at reference construction: when tool.model_call_id and model_call_ref coexist, require their IDs to agree rather than replacing the richer reference with an ID-only one.

    Preservation/acceptance: Proposed checks: ID-only and fully matching references resolve; changing only response ID, model server, or the second supplied model-call ID leaves the link unset with a scoped diagnostic. Response-ID-plus-model matching must still work. Native IDs must verify an exact reference, never select one alone.

  • responses_api_agents/opencode_agent/artifacts.py line 110: RISK — Derive conversation IDs from immutable part identity.

    Failure: Parse one running part p0 with native callID="c", retain its enriched trajectory, then complete p0 and append p1 with the same native ID. This mapping changes p0's tool_call_id from c to c:p0, although its span stays unchanged. The normalizer rejects the completed snapshot as an identity conflict, leaving the retained attempt incomplete. Counting before slicing doesn't prevent changes as the database grows. The encoding also collides with a singleton whose literal native ID is c:p0.

    Correction: Replace the Counter-dependent mapping with a persisted-part mapping, for example:

        normalized_ids = {row["id"]: f"call:{row['id']}" for row in tool_parts}

    Keep native IDs in native_tool_call_id. Apply the same mapping to observation conversations, turns, child-spawn references, and compaction references; update singleton-ID assertions accordingly.

    Preservation/acceptance: Proposed regression: enrich running p0, append the duplicate and complete p0, then re-enrich. Expect two spans, p0 completed with its end/duration, and no identity conflict. Full and filtered parses must retain identical IDs; the collision example must remain distinct. Preserve exact requesting-model joins through native IDs and incomplete status for unfinished tools.

  • responses_api_agents/opencode_agent/artifacts.py line 150: RISK — Use the tool-ID mapping for earlier compaction boundaries.

    Failure: An earlier activation contains two tool parts with native ID c, emitted as qualified conversation IDs. A later compaction's tail_start_id points to the first tool's assistant message. This earlier-message branch records raw c, so first_kept_item_id names neither emitted item. A full parse instead uses the mapped ID from the active-message branch. The boundary changes with slicing and becomes dangling without a gap.

    Correction: Use the existing complete mapping here:

                    item_id = normalized_ids[row["id"]]
    

    This dependent change remains necessary if the mapping's encoding is corrected separately.

    Preservation/acceptance: Extend the cross-activation compaction case with repeated native tool IDs. Full and sliced parses must name the same first retained tool, equal to its earlier emitted conversation ID. Text/reasoning boundaries must remain valid; a user-only boundary must remain unresolved.

  • nemo_gym/rollouts/rollout_collection.py line 1364: RISK — Enforce payload omission after failed projection too.

    Failure: include_payloads=False now merges capture with payloads, but strips them only when a trajectory exists. A call with latency_total_ms=-1 makes projection fail and leaves its request/response in capture, as shown by the existing fallback test. Collection also selects omission for multimodal_history_redacted, so projection failure can restore intentionally excluded payloads.

    Correction: After the projection attempt, independently enforce capture omission:

        if not include_payloads:
            capture = result.get("ng_model_call_capture")
            calls = capture.get("calls") if isinstance(capture, dict) else None
            for call in calls if isinstance(calls, list) else []:
                if isinstance(call, dict):
                    for key in ("request", "response", "request_raw", "response_raw"):
                        call.pop(key, None)

    Keep trajectory omission and verification gaps. Don't reuse _strip_capture_payloads here because it also removes verifier evidence needed after failure.

    Preservation/acceptance: Proposed checks: omission plus the negative-latency fixture leaves no capture payload fields despite projection failure. Include-true fallback must retain them. Successful v2 joins must remain exact and gain the payload-unavailable gap; reward, harness response, and failure verifier evidence must survive.

  • nemo_gym/rollouts/rollout_collection.py line 5437: RISK — Unwrap materialized episode responses at the shared enrichment boundary.

    Failure: Materialized /run returns {episode_id, task_id, result:{ng_agent_observations,...}}, as the environment producer shows. This callback enriches the outer envelope. Without capture, no evidence keys are found and no trajectory appears; with capture, the outer trajectory misses nested agent evidence. Full collection unwraps first.

    Correction: At the beginning of enrich_run_result, before stamping or merging, normalize recognized materialized envelopes in place:

        if _materialized_taskset(row) is not None and _is_episode_response(result):
            normalized = _native_episode_record(row, result)
            result.clear()
            result.update(normalized)

    This also covers the documented manual enrichment path and retains episode-identity validation.

    Preservation/acceptance: Proposed checks: a materialized success containing one nested invocation/tool must produce equivalent canonical evidence through opt-in run_examples, manual enrichment, and full collection. A handled failure must retain observations without inventing reward. Existing flat results must still work; enrichment-disabled calls must retain the raw envelope unchanged.

  • nemo_gym/rollouts/rollout_collection.py line 1271: RISK — Count performance evidence by the full invocation identity.

    Failure: V2 permits a/root and b/root, but _build_ng_perf keys counters only by invocation_id. With those two owners, one explicit turn each, and one tool owned by a/root, it reports four turns and two tools instead of two and one, distorting efficiency and coverage metrics.

    Correction: Use (agent_instance, invocation_id) for the owned-call increment and all tool/turn counters and lookups. The replacements include:

    owned_calls_by_invocation[(invocation.agent_instance, invocation.invocation_id)] += 1
    
    tool_calls_by_invocation = Counter((t.agent_instance, t.invocation_id) for t in trajectory.tool_calls)
    num_tool_calls = sum(tool_calls_by_invocation[(i.agent_instance, i.invocation_id)] for i in trajectory.invocations)
    turns_by_invocation = Counter((t.agent_instance, t.invocation_id) for t in trajectory.turns)
    num_turns = sum(
        turns_by_invocation[(i.agent_instance, i.invocation_id)]
        or owned_calls_by_invocation[(i.agent_instance, i.invocation_id)]
        or 1
        for i in trajectory.invocations
    )

    Preservation/acceptance: Proposed two-instance case must report two turns, one tool, and coverage 1.0 for two uniquely owned calls. Preserve call-position deduplication, including the shared-call case's two turns/once-counted tokens/0.5 coverage. Preserve per-owner explicit-turn → owned-call → one fallback, including the existing hybrid result of seven turns and coverage 4/7. Legacy null-instance owners must retain their counts.

  • nemo_gym/rollouts/rollout_collection.py line 840: RISK — Keep instance ownership when reconstructing turns.

    Failure: For explicit owners a/root → call-a and b/root → call-b, _turns_from_model_calls keeps only the string root and emits unattributed turns numbered 1 and 2. Expected identities are (a,root,1) and (b,root,1) with independent step counts. The new contract allows these owners, so supplied attribution is lost on the no-producer-trajectory path.

    Correction: Store the owning AgentInvocation, not its string ID, in both reference indexes. Resolve owner from those indexes, use key=(owner.agent_instance, owner.invocation_id) for both counters, and construct each turn with:

    invocation_id=owner.invocation_id,
    agent_instance=owner.agent_instance,
    turn_no=turn_counts[key],
    step_count=tool_counts[key],

    Increment turn_counts[key] before construction and tool_counts[key] afterward, as today.

    Preservation/acceptance: Proposed enrichment test: two v2 owners with explicit distinct call references and successful responses, no producer trajectory, must yield exactly those two composite turn identities and independent tool-step counts. Preserve legacy single-owner numbering, explicit-reference-only ownership, verifier exclusion, and HTTP-error filtering. Don't infer owners from timestamps or list order.

  • nemo_gym/rollouts/health/timeouts.py line 360: RISK — Validate clock domains before hashing them.

    Failure: Two timed spans with domains [] and "clock" reach this set comprehension and raise TypeError. The timing reader promises malformed records never raise, but evaluate_record invokes it outside the per-check guard, so one malformed row can stop the health report instead of leaving its union unknown.

    Correction: At both _model_call_span and _tool_call_span, replace the unchecked domain argument with:

    clock_domain=(
        domain if isinstance(domain := call.get("clock_domain", "legacy"), str) and domain else None
    ),

    Only nonempty strings establish comparable domains. Keep the absent-field legacy fallback; v2 already inserts explicit None for missing provenance.

    Preservation/acceptance: Proposed checks: list, dict, and empty-string domains on model calls and v2 tool measurements must not raise through the reader or full report; multi-span unions involving them must be unknown. Equal valid domains must still union overlaps correctly. Different or missing v2 domains must stay unknown, while existing legacy policy/tool splits remain measured.

  • nemo_gym/rollouts/harness_capabilities/checker.py line 853: RISK — Resolve v2 references using composite invocation ownership.

    Failure: A valid v2 trajectory with a/root and b/root, each carrying its correctly attributed tool and turn, fails tools.invocation_target and steps.invocation_target: each string ID matches twice. Request lookup, parent traversal, and step-owner comparison retain the same string-only assumption. The PR now admits v2's capture key, but these checks falsely reject its supported owner identities.

    Correction: Make invocation-target, parent, request, and step-owner resolution instance-aware for v2. An explicit instance must constrain resolution to that instance; missing attribution must remain unresolved when several owners match. Carry composite identities in the model-call owner bookkeeping too. Several checks share this assumption, so a one-line predicate replacement isn't a complete fix.

    Preservation/acceptance: Proposed complete two-instance fixture: reused invocation IDs with distinct calls/tools and consistent instances must pass target, request, parent, and step-owner checks. Remove attribution from an ambiguous reference or supply the wrong instance and expect failure. Preserve legacy single-instance resolution, genuine conflicting call-owner rejection, and missing/cyclic-parent rejection.

  • nemo_gym/rollouts/rollout_collection.py line 3167: RISK — Settle enrichment failures per drained outcome.

    Failure: aclose_and_drain transfers completed tasks out of the underlying iterator, then awaits enrichment. If one enrichment raises—for example an OSError during requested capture deletion—the entire drain raises before returning any futures. Unrelated completed outcomes have already been transferred and a second drain returns nothing, violating the settled-future contract.

    Correction: Replace this successful-task branch with per-outcome exception settlement:

                    result = task.result()
                    try:
                        if self._enrich is not None:
                            await self._enrich(result)
                    except Exception as error:
                        outcome.set_exception(error)
                        outcome.exception()
                    else:
                        outcome.set_result((result.row, result.result))

    Keep cancellation separate and retain retrieval of stored exceptions.

    Preservation/acceptance: Proposed regression: drain two completed rollouts with enrichment raising OSError for the first and succeeding for the second. Expect two settled futures: the first raises that error and the second yields its row/result. Repeated drain must be empty. Preserve original dispatch exceptions, cancelled outcomes, and the guarantee that draining admits no new work.

  • nemo_gym/trajectory/trajectory_record.py line 833: RISK — Preserve span-scoped adapter gaps through capture joins.

    Failure: OpenCode now emits per-span timing diagnostics, but this unchanged deduplication key uses only code, invocation, and detail. Two tools in one invocation with missing starts and valid completion endpoints each emit tool_start_unavailable with null detail. Capture enrichment keeps only the last diagnostic. Normalization doesn't regenerate that adapter-specific code, so the first attempt loses its gap before the new import mapping sees it.

    Correction: Include both new scope fields in this key and in join_assistant_message_calls' equivalent deduplication:

    (gap.code, gap.agent_instance, gap.invocation_id, gap.span_id, gap.detail)

    Keep exact duplicate elimination; different attempts aren't duplicate diagnostics.

    Preservation/acceptance: Proposed integration test: enrich a captured OpenCode bundle with two distinct tool parts whose starts are invalid but ends are valid. Each canonical span must retain its own start-unavailable gap through repeated enrichment. Exact duplicate gaps must still collapse. Ambiguous raw references must remain explicitly unresolved rather than being copied to multiple attempts.

  • nemo_gym/trajectory/trajectory_record.py line 643: RISK — Don't assign an ambiguous captured session to the last agent instance.

    Failure: V2 permits invocations a/root explicitly referencing call-a and b/root explicitly referencing call-b. With both captures declaring client_session_id="root", this dictionary retains only b/root. The automatic association loop then appends call-a to b/root before explicit-reference resolution; b/root keeps that incorrect extra reference with a conflict gap. This newly valid input therefore acquires contradictory ownership during capture enrichment, even though the producer supplied exact owners.

    Correction: Index all invocations for a session ID, and key each invocation's reference set by its composite identity. Honor existing exact call references before automatic session association. Infer an owner from client_session_id only when it identifies one invocation; otherwise leave an undeclared call unowned with a diagnostic instead of selecting the last record. The capture record currently lacks instance evidence, so it can't safely break that tie. A blanket tuple lookup alone wouldn't repair this inference path.

    Preservation/acceptance: Proposed regression: the two explicit owners above must retain only their original calls after repeated joins, with no manufactured ownership conflict. An undeclared call with the ambiguous session must remain unowned. Preserve the existing single-root automatic association, exact response/model-reference resolution, duplicate-reference handling, and cross-invocation compaction rejection.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

@lafrancef
lafrancef changed the base branch from main to arajfer/run-records-on-store November 3, 2026 18:50
@lafrancef

Copy link
Copy Markdown
Contributor Author

/review

@AdamRajfer

Copy link
Copy Markdown
Contributor

FYI: the base branch arajfer/run-records-on-store moved forward (merge only, no rewrite) to 4d91e9b, which brings in current main. Merge it into this branch when convenient.

@lafrancef

Copy link
Copy Markdown
Contributor Author

/review

1 similar comment
@lafrancef

Copy link
Copy Markdown
Contributor Author

/review

@AdamRajfer

Copy link
Copy Markdown
Contributor

FYI: the base branch arajfer/run-records-on-store moved forward (merge only, no rewrite) to cd8719b, which brings in current main. Merge it into this branch when convenient.

@AdamRajfer

Copy link
Copy Markdown
Contributor

FYI: the base branch arajfer/run-records-on-store moved forward again (merge only, no rewrite) to 4c31334, which brings in its own base branch's latest documentation and message fixes. Merge it into this branch when convenient.

@AdamRajfer

Copy link
Copy Markdown
Contributor

FYI: the base branch arajfer/run-records-on-store moved forward again (merge only, no rewrite) to b833ef3, which brings in its base branch's merge of current main (including #5564, which split the episode modules) and its latest fixes. Merge it into this branch when convenient.

@lafrancef
lafrancef force-pushed the flafrance/tool-timing-opencode-v2 branch from e2f1e26 to 0fb0100 Compare November 4, 2026 08:22
@lafrancef

Copy link
Copy Markdown
Contributor Author

/review

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP

No eligible findings remain from review 5755488220. I checked the prior failures against the current source and relevant regression assertions.

This was a source-only follow-up covering the prior findings and directly affected paths; I didn't execute tests. The previously reviewed tree wasn't available, so I couldn't independently compare inter-review changes. This approval isn't a fresh whole-PR review.

Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.

@lafrancef
lafrancef marked this pull request as ready for review November 4, 2026 08:43
@lafrancef

Copy link
Copy Markdown
Contributor Author

/ok to test 0fb0100

@lafrancef lafrancef added the needs-review PR is ready for code review and waiting on a reviewer label Nov 4, 2026
@nemo-automation-bot
nemo-automation-bot Bot requested review from a team November 4, 2026 08:43
@lafrancef

Copy link
Copy Markdown
Contributor Author

/ok to test c59a1d8

@lafrancef

Copy link
Copy Markdown
Contributor Author

/review

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP

No eligible findings in this source-only follow-up from review 5790966667.

I checked the current source and relevant regression assertions:

  • Missing completion-time diagnostics still require end-time evidence before normalization or enrichment clears them. The regression assertions distinguish repeated enrichment without an end timestamp from later compatible completion timing.
  • Direct HTTP /run remains raw producer evidence, while run_examples applies the selected schema. The schema-precedence assertions cover direct and nested success/failure results while preserving primary responses. The strict ATIF regression still requires unchanged legacy export after repeated default enrichment.
  • Replay retains accepted completion output and stable attempt identity. The replay assertions cover both snapshot orders and prevent older partial output from replacing completion output.
  • Runtime capture supplies separate clock domains per local launch or retained sandbox session. The busy-time assertions distinguish a shared-domain union from unknown provenance. The cancellation assertions retain partial tool output and the enclosing cancellation outcome without inventing a terminal tool outcome.

The previously reviewed tree isn't mounted, so I couldn't independently compare inter-review changes. The supplied evidence doesn't establish an eligible subsequent regression. This completes the scoped follow-up and directly affected consumer checks, not a fresh whole-PR review. All three delegated inspections finished; two report bodies weren't delivered, so I covered those assignments directly. I didn't execute tests.

Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.

@lafrancef
lafrancef force-pushed the flafrance/tool-timing-opencode-v2 branch from c59a1d8 to ecbdb53 Compare November 6, 2026 11:59
@lafrancef
lafrancef force-pushed the flafrance/tool-timing-integration-v2 branch from b176ae6 to 6e0ebde Compare November 6, 2026 11:59
@lafrancef

Copy link
Copy Markdown
Contributor Author

/ok to test ecbdb53

@lafrancef

Copy link
Copy Markdown
Contributor Author

/review

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP

No eligible findings in this source-only follow-up from review 5797616444.

I checked the current source and relevant regression assertions:

  • Missing completion-time diagnostics still require end-time evidence before normalization or enrichment clears them. The regression assertions distinguish repeated enrichment without an end timestamp from later compatible completion timing.
  • Direct HTTP /run remains raw producer evidence, while run_examples applies the selected schema. The schema-precedence assertions cover direct and nested success/failure results without changing primary responses. The strict ATIF regression requires unchanged legacy export after repeated default enrichment.
  • Replay retains accepted completion output and stable attempt identity. The replay assertions cover both snapshot orders and prevent older partial output from replacing completion output.
  • Runtime capture supplies separate clock domains per local launch or retained sandbox session. The busy-time assertions distinguish a shared-domain union from unknown provenance. The cancellation assertions retain partial tool output and the enclosing cancellation outcome without inventing a terminal tool outcome.

The previously reviewed tree isn't mounted, so I couldn't independently compare inter-review changes. The supplied evidence doesn't establish an eligible subsequent regression. This completes the scoped follow-up and directly affected consumer checks, not a fresh whole-PR review. All three delegated inspections finished; two report bodies weren't delivered, so I covered those assignments directly. I didn't execute tests.

Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.

@lafrancef

Copy link
Copy Markdown
Contributor Author

/ok to test 705abe9

@lafrancef

Copy link
Copy Markdown
Contributor Author

/review

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP

No eligible findings in this source-only follow-up from review 5799894035.

I verified the relevant source and regression assertions:

  • Missing completion-time diagnostics still require end-time evidence before normalization or enrichment clears them. The regression assertions distinguish repeated enrichment without an end timestamp from later compatible completion timing.
  • Direct HTTP /run remains raw producer evidence, while run_examples applies the selected schema. The schema-precedence assertions cover direct and nested success/failure results without changing primary responses. The strict ATIF regression requires unchanged legacy export after repeated default enrichment.
  • Replay retains accepted completion output and stable attempt identity. The replay assertions cover both snapshot orders and prevent older partial output from replacing completion output.
  • Runtime capture assigns separate clock domains per local launch or retained sandbox session. The busy-time assertions distinguish a shared-domain union from unknown provenance. The cancellation assertions retain partial tool output and the enclosing cancellation outcome without inventing a terminal tool outcome.

The previously reviewed tree isn't mounted, so I couldn't independently compare inter-review changes. The supplied evidence doesn't establish an eligible subsequent regression. This completes the scoped follow-up and directly affected consumer checks, not a fresh whole-PR review. All three delegated inspections finished; I covered completion and replay directly because that report wasn't delivered. I didn't execute tests.

Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.

@AdamRajfer AdamRajfer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I read the OpenCode side of this against failure handling, the sandbox/session boundary, recovery and the timing readers, and ran the six OpenCode test files (539 pass) plus tests/unit_tests/trajectory and tests/unit_tests/rollouts. I also fed a few sessions through parse, enrich and read_rollout_timing in both schemas: a reused native call ID, a tool still running at the stop, end before start, a bash exit 1. Timings stay per attempt and per clock domain, a Gym-cut run keeps the timeout/cancel outcome on the sandbox observation and the invocation rather than on the tool, and nothing in the capture path touches a failure event or a score. One inline note, otherwise this reads fine to me.

@lafrancef

Copy link
Copy Markdown
Contributor Author

/ok to test f814495

@lafrancef

Copy link
Copy Markdown
Contributor Author

/review

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP

No eligible findings in this source-only follow-up from review 5801657351.

I checked the ownership fix and the earlier preservation guarantees:

  • Local collection now passes a fresh agent-instance ID through the parser call. Retained sandbox collection uses the session-owned ID. The parser assigns that ID to both tool observations and invocations. The close regression assertions check matching owners and an unchanged repeated-close response.
  • Missing completion-time diagnostics still require end-time evidence before normalization or enrichment clears them. The regression assertions distinguish repeated enrichment without an end timestamp from later compatible completion timing. The replay assertions retain completed output and stable attempt identity without accepting stale partial output.
  • Caller-facing schema selection remains separate from raw HTTP producer evidence. The schema assertions cover explicit/configured precedence, nested success/failure evidence, and unchanged primary responses. The strict ATIF regression requires unchanged legacy export after repeated default enrichment.
  • The busy-time assertions still distinguish a shared-clock union from unknown provenance. The cancellation assertions retain partial output and the enclosing cancellation outcome without inventing a terminal tool outcome.

The previously reviewed tree isn't mounted, so I couldn't independently compare inter-review changes. The supplied evidence establishes no eligible subsequent regression. This completes the scoped follow-up and directly affected consumer checks, not a fresh whole-PR review. All three delegated inspections finished; two report bodies weren't delivered, so I covered their local-launch and retained-session assignments directly. I inspected source and test assertions but didn't execute tests.

Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.

Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
@lafrancef

Copy link
Copy Markdown
Contributor Author

/ok to test 4935a69

@lafrancef

Copy link
Copy Markdown
Contributor Author

/review

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP

No eligible findings in this source-only follow-up from review 5803438539.

I checked the current source and relevant regression assertions:

  • Ownership remains explicit. Local collection passes a fresh agent-instance ID, while sandbox collection passes the session-owned ID. The parser assigns it to tools and invocations. The close assertions require matching owners and an unchanged repeated-close response.
  • Missing completion-time gaps still require end-time evidence before normalization or enrichment clears them. The regression assertions distinguish repeated enrichment without an end timestamp from later compatible timing. The replay assertions retain completed output and stable identity without accepting stale partial output.
  • Caller-facing schema selection remains separate from raw HTTP producer evidence. The schema assertions cover explicit/configured precedence, nested success/failure evidence and unchanged primary responses. The strict ATIF regression requires unchanged legacy export after repeated default enrichment.
  • The busy-time assertions distinguish a shared-clock union from unknown provenance. The cancellation assertions retain partial output and the enclosing cancellation outcome without inventing a terminal tool outcome.

The previously reviewed tree isn't mounted, so I couldn't independently compare inter-review changes. The supplied evidence establishes no eligible subsequent regression. This completes the scoped follow-up and directly affected consumer checks, not a fresh whole-PR review. All three delegated inspections finished; two report bodies weren't delivered, so I covered their local-launch and retained-session topics directly. I inspected source and test assertions but didn't execute tests.

Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.

@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42824943470

This branch was previously deployed

1 inactive deployment
public — 4935a69f Deployed Nov 10, 2026 by copy-pr-bot[bot] via release / finalize / notify #11414
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:core Shared APIs, servers, telemetry, health, and registries feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants