Skip to content

feat(trajectory): add v2 tool timing schema and projection - #5660

Merged
lafrancef merged 1 commit into
mainfrom
flafrance/tool-timing-schema-v2
Nov 6, 2026
Merged

lafrancef merged 1 commit into
mainfrom
flafrance/tool-timing-schema-v2

Conversation

@lafrancef

@lafrancef lafrancef commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Adds the v2 tool-timing contract and explicit v1 projection. V2 retains attempt identity, native call IDs, requesting-model links, qualified timing intervals, failure links, and gaps. Existing readers can use project_trajectory_v1(...) during migration.

Timing boundaries describe the measured endpoints: lifecycle, executor, transcript, or event receipt. Artifact storage is independent of that meaning. Projection preserves OpenCode’s established v1 artifact source label for its lifecycle instrumentation.

Distinct attempts that reuse a native call ID remain separate, including in v1 projection when restoring that ID would collide. This intentional compatibility correction avoids merging executions; replaying one persisted attempt remains idempotent. Missing failure-detail evidence remains explicit. Failed-tool ATIF export was already unsupported and is a separate exporter concern.

Part 1 of #5660 → #5661 → #5507. The stack provides one end-to-end implementation, split for review. No separate public issue is needed; the implementation and compatibility discussion are tracked in these PRs.

@copy-pr-bot

copy-pr-bot Bot commented Nov 5, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@AdamRajfer AdamRajfer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One question inline; nothing else from me on this part.

Comment thread nemo_gym/trajectory/tool_timing.py Resolved

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HOLD

I found two BLOCKERs: valid sparse serialization can fail, and distinct producer attempts can receive the same ambiguous tool output. There’s also a RISK in response-pair ownership validation.

I inspected all 12 changed files and traced the affected export and model-call consumers. Two delegated reports weren’t available, so I independently inspected their normalization and projection topics. This was a static review; I haven’t executed the proposed checks.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

if info.context and info.context.get("defer_v1_projection"):
return result
return project_trajectory_v1(result)
result["capture_key"] = result.pop("rollout_id")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BLOCKER — Preserve sparse and filtered serialization.

Failure: TrajectoryRecord(schema_version="2.0", task_id="t", capture_key="c").model_dump(exclude_unset=True) reaches this branch without tool_calls and raises KeyError. Using include={"task_id"} instead fails on the missing rollout_id. The v2 bundle serializer similarly assumes records is present. This affects the existing export sanitizer, which uses exclude_unset=True. For v1 bundles, project_bundle_v1 has the opposite problem: it adds records=[] even when the caller explicitly excluded that field.

Correction: Transform only fields retained by the serialization handler. Replace the v2 trajectory transformation with:

if "rollout_id" in result:
    result["capture_key"] = result.pop("rollout_id")
if "tool_calls" in result:
    result["tool_calls"] = [
        canonical_tool_record(tool, include_join_evidence=False)
        for tool in result["tool_calls"]
    ]

Apply the same membership guard to the records transformation in AgentObservationBundle.serialize_version and project_bundle_v1. The invariant is that version conversion neither requires omitted fields nor reintroduces excluded fields. These are source-traced suggestions; verify them before adopting them.

Preservation and acceptance: Add proposed checks for both versions: a sparse trajectory and bundle must serialize with exclude_unset=True; include={"task_id"} must return only the trajectory task ID; AgentObservationBundle(source="test").model_dump(include={"source"}) must return exactly {"source":"test"}. Pass a sparse v2 trajectory through _rollout_for_export and assert successful sanitization. Full v2 dumps must still emit capture_key, omit rollout_id, and canonicalize tool fields. Retain the concrete v1 assertions in the existing sparse/exclusion tests.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 57c8706. Both envelope serializers transform only fields retained by the serialization handler, and v1 bundle projection no longer adds an excluded records field. Regression coverage includes sparse and filtered v1/v2 envelopes and full dumps.

Comment thread nemo_gym/trajectory/tool_timing.py Outdated
raw_attempts = defaultdict(set)
for observed in observations:
if not isinstance(observed, TrajectoryToolCall):
raw_attempts[(observed.invocation_id, observed.tool_call_id)].add(observed.span_id)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BLOCKER — Count producer-qualified attempts before assigning conversation output.

Failure: Normalize two raw observations with the same invocation_id="root", tool_call_id="request", and span_id="local", but different producer_id values. Supply one root conversation result for request. This set counts only the raw span, so its size is one and both observations receive that result. Their imported identities correctly remain distinct because the producers differ. The final output ambiguity check then emits no ambiguity gap because both outputs are already populated. One unattributable result has silently become evidence for two executions.

Correction: Include effective producer identity in this attempt count. Replace the insertion with:

raw_attempts[(observed.invocation_id, observed.tool_call_id)].add(
    (observed.producer_id or producer_id, observed.span_id)
)

This makes the existing preassignment guard leave both outputs unknown; the final one-to-many join can then emit tool_output_ambiguous. The invariant is that a conversation result cannot be assigned to several distinct imported attempts merely because their producers reused a local span. Verify this suggested change before adopting it.

Preservation and acceptance: Extend the distinct-producer test with the single conversation output above. Assert two distinct retained spans, both outputs None, and an attempt-scoped tool_output_ambiguous gap for each. A repeated snapshot with the same producer and local span must still deduplicate to one attempt and receive its unique output. A single raw attempt must retain the existing partial-instance output behavior, and canonical snapshots must retain the completed "done" output in both orders in the snapshot regression test. These checks are proposed, not executed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 57c8706. Output preassignment now counts effective producer identity together with the local span. Distinct producers retain separate attempts with unknown outputs and scoped ambiguity gaps; replay from the same producer still deduplicates and retains its unique output.

for invocation in self.invocations:
owners[invocation.invocation_id].add(invocation.agent_instance)
for reference in invocation.model_calls:
call = calls.get(reference.model_call_id)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RISK — Resolve response-pair references before checking model ownership.

Failure: Construct a v2 trajectory with an invocation attributed to "other" and model_calls=[ModelCallRef(model_ref=policy, response_id="resp")]. Include one captured call attributed to "assistant" with matching policy/response metadata. Validation accepts the contradictory ownership because this lookup uses the absent model_call_id and returns None. The same association expressed with an explicit model-call ID is correctly rejected. Both reference forms are supported by ModelCallRef, and the performance aggregator resolves response pairs and counts their calls for the invocation.

Correction: Resolve every supplied qualifier before applying the existing ownership check. A localized replacement for this lookup is:

candidates = self.model_calls
if reference.model_call_id is not None:
    candidate = calls.get(reference.model_call_id)
    candidates = [] if candidate is None else [candidate]
matches = [
    candidate
    for candidate in candidates
    if (reference.model_ref is None or candidate.response_metadata.model_ref == reference.model_ref)
    and (reference.response_id is None or candidate.response_metadata.response_id == reference.response_id)
]
call = matches[0] if len(matches) == 1 else None

The invariant is that both supported exact-reference forms enforce the same known-instance constraint without selecting an arbitrary ambiguous match. This suggestion retains the indexed explicit-ID path; verify it before adopting it.

Preservation and acceptance: Extend the invocation-instance test with the response-pair-only setup and assert rejection with owning invocation. The matching-instance case and cases with either instance unknown must validate. Keep the existing explicit-ID mismatch rejection. Unmatched or ambiguous response pairs must not select an owner, and contradictory supplied response/model qualifiers must not resolve an ID reference. These are proposed checks; I haven’t run them.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 57c8706. Ownership validation resolves all supplied model-call qualifiers and checks the instance only for a unique match. Regressions cover response pairs, explicit IDs, contradictory qualifiers, ambiguous/unmatched references and unknown instances.

@lafrancef
lafrancef force-pushed the flafrance/tool-timing-schema-v2 branch from e2b26ad to 57c8706 Compare November 5, 2026 14:49
@lafrancef

Copy link
Copy Markdown
Contributor Author

/ok to test 57c8706

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

@lafrancef

Copy link
Copy Markdown
Contributor Author

/review

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP

I verified by source inspection that all three findings from the previous bot review are fixed:

  • Envelope serialization guards omitted fields, and v1 bundle projection no longer adds an excluded records field.
  • Output preassignment counts producer-qualified attempts. Distinct producers leave the shared result ambiguous; same-producer replay still deduplicates and retains its unique output.
  • Ownership validation resolves response-pair and explicit-ID references using every supplied qualifier, then checks ownership only for a unique match.

The regression tests assert these outcomes. I also traced the directly affected export and performance consumers and the existing preservation cases. No unresolved prior findings or eligible new regressions remain.

This was a static follow-up review; I didn't execute tests. The previously reviewed tree wasn't available, so I couldn't independently compare the complete inter-review delta. The review stayed focused on the reported fixes and their directly affected paths.

Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.

@lafrancef
lafrancef force-pushed the flafrance/tool-timing-schema-v2 branch from 57c8706 to 491168e Compare November 5, 2026 19:15
@lafrancef

Copy link
Copy Markdown
Contributor Author

/ok to test 491168e

@lafrancef

Copy link
Copy Markdown
Contributor Author

/review

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP

I verified by source inspection that the three previously reported failures remain fixed:

  • Envelope serialization guards omitted fields, and v1 bundle projection leaves excluded records absent.
  • Output preassignment counts producer-qualified attempts. Distinct producers leave a shared result ambiguous; same-producer replay still deduplicates and retains its unique output.
  • Ownership validation resolves both supported reference forms using every supplied qualifier and checks ownership only for a unique match.

The regression assertions cover these failures and their preservation cases. I also checked that completion-gap filtering requires a terminal outcome and a completion endpoint, and traced the directly affected export sanitizer and performance resolver. No unresolved prior findings or eligible new regressions were established.

This was a static follow-up review; I didn't execute tests. The previously reviewed tree wasn't available, so I couldn't independently compare the complete inter-review delta. One delegated report wasn't available; I independently inspected its ownership-validation topic. The review stayed within the prior fixes and their directly affected paths.

Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.

@lafrancef
lafrancef enabled auto-merge November 6, 2026 08:07
@lafrancef

Copy link
Copy Markdown
Contributor Author

/review

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP

I verified by source inspection that the three previously reported failures remain fixed:

The regression assertions cover these failures and their preservation cases. I also traced the directly affected export sanitizer and performance resolver, and checked that completion-gap filtering still requires a terminal outcome and a completion endpoint. No unresolved prior findings or eligible new regressions were established.

This was a static follow-up review focused on the prior fixes and their directly affected paths; I didn't execute tests. The previously reviewed tree wasn't available, so I couldn't independently compare the complete inter-review delta.

Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.

@marcromeyn marcromeyn left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Frederic, re-checked at 8da94ea; suites pass for me (3148). The sparse serialization, producer-qualified attempts and response-pair ownership threads are fixed and covered in test_schema_review.py, and the earlier points hold at this layer: v2-only diagnostics are dropped from v1, snapshots advance status and output together, cycles are rejected at decode, and a terminal status keeps the missing-end gap until an end time arrives. Two things before I approve:

  1. Adam's question is still open (tool_timing.py:381-385, trajectory_format.py:33-42). tool_failure_detail_unavailable is added for any failed, timed-out or cancelled attempt without a failure_id, from every producer, and _V2_ONLY_GAPS doesn't drop it, while the v1 projection already discards failure ids and events (:110-111, :349). A normalized, otherwise complete failed tool ends up with this as its only v1 gap, which strict ATIF export rejects (atif_export.py:937). This layer's collector doesn't normalize yet, but with #5661 the collector produces it too, so the default v1 path would regress there. Dropping it in the v1 projection, keeping real coverage gaps, plus an enrich-then-strict-export test with a failed tool, would close it.
  2. A reused conversation result can go to two attempts (tool_timing.py:145, :157-158). The uniqueness check counts raw observations but not retained canonical attempts, so when a canonical attempt already owns a request id and a new raw attempt with the same id arrives, both get the same output and no tool_output_ambiguous gap (that check at :523-526 only looks at tools whose output is still empty). Repro: normalize raw span a with root/request and output "result", then normalize the retained canonical a together with raw span b on the same request id and conversation. Counting canonical and raw identities together (deduplicating snapshots of the same attempt) and a mixed canonical/raw test would pin it.

Happy to approve once these two are in.

@lafrancef

Copy link
Copy Markdown
Contributor Author

On point 1: I agree that normalization adds tool_failure_detail_unavailable to the v1 gaps. However, removing that gap alone does not restore strict export for a failed tool: gym_rollout_to_atif also rejects non-completed tool executions independently at nemo_gym/trajectory/atif_export.py:602–604, after the gap check. That restriction predates this stack. A failed tool with no gaps is therefore already rejected; adding this gap changes the first reported rejection, rather than turning a previously exportable failed-tool trajectory into an unexportable one.

We previously chose to retain the missing-failure-detail diagnostic and defer support for exporting failed tool executions to the exporter work. A successful enrich-then-strict-export regression for that case needs that exporter change as well. I would keep that policy here; if there is a previously exportable concrete fixture that this layer breaks for another reason, I am happy to check it.

Point 2 is confirmed: the mixed retained/raw case duplicates the conversation output. I am fixing the attribution check to count qualified identities across both sets, with regression coverage for both ordering and same-attempt replay.

Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
@lafrancef
lafrancef force-pushed the flafrance/tool-timing-schema-v2 branch from 8da94ea to bfd93b7 Compare November 6, 2026 14:23
@lafrancef

Copy link
Copy Markdown
Contributor Author

/ok to test bfd93b7

@lafrancef

Copy link
Copy Markdown
Contributor Author

/review

@lafrancef

Copy link
Copy Markdown
Contributor Author

Point 2 is fixed in bfd93b7. Output attribution now counts qualified identities across retained canonical and raw attempts before assigning conversation output. Repeated snapshots of the same attempt count once. In the mixed a/b example, a retains its output and b remains unresolved with tool_output_ambiguous. Regressions cover both input orders and same-attempt replay.

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP

No unresolved prior findings or eligible new regressions were established in this follow-up.

I verified by source inspection that:

The failure-detail gap doesn't establish the claimed strict-export regression: non-completed tool executions are independently rejected in both the head exporter and the base exporter. Removing the gap alone wouldn't make that failed-tool case exportable.

I traced the affected sanitizer and performance resolver and checked the relevant preservation assertions. This was a static follow-up review; I didn't execute tests. The previously reviewed tree wasn't available, so I couldn't independently compare the complete inter-review delta. One delegated report wasn't available; I independently completed its output-attribution inspection.

Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.

@lafrancef
lafrancef added this pull request to the merge queue Nov 6, 2026
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42239786681

Merged via the queue into main with commit 2c00f12 Nov 6, 2026
108 of 111 checks passed
@lafrancef
lafrancef deleted the flafrance/tool-timing-schema-v2 branch November 6, 2026 15:21

This branch was previously deployed

1 inactive deployment
public — bfd93b79 Deployed Nov 6, 2026 by copy-pr-bot[bot] via release / finalize / notify #10628
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants