Repository navigation
feat(trajectory): add v2 tool timing schema and projection - #5660
Conversation
67055eb to
e2b26ad
Compare
AdamRajfer
left a comment
There was a problem hiding this comment.
One question inline; nothing else from me on this part.
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
HOLD
I found two BLOCKERs: valid sparse serialization can fail, and distinct producer attempts can receive the same ambiguous tool output. There’s also a RISK in response-pair ownership validation.
I inspected all 12 changed files and traced the affected export and model-call consumers. Two delegated reports weren’t available, so I independently inspected their normalization and projection topics. This was a static review; I haven’t executed the proposed checks.
Note: formal approval is not available for this request, so this review is posted as a comment only.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
| if info.context and info.context.get("defer_v1_projection"): | ||
| return result | ||
| return project_trajectory_v1(result) | ||
| result["capture_key"] = result.pop("rollout_id") |
There was a problem hiding this comment.
BLOCKER — Preserve sparse and filtered serialization.
Failure: TrajectoryRecord(schema_version="2.0", task_id="t", capture_key="c").model_dump(exclude_unset=True) reaches this branch without tool_calls and raises KeyError. Using include={"task_id"} instead fails on the missing rollout_id. The v2 bundle serializer similarly assumes records is present. This affects the existing export sanitizer, which uses exclude_unset=True. For v1 bundles, project_bundle_v1 has the opposite problem: it adds records=[] even when the caller explicitly excluded that field.
Correction: Transform only fields retained by the serialization handler. Replace the v2 trajectory transformation with:
if "rollout_id" in result:
result["capture_key"] = result.pop("rollout_id")
if "tool_calls" in result:
result["tool_calls"] = [
canonical_tool_record(tool, include_join_evidence=False)
for tool in result["tool_calls"]
]Apply the same membership guard to the records transformation in AgentObservationBundle.serialize_version and project_bundle_v1. The invariant is that version conversion neither requires omitted fields nor reintroduces excluded fields. These are source-traced suggestions; verify them before adopting them.
Preservation and acceptance: Add proposed checks for both versions: a sparse trajectory and bundle must serialize with exclude_unset=True; include={"task_id"} must return only the trajectory task ID; AgentObservationBundle(source="test").model_dump(include={"source"}) must return exactly {"source":"test"}. Pass a sparse v2 trajectory through _rollout_for_export and assert successful sanitization. Full v2 dumps must still emit capture_key, omit rollout_id, and canonicalize tool fields. Retain the concrete v1 assertions in the existing sparse/exclusion tests.
There was a problem hiding this comment.
Fixed in 57c8706. Both envelope serializers transform only fields retained by the serialization handler, and v1 bundle projection no longer adds an excluded records field. Regression coverage includes sparse and filtered v1/v2 envelopes and full dumps.
| raw_attempts = defaultdict(set) | ||
| for observed in observations: | ||
| if not isinstance(observed, TrajectoryToolCall): | ||
| raw_attempts[(observed.invocation_id, observed.tool_call_id)].add(observed.span_id) |
There was a problem hiding this comment.
BLOCKER — Count producer-qualified attempts before assigning conversation output.
Failure: Normalize two raw observations with the same invocation_id="root", tool_call_id="request", and span_id="local", but different producer_id values. Supply one root conversation result for request. This set counts only the raw span, so its size is one and both observations receive that result. Their imported identities correctly remain distinct because the producers differ. The final output ambiguity check then emits no ambiguity gap because both outputs are already populated. One unattributable result has silently become evidence for two executions.
Correction: Include effective producer identity in this attempt count. Replace the insertion with:
raw_attempts[(observed.invocation_id, observed.tool_call_id)].add(
(observed.producer_id or producer_id, observed.span_id)
)This makes the existing preassignment guard leave both outputs unknown; the final one-to-many join can then emit tool_output_ambiguous. The invariant is that a conversation result cannot be assigned to several distinct imported attempts merely because their producers reused a local span. Verify this suggested change before adopting it.
Preservation and acceptance: Extend the distinct-producer test with the single conversation output above. Assert two distinct retained spans, both outputs None, and an attempt-scoped tool_output_ambiguous gap for each. A repeated snapshot with the same producer and local span must still deduplicate to one attempt and receive its unique output. A single raw attempt must retain the existing partial-instance output behavior, and canonical snapshots must retain the completed "done" output in both orders in the snapshot regression test. These checks are proposed, not executed.
There was a problem hiding this comment.
Fixed in 57c8706. Output preassignment now counts effective producer identity together with the local span. Distinct producers retain separate attempts with unknown outputs and scoped ambiguity gaps; replay from the same producer still deduplicates and retains its unique output.
| for invocation in self.invocations: | ||
| owners[invocation.invocation_id].add(invocation.agent_instance) | ||
| for reference in invocation.model_calls: | ||
| call = calls.get(reference.model_call_id) |
There was a problem hiding this comment.
RISK — Resolve response-pair references before checking model ownership.
Failure: Construct a v2 trajectory with an invocation attributed to "other" and model_calls=[ModelCallRef(model_ref=policy, response_id="resp")]. Include one captured call attributed to "assistant" with matching policy/response metadata. Validation accepts the contradictory ownership because this lookup uses the absent model_call_id and returns None. The same association expressed with an explicit model-call ID is correctly rejected. Both reference forms are supported by ModelCallRef, and the performance aggregator resolves response pairs and counts their calls for the invocation.
Correction: Resolve every supplied qualifier before applying the existing ownership check. A localized replacement for this lookup is:
candidates = self.model_calls
if reference.model_call_id is not None:
candidate = calls.get(reference.model_call_id)
candidates = [] if candidate is None else [candidate]
matches = [
candidate
for candidate in candidates
if (reference.model_ref is None or candidate.response_metadata.model_ref == reference.model_ref)
and (reference.response_id is None or candidate.response_metadata.response_id == reference.response_id)
]
call = matches[0] if len(matches) == 1 else NoneThe invariant is that both supported exact-reference forms enforce the same known-instance constraint without selecting an arbitrary ambiguous match. This suggestion retains the indexed explicit-ID path; verify it before adopting it.
Preservation and acceptance: Extend the invocation-instance test with the response-pair-only setup and assert rejection with owning invocation. The matching-instance case and cases with either instance unknown must validate. Keep the existing explicit-ID mismatch rejection. Unmatched or ambiguous response pairs must not select an owner, and contradictory supplied response/model qualifiers must not resolve an ID reference. These are proposed checks; I haven’t run them.
There was a problem hiding this comment.
Fixed in 57c8706. Ownership validation resolves all supplied model-call qualifiers and checks the instance only for a unique match. Regressions cover response pairs, explicit IDs, contradictory qualifiers, ambiguous/unmatched references and unknown instances.
e2b26ad to
57c8706
Compare
|
/ok to test 57c8706 |
|
🌿 Preview your docs: https://nvidia-preview-pr-5660-bfd93b7968fd.docs.buildwithfern.com/nemo/gym |
|
/review |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP
I verified by source inspection that all three findings from the previous bot review are fixed:
- Envelope serialization guards omitted fields, and v1 bundle projection no longer adds an excluded
recordsfield. - Output preassignment counts producer-qualified attempts. Distinct producers leave the shared result ambiguous; same-producer replay still deduplicates and retains its unique output.
- Ownership validation resolves response-pair and explicit-ID references using every supplied qualifier, then checks ownership only for a unique match.
The regression tests assert these outcomes. I also traced the directly affected export and performance consumers and the existing preservation cases. No unresolved prior findings or eligible new regressions remain.
This was a static follow-up review; I didn't execute tests. The previously reviewed tree wasn't available, so I couldn't independently compare the complete inter-review delta. The review stayed focused on the reported fixes and their directly affected paths.
Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.
57c8706 to
491168e
Compare
|
/ok to test 491168e |
|
/review |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP
I verified by source inspection that the three previously reported failures remain fixed:
- Envelope serialization guards omitted fields, and v1 bundle projection leaves excluded
recordsabsent. - Output preassignment counts producer-qualified attempts. Distinct producers leave a shared result ambiguous; same-producer replay still deduplicates and retains its unique output.
- Ownership validation resolves both supported reference forms using every supplied qualifier and checks ownership only for a unique match.
The regression assertions cover these failures and their preservation cases. I also checked that completion-gap filtering requires a terminal outcome and a completion endpoint, and traced the directly affected export sanitizer and performance resolver. No unresolved prior findings or eligible new regressions were established.
This was a static follow-up review; I didn't execute tests. The previously reviewed tree wasn't available, so I couldn't independently compare the complete inter-review delta. One delegated report wasn't available; I independently inspected its ownership-validation topic. The review stayed within the prior fixes and their directly affected paths.
Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.
|
/review |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP
I verified by source inspection that the three previously reported failures remain fixed:
- Envelope serialization and trajectory serialization guard omitted fields. V1 bundle projection leaves excluded
recordsabsent. - Output preassignment counts producer-qualified attempts. Distinct producers leave a shared output ambiguous; same-producer replay still deduplicates and retains its unique output.
- Ownership validation resolves both reference forms using every supplied qualifier and checks ownership only for a unique match.
The regression assertions cover these failures and their preservation cases. I also traced the directly affected export sanitizer and performance resolver, and checked that completion-gap filtering still requires a terminal outcome and a completion endpoint. No unresolved prior findings or eligible new regressions were established.
This was a static follow-up review focused on the prior fixes and their directly affected paths; I didn't execute tests. The previously reviewed tree wasn't available, so I couldn't independently compare the complete inter-review delta.
Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.
marcromeyn
left a comment
There was a problem hiding this comment.
Thanks Frederic, re-checked at 8da94ea; suites pass for me (3148). The sparse serialization, producer-qualified attempts and response-pair ownership threads are fixed and covered in test_schema_review.py, and the earlier points hold at this layer: v2-only diagnostics are dropped from v1, snapshots advance status and output together, cycles are rejected at decode, and a terminal status keeps the missing-end gap until an end time arrives. Two things before I approve:
- Adam's question is still open (
tool_timing.py:381-385,trajectory_format.py:33-42).tool_failure_detail_unavailableis added for any failed, timed-out or cancelled attempt without afailure_id, from every producer, and_V2_ONLY_GAPSdoesn't drop it, while the v1 projection already discards failure ids and events (:110-111,:349). A normalized, otherwise complete failed tool ends up with this as its only v1 gap, which strict ATIF export rejects (atif_export.py:937). This layer's collector doesn't normalize yet, but with #5661 the collector produces it too, so the default v1 path would regress there. Dropping it in the v1 projection, keeping real coverage gaps, plus an enrich-then-strict-export test with a failed tool, would close it. - A reused conversation result can go to two attempts (
tool_timing.py:145,:157-158). The uniqueness check counts raw observations but not retained canonical attempts, so when a canonical attempt already owns a request id and a new raw attempt with the same id arrives, both get the same output and notool_output_ambiguousgap (that check at:523-526only looks at tools whose output is still empty). Repro: normalize raw spanawithroot/requestand output"result", then normalize the retained canonicalatogether with raw spanbon the same request id and conversation. Counting canonical and raw identities together (deduplicating snapshots of the same attempt) and a mixed canonical/raw test would pin it.
Happy to approve once these two are in.
|
On point 1: I agree that normalization adds We previously chose to retain the missing-failure-detail diagnostic and defer support for exporting failed tool executions to the exporter work. A successful enrich-then-strict-export regression for that case needs that exporter change as well. I would keep that policy here; if there is a previously exportable concrete fixture that this layer breaks for another reason, I am happy to check it. Point 2 is confirmed: the mixed retained/raw case duplicates the conversation output. I am fixing the attribution check to count qualified identities across both sets, with regression coverage for both ordering and same-attempt replay. |
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
8da94ea to
bfd93b7
Compare
|
/ok to test bfd93b7 |
|
/review |
|
Point 2 is fixed in bfd93b7. Output attribution now counts qualified identities across retained canonical and raw attempts before assigning conversation output. Repeated snapshots of the same attempt count once. In the mixed a/b example, a retains its output and b remains unresolved with tool_output_ambiguous. Regressions cover both input orders and same-attempt replay. |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP
No unresolved prior findings or eligible new regressions were established in this follow-up.
I verified by source inspection that:
- The envelope serializers and trajectory serializer still guard omitted fields, and v1 bundle projection leaves excluded
recordsabsent. - Output preassignment now counts retained canonical and incoming raw attempts together. The regression assertions cover both orders, preserve the retained output, leave a distinct incoming attempt ambiguous, and retain same-attempt replay.
- Ownership validation still applies every supplied qualifier to both reference forms and checks ownership only for a unique match.
The failure-detail gap doesn't establish the claimed strict-export regression: non-completed tool executions are independently rejected in both the head exporter and the base exporter. Removing the gap alone wouldn't make that failed-tool case exportable.
I traced the affected sanitizer and performance resolver and checked the relevant preservation assertions. This was a static follow-up review; I didn't execute tests. The previously reviewed tree wasn't available, so I couldn't independently compare the complete inter-review delta. One delegated report wasn't available; I independently completed its output-attribution inspection.
Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42239786681 |
Adds the v2 tool-timing contract and explicit v1 projection. V2 retains attempt identity, native call IDs, requesting-model links, qualified timing intervals, failure links, and gaps. Existing readers can use
project_trajectory_v1(...)during migration.Timing boundaries describe the measured endpoints: lifecycle, executor, transcript, or event receipt. Artifact storage is independent of that meaning. Projection preserves OpenCode’s established v1
artifactsource label for its lifecycle instrumentation.Distinct attempts that reuse a native call ID remain separate, including in v1 projection when restoring that ID would collide. This intentional compatibility correction avoids merging executions; replaying one persisted attempt remains idempotent. Missing failure-detail evidence remains explicit. Failed-tool ATIF export was already unsupported and is a separate exporter concern.
Part 1 of #5660 → #5661 → #5507. The stack provides one end-to-end implementation, split for review. No separate public issue is needed; the implementation and compatibility discussion are tracked in these PRs.