Repository navigation
feat(opencode): capture qualified tool timing - #5507
Conversation
|
/review |
|
/review |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
HOLD
I found two BLOCKERs in requesting-model attribution and the public v1 projection, plus 15 RISKs affecting snapshot merging, OpenCode identity, enrichment, and downstream consumers.
I completed the changed-file review and traced the affected capture, replay, session, performance, health, and conformance paths. This was a source-only review; I didn't execute tests. The suggested patches and acceptance checks below still need verification.
Inline comments (escaped v1):
-
nemo_gym/trajectory/tool_timing.pyline 274: BLOCKER — Restore the model-server constraint when merging replayed tools.Failure: Enrich a v2 observation with
source_message_id="message"andsource_model_ref=Abefore capture arrives, then enrich it again with capture. The canonical serializer removes input join evidence. This merge restoressource_message_idbut notsource_model_ref, so candidate selection searches every model server. If only B has a matching session/message/native ID, the tool silently links to B despite the producer specifying A. If both exist, a resolvable request becomes ambiguous.Correction: Add the missing field to this fill-only-when-unknown tuple:
"source_message_id", "source_model_ref",The invariant is that replay must not weaken the original requesting-call scope.
Preservation/acceptance: Proposed regression: perform those two enrichment passes with source model A. Captures from A and B must resolve only to A; capture from B alone must leave the link unset. Repeating enrichment must retain span IDs and resolved links. An observation genuinely lacking model scope must remain ambiguous when both calls match. Retain failed-retry filtering and repeated-native-ID disambiguation.
-
nemo_gym/trajectory/trajectory_format.pyline 73: BLOCKER — Project valid tools whose default discriminator was omitted.Failure: The new canonical-failure fixture validates as v2, but its tool omits
kind, which defaults to"tool_call". Passing that raw trajectory toproject_trajectory_v1takes the non-tool branch of project_observation_v1. It leavesspan_id,failure_id, andtimingsinside an envelope labeled v1, which the v1 decoder rejects. The advertised compatibility API therefore fails for a valid committed wire shape.Correction: Entries in
tool_callsare tools even without the serialized default:result["tool_calls"] = [ project_observation_v1({"kind": "tool_call", **record}) for record in result.get("tool_calls", []) ]Preservation/acceptance: Proposed regression: validate the fixture as v2, project its raw dictionary, and validate the result as v1. Expect
duration_ms == 10and nospan_id,failure_id, ortimings. Explicit-kind tools must retain the same single-boundary projection; invocation, model-call, and turn projection must remain unchanged. -
nemo_gym/trajectory/trajectory_format.pyline 35: RISK — Clear flat timing when several detailed boundaries exist.Failure: A valid tool can contain flat timing such as
[1, 2]withduration_ms=1000alongside lifecycle and executor measurements. This projector overwrites flat fields only for one measurement. With two, it removestimingsbut preserves the preexisting flat interval, presenting an arbitrary boundary to legacy readers. That contradicts the documented rule that multiple boundaries produce no flat interval.Correction: Immediately after the single-measurement branch, add:
elif len(timings) > 1: for key in _TOOL_FLAT_FIELDS: record[key] = None
Preservation/acceptance: Proposed checks: two detailed boundaries plus non-null flat fields must yield null flat endpoints, duration, and source. Zero detailed measurements must preserve a legacy flat interval. One lifecycle measurement must still project its endpoints/duration with
timing_source="harness". This keeps legacy evidence without implicitly choosing among detailed boundaries. -
nemo_gym/trajectory/tool_timing.pyline 284: RISK — Retain error details from a compatible completion snapshot.Failure: Normalize two observations of the same span: first
status="incomplete", error_type=None, thenstatus="failed", error_type="exit_7". The snapshot merge advances the status but never copieserror_type. The canonical failed attempt loses supplied failure evidence.Correction: After merging the outcome, fill missing error metadata only when the incoming outcome agrees with the retained one:
if existing.error_type is None and existing.status == tool.status: existing.error_type = tool.error_type
This avoids attaching a rejected terminal snapshot's error to a different retained outcome.
Preservation/acceptance: Proposed checks: incomplete → failed must produce one failed attempt with
error_type="exit_7"; a later null value must not erase it. Completed → contradictory failed must retain completed, emittool_outcome_conflict, and not adopt the failed error label. Distinct failed/completed retry spans must remain separate, and existing timing conflict checks must remain intact. -
nemo_gym/trajectory/tool_timing.pyline 281: RISK — Don't downgrade an observed incomplete execution to unknown.Failure: Normalize same-span observations in this order: explicit
status="incomplete", then defaultstatus="unknown"from a timing-only snapshot. This branch retains unknown. Because completion-gap generation checks only incomplete, the known unfinished execution also loses its completion gap.Correction: Replace the first condition with:
if existing.status == "unknown" or ( existing.status == "incomplete" and tool.status != "unknown" ):Keep the assignment and terminal-conflict branch below it. Missing outcome evidence must not erase an explicit unfinished outcome.
Preservation/acceptance: Proposed transition checks: incomplete → unknown retains incomplete and a span-scoped
tool_completion_unavailablegap; unknown → incomplete upgrades; incomplete → completed/failed advances; terminal → unknown remains terminal; contradictory terminal outcomes still emittool_outcome_conflict. -
nemo_gym/trajectory/tool_timing.pyline 78: RISK — Enforce every supplied requesting-call qualifier.Failure: Supply
model_call_ref={model_call_id:"m", response_id:"other"}while captured m has response ID"actual"and contains the native tool ID._matches_referenceaccepts m solely by ID. A contradictory model-server qualifier is ignored too, unlike the existing shared matcher.Correction: Replace this return with:
return ( call.model_call_id == reference.model_call_id and (reference.model_ref is None or call.response_metadata.model_ref == reference.model_ref) and (reference.response_id is None or call.response_metadata.response_id == reference.response_id) )Also preserve these qualifiers at reference construction: when
tool.model_call_idandmodel_call_refcoexist, require their IDs to agree rather than replacing the richer reference with an ID-only one.Preservation/acceptance: Proposed checks: ID-only and fully matching references resolve; changing only response ID, model server, or the second supplied model-call ID leaves the link unset with a scoped diagnostic. Response-ID-plus-model matching must still work. Native IDs must verify an exact reference, never select one alone.
-
responses_api_agents/opencode_agent/artifacts.pyline 110: RISK — Derive conversation IDs from immutable part identity.Failure: Parse one running part p0 with native
callID="c", retain its enriched trajectory, then complete p0 and append p1 with the same native ID. This mapping changes p0'stool_call_idfromctoc:p0, although its span stays unchanged. The normalizer rejects the completed snapshot as an identity conflict, leaving the retained attempt incomplete. Counting before slicing doesn't prevent changes as the database grows. The encoding also collides with a singleton whose literal native ID isc:p0.Correction: Replace the Counter-dependent mapping with a persisted-part mapping, for example:
normalized_ids = {row["id"]: f"call:{row['id']}" for row in tool_parts}
Keep native IDs in
native_tool_call_id. Apply the same mapping to observation conversations, turns, child-spawn references, and compaction references; update singleton-ID assertions accordingly.Preservation/acceptance: Proposed regression: enrich running p0, append the duplicate and complete p0, then re-enrich. Expect two spans, p0 completed with its end/duration, and no identity conflict. Full and filtered parses must retain identical IDs; the collision example must remain distinct. Preserve exact requesting-model joins through native IDs and incomplete status for unfinished tools.
-
responses_api_agents/opencode_agent/artifacts.pyline 150: RISK — Use the tool-ID mapping for earlier compaction boundaries.Failure: An earlier activation contains two tool parts with native ID
c, emitted as qualified conversation IDs. A later compaction'stail_start_idpoints to the first tool's assistant message. This earlier-message branch records rawc, sofirst_kept_item_idnames neither emitted item. A full parse instead uses the mapped ID from the active-message branch. The boundary changes with slicing and becomes dangling without a gap.Correction: Use the existing complete mapping here:
item_id = normalized_ids[row["id"]]This dependent change remains necessary if the mapping's encoding is corrected separately.
Preservation/acceptance: Extend the cross-activation compaction case with repeated native tool IDs. Full and sliced parses must name the same first retained tool, equal to its earlier emitted conversation ID. Text/reasoning boundaries must remain valid; a user-only boundary must remain unresolved.
-
nemo_gym/rollouts/rollout_collection.pyline 1364: RISK — Enforce payload omission after failed projection too.Failure:
include_payloads=Falsenow merges capture with payloads, but strips them only when a trajectory exists. A call withlatency_total_ms=-1makes projection fail and leaves its request/response in capture, as shown by the existing fallback test. Collection also selects omission formultimodal_history_redacted, so projection failure can restore intentionally excluded payloads.Correction: After the projection attempt, independently enforce capture omission:
if not include_payloads: capture = result.get("ng_model_call_capture") calls = capture.get("calls") if isinstance(capture, dict) else None for call in calls if isinstance(calls, list) else []: if isinstance(call, dict): for key in ("request", "response", "request_raw", "response_raw"): call.pop(key, None)
Keep trajectory omission and verification gaps. Don't reuse
_strip_capture_payloadshere because it also removes verifier evidence needed after failure.Preservation/acceptance: Proposed checks: omission plus the negative-latency fixture leaves no capture payload fields despite projection failure. Include-true fallback must retain them. Successful v2 joins must remain exact and gain the payload-unavailable gap; reward, harness response, and failure verifier evidence must survive.
-
nemo_gym/rollouts/rollout_collection.pyline 5437: RISK — Unwrap materialized episode responses at the shared enrichment boundary.Failure: Materialized
/runreturns{episode_id, task_id, result:{ng_agent_observations,...}}, as the environment producer shows. This callback enriches the outer envelope. Without capture, no evidence keys are found and no trajectory appears; with capture, the outer trajectory misses nested agent evidence. Full collection unwraps first.Correction: At the beginning of
enrich_run_result, before stamping or merging, normalize recognized materialized envelopes in place:if _materialized_taskset(row) is not None and _is_episode_response(result): normalized = _native_episode_record(row, result) result.clear() result.update(normalized)
This also covers the documented manual enrichment path and retains episode-identity validation.
Preservation/acceptance: Proposed checks: a materialized success containing one nested invocation/tool must produce equivalent canonical evidence through opt-in
run_examples, manual enrichment, and full collection. A handled failure must retain observations without inventing reward. Existing flat results must still work; enrichment-disabled calls must retain the raw envelope unchanged. -
nemo_gym/rollouts/rollout_collection.pyline 1271: RISK — Count performance evidence by the full invocation identity.Failure: V2 permits a/root and b/root, but
_build_ng_perfkeys counters only byinvocation_id. With those two owners, one explicit turn each, and one tool owned by a/root, it reports four turns and two tools instead of two and one, distorting efficiency and coverage metrics.Correction: Use
(agent_instance, invocation_id)for the owned-call increment and all tool/turn counters and lookups. The replacements include:owned_calls_by_invocation[(invocation.agent_instance, invocation.invocation_id)] += 1 tool_calls_by_invocation = Counter((t.agent_instance, t.invocation_id) for t in trajectory.tool_calls) num_tool_calls = sum(tool_calls_by_invocation[(i.agent_instance, i.invocation_id)] for i in trajectory.invocations) turns_by_invocation = Counter((t.agent_instance, t.invocation_id) for t in trajectory.turns) num_turns = sum( turns_by_invocation[(i.agent_instance, i.invocation_id)] or owned_calls_by_invocation[(i.agent_instance, i.invocation_id)] or 1 for i in trajectory.invocations )
Preservation/acceptance: Proposed two-instance case must report two turns, one tool, and coverage 1.0 for two uniquely owned calls. Preserve call-position deduplication, including the shared-call case's two turns/once-counted tokens/0.5 coverage. Preserve per-owner explicit-turn → owned-call → one fallback, including the existing hybrid result of seven turns and coverage 4/7. Legacy null-instance owners must retain their counts.
-
nemo_gym/rollouts/rollout_collection.pyline 840: RISK — Keep instance ownership when reconstructing turns.Failure: For explicit owners a/root → call-a and b/root → call-b,
_turns_from_model_callskeeps only the stringrootand emits unattributed turns numbered 1 and 2. Expected identities are(a,root,1)and(b,root,1)with independent step counts. The new contract allows these owners, so supplied attribution is lost on the no-producer-trajectory path.Correction: Store the owning
AgentInvocation, not its string ID, in both reference indexes. Resolveownerfrom those indexes, usekey=(owner.agent_instance, owner.invocation_id)for both counters, and construct each turn with:invocation_id=owner.invocation_id, agent_instance=owner.agent_instance, turn_no=turn_counts[key], step_count=tool_counts[key],
Increment
turn_counts[key]before construction andtool_counts[key]afterward, as today.Preservation/acceptance: Proposed enrichment test: two v2 owners with explicit distinct call references and successful responses, no producer trajectory, must yield exactly those two composite turn identities and independent tool-step counts. Preserve legacy single-owner numbering, explicit-reference-only ownership, verifier exclusion, and HTTP-error filtering. Don't infer owners from timestamps or list order.
-
nemo_gym/rollouts/health/timeouts.pyline 360: RISK — Validate clock domains before hashing them.Failure: Two timed spans with domains
[]and"clock"reach this set comprehension and raiseTypeError. The timing reader promises malformed records never raise, but evaluate_record invokes it outside the per-check guard, so one malformed row can stop the health report instead of leaving its union unknown.Correction: At both
_model_call_spanand_tool_call_span, replace the unchecked domain argument with:clock_domain=( domain if isinstance(domain := call.get("clock_domain", "legacy"), str) and domain else None ),
Only nonempty strings establish comparable domains. Keep the absent-field legacy fallback; v2 already inserts explicit
Nonefor missing provenance.Preservation/acceptance: Proposed checks: list, dict, and empty-string domains on model calls and v2 tool measurements must not raise through the reader or full report; multi-span unions involving them must be unknown. Equal valid domains must still union overlaps correctly. Different or missing v2 domains must stay unknown, while existing legacy policy/tool splits remain measured.
-
nemo_gym/rollouts/harness_capabilities/checker.pyline 853: RISK — Resolve v2 references using composite invocation ownership.Failure: A valid v2 trajectory with a/root and b/root, each carrying its correctly attributed tool and turn, fails
tools.invocation_targetandsteps.invocation_target: each string ID matches twice. Request lookup, parent traversal, and step-owner comparison retain the same string-only assumption. The PR now admits v2's capture key, but these checks falsely reject its supported owner identities.Correction: Make invocation-target, parent, request, and step-owner resolution instance-aware for v2. An explicit instance must constrain resolution to that instance; missing attribution must remain unresolved when several owners match. Carry composite identities in the model-call owner bookkeeping too. Several checks share this assumption, so a one-line predicate replacement isn't a complete fix.
Preservation/acceptance: Proposed complete two-instance fixture: reused invocation IDs with distinct calls/tools and consistent instances must pass target, request, parent, and step-owner checks. Remove attribution from an ambiguous reference or supply the wrong instance and expect failure. Preserve legacy single-instance resolution, genuine conflicting call-owner rejection, and missing/cyclic-parent rejection.
-
nemo_gym/rollouts/rollout_collection.pyline 3167: RISK — Settle enrichment failures per drained outcome.Failure:
aclose_and_draintransfers completed tasks out of the underlying iterator, then awaits enrichment. If one enrichment raises—for example anOSErrorduring requested capture deletion—the entire drain raises before returning any futures. Unrelated completed outcomes have already been transferred and a second drain returns nothing, violating the settled-future contract.Correction: Replace this successful-task branch with per-outcome exception settlement:
result = task.result() try: if self._enrich is not None: await self._enrich(result) except Exception as error: outcome.set_exception(error) outcome.exception() else: outcome.set_result((result.row, result.result))
Keep cancellation separate and retain retrieval of stored exceptions.
Preservation/acceptance: Proposed regression: drain two completed rollouts with enrichment raising
OSErrorfor the first and succeeding for the second. Expect two settled futures: the first raises that error and the second yields its row/result. Repeated drain must be empty. Preserve original dispatch exceptions, cancelled outcomes, and the guarantee that draining admits no new work. -
nemo_gym/trajectory/trajectory_record.pyline 833: RISK — Preserve span-scoped adapter gaps through capture joins.Failure: OpenCode now emits per-span timing diagnostics, but this unchanged deduplication key uses only code, invocation, and detail. Two tools in one invocation with missing starts and valid completion endpoints each emit
tool_start_unavailablewith null detail. Capture enrichment keeps only the last diagnostic. Normalization doesn't regenerate that adapter-specific code, so the first attempt loses its gap before the new import mapping sees it.Correction: Include both new scope fields in this key and in join_assistant_message_calls' equivalent deduplication:
(gap.code, gap.agent_instance, gap.invocation_id, gap.span_id, gap.detail)
Keep exact duplicate elimination; different attempts aren't duplicate diagnostics.
Preservation/acceptance: Proposed integration test: enrich a captured OpenCode bundle with two distinct tool parts whose starts are invalid but ends are valid. Each canonical span must retain its own start-unavailable gap through repeated enrichment. Exact duplicate gaps must still collapse. Ambiguous raw references must remain explicitly unresolved rather than being copied to multiple attempts.
-
nemo_gym/trajectory/trajectory_record.pyline 643: RISK — Don't assign an ambiguous captured session to the last agent instance.Failure: V2 permits invocations a/root explicitly referencing call-a and b/root explicitly referencing call-b. With both captures declaring
client_session_id="root", this dictionary retains only b/root. The automatic association loop then appends call-a to b/root before explicit-reference resolution; b/root keeps that incorrect extra reference with a conflict gap. This newly valid input therefore acquires contradictory ownership during capture enrichment, even though the producer supplied exact owners.Correction: Index all invocations for a session ID, and key each invocation's reference set by its composite identity. Honor existing exact call references before automatic session association. Infer an owner from
client_session_idonly when it identifies one invocation; otherwise leave an undeclared call unowned with a diagnostic instead of selecting the last record. The capture record currently lacks instance evidence, so it can't safely break that tie. A blanket tuple lookup alone wouldn't repair this inference path.Preservation/acceptance: Proposed regression: the two explicit owners above must retain only their original calls after repeated joins, with no manufactured ownership conflict. An undeclared call with the ambiguous session must remain unowned. Preserve the existing single-root automatic association, exact response/model-reference resolution, duplicate-reference handling, and cross-invocation compaction rejection.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
|
/review |
|
FYI: the base branch arajfer/run-records-on-store moved forward (merge only, no rewrite) to 4d91e9b, which brings in current main. Merge it into this branch when convenient. |
|
/review |
1 similar comment
|
/review |
|
FYI: the base branch arajfer/run-records-on-store moved forward (merge only, no rewrite) to cd8719b, which brings in current main. Merge it into this branch when convenient. |
|
FYI: the base branch arajfer/run-records-on-store moved forward again (merge only, no rewrite) to 4c31334, which brings in its own base branch's latest documentation and message fixes. Merge it into this branch when convenient. |
e2f1e26 to
0fb0100
Compare
|
/review |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP
No eligible findings remain from review 5755488220. I checked the prior failures against the current source and relevant regression assertions.
- Requesting-model qualifiers, snapshot outcomes, error details and span-scoped diagnostics are now preserved by the normalization and merge paths.
- The v1 projector handles omitted tool discriminators and clears flat timing when multiple boundaries exist. OpenCode now uses stable persisted-part IDs, including earlier compaction references.
- Collection now enforces payload omission after projection failure, unwraps materialized replies on the supported collection paths, and settles enrichment errors per drained outcome. Malformed clock domains are validated before union comparison.
- The earlier reused-invocation-ID failures are closed by an explicitly enforced unique-ID contract, not by implementing composite invocation identity. The documentation states that limitation.
This was a source-only follow-up covering the prior findings and directly affected paths; I didn't execute tests. The previously reviewed tree wasn't available, so I couldn't independently compare inter-review changes. This approval isn't a fresh whole-PR review.
Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.
|
/ok to test 0fb0100 |
|
/ok to test c59a1d8 |
|
/review |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP
No eligible findings in this source-only follow-up from review 5790966667.
I checked the current source and relevant regression assertions:
- Missing completion-time diagnostics still require end-time evidence before normalization or enrichment clears them. The regression assertions distinguish repeated enrichment without an end timestamp from later compatible completion timing.
- Direct HTTP
/runremains raw producer evidence, whilerun_examplesapplies the selected schema. The schema-precedence assertions cover direct and nested success/failure results while preserving primary responses. The strict ATIF regression still requires unchanged legacy export after repeated default enrichment. - Replay retains accepted completion output and stable attempt identity. The replay assertions cover both snapshot orders and prevent older partial output from replacing completion output.
- Runtime capture supplies separate clock domains per local launch or retained sandbox session. The busy-time assertions distinguish a shared-domain union from unknown provenance. The cancellation assertions retain partial tool output and the enclosing cancellation outcome without inventing a terminal tool outcome.
The previously reviewed tree isn't mounted, so I couldn't independently compare inter-review changes. The supplied evidence doesn't establish an eligible subsequent regression. This completes the scoped follow-up and directly affected consumer checks, not a fresh whole-PR review. All three delegated inspections finished; two report bodies weren't delivered, so I covered those assignments directly. I didn't execute tests.
Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.
c59a1d8 to
ecbdb53
Compare
b176ae6 to
6e0ebde
Compare
|
/ok to test ecbdb53 |
|
/review |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP
No eligible findings in this source-only follow-up from review 5797616444.
I checked the current source and relevant regression assertions:
- Missing completion-time diagnostics still require end-time evidence before normalization or enrichment clears them. The regression assertions distinguish repeated enrichment without an end timestamp from later compatible completion timing.
- Direct HTTP
/runremains raw producer evidence, whilerun_examplesapplies the selected schema. The schema-precedence assertions cover direct and nested success/failure results without changing primary responses. The strict ATIF regression requires unchanged legacy export after repeated default enrichment. - Replay retains accepted completion output and stable attempt identity. The replay assertions cover both snapshot orders and prevent older partial output from replacing completion output.
- Runtime capture supplies separate clock domains per local launch or retained sandbox session. The busy-time assertions distinguish a shared-domain union from unknown provenance. The cancellation assertions retain partial tool output and the enclosing cancellation outcome without inventing a terminal tool outcome.
The previously reviewed tree isn't mounted, so I couldn't independently compare inter-review changes. The supplied evidence doesn't establish an eligible subsequent regression. This completes the scoped follow-up and directly affected consumer checks, not a fresh whole-PR review. All three delegated inspections finished; two report bodies weren't delivered, so I covered those assignments directly. I didn't execute tests.
Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.
|
/ok to test 705abe9 |
|
/review |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP
No eligible findings in this source-only follow-up from review 5799894035.
I verified the relevant source and regression assertions:
- Missing completion-time diagnostics still require end-time evidence before normalization or enrichment clears them. The regression assertions distinguish repeated enrichment without an end timestamp from later compatible completion timing.
- Direct HTTP
/runremains raw producer evidence, whilerun_examplesapplies the selected schema. The schema-precedence assertions cover direct and nested success/failure results without changing primary responses. The strict ATIF regression requires unchanged legacy export after repeated default enrichment. - Replay retains accepted completion output and stable attempt identity. The replay assertions cover both snapshot orders and prevent older partial output from replacing completion output.
- Runtime capture assigns separate clock domains per local launch or retained sandbox session. The busy-time assertions distinguish a shared-domain union from unknown provenance. The cancellation assertions retain partial tool output and the enclosing cancellation outcome without inventing a terminal tool outcome.
The previously reviewed tree isn't mounted, so I couldn't independently compare inter-review changes. The supplied evidence doesn't establish an eligible subsequent regression. This completes the scoped follow-up and directly affected consumer checks, not a fresh whole-PR review. All three delegated inspections finished; I covered completion and replay directly because that report wasn't delivered. I didn't execute tests.
Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.
AdamRajfer
left a comment
There was a problem hiding this comment.
I read the OpenCode side of this against failure handling, the sandbox/session boundary, recovery and the timing readers, and ran the six OpenCode test files (539 pass) plus tests/unit_tests/trajectory and tests/unit_tests/rollouts. I also fed a few sessions through parse, enrich and read_rollout_timing in both schemas: a reused native call ID, a tool still running at the stop, end before start, a bash exit 1. Timings stay per attempt and per clock domain, a Gym-cut run keeps the timeout/cancel outcome on the sandbox observation and the invocation rather than on the tool, and nothing in the capture path touches a failure event or a score. One inline note, otherwise this reads fine to me.
|
/ok to test f814495 |
|
/review |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP
No eligible findings in this source-only follow-up from review 5801657351.
I checked the ownership fix and the earlier preservation guarantees:
- Local collection now passes a fresh agent-instance ID through the parser call. Retained sandbox collection uses the session-owned ID. The parser assigns that ID to both tool observations and invocations. The close regression assertions check matching owners and an unchanged repeated-close response.
- Missing completion-time diagnostics still require end-time evidence before normalization or enrichment clears them. The regression assertions distinguish repeated enrichment without an end timestamp from later compatible completion timing. The replay assertions retain completed output and stable attempt identity without accepting stale partial output.
- Caller-facing schema selection remains separate from raw HTTP producer evidence. The schema assertions cover explicit/configured precedence, nested success/failure evidence, and unchanged primary responses. The strict ATIF regression requires unchanged legacy export after repeated default enrichment.
- The busy-time assertions still distinguish a shared-clock union from unknown provenance. The cancellation assertions retain partial output and the enclosing cancellation outcome without inventing a terminal tool outcome.
The previously reviewed tree isn't mounted, so I couldn't independently compare inter-review changes. The supplied evidence establishes no eligible subsequent regression. This completes the scoped follow-up and directly affected consumer checks, not a fresh whole-PR review. All three delegated inspections finished; two report bodies weren't delivered, so I covered their local-launch and retained-session assignments directly. I inspected source and test assertions but didn't execute tests.
Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
|
/ok to test 4935a69 |
|
/review |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP
No eligible findings in this source-only follow-up from review 5803438539.
I checked the current source and relevant regression assertions:
- Ownership remains explicit. Local collection passes a fresh agent-instance ID, while sandbox collection passes the session-owned ID. The parser assigns it to tools and invocations. The close assertions require matching owners and an unchanged repeated-close response.
- Missing completion-time gaps still require end-time evidence before normalization or enrichment clears them. The regression assertions distinguish repeated enrichment without an end timestamp from later compatible timing. The replay assertions retain completed output and stable identity without accepting stale partial output.
- Caller-facing schema selection remains separate from raw HTTP producer evidence. The schema assertions cover explicit/configured precedence, nested success/failure evidence and unchanged primary responses. The strict ATIF regression requires unchanged legacy export after repeated default enrichment.
- The busy-time assertions distinguish a shared-clock union from unknown provenance. The cancellation assertions retain partial output and the enclosing cancellation outcome without inventing a terminal tool outcome.
The previously reviewed tree isn't mounted, so I couldn't independently compare inter-review changes. The supplied evidence establishes no eligible subsequent regression. This completes the scoped follow-up and directly affected consumer checks, not a fresh whole-PR review. All three delegated inspections finished; two report bodies weren't delivered, so I covered their local-launch and retained-session topics directly. I inspected source and test assertions but didn't execute tests.
Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42824943470 |
Adds qualified tool timing from OpenCode’s persisted tool parts. Select
trajectory_schema_version: "2.0"in collection configuration orrun_examples(..., trajectory_schema_version="2.0")to retain attempt IDs, requesting-model links, lifecycle measurements, and observability gaps.OpenCode’s persisted
state.time.start/endvalues measure lifecycle endpoints. Database storage does not make their boundarytranscript; v1 projection retains the existingartifactsource label. Runtime collection supplies one wall-clock domain and agent instance ID per local launch or retained sandbox session, allowing tool busy-time unions and stable ownership across session snapshots. Standalone imports without clock-owner evidence remain unknown. Model capture uses Gym model-server timestamps and cannot share the OpenCode clock domain.Started tools without terminal evidence stay
incomplete, including after an enclosing timeout/cancellation. The enclosing outcome remains available to health checks. Distinct persisted parts reusing a provider call ID remain separate attempts rather than being merged, including for v1 consumers; replaying the same part remains idempotent. Failed-tool ATIF export is a pre-existing limitation, also retained by #5510.The rollout-evidence guide documents current harness mappings and migration toward semantic boundaries wherever instrumentation supports them. Legacy delivery/source labels alone do not prove executor or lifecycle scope. Pi’s legacy upgrade mapping and model-call clock-domain population remain explicit follow-ups.
Direct HTTP
/runreturns raw v2 producer observations; collection performs the selected v1/v2 projection. A terminal tool status alone does not clear a missing-completion-time diagnostic; end-time evidence is required.Schema #5660 is merged. This PR follows #5661; merge #5661 first.