Repository navigation
feat(telemetry): derive tool spans and metrics from v2 trajectory records - #6006
Conversation
…ords Each tool execution a harness records becomes a gym.tool span under the rollout's dispatch span (or its parent tool's span) and a count and per-boundary duration, without measuring again. Missing endpoints or durations are left out, not zero, and metric tool names are bounded. Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
…ns iteratively A span id is unique within its agent instance, so two instances sharing one no longer collapse into one span or borrow each other's parent. Long parent chains no longer hit the recursion limit, and over-long tool names count as other. Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
…rder A tool whose parent has no endpoints now goes under its nearest ancestor with a span in every order the record can list them, instead of under that ancestor in one order and under the dispatch span in the other. Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
|
/ok to test 6e8ee48 |
|
🌿 Preview your docs: https://nvidia-preview-pr-6006-6e8ee48c5fe4.docs.buildwithfern.com/nemo/gym |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP
One NOTE about the new tests' optional-dependency handling. I found no production correctness concerns after inspecting all five changed files and tracing trajectory validation, span parenting, metric recording, and collection gating.
This was a static review; I didn't execute tests or independently verify the validation reported in the PR.
Note: formal approval is not available for this request, so this review is posted as a comment only.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
| meter_provider.shutdown() | ||
|
|
||
|
|
||
| def test_each_timed_tool_becomes_a_span_with_its_recorded_endpoints_under_its_parent(telemetry) -> None: |
There was a problem hiding this comment.
NOTE — Suggestion: gate the Lens-dependent attribute assertions
With the OpenTelemetry SDK installed but optional nemo-lens absent, the fixture still creates spans, but Gym's safe_set_span_attributes fallback deliberately writes no attributes. This test therefore raises KeyError when building finished; the agent-instance test and both nearest-ancestor parameter cases have the same dependency. This affects SDK-only test environments, not production rollout behavior. Existing span tests explicitly account for that fallback.
Could you use the existing requires_lens marker only on the three functions that require attributes? The minimal change in this file is:
from nemo_gym.trajectory.trajectory_record import NG_TRAJECTORY_KEY
+from tests.unit_tests.telemetry.conftest import requires_lens
+@requires_lens
def test_each_timed_tool_becomes_a_span_with_its_recorded_endpoints_under_its_parent(telemetry) -> None:
+@requires_lens
def test_spans_are_keyed_and_parented_within_their_agent_instance(telemetry) -> None:
+@requires_lens
@pytest.mark.parametrize("child_first", [False, True])
def test_a_child_of_an_untimed_tool_goes_under_its_nearest_timed_ancestor_in_any_order(telemetry, child_first) -> None:Preserve the production no-op fallback and leave the metric, malformed-record, missing-endpoint, and long-chain tests runnable without Lens; don't skip the whole module.
Proposed acceptance checks, not executed here: run this file with the SDK present and Lens absent—these four parameterized cases should skip and the remaining tests should pass. With the telemetry extra installed, all cases should run, retaining the recorded-endpoint assertions, three distinct instance-scoped spans, and child-to-grandparent parenting in both record orders. Please verify the suggested patch before adopting it.
|
@lafrancef has all the context needed to review (and I see he approved). I unassigned myself from this PR. Please assign me again / comment here, if a review specifically from me is needed. |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/43989150666 |
What and why
#5660, #5661 and #5507 give a rollout's v2 trajectory record each tool execution the harness observed, with qualified timings (
executor,lifecycle,transcript,event_receipt), status, and parent links. Nothing turned that into telemetry, so a dashboard could not see which tools a run spends its time in.nemo_gym/rollouts/tool_telemetry.pyreads those records after a rollout finishes. Nothing is measured again:gym.tool.calls_totalcounts executions bynemo.gym.tool.nameandnemo.gym.tool.status.gym.tool.duration_msrecords each recorded duration, withnemo.gym.tool.timing_boundarysaying what it measures. A boundary without a duration is left out, never written as 0.gym.toolspan under the rollout's dispatch span, or under its parent tool's span. It takes the boundary closest to the execution (executor first).other; spans keep every name.Part of GYM-1187 (tool-call spans), under GYM-1170. The per-run tool summary is not in this PR.
Validation
tests/unit_tests/rollouts/test_tool_telemetry.py: counts and per-boundary durations, missing endpoints and durations left out, spans under the dispatch span and under parent tools, parents within an agent instance, a parent without endpoints in either record order, a long parent chain, v1 records ignored, bounded tool names, and an invalid record never raising.pytest tests/unit_tests/{telemetry,rollouts,trajectory} tests/unit_tests/test_fern_docs_links.py: 3,996 passed, 12 skipped.pre-commit run --fileson the changed files: passed.Rollout evidence
N/A: telemetry only. No change to rollouts, rows, rewards or agent behaviour; with metrics and traces off, the new code does not run.
Compatibility
New metrics (
gym.tool.calls_total,gym.tool.duration_ms) and a new span (gym.tool) when telemetry is on. Nothing else changes.