Skip to content

feat(telemetry): derive tool spans and metrics from v2 trajectory records - #6006

Merged
yfw merged 3 commits into
mainfrom
romeyn/tool-span-metrics
Nov 17, 2026
Merged

yfw merged 3 commits into
mainfrom
romeyn/tool-span-metrics

Conversation

@marcromeyn

Copy link
Copy Markdown
Contributor

What and why

#5660, #5661 and #5507 give a rollout's v2 trajectory record each tool execution the harness observed, with qualified timings (executor, lifecycle, transcript, event_receipt), status, and parent links. Nothing turned that into telemetry, so a dashboard could not see which tools a run spends its time in.

nemo_gym/rollouts/tool_telemetry.py reads those records after a rollout finishes. Nothing is measured again:

  • gym.tool.calls_total counts executions by nemo.gym.tool.name and nemo.gym.tool.status.
  • gym.tool.duration_ms records each recorded duration, with nemo.gym.tool.timing_boundary saying what it measures. A boundary without a duration is left out, never written as 0.
  • With traces on, each execution with both recorded endpoints becomes a gym.tool span under the rollout's dispatch span, or under its parent tool's span. It takes the boundary closest to the execution (executor first).
  • Parents are resolved within an agent instance, since a span id is unique only there. A tool whose parent has no endpoints goes under its nearest ancestor with a span, whatever order the record lists them in. Long parent chains are walked iteratively.
  • Tool names are free-form, so the metrics keep the first 64 names a process sees and count the rest, and over-long names, as other; spans keep every name.
  • It runs where the collection driver already records rollout health telemetry (off the event loop), only when metrics or the dispatch span are active, and never fails a rollout.

Part of GYM-1187 (tool-call spans), under GYM-1170. The per-run tool summary is not in this PR.

Validation

  • tests/unit_tests/rollouts/test_tool_telemetry.py: counts and per-boundary durations, missing endpoints and durations left out, spans under the dispatch span and under parent tools, parents within an agent instance, a parent without endpoints in either record order, a long parent chain, v1 records ignored, bounded tool names, and an invalid record never raising.
  • pytest tests/unit_tests/{telemetry,rollouts,trajectory} tests/unit_tests/test_fern_docs_links.py: 3,996 passed, 12 skipped.
  • pre-commit run --files on the changed files: passed.

Rollout evidence

N/A: telemetry only. No change to rollouts, rows, rewards or agent behaviour; with metrics and traces off, the new code does not run.

Compatibility

New metrics (gym.tool.calls_total, gym.tool.duration_ms) and a new span (gym.tool) when telemetry is on. Nothing else changes.

…ords

Each tool execution a harness records becomes a gym.tool span under the rollout's
dispatch span (or its parent tool's span) and a count and per-boundary duration,
without measuring again. Missing endpoints or durations are left out, not zero,
and metric tool names are bounded.

Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
…ns iteratively

A span id is unique within its agent instance, so two instances sharing one no
longer collapse into one span or borrow each other's parent. Long parent chains
no longer hit the recursion limit, and over-long tool names count as other.

Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
…rder

A tool whose parent has no endpoints now goes under its nearest ancestor with a
span in every order the record can list them, instead of under that ancestor in
one order and under the dispatch span in the other.

Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 11, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@marcromeyn

Copy link
Copy Markdown
Contributor Author

/ok to test 6e8ee48

@github-actions

Copy link
Copy Markdown
Contributor

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP

One NOTE about the new tests' optional-dependency handling. I found no production correctness concerns after inspecting all five changed files and tracing trajectory validation, span parenting, metric recording, and collection gating.

This was a static review; I didn't execute tests or independently verify the validation reported in the PR.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

meter_provider.shutdown()


def test_each_timed_tool_becomes_a_span_with_its_recorded_endpoints_under_its_parent(telemetry) -> None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NOTE — Suggestion: gate the Lens-dependent attribute assertions

With the OpenTelemetry SDK installed but optional nemo-lens absent, the fixture still creates spans, but Gym's safe_set_span_attributes fallback deliberately writes no attributes. This test therefore raises KeyError when building finished; the agent-instance test and both nearest-ancestor parameter cases have the same dependency. This affects SDK-only test environments, not production rollout behavior. Existing span tests explicitly account for that fallback.

Could you use the existing requires_lens marker only on the three functions that require attributes? The minimal change in this file is:

 from nemo_gym.trajectory.trajectory_record import NG_TRAJECTORY_KEY
+from tests.unit_tests.telemetry.conftest import requires_lens

+@requires_lens
 def test_each_timed_tool_becomes_a_span_with_its_recorded_endpoints_under_its_parent(telemetry) -> None:

+@requires_lens
 def test_spans_are_keyed_and_parented_within_their_agent_instance(telemetry) -> None:

+@requires_lens
 @pytest.mark.parametrize("child_first", [False, True])
 def test_a_child_of_an_untimed_tool_goes_under_its_nearest_timed_ancestor_in_any_order(telemetry, child_first) -> None:

Preserve the production no-op fallback and leave the metric, malformed-record, missing-endpoint, and long-chain tests runnable without Lens; don't skip the whole module.

Proposed acceptance checks, not executed here: run this file with the SDK present and Lens absent—these four parameterized cases should skip and the remaining tests should pass. With the telemetry extra installed, all cases should run, retaining the recorded-endpoint assertions, three distinct instance-scoped spans, and child-to-grandparent parenting in both record orders. Please verify the suggested patch before adopting it.

@github-actions github-actions Bot added the sla:review-overdue Review response is over the one-business-day SLA label Nov 12, 2026
@yaoyu-33
yaoyu-33 requested review from gchlebus and removed request for cwing-nvidia November 16, 2026 00:15
@iguehring-nvidia
iguehring-nvidia removed their request for review November 16, 2026 09:43
@iguehring-nvidia

Copy link
Copy Markdown
Contributor

@lafrancef has all the context needed to review (and I see he approved). I unassigned myself from this PR. Please assign me again / comment here, if a review specifically from me is needed.

@github-actions github-actions Bot removed the sla:review-overdue Review response is over the one-business-day SLA label Nov 16, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added feature New capabilities, enhancements, or enablement work area:core Shared APIs, servers, telemetry, health, and registries labels Nov 16, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the ready-to-merge PR is approved, current, and waiting for final CI or merge label Nov 16, 2026
@yfw
yfw added this pull request to the merge queue Nov 17, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/43989150666

@github-actions github-actions Bot added the sla:review-overdue Review response is over the one-business-day SLA label Nov 17, 2026
Merged via the queue into main with commit ca2dbf7 Nov 17, 2026
133 of 142 checks passed
@yfw
yfw deleted the romeyn/tool-span-metrics branch November 17, 2026 00:52

This branch had an error being deployed

1 failed deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:core Shared APIs, servers, telemetry, health, and registries feature New capabilities, enhancements, or enablement work ready-to-merge PR is approved, current, and waiting for final CI or merge sla:review-overdue Review response is over the one-business-day SLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants