Skip to content

feat(telemetry): report event-loop, process, request, driver, and resources-operation metrics - #4474

Merged
ananthsub merged 10 commits into
mainfrom
ananthsub/runtime-telemetry
Oct 14, 2026
Merged

ananthsub merged 10 commits into
mainfrom
ananthsub/runtime-telemetry

Conversation

@ananthsub

@ananthsub ananthsub commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Stack: telemetry lands before the optimizations, so the optimizations can be measured with it.

  1. feat(telemetry): report event-loop, process, request, driver, and resources-operation metrics #4474: event-loop, process, request, driver, and resources-operation metrics (base main)
  2. perf(telemetry): drop instrumentation that exports nothing #4481: drop instrumentation that exports nothing (base feat(telemetry): report event-loop, process, request, driver, and resources-operation metrics #4474)
  3. perf(server): replace HTTP middleware wrappers with ASGI middleware #2890: replace the HTTP middleware wrappers with ASGI (base perf(telemetry): drop instrumentation that exports nothing #4481)

What changed

A Gym server answers every request on one asyncio event loop. When synchronous work holds that loop, every other request waits, even though no single request looks slow, and nothing Gym exported showed it. This PR adds the event-loop, process, request, HTTP client, rollout-driver, rollout-shape, and resources-operation metrics from #3155. They run whenever telemetry exports metrics, except the event-loop task count, which is opt-in.

  • Event loop and process, from every server process, the head server, and the rollout driver: gym.event_loop.lag_ms, gym.event_loop.tasks, gym.gc.pause_duration_ms, gym.gc.collections_total, gym.gc.collected_total, gym.gc.uncollectable_total, gym.gc.count, process.cpu.time, process.memory.usage, process.thread.count, and process.open_file_descriptor.count. Each process also reports process.pid, so the Uvicorn workers of one server stay distinguishable.
  • Server requests, counted by the outermost middleware: gym.server.requests.in_flight, gym.server.requests.cancelled_total, gym.server.requests.unhandled_exceptions_total, and gym.server.responses_total by http.response.status_code, which counts handled errors such as an HTTPException as their 4xx or 5xx.
  • HTTP client: gym.http.client.requests.in_flight for outbound calls still waiting for their response, and gym.http.connection_pool.connect_duration_ms for each new pooled connection, next to the existing queue-wait metric.
  • Rollout driver: gym.driver.rollouts.offered_total, gym.driver.rollouts.resident, and gym.driver.rollouts.active, plus gym.driver.admission_duration_ms, gym.driver.output_write_duration_ms, gym.driver.output_write_bytes, and gym.driver.materialize_duration_ms. A standalone gym eval run --no-serve driver now sets up its own telemetry, which also lets it export the existing gym.job span.
  • Rollout shape, once per completed rollout: gym.driver.rollout.input_items, gym.driver.rollout.output_items, gym.driver.rollout.turns, and gym.driver.rollout.tool_calls. Turns and tool calls come from ng_perf when rollout observability produced it.
  • A shared resources-operation contract, nemo_gym.telemetry.resources_operations.resources_operation, which reports how long limited work queued (gym.resources.operation.queue_duration_ms), how long it ran (gym.resources.operation.duration_ms), and how it ended (nemo.gym.resources.operation.outcome). math_with_judge uses it for its SymPy verifier subprocesses and marks timeouts and crashes.
  • Naming follows the conventions of Gym's existing metrics: durations in milliseconds ending in _ms, counters ending in _total, and Gym attributes under nemo.gym.. The process.* metrics keep their OpenTelemetry semantic-convention names, seconds, and cpu.mode, so standard dashboards recognize them.

The design keeps the work off the request path, which is why the metrics need no switch of their own. The event loop runs one sampling coroutine per interval. Per-request and per-rollout counts are plain integer updates that the exporter thread reads through observable instruments, so a request makes no OpenTelemetry call. The GC callback only buffers timestamps and totals, because it runs inside a collection. Instruments are registered once per meter, the GC callback is removed with the last observer, and the last observer to stop exports the final totals, so a short-lived driver still reports its counts.

flowchart TB
    subgraph onloop["On the event loop"]
        requests["Each request: outermost middleware and server_utils.request"]
        rollouts["Each rollout: driver dispatch and output write"]
        sampler["Once per interval: sampler coroutine"]
    end
    totals["Plain integer totals"]
    pauses["Bounded GC pause buffer"]
    hist["Histograms"]
    subgraph offloop["Off the event loop"]
        gc["gc.callbacks hook"]
        exporter["Exporter thread, at each export"]
    end
    requests -->|"integer updates"| totals
    rollouts -->|"integer updates"| totals
    rollouts -->|"one record per rollout"| hist
    sampler -->|"lag, task count, and drained GC pauses"| hist
    gc -->|"pause timestamps"| pauses
    gc -->|"collection totals"| totals
    pauses -->|"drained once per interval"| sampler
    exporter -->|"reads totals and psutil process stats"| totals
    exporter -->|"exports"| hist
    classDef hot fill:#e3f4e1,stroke:#2e7d32
    class requests,rollouts hot
Loading

The green steps run once per request or once per rollout, and only update integers or record one histogram point per rollout.

Measured costs decided what runs by default:

Measurement Cost Where it runs Default
Request and driver counts 0.06 µs per update On the event loop, per request or rollout On
One histogram record 1.7 µs On the event loop, per rollout or sample On
GC callback 0.5 µs per collection Inside each collection On
Event-loop lag One sleep per interval On the event loop, once per second On
psutil process reads 4 to 10 µs On the exporter thread, per export On
asyncio.all_tasks for the task count 0.2 ms at 2,000 tasks, 1.7 ms at 16,000, 6.9 ms at 64,000 On the event loop, once per second Off, telemetry.event_loop_task_count

The task count walks every task on the loop while holding it, and grows with concurrency, so it is the only opt-in measurement.
The sampling interval is telemetry.event_loop_sample_interval_seconds.

Why

A capacity investigation found that a model server under bursty rollouts spent about 10 ms per call waiting on its own event loop, behind the synchronous work of other requests, with no metric showing it. These metrics make that kind of slowdown visible in any run, and the driver and resources-operation metrics show where work waits outside the servers.

Issue

Part of #3155. Per-request request and response byte histograms are not included, so that the request path makes no per-request OpenTelemetry call; output row sizes are reported by gym.driver.output_write_bytes.

Compatibility

  • With telemetry or metrics off, nothing starts, and the only additions to the request path are one global read in the outermost middleware and one in server_utils.request.
  • With metrics on, the new metrics are exported without further configuration. The task count needs telemetry.event_loop_task_count: true or NEMO_GYM_OTEL_EVENT_LOOP_TASK_COUNT=1.
  • Resource attributes gain process.pid in every process.
  • A standalone gym eval run --no-serve driver now initializes telemetry from the telemetry: block, under the service name suffix driver.
  • math_with_judge's _wait_for_library_verifier_process gains an optional keyword-only operation argument.

Validation

  • pytest tests/unit_tests/telemetry tests/unit_tests/test_rollout_collection.py tests/unit_tests/test_server_utils.py tests/unit_tests/test_cli_eval.py tests/unit_tests/test_cli.py -n 12 with the telemetry extra installed.
  • tests/unit_tests/telemetry/test_observer.py covers GC callback pairing, bounded buffering, and removal; lag from a deliberately blocked event loop; nested observers sharing one process state; no work while metrics are off; the opt-in task count; driver totals exported only by a process that offered rollouts; event-loop settings handed from the supervisor to every server; a failed start that does not stop the server; lifespan composition; request, cancellation, and failure counts; outbound in-flight counts; driver admission, residency, write, and rollout-shape metrics; and the final export of totals when the last observer stops. test_resources_operations.py covers queue wait, execution, outcomes, bounded labels, and the disabled path. test_connection_pool.py covers new-connection timing, reused connections, refused connections, and the aiohttp override signature.
  • gym env test +entrypoint=resources_servers/math_with_judge +should_validate_data=true.
  • An end-to-end run through the CPU capacity simulator from Add a high-concurrency Gym dispatch and resources capacity simulator #3156, with the console exporter, the single-agent-turn Environment Server, and 256 rollouts per run. Every expected instrument was exported by the processes that should export it. The values matched the run: offered rollouts equalled rollouts collected, resident and active returned to zero, input items equalled tasks times six messages, and resources-operation samples equalled two tool calls per rollout, including warm-up. The same check found two defects that this PR fixes: the driver's final totals were never exported, and every server exported the driver's rollout counts as constant zeros. The task gauge was exported only when NEMO_GYM_OTEL_EVENT_LOOP_TASK_COUNT=1 was set.
  • An interleaved overhead comparison through the same simulator, with near-instant inference so that Gym's own cost dominates, 2,048 rollouts per run and three runs per setting, with the engine, servers, and driver pinned to separate cores. Traces were off throughout. CPU per rollout, median with range:
Setting Server CPU per rollout Driver CPU per rollout
Base commit, telemetry off 22.2 ms (22.0–23.3) 2.77 ms (2.76–2.82)
Base commit, metrics on 22.7 ms (22.6–22.8) 2.78 ms (2.76–2.80)
This PR, telemetry off 22.2 ms (22.2–22.5) 2.81 ms (2.81–2.81)
This PR, metrics on 22.8 ms (22.7–23.9) 2.88 ms (2.87–2.89)
This PR, metrics and task count on 22.8 ms (22.8–23.1) 2.89 ms (2.88–2.95)

With telemetry off, this PR matches the base commit. With metrics on, the new server metrics add nothing measurable on top of Gym's existing metrics. The driver spends about 0.1 ms more per rollout, from its per-rollout histograms. The existing metrics pipeline costs about 2 percent of server CPU in this run, mostly from the FastAPI and aiohttp instrumentors; #4478 tracks installing them only when traces are exported.

Rollout evidence

N/A for model rollouts. The change adds observability and does not change agent, model, or verifier decisions. The capacity-simulator runs above exercise real rollouts through the driver, agent, model server, Environment Server, and resources server with a mock inference engine.

@ananthsub ananthsub added feature New capabilities, enhancements, or enablement work area:core Shared APIs, servers, telemetry, health, and registries labels Oct 12, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 12, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

@yaoyu-33 yaoyu-33 added complexity:high Cross-component or shared-contract change with a large review surface needs-review PR is ready for code review and waiting on a reviewer labels Oct 12, 2026
@ananthsub ananthsub changed the title feat(telemetry): report runtime, request, driver, and resources-operation metrics feat(telemetry): report event-loop, process, request, driver, and resources-operation metrics Oct 12, 2026
@ananthsub
ananthsub requested a review from athitten October 12, 2026 17:06
@ananthsub
ananthsub force-pushed the ananthsub/runtime-telemetry branch from 33d9c05 to e4dcd53 Compare October 12, 2026 17:24
Comment thread nemo_gym/telemetry/observer.py Outdated
"""Request totals that the server middleware updates with plain integer arithmetic.

The exporter thread reads them through observable instruments, so a request costs no OpenTelemetry call.
Only the server's own event loop updates them, so the updates never race.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: "the updates never race" holds for a server process, but not in the supervisor, where the head-server and driver loops run on different threads and share this object.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, fixed in 252b69e. In the supervisor the head server's thread updates none of these totals: it has no counting middleware and makes no outbound calls. The one count that can change on a second thread is outbound_in_flight, for example on the event loop the synchronous sandbox API runs on. The docstring now says exactly that, and why it is still safe: each update is a single += on an attribute with no call in between, which CPython with the GIL does not interleave with another thread.

Comment thread nemo_gym/server_utils.py Outdated
counts.in_flight += 1
try:
await self._handle(scope, receive, send, counts)
except BaseException as error:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: except BaseException also catches SystemExit and KeyboardInterrupt (process shutdown, not request failures), and only CancelledError is excluded by hand. except Exception would count real handler errors only, and CancelledError would no longer need the special case. Also, since handled HTTPExceptions aren't counted, failed_total reads broader than it is; unhandled_exceptions_total or a line in metrics.mdx would make that clear.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed on both, fixed in 252b69e:

  • except Exception now, so shutdown signals are not counted and CancelledError needs no special case.
  • Renamed to gym.server.requests.unhandled_exceptions_total, since it only sees exceptions no middleware turned into a response.
  • Handled errors are now counted too: the same middleware counts every response it starts sending in gym.server.responses_total, by http.response.status_code. A handled HTTPException shows up there as its 4xx or 5xx, without depending on the FastAPI instrumentor. It costs one dictionary update per response.

A test drives a real FastAPI app through the middleware: two 200s and a handled 404 land in responses_total, and an unhandled RuntimeError lands in unhandled_exceptions_total only.

…t metrics

Add an opt-in runtime observer, telemetry.runtime_metrics, that every
server process and the head server run for the lifetime of their app.
It reports event-loop lag and resident tasks, garbage-collection pauses
and totals by generation, process CPU, memory, threads, and descriptors,
and in-flight, cancelled, and failed requests.

The observer keeps work off the request path. The event loop runs one
sampling coroutine per interval, the GC callback only buffers timestamps
and totals, and request totals are plain integers that the exporter
thread reads through observable instruments. Each process also reports
process.pid so the Uvicorn workers of one server stay distinguishable.

Implements the runtime and request portion of #3155.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
…etup time

Count outbound calls through server_utils.request that are still waiting
for their response, as gym.http.client.requests.in_flight. Like the server
request totals, it is a plain integer the exporter thread reads.

Time each new pooled connection in QueueTimedTCPConnector, which is only
installed while metrics export, as
gym.http.connection_pool.connect_duration_ms. Reused connections record
nothing, so a run that keeps opening connections stands out.

Implements the HTTP client portion of #3155.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
…ut metrics

The driver now reports offered, resident, and active rollouts, the time
each rollout waits for the concurrency limit, the time and size of each
output row write, and the time to materialize the run's inputs. Its event
loop is sampled like a server's.

Counts are plain integers and the semaphore is only wrapped while runtime
metrics run, so a run without them dispatches exactly as before. A
standalone gym eval run --no-serve driver now sets up its own telemetry,
which also lets it export the existing gym.job span.

Implements the rollout-driver portion of #3155.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
…river

Record the input items, output items, turns, and tool calls of every
completed rollout, once per rollout, while runtime metrics run. Turns and
tool calls come from the rollout's ng_perf when rollout observability
produced it, so the request path does no extra work.

Implements the payload portion of #3155.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
… in math_with_judge

resources_operation wraps work a resources server limits with a semaphore
and reports how long it queued, how long it ran, how it ended, and how
many operations wait and run, by bounded operation and executor labels.

math_with_judge uses it for its SymPy verifier subprocesses and marks
timeouts and crashes, so a verifier limit that is too small shows up as
queue wait instead of an unexplained slow /verify.

Implements the resources-operation portion of #3155.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
…r stops

The observable totals, such as in-flight requests and the driver's offered,
resident, and active rollouts, report only while an observer runs. A
short-lived process, such as a driver whose run ends before the first
periodic export, stopped its observer before telemetry shutdown made the
final export, so those totals were never exported. The last observer to
stop now exports once while the totals still exist.

Found by running the capacity simulator's telemetry_e2e profile.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
The observer's metrics cost a few integer updates per request, and the exporter's thread reads the totals, so a separate switch protected nothing.
They now run whenever telemetry exports metrics, and the `runtime_metrics` block is gone.

Counting asyncio tasks walks every task on the loop, about a millisecond per sample at 16,000 tasks.
That gauge alone stays opt-in through `telemetry.event_loop_task_count`.
The sampling interval moves to `telemetry.event_loop_sample_interval_seconds`.

"Runtime" named nothing in particular.
The module becomes `nemo_gym.telemetry.observer`, and the instruments become `gym.event_loop.*` and `gym.gc.*`.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
… rollouts

The driver's observable totals are registered in every process, so each server exported constant-zero rollout counts it never updates.
They now report only after the process has offered a rollout.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
Gym's existing metrics record durations in milliseconds with a `_ms` suffix, end counter names in `_total`, and put attributes under `nemo.gym.`, as `gym.rollout.completed_total` and `gym.http.connection_pool.queue_duration_ms` do.
The new metrics now follow the same rules, so one dashboard does not mix seconds and milliseconds or bare and namespaced labels.

- Durations are in milliseconds. A thing's own duration is `<name>.duration_ms` and a phase of it is `<name>.<phase>_duration_ms`, as in `gym.rollout.duration_ms` and `gym.http.connection_pool.queue_duration_ms`: `gym.event_loop.lag_ms`, `gym.gc.pause_duration_ms`, `gym.driver.admission_duration_ms`, `gym.driver.output_write_duration_ms`, `gym.driver.materialize_duration_ms`, `gym.resources.operation.queue_duration_ms`, and `gym.resources.operation.duration_ms`, with their bucket boundaries converted.
- Counters end in `_total`: `gym.gc.collections_total`, `gym.gc.collected_total`, `gym.gc.uncollectable_total`, `gym.server.requests.cancelled_total`, `gym.server.requests.failed_total`, and `gym.driver.rollouts.offered_total`.
- Attributes are namespaced: `nemo.gym.gc.generation`, `nemo.gym.resources.operation`, `nemo.gym.resources.executor`, and `nemo.gym.resources.operation.outcome`, which no longer reads like `nemo.gym.rollout.outcome`.
- Multi-word name parts use underscores, as `gym.driver.rollout.tool_calls` already did: `gym.driver.output_write_bytes`, `gym.driver.rollout.input_items`, and `gym.driver.rollout.output_items`.

The `process.*` metrics keep their OpenTelemetry semantic-convention names, seconds, and `cpu.mode`, so standard dashboards still recognize them.
The duration tests now assert millisecond values, so a reverted conversion fails them.
The metrics table in `metrics.mdx` also loses a stray blank line that split it in two.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
…ns plainly

`gym.server.requests.failed_total` counted only exceptions that escaped every middleware, and caught `BaseException`, so `SystemExit` and `KeyboardInterrupt` during shutdown counted too while `CancelledError` needed a special case.
It is now `gym.server.requests.unhandled_exceptions_total`, catching `Exception` only.

A handled error, such as an `HTTPException`, becomes a response inside the middleware and never reached that counter.
The outermost middleware now counts every response it starts sending in `gym.server.responses_total`, by `http.response.status_code`, so 4xx and 5xx responses are visible without the FastAPI instrumentor.
It is one dictionary update per response on the serving loop, read by the exporter thread from a copy.

The `RequestCounts` docstring no longer claims only the serving loop updates the totals: the outbound count can change on another thread's loop, such as the synchronous sandbox API's, and the docstring states why a single `+=` is still safe under the GIL.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
@ananthsub
ananthsub added this pull request to the merge queue Oct 14, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/38619790337

Merged via the queue into main with commit 53eae81 Oct 14, 2026
49 of 54 checks passed
@ananthsub
ananthsub deleted the ananthsub/runtime-telemetry branch October 14, 2026 19:17
ananthsub added a commit that referenced this pull request Oct 14, 2026
> **Stack:** telemetry lands before the optimizations, so the
optimizations can be measured with it.
> 1. #4474: event-loop, process, request, driver, and
resources-operation metrics (merged)
> 2. #4481: drop instrumentation that exports nothing (base `main`)
> 3. #2890: replace the HTTP middleware wrappers with ASGI (base #4481)

## What changed

With `traces_enabled: false`, nemo-lens still enables the configured
span groups, so instrumentation whose only output is spans kept running
on every request against a tracer that records nothing.
This PR skips that work while traces are off:

- `init_telemetry` records whether the process exports traces, and
`nemo_gym.telemetry.setup.is_trace_exporter_active()` reports it. When
Gym reuses providers another library set up, it checks for a real SDK
tracer provider, as it already does for metrics.
- `SimpleServer.instrument_app_for_telemetry` installs the FastAPI
instrumentor only while traces export.
- `server_utils.request` opens the outbound CLIENT span and injects
`traceparent` only while traces export.
- The `telemetry.instrument_aiohttp` setting and its
`NEMO_GYM_OTEL_INSTRUMENT_AIOHTTP` variable are removed. Nothing in Gym
or nemo-lens calls
`nemo.lens.contrib.aiohttp.instrument_aiohttp_client`, so the setting
never did anything; its docstring described a CLIENT span that Gym in
fact creates itself in `server_utils.request`.
- `opentelemetry-instrumentation-aiohttp-client`, which only backs that
unused helper, is dropped from the `telemetry` extra and from the
packages installed into each server venv.

Span groups that also drive metrics, such as `verify`, `rollout`, and
`sandbox`, are unchanged, so `gym.verify.*`, `gym.rollout.*`, and the
sandbox metrics still report in a metrics-only run.

The FastAPI instrumentor is also the source of the `http.server.*`
metrics.
A metrics-only run no longer reports them, and the docs now say so.
The docs also named metrics the pinned instrumentation does not export:
a run exports `http.server.duration` (ms), `http.server.request.size`,
`http.server.response.size`, and `http.server.active_requests`, keyed by
`http.target`, `http.method`, and `http.status_code`, while
`metrics.mdx`, `extending.mdx`, and the telemetry README listed
`http.server.request.duration`.

## Why

Measured on one workstation with tracing off, the FastAPI instrumentor
adds about 63 µs to each inbound request (15 µs bare, 78 µs
instrumented).
The outbound path adds a no-op span and a header injection to every call
between servers.
Neither produces anything a metrics-only run can use.

## Issue

Closes #4478.

The issue also listed aiohttp client instrumentation at about 58 µs per
outbound request.
Nothing in Gym or nemo-lens installs that instrumentor; the outbound
cost Gym pays comes from its own client span, which this PR gates, and
the setting that appeared to control the instrumentor is removed.
This PR absorbs #4485.

## Compatibility

- Runs with `traces_enabled: false` and metrics on lose the
`http.server.*` metrics. Turn traces on to keep them.
- Runs with traces on are unchanged.
- Configs that still set `instrument_aiohttp` keep loading, because
`TelemetryConfig` allows extra keys; the key has no effect, as before.
The setting is removed rather than deprecated, as `export_strategy` and
`export_rank` were in #3890.
- The `telemetry` extra no longer installs
`opentelemetry-instrumentation-aiohttp-client`. `daytona` still depends
on it, so it stays in `uv.lock` for the `sandbox` extra. Code that calls
`instrument_aiohttp_client` itself can install `nemo-lens[aiohttp]`.
- Builds on #4474, now merged. In `server_utils.request`, the trace gate
chooses between the traced and untraced send, and #4474's outbound
in-flight count wraps either one.

## Validation

- `pytest tests/unit_tests/telemetry
tests/unit_tests/test_server_utils.py tests/unit_tests/test_cli.py -n
12` with the `telemetry` extra installed.
- `pytest tests/unit_tests -n 16`: the only failures are the same
environment-dependent ones as on the base commit
(`test_opensandbox_compose.py`, `test_reward_profile.py`,
`test_dataset_orchestrator.py`, `test_sandbox_stop_retry.py`), and
`test_exporters.py` needs `mlflow`.
- `test_setup.py` covers the trace flag following `traces_enabled`, the
flag for reused providers with and without an SDK tracer provider, and
the FastAPI instrumentor installed only while traces export.
- `test_config.py` checks that a config setting `instrument_aiohttp`
still loads and that `configure_telemetry_env` no longer exports the
variable; it fails on the base commit.
- In a venv synced from this branch with the `telemetry` extra,
`opentelemetry.instrumentation.aiohttp_client` is not importable, and
the full unit suite runs with the same environment-dependent failures as
the base commit, including passing propagation tests that check
`traceparent` on outbound requests.
- `uv lock --check` passes.
- `test_propagation.py` serves two real FastAPI apps and checks that,
with traces off, no CLIENT span is produced and no `traceparent` reaches
the downstream server.
- Removing either gate makes its test fail.
- `pre-commit run` on the changed files.

- An interleaved comparison through the CPU capacity simulator from
#3156 (`fast_inference_burst`, near-instant inference so that Gym's own
cost dominates, 2,048 rollouts per run, three runs per setting, engine,
servers, and driver pinned to separate cores, traces off). Server CPU
per rollout, median with range:

| Setting | Server CPU per rollout | Rollouts per second |
| --- | ---: | ---: |
| Base commit, metrics on | 22.6 ms (22.6–22.7) | 100 (97–107) |
| This PR, metrics on | 21.0 ms (21.0–21.1) | 106 (102–109) |
| This PR, telemetry off | 19.7 ms (19.4–19.7) | 114 (110–114) |

The cost of turning metrics on drops from about 2.9 ms to 1.3 ms per
rollout.
Both metrics-on runs exported the same `gym.*` metrics, including
`gym.rollout.duration_ms` and the connection-pool metrics; only the
`http.server.*` metrics are gone. No run had a failed rollout.

## Rollout evidence

`example_single_tool_call` with the simple agent and `gpt-oss-20b` (3
tasks) ran from the dependency-removal commits and from the base commit
`75bb53e6d`, both with `span_groups=all`, traces on, and console export,
through `gym env start` and then `gym eval run --no-serve`. On the base
commit every server venv had
`opentelemetry-instrumentation-aiohttp-client`; with the change, none
did, and each still had `opentelemetry-instrumentation-fastapi`.

| | Base commit | Without the aiohttp package |
| --- | --- | --- |
| Spans | 238 | 238 |
| Span types (name, kind, process) | 49 | 49, identical set |
| `HTTP POST` client spans | 26 | 26 |
| Traces | 12 | 12 |
| Traces that cross processes | 4 | 4 |
| Server spans whose parent is a client span in another process | 18 |
18 |
| Rewards | 0, 1, 1 | 0, 1, 1 |

`traceparent` still joins each outbound request to the server span that
handles it. Sandbox spans are not covered, because this environment has
no sandbox.

The trace gate does not change agent, model, or verifier decisions; the
capacity-simulator runs above exercise it through real rollouts with a
mock inference engine.

---------

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
khazic pushed a commit to khazic/Gym-lao that referenced this pull request Oct 15, 2026
## Summary
- Add the SA-V tracking verifier and its server configuration.
- Declare the SA-V environment servers used by video RL.
- Add portable five-row example data, schema, metrics, and checked
rollout artifacts.

## Why
This is the Gym-side verifier foundation for mixed SA-V tracking and
CapRL video training in NeMo-RL. No separate issue is needed; this is
part of the Super 3.5 video RL upstreaming stack.

## Validation
- Runtime evidence: the [healthy 16-node async `r18` W&B
run](https://wandb.ai/adlr/Nemotron-omni-RL-debug/runs/oci-hsg-cs-001_n3post_main_1d76d243b_vllm029_super35_video_step120_async16n_trainmb45056_expandable_uncapped_r18_20261003)
shows repeated GRPO cycles, stable TMPE, and increasing reward on the
equivalent video implementation.
- `gym dataset collate ... +mode=example_validation`: 5 examples;
generated `data/example_metrics.json`.
- Focused validation: 25 verifier tests and all 5 example rows against
`task_data.py` passed.
- Ruff format and lint passed for the changed Python files.
- Both Gym stack ranges pass `git diff --check`.

## Rollouts
`data/example_rollouts.jsonl` contains one checked rollout for each of
the five example rows. Representative task checks:

- `pt`:
`<answer>{"tracks":[{"point":1,"frame":1,"xy":[500,500],"visible":true}]}</answer>`
— `format_ok=true`, reward `1.0`, visible-frame mask membership `true`.
- `box`:
`<answer>{"tracks":[{"box":1,"frame":1,"bbox":[250,250,750,750],"visible":true}]}</answer>`
— `format_ok=true`, reward `1.0`, frame IoU `1.0`.
- `ref`:
`<answer>{"tracks":[{"id":7,"frame":1,"xy":[500,500]}]}</answer>` —
`format_ok=true`, reward `1.0`, predicted id `7` assigned to the GT
object and mask membership `true`.

The artifact also contains visible/invisible `pt` coverage and an
all-absent `ref` negative case, preserving the verifier's current
unmatched-ID behavior.

## Compatibility
Adds a new verifier/configuration and does not change unrelated
environments. Example media is embedded as a small inline JPEG, so no
employee or cluster-local paths are required.

## Stack
Gym PR 1 of 2. Consolidates and supersedes the implementation scope of
NVIDIA-NeMo#3504 after review history is preserved.

## Consolidated stack links

### Gym
1. [Gym NVIDIA-NeMo#3999 — SA-V tracking
verifier](NVIDIA-NeMo#3999)
2. [Gym NVIDIA-NeMo#4000 — R3 routing replay and rollout
identity](NVIDIA-NeMo#4000)

### NeMo-RL
1. [NeMo-RL NVIDIA-NeMo#4472 — Super video data and model
loading](NVIDIA-NeMo/RL#4472)
2. [NeMo-RL NVIDIA-NeMo#4473 — RL/MOPD video frame
contract](NVIDIA-NeMo/RL#4473)
3. [NeMo-RL NVIDIA-NeMo#4474 — sync/async Gym routing
runtime](NVIDIA-NeMo/RL#4474)
4. [NeMo-RL NVIDIA-NeMo#4475 — SA-V pin and CI
fixtures](NVIDIA-NeMo/RL#4475)
5. [NeMo-RL NVIDIA-NeMo#4476 — final dependency
pins](NVIDIA-NeMo/RL#4476)

### Child dependency refs
- [MCore integration
ref](https://github.com/ehosseiniasl/Megatron-LM/tree/ehsan/video-r3-mtp-gradient-integration-20261004),
adapting [MCore #7278](NVIDIA/Megatron-LM#7278)
and [MCore #7279](NVIDIA/Megatron-LM#7279) to
the pinned base.
- [MBridge integration
ref](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/ehsan/video-r3-mcore-pin-integration-20261004).

---------

Signed-off-by: ehosseiniasl <ehsan.hosseiniasl@gmail.com>
Co-authored-by: Pulkit Kumar <pulkitk@nvidia.com>
cmunley1 pushed a commit to Anai-Guo/Gym that referenced this pull request Oct 15, 2026
…phase (NVIDIA-NeMo#4028)

Client cancellations now extend NVIDIA-NeMo#4474's existing
`gym.server.requests.cancelled_total` counter with bounded route
templates and response-start phases.
Route attribution follows each server's prefix rules, including resource
routes containing `training-token-capture`, and cumulative labeled
buckets count each cancellation once.
Headers-only responses and SSE keepalives count as `response_started`;
stdout remains sampled once per 100 cancellations, with elapsed time and
phase available through the existing request-debug override.
The metrics documentation explains aggregation across route/phase series
and the distinction between per-server cancellations and unique
rollouts.
Validation passed: 332 native targeted tests and full pre-commit, plus
the earlier 23 documentation link tests and Fern checks; Fern's
authenticated redirect check was unavailable.
Model rollouts are N/A for this diagnostics-only change, and no separate
issue is needed for this focused review revision.

---------

Signed-off-by: Lin Jia <linj@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

This branch was previously deployed

1 inactive deployment
public — 252b69ed Deployed Oct 14, 2026 by copy-pr-bot[bot] via release / finalize / notify #5324
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:core Shared APIs, servers, telemetry, health, and registries complexity:high Cross-component or shared-contract change with a large review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer sla:review-overdue Review response is over the one-business-day SLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants