Repository navigation
feat(telemetry): report event-loop, process, request, driver, and resources-operation metrics - #4474
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
🌿 Preview your docs: https://nvidia-preview-pr-4474-252b69ed7e1b.docs.buildwithfern.com/nemo/gym |
33d9c05 to
e4dcd53
Compare
| """Request totals that the server middleware updates with plain integer arithmetic. | ||
|
|
||
| The exporter thread reads them through observable instruments, so a request costs no OpenTelemetry call. | ||
| Only the server's own event loop updates them, so the updates never race. |
There was a problem hiding this comment.
nit: "the updates never race" holds for a server process, but not in the supervisor, where the head-server and driver loops run on different threads and share this object.
There was a problem hiding this comment.
Right, fixed in 252b69e. In the supervisor the head server's thread updates none of these totals: it has no counting middleware and makes no outbound calls. The one count that can change on a second thread is outbound_in_flight, for example on the event loop the synchronous sandbox API runs on. The docstring now says exactly that, and why it is still safe: each update is a single += on an attribute with no call in between, which CPython with the GIL does not interleave with another thread.
| counts.in_flight += 1 | ||
| try: | ||
| await self._handle(scope, receive, send, counts) | ||
| except BaseException as error: |
There was a problem hiding this comment.
nit: except BaseException also catches SystemExit and KeyboardInterrupt (process shutdown, not request failures), and only CancelledError is excluded by hand. except Exception would count real handler errors only, and CancelledError would no longer need the special case. Also, since handled HTTPExceptions aren't counted, failed_total reads broader than it is; unhandled_exceptions_total or a line in metrics.mdx would make that clear.
There was a problem hiding this comment.
Agreed on both, fixed in 252b69e:
except Exceptionnow, so shutdown signals are not counted andCancelledErrorneeds no special case.- Renamed to
gym.server.requests.unhandled_exceptions_total, since it only sees exceptions no middleware turned into a response. - Handled errors are now counted too: the same middleware counts every response it starts sending in
gym.server.responses_total, byhttp.response.status_code. A handledHTTPExceptionshows up there as its 4xx or 5xx, without depending on the FastAPI instrumentor. It costs one dictionary update per response.
A test drives a real FastAPI app through the middleware: two 200s and a handled 404 land in responses_total, and an unhandled RuntimeError lands in unhandled_exceptions_total only.
…t metrics Add an opt-in runtime observer, telemetry.runtime_metrics, that every server process and the head server run for the lifetime of their app. It reports event-loop lag and resident tasks, garbage-collection pauses and totals by generation, process CPU, memory, threads, and descriptors, and in-flight, cancelled, and failed requests. The observer keeps work off the request path. The event loop runs one sampling coroutine per interval, the GC callback only buffers timestamps and totals, and request totals are plain integers that the exporter thread reads through observable instruments. Each process also reports process.pid so the Uvicorn workers of one server stay distinguishable. Implements the runtime and request portion of #3155. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
…etup time Count outbound calls through server_utils.request that are still waiting for their response, as gym.http.client.requests.in_flight. Like the server request totals, it is a plain integer the exporter thread reads. Time each new pooled connection in QueueTimedTCPConnector, which is only installed while metrics export, as gym.http.connection_pool.connect_duration_ms. Reused connections record nothing, so a run that keeps opening connections stands out. Implements the HTTP client portion of #3155. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
…ut metrics The driver now reports offered, resident, and active rollouts, the time each rollout waits for the concurrency limit, the time and size of each output row write, and the time to materialize the run's inputs. Its event loop is sampled like a server's. Counts are plain integers and the semaphore is only wrapped while runtime metrics run, so a run without them dispatches exactly as before. A standalone gym eval run --no-serve driver now sets up its own telemetry, which also lets it export the existing gym.job span. Implements the rollout-driver portion of #3155. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
…river Record the input items, output items, turns, and tool calls of every completed rollout, once per rollout, while runtime metrics run. Turns and tool calls come from the rollout's ng_perf when rollout observability produced it, so the request path does no extra work. Implements the payload portion of #3155. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
… in math_with_judge resources_operation wraps work a resources server limits with a semaphore and reports how long it queued, how long it ran, how it ended, and how many operations wait and run, by bounded operation and executor labels. math_with_judge uses it for its SymPy verifier subprocesses and marks timeouts and crashes, so a verifier limit that is too small shows up as queue wait instead of an unexplained slow /verify. Implements the resources-operation portion of #3155. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
…r stops The observable totals, such as in-flight requests and the driver's offered, resident, and active rollouts, report only while an observer runs. A short-lived process, such as a driver whose run ends before the first periodic export, stopped its observer before telemetry shutdown made the final export, so those totals were never exported. The last observer to stop now exports once while the totals still exist. Found by running the capacity simulator's telemetry_e2e profile. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
The observer's metrics cost a few integer updates per request, and the exporter's thread reads the totals, so a separate switch protected nothing. They now run whenever telemetry exports metrics, and the `runtime_metrics` block is gone. Counting asyncio tasks walks every task on the loop, about a millisecond per sample at 16,000 tasks. That gauge alone stays opt-in through `telemetry.event_loop_task_count`. The sampling interval moves to `telemetry.event_loop_sample_interval_seconds`. "Runtime" named nothing in particular. The module becomes `nemo_gym.telemetry.observer`, and the instruments become `gym.event_loop.*` and `gym.gc.*`. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
… rollouts The driver's observable totals are registered in every process, so each server exported constant-zero rollout counts it never updates. They now report only after the process has offered a rollout. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
Gym's existing metrics record durations in milliseconds with a `_ms` suffix, end counter names in `_total`, and put attributes under `nemo.gym.`, as `gym.rollout.completed_total` and `gym.http.connection_pool.queue_duration_ms` do. The new metrics now follow the same rules, so one dashboard does not mix seconds and milliseconds or bare and namespaced labels. - Durations are in milliseconds. A thing's own duration is `<name>.duration_ms` and a phase of it is `<name>.<phase>_duration_ms`, as in `gym.rollout.duration_ms` and `gym.http.connection_pool.queue_duration_ms`: `gym.event_loop.lag_ms`, `gym.gc.pause_duration_ms`, `gym.driver.admission_duration_ms`, `gym.driver.output_write_duration_ms`, `gym.driver.materialize_duration_ms`, `gym.resources.operation.queue_duration_ms`, and `gym.resources.operation.duration_ms`, with their bucket boundaries converted. - Counters end in `_total`: `gym.gc.collections_total`, `gym.gc.collected_total`, `gym.gc.uncollectable_total`, `gym.server.requests.cancelled_total`, `gym.server.requests.failed_total`, and `gym.driver.rollouts.offered_total`. - Attributes are namespaced: `nemo.gym.gc.generation`, `nemo.gym.resources.operation`, `nemo.gym.resources.executor`, and `nemo.gym.resources.operation.outcome`, which no longer reads like `nemo.gym.rollout.outcome`. - Multi-word name parts use underscores, as `gym.driver.rollout.tool_calls` already did: `gym.driver.output_write_bytes`, `gym.driver.rollout.input_items`, and `gym.driver.rollout.output_items`. The `process.*` metrics keep their OpenTelemetry semantic-convention names, seconds, and `cpu.mode`, so standard dashboards still recognize them. The duration tests now assert millisecond values, so a reverted conversion fails them. The metrics table in `metrics.mdx` also loses a stray blank line that split it in two. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
…ns plainly `gym.server.requests.failed_total` counted only exceptions that escaped every middleware, and caught `BaseException`, so `SystemExit` and `KeyboardInterrupt` during shutdown counted too while `CancelledError` needed a special case. It is now `gym.server.requests.unhandled_exceptions_total`, catching `Exception` only. A handled error, such as an `HTTPException`, becomes a response inside the middleware and never reached that counter. The outermost middleware now counts every response it starts sending in `gym.server.responses_total`, by `http.response.status_code`, so 4xx and 5xx responses are visible without the FastAPI instrumentor. It is one dictionary update per response on the serving loop, read by the exporter thread from a copy. The `RequestCounts` docstring no longer claims only the serving loop updates the totals: the outbound count can change on another thread's loop, such as the synchronous sandbox API's, and the docstring states why a single `+=` is still safe under the GIL. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
9564fec to
252b69e
Compare
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/38619790337 |
> **Stack:** telemetry lands before the optimizations, so the optimizations can be measured with it. > 1. #4474: event-loop, process, request, driver, and resources-operation metrics (merged) > 2. #4481: drop instrumentation that exports nothing (base `main`) > 3. #2890: replace the HTTP middleware wrappers with ASGI (base #4481) ## What changed With `traces_enabled: false`, nemo-lens still enables the configured span groups, so instrumentation whose only output is spans kept running on every request against a tracer that records nothing. This PR skips that work while traces are off: - `init_telemetry` records whether the process exports traces, and `nemo_gym.telemetry.setup.is_trace_exporter_active()` reports it. When Gym reuses providers another library set up, it checks for a real SDK tracer provider, as it already does for metrics. - `SimpleServer.instrument_app_for_telemetry` installs the FastAPI instrumentor only while traces export. - `server_utils.request` opens the outbound CLIENT span and injects `traceparent` only while traces export. - The `telemetry.instrument_aiohttp` setting and its `NEMO_GYM_OTEL_INSTRUMENT_AIOHTTP` variable are removed. Nothing in Gym or nemo-lens calls `nemo.lens.contrib.aiohttp.instrument_aiohttp_client`, so the setting never did anything; its docstring described a CLIENT span that Gym in fact creates itself in `server_utils.request`. - `opentelemetry-instrumentation-aiohttp-client`, which only backs that unused helper, is dropped from the `telemetry` extra and from the packages installed into each server venv. Span groups that also drive metrics, such as `verify`, `rollout`, and `sandbox`, are unchanged, so `gym.verify.*`, `gym.rollout.*`, and the sandbox metrics still report in a metrics-only run. The FastAPI instrumentor is also the source of the `http.server.*` metrics. A metrics-only run no longer reports them, and the docs now say so. The docs also named metrics the pinned instrumentation does not export: a run exports `http.server.duration` (ms), `http.server.request.size`, `http.server.response.size`, and `http.server.active_requests`, keyed by `http.target`, `http.method`, and `http.status_code`, while `metrics.mdx`, `extending.mdx`, and the telemetry README listed `http.server.request.duration`. ## Why Measured on one workstation with tracing off, the FastAPI instrumentor adds about 63 µs to each inbound request (15 µs bare, 78 µs instrumented). The outbound path adds a no-op span and a header injection to every call between servers. Neither produces anything a metrics-only run can use. ## Issue Closes #4478. The issue also listed aiohttp client instrumentation at about 58 µs per outbound request. Nothing in Gym or nemo-lens installs that instrumentor; the outbound cost Gym pays comes from its own client span, which this PR gates, and the setting that appeared to control the instrumentor is removed. This PR absorbs #4485. ## Compatibility - Runs with `traces_enabled: false` and metrics on lose the `http.server.*` metrics. Turn traces on to keep them. - Runs with traces on are unchanged. - Configs that still set `instrument_aiohttp` keep loading, because `TelemetryConfig` allows extra keys; the key has no effect, as before. The setting is removed rather than deprecated, as `export_strategy` and `export_rank` were in #3890. - The `telemetry` extra no longer installs `opentelemetry-instrumentation-aiohttp-client`. `daytona` still depends on it, so it stays in `uv.lock` for the `sandbox` extra. Code that calls `instrument_aiohttp_client` itself can install `nemo-lens[aiohttp]`. - Builds on #4474, now merged. In `server_utils.request`, the trace gate chooses between the traced and untraced send, and #4474's outbound in-flight count wraps either one. ## Validation - `pytest tests/unit_tests/telemetry tests/unit_tests/test_server_utils.py tests/unit_tests/test_cli.py -n 12` with the `telemetry` extra installed. - `pytest tests/unit_tests -n 16`: the only failures are the same environment-dependent ones as on the base commit (`test_opensandbox_compose.py`, `test_reward_profile.py`, `test_dataset_orchestrator.py`, `test_sandbox_stop_retry.py`), and `test_exporters.py` needs `mlflow`. - `test_setup.py` covers the trace flag following `traces_enabled`, the flag for reused providers with and without an SDK tracer provider, and the FastAPI instrumentor installed only while traces export. - `test_config.py` checks that a config setting `instrument_aiohttp` still loads and that `configure_telemetry_env` no longer exports the variable; it fails on the base commit. - In a venv synced from this branch with the `telemetry` extra, `opentelemetry.instrumentation.aiohttp_client` is not importable, and the full unit suite runs with the same environment-dependent failures as the base commit, including passing propagation tests that check `traceparent` on outbound requests. - `uv lock --check` passes. - `test_propagation.py` serves two real FastAPI apps and checks that, with traces off, no CLIENT span is produced and no `traceparent` reaches the downstream server. - Removing either gate makes its test fail. - `pre-commit run` on the changed files. - An interleaved comparison through the CPU capacity simulator from #3156 (`fast_inference_burst`, near-instant inference so that Gym's own cost dominates, 2,048 rollouts per run, three runs per setting, engine, servers, and driver pinned to separate cores, traces off). Server CPU per rollout, median with range: | Setting | Server CPU per rollout | Rollouts per second | | --- | ---: | ---: | | Base commit, metrics on | 22.6 ms (22.6–22.7) | 100 (97–107) | | This PR, metrics on | 21.0 ms (21.0–21.1) | 106 (102–109) | | This PR, telemetry off | 19.7 ms (19.4–19.7) | 114 (110–114) | The cost of turning metrics on drops from about 2.9 ms to 1.3 ms per rollout. Both metrics-on runs exported the same `gym.*` metrics, including `gym.rollout.duration_ms` and the connection-pool metrics; only the `http.server.*` metrics are gone. No run had a failed rollout. ## Rollout evidence `example_single_tool_call` with the simple agent and `gpt-oss-20b` (3 tasks) ran from the dependency-removal commits and from the base commit `75bb53e6d`, both with `span_groups=all`, traces on, and console export, through `gym env start` and then `gym eval run --no-serve`. On the base commit every server venv had `opentelemetry-instrumentation-aiohttp-client`; with the change, none did, and each still had `opentelemetry-instrumentation-fastapi`. | | Base commit | Without the aiohttp package | | --- | --- | --- | | Spans | 238 | 238 | | Span types (name, kind, process) | 49 | 49, identical set | | `HTTP POST` client spans | 26 | 26 | | Traces | 12 | 12 | | Traces that cross processes | 4 | 4 | | Server spans whose parent is a client span in another process | 18 | 18 | | Rewards | 0, 1, 1 | 0, 1, 1 | `traceparent` still joins each outbound request to the server span that handles it. Sandbox spans are not covered, because this environment has no sandbox. The trace gate does not change agent, model, or verifier decisions; the capacity-simulator runs above exercise it through real rollouts with a mock inference engine. --------- Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
## Summary - Add the SA-V tracking verifier and its server configuration. - Declare the SA-V environment servers used by video RL. - Add portable five-row example data, schema, metrics, and checked rollout artifacts. ## Why This is the Gym-side verifier foundation for mixed SA-V tracking and CapRL video training in NeMo-RL. No separate issue is needed; this is part of the Super 3.5 video RL upstreaming stack. ## Validation - Runtime evidence: the [healthy 16-node async `r18` W&B run](https://wandb.ai/adlr/Nemotron-omni-RL-debug/runs/oci-hsg-cs-001_n3post_main_1d76d243b_vllm029_super35_video_step120_async16n_trainmb45056_expandable_uncapped_r18_20261003) shows repeated GRPO cycles, stable TMPE, and increasing reward on the equivalent video implementation. - `gym dataset collate ... +mode=example_validation`: 5 examples; generated `data/example_metrics.json`. - Focused validation: 25 verifier tests and all 5 example rows against `task_data.py` passed. - Ruff format and lint passed for the changed Python files. - Both Gym stack ranges pass `git diff --check`. ## Rollouts `data/example_rollouts.jsonl` contains one checked rollout for each of the five example rows. Representative task checks: - `pt`: `<answer>{"tracks":[{"point":1,"frame":1,"xy":[500,500],"visible":true}]}</answer>` — `format_ok=true`, reward `1.0`, visible-frame mask membership `true`. - `box`: `<answer>{"tracks":[{"box":1,"frame":1,"bbox":[250,250,750,750],"visible":true}]}</answer>` — `format_ok=true`, reward `1.0`, frame IoU `1.0`. - `ref`: `<answer>{"tracks":[{"id":7,"frame":1,"xy":[500,500]}]}</answer>` — `format_ok=true`, reward `1.0`, predicted id `7` assigned to the GT object and mask membership `true`. The artifact also contains visible/invisible `pt` coverage and an all-absent `ref` negative case, preserving the verifier's current unmatched-ID behavior. ## Compatibility Adds a new verifier/configuration and does not change unrelated environments. Example media is embedded as a small inline JPEG, so no employee or cluster-local paths are required. ## Stack Gym PR 1 of 2. Consolidates and supersedes the implementation scope of NVIDIA-NeMo#3504 after review history is preserved. ## Consolidated stack links ### Gym 1. [Gym NVIDIA-NeMo#3999 — SA-V tracking verifier](NVIDIA-NeMo#3999) 2. [Gym NVIDIA-NeMo#4000 — R3 routing replay and rollout identity](NVIDIA-NeMo#4000) ### NeMo-RL 1. [NeMo-RL NVIDIA-NeMo#4472 — Super video data and model loading](NVIDIA-NeMo/RL#4472) 2. [NeMo-RL NVIDIA-NeMo#4473 — RL/MOPD video frame contract](NVIDIA-NeMo/RL#4473) 3. [NeMo-RL NVIDIA-NeMo#4474 — sync/async Gym routing runtime](NVIDIA-NeMo/RL#4474) 4. [NeMo-RL NVIDIA-NeMo#4475 — SA-V pin and CI fixtures](NVIDIA-NeMo/RL#4475) 5. [NeMo-RL NVIDIA-NeMo#4476 — final dependency pins](NVIDIA-NeMo/RL#4476) ### Child dependency refs - [MCore integration ref](https://github.com/ehosseiniasl/Megatron-LM/tree/ehsan/video-r3-mtp-gradient-integration-20261004), adapting [MCore #7278](NVIDIA/Megatron-LM#7278) and [MCore #7279](NVIDIA/Megatron-LM#7279) to the pinned base. - [MBridge integration ref](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/ehsan/video-r3-mcore-pin-integration-20261004). --------- Signed-off-by: ehosseiniasl <ehsan.hosseiniasl@gmail.com> Co-authored-by: Pulkit Kumar <pulkitk@nvidia.com>
…phase (NVIDIA-NeMo#4028) Client cancellations now extend NVIDIA-NeMo#4474's existing `gym.server.requests.cancelled_total` counter with bounded route templates and response-start phases. Route attribution follows each server's prefix rules, including resource routes containing `training-token-capture`, and cumulative labeled buckets count each cancellation once. Headers-only responses and SSE keepalives count as `response_started`; stdout remains sampled once per 100 cancellations, with elapsed time and phase available through the existing request-debug override. The metrics documentation explains aggregation across route/phase series and the distinction between per-server cancellations and unique rollouts. Validation passed: 332 native targeted tests and full pre-commit, plus the earlier 23 documentation link tests and Fern checks; Fern's authenticated redirect check was unavailable. Model rollouts are N/A for this diagnostics-only change, and no separate issue is needed for this focused review revision. --------- Signed-off-by: Lin Jia <linj@nvidia.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
What changed
A Gym server answers every request on one asyncio event loop. When synchronous work holds that loop, every other request waits, even though no single request looks slow, and nothing Gym exported showed it. This PR adds the event-loop, process, request, HTTP client, rollout-driver, rollout-shape, and resources-operation metrics from #3155. They run whenever telemetry exports metrics, except the event-loop task count, which is opt-in.
gym.event_loop.lag_ms,gym.event_loop.tasks,gym.gc.pause_duration_ms,gym.gc.collections_total,gym.gc.collected_total,gym.gc.uncollectable_total,gym.gc.count,process.cpu.time,process.memory.usage,process.thread.count, andprocess.open_file_descriptor.count. Each process also reportsprocess.pid, so the Uvicorn workers of one server stay distinguishable.gym.server.requests.in_flight,gym.server.requests.cancelled_total,gym.server.requests.unhandled_exceptions_total, andgym.server.responses_totalbyhttp.response.status_code, which counts handled errors such as anHTTPExceptionas their 4xx or 5xx.gym.http.client.requests.in_flightfor outbound calls still waiting for their response, andgym.http.connection_pool.connect_duration_msfor each new pooled connection, next to the existing queue-wait metric.gym.driver.rollouts.offered_total,gym.driver.rollouts.resident, andgym.driver.rollouts.active, plusgym.driver.admission_duration_ms,gym.driver.output_write_duration_ms,gym.driver.output_write_bytes, andgym.driver.materialize_duration_ms. A standalonegym eval run --no-servedriver now sets up its own telemetry, which also lets it export the existinggym.jobspan.gym.driver.rollout.input_items,gym.driver.rollout.output_items,gym.driver.rollout.turns, andgym.driver.rollout.tool_calls. Turns and tool calls come fromng_perfwhen rollout observability produced it.nemo_gym.telemetry.resources_operations.resources_operation, which reports how long limited work queued (gym.resources.operation.queue_duration_ms), how long it ran (gym.resources.operation.duration_ms), and how it ended (nemo.gym.resources.operation.outcome).math_with_judgeuses it for its SymPy verifier subprocesses and marks timeouts and crashes._ms, counters ending in_total, and Gym attributes undernemo.gym.. Theprocess.*metrics keep their OpenTelemetry semantic-convention names, seconds, andcpu.mode, so standard dashboards recognize them.The design keeps the work off the request path, which is why the metrics need no switch of their own. The event loop runs one sampling coroutine per interval. Per-request and per-rollout counts are plain integer updates that the exporter thread reads through observable instruments, so a request makes no OpenTelemetry call. The GC callback only buffers timestamps and totals, because it runs inside a collection. Instruments are registered once per meter, the GC callback is removed with the last observer, and the last observer to stop exports the final totals, so a short-lived driver still reports its counts.
flowchart TB subgraph onloop["On the event loop"] requests["Each request: outermost middleware and server_utils.request"] rollouts["Each rollout: driver dispatch and output write"] sampler["Once per interval: sampler coroutine"] end totals["Plain integer totals"] pauses["Bounded GC pause buffer"] hist["Histograms"] subgraph offloop["Off the event loop"] gc["gc.callbacks hook"] exporter["Exporter thread, at each export"] end requests -->|"integer updates"| totals rollouts -->|"integer updates"| totals rollouts -->|"one record per rollout"| hist sampler -->|"lag, task count, and drained GC pauses"| hist gc -->|"pause timestamps"| pauses gc -->|"collection totals"| totals pauses -->|"drained once per interval"| sampler exporter -->|"reads totals and psutil process stats"| totals exporter -->|"exports"| hist classDef hot fill:#e3f4e1,stroke:#2e7d32 class requests,rollouts hotThe green steps run once per request or once per rollout, and only update integers or record one histogram point per rollout.
Measured costs decided what runs by default:
psutilprocess readsasyncio.all_tasksfor the task counttelemetry.event_loop_task_countThe task count walks every task on the loop while holding it, and grows with concurrency, so it is the only opt-in measurement.
The sampling interval is
telemetry.event_loop_sample_interval_seconds.Why
A capacity investigation found that a model server under bursty rollouts spent about 10 ms per call waiting on its own event loop, behind the synchronous work of other requests, with no metric showing it. These metrics make that kind of slowdown visible in any run, and the driver and resources-operation metrics show where work waits outside the servers.
Issue
Part of #3155. Per-request request and response byte histograms are not included, so that the request path makes no per-request OpenTelemetry call; output row sizes are reported by
gym.driver.output_write_bytes.Compatibility
server_utils.request.telemetry.event_loop_task_count: trueorNEMO_GYM_OTEL_EVENT_LOOP_TASK_COUNT=1.process.pidin every process.gym eval run --no-servedriver now initializes telemetry from thetelemetry:block, under the service name suffixdriver.math_with_judge's_wait_for_library_verifier_processgains an optional keyword-onlyoperationargument.Validation
pytest tests/unit_tests/telemetry tests/unit_tests/test_rollout_collection.py tests/unit_tests/test_server_utils.py tests/unit_tests/test_cli_eval.py tests/unit_tests/test_cli.py -n 12with thetelemetryextra installed.tests/unit_tests/telemetry/test_observer.pycovers GC callback pairing, bounded buffering, and removal; lag from a deliberately blocked event loop; nested observers sharing one process state; no work while metrics are off; the opt-in task count; driver totals exported only by a process that offered rollouts; event-loop settings handed from the supervisor to every server; a failed start that does not stop the server; lifespan composition; request, cancellation, and failure counts; outbound in-flight counts; driver admission, residency, write, and rollout-shape metrics; and the final export of totals when the last observer stops.test_resources_operations.pycovers queue wait, execution, outcomes, bounded labels, and the disabled path.test_connection_pool.pycovers new-connection timing, reused connections, refused connections, and the aiohttp override signature.gym env test +entrypoint=resources_servers/math_with_judge +should_validate_data=true.NEMO_GYM_OTEL_EVENT_LOOP_TASK_COUNT=1was set.With telemetry off, this PR matches the base commit. With metrics on, the new server metrics add nothing measurable on top of Gym's existing metrics. The driver spends about 0.1 ms more per rollout, from its per-rollout histograms. The existing metrics pipeline costs about 2 percent of server CPU in this run, mostly from the FastAPI and aiohttp instrumentors; #4478 tracks installing them only when traces are exported.
Rollout evidence
N/A for model rollouts. The change adds observability and does not change agent, model, or verifier decisions. The capacity-simulator runs above exercise real rollouts through the driver, agent, model server, Environment Server, and resources server with a mock inference engine.