Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
1e228e7
Expose aiohttp pool wait telemetry
athitten Sep 22, 2026
1a45570
fix(core): attribute pool pressure and harden capacity reporting
athitten Sep 23, 2026
b111268
fix(core): record benchmark errors and use the production client
athitten Sep 23, 2026
9b3f3d2
Remove benchmark script
athitten Sep 23, 2026
6c46eae
Merge latest main
athitten Sep 23, 2026
db5d90e
Rename pool queue constraint attribute
athitten Sep 23, 2026
d7673d7
fix(core): address pool telemetry review
athitten Sep 29, 2026
61269a2
docs(agent): clarify remote retry ownership
athitten Sep 29, 2026
bb73579
Merge origin/main into athitten/issue-3008
athitten Sep 29, 2026
9a6605f
perf: reduce connection queue metric overhead
athitten Sep 30, 2026
e40867e
fix: avoid misleading descriptor warnings
athitten Sep 30, 2026
16a21a9
fix: label first outbound connection attempt
athitten Sep 30, 2026
ab01a21
fix: preserve client initialization tracing
athitten Sep 30, 2026
d1af7a6
test: cover pool tracing and destination labels
athitten Oct 1, 2026
2bf2734
test: bound connection pool test cleanup
athitten Oct 1, 2026
e62eb24
refactor: clarify metrics exporter state
athitten Oct 1, 2026
0cb5555
fix: identify HTTP metric destinations
athitten Oct 1, 2026
6afdfda
test: verify queued redirect sampling
athitten Oct 1, 2026
c569325
docs: clarify sustained connector overshoot
athitten Oct 1, 2026
0d5f727
docs: clarify pool metrics process coverage
athitten Oct 1, 2026
7c255d6
fix: label per-worker pool capacity
athitten Oct 1, 2026
7b0439d
docs: explain pool capacity and destinations
athitten Oct 1, 2026
e74b8e8
docs: clarify pool sizing guidance
athitten Oct 1, 2026
10f4317
test: cover pool limit edge cases and startup report
athitten Oct 1, 2026
1b1f171
test: assert remote agent pool destination label
athitten Oct 1, 2026
21f2b12
test: bound pool tests and guard connect override
athitten Oct 1, 2026
3639e88
Apply batched suggestions from code review
athitten Oct 1, 2026
de00582
fix(core): address pool telemetry follow-up review
athitten Oct 1, 2026
d6b15ff
Merge origin/main into athitten/issue-3008
athitten Oct 1, 2026
a8c03d7
fix(core): record pool metrics when reusing lens providers
athitten Oct 1, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,12 @@ traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
version
```

CLIENT spans measure total request time. When metrics export is enabled, the separate
`gym.http.connection_pool.queue_duration_ms` histogram measures connection-pool wait without
requiring a span for every request. See
[Configuration](/main/reference/configuration#http-connection-pool) for the metric and pool-sizing
reference.

**Inbound.** Every NeMo Gym server applies OpenTelemetry FastAPI auto-instrumentation,
which reads `traceparent` off the incoming request and makes the caller's span the parent
of its own SERVER span.
Expand Down
24 changes: 24 additions & 0 deletions fern/versions/latest/pages/observability/metrics.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ would disagree the first time one dropped a sample.
| `gym.verify.duration_ms` | Histogram | ms | Wall-clock duration of one verification. |
| `gym.verify.success_rate` | Gauge | fraction | Cumulative fraction of verifications that completed. See below. |
| `gym.servers.active` | Gauge | count | Number of servers the orchestrator started. |
| `gym.http.connection_pool.queue_duration_ms` | Histogram | ms | Time one queued connection acquisition waited for an aiohttp pool slot. |
| `gym.http.connection_pool.connect_total` | Observable counter | count | All outbound aiohttp connection attempts (`connect()` calls), including attempts that later fail or are abandoned. |
| `gym.server.memory.usage` | Gauge | bytes | Current RSS or PSS for one managed server and its descendants. |
| `gym.server.memory.sampled_peak` | Gauge | bytes | Largest value observed by the sampler for one managed server and its descendants. |
| `gym.server.processes` | Gauge | count | Processes currently attributed to one managed server. |
Expand Down Expand Up @@ -84,6 +86,28 @@ server:
Use these for request latency. They are dimensioned, so you can compare `/verify` against
`/v1/responses`, or isolate 500s, which is what latency questions usually require.

The Gym-specific `gym.http.connection_pool.queue_duration_ms` histogram measures client-side
connection acquisition separately from request latency. Its bounded attributes identify the
destination server, queue outcome (`ok` or `abandoned`), and which pool constraint caused the
wait (`total`, `per_host`, or `unknown`). It records one sample for each queued connection
acquisition (`connect()`), combining repeated waiter wakeups within that acquisition.
Redirect hops, aiohttp's internal reconnects, and Gym retries record separate samples when
their acquisitions queue. Connection attempts that acquire a slot immediately do not record
a queue-duration histogram sample.
`abandoned` means the wait ended before a slot became available, including timeout and cancellation.
Divide the number of `gym.http.connection_pool.queue_duration_ms` histogram samples by
`gym.http.connection_pool.connect_total` over the same interval and destination to calculate the
fraction of connection attempts that queued.

Both instruments use `nemo.gym.http.destination.server.name` for the request destination:
the configured server name for `ServerClient` calls, `remote_agent_service` for calls to the
remote agent's external service, or `external` for unlabeled direct calls. This differs from
the resource attribute `nemo.gym.server.name`, which identifies the Gym process recording the metric.

These pool metrics require telemetry initialization before client construction. Standalone
collection does not initialize telemetry automatically, so missing metrics do not mean no
queueing. See [HTTP Connection Pool](/reference/configuration#http-connection-pool) for process coverage.

### A Note on `gym.server.request_duration_ms`

`nemo-lens` declares a `gym.server.request_duration_ms` histogram. NeMo Gym does not use
Expand Down
61 changes: 61 additions & 0 deletions fern/versions/latest/pages/reference/configuration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,67 @@ server_id: # Your unique name for this server
# ... additional fields vary by server type
```

### HTTP Connection Pool

Gym uses one shared aiohttp connection pool per process. Each top-level limit is a budget for
one server process group (or the rollout CLI), divided across that server's FastAPI workers. It
is not a deployment-wide budget: every agent, model, and resources server gets its own pool.

```yaml
global_aiohttp_connector_limit: 102400
global_aiohttp_connector_limit_per_host: 1024

# Optional sizing checks. Set these when the expected outbound concurrency is known.
global_aiohttp_intended_concurrency: 4096
global_aiohttp_intended_concurrency_per_host: 1024
```

At startup, Gym logs the configured and effective per-worker limits once per server. An explicit
connector limit of `0` keeps aiohttp's existing meaning of unlimited. Gym rejects a positive
limit that becomes zero after worker division, because that would accidentally turn a bounded
pool into an unlimited one. The optional intended-concurrency values also apply separately to
each server or CLI pool. When intended demand is larger than effective capacity or can exhaust the
file-descriptor budget, Gym logs an actionable warning. The connector limit and file-descriptor
soft limit remain in the startup information line, but the connector ceiling alone does not trigger
a file-descriptor warning because it is not an allocation.

The total aiohttp limit is a scheduling limit, not a strict open-socket cap across multiple
hosts. Idle-connection reuse and aiohttp's waiter wake-up order can exceed the total limit for

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

configuration.mdx:121

Non-blocking: one more sentence here would explain the bursty total-limit waits in the PR's own benchmark table.

connect() tries to reuse an idle pooled connection before it checks any limit or joins the wait queue (connector.py#L627-L634).
A new request for a host with an idle socket therefore skips requests that are already waiting.
Waits behind the total limit are not first come, first served, so a few requests can wait much longer than the backend latency.
Please add a sentence saying that total-limit waits are not FIFO, because requests that reuse pooled sockets bypass waiting requests.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in fern/versions/latest/pages/reference/configuration.mdx:122. Added that pool waits are not first come, first served: a request that can reuse an idle connection skips waiting requests, and aiohttp wakes waiting hosts in random order.

as long as requests to more than one host are in flight. Pool waits are also not first come, first
served: a new request that can reuse an idle pooled connection skips requests that are already
waiting, and aiohttp wakes waiting hosts in random order, so a few requests can wait much longer than
the backend latency. When sizing pools, keep file-descriptor, ephemeral-port, and downstream
connection budgets in mind.

Pool metrics require an active metrics exporter before the shared client is constructed.
Gym server processes and the orchestrator used by `gym eval run` initialize telemetry
automatically. When Gym telemetry is enabled and another library, such as NeMo RL, already set
up nemo-lens in the same process, Gym reuses its providers, and pool metrics follow that library's
metrics setting.
Standalone `gym eval run --no-serve` and library calls to `setup_server_client()`
do not initialize telemetry; callers must initialize it explicitly before creating the client.
Initializing telemetry later does not replace an existing connector. Missing metrics from
these standalone processes do not mean that no requests queued.

When this instrumentation is installed, Gym records one histogram sample per queued aiohttp
connection acquisition. This metric is independent of the `http_client` tracing span group:

| Metric or attribute | Meaning |
| --- | --- |
| `gym.http.connection_pool.queue_duration_ms` | Histogram of wait time for queued connects; immediate connects do not record a sample |
| `gym.http.connection_pool.connect_total` | Observable counter of all connection attempts (`connect()` calls), including attempts that later fail or are abandoned; use it as the denominator for the queued fraction |
| `nemo.gym.http.connection_pool.queue_constraint` | Binding limit: `total`, `per_host`, or `unknown` |
| `nemo.gym.http.connection_pool.queue_outcome` | Queue outcome: `ok`, or `abandoned` when the wait ends before a slot is available, including timeout and cancellation |
| `nemo.gym.http.destination.server.name` | Configured destination server name for `ServerClient` calls, `remote_agent_service` for the remote agent's external service, or `external` for unlabeled direct calls |

The destination attribute identifies the configured or fixed request destination, including
across redirects. The resource attribute `nemo.gym.server.name` identifies the Gym process
recording the metric; they are not the same.

When metrics export is active at client construction, Gym uses `QueueTimedTCPConnector` to count
connection attempts and measure queue waits. Otherwise, Gym uses aiohttp's `TCPConnector` directly
without collecting these metrics.

### Reverse Proxy Headers

Gym disables Uvicorn proxy-header processing by default. Direct Gym traffic does not pass through a reverse proxy, so
Expand Down
Loading
Loading