Skip to content

fix: bound how long a socket connect can hang - #2365

Closed
ananthsub wants to merge 1 commit into
ananthsub/m1-classify-and-reportfrom
ananthsub/m5-connect-deadline
Closed

ananthsub wants to merge 1 commit into
ananthsub/m1-classify-and-reportfrom
ananthsub/m5-connect-deadline

Conversation

@ananthsub

@ananthsub ananthsub commented Aug 5, 2026 •

Copy link
Copy Markdown
Contributor

Partially addresses #2216. Stacked directly on #2361. This change bounds each socket connection attempt; #2363 remains separate and is not inherited by this branch.

set_global_aiohttp_client passes a bare ClientTimeout(), which sets
total, connect, sock_read and sock_connect all to None. aiohttp's
default when the argument is omitted entirely is ClientTimeout(total=5*60, sock_connect=30), so passing an empty one turns off both.

Disabling total is deliberate and correct: a generation can legitimately run
for minutes. Disabling sock_connect looks like collateral damage rather than
a decision.

Without it, a host that silently drops connection attempts rather than
refusing them, such as a firewall DROP rule or a machine that went away, is
bounded only by the kernel's SYN retry budget. At the default tcp_syn_retries = 6 that is roughly 127 seconds per attempt, inside a retry loop that does
not stop.

This sets sock_connect and leaves the rest alone: 30s for external
endpoints, which is aiohttp's own default and therefore a restoration rather
than a new number, and 10s for calls to other Gym servers, which are near or
on the same machine.

Not connect. The aiohttp docs describe connect as covering acquisition of
a connection from the pool, which "consists connection establishment for a new
connection or waiting for a free connection from a pool if pool connection
limits are exceeded", while sock_connect covers "connecting to a peer for a
new connection, not given from a pool". The source agrees: connect wraps all
of _get_connection, sock_connect wraps only the socket connect. At Gym's
concurrency, waiting for a pool slot is normal and must not be counted as a
failure.

Not below 10s for internal calls either. Under load a localhost accept queue
can fill, the kernel drops the SYN, and the retransmit does not arrive for a
full second. A one-second deadline would manufacture failures under exactly
the conditions this is meant to survive.

ClientTimeout is set per session rather than per call, so the session
carries the external value and ServerClient.request passes the internal one
explicitly. A caller that supplies its own timeout keeps it.

@copy-pr-bot

copy-pr-bot Bot commented Aug 5, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@ananthsub
ananthsub force-pushed the ananthsub/m5-connect-deadline branch from fd2ab25 to e842533 Compare August 5, 2026 22:08
@ananthsub
ananthsub force-pushed the ananthsub/m5-connect-deadline branch from e842533 to 4d89167 Compare August 24, 2026 20:31
@copy-pr-bot

copy-pr-bot Bot commented Aug 24, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@ananthsub
ananthsub changed the base branch from ananthsub/m3-route-failures-to-sidecar to ananthsub/m1-classify-and-report August 24, 2026 21:03
Partially addresses #2216. Stacked on #2363, which is what lets the resulting
failure become a failed rollout rather than a dead run.

`set_global_aiohttp_client` passes a bare `ClientTimeout()`, which sets
`total`, `connect`, `sock_read` and `sock_connect` all to `None`. aiohttp's
default when the argument is omitted entirely is `ClientTimeout(total=5*60,
sock_connect=30)`, so passing an empty one turns off both.

Disabling `total` is deliberate and correct: a generation can legitimately run
for minutes. Disabling `sock_connect` looks like collateral damage rather than
a decision.

Without it, a host that silently drops connection attempts rather than
refusing them, such as a firewall DROP rule or a machine that went away, is
bounded only by the kernel's SYN retry budget. At the default `tcp_syn_retries
= 6` that is roughly 127 seconds per attempt, inside a retry loop that does
not stop.

This sets `sock_connect` and leaves the rest alone: 30s for external
endpoints, which is aiohttp's own default and therefore a restoration rather
than a new number, and 10s for calls to other Gym servers, which are near or
on the same machine.

Not `connect`. The aiohttp docs describe `connect` as covering acquisition of
a connection from the pool, which "consists connection establishment for a new
connection or waiting for a free connection from a pool if pool connection
limits are exceeded", while `sock_connect` covers "connecting to a peer for a
new connection, not given from a pool". The source agrees: `connect` wraps all
of `_get_connection`, `sock_connect` wraps only the socket connect. At Gym's
concurrency, waiting for a pool slot is normal and must not be counted as a
failure.

Not below 10s for internal calls either. Under load a localhost accept queue
can fill, the kernel drops the SYN, and the retransmit does not arrive for a
full second. A one-second deadline would manufacture failures under exactly
the conditions this is meant to survive.

`ClientTimeout` is set per session rather than per call, so the session
carries the external value and `ServerClient.request` passes the internal one
explicitly. A caller that supplies its own `timeout` keeps it.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
@ananthsub

Copy link
Copy Markdown
Contributor Author

The finite socket-connect deadline and tests remain part of #3159, but this stacked branch depends on the superseded #2361 implementation and does not provide the shared request-budget or replay contract. Closing as superseded; retain the connect-timeout implementation and tests when #3159 is rebuilt.

@ananthsub ananthsub closed this Sep 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant