Repository navigation
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
ananthsub
force-pushed
the
ananthsub/m5-connect-deadline
branch
from
August 5, 2026 22:08
fd2ab25 to
e842533
Compare
ananthsub
force-pushed
the
ananthsub/m5-connect-deadline
branch
from
August 24, 2026 20:31
e842533 to
4d89167
Compare
ananthsub
changed the base branch from
ananthsub/m3-route-failures-to-sidecar
to
ananthsub/m1-classify-and-report
August 24, 2026 21:03
8 tasks
Partially addresses #2216. Stacked on #2363, which is what lets the resulting failure become a failed rollout rather than a dead run. `set_global_aiohttp_client` passes a bare `ClientTimeout()`, which sets `total`, `connect`, `sock_read` and `sock_connect` all to `None`. aiohttp's default when the argument is omitted entirely is `ClientTimeout(total=5*60, sock_connect=30)`, so passing an empty one turns off both. Disabling `total` is deliberate and correct: a generation can legitimately run for minutes. Disabling `sock_connect` looks like collateral damage rather than a decision. Without it, a host that silently drops connection attempts rather than refusing them, such as a firewall DROP rule or a machine that went away, is bounded only by the kernel's SYN retry budget. At the default `tcp_syn_retries = 6` that is roughly 127 seconds per attempt, inside a retry loop that does not stop. This sets `sock_connect` and leaves the rest alone: 30s for external endpoints, which is aiohttp's own default and therefore a restoration rather than a new number, and 10s for calls to other Gym servers, which are near or on the same machine. Not `connect`. The aiohttp docs describe `connect` as covering acquisition of a connection from the pool, which "consists connection establishment for a new connection or waiting for a free connection from a pool if pool connection limits are exceeded", while `sock_connect` covers "connecting to a peer for a new connection, not given from a pool". The source agrees: `connect` wraps all of `_get_connection`, `sock_connect` wraps only the socket connect. At Gym's concurrency, waiting for a pool slot is normal and must not be counted as a failure. Not below 10s for internal calls either. Under load a localhost accept queue can fill, the kernel drops the SYN, and the retransmit does not arrive for a full second. A one-second deadline would manufacture failures under exactly the conditions this is meant to survive. `ClientTimeout` is set per session rather than per call, so the session carries the external value and `ServerClient.request` passes the internal one explicitly. A caller that supplies its own `timeout` keeps it. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
ananthsub
force-pushed
the
ananthsub/m5-connect-deadline
branch
from
August 25, 2026 20:15
4d89167 to
4f3a238
Compare
6 tasks
Contributor
Author
|
The finite socket-connect deadline and tests remain part of #3159, but this stacked branch depends on the superseded #2361 implementation and does not provide the shared request-budget or replay contract. Closing as superseded; retain the connect-timeout implementation and tests when #3159 is rebuilt. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Partially addresses #2216. Stacked directly on #2361. This change bounds each socket connection attempt; #2363 remains separate and is not inherited by this branch.
set_global_aiohttp_clientpasses a bareClientTimeout(), which setstotal,connect,sock_readandsock_connectall toNone. aiohttp'sdefault when the argument is omitted entirely is
ClientTimeout(total=5*60, sock_connect=30), so passing an empty one turns off both.Disabling
totalis deliberate and correct: a generation can legitimately runfor minutes. Disabling
sock_connectlooks like collateral damage rather thana decision.
Without it, a host that silently drops connection attempts rather than
refusing them, such as a firewall DROP rule or a machine that went away, is
bounded only by the kernel's SYN retry budget. At the default
tcp_syn_retries = 6that is roughly 127 seconds per attempt, inside a retry loop that doesnot stop.
This sets
sock_connectand leaves the rest alone: 30s for externalendpoints, which is aiohttp's own default and therefore a restoration rather
than a new number, and 10s for calls to other Gym servers, which are near or
on the same machine.
Not
connect. The aiohttp docs describeconnectas covering acquisition ofa connection from the pool, which "consists connection establishment for a new
connection or waiting for a free connection from a pool if pool connection
limits are exceeded", while
sock_connectcovers "connecting to a peer for anew connection, not given from a pool". The source agrees:
connectwraps allof
_get_connection,sock_connectwraps only the socket connect. At Gym'sconcurrency, waiting for a pool slot is normal and must not be counted as a
failure.
Not below 10s for internal calls either. Under load a localhost accept queue
can fill, the kernel drops the SYN, and the retransmit does not arrive for a
full second. A one-second deadline would manufacture failures under exactly
the conditions this is meant to survive.
ClientTimeoutis set per session rather than per call, so the sessioncarries the external value and
ServerClient.requestpasses the internal oneexplicitly. A caller that supplies its own
timeoutkeeps it.