Repository navigation
fix(runtime): set TCP_NODELAY on accepted request-plane connections - #18378
Conversation
The worker writes a small ACK frame for each request on its request-plane connection, and the frontend waits for it before awaiting the response stream. With Nagle on, an ACK written within one round trip of the previous one waits for the frontend's delayed ACK (~40 ms), which lands in that request's TTFT. Signed-off-by: jthomson04 <jwillthomson19@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
No actionable comments were generated in the recent review. WalkthroughSharedTcpServer now attempts to enable TCP_NODELAY on each accepted request-plane connection. If setting the option fails, the server logs a warning with the peer address and error, then continues handling the connection. ChangesTCP ingress
Estimated code review effort: 2 (Simple) | ~5 minutes Merge Risk:
|
✅ Dynamo PR CI passed — run 42408675231 (attempt 1) on
|
Absorbs 19 upstream commits (c5680e5..3a8226a). One conflict, the root Cargo.toml: kept upstream's dynamo-tokenizers 4.1.2, dynamo-renderer 9.0.0 and dynamo-protocols 8.5.0 bumps together with the stack's workspace members and path crates; the four lockfiles were regenerated with cargo metadata (no drift from the auto-merge). - ai-dynamo#17923 perf(kv-router) no block-lookup growth for existing-key updates: lands in lib/router/kv-indexer/src/concurrent_radix_tree_compressed/block_lookup.rs (rename-detected; the stack copy differs from upstream only by the crate::prelude import). - ai-dynamo#17741 build(deps) dynamo-tokenizers 4.1.2, dynamo-renderer 9.0.0, tokenizers 0.23.2 in lib/llm and lib/bench: root Cargo.toml conflict resolved as above; lib/llm/tests/tokenizer_cache.rs. - ai-dynamo#17520 feat(vllm) realtime text tools, held-request prefill, chat completions and sampling options: Python vllm, dynamo-protocols 8.5.0 bump. - ai-dynamo#18380 perf(bindings) direct token_ids/extra_args reads in RoutedEngine.generate and ai-dynamo#18448 refactor(bindings) py_convert: new lib/bindings/python/rust/llm/py_convert.rs, routed_engine.rs, engine.rs, _convert_preprocessed_request registered in lib.rs, _core.pyi, tests/test_preprocessed_request_conversion.py; llm/kv.rs extract_token_ids now delegates to py_convert::read_exact_u32_list with the stack's owner-crate imports kept. - ai-dynamo#18454 fix(frontend) named passthrough fields to Python chat processors: lib/llm protocols/openai/chat_completions.rs, Python sglang and vllm processors. - ai-dynamo#18253 fix(frontend) 16-bit image narrowing and EXIF opt-in: lib/llm preprocessor/media decoders, image_reader, loader. - ai-dynamo#18185 fix(frontend) rendering context in speculative prefill: lib/llm preprocessor/speculative_prefill.rs. - ai-dynamo#18428 fix(frontend) image step count without accounting: lib/llm protocols/openai/images aggregator.rs and nvext.rs. - ai-dynamo#17656 fix(frontend) recipe-based DeepSeek encoding with SGLang: Python frontend sglang_deepseek.py, sglang_prompt_renderer.py, sglang_prepost.py, sglang_processor.py. - ai-dynamo#18378 fix(runtime) TCP_NODELAY on accepted request-plane connections: lib/runtime pipeline/network/ingress/shared_tcp_endpoint.rs. - ai-dynamo#18007 test(router) routed-worker assertion for cache-salt isolation repeats: tests/router/common.py and helper.py. - ai-dynamo#18452 feat(backend) embedded engines skip Dynamo logging setup, ai-dynamo#18170 feat(trtllm) per-request metrics opt-in, ai-dynamo#18135 and ai-dynamo#16164 fix(vllm), ai-dynamo#16879 and ai-dynamo#18437 fix(recipes), c5680e5 docs community events: Python, recipes, docs; all auto-merged. No change to the router kernel, admission, worker-selection, slot-tracker, session-affinity, kv-events or kv-hints crates; no [features] table or Dynamo path-crate dependency edge changed. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Overview
The worker's TCP request-plane server never set
TCP_NODELAYon the connections it accepts. The worker writes a small ACK frame for every request, and the frontend waits for that ACK before it waits for the response stream. When two requests reach the same worker within one round trip, Nagle holds the second ACK until the frontend's delayed ACK fires, which adds up to about 40 ms to that request's TTFT. This PR setsTCP_NODELAYon each accepted request-plane connection; every other TCP socket on the request path already sets it.Key Changes
3 lines of core logic.
shared_tcp_endpoint.rs:SharedTcpServer::accept_loopsetsTCP_NODELAYon each accepted stream. It is set before the optional TLS handshake, so TLS connections get it too.Validation
send_requesttook over 10 msSetup: Tyche; one frontend (
--router-mode kv, jemalloc,DYN_TOKENIZER=fastokens) in front of 32dynamo.mockerworkers with 2.5 ms decode passes; closed loop at concurrency 32; 8,205-token prompts sharing a 4,096-token prefix, 128 output tokens; 60 s window, one run per arm. The default HF tokenizer was swapped for fastokens because its tokenize time otherwise hides this tail.dynamo_request_plane_send_seconds(dispatch plus the ACK wait) was 10–50 ms. Tokenization and request-plane queueing were under 0.5 ms for all of them.TCP_NODELAYthe second ACK frame leaves as soon as it is written. All 6,016 requests finishsend_requestin under 0.5 ms, and CPU and throughput are unchanged.Issues and Ongoing Work
service_v2.rs).Pull Request Checklist
/pr-checklistskill.🤖 Generated with Claude Code
Summary by CodeRabbit