Skip to content

fix(runtime): set TCP_NODELAY on accepted request-plane connections - #18378

Merged
jthomson04 merged 1 commit into
mainfrom
jthomson04/request-plane-nodelay
Nov 9, 2026
Merged

jthomson04 merged 1 commit into
mainfrom
jthomson04/request-plane-nodelay

Conversation

@jthomson04

@jthomson04 jthomson04 commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Overview

The worker's TCP request-plane server never set TCP_NODELAY on the connections it accepts. The worker writes a small ACK frame for every request, and the frontend waits for that ACK before it waits for the response stream. When two requests reach the same worker within one round trip, Nagle holds the second ACK until the frontend's delayed ACK fires, which adds up to about 40 ms to that request's TTFT. This PR sets TCP_NODELAY on each accepted request-plane connection; every other TCP socket on the request path already sets it.

Key Changes

3 lines of core logic.

  • shared_tcp_endpoint.rs: SharedTcpServer::accept_loop sets TCP_NODELAY on each accepted stream. It is set before the optional TLS handshake, so TLS connections get it too.

Validation

Concurrency 32 main this PR
TTFT p50 / p90 / p99 1.6 / 5.8 / 42.2 ms 1.5 / 1.6 / 1.7 ms
Requests whose send_request took over 10 ms 467 of 6,016 (7.8%) 0 of 6,016
Router CPU per request 7.5 ms 7.5 ms
Throughput 100 req/s 100 req/s

Setup: Tyche; one frontend (--router-mode kv, jemalloc, DYN_TOKENIZER=fastokens) in front of 32 dynamo.mocker workers with 2.5 ms decode passes; closed loop at concurrency 32; 8,205-token prompts sharing a 4,096-token prefix, 128 output tokens; 60 s window, one run per arm. The default HF tokenizer was swapped for fastokens because its tokenize time otherwise hides this tail.

  • Cause: on main, the 467 slow requests are exactly the ones whose dynamo_request_plane_send_seconds (dispatch plus the ACK wait) was 10–50 ms. Tokenization and request-plane queueing were under 0.5 ms for all of them.
  • Fix: with TCP_NODELAY the second ACK frame leaves as soon as it is written. All 6,016 requests finish send_request in under 0.5 ms, and CPU and throughput are unchanged.
  • No unit test: the accepted socket is internal to the accept loop, and the only observable effect is ACK timing on the wire, so a test would have to assert on the kernel's delayed-ACK timing.

Issues and Ongoing Work

Pull Request Checklist
  • I used Dynamo's /pr-checklist skill.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Performance
    • Enabled TCP_NODELAY on accepted request connections to reduce delays in sending data.

The worker writes a small ACK frame for each request on its request-plane
connection, and the frontend waits for it before awaiting the response
stream. With Nagle on, an ACK written within one round trip of the previous
one waits for the frontend's delayed ACK (~40 ms), which lands in that
request's TTFT.

Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jthomson04
jthomson04 requested a review from a team as a code owner November 7, 2026 19:57
@github-actions github-actions Bot added the fix label Nov 7, 2026
@coderabbitai

coderabbitai Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

SharedTcpServer now attempts to enable TCP_NODELAY on each accepted request-plane connection. If setting the option fails, the server logs a warning with the peer address and error, then continues handling the connection.

Changes

TCP ingress

Layer / File(s) Summary
Accepted connection setup
lib/runtime/src/pipeline/network/ingress/shared_tcp_endpoint.rs
Each accepted connection now attempts to enable TCP_NODELAY. On failure, the server logs a warning with the peer address and error, then continues handling the connection.

Estimated code review effort: 2 (Simple) | ~5 minutes

Merge Risk: Minimal · up to b1838

Connections continue even if TCP_NODELAY cannot be set. No merge-blocking risk is evident.

Pre-merge checks: Passed 5

Check name Status Explanation
Title check Passed The title "fix(runtime): set TCP_NODELAY on accepted request-plane connections" matches the single change in the changeset. The raw summary says SharedTcpServer::accept_loop now enables TCP_NODELAY on…
Docstring Coverage Passed No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description includes all required sections, explains the change, provides validation results, and includes the checklist. It also identifies related issues, although it does not state that this PR…
Full details: Title check

Explanation

The title "fix(runtime): set TCP_NODELAY on accepted request-plane connections" matches the single change in the changeset. The raw summary says SharedTcpServer::accept_loop now enables TCP_NODELAY on each accepted request-plane connection and logs a warning if that fails. The PR objectives describe the same change. The title is specific and uses the conventional commit format. It is 67 characters, which is within a reasonable length.

Full details: Docstring Coverage

Explanation

No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 1 files.

Full details: Description check

Explanation

The description includes all required sections, explains the change, provides validation results, and includes the checklist. It also identifies related issues, although it does not state that this PR closes an issue.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 1dd1eef and b18386e.

Files selected for processing: 1

  • lib/runtime/src/pipeline/network/ingress/shared_tcp_endpoint.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@devin-ai-integration

Copy link
Copy Markdown
Contributor

✅ Dynamo PR CI passed — run 42408675231 (attempt 1) on b18386ed7a

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Posted automatically by Devin for run 42408675231. Updated on every full-CI run of this PR.

@jthomson04
jthomson04 merged commit d477bfb into main Nov 9, 2026
140 checks passed
@jthomson04
jthomson04 deleted the jthomson04/request-plane-nodelay branch November 9, 2026 17:47
ishandhanani added a commit to ishandhanani/dynamo that referenced this pull request Nov 10, 2026
Absorbs 19 upstream commits (c5680e5..3a8226a). One conflict, the root Cargo.toml: kept upstream's dynamo-tokenizers 4.1.2, dynamo-renderer 9.0.0 and dynamo-protocols 8.5.0 bumps together with the stack's workspace members and path crates; the four lockfiles were regenerated with cargo metadata (no drift from the auto-merge).

- ai-dynamo#17923 perf(kv-router) no block-lookup growth for existing-key updates: lands in lib/router/kv-indexer/src/concurrent_radix_tree_compressed/block_lookup.rs (rename-detected; the stack copy differs from upstream only by the crate::prelude import).
- ai-dynamo#17741 build(deps) dynamo-tokenizers 4.1.2, dynamo-renderer 9.0.0, tokenizers 0.23.2 in lib/llm and lib/bench: root Cargo.toml conflict resolved as above; lib/llm/tests/tokenizer_cache.rs.
- ai-dynamo#17520 feat(vllm) realtime text tools, held-request prefill, chat completions and sampling options: Python vllm, dynamo-protocols 8.5.0 bump.
- ai-dynamo#18380 perf(bindings) direct token_ids/extra_args reads in RoutedEngine.generate and ai-dynamo#18448 refactor(bindings) py_convert: new lib/bindings/python/rust/llm/py_convert.rs, routed_engine.rs, engine.rs, _convert_preprocessed_request registered in lib.rs, _core.pyi, tests/test_preprocessed_request_conversion.py; llm/kv.rs extract_token_ids now delegates to py_convert::read_exact_u32_list with the stack's owner-crate imports kept.
- ai-dynamo#18454 fix(frontend) named passthrough fields to Python chat processors: lib/llm protocols/openai/chat_completions.rs, Python sglang and vllm processors.
- ai-dynamo#18253 fix(frontend) 16-bit image narrowing and EXIF opt-in: lib/llm preprocessor/media decoders, image_reader, loader.
- ai-dynamo#18185 fix(frontend) rendering context in speculative prefill: lib/llm preprocessor/speculative_prefill.rs.
- ai-dynamo#18428 fix(frontend) image step count without accounting: lib/llm protocols/openai/images aggregator.rs and nvext.rs.
- ai-dynamo#17656 fix(frontend) recipe-based DeepSeek encoding with SGLang: Python frontend sglang_deepseek.py, sglang_prompt_renderer.py, sglang_prepost.py, sglang_processor.py.
- ai-dynamo#18378 fix(runtime) TCP_NODELAY on accepted request-plane connections: lib/runtime pipeline/network/ingress/shared_tcp_endpoint.rs.
- ai-dynamo#18007 test(router) routed-worker assertion for cache-salt isolation repeats: tests/router/common.py and helper.py.
- ai-dynamo#18452 feat(backend) embedded engines skip Dynamo logging setup, ai-dynamo#18170 feat(trtllm) per-request metrics opt-in, ai-dynamo#18135 and ai-dynamo#16164 fix(vllm), ai-dynamo#16879 and ai-dynamo#18437 fix(recipes), c5680e5 docs community events: Python, recipes, docs; all auto-merged.

No change to the router kernel, admission, worker-selection, slot-tracker, session-affinity, kv-events or kv-hints crates; no [features] table or Dynamo path-crate dependency edge changed.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants