fix(router): use full conversation for PD chat cache-aware routing (#26263)

[pjdurden]

Motivation

Closes #26263.

The PD router's cache-aware routing builds its routing text for chat
completions from messages.first() only. For multi-turn conversations this
ignores the rest of the conversation (system prompt, prior turns, the current
message, tool context) — exactly the content that drives KV-cache reuse — so
prefix matching is dominated by system-prompt similarity rather than the real
cached prefix.

As reported in the issue, this materially hurts cache locality:

• first-message-only: ~69% cache hits, ~678 output TPS
• full-context: ~96% cache hits, ~1078–1085 output TPS

Modifications

• route_chat in sgl-model-gateway/src/routers/http/pd_router.rs now builds
  routing text via ChatCompletionRequest::extract_text_for_routing() — the
  same method the regular (non-PD) HTTP router already uses
  (routers/http/router.rs) — so both routers derive routing text identically
  from the full conversation.
• Extracted the logic into a small build_chat_request_text helper and removed
  the now-unused first-message ChatMessage/MessageContent matching.

Tests

• Added regression test test_chat_request_text_uses_full_conversation,
  asserting later turns (not just the first message) are included in the
  routing text.
• Full library test suite passes (cargo test --lib, 384 tests).
• cargo fmt --check and cargo clippy are clean.

---

CI States

Latest PR Test (Base): ✅ Run #27057060542
Latest PR Test (Extra): ❌ Run #27930620068

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR fixes cache-aware routing for PD chat requests by ensuring the routing text is derived from the full conversation (not just the first message), improving KV-cache prefix matching accuracy.

Changes:

• Switch chat request routing text generation to reuse extract_text_for_routing() over the full conversation.
• Add a dedicated helper (build_chat_request_text) for PD chat routing text creation.
• Add a regression test covering multi-turn chat routing text behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[gemini-code-assist]

Code Review

This pull request updates the PD router to build routing text from the full conversation history rather than just the first message, ensuring consistent cache-aware routing in multi-turn chats. It also adds a regression test for this behavior. The review feedback points out an unused import of GenerationRequest and suggests returning None instead of Some("") when the extracted routing text is empty to prevent routing issues.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[pjdurden]

@slin1237 gentle nudge when you have time. The PD router cache routing fix is tested and lint clean, it just needs the run ci label to unblock the gate. Happy to adjust anything.

[petrpan26]

woulld love to have this merged in as well ! @slin1237

[pjdurden]

@CatherineSue could you take a look when you get a chance? This is a small one-file fix in src/routers for PD chat cache-aware routing (#26263). Approved already and lint clean, just needs the run ci label to clear the gate. Thanks!

[slin1237]

i thought I fixed it a lifetime ago. Why is this broken again 😓

[petrpan26]

wooo merged ! would love to see this push into new version