Bug Description
In the SGLang PD router path for OpenAI-compatible chat completions, the cache-aware routing text is currently built only from the first chat message:
see https://github.com/lightseekorg/smg/blob/main/model_gateway/src/routers/http/pd_router.rs#L1445
sglang router also has this problem but already fixed by sgl-project/sglang#27430
Steps to Reproduce
- Config smg as http pd router.
- Continuous chat multi turn
- Cache hit rate is extremely low
Expected Behavior
routing with full conversation
Actual Behavior
only route with first message
Component
model-gateway (core routing)
Routing Policy (if applicable)
None
Connection Mode
None
Configuration
Logs / Error Output
Environment
os: linux
Deployment Environment
Docker
Streaming Context
Additional Context
No response
Pre-submission Checklist
Bug Description
In the SGLang PD router path for OpenAI-compatible chat completions, the cache-aware routing text is currently built only from the first chat message:
see https://github.com/lightseekorg/smg/blob/main/model_gateway/src/routers/http/pd_router.rs#L1445
sglang router also has this problem but already fixed by sgl-project/sglang#27430
Steps to Reproduce
Expected Behavior
routing with full conversation
Actual Behavior
only route with first message
Component
model-gateway (core routing)
Routing Policy (if applicable)
None
Connection Mode
None
Configuration
Logs / Error Output
Environment
os: linux
Deployment Environment
Docker
Streaming Context
Additional Context
No response
Pre-submission Checklist