Skip to content

[Bug]: http pd router only use the first message #1928

Description

@Moersity

Bug Description

In the SGLang PD router path for OpenAI-compatible chat completions, the cache-aware routing text is currently built only from the first chat message:
see https://github.com/lightseekorg/smg/blob/main/model_gateway/src/routers/http/pd_router.rs#L1445

sglang router also has this problem but already fixed by sgl-project/sglang#27430

Steps to Reproduce

  1. Config smg as http pd router.
  2. Continuous chat multi turn
  3. Cache hit rate is extremely low

Expected Behavior

routing with full conversation

Actual Behavior

only route with first message

Component

model-gateway (core routing)

Routing Policy (if applicable)

None

Connection Mode

None

Configuration

Logs / Error Output

Environment

os: linux

Deployment Environment

Docker

Streaming Context

  • Issue occurs with streaming (SSE) responses
  • Issue occurs with non-streaming responses
  • Issue occurs in both modes

Additional Context

No response

Pre-submission Checklist

  • I have searched existing issues and discussions
  • I can reproduce this issue consistently
  • I am using the latest version of SMG

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions