[Rust Frontend] Add OpenAI Responses API endpoint by gcanlin · Pull Request #53380 · vllm-project/vllm

gcanlin · 2026-08-22T12:28:30Z

Note: This PR is totally vibe coded.

Summary

Add the OpenAI Responses API to the Rust frontend, tracking the feature-parity roadmap in #44280:

POST /v1/responses — string or item-list input (messages, function calls, function call outputs, reasoning), streaming and non-streaming, with reasoning items and function tool calls, mapped onto the shared vllm-chat pipeline.
Streaming emits the full Python parity event sequence (response.created → response.in_progress → per-item response.output_item.* / part deltas → response.completed), with both event: and data: lines and a globally assigned sequence_number. Item IDs stay consistent between streamed events and the terminal payload.
Store-dependent features follow the Python store-off behavior: store=true executes without retention, background=true is rejected, previous_response_id and GET/cancel return 404, item_reference inputs are rejected. Only function tools are supported; builtin tool types return 400.
Reasoning is controlled via reasoning.effort (injects enable_thinking template kwarg like the Python frontend) and include_reasoning.

Out of scope (TODOs left in code): harmony channel analysis, builtin-tool execution, response store, logprobs on output items, truncation="auto", allowed_tools choices.

Searched open PRs for Responses-API/Rust-frontend work; none covers this endpoint (only tracked as pending in #44280).

Tests

cd rust && cargo nextest run -p vllm-server → 359 passed (12 new Responses API integration tests in routes/tests.rs covering non-streaming, streaming event order, reasoning include/exclude, function-call roundtrip across turns, and store/tool error paths), clippy/fmt clean.
Hardware check on 8×B200 (DeepSeek-V4-Flash fp8, TP8, --tool-call-parser deepseek_v4 --reasoning-parser deepseek_v4): verified end-to-end via curl — non-streaming response with reasoning+message items and usage, streaming event sequence, a real get_weather function call, function-call-output multi-turn replay, and all rejection paths. No output/accuracy-affecting change to existing endpoints.

AI assistance

This change was implemented with AI (Codex CLI) assistance; all changes were reviewed and tested by the human submitter.

Implement POST /v1/responses (plus GET/cancel stubs) in the Rust frontend, mirroring the stateless non-harmony path of the Python frontend: message/function-call/reasoning input items, streaming SSE with the full output-item event sequence, reasoning items, and function tool calls. Store-dependent features follow store-off Python behavior: store=true executes without retention, background=true is rejected, previous_response_id and retrieval/cancellation return 404. Co-authored-by: Codex Signed-off-by: Canlin Guo <canlinguosdu@gmail.com>

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

gcanlin · 2026-08-23T07:47:11Z

@BugenZhao Hi, do we have plan to support this API? It seems that codex has deprecated /chat api and we need /responses api when using rust frontend. I'm not familiar with rust so this PR may not be production ready.

franciscojavierarceo

thanks for this PR!

i maintain the vLLM Agentic API project, which is written in Rust, and recently worked on the Responses API here. i think we should keep Responses stateless, as this implementation already does, and leave response storage out of scope. i've left a few inline comments on streaming behavior, stateless replay, and request handling.

franciscojavierarceo · 2026-09-14T20:26:20Z

+                    AssistantBlockKind::Reasoning => {
+                        let mut events = self.open_reasoning_if_needed();
+                        events.push(self.reasoning_delta(delta.clone()));
+                        events


include_reasoning=false still emits reasoning over SSE, even though the final response excludes it. this also leaves the streamed message at index 1 while it appears at index 0 in the final output. Python suppresses those deltas. we should filter them before opening output items.

franciscojavierarceo · 2026-09-14T20:26:20Z

+    #[serde(default)]
+    pub metadata: Option<Value>,
+    #[serde(default)]
+    pub previous_response_id: Option<String>,


previous_input_messages is missing from the request type and silently ignored during deserialization, so clients using Python’s stateless Harmony replay format lose their supplied history. supporting that format can stay out of scope, but we should explicitly reject the field until it is supported.

franciscojavierarceo · 2026-09-14T20:26:20Z

+
+    if let Some(truncation) = &request.truncation {
+        // TODO: implement context-length truncation for `truncation="auto"`.
+        if !matches!(truncation.as_str(), "auto" | "disabled") {


truncation="auto" is accepted and echoed back without being applied, so long conversations still hit the normal context-length error. we should either implement truncation or reject the option explicitly for now.

franciscojavierarceo · 2026-09-14T20:26:20Z

+    let request_id = request
+        .request_id
+        .clone()
+        .unwrap_or_else(|| format!("resp_{}", Uuid::new_v4().simple()));


we resolve X-Request-ID in the handler, then ignore it here and use the body ID or a new UUID. this breaks request correlation and differs from the other Rust endpoints. we should use the resolved request context here.

gcanlin · 2026-09-24T03:34:31Z

@franciscojavierarceo Thank you for your review! But I have a little concern about my totally vibe PR. It's better that someone who's familiar with rust can take it. Feel free to open one new PR!

BugenZhao · 2026-09-24T10:21:22Z

Hi @gcanlin and @franciscojavierarceo, thanks so much for your work and review.

We've recently started discussing the future direction of integration between agentic-api and the Rust frontend in the community Slack server: https://vllm-dev.slack.com/archives/C0ABLP9A4RH/p1789756976480809. So far, our conclusion is that full support for the stateful Responses API will live in agentic-api, while the Rust frontend may still need thin, stateless support built around vllm_chat::ChatLlm in this main repo.

I agree with what @franciscojavierarceo said in #57499 (comment) — perhaps it would be best if we could share the protocol definitions, lifecycle, and so on between both sides. That said, I think it is also fine to temporarily provide a minimal subset on the Rust frontend side that can power use cases like Codex (just like what DeepSeek provides in their official API platform): the code would only be 2–3k lines, and it would give us an opportunity to see what changes the vllm_chat interface needs to make for this.

If this direction looks good to you, I believe I can help take over this PR and bring it to a milestone!

franciscojavierarceo · 2026-09-24T11:05:08Z

Yeah that sounds good to me. @BugenZhao. I can test wiring it up with Agentic API as well and identify any issues.

franciscojavierarceo · 2026-09-25T18:46:17Z

I prepared a validated follow-up commit for the Rust Responses frontend: franciscojavierarceo@4f80bcad71

The branch is franciscojavierarceo:codex/rust-responses-followups. I do not have push permission to gcanlin:feat/rust-responses-api, so this should be cherry-picked or merged into this draft rather than opened as a competing PR.

It closes the parity gaps identified in review:

derive Responses and engine IDs from the transport request ID;
include structured server_error details on failed SSE terminal responses;
reject unsupported model-visible/history controls instead of silently accepting them;
suppress reasoning SSE events when include_reasoning=false;
always emit the function-call arguments completion event, including empty arguments.

Validated with:

cargo fmt --check
OPENSSL_DIR=... cargo test -p vllm-server responses_ (14 passed)
OPENSSL_DIR=... cargo clippy -p vllm-server --tests -- -D warnings
relevant pre-commit hooks during commit

This is complementary to the shared Agentic API Responses event contract in vllm-project/agentic-api#382; it deliberately does not create a second vLLM Responses endpoint PR.

franciscojavierarceo · 2026-09-25T19:47:33Z

Update: the maintainer handoff is now captured as draft #58783, preserving @gcanlin’s original commit as the first commit and adding the validated parity follow-up as the second. Please continue review there; this original draft can remain as historical context or be closed once the takeover is accepted.

mergify · 2026-10-06T06:14:38Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @gcanlin.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

gcanlin requested review from BugenZhao and njhill as code owners August 22, 2026 12:28

claude Bot reviewed Aug 22, 2026

View reviewed changes

mergify Bot added the rust label Aug 22, 2026

gcanlin marked this pull request as draft August 22, 2026 12:50

franciscojavierarceo reviewed Sep 14, 2026

View reviewed changes

ai-jz mentioned this pull request Sep 18, 2026

[RFC]: Stateless Responses API in the Rust frontend #57499

Open

franciscojavierarceo mentioned this pull request Sep 18, 2026

feat(responses): add WebSocket mode for Responses API #35492

Draft

6 of 8 tasks

shiweijiezero mentioned this pull request Sep 24, 2026

feat(frontend): add native Messages and Responses APIs shiweijiezero/foretoken#184

Merged

franciscojavierarceo mentioned this pull request Sep 25, 2026

feat: add shared vLLM Responses event contract vllm-project/agentic-api#382

Draft

franciscojavierarceo mentioned this pull request Sep 25, 2026

[Rust Frontend] Add OpenAI Responses API endpoint #58783

Closed

QHarshil mentioned this pull request Sep 30, 2026

[Rust][Benchmark] Add openai-responses backend to vllm-bench #59489

Merged

4 tasks done

mergify Bot added the needs-rebase label Oct 6, 2026

gcanlin closed this Oct 19, 2026

Conversation

gcanlin commented Aug 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Tests

AI assistance

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

gcanlin commented Aug 23, 2026

Uh oh!

franciscojavierarceo left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

franciscojavierarceo Sep 14, 2026

Choose a reason for hiding this comment

Uh oh!

franciscojavierarceo Sep 14, 2026

Choose a reason for hiding this comment

Uh oh!

franciscojavierarceo Sep 14, 2026

Choose a reason for hiding this comment

Uh oh!

franciscojavierarceo Sep 14, 2026

Choose a reason for hiding this comment

Uh oh!

gcanlin commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BugenZhao commented Sep 24, 2026

Uh oh!

franciscojavierarceo commented Sep 24, 2026

Uh oh!

franciscojavierarceo commented Sep 25, 2026

Uh oh!

franciscojavierarceo commented Sep 25, 2026

Uh oh!

mergify Bot commented Oct 6, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gcanlin commented Aug 22, 2026 •

edited

Loading

franciscojavierarceo left a comment •

edited

Loading

gcanlin commented Sep 24, 2026 •

edited

Loading