feat(responses): add WebSocket mode for Responses API by rasonyang · Pull Request #35492 · vllm-project/vllm

rasonyang · 2026-02-27T07:42:50Z

Summary

Add WebSocket transport for the OpenAI Responses API, enabling persistent bidirectional connections for lower-latency streaming.

Add --max-websocket-connections CLI argument (default: 100, enforced per API server process) to control server capacity
Initialize WebSocket server state (active connection counter + lock) in init_app_state()
Add WebSocket route /v1/responses with connection limit enforcement
New websocket.py module: ConnectionContext, uuid7-based event IDs, JSON message parsing, streaming response events, warmup/error handling, and graceful shutdown

Follows the OpenAI Responses API WebSocket protocol: client connects to ws(s)://host/v1/responses, sends {"type": "response.create", ...}, and the server streams back the same event objects as SSE mode.

Files Changed

File	Change	Description
`vllm/entrypoints/launchers/cli_args.py`	Modified	`--max-websocket-connections` arg
`vllm/entrypoints/launchers/api_server/app_state.py`	Modified	WebSocket state init in `init_app_state()`
`vllm/entrypoints/openai/responses/api_router.py`	Modified	WebSocket route + connection limit
`vllm/entrypoints/openai/responses/websocket.py`	New	Core WebSocket handler
`tests/.../test_websocket.py`	New	Unit tests
`tests/.../test_websocket_integration.py`	New	Mock integration tests (Starlette TestClient)
`tests/.../test_websocket_real.py`	New	Real e2e tests (CPU server + `Qwen/Qwen2-0.5B-Instruct`)
`tests/.../test_websocket_gpu.py`	New	GPU integration tests (`Qwen/Qwen2-0.5B-Instruct`)
`scripts/run_websocket_gpu_tests.sh`	New	GPU test setup script

Notes

Connection limit is per API server process. With --api-server-count=N, the effective total is N * --max-websocket-connections. Documented in the CLI arg docstring.
previous_response_id continuation is connection-local. It resolves against a per-connection cache rather than the shared response store. A WebSocket connection is pinned to one process for its lifetime, so this stays consistent under multiple API servers.
State init moved. init_app_state() now lives in vllm/entrypoints/launchers/api_server/app_state.py — upstream turned vllm/entrypoints/openai/api_server.py into a deprecation shim, so this PR no longer touches that file.
CLI args moved. Upstream [Frontend] Move cli_args.py and dp_supervisor.py out openai folder #53659 relocated cli_args.py from vllm/entrypoints/openai/ to vllm/entrypoints/launchers/, so --max-websocket-connections now lands in the new path.

Test Plan

Results below are from a full run before the rebase onto current main (commit d3ee956b0):

Unit tests: ConnectionContext, uuid7, message parsing, event formatting
Mock integration tests: full connection lifecycle with mocked engine
Real e2e tests (13 tests): live CPU server with Qwen/Qwen2-0.5B-Instruct covering streaming, multi-turn, cancellation, concurrent connections, and error handling
GPU integration tests (12 tests on NVIDIA A10): live GPU server with Qwen/Qwen2-0.5B-Instruct, same coverage

.venv/bin/python -m pytest tests/entrypoints/openai/responses/test_websocket.py -v
.venv/bin/python -m pytest tests/entrypoints/openai/responses/test_websocket_integration.py -v
.venv/bin/python -m pytest tests/entrypoints/openai/responses/test_websocket_real.py -v
bash scripts/run_websocket_gpu_tests.sh

Re-run after the rebase onto current main:

ruff check / ruff format --check (v0.14.0, matching .pre-commit-config.yaml) on all changed files — clean
Unit tests (18) and mock integration tests (4) — all pass. These two suites are written to run without torch; the dev machine has no CUDA build, and tests/entrypoints/openai/responses/conftest.py imports tests.utils (which needs torch), so they were executed by importing the modules directly rather than through pytest.
test_websocket_real.py and test_websocket_gpu.py — not re-run post-rebase; they need a server and a GPU box.
CI — pre-run-check fails with PR must have the 'verified', 'ready', or 'ready-run-all-tests' label to run pre-commit, or the author must have at least 4 merged PRs (found 0), which skips pre-commit and gates Buildkite off entirely. This PR has therefore never had a CI run. A maintainer applying ready would let CI validate it; the workflow triggers on labeled, so no new push is needed.

Model evaluation

Not applicable — this adds a transport for existing Responses API functionality. Generation goes through the unchanged create_responses() serving pipeline, so model output and accuracy are unaffected. No sampling, tokenization, or scheduling code is touched.

Duplicate-work check

gh pr list --repo vllm-project/vllm --state open --search "websocket responses" returns only this PR. The one adjacent open PR, #38084 (/v1/responses/render endpoint), is a separate feature and does not overlap.

AI assistance

AI assistance was used for parts of this change, including the rebases onto current main, the conflict resolution, the sanitize_message fix for the error paths flagged in review, and the test fixes that followed upstream's module relocations.

gemini-code-assist

Code Review

This pull request introduces WebSocket support for the Responses API, which is a significant feature enhancement. The implementation is comprehensive, including a new CLI argument for connection limits, state management, a new WebSocket handler module, and extensive tests (unit, integration, and e2e). The code is generally well-structured and robust. I've found one critical issue in the WebSocket connection handling logic that could lead to a runtime error, and a related redundancy. My review includes suggestions to fix this. Overall, this is a great contribution.

gemini-code-assist · 2026-02-27T07:44:52Z

+    app = websocket.app
+    serving = app.state.openai_serving_responses
+
+    if serving is None:
+        await websocket.close(code=1008,
+                              reason="Responses API not supported")
+        return
+
+    # Connection limit check
+    async with app.state.ws_responses_lock:
+        if (app.state.ws_responses_active_connections
+                >= app.state.ws_responses_max_connections):
+            await websocket.accept()
+            await websocket.send_text(json.dumps({
+                "type": "error",
+                "status": 429,
+                "error": {
+                    "code": "websocket_connection_limit_reached",
+                    "message": "Maximum WebSocket connections reached",
+                },
+            }))
+            await websocket.close()
+            return
+        app.state.ws_responses_active_connections += 1


The current WebSocket connection handling logic has a bug and can be simplified.

Bug: If serving is None, websocket.close() is called before websocket.accept(), which will raise a RuntimeError: Cannot close a WebSocket connection that is not open.

Redundancy: websocket.accept() is called in the connection limit check path, and also inside WebSocketResponsesConnection.handle_connection(). This is redundant.

To fix this, you should accept the connection once at the beginning of the create_responses_websocket function. This ensures the connection is open before any other operations and simplifies the logic.

await websocket.accept() app = websocket.app serving = app.state.openai_serving_responses if serving is None: await websocket.close(code=1008, reason="Responses API not supported") return # Connection limit check async with app.state.ws_responses_lock: if (app.state.ws_responses_active_connections >= app.state.ws_responses_max_connections): await websocket.send_text(json.dumps({ "type": "error", "status": 429, "error": { "code": "websocket_connection_limit_reached", "message": "Maximum WebSocket connections reached", }, })) await websocket.close() return app.state.ws_responses_active_connections += 1

Thanks for catching this! Both issues are already resolved in the current code:

Bug fixed: websocket.accept() is now called first (line 155), before the serving is None check — so we never attempt to close an unaccepted connection.

No redundancy: websocket.py (handle_connection) does not call accept() — it's only called once in the router.

The current flow matches exactly what you suggested.

gemini-code-assist · 2026-02-27T07:44:52Z

+        """Main WebSocket connection loop."""
+        from starlette.websockets import WebSocketDisconnect
+
+        await self.websocket.accept()


Following the proposed change in api_router.py to accept the WebSocket connection in the router, this accept() call becomes redundant and should be removed. The connection will already be accepted before handle_connection is called.

Already addressed — handle_connection() in websocket.py does not call accept(). The connection is accepted once in the router before handle_connection is called, so there's no redundancy.

github-actions · 2026-02-27T07:45:25Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors.

You ask your reviewers to trigger select CI tests on top of fastcheck CI.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

🚀

chaunceyjiang · 2026-02-27T07:54:37Z

@@ -137,5 +137,55 @@ async def cancel_responses(response_id: str, raw_request: Request):
    return JSONResponse(content=response.model_dump())


+@router.websocket("/v1/responses")


Just a quick question: is this compatible with the OpenAI Responses API WebSocket standard?

Could you also write some E2E tests?

Thanks for the review @chaunceyjiang!

OpenAI compatibility — yes, this implementation follows the OpenAI Responses API WebSocket standard. Specifically:

Spec requirement Implementation

Endpoint ws(s)://host/v1/responses @router.websocket("/v1/responses") in api_router.py

Client sends {"type": "response.create", ...} handle_event() dispatches on type == "response.create"

Server streams same events as SSE mode _process_response_create() forwards the existing streaming pipeline events over WebSocket frames

stream / background fields omitted from WS payload Stripped in _process_response_create() (payload["stream"] = True, payload.pop("background", None))

One in-flight request per connection ctx.inflight guard with concurrent_request error (409)

previous_response_id for continuation via connection-local cache ConnectionContext.last_response_id/last_response with previous_response_not_found error

generate: false warmup mode Returns response.created + response.completed with no inference

60-minute connection lifetime _enforce_deadline() background task (warning at 55 min, close at 60 min)

Error format {type: "error", status, error: {code, message}} send_error() method

E2E tests — already included in this PR at tests/entrypoints/openai/responses/test_websocket_real.py. These spin up a real vLLM server with a tiny model and exercise the full WebSocket flow end-to-end. Coverage includes:

Basic streaming (connect → response.create → receive events → response.completed)

Streaming event ordering (created → in_progress → output_item.added → deltas → done → completed)

Text consistency (concatenated deltas == final output text)

Warmup mode (generate: false)

Continuation with previous_response_id

Invalid previous_response_id error

Unknown event type error (connection stays open)

Invalid JSON error (connection stays open)

Multiple sequential requests on same connection

Multiple concurrent connections

Warmup → real request continuation

Response usage/token counts

In addition, there are unit tests (test_websocket.py) and integration tests with mock serving (test_websocket_integration.py).

@rasonyang Thanks~

I have another question: when using --api-server-count=2, does WebSocket work properly?

Good question. WebSocket works with --api-server-count=2, but with two caveats:

--max-websocket-connections is per-process, not global. With 2 servers and limit=100, the actual max is 200.

Connection-local state (previous_response_id cache) is per-process. This is fine in practice — a single WebSocket connection is a persistent TCP connection pinned to one process for its entire lifetime, so the cache stays consistent. The state only matters within the same connection, and connections don't migrate between processes.

So it works correctly — the only nuance is the per-process connection limit. I can add a note in the docstring if you'd like.

mergify · 2026-02-27T08:00:07Z

Hi @rasonyang, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

mergify · 2026-02-27T08:22:53Z

Hi @rasonyang, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

chaunceyjiang · 2026-02-27T13:39:05Z

+                json.dumps(
+                    {
+                        "type": "error",
+                        "status": 429,


Could we use a constant instead of a magic number?

Good catch — replaced 429 with HTTPStatus.TOO_MANY_REQUESTS. Fixed in the latest push.

mergify · 2026-02-28T08:41:55Z

Hi @rasonyang, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

mergify · 2026-03-03T07:20:17Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @rasonyang.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

github-actions · 2026-08-22T02:02:11Z

This pull request has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this pull request should remain open. Thank you!

mergify · 2026-08-26T02:04:59Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @rasonyang.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Add WebSocket transport for the OpenAI Responses API, enabling persistent bidirectional connections for lower-latency streaming. Changes: - Add `--max-websocket-connections` CLI argument (default: 100) - Initialize WebSocket server state (semaphore, active connections counter) in `init_app_state()` - Add WebSocket route `/v1/responses/websocket` with connection limit enforcement in the responses API router - New `websocket.py` module with full connection lifecycle: ConnectionContext, uuid7-based event IDs, JSON message parsing, streaming response events, warmup/error handling, and graceful shutdown Tests: - Unit tests (test_websocket.py): ConnectionContext, uuid7, message parsing, event formatting — no GPU/torch required - Mock integration tests (test_websocket_integration.py): full connection lifecycle using Starlette TestClient with mocked engine - Real e2e tests (test_websocket_real.py): live CPU server with facebook/opt-125m, covering streaming, multi-turn, cancellation, and concurrent connections Signed-off-by: Rason Yang <rason.yang@gmail.com>

Add real GPU integration tests for the WebSocket mode of the Responses API. These tests run against a live vLLM server with GPU inference using the Qwen/Qwen2-0.5B-Instruct model. The test suite covers: - Basic streaming - Event ordering - Text consistency - Warmup requests - Response continuation with previous_response_id - Error handling (invalid JSON, unknown events) - Sequential and concurrent requests - Usage statistics This is the GPU counterpart to test_websocket_real.py (CPU version). Signed-off-by: Rason Yang <rason.yang@gmail.com>

…GPU tests - Update shell script and docstring references from facebook/opt-125m to Qwen/Qwen2-0.5B-Instruct to match the actual MODEL_NAME constant - Rename _ws to ws since the variable is actively used Signed-off-by: Rason Yang <rason.yang@gmail.com>

…ff E501 Signed-off-by: Rason Yang <rason.yang@gmail.com>

Upstream moved `load_aware_call`, `validate_json_request` and `with_cancellation` from `vllm.entrypoints.utils` / `vllm.entrypoints.openai.utils` into `vllm.entrypoints.serve.utils.api_utils`. The torch-free import stubs in the WebSocket tests still registered the old module paths, so importing `responses/api_router.py` no longer resolved through them. Point the stubs at the new module. Signed-off-by: rason yang <rason.yang@gmail.com>

…lient Raw exception text was forwarded to remote WebSocket clients from both `processing_error` paths, which can leak server file paths, memory addresses and tracebacks. Route both through `sanitize_message()`, matching the equivalent handling in `speech_to_text/realtime/connection.py`. The unsanitized detail is still recorded server-side via `logger.exception`. Errors arriving as an `ErrorResponse` from the serving layer are already sanitized by `create_error_response`, so they are unchanged. Importing `sanitize_message` executes `vllm.entrypoints.serve.__init__`, which pulls `ErrorInfo` and `GenerationError` from the engine protocol module, so extend the torch-free import stubs in the tests accordingly. Add a regression test asserting paths and memory addresses are stripped while the underlying cause stays visible. Signed-off-by: rason yang <rason.yang@gmail.com>

@chaunceyjiang

…rocess The limit is enforced independently in each API server process, so with --api-server-count=N the effective total is N times the configured value. Raised by @chaunceyjiang in review. Signed-off-by: rason yang <rason.yang@gmail.com>

…orts Upstream vllm-project#53659 moved cli_args.py from vllm/entrypoints/openai/ to vllm/entrypoints/launchers/, which broke the AST check that reads the file by path. Also install the module stubs at import time. websocket.py imports sanitize_message, which executes vllm/entrypoints/serve/__init__.py and pulls in torch, so the tests above only ran on torch-less machines by accident of an earlier failed import leaving sys.modules populated. Guarded on torch being absent, so CI behaviour is unchanged. Signed-off-by: rason yang <rason.yang@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>

rasonyang · 2026-08-26T06:46:52Z

Rebased onto current main — merge conflicts are cleared and the PR is MERGEABLE again (head d8ce8bd45).

Upstream #53659 moved cli_args.py from vllm/entrypoints/openai/ to vllm/entrypoints/launchers/, which is what caused the conflict. --max-websocket-connections now lands in the new location, and a test that read the old path directly was updated to match.

Addressed since the last review pass

Unsanitized error messages (both findings from @depthfirst-app): the two processing_error paths in websocket.py now route through sanitize_message() before anything reaches the client, matching how speech_to_text/realtime/connection.py already handles this. Added a regression test asserting that server file paths and memory addresses are scrubbed. Both threads are resolved.
Per-process connection limit (@chaunceyjiang): the --max-websocket-connections docstring now states that the limit is enforced per API server process, so with --api-server-count=N the effective total is N × the configured value.
Test stubs: updated to follow upstream's relocation of load_aware_call / validate_json_request / with_cancellation into vllm.entrypoints.serve.utils.api_utils.

The earlier questions on OpenAI WebSocket-mode spec compatibility and --api-server-count=2 behaviour were answered inline on the original threads.

Testing

ruff check / ruff format --check (v0.14.0) clean on all changed files. The 18 unit tests and 4 mock integration tests pass on the rebased tree. The live-server and GPU suites have not been re-run since the rebase — those results in the description are from the pre-rebase run.

CI

pre-run-check fails with:

PR must have the 'verified', 'ready', or 'ready-run-all-tests' label to run pre-commit, or the author must have at least 4 merged PRs (found 0).

That skips pre-commit and gates Buildkite off, so this PR has never had a CI run — the failing check is the label policy, not the code. Could a maintainer add ready? The workflow triggers on labeled, so it will pick up the current head without a new push.

I have reviewed every changed line in this PR. AI assistance was used, as noted in the description.

rasonyang · 2026-09-17T08:05:16Z

@chaunceyjiang could you add the ready label? Only pre-run-check is failing, it needs verified/ready/ready-run-all-tests or 4 merged PRs. Code checks and local pre-commit pass, no conflicts. Thanks.

franciscojavierarceo · 2026-09-18T17:09:07Z

we should coordinate this with Agentic API before adding another WebSocket continuation implementation. Agentic API already handles Responses WebSockets, connection-local state, prewarming, persistence, and tool orchestration. its upstream path uses stateless /v1/responses over HTTP/SSE, so it needs Responses inference in the Rust frontend, but doesn't require upstream WebSockets.

the non-stateful Responses scope in #44280 and the work in #53380 look like the right backend boundary for that integration. standalone vllm serve users may still benefit from WebSockets here; we should agree on that deployment use case and how the Python frontend, Rust frontend, and Agentic API will share responsibility for compatibility and maintenance.

there are also a few correctness issues in d8ce8bd45 that we should address:

continuation drops the prior context. previous_response_id is validated and then removed, but the cached input/output is never passed into the serving pipeline. the next inference receives only the new input, breaking follow-up questions, function-call continuations, and generate: false prewarming. we should restore the complete prior item history and add tests that assert the context reaching inference, rather than just successful completion.
the socket reader waits for the entire generation, so disconnects go unread while inference is blocked. the lifetime timer also closes the socket without cancelling active generation. we should keep receiving independently and cancel/join active work on disconnect or expiry.
event serialization omits by_alias=True. structured-output events emit text.format.schema_ instead of schema. we should match the existing SSE serialization.
request validation falls into the generic exception handler. a request missing input returns a 500 and clears the previous cached response even when it doesn't reference that response. we should return a client validation error and avoid evicting unrelated continuation state.

rasonyang · 2026-09-19T01:13:12Z

Thanks @franciscojavierarceo. I verified all four issues against d8ce8bd45; they're valid, and #1 means continuation is currently broken.

Agreed on coordinating with Agentic API. Since it only needs stateless /v1/responses from vLLM, I'll move this PR to draft until the boundary in #44280 / #53380 settles, then we can revisit whether standalone vllm serve needs WebSockets. If it does, I'll fix the four issues (with tests asserting the context reaching inference) before re-requesting review.

Is there a thread tracking the Python / Rust frontend / Agentic API split?

franciscojavierarceo · 2026-09-20T05:16:08Z

I raised a discussion in the sig-frontend slack channel here

rasonyang requested review from DarkLight1337, NickLucche, aarnphm, chaunceyjiang, robertgshaw2-redhat and russellb as code owners February 27, 2026 07:42

mergify Bot added the frontend label Feb 27, 2026

gemini-code-assist Bot reviewed Feb 27, 2026

View reviewed changes

rasonyang force-pushed the feat/responses-websocket-mode-clean branch 3 times, most recently from 437831b to 2128b46 Compare February 27, 2026 07:53

chaunceyjiang reviewed Feb 27, 2026

View reviewed changes

rasonyang force-pushed the feat/responses-websocket-mode-clean branch 4 times, most recently from b938069 to 399a6e4 Compare February 27, 2026 08:15

rasonyang force-pushed the feat/responses-websocket-mode-clean branch from fc8757d to a058383 Compare February 27, 2026 08:50

chaunceyjiang reviewed Feb 27, 2026

View reviewed changes

rasonyang force-pushed the feat/responses-websocket-mode-clean branch from a058383 to e6d4125 Compare February 28, 2026 02:12

rasonyang force-pushed the feat/responses-websocket-mode-clean branch from 65c798d to 4ea86ae Compare February 28, 2026 08:45

mergify Bot added the needs-rebase label Mar 3, 2026

rasonyang force-pushed the feat/responses-websocket-mode-clean branch 2 times, most recently from b769df5 to 12e1802 Compare March 3, 2026 14:18

mergify Bot removed the needs-rebase label Mar 3, 2026

rasonyang force-pushed the feat/responses-websocket-mode-clean branch from 12e1802 to a272270 Compare March 3, 2026 14:22

github-actions Bot added the stale Over 90 days of inactivity label Aug 22, 2026

rasonyang force-pushed the feat/responses-websocket-mode-clean branch from d3ee956 to 7a96299 Compare August 22, 2026 02:26

rasonyang requested a review from AndreasKaratzas as a code owner August 22, 2026 02:26

mergify Bot removed the needs-rebase label Aug 22, 2026

depthfirst-app Bot reviewed Aug 22, 2026

View reviewed changes

Comment thread vllm/entrypoints/openai/responses/websocket.py Resolved

Comment thread vllm/entrypoints/openai/responses/websocket.py Resolved

rasonyang force-pushed the feat/responses-websocket-mode-clean branch from 70dd9c1 to 969c2a8 Compare August 22, 2026 03:20

github-actions Bot added unstale Recieved activity after being labelled stale and removed stale Over 90 days of inactivity labels Aug 26, 2026

mergify Bot added the needs-rebase label Aug 26, 2026

rasonyang and others added 8 commits August 26, 2026 14:21

fix(ci): wrap long line in test_websocket_gpu.py docstring to pass ru…

b29d3ed

…ff E501 Signed-off-by: Rason Yang <rason.yang@gmail.com>

rasonyang force-pushed the feat/responses-websocket-mode-clean branch from caec653 to d8ce8bd Compare August 26, 2026 06:34

mergify Bot removed the needs-rebase label Aug 26, 2026

jperezdealgaba mentioned this pull request Sep 13, 2026

[Security] Strip uppercase hex addresses from client error messages #56728

Closed

franciscojavierarceo mentioned this pull request Sep 18, 2026

[Frontend] Add Responses API compaction endpoint #56970

Draft

4 tasks

rasonyang marked this pull request as draft September 19, 2026 01:13

leseb mentioned this pull request Nov 12, 2026

Spike: Application-managed WebSocket sessions praxis-proxy/praxis#894

Open

6 tasks

		@@ -137,5 +137,55 @@ async def cancel_responses(response_id: str, raw_request: Request):
		return JSONResponse(content=response.model_dump())


		@router.websocket("/v1/responses")

Spec requirement	Implementation
Endpoint `ws(s)://host/v1/responses`	`@router.websocket("/v1/responses")` in `api_router.py`
Client sends `{"type": "response.create", ...}`	`handle_event()` dispatches on `type == "response.create"`
Server streams same events as SSE mode	`_process_response_create()` forwards the existing streaming pipeline events over WebSocket frames
`stream` / `background` fields omitted from WS payload	Stripped in `_process_response_create()` (`payload["stream"] = True`, `payload.pop("background", None)`)
One in-flight request per connection	`ctx.inflight` guard with `concurrent_request` error (409)
`previous_response_id` for continuation via connection-local cache	`ConnectionContext.last_response_id/last_response` with `previous_response_not_found` error
`generate: false` warmup mode	Returns `response.created` + `response.completed` with no inference
60-minute connection lifetime	`_enforce_deadline()` background task (warning at 55 min, close at 60 min)
Error format `{type: "error", status, error: {code, message}}`	`send_error()` method

Conversation

rasonyang commented Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Files Changed

Notes

Test Plan

Model evaluation

Duplicate-work check

AI assistance

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Feb 27, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Feb 27, 2026

Uh oh!

mergify Bot commented Feb 27, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Feb 28, 2026

Uh oh!

mergify Bot commented Mar 3, 2026

Uh oh!

github-actions Bot commented Aug 22, 2026

Uh oh!

Uh oh!

Uh oh!

mergify Bot commented Aug 26, 2026

Uh oh!

rasonyang commented Aug 26, 2026

Addressed since the last review pass

Testing

CI

Uh oh!

rasonyang commented Sep 17, 2026

Uh oh!

franciscojavierarceo commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rasonyang commented Sep 19, 2026

Uh oh!

franciscojavierarceo commented Sep 20, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rasonyang commented Feb 27, 2026 •

edited

Loading

franciscojavierarceo commented Sep 18, 2026 •

edited

Loading