Skip to content

fix(model): keepalive cleanup and conversion option checks - #5205

Merged
mpatel31415 merged 26 commits into
NVIDIA-NeMo:mainfrom
gwarmstrong:georgea/vllm-model-keepalive-followups
Nov 10, 2026
Merged

mpatel31415 merged 26 commits into
NVIDIA-NeMo:mainfrom
gwarmstrong:georgea/vllm-model-keepalive-followups

Conversation

@gwarmstrong

@gwarmstrong gwarmstrong commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

A client disconnecting from a keepalive stream can leave its backend request running or interrupt asynchronous teardown. The response now owns that request, cancels it, and waits for cleanup with protection from repeated AnyIO cancellation. Both Responses and Chat Completions are covered during the first interval, before the first chunk, and after keepalives.

Follow-ups to #5155 also:

  • Apply the configured text placeholder to URL-less tool images when inline_tool_output_images and drop_unrepresentable_request_fields are enabled. Strict conversion still rejects them; valid images still inline.
  • Reject stream_keepalive_s with is_responses_native, and inline_tool_output_images with use_completions_api, at configuration load.
  • Preserve the response ID recorded by capture on keepalive streams, including checkpoint-gated responses.

The branch includes main 50f093214498689812e997b15a63e5eb137df07a. The overlapping AgenticIF driver fixture now matches main's #5909 exactly, so this PR has no AgenticIF file changes.

The focused #5895 test follow-up makes the recovered-response test's intended first timeout deterministic after a confirmed real HTTP upload. Real aiohttp response-start classification, upload tracking, retry and typed reporting remain exercised. Both wire payloads and exactly two attempts are checked. A 30-second request/recovery watchdog bounds stuck I/O.

No separate issue for the model changes: these are review follow-ups to #5155.

Validation

Current source: 4386664ac8c9351588f1662a08c35a7bd1714a73. This refresh resolves the overlapping adaptive-driver change using main's complete file and preserves the disjoint vLLM routing, budget-refit, keepalive and conversion changes.

  • Four shared model/converter/streaming modules: 609 passed, including the real-disconnect middleware matrix.
  • Full vLLM test suite: 1,049 cases passed, using the current source with the installed component dependencies. The local source locations were checked explicitly to avoid mixing an older editable installation with current tests.
  • Recovered-response timeout regression: 1 passed, exercising real aiohttp upload, retry and typed reporting.
  • Test-quality review found no additional issue. Disconnect tests require completed asynchronous backend cleanup; the conversion/config/capture tests assert observable results. The resolved driver file keeps main's real fingerprint guard and no-dispatch/artifact-mutation checks.
  • pre-commit run --all-files passed with no changes, including guardwords and import-linter. Merge-commit hooks passed again, and the merge commit has DCO sign-off.
  • Final-head AgenticIF driver/contract modules: 53 passed (703.74 s) in a fresh isolated environment.
  • Hosted CI has been authorized for this full SHA and is rerunning; final hosted status and renewed reviewer approval determine merge readiness.

Rollouts

N/A: model-server fixes and test-fixture changes. Tests exercise real dispatch, streaming, disconnect middleware, capture, conversion and HTTP transport with controlled external backends. Full CLI harness conformance runs in Linux CI; local macOS conformance stops at the explicit Linux guard.

Compatibility and benchmark impact

The two unsupported configuration combinations now fail at configuration load. URL-less images follow the existing strict/drop policy. Keepalive disconnects wait for the owned backend's asynchronous teardown. Production benchmark fingerprint validation is unchanged.

Streaming /v1/responses requests are buffered: the converted response is
re-emitted as SSE only once responses() returns, so the client sees no
event while a long generation runs. A client that bounds the silence
between events abandons the turn and re-sends it. The Codex CLI does
this after stream_idle_timeout_ms (300 s by default), which a high
reasoning effort turn can easily outlast. SSE comments, which the chat
streaming path sends, do not help here: Codex's event parser drops them
before its idle timer sees them.

Add stream_keepalive_s to BaseResponsesAPIModelConfig (default null,
today's behavior). When set and a call is still running after that many
seconds, the stream opens with response.created and then carries
response.in_progress every interval for the same placeholder response
id. Once responses() returns, the item and terminal events follow
exactly as the buffered path sends them, minus the response.created the
keepalives already sent.

A call that finishes within the first interval streams exactly as
before, HTTP error statuses included. A failure after the stream opened
ends with response.failed for the announced response id, carrying what
the client would have seen before the stream opened: a refusal the
server propagates keeps its OpenAI code (context_length_exceeded), an
upstream HTTP error follows the server's propagation policy, and no
upstream request URL is reported.

Model-call capture measured time to first token from the first body
chunk, which with keepalives would be the opening response.created one
interval in. The Responses lifecycle events (response.created and
response.in_progress) carry no model output, so they no longer count
toward it; on the buffered path they are sent in the same instant as the
first item, so its measurement is unchanged.

Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
…text

The Codex CLI sends reasoning efforts its model catalog offers, and a
catalog may offer one the pinned OpenAI SDK does not list, such as
"max". Request validation types reasoning.effort with the SDK's literal,
so vllm_model answers such a request with 422 before any conversion,
although the engine behind the chat route accepts the level.

Add forward_unvalidated_reasoning_effort to VLLMModelConfig (default
false, today's behavior; requires is_responses_native=false). When set,
a string effort the SDK does not list is taken out of the raw Responses
body before validation and put verbatim on the converted chat request
as reasoning_effort; the engine validates it. Efforts the SDK lists,
"none" included, keep their existing path, and an empty effort is still
refused.

Codex also sends reasoning.context "all_turns" when its catalog enables
reasoning context, which drop_unrepresentable_request_fields still
refuses. The conversion passes every reasoning item the request carries
to the engine and leaves what is rendered to its chat template, which is
what "all_turns" (render them all) and "auto" (the provider's choice)
ask for. With drop_unrepresentable_request_fields those two values are
now dropped like the summary fields. "current_turn" asks for earlier
turns' reasoning to be left out, which the conversion does not do, so
it is still refused, and every value is still refused by default.

Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Two things the Codex CLI sends cannot reach a vLLM chat route through
the Responses to Chat Completions conversion today.

Codex's apply_patch is a Responses custom (freeform) tool with a lark
grammar. The conversion forwards it as a Chat Completions custom tool,
which a vLLM chat route rejects, so the only option was
drop_custom_tools: the model then has no apply_patch, tries it in the
shell, and falls back to editing with sed.

Add custom_tools_as_functions (VLLMModelConfig and ResponsesConverter,
default false). Each custom tool is presented to the engine as a
function tool with one required string parameter, input; its
description is the tool's description, a note to pass the raw input as
input, and the grammar when the tool has one, since a string parameter
cannot constrain the input. The engine's calls to that function come
back as custom_tool_call items (call_id, name, input, status) on the
non-streaming and the streaming path alike. Arguments that are not an
object with a string input are passed through as the input, so the
client's tool reports the bad call to the model. Replayed
custom_tool_call and custom_tool_call_output items become the chat tool
call (arguments {"input": ...}) and tool message, on the same assistant
turn as the replayed reasoning. A named tool_choice of a custom tool
selects its function. The option takes precedence over
drop_custom_tools, and a custom tool named like a function tool fails
the request. It needs is_responses_native=false and
return_token_id_information=false: custom_tool_call has no variant that
carries token IDs. Saved model-call records now list custom_tool_call
items among a response's tool calls.

When the model catalog lists image input, Codex offers view_image,
whose output is a function_call_output holding an input_image part.
Chat tool messages accept image parts since NVIDIA-NeMo#4688, but the conversion
still refuses the part, or replaces it with a text placeholder under
drop_unrepresentable_request_fields, so the model never sees the image.

Add inline_tool_output_images (VLLMModelConfig and ResponsesConverter,
default false; needs is_responses_native=false). Each image part of a
tool call output becomes an image_url part of the chat tool message,
which vLLM accepts and leaves to the chat template, and it takes
precedence over the placeholder. Detail "original", which Codex sends
when the model asks view_image for full resolution and which chat
lacks, goes as "high". Other part types a tool message cannot carry
are still refused.

Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
… cannot apply

The completions path builds its body from a fixed list of chat fields, and
reasoning_effort is not one of them, so with use_completions_api=true the
forwarded effort was dropped without a word. The compaction server has its
own dispatch, which never sets the unlisted effort aside. Both now refuse
the option at startup instead of ignoring it.

Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
NVIDIA-NeMo#4880 added custom_tool_strategy: function_bridge, which lowers each
Responses custom tool to a function tool with one string input and lifts
the calls back, on the same conversion this branch extended. Drop this
branch's custom_tools_as_functions, which did the same job: the config
field, the converter helpers and restore step, the name-clash check,
the streaming drop override and their tests.

The Codex tests now run apply_patch through function_bridge, streaming
and non-streaming, with drop_custom_tools off: it is still the kill
switch and would drop the tool before the bridge sees it.

Saved model-call records keep listing custom_tool_call items, which the
bridge returns too.

Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
… forwards

Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
…leaves early

Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
…dropping

Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>

# Conflicts:
#	nemo_gym/base_responses_api_model.py
#	nemo_gym/responses_converter.py
#	responses_api_models/vllm_model/app.py
#	responses_api_models/vllm_model/tests/test_app.py
#	tests/unit_tests/test_responses_api_model_streaming.py
@gwarmstrong
gwarmstrong requested a review from yaoyu-33 October 28, 2026 18:16
@copy-pr-bot

copy-pr-bot Bot commented Oct 28, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@gwarmstrong

Copy link
Copy Markdown
Contributor Author

/ok to test 34e7786

@nemo-automation-bot
nemo-automation-bot Bot requested review from a team October 28, 2026 18:17
@svcnvidia-nemo-ci svcnvidia-nemo-ci added area:model Model servers, inference providers, and model adapters bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer labels Oct 28, 2026
@mpatel31415

Copy link
Copy Markdown
Contributor

/review reset=false mode=strict prompt="Review exact dac0820: recovered-timeout fixture with real response-start, upload, retry and report paths; cleanup and 30-second watchdog. Check the validation evidence and macOS baseline limitations."

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP

LGTM — no concerns found.

This follow-up covers changes since d6e4f857. Whole-file digests confirm that the other seven PR files match that reviewed head. The latest supplied review had no unresolved findings.

The recovered-timeout fixture waits for the server to read the uploaded JSON before injecting the first timeout beneath the real response-start call. I traced the upload evidence, retry and failure-report paths. The assertions require both payloads, exactly two attempts, and one typed confirmed-upload observation without a terminal failure. The scoped protocol patch restores on unwinding, and the finally block releases the first handler before session/server teardown. The 30-second watchdog covers request/recovery I/O, not context-manager shutdown.

The neighboring body-read fixture changes match the supplied upstream base. The two connection-backlog cases are unchanged from the previous reviewed source; the reuse-port suite retains its explicit Linux-only guard.

Static source review only: I did not execute tests or verify hosted CI artifacts. The reported counts, controlled probes, macOS reproductions and final-head Linux results remain author-reported. I checked the CI aggregation rules; harness conformance is separately non-blocking, so the final aggregate alone doesn't establish its result. One delegated transport report wasn't delivered; I completed that source trace directly.

Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.

Preserve upstream runtime snapshot isolation from NVIDIA-NeMo#5088 and retarget the intentional revision-mismatch regression to that environment boundary. Keep the real resume guard and no-dispatch/artifact assertions.

Refs NVIDIA-NeMo#5893

Signed-off-by: Martyna Patelka <mpatelka@nvidia.com>
@mpatel31415

Copy link
Copy Markdown
Contributor

/ok to test 0001b65

@mpatel31415

Copy link
Copy Markdown
Contributor

/review reset=false mode=strict prompt="Review exact 0001b65: main refresh preserves upstream runtime snapshot; revision mismatch changes only gym_commit at that boundary while the resume guard stays real. Check vLLM merge integration and validation limits."

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP

LGTM — no concerns found.

This follow-up covers changes since reviewed dac0820b. The latest supplied review had no unresolved findings. I compared all eight listed files; five are byte-identical to that baseline.

  • The adaptive fixture retains upstream’s one-time runtime snapshot. The revision-mismatch case copies that snapshot and changes only gym_commit. I traced it through resume preflight to the real journal fingerprint guard, before generation or publication. The assertions retain successful same-runtime resume, unchanged policy/judge dispatch counts, and unchanged artifact bytes and capture/receipt digests.
  • The vLLM merge integration retains upstream’s shared session-key routing and cookie-session fallback. The routing assertions check endpoint selection, outbound-header agreement, rollout affinity and fallback behavior. The merge retains the existing conversion options, startup validation and keepalive capture-ID regression coverage.

Static source review only: I did not execute tests or verify hosted CI artifacts. The reported current-head counts and macOS baseline reproductions remain author-reported; historical Linux results do not establish a current-head pass. Harness conformance remains separately non-blocking, so a green final aggregate alone would not establish its result. One delegated adaptive-resume report wasn't delivered; I completed that source trace directly.

Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.

@AdamRajfer

AdamRajfer commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

🤖 Adam’s AI assistant

I reviewed this for Adam. The keepalive teardown, URL-less image handling and config-load checks look right and are well covered. The branch now conflicts with main in resources_servers/agentic_if_realworld/tests/test_driver.py: #5909 already covers that hunk (#5893), so please merge main keeping main's version of that file. I'll approve once CI is green on the new head.

Resolve the overlapping AgenticIF fixture change with the main version, where NVIDIA-NeMo#5909 covers the resume identity boundary. Preserve the keepalive, conversion and model option changes while incorporating current main.

Signed-off-by: Martyna Patelka <mpatelka@nvidia.com>
@AdamRajfer

Copy link
Copy Markdown
Contributor

/ok to test 4386664

@mpatel31415
mpatel31415 added this pull request to the merge queue Nov 10, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42899666435

@AdamRajfer AdamRajfer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Adam’s AI assistant

I reviewed this for Adam. The conflict resolution keeps test_driver.py identical to main, and the rest matches the head reviewed earlier. CI is green.

Merged via the queue into NVIDIA-NeMo:main with commit 5365700 Nov 10, 2026
68 checks passed

This branch was previously deployed

1 inactive deployment
public — 4386664a Deployed Nov 10, 2026 by copy-pr-bot[bot] via release / finalize / notify #11569
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:model Model servers, inference providers, and model adapters bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer sla:review-overdue Review response is over the one-business-day SLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants