Skip to content

draft: one shared plan/run pipeline for Gym chat calls (sketch for #4979) - #13

Draft
terrykong wants to merge 1 commit into
jthomson04:jthomson04/dynamo-gymfrom
terrykong:terryk/pr4979-chat-call-sketch
Draft

terrykong wants to merge 1 commit into
jthomson04:jthomson04/dynamo-gymfrom
terrykong:terryk/pr4979-chat-call-sketch

Conversation

@terrykong

@terrykong terrykong commented Nov 11, 2026 •

Copy link
Copy Markdown

A sketch to copy from, not to merge. It goes with review comment C88 on NVIDIA-NeMo#4979.

It is one commit on 5c7fa69, the current head of jthomson04/dynamo-gym, so it includes the NVIDIA-NeMo#4506 cut-and-resume port and covers C88's AI-6.

It shares the steps of a Gym chat call after its render between the native vLLM server and the Dynamo path. All of it is in the new nemo_rl/models/generation/vllm/prompt_and_capture.py, plus one reply mixin in chat_server.py:

  • prepare_prompt → PreparedPrompt: TQ prefix read, splice and clamp. It runs where the render runs: the worker natively, the frontend behind Dynamo. Dynamo sends it to the engine with PreparedPrompt.to_json() / from_json(), inside a ForwardedCall.
  • start_capture: always on the worker, in one fixed order: resolve a restore from a checkpoint cut → swap in the saved prompt → check for a finished prefix → set the budget (unless finished) → snapshot media (skipped for a restore) → remap placeholders → begin_call with the cut → serve a finished prefix. Native preprocess_chat and the Dynamo adapter both call it; neither keeps its own copy (AI-6).
  • save_capture: the one TQ write, on both paths. Dynamo builds the staged message off the event loop.
  • error_reply / abort_capture: one error → status, abort reason and acknowledgement map, used by the native route, the adapter and the frontend.
  • ResumedCallReplyMixin: the reply fixes for a resumed call (restored text, parser, usage), used by the native serving chat and the render pool's FrontendServingChat (AI-6).
  • on_policy_sampling also refuses n != 1 and logprobs without top_logprobs, on both paths.

Deleted: the frontend's _splice, both native clamp branches, _clamp_max_tokens, _enter_request_prefix, the adapter's own restore → begin_call copy, the render pool's own restore calls, and PrefixRecovery.restore_reply_text (no caller left).

Tested (macOS): ruff check / ruff format (0.9.9) and py_compile on every changed file. 52 native chat-path tests that run against the repo's fake vLLM module tree (test_vllm_chat_template_wiring.py and the chat-route tests in test_vllm_generation.py): 52 pass, the same 52 that pass at 5c7fa69. test_vllm_token_capture_hosting.py with Gym on the path: 102 pass, 9 fail, 7 skip, the same as at 5c7fa69 (the 9 fail on this Mac's Python 3.12, which has no sys._is_gil_enabled).

Not tested: every Dynamo test, the parity test test_gym_chat_path.py with its 4 new failure-path cases, the render-pool tests, and test_prefix_restore_cases.py. There is no dynamo wheel and no real vLLM on macOS, so the Dynamo-side code and the shared reply mixin under real vLLM have never run.

🤖 Generated with Claude Code

terrykong added a commit to terrykong/gh-pages-poc that referenced this pull request Nov 11, 2026
Re-pin every link to 5c7fa69, add the restore step order and the reply
builder as rows the author now writes twice, put restore resolution in B,
and link the sketch as draft PR jthomson04/RL#13.

Signed-off-by: Terry Kong <terryk@nvidia.com>
…(sketch for NVIDIA-NeMo#4979)

A sketch for review comment C88 on NVIDIA-NeMo#4979, not for merge. Built on
5c7fa69. Both serving paths now share the steps of a Gym chat call
after its render, in nemo_rl/models/generation/vllm/prompt_and_capture.py:

- prepare_prompt -> PreparedPrompt: TQ prefix read, splice and clamp.
  Runs where the render runs (the worker natively, the frontend behind
  Dynamo). PreparedPrompt.to_json / from_json carry it to the engine.
- start_capture: on the worker, in one fixed order: resolve a restore
  from a checkpoint cut, swap in the saved prompt, check for a finished
  prefix, set the budget, snapshot media, remap placeholders, begin_call,
  and serve a finished prefix. Native preprocess_chat and the Dynamo
  adapter both call it; neither keeps its own copy.
- save_capture: the one TQ write, on both paths.
- error_reply / abort_capture: one error -> status, abort reason and
  acknowledgement map for the native route, the adapter and the frontend.
- ResumedCallReplyMixin: the reply fixes for a resumed call (text,
  parser, usage), used by the native serving chat and the render pool's.

on_policy_sampling also refuses n != 1 and logprobs without top_logprobs
on both paths. Dynamo checks the queue before capture begins and builds
the staged message off the event loop.

Signed-off-by: Terry Kong <terryk@nvidia.com>
@terrykong
terrykong force-pushed the terryk/pr4979-chat-call-sketch branch from 63916ed to bbca955 Compare November 11, 2026 08:08
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Nov 11, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant