Repository navigation
Conversation
terrykong
added a commit
to terrykong/gh-pages-poc
that referenced
this pull request
Nov 11, 2026
Re-pin every link to 5c7fa69, add the restore step order and the reply builder as rows the author now writes twice, put restore resolution in B, and link the sketch as draft PR jthomson04/RL#13. Signed-off-by: Terry Kong <terryk@nvidia.com>
…(sketch for NVIDIA-NeMo#4979) A sketch for review comment C88 on NVIDIA-NeMo#4979, not for merge. Built on 5c7fa69. Both serving paths now share the steps of a Gym chat call after its render, in nemo_rl/models/generation/vllm/prompt_and_capture.py: - prepare_prompt -> PreparedPrompt: TQ prefix read, splice and clamp. Runs where the render runs (the worker natively, the frontend behind Dynamo). PreparedPrompt.to_json / from_json carry it to the engine. - start_capture: on the worker, in one fixed order: resolve a restore from a checkpoint cut, swap in the saved prompt, check for a finished prefix, set the budget, snapshot media, remap placeholders, begin_call, and serve a finished prefix. Native preprocess_chat and the Dynamo adapter both call it; neither keeps its own copy. - save_capture: the one TQ write, on both paths. - error_reply / abort_capture: one error -> status, abort reason and acknowledgement map for the native route, the adapter and the frontend. - ResumedCallReplyMixin: the reply fixes for a resumed call (text, parser, usage), used by the native serving chat and the render pool's. on_policy_sampling also refuses n != 1 and logprobs without top_logprobs on both paths. Dynamo checks the queue before capture begins and builds the staged message off the event loop. Signed-off-by: Terry Kong <terryk@nvidia.com>
terrykong
force-pushed
the
terryk/pr4979-chat-call-sketch
branch
from
November 11, 2026 08:08
63916ed to
bbca955
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A sketch to copy from, not to merge. It goes with review comment C88 on NVIDIA-NeMo#4979.
It is one commit on 5c7fa69, the current head of
jthomson04/dynamo-gym, so it includes the NVIDIA-NeMo#4506 cut-and-resume port and covers C88's AI-6.It shares the steps of a Gym chat call after its render between the native vLLM server and the Dynamo path. All of it is in the new
nemo_rl/models/generation/vllm/prompt_and_capture.py, plus one reply mixin inchat_server.py:prepare_prompt→PreparedPrompt: TQ prefix read, splice and clamp. It runs where the render runs: the worker natively, the frontend behind Dynamo. Dynamo sends it to the engine withPreparedPrompt.to_json()/from_json(), inside aForwardedCall.start_capture: always on the worker, in one fixed order: resolve a restore from a checkpoint cut → swap in the saved prompt → check for a finished prefix → set the budget (unless finished) → snapshot media (skipped for a restore) → remap placeholders →begin_callwith the cut → serve a finished prefix. Nativepreprocess_chatand the Dynamo adapter both call it; neither keeps its own copy (AI-6).save_capture: the one TQ write, on both paths. Dynamo builds the staged message off the event loop.error_reply/abort_capture: one error → status, abort reason and acknowledgement map, used by the native route, the adapter and the frontend.ResumedCallReplyMixin: the reply fixes for a resumed call (restored text, parser, usage), used by the native serving chat and the render pool'sFrontendServingChat(AI-6).on_policy_samplingalso refusesn != 1andlogprobswithouttop_logprobs, on both paths.Deleted: the frontend's
_splice, both native clamp branches,_clamp_max_tokens,_enter_request_prefix, the adapter's own restore →begin_callcopy, the render pool's own restore calls, andPrefixRecovery.restore_reply_text(no caller left).Tested (macOS):
ruff check/ruff format(0.9.9) andpy_compileon every changed file. 52 native chat-path tests that run against the repo's fake vLLM module tree (test_vllm_chat_template_wiring.pyand the chat-route tests intest_vllm_generation.py): 52 pass, the same 52 that pass at 5c7fa69.test_vllm_token_capture_hosting.pywith Gym on the path: 102 pass, 9 fail, 7 skip, the same as at 5c7fa69 (the 9 fail on this Mac's Python 3.12, which has nosys._is_gil_enabled).Not tested: every Dynamo test, the parity test
test_gym_chat_path.pywith its 4 new failure-path cases, the render-pool tests, andtest_prefix_restore_cases.py. There is nodynamowheel and no real vLLM on macOS, so the Dynamo-side code and the shared reply mixin under real vLLM have never run.🤖 Generated with Claude Code