Skip to content

feat(sglang): add token-exact SGLang model server for RL training - #1787

Draft
Kh4L wants to merge 13 commits into
NVIDIA-NeMo:mainfrom
Kh4L:sglang-splice-fix
Draft

Kh4L wants to merge 13 commits into
NVIDIA-NeMo:mainfrom
Kh4L:sglang-splice-fix

Conversation

@Kh4L

@Kh4L Kh4L commented Jun 26, 2026 •

Copy link
Copy Markdown
Contributor

Adds a text-only SGLang model server for NeMo Gym rollouts. It calls the native /generate endpoint to preserve sampled token IDs and logprobs, then splices those exact IDs into later turns. Changed rendering inputs fail explicitly, and a full context window ends generation without truncating the prompt.

The adapter uses Gym's current public package imports and shared HTTP client. Its component lock inherits the repository's resolver policy and pins Transformers 5.17.0. The tokenizer and chat template must match training; serving can use its own environment because requests carry integer token IDs.

Validation on the current main merge and dependency fix:

  • 146 adapter/client, dependency-policy, and targeted CI-regression tests passed in the locked component environment, including real loopback HTTP tests.
  • All 277 managed component-lock checks passed.
  • pre-commit run --all-files and git diff --check passed; commits are signed off.

This remains a draft pending a representative model-backed smoke on the final head. Earlier GPU runs do not validate this refreshed dependency set. Single-controller and multimodal integration remain separate work.

Supports NVIDIA-NeMo/RL#3127. Episode truncation is tracked separately in NVIDIA-NeMo/RL#3063; no additional issue is needed for this existing integration.

@copy-pr-bot

copy-pr-bot Bot commented Jun 26, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@cwing-nvidia

Copy link
Copy Markdown
Contributor

@Kh4L can you also take a look at #1557

@Kh4L

Kh4L commented Jul 1, 2026

Copy link
Copy Markdown
Contributor Author

Pushed 261615e: added an over-context guard to the SGLang /generate path.

SGLang hard-rejects input >= context_length with a 500 (vLLM tolerates it). In a multi-turn rollout the spliced prompt can already fill the window; the 500 then corrupts the trajectory and trips the downstream contiguity assert, stalling the async step. The guard pre-checks the prompt against sglang_max_total_sequence_length and defensively catches the ClientResponseError, ending the turn cleanly with finish_reason="length" — mirroring the existing vLLM context-length handling.

The splice fix + this guard have now been exercised end-to-end by a 16-node R=32 SGLang async-GRPO run: 0 chat-500s, 15+ training steps with graceful trajectory truncation and no stall. Still draft pending a full convergence-vs-baseline comparison.

@Kh4L

Kh4L commented Jul 2, 2026

Copy link
Copy Markdown
Contributor Author

Convergence validation complete. The splice fix + over-context guard in this PR (as carried on the fork branch) have now been validated by a full-scale end-to-end run: Qwen3-30B-A3B-Thinking SWE-bench async-GRPO on SGLang, 16 nodes (8 train + 8 gen, 32 generation replicas), 63 training steps across four clean checkpoint+resume segments.

  • train/total_reward/mean rose 0.10 → 0.19 → 0.22 and then held ~0.22 for two independent segments — the model learns and saturates; no stalls.
  • Zero over-context 500s (the guard's pre-check + error catch both exercised).
  • Rare non-contiguous turns were handled gracefully throughout (handled on the NeMo-RL side in the fork; the assert-side resilience is a separate NeMo-RL change).

Remaining known caveat: this port of the fix onto refactored Gym main is still not GPU-tested in this exact form (the validated code is the fork's pre-refactor equivalent) — keeping the PR draft until that's run.

Kh4L added 8 commits July 23, 2026 14:56
Signed-off-by: Serge Panev <spanev@nvidia.com>
SGLang /generate hard-rejects input tokens >= context_length (HTTP 500),
unlike vLLM which tolerates it. In a multi-turn SWE rollout the spliced
prompt can already fill the window; the 500 then corrupts the trajectory
and trips the downstream contiguity assert, stalling the whole async step.

Mirror the vLLM context-length handling: pre-check the prompt length
against sglang_max_total_sequence_length, and defensively catch the
ClientResponseError, ending the turn cleanly with finish_reason="length"
(empty completion) in both cases. Validated by a 16-node R=32 SGLang
GRPO run (0 chat-500s; 15+ steps, graceful truncation, no stall).

Signed-off-by: Serge Panev <spanev@nvidia.com>
The SGLang engine path re-parses tool calls from raw generated text
client-side. The existing parser only handled the hermes JSON format
(Qwen3-thinking). Add a dependency-free parser for the qwen3_coder
XML-ish format (<tool_call><function=NAME><parameter=KEY>...) used by
e.g. Qwen3-Coder-style chat templates, with schema-aware argument
type coercion from the request's tools, selectable via a new
sglang_tool_format config field (default "hermes" = no behavior
change). Unit tests mirror the template's canonical example.

Signed-off-by: Serge Panev <spanev@nvidia.com>
Chat templates iterate assistant tool-call arguments as a mapping
(tool_call.arguments|items), but OpenAI-format history carries
function.arguments as a JSON string - exactly what our /generate-path
parsers emit. When a request misses the token-splice cache and falls
back to a full template render of a tool-call-bearing conversation,
jinja raises TypeError('Can only get item pairs from a mapping.') and
the request 500s.

Normalize on the full-render path only: decode string arguments that
parse to a JSON object, leave everything else (and the splice path,
which never re-renders old assistant turns) untouched. Inputs are not
mutated - the splice session cache holds the original messages.

Signed-off-by: Serge Panev <spanev@nvidia.com>
Signed-off-by: Serge Panev <spanev@nvidia.com>
Mechanical ruff check --fix (I001 import order) + ruff format on the SGLang
proxy files; no logic changes. Fixes the failing Lint check on the PR.

Signed-off-by: Serge Panev <spanev@nvidia.com>
Signed-off-by: Serge Panev <spanev@nvidia.com>
Signed-off-by: Serge Panev <spanev@nvidia.com>
@Kh4L
Kh4L force-pushed the sglang-splice-fix branch from c223c16 to 65ddaa2 Compare July 23, 2026 22:27
@copy-pr-bot

copy-pr-bot Bot commented Jul 23, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@Kh4L Kh4L changed the title fix(sglang): keep multi-turn prompts prefix-stable via token-splicing feat(sglang): add token-exact SGLang model server for RL training Jul 23, 2026
The adapter renders the chat template and tokenizes locally before calling the
SGLang server, so its transformers version has to agree with the server's. It
was pinned to 5.6.0 to match the NeMo-RL sglang extra; that extra moves to
5.8.1 alongside sglang 0.5.13.post1, so this pin follows.

A mismatch here does not fail loudly at import. It surfaces as a rollout
contiguity failure when the locally rendered prefix stops matching what the
server tokenized, which is far harder to attribute, hence the comment on the
pin.

Signed-off-by: Serge Panev <spanev@nvidia.com>
@svcnvidia-nemo-ci svcnvidia-nemo-ci added area:model Model servers, inference providers, and model adapters feature New capabilities, enhancements, or enablement work labels Oct 20, 2026
Signed-off-by: Serge Panev <spanev@nvidia.com>
@Kh4L

Kh4L commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 3122202

Signed-off-by: Serge Panev <spanev@nvidia.com>
@Kh4L

Kh4L commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 9537b32

Signed-off-by: Serge Panev <spanev@nvidia.com>
@Kh4L

Kh4L commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test caa72c5

Signed-off-by: Serge Panev <spanev@nvidia.com>
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the risk:high Autobot PR risk: high label Nov 19, 2026
@Kh4L

Kh4L commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 32db78d

This branch was previously deployed

1 inactive (outdated) deployment
public — caa72c5f Deployed Nov 10, 2026 by copy-pr-bot[bot] via release / finalize / notify #11259
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:model Model servers, inference providers, and model adapters feature New capabilities, enhancements, or enablement work risk:high Autobot PR risk: high

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants