Skip to content

[Bugfix] Harden DeepSeek V4 reasoning and DSML tool-call parsing  - #40833

Open
SilverNarcissus wants to merge 15 commits into
sgl-project:release/v0.5.20from
SilverNarcissus:fix/dsv4-dsml-v0.5.18
Open

SilverNarcissus wants to merge 15 commits into
sgl-project:release/v0.5.20from
SilverNarcissus:fix/dsv4-dsml-v0.5.18

Conversation

@SilverNarcissus

@SilverNarcissus SilverNarcissus commented Sep 23, 2026 •

Copy link
Copy Markdown

Motivation

DeepSeek V4 streaming parsing can emit unstable non-string arguments, confuse protocol boundaries with literal text,
or report completion before malformed tool arguments are detected.

This PR preserves append-only argument semantics and validates actual tool frames. It does not repair arbitrary
generation or forbid protocol-related strings in otherwise valid content.

Modifications

  • Rebased onto release/v0.5.20 (94602c9c2b7cbdb8efd5c52802dac6a1c180089e).
  • Delay non-string DSML parameters until their values are complete; reject incomplete EOF rather than guess values.
  • Validate DSML structure, duplicate parameters/JSON keys, non-finite JSON values, append-only prefixes, and the
    supplied tool schema by default. No coercion, missing-field insertion, or external schema retrieval.
  • Preserve quoted/code literals and normalize redundant/orphan closers only at confirmed reasoning/tool boundaries.
  • Remove the four experimental strict/schema/marker environment switches. Remove blanket marker-content rejection
    and its ordinary-text buffering, so legitimate protocol examples remain usable.
  • Keep raw pre-parser output tracing independently opt-in: both SGLANG_ENABLE_MODEL_OUTPUT_LOGGING=1 and
    SGLANG_MODEL_OUTPUT_TRACE_DIR are required. Output text is not redacted; request bodies and authentication headers
    are not copied directly into this trace.

Malformed or schema-invalid tool calls now fail explicitly instead of being forwarded as successful calls.
Ordinary text is not globally scanned for forbidden words. Middle-of-reasoning orphan markers remain outside the
confirmed-boundary cleanup scope.

Accuracy Tests

No kernel, model-forward, or weight-quantization implementation is changed by this patch.

python3 -m pytest -q \
  test/registered/unit/parser/test_reasoning_parser.py \
  test/registered/unit/function_call/test_deepseekv4_detector.py \
  test/registered/unit/test_model_output_trace.py

230 tests passed, including 3,953 subtests, on the v0.5.20 code and again inside the resulting image.
The final-image run copied test files only; runtime source hashes were verified unchanged.

Coverage includes chunk boundaries, EOF, malformed structure, mandatory schema checks, valid protocol literals,
reasoning boundaries, and disabled/enabled raw logging.

Live smoke validation on a four-GPU FP8 setup completed 5/6 fixed API probes successfully. A compound literal request
was rejected for incomplete DSML framing. Two additional, separately recorded diagnostics passed 1/2; one requested
literal string was generated with a different value. A separate opt-in trace run confirmed that the malformed frame
and changed string were already present in decoded output before the reasoning/tool parsers; replay reproduced the
observed parsing behavior.

These failures are retained rather than replaced with successful reruns. This is limited integration evidence, not
a model-accuracy claim. Earlier v0.5.18 task scores are not attributed to this revision.

Speed Tests and Profiling

No isolated TTFT, TPOT, throughput benchmark, or profiler run was performed. No speedup is claimed.
Validation and enabled tracing can add CPU/I/O work; tracing is disabled by default.

Checklist

Black formatting and code-level documentation are updated. Full pre-commit, public-facing documentation, and
isolated inference-performance validation remain pending.

Review and Merge Process

  1. Ping Merge Oncalls according to the PR Merge Process.
  2. Obtain CODEOWNERS and reviewer approvals.
  3. Run the required CI through authorized maintainers.
  4. Merge only after required approvals and green CI.

CI States

Latest PR Test (Base): ❌ Missing run-ci label -- add it to run CI tests.
Latest PR Test (Extra): ❌ Blocked -- run-ci is required first.
Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.

@github-actions github-actions Bot added dependencies Pull requests that update a dependency file deepseek npu labels Sep 23, 2026
xuekaifeng.silver and others added 15 commits September 23, 2026 12:04
Backport upstream PR sgl-project#36285 (c79efab) with its regression tests. Preserve the existing guards for forced reasoning and already-closed reasoning blocks.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52fd8e11-8b90-4cbc-a227-032e000bae8d
Preserve append-only argument deltas for numeric and structured values. Reject an incomplete non-string parameter at EOF and retain incremental string arguments. Include V3.2/V4 chunk-boundary regressions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52fd8e11-8b90-4cbc-a227-032e000bae8d
Keep malformed DSML wrappers out of ordinary preambles once an invoke is parsed. Preserve quoted examples and use one parser path for streaming and nonstreaming. Add opt-in strict validation and EOF checks, with boundary, literal and public-parser regressions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52fd8e11-8b90-4cbc-a227-032e000bae8d
Add opt-in decoded-output delta tracing with request/choice correlation and replacement events. Exclude request bodies, input IDs and credentials. Wire streaming and nonstreaming Chat paths without changing sampling or parser results; include privacy and reconstruction tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52fd8e11-8b90-4cbc-a227-032e000bae8d
Reject unquoted malformed parameter-close fragments in strict XML string arguments before dispatch. Preserve quoted, fenced, heredoc and escaped literals. Validate direct-JSON string values before releasing their arguments. Add minimal regressions for the observed boundary defect without committing benchmark captures.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52fd8e11-8b90-4cbc-a227-032e000bae8d
Gate V4 reasoning-to-tool boundary handling behind active tool parsing and strict output mode. Preserve post-reasoning prose and quoted examples, remove only standalone redundant closers immediately before a native tool block, and never reinterpret think literals inside tool payloads.

Share the existing literal masking helper with the V4 reasoning detector and cover streaming chunk boundaries, nonstreaming parity, continuation behavior, and end-to-end argument preservation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 52fd8e11-8b90-4cbc-a227-032e000bae8d
Recognize a redundant unquoted trailing think closer at an established native tool boundary even when it is attached to prose. Keep escaped markers, quoted examples, intervening prose, and all tool payload untouched; retain the existing strict active-tool gate.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 52fd8e11-8b90-4cbc-a227-032e000bae8d
In strict V4 mode reject unexpected parameter boundaries, unclosed or duplicate parameters, invalid type flags, ambiguous JSON keys and non-JSON constants. Validate emitted-prefix consistency and completed JSON objects before marking a tool call complete. Keep V3.2 and non-strict behavior unchanged through a no-op validation hook.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 52fd8e11-8b90-4cbc-a227-032e000bae8d
Validate completed strict V4 calls against their supplied tool schemas without coercion. Reuse schema type normalization, preserve explicit unknown-tool forwarding, and disable all external schema retrieval. The policy defaults off because rejection is not generation repair and may change recovery behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 52fd8e11-8b90-4cbc-a227-032e000bae8d
Reject literal think delimiters in parsed tool arguments when explicitly enabled, including quoted document strings and nested JSON values. Keep the policy off by default, preserve legitimate literals when disabled, and never rewrite commands or claim to sanitize arbitrary program outputs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 52fd8e11-8b90-4cbc-a227-032e000bae8d
Add an explicit strict-only, default-off lexical rejection policy covering normal content, tool names and parsed arguments. Preserve native partial parameter framing. The opt-in policy deliberately rejects quoted protocol examples instead of rewriting them; separate reasoning remains untouched.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 52fd8e11-8b90-4cbc-a227-032e000bae8d
…daries

Gate cleanup on strict active-tool parsing and the explicit protocol-marker policy. Preserve quoted and paired markup, incomplete EOF, interrupted reasoning, and tool payloads.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52fd8e11-8b90-4cbc-a227-032e000bae8d
Keep raw output collection disabled even when a trace directory is configured. Preserve all parsing and validation behavior; document the independent logging gate and cover disabled/enabled configurations.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52fd8e11-8b90-4cbc-a227-032e000bae8d
Keep executable code, protocol markers, and test data unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52fd8e11-8b90-4cbc-a227-032e000bae8d
Remove experimental strict/schema/marker switches. Enforce structural, JSON, schema, and append-only checks by default; retain confirmed reasoning-boundary normalization. Drop blanket marker rejection and its ordinary-text holdback so valid protocol examples remain usable. Raw output logging stays explicitly opt-in.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52fd8e11-8b90-4cbc-a227-032e000bae8d

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

amd apple-silicon blackwell SM100/SM120 deepseek dependencies Pull requests that update a dependency file diffusion SGLang Diffusion documentation Improvements or additions to documentation hicache Hierarchical Caching for SGLang jit-kernel lora mthreads Multi-modal multi-modal language model npu quant LLM Quantization sgl-kernel speculative-decoding

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant