Skip to content

fix(frontend): keep buffering tool text while a later tool call is still open - #15829

Merged
rmccorm4 merged 7 commits into
ai-dynamo:mainfrom
mifegui:fix/tool-buffer-parallel-calls
Oct 16, 2026
Merged

rmccorm4 merged 7 commits into
ai-dynamo:mainfrom
mifegui:fix/tool-buffer-parallel-calls

Conversation

@mifegui

@mifegui mifegui commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Summary

When reasoning ends in the same chunk as a tool call starts, the vLLM chat processor buffers the tool text and parsed it as soon as the buffer held an end marker, handing the rest of the response to the streaming parser. With multi-token steps (e.g. MTP) the chunk that closes one call can also open the next: that call was dropped and streamed as content, and when the split fell between calls, the streaming parser numbered the next call from 0 again, so clients merged two calls into one.

The buffered text is now parsed once, when the response finishes, so all calls come from one parse with sequential indices. Normal streaming (no reasoning/tool collision) is unchanged.

Fixes #15828

Validation

  • New test_parallel_tool_calls_survive_multi_token_steps and test_parallel_tool_calls_survive_split_start_marker: stream two GLM tool calls after </think> in multi-token steps and with a split start marker, rebuilding calls by index as a client does. They fail on main and pass here.
  • pytest components/src/dynamo/frontend/tests/test_vllm_processor_unit.py: 176 passed
  • Also checked with the GLM-4.7 tokenizer: two and three calls, a single call, and calls without reasoning
  • pre-commit run --files on both files
  • GPU, same request under load with MTP 3 and streaming: 17/40 → 0/40 leaks. Plain text, reasoning, non-streaming and image requests unchanged.

A call cut off by max_tokens comes back the way a non-streaming request returns it, with the arguments parsed so far and finish_reason: length; dropping unfinished calls on length is left for a follow-up.

When reasoning ends in the same chunk as a tool call starts, the vLLM
chat processor buffers the tool text and parses it on the first end
marker. With multi-token steps (e.g. MTP) that chunk can also open the
next call, which was dropped, and the rest of that call then streamed
as content. Close the buffer only when no call opened after the last
end marker.

Signed-off-by: Eric Guimarães <ericg@telnyx.com>
@mifegui
mifegui requested a review from a team as a code owner October 7, 2026 19:57
@copy-pr-bot

copy-pr-bot Bot commented Oct 7, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@mifegui
mifegui temporarily deployed to external_collaborator October 7, 2026 19:57 — with GitHub Actions Inactive
@mifegui
mifegui temporarily deployed to external_collaborator October 7, 2026 19:57 — with GitHub Actions Inactive
@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

👋 Hi mifegui! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

@github-actions github-actions Bot added fix external-contribution Pull request is from an external contributor frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` labels Oct 7, 2026
@coderabbitai

coderabbitai Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: ai-dynamo/dynamo/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: a1c7234d-2882-4c2b-9ced-540c54eed40c
📥 Commits

Reviewing files that changed from the base of the PR and between 202d552 and 04571a0.

📒 Files selected for processing (2)
  • components/src/dynamo/frontend/prepost.py
  • components/src/dynamo/frontend/tests/test_vllm_processor_unit.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.


Walkthrough

The streaming processor now completes buffered tool-call parsing only when the final tool call is closed, unless a finish reason completes it first. A regression test checks two consecutive calls across token batch sizes 1–6.

Changes

Tool-call streaming

Layer / File(s) Summary
Buffered tool-call parsing
components/src/dynamo/frontend/prepost.py, components/src/dynamo/frontend/tests/test_vllm_processor_unit.py
The processor keeps the buffer open when a tool-start marker follows the last end marker. The regression test checks that two calls retain their names and arguments, with no tool-call text emitted as content.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Severity of issue fixed: Medium

Merge Risk: ⚪ Minimal · up to 04571

No actionable merge-blocking risk is established for this change; it is mergeable after normal checks.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 4 functions across 2 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Issue [#15828] requires the vLLM streaming processor to preserve two parallel tool calls when one chunk closes the first call and opens the second, with no tool-call text in content. The change adds…
Out of Scope Changes check ✅ Passed The pull request changes only StreamingPostProcessor buffer-completion logic and adds a focused regression test for issue [#15828]. Both changes directly support the linked issue. No unrelated sourc…
Title check ✅ Passed The title clearly describes the primary fix: keeping tool-call text buffered while a later tool call remains open.
Description check ✅ Passed The description clearly explains the problem, solution, validation, test results, and linked issue. It does not use the template headings exactly and does not include a separate reviewer-start section…
  • Fix all pre-merge checks with AI
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

Address review: a start marker split across steps left an incomplete
prefix after the last end marker. Drop the step sizes that never
reached the buffer path.

Signed-off-by: Eric Guimarães <ericg@telnyx.com>
@pull-request-size pull-request-size Bot added size/L and removed size/M labels Oct 7, 2026
@mifegui
mifegui temporarily deployed to external_collaborator October 7, 2026 21:04 — with GitHub Actions Inactive
Closing the buffer mid-response handed later calls to the streaming
parser, which numbered them from 0 again, so two calls could share an
index. Parse the whole buffer when the response finishes instead; this
also removes the marker scan. The test now rebuilds calls by index.

Signed-off-by: Eric Guimarães <ericg@telnyx.com>

@dynamo-review-agent dynamo-review-agent Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previously reported defects still present:

  • Original discussion: Still present: _assert_two_glm_calls_streamed aggregates emitted tool calls by append order and ignores tool_call.index, so cases where both calls are emitted with index 0 can still pass despite clients merging them into one malformed call.
  • Original discussion: The helper still reconstructs streamed tool calls by appending each named function to positional lists and never uses tool_call.index, so emitted deltas that reuse index 0 for both calls can still satisfy the new assertions even though clients would merge them into one malformed call.
  • Original discussion: The new test helper still reconstructs streamed tool calls by appending arguments to the most recent name instead of grouping deltas by tool_call.index, so cases where both emitted calls use index 0 continue to pass while clients would merge them into one malformed call.
  • Original discussion: The test helper still reconstructs streamed tool calls by appending a new argument buffer whenever a delta carries a function name, instead of grouping deltas by tool_call.index. As previously reported, some step/offset cases can emit both calls at index 0 while this helper still passes, so the new test can miss the client-visible merge into one malformed call.
  • Original discussion: The test helper still reconstructs streamed tool calls by appending each fragment to the most recent name instead of grouping by tool_call.index. When the processor emits both GLM calls with index 0 for some step/offset cases, this assertion can still pass even though OpenAI-stream clients would merge the fragments into one malformed call, so the regression coverage remains weaker than claimed.

@mifegui
mifegui temporarily deployed to external_collaborator October 7, 2026 22:42 — with GitHub Actions Inactive
Signed-off-by: Eric Guimarães <ericg@telnyx.com>
@pull-request-size pull-request-size Bot removed the size/L label Oct 7, 2026
@mifegui
mifegui temporarily deployed to external_collaborator October 7, 2026 22:44 — with GitHub Actions Inactive
@rmccorm4
rmccorm4 temporarily deployed to external_collaborator October 14, 2026 00:20 — with GitHub Actions Inactive
@rmccorm4

Copy link
Copy Markdown
Contributor

/ok to test ccc5ece

@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test ccc5ece

@devin-ai-integration

devin-ai-integration Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

❌ Dynamo PR CI failed — run 38790719904 (attempt 1) on afdeccc004

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ❌ dynamo-status-check

Other Jobs
dynamo-runtime ❌ 2 ✅ 6

Failure details

2 jobs failed (amd64 and arm64) on the same single test: tests/test_kustomize_matrix.py::test_user_facing_manifests_use_short_vllm_component_names rejects the legacy vLLM component name VllmWorker at recipes/muse-spark-open-816b-a42b/vllm/agg-gb300-agentic/deploy.yaml:67. No jobs were cancelled.

❌ dynamo-runtime / test / sequential cuda13.0, amd64: legacy component name VllmWorker in a recipe manifest

Job: dynamo-runtime / test / sequential cuda13.0, amd64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 116622069276 -R ai-dynamo/dynamo --log-failed

FAILED tests/test_kustomize_matrix.py::test_user_facing_manifests_use_short_vllm_component_names - AssertionError: user-facing manifests must use worker, prefill, or decode instead of legacy vLLM component names:
  recipes/muse-spark-open-816b-a42b/vllm/agg-gb300-agentic/deploy.yaml:67: - name: VllmWorker
assert not ['recipes/muse-spark-open-816b-a42b/vllm/agg-gb300-agentic/deploy.yaml:67: - name: VllmWorker']
== 1 failed, 5280 passed, 153 skipped, 2525 deselected in 1133.11s (0:18:53) ===
❌ pre_merge_cpu: step outcome 'failure'

In the pre_merge_cpu stage, 1 test failed (5280 passed, all retries failed too): tests/test_kustomize_matrix.py::test_user_facing_manifests_use_short_vllm_component_names asserts that user-facing manifests use worker, prefill or decode for vLLM components, but recipes/muse-spark-open-816b-a42b/vllm/agg-gb300-agentic/deploy.yaml line 67 still has - name: VllmWorker. The later pytest stages did not run because this stage failed.

❌ dynamo-runtime / test / sequential cuda13.0, arm64: legacy component name VllmWorker in a recipe manifest

Job: dynamo-runtime / test / sequential cuda13.0, arm64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 116622069347 -R ai-dynamo/dynamo --log-failed

FAILED tests/test_kustomize_matrix.py::test_user_facing_manifests_use_short_vllm_component_names - AssertionError: user-facing manifests must use worker, prefill, or decode instead of legacy vLLM component names:
  recipes/muse-spark-open-816b-a42b/vllm/agg-gb300-agentic/deploy.yaml:67: - name: VllmWorker
assert not ['recipes/muse-spark-open-816b-a42b/vllm/agg-gb300-agentic/deploy.yaml:67: - name: VllmWorker']
== 1 failed, 5280 passed, 153 skipped, 2525 deselected in 1278.37s (0:21:18) ===
❌ pre_merge_cpu: step outcome 'failure'

In the pre_merge_cpu stage, 1 test failed (5280 passed, all retries failed too): tests/test_kustomize_matrix.py::test_user_facing_manifests_use_short_vllm_component_names asserts that user-facing manifests use worker, prefill or decode for vLLM components, but recipes/muse-spark-open-816b-a42b/vllm/agg-gb300-agentic/deploy.yaml line 67 still has - name: VllmWorker. The later pytest stages did not run because this stage failed.

For agents
{"pr": 15829, "run_id": 38790719904, "run_attempt": 1, "head_sha": "afdeccc004b83b0d2baf0bdf4d29443c04f413ac", "failures": [{"job": "dynamo-runtime / test / sequential cuda13.0, amd64", "job_id": 116622069276, "failed_step": "Run CPU-only tests (parallelized)", "signature": "test_user_facing_manifests_use_short_vllm_component_names: legacy component name VllmWorker in recipes/muse-spark-open-816b-a42b/vllm/agg-gb300-agentic/deploy.yaml:67", "tests": ["tests/test_kustomize_matrix.py::test_user_facing_manifests_use_short_vllm_component_names"], "log_cmd": "gh run view --job 116622069276 -R ai-dynamo/dynamo --log-failed"}, {"job": "dynamo-runtime / test / sequential cuda13.0, arm64", "job_id": 116622069347, "failed_step": "Run CPU-only tests (parallelized)", "signature": "test_user_facing_manifests_use_short_vllm_component_names: legacy component name VllmWorker in recipes/muse-spark-open-816b-a42b/vllm/agg-gb300-agentic/deploy.yaml:67", "tests": ["tests/test_kustomize_matrix.py::test_user_facing_manifests_use_short_vllm_component_names"], "log_cmd": "gh run view --job 116622069347 -R ai-dynamo/dynamo --log-failed"}]}

Posted automatically by Devin for run 38790719904. Updated on every full-CI run of this PR.

@rmccorm4
rmccorm4 temporarily deployed to external_collaborator October 15, 2026 19:02 — with GitHub Actions Inactive
@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test afdeccc

@rmccorm4
rmccorm4 temporarily deployed to external_collaborator October 16, 2026 06:03 — with GitHub Actions Inactive
@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test 3a2dcaa

@rmccorm4
rmccorm4 enabled auto-merge (squash) October 16, 2026 06:04
@rmccorm4
rmccorm4 merged commit 178cc2f into ai-dynamo:main Oct 16, 2026
110 checks passed

This branch was previously deployed

1 inactive deployment
external_collaborator — 3a2dcaa4 Deployed Oct 16, 2026 by rmccorm4 via ok-to-test #33995
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

external-contribution Pull request is from an external contributor fix frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` size/M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: vLLM chat processor streams the second parallel tool call as content when a step closes one call and opens the next (MTP)

3 participants