Repository navigation
fix(frontend): keep buffering tool text while a later tool call is still open - #15829
Conversation
When reasoning ends in the same chunk as a tool call starts, the vLLM chat processor buffers the tool text and parses it on the first end marker. With multi-token steps (e.g. MTP) that chunk can also open the next call, which was dropped, and the rest of that call then streamed as content. Close the buffer only when no call opened after the last end marker. Signed-off-by: Eric Guimarães <ericg@telnyx.com>
|
👋 Hi mifegui! Thank you for contributing to ai-dynamo/dynamo. Just a reminder: The 🚀 |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configuration
📒 Files selected for processing (2)
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review. WalkthroughThe streaming processor now completes buffered tool-call parsing only when the final tool call is closed, unless a finish reason completes it first. A regression test checks two consecutive calls across token batch sizes 1–6. ChangesTool-call streaming
Priority: ➖ Normal Estimated code review effort: 2 (Simple) | ~10 minutes Severity of issue fixed: Medium Merge Risk: ⚪ Minimal · up to No actionable merge-blocking risk is established for this change; it is mergeable after normal checks. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Comment |
Address review: a start marker split across steps left an incomplete prefix after the last end marker. Drop the step sizes that never reached the buffer path. Signed-off-by: Eric Guimarães <ericg@telnyx.com>
Closing the buffer mid-response handed later calls to the streaming parser, which numbered them from 0 again, so two calls could share an index. Parse the whole buffer when the response finishes instead; this also removes the marker scan. The test now rebuilds calls by index. Signed-off-by: Eric Guimarães <ericg@telnyx.com>
There was a problem hiding this comment.
Previously reported defects still present:
- Original discussion: Still present:
_assert_two_glm_calls_streamedaggregates emitted tool calls by append order and ignorestool_call.index, so cases where both calls are emitted with index 0 can still pass despite clients merging them into one malformed call. - Original discussion: The helper still reconstructs streamed tool calls by appending each named function to positional lists and never uses
tool_call.index, so emitted deltas that reuse index 0 for both calls can still satisfy the new assertions even though clients would merge them into one malformed call. - Original discussion: The new test helper still reconstructs streamed tool calls by appending arguments to the most recent name instead of grouping deltas by
tool_call.index, so cases where both emitted calls use index 0 continue to pass while clients would merge them into one malformed call. - Original discussion: The test helper still reconstructs streamed tool calls by appending a new argument buffer whenever a delta carries a function name, instead of grouping deltas by tool_call.index. As previously reported, some step/offset cases can emit both calls at index 0 while this helper still passes, so the new test can miss the client-visible merge into one malformed call.
- Original discussion: The test helper still reconstructs streamed tool calls by appending each fragment to the most recent name instead of grouping by
tool_call.index. When the processor emits both GLM calls with index 0 for some step/offset cases, this assertion can still pass even though OpenAI-stream clients would merge the fragments into one malformed call, so the regression coverage remains weaker than claimed.
Signed-off-by: Eric Guimarães <ericg@telnyx.com>
|
/ok to test ccc5ece |
|
/ok to test ccc5ece |
❌ Dynamo PR CI failed — run 38790719904 (attempt 1) on
|
| Other | Jobs |
|---|---|
| dynamo-runtime | ❌ 2 ✅ 6 |
Failure details
2 jobs failed (amd64 and arm64) on the same single test: tests/test_kustomize_matrix.py::test_user_facing_manifests_use_short_vllm_component_names rejects the legacy vLLM component name VllmWorker at recipes/muse-spark-open-816b-a42b/vllm/agg-gb300-agentic/deploy.yaml:67. No jobs were cancelled.
❌ dynamo-runtime / test / sequential cuda13.0, amd64: legacy component name VllmWorker in a recipe manifest
Job: dynamo-runtime / test / sequential cuda13.0, amd64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 116622069276 -R ai-dynamo/dynamo --log-failed
FAILED tests/test_kustomize_matrix.py::test_user_facing_manifests_use_short_vllm_component_names - AssertionError: user-facing manifests must use worker, prefill, or decode instead of legacy vLLM component names:
recipes/muse-spark-open-816b-a42b/vllm/agg-gb300-agentic/deploy.yaml:67: - name: VllmWorker
assert not ['recipes/muse-spark-open-816b-a42b/vllm/agg-gb300-agentic/deploy.yaml:67: - name: VllmWorker']
== 1 failed, 5280 passed, 153 skipped, 2525 deselected in 1133.11s (0:18:53) ===
❌ pre_merge_cpu: step outcome 'failure'
In the pre_merge_cpu stage, 1 test failed (5280 passed, all retries failed too): tests/test_kustomize_matrix.py::test_user_facing_manifests_use_short_vllm_component_names asserts that user-facing manifests use worker, prefill or decode for vLLM components, but recipes/muse-spark-open-816b-a42b/vllm/agg-gb300-agentic/deploy.yaml line 67 still has - name: VllmWorker. The later pytest stages did not run because this stage failed.
❌ dynamo-runtime / test / sequential cuda13.0, arm64: legacy component name VllmWorker in a recipe manifest
Job: dynamo-runtime / test / sequential cuda13.0, arm64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 116622069347 -R ai-dynamo/dynamo --log-failed
FAILED tests/test_kustomize_matrix.py::test_user_facing_manifests_use_short_vllm_component_names - AssertionError: user-facing manifests must use worker, prefill, or decode instead of legacy vLLM component names:
recipes/muse-spark-open-816b-a42b/vllm/agg-gb300-agentic/deploy.yaml:67: - name: VllmWorker
assert not ['recipes/muse-spark-open-816b-a42b/vllm/agg-gb300-agentic/deploy.yaml:67: - name: VllmWorker']
== 1 failed, 5280 passed, 153 skipped, 2525 deselected in 1278.37s (0:21:18) ===
❌ pre_merge_cpu: step outcome 'failure'
In the pre_merge_cpu stage, 1 test failed (5280 passed, all retries failed too): tests/test_kustomize_matrix.py::test_user_facing_manifests_use_short_vllm_component_names asserts that user-facing manifests use worker, prefill or decode for vLLM components, but recipes/muse-spark-open-816b-a42b/vllm/agg-gb300-agentic/deploy.yaml line 67 still has - name: VllmWorker. The later pytest stages did not run because this stage failed.
For agents
{"pr": 15829, "run_id": 38790719904, "run_attempt": 1, "head_sha": "afdeccc004b83b0d2baf0bdf4d29443c04f413ac", "failures": [{"job": "dynamo-runtime / test / sequential cuda13.0, amd64", "job_id": 116622069276, "failed_step": "Run CPU-only tests (parallelized)", "signature": "test_user_facing_manifests_use_short_vllm_component_names: legacy component name VllmWorker in recipes/muse-spark-open-816b-a42b/vllm/agg-gb300-agentic/deploy.yaml:67", "tests": ["tests/test_kustomize_matrix.py::test_user_facing_manifests_use_short_vllm_component_names"], "log_cmd": "gh run view --job 116622069276 -R ai-dynamo/dynamo --log-failed"}, {"job": "dynamo-runtime / test / sequential cuda13.0, arm64", "job_id": 116622069347, "failed_step": "Run CPU-only tests (parallelized)", "signature": "test_user_facing_manifests_use_short_vllm_component_names: legacy component name VllmWorker in recipes/muse-spark-open-816b-a42b/vllm/agg-gb300-agentic/deploy.yaml:67", "tests": ["tests/test_kustomize_matrix.py::test_user_facing_manifests_use_short_vllm_component_names"], "log_cmd": "gh run view --job 116622069347 -R ai-dynamo/dynamo --log-failed"}]}Posted automatically by Devin for run 38790719904. Updated on every full-CI run of this PR.
|
/ok to test afdeccc |
|
/ok to test 3a2dcaa |
Summary
When reasoning ends in the same chunk as a tool call starts, the vLLM chat processor buffers the tool text and parsed it as soon as the buffer held an end marker, handing the rest of the response to the streaming parser. With multi-token steps (e.g. MTP) the chunk that closes one call can also open the next: that call was dropped and streamed as
content, and when the split fell between calls, the streaming parser numbered the next call from 0 again, so clients merged two calls into one.The buffered text is now parsed once, when the response finishes, so all calls come from one parse with sequential indices. Normal streaming (no reasoning/tool collision) is unchanged.
Fixes #15828
Validation
test_parallel_tool_calls_survive_multi_token_stepsandtest_parallel_tool_calls_survive_split_start_marker: stream two GLM tool calls after</think>in multi-token steps and with a split start marker, rebuilding calls byindexas a client does. They fail on main and pass here.pytest components/src/dynamo/frontend/tests/test_vllm_processor_unit.py: 176 passedpre-commit run --fileson both filesA call cut off by
max_tokenscomes back the way a non-streaming request returns it, with the arguments parsed so far andfinish_reason: length; dropping unfinished calls onlengthis left for a follow-up.