Skip to content

perf(llm): batch the per-token output-token counter per request - #17067

Merged
jthomson04 merged 2 commits into
mainfrom
jthomson04/local-output-token-counter
Oct 23, 2026
Merged

jthomson04 merged 2 commits into
mainfrom
jthomson04/local-output-token-counter

Conversation

@jthomson04

@jthomson04 jthomson04 commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Overview:

dynamo_frontend_output_tokens_total is one atomic per model, and every in-flight request added to it on every output token, so its cache line bounced across the frontend's cores. Each request now accumulates output tokens in its RequestMetrics and publishes them every 64 tokens and when it finishes, the same batching the ITL histogram already uses. Writes to the shared atomic drop ~60× (~2M → ~35K adds/s per frontend below).

observe_response self time base this PR
numa0 frontend 0.50% 0.19%
numa1 frontend 0.50% 0.14%

Tyche agentx-dual-numa: 2 frontends, 2,048 mockers, c=6,144, Qwen3-0.6B, cache-bust. Stack: main db6d5b5434 + #16728. On-CPU, -c 2500000. These numbers are from before the rebase onto #13187. That PR moved output-token counting into RequestMetrics, so the batching moved there too.

Details:

  • RequestMetrics::add_output_tokens adds to a per-request pending_output_tokens. The pending count is published to the shared counter at LOCAL_FLUSH_TOKENS (64, was ITL_LOCAL_FLUSH_TOKENS), in finish, and in Drop. SGLang /generate counts through RequestMetrics directly, so it is batched too.
  • ITL batching uses LocalHistogram::get_sample_count() instead of the itl_pending_tokens field.
  • Totals are exact once a request finishes. While a request is live, /metrics and the serving-load frame lag by at most 63 of its tokens.
  • Validation: cargo test -p dynamo-llm --lib -- http::service::metrics http::service::generate http::service::sglang_generate, cargo test -p dynamo-llm --test http_metrics, clippy, fmt.
  • Reproduce (base arm omits the second --patch):
frontend-mocker run agentx-dual-numa --site tyche --ref db6d5b5434 --patch pr16728.patch --patch local-output-token-counter.patch --profile oncpu --set runtime.tokenizer_cache_bytes=34359738368 --set 'load.cache_bust={target: first_turn_prefix}'

Where should the reviewer start?

RequestMetrics::add_output_tokens, finish, and its Drop in lib/llm/src/http/service/metrics.rs.

Related Issues

  • None.

🤖 Generated with Claude Code

@jthomson04
jthomson04 requested review from a team as code owners October 21, 2026 23:46
@jthomson04

Copy link
Copy Markdown
Contributor Author

/ok to test 0f0204a

@github-actions github-actions Bot added perf frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` labels Oct 21, 2026
@coderabbitai

coderabbitai Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: ai-dynamo/dynamo/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: 86680f14-dcac-44eb-bf5b-f0f69ef3326e
📥 Commits

Reviewing files that changed from the base of the PR and between b434e94 and 0f0204a.

📒 Files selected for processing: 1
  • lib/llm/src/http/service/metrics.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

Walkthrough

ResponseMetricCollector now batches output-token increments in a per-model local counter. It publishes accumulated tokens at 64-token thresholds and when the collector is dropped. Tests cover deferred publication, threshold and drop flushing, zero-token observations, and per-model isolation.

Changes

Output-token metrics

Layer / File(s) Summary
Counter batching and flush
lib/llm/src/http/service/metrics.rs
The collector stores output-token increments locally and publishes them at the 64-token threshold or on drop. Tests verify deferred publication, threshold and drop flushing, zero-token behavior, and per-model totals.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to 0f020

The supplied evidence identifies no issue that needs resolution before merge. Normal checks remain appropriate.

Pre-merge checks: Passed 3 · Failed 2

Failed Failed checks: 2 warnings

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 71.43% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 7 functions across 1 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check Warning The description follows the template's Overview, Details, and Where should the reviewer start sections with substantive content. It explains the motivation (cache-line bouncing on a shared atomic), th… Add a Related Issues reference in the template's format, for example "Closes #XXXX" for a closed issue or "Relates to #XXXX" for a partial link. A Linear reference such as "Closes DYN-1234" also works. If no issue exists, create one and lin…
Passed Passed checks: 3 passed
Check name Status Explanation
Title check Passed The title clearly and concisely describes the main change: batching per-token output-token counter updates per request.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Description check

Explanation

The description follows the template's Overview, Details, and Where should the reviewer start sections with substantive content. It explains the motivation (cache-line bouncing on a shared atomic), the batching change (flush every 64 tokens, on finish, and on drop), benchmark numbers, validation commands, and reviewer starting points. The Related Issues section is required by the template, but it only says "None." and has no "Closes #XXXX" or "Relates to #XXXX" reference. The template marks this section as required, so the description is missing required information. The rest of the description is complete enough that the check does not fail outright.

Resolution

Add a Related Issues reference in the template's format, for example "Closes #XXXX" for a closed issue or "Relates to #XXXX" for a partial link. A Linear reference such as "Closes DYN-1234" also works. If no issue exists, create one and link it before merge. Also consider noting that the PR objectives reference PR #17067 and the description references #13187 and #16728, so the author should confirm which PR this change targets and link any tracking issue accordingly.

✨ Finishing Touches: 💡 1
🛠️ Fix failing CI checks: 💡
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

Comment thread lib/llm/src/http/service/metrics.rs Resolved
Comment thread lib/llm/src/http/service/metrics.rs Resolved
Comment thread lib/llm/src/http/service/metrics.rs Resolved
@devin-ai-integration

devin-ai-integration Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

❌ Dynamo PR CI failed — run 39901524171 (attempt 1) on d92fea550f

Gate checks: ❌ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Framework 1-GPU amd64
SGLang ❌ 1

Failure details

1 job failed: in the SGLang 1-GPU amd64 test job, test_frontend_api_surface_compliance fails because the Codex CLI it shells out to cannot start (missing npm platform package @openai/codex-linux-x64). All other GPU-parallel tests passed (15 passed, 2 skipped), and the CPU and sequential GPU stages passed.

❌ sglang-runtime / Test cuda13.0, amd64: codex exec failed: Missing optional dependency @openai/codex-linux-x64

Job: sglang-runtime / Test cuda13.0, amd64 · Failed step: Run GPU tests (parallel) · Logs: gh run view --job 120247103791 -R ai-dynamo/dynamo --log-failed

[w0] tests/frontend/test_frontend_api_surface_compliance.py::test_frontend_api_surface_compliance RETRY FAILED (Auto Test Retries) [100%]
[w0] tests/frontend/test_frontend_api_surface_compliance.py::test_frontend_api_surface_compliance FAILED [100%]
[w0] FAILED tests/frontend/test_frontend_api_surface_compliance.py::test_frontend_api_surface_compliance - Failed: codex exec failed (exit=1).
[w0] stderr:
[w0] file:///scratch/pytest_temp/.../_frontend_api_surface_tools/codex/node_modules/@openai/codex/bin/codex.js:107
[w0]   throw new Error(
[w0] Error: Missing optional dependency @openai/codex-linux-x64. Reinstall Codex: npm install -g @openai/codex@latest
[w0]     at findCodexExecutable (file:///scratch/pytest_temp/.../codex/node_modules/@openai/codex/bin/codex.js:107:9)
[w0] Node.js v20.19.0
== 1 failed, 15 passed, 2 skipped in 1035.51s (17:15) (vs 1442s seq, 1.4x) ==
❌ pre_merge_gpu_parallel: step outcome 'failure'
##[error]A pytest stage did not record a passing result. See this step's log for the stage.

tests/frontend/test_frontend_api_surface_compliance.py::test_frontend_api_surface_compliance failed on the first attempt and all 3 auto-retries with codex exec failed (exit=1). Codex's launcher codex.js throws because the optional platform binary package @openai/codex-linux-x64 is missing from the Codex install the test downloads into _frontend_api_surface_tools/codex. So the CLI never reached the Dynamo frontend. This is an incomplete external-tool install, not an error from a Dynamo server. If the package failed to download during the test's tool install, rerunning the failed job may be enough.

For agents
{"pr": 17067, "run_id": 39901524171, "run_attempt": 1, "head_sha": "d92fea550f91b404a6645b74f9fa42081b749b0a", "failures": [{"job": "sglang-runtime / Test cuda13.0, amd64", "job_id": 120247103791, "failed_step": "Run GPU tests (parallel)", "signature": "codex exec failed: Missing optional dependency @openai/codex-linux-x64", "tests": ["tests/frontend/test_frontend_api_surface_compliance.py::test_frontend_api_surface_compliance"], "log_cmd": "gh run view --job 120247103791 -R ai-dynamo/dynamo --log-failed"}]}

Posted automatically by Devin for run 39901524171. Updated on every full-CI run of this PR.

@jthomson04

Copy link
Copy Markdown
Contributor Author

/ok to test 5b0184a

jthomson04 and others added 2 commits October 22, 2026 21:31
ResponseMetricCollector added every output token to the per-model
output_tokens_total counter, one atomic shared by all in-flight requests,
so each token bounced its cache line between cores. Count locally with a
LocalIntCounter and flush every ITL_LOCAL_FLUSH_TOKENS (64) tokens and on
drop, as the ITL histogram already does. Totals are unchanged; a request's
unpublished tokens lag by at most 63.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
…batching

Rename ITL_LOCAL_FLUSH_TOKENS to LOCAL_FLUSH_TOKENS now that it also
gates the output-token counter, and drop itl_pending_tokens: the local
histogram's get_sample_count() already tracks the unflushed count, so
both per-token metrics flush the same way. Fix the field comment and
remove comments and a test assert that only restated the code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@jthomson04
jthomson04 force-pushed the jthomson04/local-output-token-counter branch from 5b0184a to d92fea5 Compare October 23, 2026 04:39
@jthomson04

Copy link
Copy Markdown
Contributor Author

/ok to test d92fea5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` perf size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants