Repository navigation
perf(llm): batch the per-token output-token counter per request - #17067
Conversation
|
/ok to test 0f0204a |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
Walkthrough
ChangesOutput-token metrics
Priority: ⬇️ Low Estimated code review effort: 2 (Simple) | ~10 minutes Merge Risk: ⚪ Minimal · up to
|
| Check name | Status | Explanation | Resolution |
|---|---|---|---|
| Docstring Coverage | Docstring coverage is 71.43% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 7 functions across 1 files. | Write docstrings for the functions missing them to satisfy the coverage threshold. | |
| Description check | The description follows the template's Overview, Details, and Where should the reviewer start sections with substantive content. It explains the motivation (cache-line bouncing on a shared atomic), th… | Add a Related Issues reference in the template's format, for example "Closes #XXXX" for a closed issue or "Relates to #XXXX" for a partial link. A Linear reference such as "Closes DYN-1234" also works. If no issue exists, create one and lin… |

Passed checks: 3 passed
Full details: Description check
-
Explanation
The description follows the template's Overview, Details, and Where should the reviewer start sections with substantive content. It explains the motivation (cache-line bouncing on a shared atomic), the batching change (flush every 64 tokens, on finish, and on drop), benchmark numbers, validation commands, and reviewer starting points. The Related Issues section is required by the template, but it only says "None." and has no "Closes
#XXXX" or "Relates to#XXXX" reference. The template marks this section as required, so the description is missing required information. The rest of the description is complete enough that the check does not fail outright.Resolution
Add a Related Issues reference in the template's format, for example "Closes
#XXXX" for a closed issue or "Relates to#XXXX" for a partial link. A Linear reference such as "Closes DYN-1234" also works. If no issue exists, create one and link it before merge. Also consider noting that the PR objectives reference PR#17067and the description references#13187and#16728, so the author should confirm which PR this change targets and link any tracking issue accordingly.
✨ Finishing Touches: 💡 1
-
🛠️ Fix failing CI checks: 💡
-
- Commit to this branch
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Comment @coderabbitai help to get the list of available commands.
❌ Dynamo PR CI failed — run 39901524171 (attempt 1) on
|
| Framework | 1-GPU amd64 |
|---|---|
| SGLang | ❌ 1 |
Failure details
1 job failed: in the SGLang 1-GPU amd64 test job, test_frontend_api_surface_compliance fails because the Codex CLI it shells out to cannot start (missing npm platform package @openai/codex-linux-x64). All other GPU-parallel tests passed (15 passed, 2 skipped), and the CPU and sequential GPU stages passed.
❌ sglang-runtime / Test cuda13.0, amd64: codex exec failed: Missing optional dependency @openai/codex-linux-x64
Job: sglang-runtime / Test cuda13.0, amd64 · Failed step: Run GPU tests (parallel) · Logs: gh run view --job 120247103791 -R ai-dynamo/dynamo --log-failed
[w0] tests/frontend/test_frontend_api_surface_compliance.py::test_frontend_api_surface_compliance RETRY FAILED (Auto Test Retries) [100%]
[w0] tests/frontend/test_frontend_api_surface_compliance.py::test_frontend_api_surface_compliance FAILED [100%]
[w0] FAILED tests/frontend/test_frontend_api_surface_compliance.py::test_frontend_api_surface_compliance - Failed: codex exec failed (exit=1).
[w0] stderr:
[w0] file:///scratch/pytest_temp/.../_frontend_api_surface_tools/codex/node_modules/@openai/codex/bin/codex.js:107
[w0] throw new Error(
[w0] Error: Missing optional dependency @openai/codex-linux-x64. Reinstall Codex: npm install -g @openai/codex@latest
[w0] at findCodexExecutable (file:///scratch/pytest_temp/.../codex/node_modules/@openai/codex/bin/codex.js:107:9)
[w0] Node.js v20.19.0
== 1 failed, 15 passed, 2 skipped in 1035.51s (17:15) (vs 1442s seq, 1.4x) ==
❌ pre_merge_gpu_parallel: step outcome 'failure'
##[error]A pytest stage did not record a passing result. See this step's log for the stage.
tests/frontend/test_frontend_api_surface_compliance.py::test_frontend_api_surface_compliance failed on the first attempt and all 3 auto-retries with codex exec failed (exit=1). Codex's launcher codex.js throws because the optional platform binary package @openai/codex-linux-x64 is missing from the Codex install the test downloads into _frontend_api_surface_tools/codex. So the CLI never reached the Dynamo frontend. This is an incomplete external-tool install, not an error from a Dynamo server. If the package failed to download during the test's tool install, rerunning the failed job may be enough.
For agents
{"pr": 17067, "run_id": 39901524171, "run_attempt": 1, "head_sha": "d92fea550f91b404a6645b74f9fa42081b749b0a", "failures": [{"job": "sglang-runtime / Test cuda13.0, amd64", "job_id": 120247103791, "failed_step": "Run GPU tests (parallel)", "signature": "codex exec failed: Missing optional dependency @openai/codex-linux-x64", "tests": ["tests/frontend/test_frontend_api_surface_compliance.py::test_frontend_api_surface_compliance"], "log_cmd": "gh run view --job 120247103791 -R ai-dynamo/dynamo --log-failed"}]}Posted automatically by Devin for run 39901524171. Updated on every full-CI run of this PR.
|
/ok to test 5b0184a |
ResponseMetricCollector added every output token to the per-model output_tokens_total counter, one atomic shared by all in-flight requests, so each token bounced its cache line between cores. Count locally with a LocalIntCounter and flush every ITL_LOCAL_FLUSH_TOKENS (64) tokens and on drop, as the ITL histogram already does. Totals are unchanged; a request's unpublished tokens lag by at most 63. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
…batching Rename ITL_LOCAL_FLUSH_TOKENS to LOCAL_FLUSH_TOKENS now that it also gates the output-token counter, and drop itl_pending_tokens: the local histogram's get_sample_count() already tracks the unflushed count, so both per-token metrics flush the same way. Fix the field comment and remove comments and a test assert that only restated the code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
5b0184a to
d92fea5
Compare
|
/ok to test d92fea5 |
Overview:
dynamo_frontend_output_tokens_totalis one atomic per model, and every in-flight request added to it on every output token, so its cache line bounced across the frontend's cores. Each request now accumulates output tokens in itsRequestMetricsand publishes them every 64 tokens and when it finishes, the same batching the ITL histogram already uses. Writes to the shared atomic drop ~60× (~2M → ~35K adds/s per frontend below).observe_responseself timeTyche
agentx-dual-numa: 2 frontends, 2,048 mockers, c=6,144, Qwen3-0.6B, cache-bust. Stack: maindb6d5b5434+ #16728. On-CPU,-c 2500000. These numbers are from before the rebase onto #13187. That PR moved output-token counting intoRequestMetrics, so the batching moved there too.Details:
RequestMetrics::add_output_tokensadds to a per-requestpending_output_tokens. The pending count is published to the shared counter atLOCAL_FLUSH_TOKENS(64, wasITL_LOCAL_FLUSH_TOKENS), infinish, and inDrop. SGLang/generatecounts throughRequestMetricsdirectly, so it is batched too.LocalHistogram::get_sample_count()instead of theitl_pending_tokensfield./metricsand the serving-load frame lag by at most 63 of its tokens.cargo test -p dynamo-llm --lib -- http::service::metrics http::service::generate http::service::sglang_generate,cargo test -p dynamo-llm --test http_metrics, clippy, fmt.--patch):frontend-mocker run agentx-dual-numa --site tyche --ref db6d5b5434 --patch pr16728.patch --patch local-output-token-counter.patch --profile oncpu --set runtime.tokenizer_cache_bytes=34359738368 --set 'load.cache_bust={target: first_turn_prefix}'Where should the reviewer start?
RequestMetrics::add_output_tokens,finish, and itsDropinlib/llm/src/http/service/metrics.rs.Related Issues
🤖 Generated with Claude Code