Skip to content

feat(bindings): mark Python response encoding with an NVTX range - #17707

Merged
nv-kmcgill53 merged 1 commit into
mainfrom
kmcgill/nvtx-encode-range
Nov 5, 2026
Merged

nv-kmcgill53 merged 1 commit into
mainfrom
kmcgill/nvtx-encode-range

Conversation

@nv-kmcgill53

@nv-kmcgill53 nv-kmcgill53 commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Wraps Python response encoding (encode_python_item) in a pybridge.encode_response NVTX range. Without it, an Nsight capture of a Python worker misattributes this time. The encoding runs inside the response stream's next(), before worker.egress.chunk opens. Its GIL-held parse and serialization therefore lands in the gap between chunk ranges and reads as engine time. tanmayv25 raised this on #17334 (review thread), and that PR deferred it to this follow-up.

Key Changes

  • lib/bindings/python/rust/python_payload.rs: adds let _nvtx = dynamo_nvtx_range!("pybridge.encode_response"); inside the spawn_blocking closure in encode_python_item. The range covers encode_python_response and nothing else.

Validation

  • cargo check --features nvtx passes in lib/bindings/python.
  • Real capture: Qwen3-0.6B on vLLM (eager mode, one RTX PRO 6000), streaming requests, DYN_NVTX=1 under nsys -t nvtx,cuda. The range fires once per chunk, with a median of about 22 µs. It ends about 13 µs before worker.egress.chunk opens. The last 150 µs before a chunk:
pybridge.anext_call                                                                  [--]
pybridge.encode_response                                                  [----]
worker.egress.chunk                                                               [--]
worker.egress.encode                                                                |
worker.egress.publish                                                               []
GPU busy (vLLM kernels)  .    :.    +           :                      ++               #+    +
                         -150 us                                                         +36.9 us
                         [--] range  | under 1 column  ^ mark  GPU . <25% : <50% + <75% # more
                         1 column = 2.6 us; GPU busy 14.0 us of 186.9 us (7%)
  • Overhead: one range per chunk, and a no-op unless the feature is built and DYN_NVTX=1. Under nsys it costs about 1.3–1.7 µs, against the 22 µs it measures.

Issues and Ongoing Work

Pull Request Checklist
  • I used Dynamo's /pr-checklist skill.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Performance Diagnostics
    • Python response-encoding operations are now marked in performance traces for easier visibility.

encode_python_item runs inside the response stream's next(), before
worker.egress.chunk opens, so its GIL-held parse and serialization showed
up as engine time in the gap between chunk ranges. Wrap it in
pybridge.encode_response.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Kyle McGill <kmcgill@nvidia.com>
@coderabbitai

coderabbitai Bot commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between c10c879 and fa615fa.

Files selected for processing: 1
  • lib/bindings/python/rust/python_payload.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

Walkthrough

The Python response-encoding task now marks each encoding operation with the pybridge.encode_response NVTX range. The encoding call and its existing error handling remain unchanged.

Changes

Python response encoding

Layer / File(s) Summary
Mark response encoding
lib/bindings/python/rust/python_payload.rs
The code imports dynamo_nvtx_range and applies the pybridge.encode_response range around response encoding in the blocking task. Encoding behavior and error handling remain unchanged.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Merge Risk: Minimal · up to fa615

The instrumentation is isolated and does not change response encoding behavior; the PR is mergeable.

Pre-merge checks: Passed 5
Check name Status Explanation
Title check Passed The title "feat(bindings): mark Python response encoding with an NVTX range" matches the single change in the changeset. The diff imports dynamo_nvtx_range and wraps encode_python_response in a pybrid…
Description check Passed The description follows the repository template. It has Overview, Key Changes, Validation, and Issues and Ongoing Work sections, plus the Pull Request Checklist block. The Overview explains the motiva…
Docstring Coverage Passed No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title "feat(bindings): mark Python response encoding with an NVTX range" matches the single change in the changeset. The diff imports dynamo_nvtx_range and wraps encode_python_response in a pybridge.encode_response NVTX range inside the spawn_blocking closure of encode_python_item. The title is concise, specific, and names the primary change and its scope.

Full details: Description check

Explanation

The description follows the repository template. It has Overview, Key Changes, Validation, and Issues and Ongoing Work sections, plus the Pull Request Checklist block. The Overview explains the motivation: encoding runs before worker.egress.chunk opens, so its time was misattributed. Key Changes names the file and the added line. Validation gives a cargo check result, a captured profile with a median of about 22 µs, and overhead figures. Issues lists the related PRs and DIS-3260. The template's Closes #XXXX line is replaced by a Relates to DIS-3260 reference, and the checklist item remains unchecked. These are minor gaps; the description is mostly complete.

Full details: Docstring Coverage

Explanation

No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 1 files.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@devin-ai-integration

Copy link
Copy Markdown
Contributor

✅ Dynamo PR CI passed — run 41123793555 (attempt 1) on fa615fa46c

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Posted automatically by Devin for run 41123793555. Updated on every full-CI run of this PR.

@nv-kmcgill53
nv-kmcgill53 merged commit 53015f9 into main Nov 5, 2026
146 checks passed
@nv-kmcgill53
nv-kmcgill53 deleted the kmcgill/nvtx-encode-range branch November 5, 2026 00:39
ishandhanani added a commit to ishandhanani/dynamo that referenced this pull request Nov 5, 2026
Second upstream merge for the split v2 PR: 36 commits past the first
merge's base 182039d, 17 conflicted paths, each upstream change placed
in the crate that owns the code at the stack head, upstream behavior whole,
the stack's moves intact, no shims.

- ai-dynamo#17723 flatten policy classes: flat PolicyProfile (from_classes,
  UnknownPolicyClassError) in lib/router/admission profile.rs; the YAML
  half (default_policy_class, removed keys rejected) in lib/router/kv-router
  policy_config.rs; KvSchedulerError::UnknownPolicyClass via From in the
  kernel; 400/InvalidArgument in lib/llm (UnknownPolicyClassRefusal) and
  ext-proc.
- ai-dynamo#17724 EDF ordering: RouterQueuePolicy::Edf in admission profile.rs;
  due-time order, DueKey and class budgets in admission queue.rs; the
  restored score keys (RouterSchedulingPolicy::score_policy) as the
  kernel's policy.rs (ordered-float back in dynamo-scheduler).
- ai-dynamo#17725 controlled delay: head expiry and with_class_budget_expiry in
  admission queue.rs; ClassTimers, due-time checks, DRR charged on
  admission (extract_next/settle) and the retained overlap refresh in the
  kernel's queue.rs and overlap_refresh.rs; metrics in lib/llm.
- ai-dynamo#17437 image-aware KV routing for TRT-LLM sidecars: zmq_wire/trtllm.rs
  in lib/kv-events; the sidecar depends on kv-events, kv-hashing,
  multimodal and protocol instead of kv-router.
- ai-dynamo#14775 prefill cancellation, ai-dynamo#17716/ai-dynamo#18077 pool relay, ai-dynamo#17707 NVTX
  range, ai-dynamo#18102 LiveEngine buffering, ai-dynamo#17887 shared replay clock: lib/llm,
  bindings, mocker, bench, auto-merged with import re-points.

Tests: upstream's queue tests split as the stack does (YAML-driven policy
queue tests in kv-router through testing hooks; kernel tests build the
flat profile through PolicyProfile::from_classes); every upstream test in
the touched modules is present (97 kernel queue, 40 policy queue, 21
policy config, 53 config, 15 policy, 52 zmq_wire). AGENTS.md text for the
queue placed in the two owner crates. CODEOWNERS regenerated; four
lockfiles regenerated and --locked clean.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants