Repository navigation
feat(bindings): mark Python response encoding with an NVTX range - #17707
Conversation
encode_python_item runs inside the response stream's next(), before worker.egress.chunk opens, so its GIL-held parse and serialization showed up as engine time in the gap between chunk ranges. Wrap it in pybridge.encode_response. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Kyle McGill <kmcgill@nvidia.com>
|
No actionable comments were generated in the recent review.
|
| Layer / File(s) | Summary |
|---|---|
Mark response encoding lib/bindings/python/rust/python_payload.rs |
The code imports dynamo_nvtx_range and applies the pybridge.encode_response range around response encoding in the blocking task. Encoding behavior and error handling remain unchanged. |
Estimated code review effort: 1 (Trivial) | ~2 minutes
Merge Risk: 
Minimal · up to fa615
-
The instrumentation is isolated and does not change response encoding behavior; the PR is mergeable.
Pre-merge checks: 
5
-
Full details: Title check
-
Explanation
The title "feat(bindings): mark Python response encoding with an NVTX range" matches the single change in the changeset. The diff imports dynamo_nvtx_range and wraps encode_python_response in a pybridge.encode_response NVTX range inside the spawn_blocking closure of encode_python_item. The title is concise, specific, and names the primary change and its scope.
Full details: Description check
-
Explanation
The description follows the repository template. It has Overview, Key Changes, Validation, and Issues and Ongoing Work sections, plus the Pull Request Checklist block. The Overview explains the motivation: encoding runs before worker.egress.chunk opens, so its time was misattributed. Key Changes names the file and the added line. Validation gives a cargo check result, a captured profile with a median of about 22 µs, and overhead figures. Issues lists the related PRs and DIS-3260. The template's Closes
#XXXXline is replaced by a Relates to DIS-3260 reference, and the checklist item remains unchecked. These are minor gaps; the description is mostly complete.
Full details: Docstring Coverage
-
Explanation
No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 1 files.
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Comment @coderabbitai help to get the list of available commands.
✅ Dynamo PR CI passed — run 41123793555 (attempt 1) on
|
Second upstream merge for the split v2 PR: 36 commits past the first merge's base 182039d, 17 conflicted paths, each upstream change placed in the crate that owns the code at the stack head, upstream behavior whole, the stack's moves intact, no shims. - ai-dynamo#17723 flatten policy classes: flat PolicyProfile (from_classes, UnknownPolicyClassError) in lib/router/admission profile.rs; the YAML half (default_policy_class, removed keys rejected) in lib/router/kv-router policy_config.rs; KvSchedulerError::UnknownPolicyClass via From in the kernel; 400/InvalidArgument in lib/llm (UnknownPolicyClassRefusal) and ext-proc. - ai-dynamo#17724 EDF ordering: RouterQueuePolicy::Edf in admission profile.rs; due-time order, DueKey and class budgets in admission queue.rs; the restored score keys (RouterSchedulingPolicy::score_policy) as the kernel's policy.rs (ordered-float back in dynamo-scheduler). - ai-dynamo#17725 controlled delay: head expiry and with_class_budget_expiry in admission queue.rs; ClassTimers, due-time checks, DRR charged on admission (extract_next/settle) and the retained overlap refresh in the kernel's queue.rs and overlap_refresh.rs; metrics in lib/llm. - ai-dynamo#17437 image-aware KV routing for TRT-LLM sidecars: zmq_wire/trtllm.rs in lib/kv-events; the sidecar depends on kv-events, kv-hashing, multimodal and protocol instead of kv-router. - ai-dynamo#14775 prefill cancellation, ai-dynamo#17716/ai-dynamo#18077 pool relay, ai-dynamo#17707 NVTX range, ai-dynamo#18102 LiveEngine buffering, ai-dynamo#17887 shared replay clock: lib/llm, bindings, mocker, bench, auto-merged with import re-points. Tests: upstream's queue tests split as the stack does (YAML-driven policy queue tests in kv-router through testing hooks; kernel tests build the flat profile through PolicyProfile::from_classes); every upstream test in the touched modules is present (97 kernel queue, 40 policy queue, 21 policy config, 53 config, 15 policy, 52 zmq_wire). AGENTS.md text for the queue placed in the two owner crates. CODEOWNERS regenerated; four lockfiles regenerated and --locked clean. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Overview
Wraps Python response encoding (
encode_python_item) in apybridge.encode_responseNVTX range. Without it, an Nsight capture of a Python worker misattributes this time. The encoding runs inside the response stream'snext(), beforeworker.egress.chunkopens. Its GIL-held parse and serialization therefore lands in the gap between chunk ranges and reads as engine time.tanmayv25raised this on #17334 (review thread), and that PR deferred it to this follow-up.Key Changes
lib/bindings/python/rust/python_payload.rs: addslet _nvtx = dynamo_nvtx_range!("pybridge.encode_response");inside thespawn_blockingclosure inencode_python_item. The range coversencode_python_responseand nothing else.Validation
cargo check --features nvtxpasses inlib/bindings/python.DYN_NVTX=1undernsys -t nvtx,cuda. The range fires once per chunk, with a median of about 22 µs. It ends about 13 µs beforeworker.egress.chunkopens. The last 150 µs before a chunk:DYN_NVTX=1. Undernsysit costs about 1.3–1.7 µs, against the 22 µs it measures.Issues and Ongoing Work
dynamo/vllm/handlers.py), which these markers do not split further.Pull Request Checklist
/pr-checklistskill.🤖 Generated with Claude Code
Summary by CodeRabbit