Repository navigation
perf(llm): edit the preprocessor's response items in place - #18406
Conversation
The preprocessor's response transform went through `unfold` and `fuse`. Its state holds a buffered usage chunk the size of an item, so `unfold` moved about two items' worth of bytes through a new future on every item. The transform is now a stream that owns its state, and it still drops that state, and so flushes the detokenize metrics, when the response ends. The tool-call policy and the role fix-up after the parsers took each item by value through `map` and returned it. They now edit it through `&mut` with `edit_items`, which uses pin-project-lite (already in the lock through futures and tokio) so that streams that are not `Unpin` still work. On pipeline A (aarch64) these stages' memcpy calls per item fall from 22 to 8. Signed-off-by: Ryan Olson <rolson@nvidia.com>
When no reasoning parser and no tool parser runs, the chat response stream was boxed again while it was already a boxed stream: one allocation per request and one more call per item for each. Pass the boxed stream on as it is. Pipelines A and B, both forms, make 2 allocations and 96 bytes fewer per request; the allocation records are lowered to match. Signed-off-by: Ryan Olson <rolson@nvidia.com>
The preprocessor's response transform boxed the stream it reads. That stream is already a box (`ManyOut`), so each chat and completions request made one allocation for a box around a box. The transform now holds the stream as it is. Per request: one allocation fewer on pipelines A and B, and about 1,200 fewer instructions. memcpy per request and per item do not change. Signed-off-by: Ryan Olson <rolson@nvidia.com>
|
👋 Hi ryanolson! Thank you for contributing to ai-dynamo/dynamo. Just a reminder: The 🚀 |
|
/ok to test db7f20b |
✅ Dynamo PR CI passed — run 42799200259 (attempt 1) on
|
|
No actionable comments were generated in the recent review. WalkthroughThe change adds ChangesResponse stream processing
Estimated code review effort: 3 (Moderate) | ~25 minutes Merge Risk:
|
…is pending The existing tests feed chunks through stream::iter, which is always ready, so the Pending path of the response transform never ran. Signed-off-by: Ryan Olson <rolson@nvidia.com>
Every production input to edit_items is already boxed, so EditItems can require Unpin and poll with poll_next_unpin, as Postprocessed does. This drops the new direct dependency. Tests that pass an async_stream box it. Also make the response transform's comments describe the code as it is: the state drops when the stream ends, as it did with unfold. Remove a commented-out line and a redundant use<> bound. Signed-off-by: Ryan Olson <rolson@nvidia.com>
|
/ok to test e7e915d |
Absorbs six upstream commits, no conflicts, no router-owned path, no Cargo.toml or lockfile change: - fb2d4a0 fix(request-trace): record public Responses tool names and namespaces (ai-dynamo#18464): lib/llm/src/request_trace/{agent_context, integration, mod, types}.rs, lib/llm/src/http/service/openai.rs, lib/llm/src/protocols/openai/responses/tool_names.rs and the agent-tracing.md page, auto-merged and taken whole. The split's re-pointed imports in openai.rs (dynamo_admission::host::QueueRejection) and request_trace/mod.rs (dynamo_worker_selection::host:: SelectionTraceSpec) are intact. - 7c45ddb fix(vllm): report queued requests in FPM heartbeats during long passes (ai-dynamo#18321): Python only, taken whole: components/src/dynamo/common/forward_pass_metrics.py, components/src/dynamo/vllm/instrumented_scheduler.py, the new components/src/dynamo/vllm/tests/test_vllm_instrumented_scheduler.py and the planner-design.md page. - 7424cc8 perf(llm): read Migration's response items in place (ai-dynamo#18404): lib/llm/src/migration.rs auto-merged; the split's one re-point (dynamo_admission::host::AbortCause) is intact. - 2e32ec8 perf(llm): detokenize Backend's response items in place (ai-dynamo#18405): lib/llm/src/backend.rs, taken whole (the split does not touch the file). - 8ddf37c perf(llm): edit the preprocessor's response items in place (ai-dynamo#18406): lib/llm/src/preprocessor.rs auto-merged with the split's sixteen dynamo_kv_hashing::multimodal and dynamo_kv_events::zmq_wire re-points intact; the new lib/llm/src/preprocessor/edit.rs and lib/llm/tests/test_streaming_usage.rs taken whole. - 104e488 fix(multimodal): reject unsupported modalities before decoding (ai-dynamo#18416): Python only, taken whole: components/src/dynamo/vllm/multimodal_utils/request_processor.py and its test test_vllm_request_processor.py. Validation: Tier C slim chain (router crates clippy default and all-features, tests and rustdoc; kv-router clippy, bench check and tests; workspace clippy --all-targets -D warnings; dynamo-llm lib alone, kv_router 517, discovery 283, pool_relay 269; batched host clippy --tests; bindings clippy; cargo hack --each-feature; router layering gates plain and --host; codeowners strict; fmt; machete; four lockfiles --locked), 2,682 passed / 0 failed; cargo check --workspace --all-targets; by name cargo test -p dynamo-llm --test test_streaming_usage 15, --lib migration 67, backend 117, preprocessor 417, request_trace 88, http::service::openai 171, all 0 failed; cargo check -p dynamo-llm --features mm-routing --all-targets; pre-commit on the merged files and over origin/main..HEAD; in the worktree venv test_recv_forward_pass_metrics.py 14 passed, while the two new test_vllm_* files need vLLM, which that venv does not carry. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Resolutions: - Policy classes follow upstream's flat model (ai-dynamo#17723). Our cache-bucket admission (bucket_key: cached_tokens, 538367a) goes with the family/bucket matrix it extended; a profile that still sets default_policy_family, uncached_isl_buckets, policy_family or cache_bucket now fails to load. - Upstream's PendingSinkGuard replaces our SinkEntryGuard (cb8ba31) for the same pipeline sink leak. - Worker-scoped shared-cache hits: the worker id moves from WorkerCacheData to the borrowed WorkerCacheInput view, so CandidateData keeps upstream's 128 bytes. - The shared-cache credit on a queued request's overlap refresh moves into upstream's refresh arm. - Upstream's accounting attribution (ai-dynamo#17895) and our engine_block_hashes both ride on find_best_match, the scheduling request and the booked sequence event; both are kept, and the booked lora_name goes through accounting_name. - Upstream's AttemptLease replaces the LoraLoadGuard the request guard held; our cache_prediction stays beside it. - chat_completions tests: upstream's passthrough_extra_fields test is added; our over-max stop test keeps its truncation body and name. - preprocessor: upstream's in-place response stream (ai-dynamo#18406) moves the per-chunk code onto self; our per-decode-worker logprob histogram (3e7a796) moves with it. - Our shared-cache tests: the load-aware one expects upstream's --load-aware deprecation warning (ai-dynamo#16620), and the planner's router-field coverage test (ai-dynamo#17483) sets shared_cache_hit_weight as it does shared_cache_multiplier, since serde skips both at their defaults. - Upstream's new tests take engine_block_hashes: None, model_eos: Vec::new() and the SharedCacheQuery form of check_blocks.
Resolutions: - Policy classes follow upstream's flat model (ai-dynamo#17723). Our cache-bucket admission (bucket_key: cached_tokens, 538367a) goes with the family/bucket matrix it extended; a profile that still sets default_policy_family, uncached_isl_buckets, policy_family or cache_bucket now fails to load. - Upstream's PendingSinkGuard replaces our SinkEntryGuard (cb8ba31) for the same pipeline sink leak. - Worker-scoped shared-cache hits: the worker id moves from WorkerCacheData to the borrowed WorkerCacheInput view, so CandidateData keeps upstream's 128 bytes. - The shared-cache credit on a queued request's overlap refresh moves into upstream's refresh arm. - Upstream's accounting attribution (ai-dynamo#17895) and our engine_block_hashes both ride on find_best_match, the scheduling request and the booked sequence event; both are kept, and the booked lora_name goes through accounting_name. - Upstream's AttemptLease replaces the LoraLoadGuard the request guard held; our cache_prediction stays beside it. - chat_completions tests: upstream's passthrough_extra_fields test is added; our over-max stop test keeps its truncation body and name. - preprocessor: upstream's in-place response stream (ai-dynamo#18406) moves the per-chunk code onto self; our per-decode-worker logprob histogram (3e7a796) moves with it. - Our shared-cache tests: the load-aware one expects upstream's --load-aware deprecation warning (ai-dynamo#16620), and the planner's router-field coverage test (ai-dynamo#17483) sets shared_cache_hit_weight as it does shared_cache_multiplier, since serde skips both at their defaults. - Upstream's new tests take engine_block_hashes: None and the SharedCacheQuery form of check_blocks. - Upstream's router split (ai-dynamo#18055) moves our kv-router changes into the new crates: per-worker SharedCacheHits into dynamo-worker-selection, SharedCacheQuery and observe_stored onto the scheduler's SharedKvCache, the mooncake-store cache type and its default multiplier onto SharedCacheType, and engine_block_hashes onto the slot tracker's wire event. - shared_cache_hit_weight reaches the kernel as SelectionConfig's shared_cache_hit_weight and CacheHitWeights::shared, which the selection run and the overlap refresher both read. - Our lib/llm code imports these from the stage crates' host modules, not the deprecated kv-router paths; the scheduler's host module also exports SharedCacheQuery and the slot tracker's exports ReplicaPromptObserver.
Resolutions: - Policy classes follow upstream's flat model (ai-dynamo#17723). Our cache-bucket admission (bucket_key: cached_tokens, 538367a) goes with the family/bucket matrix it extended; a profile that still sets default_policy_family, uncached_isl_buckets, policy_family or cache_bucket now fails to load. - Upstream's PendingSinkGuard replaces our SinkEntryGuard (cb8ba31) for the same pipeline sink leak. - Worker-scoped shared-cache hits: the worker id moves from WorkerCacheData to the borrowed WorkerCacheInput view, so CandidateData keeps upstream's 128 bytes. - The shared-cache credit on a queued request's overlap refresh moves into upstream's refresh arm. - Upstream's accounting attribution (ai-dynamo#17895) and our engine_block_hashes both ride on find_best_match, the scheduling request and the booked sequence event; both are kept, and the booked lora_name goes through accounting_name. - Upstream's AttemptLease replaces the LoraLoadGuard the request guard held; our cache_prediction stays beside it. - chat_completions tests: upstream's passthrough_extra_fields test is added; our over-max stop test keeps its truncation body and name. - preprocessor: upstream's in-place response stream (ai-dynamo#18406) moves the per-chunk code onto self; our per-decode-worker logprob histogram (3e7a796) moves with it. - Our shared-cache tests: the load-aware one expects upstream's --load-aware deprecation warning (ai-dynamo#16620), and the planner's router-field coverage test (ai-dynamo#17483) sets shared_cache_hit_weight as it does shared_cache_multiplier, since serde skips both at their defaults. - Upstream's new tests take engine_block_hashes: None and the SharedCacheQuery form of check_blocks. - Upstream's router split (ai-dynamo#18055) moves our kv-router changes into the new crates: per-worker SharedCacheHits into dynamo-worker-selection, SharedCacheQuery and observe_stored onto the scheduler's SharedKvCache, the mooncake-store cache type and its default multiplier onto SharedCacheType, and engine_block_hashes onto the slot tracker's wire event. - shared_cache_hit_weight reaches the kernel as SelectionConfig's shared_cache_hit_weight and CacheHitWeights::shared, which the selection run and the overlap refresher both read. - Our lib/llm code imports these from the stage crates' host modules, not the deprecated kv-router paths; the scheduler's host module also exports SharedCacheQuery and the slot tracker's exports ReplicaPromptObserver. - Upstream's kvcr policy (ai-dynamo#17647) takes KV-copy credit out of the default selector and drops its per-request shared_cache_multiplier override; the default multiplier still comes from SharedCacheType::default_multiplier, so mooncake-store keeps its 0.5. - Upstream's live per-rank resident block counts (ai-dynamo#16078) and our tracked block count and worker listing (235a17b) share the indexer actor. - Upstream's needs_uncached_prefill (ai-dynamo#17441) rides on find_best_match beside engine_block_hashes. - The embedded partition config carries upstream's hint policies (ai-dynamo#17359) beside our peer prompt observer. - Upstream's prepare_inputs gate decides whether a shared-cache client is created, mooncake-store included.
Resolutions: - Policy classes follow upstream's flat model (ai-dynamo#17723). Our cache-bucket admission (bucket_key: cached_tokens, 538367a) goes with the family/bucket matrix it extended; a profile that still sets default_policy_family, uncached_isl_buckets, policy_family or cache_bucket now fails to load. - Upstream's PendingSinkGuard replaces our SinkEntryGuard (cb8ba31) for the same pipeline sink leak. - Worker-scoped shared-cache hits: the worker id moves from WorkerCacheData to the borrowed WorkerCacheInput view, so CandidateData keeps upstream's 128 bytes. - The shared-cache credit on a queued request's overlap refresh moves into upstream's refresh arm. - Upstream's accounting attribution (ai-dynamo#17895) and our engine_block_hashes both ride on find_best_match, the scheduling request and the booked sequence event; both are kept, and the booked lora_name goes through accounting_name. - Upstream's AttemptLease replaces the LoraLoadGuard the request guard held; our cache_prediction stays beside it. - chat_completions tests: upstream's passthrough_extra_fields test is added; our over-max stop test keeps its truncation body and name. - preprocessor: upstream's in-place response stream (ai-dynamo#18406) moves the per-chunk code onto self; our per-decode-worker logprob histogram (3e7a796) moves with it. - Our shared-cache tests: the load-aware one expects upstream's --load-aware deprecation warning (ai-dynamo#16620), and the planner's router-field coverage test (ai-dynamo#17483) sets shared_cache_hit_weight as it does shared_cache_multiplier, since serde skips both at their defaults. - Upstream's new tests take engine_block_hashes: None and the SharedCacheQuery form of check_blocks. - Upstream's router split (ai-dynamo#18055) moves our kv-router changes into the new crates: per-worker SharedCacheHits into dynamo-worker-selection, SharedCacheQuery and observe_stored onto the scheduler's SharedKvCache, the mooncake-store cache type and its default multiplier onto SharedCacheType, and engine_block_hashes onto the slot tracker's wire event. - shared_cache_hit_weight reaches the kernel as SelectionConfig's shared_cache_hit_weight and CacheHitWeights::shared, which the selection run and the overlap refresher both read. - Our lib/llm code imports these from the stage crates' host modules, not the deprecated kv-router paths; the scheduler's host module also exports SharedCacheQuery and the slot tracker's exports ReplicaPromptObserver. - Upstream's kvcr policy (ai-dynamo#17647) takes KV-copy credit out of the default selector and drops its per-request shared_cache_multiplier override; the default multiplier still comes from SharedCacheType::default_multiplier, so mooncake-store keeps its 0.5. - Upstream's live per-rank resident block counts (ai-dynamo#16078) and our tracked block count and worker listing (235a17b) share the indexer actor. - Upstream's needs_uncached_prefill (ai-dynamo#17441) rides on find_best_match beside engine_block_hashes. - The embedded partition config carries upstream's hint policies (ai-dynamo#17359) beside our peer prompt observer. - Upstream's prepare_inputs gate decides whether a shared-cache client is created, mooncake-store included.
Overview
Three per-item fixes in the chat and completions preprocessor. The response transform is a stream that owns its state, in place of
unfold. The role and tool-call edits change items through&mut. And the parsed chat stream and the transform's input are no longer boxed twice.Key Changes
Validation
cargo test -p dynamo-llm(all targets) and clippy-D warningspass on this branch (aarch64).Pipelines in the measurements: A is in-process chat (preprocessor, Backend, engine); B is routed chat (preprocessor, Migration, Backend, encoder and prefill routers); C is a routed preprocessed request (Migration and the routers). Per streamed item means callgrind's (16-item count − 1-item count) / 15.
Issues and Ongoing Work
Pull Request Checklist
/pr-checklistskill.Summary by CodeRabbit
Closes STRGT-1108