Repository navigation
feat(lora): integrate safe runtime routing - #17895
Conversation
✅ Dynamo PR CI passed — run 42763466643 (attempt 2) on
|
WalkthroughThe changes add runtime LoRA metadata and candidate-policy routing. Routing hosts filter and revalidate eligible workers. Prefill and encoder stages check runtime LoRA requests before dispatch. Retry behavior and accounting use request routing requirements. ChangesRuntime LoRA Routing
Estimated code review effort: 4 (Complex) | ~45 minutes Merge Risk:
|
023ecb1 to
67b0dd0
Compare
tedzhouhk
left a comment
There was a problem hiding this comment.
Reviewed runtime identity validation, worker eligibility rechecks, pins and DP rank, replay gating, and cleanup at 67b0dd0. Local validation passed: 181 dynamo-lora unit tests and 21 focused LLM regressions. GPU-backed resolver/load E2E was not run.
67b0dd0 to
98f7064
Compare
c98be23 to
0f8497a
Compare
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
0f8497a to
0ffaae6
Compare
ishandhanani
left a comment
There was a problem hiding this comment.
Post-merge note on the plugin-facing surface. The feature and its placement look right: the kernel gains one generic enum and the LoRA logic stays in lib/llm and lib/lora. One API point below, raised after the fact because it affects out-of-tree policy crates.
| pub block_hashes: Option<Vec<LocalBlockHash>>, | ||
| pub isl_tokens: usize, | ||
| pub lora_name: Option<String>, | ||
| pub accounting_attribution: AccountingAttribution, |
There was a problem hiding this comment.
Adding a pub field to ScheduleRequest (and to SelectionOperation in operation.rs) breaks every out-of-tree constructor that builds these by struct literal, even though the field has a Default. This PR had to patch both in-repo example policy crates with accounting_attribution: Default::default() for exactly that reason, and external policy crates that construct ScheduleRequest in their tests will hit the same compile error on upgrade.
Suggestion for a follow-up: keep additive fields non-breaking for plugin authors, either by giving ScheduleRequest a constructor or builder that plugin code is told to use, or by making newly added knobs pub(crate) with a public setter on the request. Any of those lets the next accounting or placement field land without touching every policy crate.
|
|
||
| fn request(affinity_target: Option<WorkerAffinityTarget>) -> SchedulingRequest { | ||
| SchedulingRequest { | ||
| accounting_attribution: Default::default(), |
There was a problem hiding this comment.
This line is the symptom: an example policy had to change because a kernel request type gained a public field. External policy crates with the same literal need the same edit.
- ai-dynamo#17895 feat(lora): integrate safe runtime routing: the kernel's new AccountingAttribution enum and its threading through ScheduleRequest, SchedulingRequest, SelectionOperation, PendingSelection and ReservationBooking land whole in dynamo-scheduler (lib/router/scheduler/src/{types,host,local,policy,queue}.rs, selection/core/{operation,reservations,run,tests}.rs, selection/pending.rs, selector/{mod,reference}.rs); dynamo_kv_router::plugins::prelude exports AccountingAttribution and compat.rs keeps the 1.7 path dynamo_kv_router::scheduling::AccountingAttribution; lib/llm (routing_policy.rs, lora_routing.rs, kv_router.rs, routing_host/{hosted,kv_selection}.rs) and lib/lora (tests/accounting.rs) keep upstream's content with owner-crate re-points to dynamo_scheduler::host, dynamo_worker_selection::host, dynamo_slot_tracker::host and dynamo_router_protocol::worker plus the hosted picker's KvSchedulerError adaptation; the example policy crates, builtin policies, mocker and the kv-router worker_selection bench take the new field via Default::default(). Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
ishandhanani
left a comment
There was a problem hiding this comment.
Requesting changes on the AccountingAttribution design, to land as a follow-up since this is merged.
The host decides whether a request's lora_name counts toward per-adapter demand (one if in lora_routing::execution_requirements), but instead of passing the resolved answer it passes a two-variant flag that the scheduler kernel threads through ScheduleRequest, SchedulingRequest, SelectionOperation, PendingSelection, and ReservationBooking, then interprets at the two booking sites by masking lora_name. The kernel has no information at those sites that the host lacked when it set the flag, so the enum, its impl, and the five field threads are carrying a decision that was already made.
Ask for the follow-up:
- Delete
AccountingAttributionand the field from all five structs. - Have
lib/llmpass the resolved accounting identity instead: either alora_namethat is alreadyNoneon the paths that must not be charged, or a separateaccounting_lora_name: Option<String>if the routing identity still needs to reach selection. The kernel books what it is given. - Whatever field shape survives, do not add it as a new
pubfield onScheduleRequest. Both in-repo example policy crates neededaccounting_attribution: Default::default()in this PR, and every out-of-tree policy crate that builds the request by struct literal breaks the same way on upgrade. Land request-side additions behind a constructor or builder.
Inline notes point at each piece.
|
|
||
| /// Whether routing identity contributes to per-adapter demand accounting. | ||
| #[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] | ||
| pub enum AccountingAttribution { |
There was a problem hiding this comment.
This is a bool spelled as an enum, and its only consumer is accounting_name, which maps it straight to Option<String>. The one place it is set (lora_routing::execution_requirements) already knows the resolved value. Pass that value and this type, the impl, and the field threads through ScheduleRequest, SchedulingRequest, SelectionOperation, PendingSelection, and ReservationBooking all go away.
| pub block_hashes: Option<Vec<LocalBlockHash>>, | ||
| pub isl_tokens: usize, | ||
| pub lora_name: Option<String>, | ||
| pub accounting_attribution: AccountingAttribution, |
There was a problem hiding this comment.
A pub field on the plugin-facing request for a value no policy reads or sets. Every external policy crate that constructs ScheduleRequest by struct literal stops compiling on upgrade; this PR had to patch both in-repo examples for exactly that. If a new request-side knob is unavoidable it should land behind a constructor or builder, not as another public field.
| prefill_load_hint, | ||
| worker, | ||
| lora_name, | ||
| lora_name: accounting_attribution.accounting_name(lora_name), |
There was a problem hiding this comment.
The booking decision is made here by masking lora_name with the flag, but nothing is known at this point that was not known in lib/llm when the flag was set. Resolve it there and book the name you are handed.
| lora_name: request.lora_name.take(), | ||
| lora_name: request | ||
| .accounting_attribution | ||
| .accounting_name(request.lora_name.take()), |
There was a problem hiding this comment.
Second copy of the same booking-time mask. With the host passing the resolved identity, this line and the one in reservations.rs become plain lora_name again.
| let policy = policy(request); | ||
| ExecutionRequirements { | ||
| allows_replay: policy.allows_replay(), | ||
| policy_required: policy.requires_candidate_policy(), |
There was a problem hiding this comment.
This is the only decision site: tracks_legacy_load() means charge lora_name, otherwise charge nothing. Emit the resolved identity here (None, or the name) instead of a flag the kernel has to interpret five structs later.
Resolutions: - Policy classes follow upstream's flat model (ai-dynamo#17723). Our cache-bucket admission (bucket_key: cached_tokens, 538367a) goes with the family/bucket matrix it extended; a profile that still sets default_policy_family, uncached_isl_buckets, policy_family or cache_bucket now fails to load. - Upstream's PendingSinkGuard replaces our SinkEntryGuard (cb8ba31) for the same pipeline sink leak. - Worker-scoped shared-cache hits: the worker id moves from WorkerCacheData to the borrowed WorkerCacheInput view, so CandidateData keeps upstream's 128 bytes. - The shared-cache credit on a queued request's overlap refresh moves into upstream's refresh arm. - Upstream's accounting attribution (ai-dynamo#17895) and our engine_block_hashes both ride on find_best_match, the scheduling request and the booked sequence event; both are kept, and the booked lora_name goes through accounting_name. - Upstream's AttemptLease replaces the LoraLoadGuard the request guard held; our cache_prediction stays beside it. - chat_completions tests: upstream's passthrough_extra_fields test is added; our over-max stop test keeps its truncation body and name. - Upstream's new tests take engine_block_hashes: None and the SharedCacheQuery form of check_blocks. Signed-off-by: Mert Saglam <mert@deepinfra.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dp-rank Resolve conflicts with the LoRA candidate-policy refactor (ai-dynamo#17889, ai-dynamo#17895) in routing_host.rs and hosted.rs: take main's candidate_policy field and the removal of select_lora_target, and keep the hard affinity DP rank helper. Hold::session_id moved with the session-affinity crate split (ai-dynamo#18055). Signed-off-by: pyextreme <45779665+PyExtreme@users.noreply.github.com>
…e-decode-selection Resolve the conflicts with ai-dynamo#18055 and move kv_capacity_aware, its lib.rs test and its benchmark to the new crate layout: import from dynamo_kv_router::plugins::prelude, build policies from SelectionConfig. Add accounting_attribution, the SchedulingRequest field from ai-dynamo#17895, to the kv_capacity_aware test helper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Yuehua Liu <liujohn@nvidia.com>
Resolutions: - Policy classes follow upstream's flat model (ai-dynamo#17723). Our cache-bucket admission (bucket_key: cached_tokens, 538367a) goes with the family/bucket matrix it extended; a profile that still sets default_policy_family, uncached_isl_buckets, policy_family or cache_bucket now fails to load. - Upstream's PendingSinkGuard replaces our SinkEntryGuard (cb8ba31) for the same pipeline sink leak. - Worker-scoped shared-cache hits: the worker id moves from WorkerCacheData to the borrowed WorkerCacheInput view, so CandidateData keeps upstream's 128 bytes. - The shared-cache credit on a queued request's overlap refresh moves into upstream's refresh arm. - Upstream's accounting attribution (ai-dynamo#17895) and our engine_block_hashes both ride on find_best_match, the scheduling request and the booked sequence event; both are kept, and the booked lora_name goes through accounting_name. - Upstream's AttemptLease replaces the LoraLoadGuard the request guard held; our cache_prediction stays beside it. - chat_completions tests: upstream's passthrough_extra_fields test is added; our over-max stop test keeps its truncation body and name. - preprocessor: upstream's in-place response stream (ai-dynamo#18406) moves the per-chunk code onto self; our per-decode-worker logprob histogram (3e7a796) moves with it. - Our shared-cache tests: the load-aware one expects upstream's --load-aware deprecation warning (ai-dynamo#16620), and the planner's router-field coverage test (ai-dynamo#17483) sets shared_cache_hit_weight as it does shared_cache_multiplier, since serde skips both at their defaults. - Upstream's new tests take engine_block_hashes: None, model_eos: Vec::new() and the SharedCacheQuery form of check_blocks.
Resolutions: - Policy classes follow upstream's flat model (ai-dynamo#17723). Our cache-bucket admission (bucket_key: cached_tokens, 538367a) goes with the family/bucket matrix it extended; a profile that still sets default_policy_family, uncached_isl_buckets, policy_family or cache_bucket now fails to load. - Upstream's PendingSinkGuard replaces our SinkEntryGuard (cb8ba31) for the same pipeline sink leak. - Worker-scoped shared-cache hits: the worker id moves from WorkerCacheData to the borrowed WorkerCacheInput view, so CandidateData keeps upstream's 128 bytes. - The shared-cache credit on a queued request's overlap refresh moves into upstream's refresh arm. - Upstream's accounting attribution (ai-dynamo#17895) and our engine_block_hashes both ride on find_best_match, the scheduling request and the booked sequence event; both are kept, and the booked lora_name goes through accounting_name. - Upstream's AttemptLease replaces the LoraLoadGuard the request guard held; our cache_prediction stays beside it. - chat_completions tests: upstream's passthrough_extra_fields test is added; our over-max stop test keeps its truncation body and name. - preprocessor: upstream's in-place response stream (ai-dynamo#18406) moves the per-chunk code onto self; our per-decode-worker logprob histogram (3e7a796) moves with it. - Our shared-cache tests: the load-aware one expects upstream's --load-aware deprecation warning (ai-dynamo#16620), and the planner's router-field coverage test (ai-dynamo#17483) sets shared_cache_hit_weight as it does shared_cache_multiplier, since serde skips both at their defaults. - Upstream's new tests take engine_block_hashes: None and the SharedCacheQuery form of check_blocks. - Upstream's router split (ai-dynamo#18055) moves our kv-router changes into the new crates: per-worker SharedCacheHits into dynamo-worker-selection, SharedCacheQuery and observe_stored onto the scheduler's SharedKvCache, the mooncake-store cache type and its default multiplier onto SharedCacheType, and engine_block_hashes onto the slot tracker's wire event. - shared_cache_hit_weight reaches the kernel as SelectionConfig's shared_cache_hit_weight and CacheHitWeights::shared, which the selection run and the overlap refresher both read. - Our lib/llm code imports these from the stage crates' host modules, not the deprecated kv-router paths; the scheduler's host module also exports SharedCacheQuery and the slot tracker's exports ReplicaPromptObserver.
Resolutions: - Policy classes follow upstream's flat model (ai-dynamo#17723). Our cache-bucket admission (bucket_key: cached_tokens, 538367a) goes with the family/bucket matrix it extended; a profile that still sets default_policy_family, uncached_isl_buckets, policy_family or cache_bucket now fails to load. - Upstream's PendingSinkGuard replaces our SinkEntryGuard (cb8ba31) for the same pipeline sink leak. - Worker-scoped shared-cache hits: the worker id moves from WorkerCacheData to the borrowed WorkerCacheInput view, so CandidateData keeps upstream's 128 bytes. - The shared-cache credit on a queued request's overlap refresh moves into upstream's refresh arm. - Upstream's accounting attribution (ai-dynamo#17895) and our engine_block_hashes both ride on find_best_match, the scheduling request and the booked sequence event; both are kept, and the booked lora_name goes through accounting_name. - Upstream's AttemptLease replaces the LoraLoadGuard the request guard held; our cache_prediction stays beside it. - chat_completions tests: upstream's passthrough_extra_fields test is added; our over-max stop test keeps its truncation body and name. - preprocessor: upstream's in-place response stream (ai-dynamo#18406) moves the per-chunk code onto self; our per-decode-worker logprob histogram (3e7a796) moves with it. - Our shared-cache tests: the load-aware one expects upstream's --load-aware deprecation warning (ai-dynamo#16620), and the planner's router-field coverage test (ai-dynamo#17483) sets shared_cache_hit_weight as it does shared_cache_multiplier, since serde skips both at their defaults. - Upstream's new tests take engine_block_hashes: None and the SharedCacheQuery form of check_blocks. - Upstream's router split (ai-dynamo#18055) moves our kv-router changes into the new crates: per-worker SharedCacheHits into dynamo-worker-selection, SharedCacheQuery and observe_stored onto the scheduler's SharedKvCache, the mooncake-store cache type and its default multiplier onto SharedCacheType, and engine_block_hashes onto the slot tracker's wire event. - shared_cache_hit_weight reaches the kernel as SelectionConfig's shared_cache_hit_weight and CacheHitWeights::shared, which the selection run and the overlap refresher both read. - Our lib/llm code imports these from the stage crates' host modules, not the deprecated kv-router paths; the scheduler's host module also exports SharedCacheQuery and the slot tracker's exports ReplicaPromptObserver. - Upstream's kvcr policy (ai-dynamo#17647) takes KV-copy credit out of the default selector and drops its per-request shared_cache_multiplier override; the default multiplier still comes from SharedCacheType::default_multiplier, so mooncake-store keeps its 0.5. - Upstream's live per-rank resident block counts (ai-dynamo#16078) and our tracked block count and worker listing (235a17b) share the indexer actor. - Upstream's needs_uncached_prefill (ai-dynamo#17441) rides on find_best_match beside engine_block_hashes. - The embedded partition config carries upstream's hint policies (ai-dynamo#17359) beside our peer prompt observer. - Upstream's prepare_inputs gate decides whether a shared-cache client is created, mooncake-store included.
Resolutions: - Policy classes follow upstream's flat model (ai-dynamo#17723). Our cache-bucket admission (bucket_key: cached_tokens, 538367a) goes with the family/bucket matrix it extended; a profile that still sets default_policy_family, uncached_isl_buckets, policy_family or cache_bucket now fails to load. - Upstream's PendingSinkGuard replaces our SinkEntryGuard (cb8ba31) for the same pipeline sink leak. - Worker-scoped shared-cache hits: the worker id moves from WorkerCacheData to the borrowed WorkerCacheInput view, so CandidateData keeps upstream's 128 bytes. - The shared-cache credit on a queued request's overlap refresh moves into upstream's refresh arm. - Upstream's accounting attribution (ai-dynamo#17895) and our engine_block_hashes both ride on find_best_match, the scheduling request and the booked sequence event; both are kept, and the booked lora_name goes through accounting_name. - Upstream's AttemptLease replaces the LoraLoadGuard the request guard held; our cache_prediction stays beside it. - chat_completions tests: upstream's passthrough_extra_fields test is added; our over-max stop test keeps its truncation body and name. - preprocessor: upstream's in-place response stream (ai-dynamo#18406) moves the per-chunk code onto self; our per-decode-worker logprob histogram (3e7a796) moves with it. - Our shared-cache tests: the load-aware one expects upstream's --load-aware deprecation warning (ai-dynamo#16620), and the planner's router-field coverage test (ai-dynamo#17483) sets shared_cache_hit_weight as it does shared_cache_multiplier, since serde skips both at their defaults. - Upstream's new tests take engine_block_hashes: None and the SharedCacheQuery form of check_blocks. - Upstream's router split (ai-dynamo#18055) moves our kv-router changes into the new crates: per-worker SharedCacheHits into dynamo-worker-selection, SharedCacheQuery and observe_stored onto the scheduler's SharedKvCache, the mooncake-store cache type and its default multiplier onto SharedCacheType, and engine_block_hashes onto the slot tracker's wire event. - shared_cache_hit_weight reaches the kernel as SelectionConfig's shared_cache_hit_weight and CacheHitWeights::shared, which the selection run and the overlap refresher both read. - Our lib/llm code imports these from the stage crates' host modules, not the deprecated kv-router paths; the scheduler's host module also exports SharedCacheQuery and the slot tracker's exports ReplicaPromptObserver. - Upstream's kvcr policy (ai-dynamo#17647) takes KV-copy credit out of the default selector and drops its per-request shared_cache_multiplier override; the default multiplier still comes from SharedCacheType::default_multiplier, so mooncake-store keeps its 0.5. - Upstream's live per-rank resident block counts (ai-dynamo#16078) and our tracked block count and worker listing (235a17b) share the indexer actor. - Upstream's needs_uncached_prefill (ai-dynamo#17441) rides on find_best_match beside engine_block_hashes. - The embedded partition config carries upstream's hint policies (ai-dynamo#17359) beside our peer prompt observer. - Upstream's prepare_inputs gate decides whether a shared-cache client is created, mooncake-store included.
Summary
accounting_nameand cover add/free attribution and load cleanup.Part of #14547, #15018, and DIS-2937. Based on #17891.
Validation
24c7f0c38ethrough refactor(router): add generic candidate policy hooks #17889 and feat(lora): add runtime placement policy #17891, preserving adjacent upstream changes.cargo test --locked -p dynamo-llm --no-default-features --lib lora_: 39 passed.standalone-selection,runtime-protocols): 107 passed, including explicit-reservation accounting and opt-out/replay coverage.cargo clippy --locked -p dynamo-llm --no-default-features --tests -- -D warnings: passed.cargo clippy --locked -p dynamo-kv-router --features standalone-selection,runtime-protocols --tests -- -D warnings: passed.