Skip to content

feat(lora): integrate safe runtime routing - #17895

Merged
biswapanda merged 2 commits into
mainfrom
bis/dis-2937-lora-routing-final
Nov 10, 2026
Merged

biswapanda merged 2 commits into
mainfrom
bis/dis-2937-lora-routing-final

Conversation

@biswapanda

@biswapanda biswapanda commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Integrate runtime LoRA placement with hosted and KV-aware routing while preserving legacy adapter behavior.
  • Keep hosted selection/revalidation and KV restriction/admission/revalidation in separate traits and concrete policies. KV hosts do not carry an unused hosted policy.
  • Carry opaque adapter identity without interpreting downloader URIs in Rust; fail closed for incomplete metadata and revalidate current capability, eligibility, caller constraints, and phase-specific pins before dispatch.
  • Introduce accounting attribution here, at its first production opt-out consumer. Preserve routing/cache identity while excluding runtime adapters from legacy demand accounting. Explicit legacy reservations retain routing-name accounting; the public reservation API carries no runtime identity. Rename the helper to accounting_name and cover add/free attribution and load cleanup.

Part of #14547, #15018, and DIS-2937. Based on #17891.

Validation

  • Restacked onto main 24c7f0c38e through refactor(router): add generic candidate policy hooks #17889 and feat(lora): add runtime placement policy #17891, preserving adjacent upstream changes.
  • cargo test --locked -p dynamo-llm --no-default-features --lib lora_: 39 passed.
  • RoutingHost suite: 82 passed, including queued worker revalidation and pinned fallback.
  • Diff-scoped pre-commit hooks, formatting, and whitespace checks passed.
  • KV selection-core suite (standalone-selection,runtime-protocols): 107 passed, including explicit-reservation accounting and opt-out/replay coverage.
  • cargo clippy --locked -p dynamo-llm --no-default-features --tests -- -D warnings: passed.
  • cargo clippy --locked -p dynamo-kv-router --features standalone-selection,runtime-protocols --tests -- -D warnings: passed.
  • GPU live serving has not been rerun for this revision.

@github-actions github-actions Bot added the feat label Nov 3, 2026
@github-actions github-actions Bot added frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` router Relates to routing, KV-aware routing, etc. labels Nov 3, 2026
@biswapanda
biswapanda added this pull request to stack #17906 November 3, 2026 02:26
@devin-ai-integration

devin-ai-integration Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

✅ Dynamo PR CI passed — run 42763466643 (attempt 2) on 0f8497a690

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Posted automatically by Devin for run 42763466643. Updated on every full-CI run of this PR.

biswapanda

This comment was marked as outdated.

@biswapanda
biswapanda marked this pull request as ready for review November 3, 2026 17:24
@biswapanda
biswapanda requested review from a team as code owners November 3, 2026 17:24
@coderabbitai

coderabbitai Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Walkthrough

The changes add runtime LoRA metadata and candidate-policy routing. Routing hosts filter and revalidate eligible workers. Prefill and encoder stages check runtime LoRA requests before dispatch. Retry behavior and accounting use request routing requirements.

Changes

Runtime LoRA Routing

Layer / File(s) Summary
Runtime metadata and policy contract
lib/llm/src/protocols/common/preprocessor.rs, lib/llm/src/preprocessor.rs, lib/llm/src/routing_policy.rs, lib/llm/src/lora_routing.rs, lib/lora/src/runtime.rs, lib/lora/src/policy.rs
Routing hints add optional runtime LoRA identity fields. Runtime worker checks validate capability and identity. Routing policy adds execution requirements and worker-eligibility checks.
Candidate-policy construction and selection
lib/llm/src/entrypoint/input/common.rs, lib/llm/src/discovery/model_manager.rs, lib/llm/src/kv_router.rs, lib/llm/src/kv_router/routing_host/builder.rs, lib/llm/src/lora_routing.rs
Endpoint setup supplies KV or hosted LoRA candidate policies. Hosted selection uses pins, runtime configuration, routing constraints, and available capacity.
Worker selection and dispatch validation
lib/llm/src/kv_router/routing_host.rs, lib/llm/src/kv_router/routing_host/kv_selection.rs, lib/llm/src/kv_router/routing_host/builtin.rs, lib/llm/src/kv_router/routing_host/kv.rs, lib/llm/src/kv_router/routing_host/request_guard.rs, lib/llm/src/kv_router/routing_host/tests.rs, lib/llm/src/lora_routing.rs
Routing hosts restrict candidates and recheck worker eligibility before dispatch, including after transport fallback. Tests cover runtime capability changes, worker pins, and rejected dispatches.
Execution-stage checks and replay behavior
lib/llm/src/kv_router/encoder_router.rs, lib/llm/src/kv_router/prefill_router/*, lib/llm/src/kv_router/routing_host/kv.rs, lib/llm/src/kv_router/routing_host/request_guard.rs, lib/llm/src/migration.rs, lib/llm/src/lora_routing.rs
Prefill and encoder routing reject unsupported runtime LoRA requests before dispatch. Requests that disallow replay do not retain migration state or retry.
Accounting attribution and metrics validation
lib/llm/src/kv_router.rs, lib/llm/src/lora_routing.rs, lib/llm/src/lora.rs, lib/lora/tests/accounting.rs
Accounting attribution distinguishes legacy LoRA aliases from complete runtime LoRA metadata. Tests check event counts, controller allocation, metric labels, and inflight cleanup.

Estimated code review effort: 4 (Complex) | ~45 minutes

Merge Risk: Low · up to 585e9

Setting DYN_LORA_RUNTIME_LOAD_ENABLED without DYN_LORA_ENABLED makes the frontend wait for the initial workers before it reports the configuration error. The delay affects startup only. Parse the configuration earlier, and confirm the failing rust-tests CI job before merging.

Pre-merge checks: Passed 4 · Inconclusive 1

Failed Failed checks: 1 inconclusive

Check name Status Explanation Resolution
Docstring Coverage Inconclusive Docstring coverage is 13.46% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed

Check name Status Explanation
Title check Passed The title clearly identifies the main change: integrating safe runtime LoRA routing.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description gives a clear summary of the changes and includes detailed validation results. It also notes that GPU live serving has not been rerun. The required information is mostly present, but t…
Full details: Description check

Explanation

The description gives a clear summary of the changes and includes detailed validation results. It also notes that GPU live serving has not been rerun. The required information is mostly present, but the template headings differ, and the Issues and Ongoing Work section and checklist are absent.

Full details: Docstring Coverage

Explanation

Docstring coverage is 13.46% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 52 functions across 21 files. (1 skipped: 1 too large.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between d832552 and 585e9c0.

Files selected for processing: 22

  • lib/llm/src/discovery/model_manager.rs
  • lib/llm/src/entrypoint/input/common.rs
  • lib/llm/src/kv_router.rs
  • lib/llm/src/kv_router/encoder_router.rs
  • lib/llm/src/kv_router/prefill_router/activation.rs
  • lib/llm/src/kv_router/prefill_router/mod.rs
  • lib/llm/src/kv_router/routing_host.rs
  • lib/llm/src/kv_router/routing_host/builder.rs
  • lib/llm/src/kv_router/routing_host/builtin.rs
  • lib/llm/src/kv_router/routing_host/kv.rs
  • lib/llm/src/kv_router/routing_host/kv_selection.rs
  • lib/llm/src/kv_router/routing_host/request_guard.rs
  • lib/llm/src/kv_router/routing_host/tests.rs
  • lib/llm/src/lora.rs
  • lib/llm/src/lora_routing.rs
  • lib/llm/src/migration.rs
  • lib/llm/src/preprocessor.rs
  • lib/llm/src/protocols/common/preprocessor.rs
  • lib/llm/src/routing_policy.rs
  • lib/lora/src/policy.rs
  • lib/lora/src/runtime.rs
  • lib/lora/tests/accounting.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 3 remain after this review.

@coderabbitai coderabbitai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Fix comments: Fix CodeRabbit comments on this PR

Comment thread lib/llm/src/entrypoint/input/common.rs Resolved
Comment thread lib/llm/src/lora_routing.rs Resolved
Comment thread lib/lora/src/policy.rs Resolved
@biswapanda
biswapanda requested review from a team as code owners November 8, 2026 01:36
@biswapanda
biswapanda force-pushed the bis/dis-2937-lora-routing-final branch from 023ecb1 to 67b0dd0 Compare November 8, 2026 01:48

@tedzhouhk tedzhouhk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed runtime identity validation, worker eligibility rechecks, pins and DP rank, replay gating, and cleanup at 67b0dd0. Local validation passed: 181 dynamo-lora unit tests and 21 focused LLM regressions. GPU-backed resolver/load E2E was not run.

@biswapanda
biswapanda removed request for a team November 10, 2026 02:19
@biswapanda
biswapanda removed request for a team November 10, 2026 02:19
@biswapanda
biswapanda force-pushed the bis/dis-2937-lora-routing-final branch from 67b0dd0 to 98f7064 Compare November 10, 2026 03:04
@biswapanda
biswapanda force-pushed the bis/dis-2937-lora-routing-final branch from c98be23 to 0f8497a Compare November 10, 2026 05:39
Base automatically changed from bis/dis-2937-lora-policy to main November 10, 2026 08:05
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
@biswapanda
biswapanda force-pushed the bis/dis-2937-lora-routing-final branch from 0f8497a to 0ffaae6 Compare November 10, 2026 08:05
@biswapanda
biswapanda enabled stacked auto-merge (squash) November 10, 2026 08:05
@biswapanda
biswapanda merged commit 3a524d0 into main Nov 10, 2026
126 checks passed
@biswapanda
biswapanda deleted the bis/dis-2937-lora-routing-final branch November 10, 2026 08:54

@ishandhanani ishandhanani left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Post-merge note on the plugin-facing surface. The feature and its placement look right: the kernel gains one generic enum and the LoRA logic stays in lib/llm and lib/lora. One API point below, raised after the fact because it affects out-of-tree policy crates.

pub block_hashes: Option<Vec<LocalBlockHash>>,
pub isl_tokens: usize,
pub lora_name: Option<String>,
pub accounting_attribution: AccountingAttribution,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding a pub field to ScheduleRequest (and to SelectionOperation in operation.rs) breaks every out-of-tree constructor that builds these by struct literal, even though the field has a Default. This PR had to patch both in-repo example policy crates with accounting_attribution: Default::default() for exactly that reason, and external policy crates that construct ScheduleRequest in their tests will hit the same compile error on upgrade.

Suggestion for a follow-up: keep additive fields non-breaking for plugin authors, either by giving ScheduleRequest a constructor or builder that plugin code is told to use, or by making newly added knobs pub(crate) with a public setter on the request. Any of those lets the next accounting or placement field land without touching every policy crate.


fn request(affinity_target: Option<WorkerAffinityTarget>) -> SchedulingRequest {
SchedulingRequest {
accounting_attribution: Default::default(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line is the symptom: an example policy had to change because a kernel request type gained a public field. External policy crates with the same literal need the same edit.

ishandhanani added a commit to ishandhanani/dynamo that referenced this pull request Nov 10, 2026
- ai-dynamo#17895 feat(lora): integrate safe runtime routing: the kernel's new AccountingAttribution enum and its threading through ScheduleRequest, SchedulingRequest, SelectionOperation, PendingSelection and ReservationBooking land whole in dynamo-scheduler (lib/router/scheduler/src/{types,host,local,policy,queue}.rs, selection/core/{operation,reservations,run,tests}.rs, selection/pending.rs, selector/{mod,reference}.rs); dynamo_kv_router::plugins::prelude exports AccountingAttribution and compat.rs keeps the 1.7 path dynamo_kv_router::scheduling::AccountingAttribution; lib/llm (routing_policy.rs, lora_routing.rs, kv_router.rs, routing_host/{hosted,kv_selection}.rs) and lib/lora (tests/accounting.rs) keep upstream's content with owner-crate re-points to dynamo_scheduler::host, dynamo_worker_selection::host, dynamo_slot_tracker::host and dynamo_router_protocol::worker plus the hosted picker's KvSchedulerError adaptation; the example policy crates, builtin policies, mocker and the kv-router worker_selection bench take the new field via Default::default().

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>

@ishandhanani ishandhanani left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes on the AccountingAttribution design, to land as a follow-up since this is merged.

The host decides whether a request's lora_name counts toward per-adapter demand (one if in lora_routing::execution_requirements), but instead of passing the resolved answer it passes a two-variant flag that the scheduler kernel threads through ScheduleRequest, SchedulingRequest, SelectionOperation, PendingSelection, and ReservationBooking, then interprets at the two booking sites by masking lora_name. The kernel has no information at those sites that the host lacked when it set the flag, so the enum, its impl, and the five field threads are carrying a decision that was already made.

Ask for the follow-up:

  1. Delete AccountingAttribution and the field from all five structs.
  2. Have lib/llm pass the resolved accounting identity instead: either a lora_name that is already None on the paths that must not be charged, or a separate accounting_lora_name: Option<String> if the routing identity still needs to reach selection. The kernel books what it is given.
  3. Whatever field shape survives, do not add it as a new pub field on ScheduleRequest. Both in-repo example policy crates needed accounting_attribution: Default::default() in this PR, and every out-of-tree policy crate that builds the request by struct literal breaks the same way on upgrade. Land request-side additions behind a constructor or builder.

Inline notes point at each piece.


/// Whether routing identity contributes to per-adapter demand accounting.
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub enum AccountingAttribution {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bool spelled as an enum, and its only consumer is accounting_name, which maps it straight to Option<String>. The one place it is set (lora_routing::execution_requirements) already knows the resolved value. Pass that value and this type, the impl, and the field threads through ScheduleRequest, SchedulingRequest, SelectionOperation, PendingSelection, and ReservationBooking all go away.

pub block_hashes: Option<Vec<LocalBlockHash>>,
pub isl_tokens: usize,
pub lora_name: Option<String>,
pub accounting_attribution: AccountingAttribution,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A pub field on the plugin-facing request for a value no policy reads or sets. Every external policy crate that constructs ScheduleRequest by struct literal stops compiling on upgrade; this PR had to patch both in-repo examples for exactly that. If a new request-side knob is unavoidable it should land behind a constructor or builder, not as another public field.

prefill_load_hint,
worker,
lora_name,
lora_name: accounting_attribution.accounting_name(lora_name),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The booking decision is made here by masking lora_name with the flag, but nothing is known at this point that was not known in lib/llm when the flag was set. Resolve it there and book the name you are handed.

lora_name: request.lora_name.take(),
lora_name: request
.accounting_attribution
.accounting_name(request.lora_name.take()),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Second copy of the same booking-time mask. With the host passing the resolved identity, this line and the one in reservations.rs become plain lora_name again.

let policy = policy(request);
ExecutionRequirements {
allows_replay: policy.allows_replay(),
policy_required: policy.requires_candidate_policy(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the only decision site: tracks_legacy_load() means charge lora_name, otherwise charge nothing. Emit the resolved identity here (None, or the name) instead of a flag the kernel has to interpret five structs later.

mnsaglam added a commit to deepinfra/dynamo that referenced this pull request Nov 10, 2026
Resolutions:
- Policy classes follow upstream's flat model (ai-dynamo#17723). Our cache-bucket
  admission (bucket_key: cached_tokens, 538367a) goes with the
  family/bucket matrix it extended; a profile that still sets
  default_policy_family, uncached_isl_buckets, policy_family or cache_bucket
  now fails to load.
- Upstream's PendingSinkGuard replaces our SinkEntryGuard (cb8ba31) for the
  same pipeline sink leak.
- Worker-scoped shared-cache hits: the worker id moves from WorkerCacheData to
  the borrowed WorkerCacheInput view, so CandidateData keeps upstream's 128
  bytes.
- The shared-cache credit on a queued request's overlap refresh moves into
  upstream's refresh arm.
- Upstream's accounting attribution (ai-dynamo#17895) and our engine_block_hashes both
  ride on find_best_match, the scheduling request and the booked sequence
  event; both are kept, and the booked lora_name goes through
  accounting_name.
- Upstream's AttemptLease replaces the LoraLoadGuard the request guard held;
  our cache_prediction stays beside it.
- chat_completions tests: upstream's passthrough_extra_fields test is added;
  our over-max stop test keeps its truncation body and name.
- Upstream's new tests take engine_block_hashes: None and the SharedCacheQuery
  form of check_blocks.

Signed-off-by: Mert Saglam <mert@deepinfra.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PyExtreme added a commit to PyExtreme/dynamo that referenced this pull request Nov 11, 2026
…dp-rank

Resolve conflicts with the LoRA candidate-policy refactor (ai-dynamo#17889, ai-dynamo#17895)
in routing_host.rs and hosted.rs: take main's candidate_policy field and
the removal of select_lora_target, and keep the hard affinity DP rank
helper. Hold::session_id moved with the session-affinity crate split
(ai-dynamo#18055).

Signed-off-by: pyextreme <45779665+PyExtreme@users.noreply.github.com>
john-liu-dev added a commit to john-liu-dev/dynamo that referenced this pull request Nov 11, 2026
…e-decode-selection

Resolve the conflicts with ai-dynamo#18055 and move kv_capacity_aware, its
lib.rs test and its benchmark to the new crate layout: import from
dynamo_kv_router::plugins::prelude, build policies from SelectionConfig.
Add accounting_attribution, the SchedulingRequest field from ai-dynamo#17895,
to the kv_capacity_aware test helper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yuehua Liu <liujohn@nvidia.com>
mnsaglam added a commit to deepinfra/dynamo that referenced this pull request Nov 11, 2026
Resolutions:
- Policy classes follow upstream's flat model (ai-dynamo#17723). Our cache-bucket
  admission (bucket_key: cached_tokens, 538367a) goes with the
  family/bucket matrix it extended; a profile that still sets
  default_policy_family, uncached_isl_buckets, policy_family or cache_bucket
  now fails to load.
- Upstream's PendingSinkGuard replaces our SinkEntryGuard (cb8ba31) for the
  same pipeline sink leak.
- Worker-scoped shared-cache hits: the worker id moves from WorkerCacheData to
  the borrowed WorkerCacheInput view, so CandidateData keeps upstream's 128
  bytes.
- The shared-cache credit on a queued request's overlap refresh moves into
  upstream's refresh arm.
- Upstream's accounting attribution (ai-dynamo#17895) and our engine_block_hashes both
  ride on find_best_match, the scheduling request and the booked sequence
  event; both are kept, and the booked lora_name goes through
  accounting_name.
- Upstream's AttemptLease replaces the LoraLoadGuard the request guard held;
  our cache_prediction stays beside it.
- chat_completions tests: upstream's passthrough_extra_fields test is added;
  our over-max stop test keeps its truncation body and name.
- preprocessor: upstream's in-place response stream (ai-dynamo#18406) moves the
  per-chunk code onto self; our per-decode-worker logprob histogram
  (3e7a796) moves with it.
- Our shared-cache tests: the load-aware one expects upstream's --load-aware
  deprecation warning (ai-dynamo#16620), and the planner's router-field coverage
  test (ai-dynamo#17483) sets shared_cache_hit_weight as it does
  shared_cache_multiplier, since serde skips both at their defaults.
- Upstream's new tests take engine_block_hashes: None, model_eos: Vec::new()
  and the SharedCacheQuery form of check_blocks.
mnsaglam added a commit to deepinfra/dynamo that referenced this pull request Nov 11, 2026
Resolutions:
- Policy classes follow upstream's flat model (ai-dynamo#17723). Our cache-bucket
  admission (bucket_key: cached_tokens, 538367a) goes with the
  family/bucket matrix it extended; a profile that still sets
  default_policy_family, uncached_isl_buckets, policy_family or cache_bucket
  now fails to load.
- Upstream's PendingSinkGuard replaces our SinkEntryGuard (cb8ba31) for the
  same pipeline sink leak.
- Worker-scoped shared-cache hits: the worker id moves from WorkerCacheData to
  the borrowed WorkerCacheInput view, so CandidateData keeps upstream's 128
  bytes.
- The shared-cache credit on a queued request's overlap refresh moves into
  upstream's refresh arm.
- Upstream's accounting attribution (ai-dynamo#17895) and our engine_block_hashes both
  ride on find_best_match, the scheduling request and the booked sequence
  event; both are kept, and the booked lora_name goes through
  accounting_name.
- Upstream's AttemptLease replaces the LoraLoadGuard the request guard held;
  our cache_prediction stays beside it.
- chat_completions tests: upstream's passthrough_extra_fields test is added;
  our over-max stop test keeps its truncation body and name.
- preprocessor: upstream's in-place response stream (ai-dynamo#18406) moves the
  per-chunk code onto self; our per-decode-worker logprob histogram
  (3e7a796) moves with it.
- Our shared-cache tests: the load-aware one expects upstream's --load-aware
  deprecation warning (ai-dynamo#16620), and the planner's router-field coverage
  test (ai-dynamo#17483) sets shared_cache_hit_weight as it does
  shared_cache_multiplier, since serde skips both at their defaults.
- Upstream's new tests take engine_block_hashes: None and the
  SharedCacheQuery form of check_blocks.
- Upstream's router split (ai-dynamo#18055) moves our kv-router changes into the new
  crates: per-worker SharedCacheHits into dynamo-worker-selection,
  SharedCacheQuery and observe_stored onto the scheduler's SharedKvCache, the
  mooncake-store cache type and its default multiplier onto SharedCacheType,
  and engine_block_hashes onto the slot tracker's wire event.
- shared_cache_hit_weight reaches the kernel as SelectionConfig's
  shared_cache_hit_weight and CacheHitWeights::shared, which the selection
  run and the overlap refresher both read.
- Our lib/llm code imports these from the stage crates' host modules, not
  the deprecated kv-router paths; the scheduler's host module also exports
  SharedCacheQuery and the slot tracker's exports ReplicaPromptObserver.
mnsaglam added a commit to deepinfra/dynamo that referenced this pull request Nov 12, 2026
Resolutions:
- Policy classes follow upstream's flat model (ai-dynamo#17723). Our cache-bucket
  admission (bucket_key: cached_tokens, 538367a) goes with the
  family/bucket matrix it extended; a profile that still sets
  default_policy_family, uncached_isl_buckets, policy_family or cache_bucket
  now fails to load.
- Upstream's PendingSinkGuard replaces our SinkEntryGuard (cb8ba31) for the
  same pipeline sink leak.
- Worker-scoped shared-cache hits: the worker id moves from WorkerCacheData to
  the borrowed WorkerCacheInput view, so CandidateData keeps upstream's 128
  bytes.
- The shared-cache credit on a queued request's overlap refresh moves into
  upstream's refresh arm.
- Upstream's accounting attribution (ai-dynamo#17895) and our engine_block_hashes both
  ride on find_best_match, the scheduling request and the booked sequence
  event; both are kept, and the booked lora_name goes through
  accounting_name.
- Upstream's AttemptLease replaces the LoraLoadGuard the request guard held;
  our cache_prediction stays beside it.
- chat_completions tests: upstream's passthrough_extra_fields test is added;
  our over-max stop test keeps its truncation body and name.
- preprocessor: upstream's in-place response stream (ai-dynamo#18406) moves the
  per-chunk code onto self; our per-decode-worker logprob histogram
  (3e7a796) moves with it.
- Our shared-cache tests: the load-aware one expects upstream's --load-aware
  deprecation warning (ai-dynamo#16620), and the planner's router-field coverage
  test (ai-dynamo#17483) sets shared_cache_hit_weight as it does
  shared_cache_multiplier, since serde skips both at their defaults.
- Upstream's new tests take engine_block_hashes: None and the
  SharedCacheQuery form of check_blocks.
- Upstream's router split (ai-dynamo#18055) moves our kv-router changes into the new
  crates: per-worker SharedCacheHits into dynamo-worker-selection,
  SharedCacheQuery and observe_stored onto the scheduler's SharedKvCache, the
  mooncake-store cache type and its default multiplier onto SharedCacheType,
  and engine_block_hashes onto the slot tracker's wire event.
- shared_cache_hit_weight reaches the kernel as SelectionConfig's
  shared_cache_hit_weight and CacheHitWeights::shared, which the selection
  run and the overlap refresher both read.
- Our lib/llm code imports these from the stage crates' host modules, not
  the deprecated kv-router paths; the scheduler's host module also exports
  SharedCacheQuery and the slot tracker's exports ReplicaPromptObserver.
- Upstream's kvcr policy (ai-dynamo#17647) takes KV-copy credit out of the default
  selector and drops its per-request shared_cache_multiplier override; the
  default multiplier still comes from SharedCacheType::default_multiplier,
  so mooncake-store keeps its 0.5.
- Upstream's live per-rank resident block counts (ai-dynamo#16078) and our tracked
  block count and worker listing (235a17b) share the indexer actor.
- Upstream's needs_uncached_prefill (ai-dynamo#17441) rides on find_best_match beside
  engine_block_hashes.
- The embedded partition config carries upstream's hint policies (ai-dynamo#17359)
  beside our peer prompt observer.
- Upstream's prepare_inputs gate decides whether a shared-cache client is
  created, mooncake-store included.
mnsaglam added a commit to deepinfra/dynamo that referenced this pull request Nov 12, 2026
Resolutions:
- Policy classes follow upstream's flat model (ai-dynamo#17723). Our cache-bucket
  admission (bucket_key: cached_tokens, 538367a) goes with the
  family/bucket matrix it extended; a profile that still sets
  default_policy_family, uncached_isl_buckets, policy_family or cache_bucket
  now fails to load.
- Upstream's PendingSinkGuard replaces our SinkEntryGuard (cb8ba31) for the
  same pipeline sink leak.
- Worker-scoped shared-cache hits: the worker id moves from WorkerCacheData to
  the borrowed WorkerCacheInput view, so CandidateData keeps upstream's 128
  bytes.
- The shared-cache credit on a queued request's overlap refresh moves into
  upstream's refresh arm.
- Upstream's accounting attribution (ai-dynamo#17895) and our engine_block_hashes both
  ride on find_best_match, the scheduling request and the booked sequence
  event; both are kept, and the booked lora_name goes through
  accounting_name.
- Upstream's AttemptLease replaces the LoraLoadGuard the request guard held;
  our cache_prediction stays beside it.
- chat_completions tests: upstream's passthrough_extra_fields test is added;
  our over-max stop test keeps its truncation body and name.
- preprocessor: upstream's in-place response stream (ai-dynamo#18406) moves the
  per-chunk code onto self; our per-decode-worker logprob histogram
  (3e7a796) moves with it.
- Our shared-cache tests: the load-aware one expects upstream's --load-aware
  deprecation warning (ai-dynamo#16620), and the planner's router-field coverage
  test (ai-dynamo#17483) sets shared_cache_hit_weight as it does
  shared_cache_multiplier, since serde skips both at their defaults.
- Upstream's new tests take engine_block_hashes: None and the
  SharedCacheQuery form of check_blocks.
- Upstream's router split (ai-dynamo#18055) moves our kv-router changes into the new
  crates: per-worker SharedCacheHits into dynamo-worker-selection,
  SharedCacheQuery and observe_stored onto the scheduler's SharedKvCache, the
  mooncake-store cache type and its default multiplier onto SharedCacheType,
  and engine_block_hashes onto the slot tracker's wire event.
- shared_cache_hit_weight reaches the kernel as SelectionConfig's
  shared_cache_hit_weight and CacheHitWeights::shared, which the selection
  run and the overlap refresher both read.
- Our lib/llm code imports these from the stage crates' host modules, not
  the deprecated kv-router paths; the scheduler's host module also exports
  SharedCacheQuery and the slot tracker's exports ReplicaPromptObserver.
- Upstream's kvcr policy (ai-dynamo#17647) takes KV-copy credit out of the default
  selector and drops its per-request shared_cache_multiplier override; the
  default multiplier still comes from SharedCacheType::default_multiplier,
  so mooncake-store keeps its 0.5.
- Upstream's live per-rank resident block counts (ai-dynamo#16078) and our tracked
  block count and worker listing (235a17b) share the indexer actor.
- Upstream's needs_uncached_prefill (ai-dynamo#17441) rides on find_best_match beside
  engine_block_hashes.
- The embedded partition config carries upstream's hint policies (ai-dynamo#17359)
  beside our peer prompt observer.
- Upstream's prepare_inputs gate decides whether a shared-cache client is
  created, mooncake-store included.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feat frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` router Relates to routing, KV-aware routing, etc. size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants