Skip to content

refactor(router): open the admission stage and move it to lib/router/admission - #182

Closed
ishandhanani wants to merge 5 commits into
idhanani/rs2-08-worker-selectionfrom
idhanani/rs2-09-admission
Closed

ishandhanani wants to merge 5 commits into
idhanani/rs2-08-worker-selectionfrom
idhanani/rs2-09-admission

Conversation

@ishandhanani

Copy link
Copy Markdown
Owner

Router split v2, unit 09 of 17. Stacked on #181 (unit 08).

Summary

dynamo-admission (lib/router/admission) admits, holds, or rejects now: the per-class policy queue, resolved profiles, the classifier runtime and request lifecycle, the classifier plugin API with its own WorkerView trait, AdmissionConfig, ClassifierConfig, AdmissionError; it depends on protocol only, with a testing feature, and its classifier API is tier 1 through kv-router's prelude. Four boundary changes first (#129 WorkerInputs and WorkerInfo to protocol's columns; #130 classifiers read workers through an admission-owned view; #131 the stage's own resolved config, with BusyThresholds to protocol; #132 ClassificationError reported by the runtime), then #133 moves the files. Rule applied: the producer owns its output; the one plugin-API signature change (#130, classify(request, &dyn WorkerView)) is what keeps the stage from reaching the kernel's candidate table. kv-router re-exports every name at the paths plugins and hosts already use. Replaces fork PRs #129, #130, #131, #132, #133.

What moves

  • refactor(router): move WorkerInputs and WorkerInfo to the protocol crate #129 12dd91e993 refactor(router): move WorkerInputs and WorkerInfo to the protocol crate: from worker-selection's inputs.rs to lib/router/protocol/src/columns.rs; worker-selection re-exports them. Move-diff (its body): 458 common / 1 new / 1 dropped, the one pair being pub(crate) fn without becoming pub fn without.
  • refactor(router): classifiers read workers through an admission-owned view #130 a90f585625 refactor(router): classifiers read workers through an admission-owned view: RequestClassifier::classify(request, &dyn WorkerView) replaces WorkerCandidates<'_>; view.rs, classifier_rows.rs. Move-diff (its body): 9,243 common / 124 new / 39 dropped (new: view.rs, classifier_rows.rs, the &dyn WorkerView parameters, the ported capacity-check test; dropped: the WorkerCandidates<'_> parameters, the read(|workers| classify(..)) closures, the iter() form of that test).
  • refactor(router): give the admission stage its own resolved config #131 897923e4ff refactor(router): give the admission stage its own resolved config: BusyThresholds to protocol load (the slot tracker re-exports it and keeps the rank checks as BusyRanksExt); scheduling/admission_config.rs with RouterQueuePolicy, PolicyClassConfig, PolicyProfile, PolicyProfile::family_buckets, SYNTHETIC_POLICY_CLASS. Move-diff (its body): 3,878 common / 109 new / 10 dropped (new: the family_buckets constructor and its checks, FamilyBucketTable, AdmissionConfigError and its From, the BusyRanksExt trait, two #[inline] hints; dropped: the struct-literal profile construction, the UncachedIslBucket vector in the resolver, pub(crate) on resolve_class_index_strict).
  • refactor(router): classifier runtime reports its own error and takes the view #132 5b8c1f3ebb refactor(router): classifier runtime reports its own error and takes the view: ClassificationError with From into KvSchedulerError; classify_with(request, &dyn WorkerView). Move-diff (its body): 3,788 common / 91 new / 64 dropped (new: the error enum, the From impl, the view parameter and the two call-site wraps, NoWorkers in the runtime tests, the Sync bound; dropped: the KvSchedulerError variant constructions and signatures in the runtime, the WorkerInputRows::empty() arguments, the in-runtime ClassifierRows wrap).
  • refactor(router): move the admission stage to lib/router/admission #133 b964520ae4 refactor(router): move the admission stage to lib/router/admission: policy_queue.rs to queue.rs, queue_admission/mod.rs to placement.rs, admission_config.rs to profile.rs, the classifier plugin API to classifier/mod.rs with inputs.rs, progress.rs, view.rs, and the runtime to classifier/runtime.rs; the 24 policy-queue tests stay in kv-router as policy_queue_tests.rs over three testing-gated accessors. Move-diff (its body): 3,941 common / 55 new / 49 dropped (the pub(crate) to pub promotions, ClassifyRequest's three fields losing pub(crate), the queue tests reading due_entry_count/class_deficit/class_blocks_worker instead of the private fields).

Dependency edges

  • dynamo-admission -> dynamo-router-protocol (router deps; dynamo-runtime, dynamo-kv-router, dynamo-kv-indexer, dynamo-slot-tracker, dynamo-session-affinity, dynamo-worker-selection, dynamo-kv-events, dynamo-kv-hashing, dynamo-zmq, axum, hyper, reqwest, zmq, prometheus forbidden in its tree; direct column pinned in unit 14)

Validation

Unit-level move-diff, run from the worktree at the unit head b964520ae4: python3 -I scripts/router_move_diff.py 7598a008de b964520ae4 --from <17 paths> --to <28 paths>, the union of the replaced fork PRs' --from/--to lists filtered to the paths present at each end (full command in the artifact /tmp/rr-split/unit-prs/09-move-diff.txt).

summary from_normalized=16089 to_normalized=16306 common=15935 new=371 dropped=154
result DIFFERS

371 new / 154 dropped (the per-PR runs sum to 380 / 163). By category: ClassificationError (the enum, its From into KvSchedulerError, the map_err sites, the Err(ClassificationError::LifecycleEnded(_)) assertions) replacing the KvSchedulerError variant constructions (#132); the &dyn WorkerView parameter (&NoWorkers 15x, ClassifierRows(workers)) replacing WorkerInputRows::empty() (15x) and _workers: WorkerCandidates<'_> (13x) (#130); PolicyProfile::family_buckets, FamilyBucketTable, AdmissionConfigError and its validation messages replacing the struct-literal profile construction and the UncachedIslBucket vector (#131); BusyRanksExt and the two #[inline] hints (#131); visibility promotions for the move (64 / 39) and the queue tests reading due_entry_count/class_deficit/class_blocks_worker instead of the private fields (#133); pub(crate) fn without to pub fn without (#129). Each fork PR body explains its own lines.

New and dropped normalized lines (371 new, 154 dropped).
new 1x "class table has {} cells for {families} families and {} buckets",
new 1x "default family index {default_family_index} is out of range for {families} families"
new 1x "uncached_isl_buckets min_tokens must be strictly increasing".to_string(),
new 1x #[derive(Clone, Copy, Debug, Default)]
new 2x #[derive(Clone, Copy, Debug, Default, PartialEq)]
new 1x #[derive(Debug, Clone, PartialEq)]
new 1x #[derive(Debug, thiserror::Error)]
new 1x #[derive(Debug, thiserror::Error, PartialEq, Eq)]
new 1x #[error("invalid request classification metadata: {0}")]
new 1x #[error("invalid router policy config: {0}")]
new 1x #[error("request classifier failed: {0}")]
new 1x #[error("request classifier is already tracking request ID {0:?}")]
new 1x #[error("request classifier panicked: {0}")]
new 1x #[error("request classifier runtime shut down")]
new 1x #[error("request lifecycle for request ID {0:?} ended before classification completed")]
new 2x #[inline]
new 15x &NoWorkers,
new 2x &self,
new 1x (0..workers.len())
new 1x ) -> Result<ClassifyRequest, ClassificationError> {
new 1x ) -> Result<RequestLifecycle, ClassificationError> {
new 1x ) -> Result<tokio::sync::MutexGuard<'_, Box<dyn RequestClassifier>>, ClassificationError> {
new 2x ) -> bool;
new 1x );
new 2x .any(|&index| index >= classes.len())
new 1x .begin_request(request_id)
new 1x .cache(row)
new 4x .classify_with(ClassifyRequest::new(1, 0), &NoWorkers)
new 1x .classify_with(ClassifyRequest::new(1, 1), &NoWorkers)
new 2x .classify_with(ClassifyRequest::new(2, 0), &NoWorkers)
new 1x .classify_with(classification, &ClassifierRows(workers))
new 1x .collect(),
new 1x .device_overlap_blocks,
new 2x .flatten()
new 1x .get(request_id)
new 1x .into_iter()
new 1x .map(|classifier| {
new 1x .map(|min_tokens| UncachedIslBucket { min_tokens })
new 1x .map(|row| {
new 2x .map_err(KvSchedulerError::from)
new 1x .map_err(|error| ClassificationError::Failed(Arc::from(error)))?;
new 2x .map_err(|panic| ClassificationError::Panicked(panic_message(panic)))?
new 1x .ok_or_else(|| ClassificationError::LifecycleEnded(request_id.to_owned()))?;
new 1x .values()
new 1x 0
new 1x AdmissionConfigError::Validation(message) => Self::Validation(message),
new 1x ClassificationError::DuplicateRequestId(id) => {
new 1x ClassificationError::DuplicateRequestId(request_id)
new 1x ClassificationError::Failed(error) => Self::RequestClassifierFailed(error),
new 1x ClassificationError::Failed(source)
new 1x ClassificationError::InvalidMetadata(message)
new 1x ClassificationError::InvalidMetadata(message) => {
new 1x ClassificationError::LifecycleEnded(id) => Self::ClassificationLifecycleEnded(id),
new 2x ClassificationError::Panicked(message)
new 1x ClassificationError::Panicked(message) => Self::RequestClassifierPanicked(message),
new 1x ClassificationError::Shutdown => Self::SubscriberShutdown,
new 1x DuplicateRequestId(String),
new 5x Err(ClassificationError::LifecycleEnded(_))
new 1x Failed(Arc<ClassifierError>),
new 1x InvalidMetadata(String),
new 1x LifecycleEnded(String),
new 3x None
new 1x Ok(PolicyProfile::family_buckets(FamilyBucketTable {
new 1x Ok(Self {
new 1x Panicked(String),
new 1x Self::DuplicateClassificationRequestId(id)
new 1x Self::InvalidClassificationMetadata(message)
new 1x Shutdown,
new 1x Validation(String),
new 1x WorkerCacheView {
new 2x _ = self.shutdown.cancelled() => return Err(ClassificationError::Shutdown),
new 12x _workers: &dyn WorkerView,
new 1x active_decode_blocks: load.active_decode_blocks(),
new 1x active_prefill_tokens: load.active_prefill_tokens(),
new 1x active_requests: load.active_requests(),
new 1x assert!(queue.class_blocks_worker(0, worker));
new 1x assert_eq!(queue.class_deficit(0), 0);
new 1x assert_eq!(queue.class_deficit(0), 3);
new 2x assert_eq!(queue.class_deficit(0), 5);
new 1x assert_eq!(queue.class_deficit(1), 0);
new 1x assert_eq!(queue.class_deficit(1), blocked_deficit);
new 2x assert_eq!(queue.due_entry_count(), 0);
new 1x assert_eq!(queue.due_entry_count(), 1);
new 1x async fn await_terminal_delivery(&self, request_id: &str) -> Result<(), ClassificationError> {
new 1x available: load.is_available(),
new 1x bucket_min_tokens,
new 1x bucket_min_tokens.len()
new 1x bucket_min_tokens: resolved_buckets.buckets,
new 1x buckets.push(raw.min_tokens);
new 1x buckets: Vec<usize>,
new 1x buckets: bucket_min_tokens
new 1x cache.accounting_cache_estimate();
new 1x candidate.cache().map(|cache| {
new 1x candidate.load().map(|load| WorkerLoadView {
new 2x capacity: WorkerCapacity,
new 1x catch_unwind(AssertUnwindSafe(|| classifier.classify(request, workers)))
new 2x class_by_family_bucket,
new 1x class_by_family_bucket.len(),
new 2x classes,
new 1x classification_id: u64,
new 1x classifier
new 1x decode_cost_blocks: load.decode_cost_blocks(),
new 2x default_family_index,
new 1x device_overlap_blocks: cache.device_overlap_blocks(),
new 1x disk_overlap_blocks: cache.disk_overlap_blocks(),
new 1x dp_ranks: Range<u32>,
new 1x effective_overlap_blocks,
new 1x estimated_cached_tokens,
new 2x explicit_class_indices,
new 2x family_indices,
new 1x fn active_decode_blocks(&self) -> usize {
new 1x fn active_prefill_tokens(&self) -> usize {
new 1x fn cache(&self, _row: usize) -> Option<WorkerCacheView> {
new 1x fn cache(&self, row: usize) -> Option<WorkerCacheView> {
new 1x fn cache(&self, row: usize) -> Option<WorkerCacheView>;
new 1x fn classify(&mut self, request: ClassifyRequest, _workers: &dyn WorkerView) -> ClassifyFuture {
new 1x fn from(error: AdmissionConfigError) -> Self {
new 1x fn from(error: super::request_classifier::ClassificationError) -> Self {
new 1x fn info(&self, _row: usize) -> Option<WorkerInfo> {
new 1x fn info(&self, row: usize) -> Option<WorkerInfo> {
new 1x fn info(&self, row: usize) -> Option<WorkerInfo>;
new 1x fn is_empty(&self) -> bool {
new 2x fn len(&self) -> usize {
new 1x fn len(&self) -> usize;
new 1x fn load(&self, _row: usize) -> Option<WorkerLoadView> {
new 1x fn load(&self, row: usize) -> Option<WorkerLoadView> {
new 1x fn load(&self, row: usize) -> Option<WorkerLoadView>;
new 2x fn rank_is_full(
new 1x fn with_row<R>(&self, row: usize, read: impl FnOnce(WorkerCandidate<'_>) -> R) -> Option<R> {
new 2x fn worker_is_full_in_ranks(
new 1x host_overlap_blocks: cache.host_overlap_blocks(),
new 1x if bucket_min_tokens.first() != Some(&0) {
new 1x if bucket_min_tokens.windows(2).any(|pair| pair[1] <= pair[0]) {
new 1x if class_by_family_bucket
new 1x if class_by_family_bucket.len() != families * bucket_min_tokens.len() {
new 1x if classes.is_empty() {
new 1x if default_family_index >= families {
new 1x if explicit_class_indices
new 1x if family_indices.values().any(|&index| index >= families) {
new 1x impl BusyRanksExt for BusyThresholds {
new 1x impl ClassifierRows {
new 1x impl From<AdmissionConfigError> for RouterPolicyConfigError {
new 1x impl From<super::request_classifier::ClassificationError> for KvSchedulerError {
new 1x impl RankLoad for WorkerLoadView {
new 1x impl WorkerView for ClassifierRows {
new 1x impl WorkerView for NoWorkers {
new 1x info.worker(),
new 1x let (effective_overlap_blocks, estimated_cached_tokens) =
new 1x let FamilyBucketTable {
new 1x let blocked_deficit = queue.class_deficit(1);
new 1x let families = family_indices.len();
new 1x let info = workers.info(row).expect("row in range");
new 1x let invalid = |message: String| Err(AdmissionConfigError::Validation(message));
new 1x let live = live_requests
new 1x let load = workers.load(row).expect("LOAD was declared");
new 1x let mut future = classifier.classify(input, &ClassifierRows(workers));
new 1x let pending = classifier.classify(classified, &ClassifierRows(workers));
new 1x let result = PassThrough.classify(request, &NoWorkers).await.unwrap();
new 1x load.active_requests,
new 1x load.available,
new 2x loads: &FxHashMap<WorkerWithDpRank, WorkerLoadProjection>,
new 2x match error {
new 1x overrides: ClassificationOverrides,
new 1x progress: RequestProgress,
new 1x pub active_decode_blocks: usize,
new 1x pub active_prefill_tokens: usize,
new 1x pub active_requests: usize,
new 1x pub arrival_offset_secs: f64,
new 1x pub async fn classify_with(
new 1x pub available: bool,
new 1x pub bucket_min_tokens: Vec<usize>,
new 1x pub class_by_family_bucket: Vec<usize>,
new 1x pub class_index: usize,
new 1x pub classes: Vec<PolicyClassConfig>,
new 1x pub const SYNTHETIC_POLICY_CLASS: &str = "default";
new 1x pub decode_cost_blocks: f64,
new 1x pub default_family_index: usize,
new 1x pub device_overlap_blocks: f64,
new 1x pub disk_overlap_blocks: f64,
new 2x pub due_at: Option<Instant>,
new 1x pub effective_overlap_blocks: f64,
new 1x pub enum AdmissionConfigError {
new 1x pub enum ClassificationError {
new 1x pub estimated_cached_tokens: usize,
new 1x pub explicit_class_indices: HashMap<String, usize>,
new 1x pub family_indices: HashMap<String, usize>,
new 1x pub fn any_ready_head(
new 1x pub fn begin_request(
new 1x pub fn cached_overrides(&self, request_id: &str) -> Option<bool> {
new 1x pub fn class_blocks_worker(&self, class_index: usize, worker: WorkerWithDpRank) -> bool {
new 1x pub fn class_deficit(&self, class_index: usize) -> usize {
new 1x pub fn due_entry_count(&self) -> usize {
new 1x pub fn enqueue_with_due_at(
new 1x pub fn family_buckets(table: FamilyBucketTable) -> Result<Self, AdmissionConfigError> {
new 1x pub fn has_ready(&self) -> bool {
new 1x pub fn into_queue_inputs(self) -> ClassificationOverrides {
new 1x pub fn new(classifier: Box<dyn RequestClassifier>, shutdown: CancellationToken) -> Arc<Self> {
new 1x pub fn new(input_tokens: usize, initial_cached_tokens: usize) -> Self {
new 1x pub fn next_due_at(&self) -> Option<Instant> {
new 1x pub fn recheck_worker(&mut self, worker: WorkerWithDpRank) {
new 1x pub fn resolve_class_index_strict(
new 1x pub fn take_expired(&mut self, now: Instant) -> Vec<PolicyQueueEntry<T>> {
new 1x pub fn take_if_in_class(
new 1x pub fn with_busy_thresholds(
new 1x pub fn with_initial_policy_class(mut self, policy_class: impl Into<String>) -> Self {
new 1x pub fn with_pinned_worker(mut self, pinned_worker: Option<WorkerWithDpRank>) -> Self {
new 1x pub fn with_request_id(mut self, request_id: impl Into<String>) -> Self {
new 1x pub fn with_sequence_hashes(mut self, hashes: Option<&[SequenceHash]>) -> Self {
new 1x pub fn with_session_context(mut self, session_context: SessionContext) -> Self {
new 1x pub fn with_timing(
new 1x pub fn without(self, other: Self) -> Self {
new 1x pub fn worker_inputs(&self) -> WorkerInputs {
new 1x pub host_overlap_blocks: f64,
new 1x pub policy_class: Option<String>,
new 1x pub scheduling_cost_tokens: Option<usize>,
new 1x pub snapshot: QueueSnapshot,
new 1x pub struct ClassificationOverrides {
new 1x pub struct FamilyBucketTable {
new 1x pub struct NoWorkers;
new 1x pub struct QueueMetadata {
new 1x pub struct RequestClassifierRuntime {
new 1x pub struct WorkerCacheView {
new 1x pub struct WorkerLoadView {
new 1x pub trait BusyRanksExt {
new 1x pub trait WorkerView: Sync {
new 1x pub worker_selection_target: Option<Option<WorkerAffinityTarget>>,
new 1x pub(crate) struct ClassifierRows(pub(crate) WorkerInputRows);
new 1x return Err(ClassificationError::DuplicateRequestId(
new 1x return Err(ClassificationError::InvalidMetadata(
new 2x return Err(ClassificationError::LifecycleEnded(request_id.to_owned()));
new 2x return Err(ClassificationError::Shutdown);
new 1x return invalid(
new 1x return invalid("class table names a policy class index out of range".to_string());
new 1x return invalid("explicit policy class index is out of range".to_string());
new 1x return invalid("policy family index is out of range".to_string());
new 1x return invalid("profile has no policy classes".to_string());
new 1x return invalid("uncached_isl_buckets must start at min_tokens 0".to_string());
new 2x return invalid(format!(
new 1x self.0.read(|candidates| candidates.get(row).map(read))
new 1x self.0.read(|candidates| candidates.len())
new 1x self.active_decode_blocks
new 1x self.active_prefill_tokens
new 1x self.classes[class_index].blocked_workers.contains(&worker)
new 1x self.classes[class_index].deficit
new 1x self.due_entries.len()
new 1x self.len() == 0
new 1x self.with_row(row, |candidate| candidate.info())
new 2x self.with_row(row, |candidate| {
new 1x thresholds.has_room(&load, info.capacity()),
new 1x worker: WorkerWithDpRank,
new 1x worker_id: WorkerId,
new 3x workers: &dyn WorkerView,
new 2x {
new 45x }
new 1x } = table;
new 5x })
new 1x })?)
dropped 1x &mut self,
dropped 1x ) -> Arc<Self> {
dropped 1x ) -> ClassifyFuture {
dropped 1x ) -> Result<ClassifyRequest, KvSchedulerError> {
dropped 1x ) -> Result<RequestLifecycle, KvSchedulerError> {
dropped 1x ) -> Result<tokio::sync::MutexGuard<'_, Box<dyn RequestClassifier>>, KvSchedulerError> {
dropped 1x .await
dropped 1x .cache()
dropped 4x .classify_with(ClassifyRequest::new(1, 0), WorkerInputRows::empty())
dropped 1x .classify_with(ClassifyRequest::new(1, 1), WorkerInputRows::empty())
dropped 2x .classify_with(ClassifyRequest::new(2, 0), WorkerInputRows::empty())
dropped 1x .classify_with(classification, workers)
dropped 1x .device_overlap_blocks(),
dropped 1x .map(|candidate| {
dropped 1x .map(|classifier| classifier.begin_request(request_id))
dropped 1x .map_err(|error| KvSchedulerError::RequestClassifierFailed(Arc::from(error)))?;
dropped 1x .map_err(|panic| KvSchedulerError::RequestClassifierPanicked(panic_message(panic)))?
dropped 1x .map_err(|panic| KvSchedulerError::RequestClassifierPanicked(panic_message(panic)))?;
dropped 1x .read(|workers| PassThrough.classify(request, workers))
dropped 1x .unwrap();
dropped 5x Err(KvSchedulerError::ClassificationLifecycleEnded(_))
dropped 1x KvSchedulerError::ClassificationLifecycleEnded(request_id.to_owned())
dropped 1x KvSchedulerError::DuplicateClassificationRequestId(request_id)
dropped 1x KvSchedulerError::InvalidClassificationMetadata(message)
dropped 1x KvSchedulerError::RequestClassifierFailed(source)
dropped 2x KvSchedulerError::RequestClassifierPanicked(message)
dropped 1x Ok(PolicyProfile {
dropped 15x WorkerInputRows::empty(),
dropped 2x _ = self.shutdown.cancelled() => return Err(KvSchedulerError::SubscriberShutdown),
dropped 13x _workers: WorkerCandidates<'_>,
dropped 1x assert!(queue.classes[0].blocked_workers.contains(&worker));
dropped 2x assert!(queue.due_entries.is_empty());
dropped 1x assert_eq!(queue.classes[0].deficit, 0);
dropped 1x assert_eq!(queue.classes[0].deficit, 3);
dropped 2x assert_eq!(queue.classes[0].deficit, 5);
dropped 1x assert_eq!(queue.classes[1].deficit, 0);
dropped 1x assert_eq!(queue.classes[1].deficit, blocked_deficit);
dropped 1x assert_eq!(queue.due_entries.len(), 1);
dropped 1x async fn await_terminal_delivery(&self, request_id: &str) -> Result<(), KvSchedulerError> {
dropped 1x buckets.push(UncachedIslBucket {
dropped 1x buckets: Vec<UncachedIslBucket>,
dropped 1x buckets: resolved_buckets.buckets,
dropped 1x candidate
dropped 1x candidate.worker(),
dropped 1x classification
dropped 1x classifier: Box<dyn RequestClassifier>,
dropped 1x const SYNTHETIC_POLICY_CLASS: &str = "default";
dropped 1x drop(workers);
dropped 1x fn classify(
dropped 1x let blocked_deficit = queue.classes[1].deficit;
dropped 1x let classification = catch_unwind(AssertUnwindSafe(|| {
dropped 1x let live = live_requests.get(request_id).ok_or_else(|| {
dropped 1x let load = candidate.load().expect("LOAD was declared");
dropped 1x let mut future = workers.read(|workers| classifier.classify(input, workers));
dropped 1x let pending = workers.read(|workers| classifier.classify(classified, workers));
dropped 1x let result = WorkerInputRows::empty()
dropped 1x load.active_requests(),
dropped 1x load.is_available(),
dropped 1x min_tokens: raw.min_tokens,
dropped 1x pub fn rank_is_full(
dropped 1x pub fn worker_is_full_in_ranks(
dropped 1x pub(crate) arrival_offset_secs: f64,
dropped 1x pub(crate) async fn classify_with(
dropped 1x pub(crate) class_index: usize,
dropped 1x pub(crate) classification_id: u64,
dropped 2x pub(crate) due_at: Option<Instant>,
dropped 1x pub(crate) fn any_ready_head(
dropped 1x pub(crate) fn begin_request(
dropped 1x pub(crate) fn cached_overrides(&self, request_id: &str) -> Option<bool> {
dropped 1x pub(crate) fn enqueue_with_due_at(
dropped 1x pub(crate) fn has_ready(&self) -> bool {
dropped 1x pub(crate) fn into_queue_inputs(self) -> ClassificationOverrides {
dropped 1x pub(crate) fn new(
dropped 1x pub(crate) fn new(input_tokens: usize, initial_cached_tokens: usize) -> Self {
dropped 1x pub(crate) fn next_due_at(&self) -> Option<Instant> {
dropped 1x pub(crate) fn recheck_worker(&mut self, worker: WorkerWithDpRank) {
dropped 1x pub(crate) fn resolve_class_index_strict(
dropped 1x pub(crate) fn take_expired(&mut self, now: Instant) -> Vec<PolicyQueueEntry<T>> {
dropped 1x pub(crate) fn take_if_in_class(
dropped 1x pub(crate) fn with_busy_thresholds(
dropped 1x pub(crate) fn with_initial_policy_class(mut self, policy_class: impl Into<String>) -> Self {
dropped 1x pub(crate) fn with_pinned_worker(mut self, pinned_worker: Option<WorkerWithDpRank>) -> Self {
dropped 1x pub(crate) fn with_request_id(mut self, request_id: impl Into<String>) -> Self {
dropped 1x pub(crate) fn with_sequence_hashes(mut self, hashes: Option<&[SequenceHash]>) -> Self {
dropped 1x pub(crate) fn with_session_context(mut self, session_context: SessionContext) -> Self {
dropped 1x pub(crate) fn with_timing(
dropped 1x pub(crate) fn without(self, other: Self) -> Self {
dropped 1x pub(crate) fn worker_inputs(&self) -> WorkerInputs {
dropped 1x pub(crate) overrides: ClassificationOverrides,
dropped 1x pub(crate) policy_class: Option<String>,
dropped 1x pub(crate) progress: RequestProgress,
dropped 1x pub(crate) scheduling_cost_tokens: Option<usize>,
dropped 1x pub(crate) snapshot: QueueSnapshot,
dropped 1x pub(crate) struct ClassificationOverrides {
dropped 1x pub(crate) struct QueueMetadata {
dropped 1x pub(crate) struct RequestClassifierRuntime {
dropped 1x pub(crate) worker_selection_target: Option<Option<WorkerAffinityTarget>>,
dropped 1x request: ClassifyRequest,
dropped 2x request_id.to_owned(),
dropped 2x return Err(KvSchedulerError::ClassificationLifecycleEnded(
dropped 1x return Err(KvSchedulerError::DuplicateClassificationRequestId(
dropped 1x return Err(KvSchedulerError::InvalidClassificationMetadata(
dropped 2x return Err(KvSchedulerError::SubscriberShutdown);
dropped 1x shutdown: CancellationToken,
dropped 1x thresholds.has_room(load, candidate.info().capacity()),
dropped 1x workers.read(|workers| classifier.classify(request, workers))
dropped 2x workers: WorkerCandidates<'_>,
dropped 1x workers: WorkerInputRows,
dropped 1x }))
dropped 1x });
dropped 1x })?;

Chain v2, full mode, on the rebased stack head ebd31cdf4d (every unit's commits are ancestors of it; the chain ran once on the head, not per unit), log /tmp/rr-split/srebase-validation.log: stage 1, the 13 router crates (clippy under default and all features, cargo test 1,247 passed / 5 ignored, rustdoc with all features), 1 min 50 s; stage 2, kv-router (clippy, bench check, 229 tests including the prelude test), 1 min 20 s; stage 3, workspace clippy, 4 min 58 s; stage 4, hosts (dynamo-llm lib alone, its kv_router 492 / discovery 275 / pool_relay 261 tests, 13 host crates clippy, bindings clippy; the pre-existing kvbm-consolidator manifest failure is the one non-zero exit), 29 min 21 s; stage 5, gates (every kv-router feature alone and together, 6 public-API snapshots unchanged with kv-router at 918 items, layering gate 47 checks and --host 66 checks with 0 failures, codeowners 7590/7590, fmt, 3 lockfiles --locked), 3 min 06 s; 40 min 35 s total, exit 0.

Each replaced fork PR carried its own validation chain; see its body.

Tickets

Part of DYNO-178

Step 5e-0 of the router split (DYNO-178). The column-group bitset a
stage component declares (WorkerInputs) and the static facts about a
rank every column carries (WorkerInfo: identity plus advertised
capacity) move from dynamo-worker-selection to a new columns module in
dynamo-router-protocol. Worker selection builds its rows from them and
admission's classifier will read workers through them, so neither
stage should have to name the other. Worker selection re-exports both,
so every existing path in kv-router, lib/llm, and the plugins holds.
WorkerInputs::without, which only the composed policy used, is public
in its new home.

Validation: protocol 10 and worker-selection 41 tests; kv-router
554 + 2 + 7; builtin plugin and lib/llm kv_router 485 tests; clippy clean
on the stage crates and lib/llm, mocker, ext-proc, bench; layering gate
22 checks; move-diff 458 common / 1 new / 1 dropped.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
… view

Step 5e-1 of the router split (DYNO-178). RequestClassifier::classify
takes `&dyn WorkerView` instead of worker selection's
`WorkerCandidates<'_>`. The view is the admission stage's own vocabulary:
one row per eligible rank, `info(row)` for identity and capacity, and
`load(row)` / `cache(row)` as plain column structs (WorkerLoadView,
WorkerCacheView) that read as None unless the classifier declared the
group. The kernel implements it over worker selection's rows with the
ClassifierRows newtype (scheduling/classifier_rows.rs), so each column
means what it means in a filter or scorer, and the admission crate will
depend on the protocol crate alone.

This is the one plugin-API signature change of the admission step. The
in-tree implementations (the ThunderAgent classifier, the lib/llm
watcher and routing-host test classifiers, the kv-router and builtin
test classifiers) change their parameter type and nothing else; none
of them read the table beyond `is_empty`. Groups other than CACHE and
LOAD are selection's and read as None through the view; no in-tree
classifier declared them. NoWorkers is the empty view.

Validation: kv-router 554 + 2 + 7; builtin plugin all targets; lib/llm
kv_router 485 and discovery watcher 36 tests; clippy clean on the stage
crates and lib/llm, mocker, ext-proc, bench; downstream check clean;
layering gate 22 checks; move-diff over the touched kv-router files
9,243 common / 124 new / 39 dropped (the view module, the newtype, and
the signature and test-site changes).

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Step 5e-2 of the router split (DYNO-178). The resolved half of the
policy config leaves the YAML behind.

- BusyThresholds (the per-rank limits, is_configured, has_room) moves
  to dynamo-router-protocol's load module beside WorkerCapacity and
  RankLoad: admission configures it per class, the kernel's gate
  evaluates it, classifiers read it. The slot tracker re-exports it and
  keeps the rank checks over its own projection as BusyRanksExt
  (rank_is_full, worker_is_full_in_ranks).
- RouterQueuePolicy, PolicyClassConfig, PolicyProfile, and the family-
  bucket classifier move to scheduling/admission_config.rs, the stage's
  resolved config, with a public validated constructor:
  PolicyProfile::family_buckets(FamilyBucketTable) refuses an index out
  of range, a bucket list that does not start at zero or is not
  strictly increasing, or a table whose size is not families times
  buckets, returning AdmissionConfigError::Validation. The raw YAML
  forms, the file reader, RouterSettings, the identifier rules, and
  every message the resolver emits today stay in policy_config.rs; the
  resolver builds the table and calls the constructor, and
  RouterPolicyConfigError wraps the stage's error.
- config.rs and policy_config.rs re-export every moved name, so
  dynamo_kv_router::RouterQueuePolicy, scheduling::{BusyThresholds,
  PolicyClassConfig, PolicyProfile, RouterPolicyConfigError}, and the
  slot tracker's busy::BusyThresholds all still resolve. The mocker's
  offline replay, the one caller of rank_is_full outside the slot
  tracker, imports BusyRanksExt beside BusyThresholdsExt.

Validation: protocol 10, slot-tracker 130, worker-selection 41,
kv-router 554 + 2 + 7 (resolver messages unchanged), builtin plugin,
lib/llm kv_router 485, mocker 304; clippy clean on the stage crates and
kv-router, lib/llm, mocker, ext-proc, bench; layering gate 22 checks;
move-diff 3878 common / 109 new / 10 dropped.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…the view

Step 5e-3 of the router split (DYNO-178). RequestClassifierRuntime no
longer names KvSchedulerError: classify_with, begin_request, and the
terminal-delivery wait return ClassificationError, with one variant per
failure the runtime can detect (Panicked, Failed, DuplicateRequestId,
InvalidMetadata, LifecycleEnded, Shutdown) and the same messages the
kernel's variants carried. The kernel converts with
From<ClassificationError> for KvSchedulerError, so its error surface,
the selection service's status mapping, and every host path are
unchanged. classify_with takes `&dyn WorkerView`; the scheduler wraps
the rows it built in ClassifierRows at its two call sites, and the
runtime's tests classify over the empty view. WorkerView gains a Sync
supertrait: views are read-only snapshots, and a classification future
that borrows one must stay Send for spawned callers.

Validation: kv-router 554 + 2 + 7; builtin plugin all targets; lib/llm
kv_router 485 and discovery watcher 36; clippy clean on kv-router,
builtin, lib/llm, mocker, ext-proc, bench; downstream check clean;
layering gate 22 checks; move-diff 3,788 common / 91 new / 64 dropped.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Step 5e-4 of the router split (DYNO-178): dynamo-admission, a stage
crate that depends on dynamo-router-protocol and nothing router-side
beyond it. A move: the files that made up the stage inside
lib/kv-router move as they are, and lib/kv-router re-exports every name
at the path plugins and hosts already use.

Moved: scheduling/policy_queue.rs as queue, scheduling/queue_admission
as placement, scheduling/admission_config.rs as profile, and the
classifier side as classifier/{mod (the plugin API), inputs, progress,
view, runtime (scheduling/request_classifier.rs: the runtime and the
request lifecycle)}.

Stays in lib/kv-router: the YAML policy file and its resolver
(policy_config.rs), the classifier instance config and the registries,
the gate over the slot projection, ClassifierRows (the WorkerView over
worker selection's rows), and the ClassificationError conversion. The
24 policy-queue tests build their profiles through the YAML resolver,
so they stay as scheduling/policy_queue_tests.rs and read the queue's
internals through three testing-gated accessors (due_entry_count,
class_deficit, class_blocks_worker); the entry-size test moves with the
queue. The classifier runtime's tests move with it.

Visibility the crate boundary required: the kernel-facing constructors
and scheduling methods are pub (ClassifyRequest::with_*, into_queue_
inputs; ClassificationOverrides and QueueMetadata with their fields;
PolicyQueue::{has_ready, any_ready_head, next_due_at, recheck_worker,
enqueue_with_due_at, take_if_in_class, take_expired};
RequestClassifierRuntime and its methods), documented as kernel-facing
in the crate docs instead of #[doc(hidden)]. ClassifyRequest's fields
stay private: the runtime that reads them is a child module.

Layering gate: a row for dynamo-admission (24 checks). CODEOWNERS gains
lib/router/admission/ under the router area.

Validation: admission 22 tests; kv-router 532 + 2 + 7 (532 + 22 moved = the
554 before); builtin plugin, lib/llm kv_router 485 and watcher 36,
mocker 304; clippy clean on admission, kv-router, builtin, lib/llm,
mocker, ext-proc, bench; layering gate 24 checks; move-diff 3,941
common / 55 new / 49 dropped (visibility and the test hooks).

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani

Copy link
Copy Markdown
Owner Author

Landed upstream as ai-dynamo#18055 (squash commit 79692cf on main, 2026-11-11). Closing this fork review unit.

This branch had an error being deployed

1 failed deployment
external_collaborator — b964520a Deployed Nov 2, 2026 by ishandhanani via ok-to-test #521
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation refactor router

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant