Repository navigation
feat(router): connect request classifier lifecycle - #13801
Closed
ishandhanani wants to merge 1 commit into
Closed
ishandhanani wants to merge 1 commit into
ishandhanani wants to merge 1 commit into
Conversation
This was referenced Aug 26, 2026
ishandhanani
force-pushed
the
idhanani/dyn-classify-lifecycle
branch
from
August 26, 2026 01:52
9a2310e to
d60639c
Compare
ishandhanani
force-pushed
the
idhanani/dyn-classify-lifecycle
branch
from
August 26, 2026 02:30
a36450a to
efde077
Compare
ishandhanani
force-pushed
the
idhanani/dyn-classify-lifecycle
branch
from
August 26, 2026 08:26
efde077 to
9a5fc5d
Compare
ishandhanani
force-pushed
the
idhanani/dyn-classify-lifecycle
branch
3 times, most recently
from
August 26, 2026 21:50
6d94dff to
9a9e41f
Compare
sara4dev
force-pushed
the
idhanani/dyn-classify-lifecycle
branch
from
August 26, 2026 23:20
9a9e41f to
0dc6b84
Compare
ishandhanani
force-pushed
the
idhanani/dyn-classify-lifecycle
branch
from
August 27, 2026 02:51
0dc6b84 to
7986f11
Compare
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
ishandhanani
force-pushed
the
idhanani/dyn-classify-lifecycle
branch
from
August 27, 2026 03:18
7986f11 to
811dc50
Compare
Contributor
Author
|
Closing as superseded. We are restarting from current main using DEP #13891 as the guide and replacing this three-PR stack with two focused PRs: first the complete generic request-classifier and lifecycle seam with an independently pollable classify future, then the ThunderAgent algorithm built on that public seam. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Connects the classifier lifecycle from #13800 to the production KV-router request and response paths. This lets pending classifiers learn when their logical requests are sent, begin responding, complete, or abort across retries without changing worker-selection policy.
Related DEP: #13891. Stack: #13800 → #13801 → #13802.
How This Was Implemented
Sentafter dispatch,Respondingon the first response,Completedwith authoritative context tokens when available, andAbortedfor cancellation, errors, or abandoned streams.Walkthrough
Mental model
The scheduler owns classification and queueing, while the existing response guard owns the only state that can determine terminal request outcome. A small lifecycle handle connects those owners without giving the classifier dispatch authority.
sequenceDiagram participant R as RoutingHost participant C as Classifier runtime participant S as Existing scheduler and selector participant G as Response guard R->>C: register logical request R->>S: classify, schedule, select S-->>G: worker plus lifecycle handle G->>C: Sent, then Responding G->>C: Completed or AbortedRequest lifecycle
Lifecycle registration occurs only for tracked requests. Selection records the chosen worker, addressed dispatch emits
Sent, the first backend item emitsResponding, and normal stream completion emitsCompleted. Backend usage metadata supplies total context tokens when present; otherwise the guard falls back to authoritative input tokens plus locally observed output tokens.Retryable migration failures move the lifecycle into migration state after cleaning up the failed attempt. The next attempt reuses that handle and re-enters ordinary scheduling without calling
classify()again; final failure emits oneAbortednotification with the last known worker and typed error when available.Boundaries and limitations
on_event()is synchronous and notification-only.Receivedremains available in the event vocabulary but is not emitted because the current transport has no authoritative receipt signal; this PR adds no affinity, placement, durable plugin state, mailbox, reconciliation protocol, or compatibility path.Validation
cargo test -q -p dynamo-llm kv_router::routing_host::tests --lib(24 passed)cargo test -q -p dynamo-llm test_retry_manager_generic_over_llm_engine_output --lib(1 passed)cargo test -q -p dynamo-llm classifier_cache_inputs_retain_cache_providers --lib(1 passed)cargo clippy -p dynamo-kv-router --all-targets -- -D warningscargo clippy -p dynamo-llm --lib -- -D warningsgit diff --checkBenchmark Results
cargo bench -p dynamo-kv-router --bench request_classifierschedules 128 tracked requests per batch and compares the disabled path with a trivial pass-through classifier in the same run.