Skip to content

feat(router): connect request classifier lifecycle - #13801

Closed
ishandhanani wants to merge 1 commit into
idhanani/dyn-classify-orderfrom
idhanani/dyn-classify-lifecycle
Closed

ishandhanani wants to merge 1 commit into
idhanani/dyn-classify-orderfrom
idhanani/dyn-classify-lifecycle

Conversation

@ishandhanani

@ishandhanani ishandhanani commented Aug 26, 2026 •

Copy link
Copy Markdown
Contributor

Connects the classifier lifecycle from #13800 to the production KV-router request and response paths. This lets pending classifiers learn when their logical requests are sent, begin responding, complete, or abort across retries without changing worker-selection policy.

Related DEP: #13891. Stack: #13800 → #13801 → #13802.

How This Was Implemented

  • Register one lifecycle for each tracked logical request and carry its RAII handle from routing through dispatch and response handling.
  • Emit Sent after dispatch, Responding on the first response, Completed with authoritative context tokens when available, and Aborted for cancellation, errors, or abandoned streams.
  • Keep classification outside native session-affinity retries so a stale affinity target is retried without classifying twice or emitting an intermediate abort.
  • Preserve the lifecycle across migration retries while releasing attempt-local scheduler state before the next attempt.
  • Keep advisory query-only requests outside classifier and lifecycle accounting.
  • Add a Criterion benchmark for disabled versus trivial pass-through classifier paths.
Walkthrough

Mental model

The scheduler owns classification and queueing, while the existing response guard owns the only state that can determine terminal request outcome. A small lifecycle handle connects those owners without giving the classifier dispatch authority.

sequenceDiagram
    participant R as RoutingHost
    participant C as Classifier runtime
    participant S as Existing scheduler and selector
    participant G as Response guard
    R->>C: register logical request
    R->>S: classify, schedule, select
    S-->>G: worker plus lifecycle handle
    G->>C: Sent, then Responding
    G->>C: Completed or Aborted
Loading

Request lifecycle

Lifecycle registration occurs only for tracked requests. Selection records the chosen worker, addressed dispatch emits Sent, the first backend item emits Responding, and normal stream completion emits Completed. Backend usage metadata supplies total context tokens when present; otherwise the guard falls back to authoritative input tokens plus locally observed output tokens.

Retryable migration failures move the lifecycle into migration state after cleaning up the failed attempt. The next attempt reuses that handle and re-enters ordinary scheduling without calling classify() again; final failure emits one Aborted notification with the last known worker and typed error when available.

Boundaries and limitations

on_event() is synchronous and notification-only. Received remains available in the event vocabulary but is not emitted because the current transport has no authoritative receipt signal; this PR adds no affinity, placement, durable plugin state, mailbox, reconciliation protocol, or compatibility path.

Validation

  • cargo test -q -p dynamo-llm kv_router::routing_host::tests --lib (24 passed)
  • cargo test -q -p dynamo-llm test_retry_manager_generic_over_llm_engine_output --lib (1 passed)
  • cargo test -q -p dynamo-llm classifier_cache_inputs_retain_cache_providers --lib (1 passed)
  • cargo clippy -p dynamo-kv-router --all-targets -- -D warnings
  • cargo clippy -p dynamo-llm --lib -- -D warnings
  • git diff --check

Benchmark Results

cargo bench -p dynamo-kv-router --bench request_classifier schedules 128 tracked requests per batch and compares the disabled path with a trivial pass-through classifier in the same run.

Path Disabled Pass-through Delta
Classifier scheduling 23.945 µs/request 23.945 µs/request approximately 0%
Full lifecycle 24.414 µs/request 25.030 µs/request +0.616 µs/request (+2.52%)

@github-actions github-actions Bot added feat router Relates to routing, KV-aware routing, etc. labels Aug 26, 2026
@ishandhanani
ishandhanani force-pushed the idhanani/dyn-classify-lifecycle branch from 9a2310e to d60639c Compare August 26, 2026 01:52
@ishandhanani
ishandhanani force-pushed the idhanani/dyn-classify-lifecycle branch from a36450a to efde077 Compare August 26, 2026 02:30
@ishandhanani
ishandhanani force-pushed the idhanani/dyn-classify-lifecycle branch from efde077 to 9a5fc5d Compare August 26, 2026 08:26
@ishandhanani ishandhanani changed the title feat(router): supervise flow control lifecycle feat(router): connect request classifier lifecycle Aug 26, 2026
@ishandhanani
ishandhanani force-pushed the idhanani/dyn-classify-lifecycle branch 3 times, most recently from 6d94dff to 9a9e41f Compare August 26, 2026 21:50
@github-actions github-actions Bot added the frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` label Aug 26, 2026
@sara4dev
sara4dev force-pushed the idhanani/dyn-classify-lifecycle branch from 9a9e41f to 0dc6b84 Compare August 26, 2026 23:20
@ishandhanani
ishandhanani force-pushed the idhanani/dyn-classify-lifecycle branch from 0dc6b84 to 7986f11 Compare August 27, 2026 02:51
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani
ishandhanani force-pushed the idhanani/dyn-classify-lifecycle branch from 7986f11 to 811dc50 Compare August 27, 2026 03:18
@ishandhanani

Copy link
Copy Markdown
Contributor Author

Closing as superseded. We are restarting from current main using DEP #13891 as the guide and replacing this three-PR stack with two focused PRs: first the complete generic request-classifier and lifecycle seam with an independently pollable classify future, then the ThunderAgent algorithm built on that public seam.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feat frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` router Relates to routing, KV-aware routing, etc. size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant