Skip to content

refactor(llm): stop protocols importing preprocessor, admission, router and lora modules - #212

Draft
ishandhanani wants to merge 7 commits into
idhanani/dyn-llm-split-http-serverfrom
idhanani/dyn-llm-split-protocols-edges
Draft

ishandhanani wants to merge 7 commits into
idhanani/dyn-llm-split-http-serverfrom
idhanani/dyn-llm-split-protocols-edges

Conversation

@ishandhanani

@ishandhanani ishandhanani commented Nov 12, 2026 •

Copy link
Copy Markdown
Owner

Summary

Prepares step 4 of splitting lib/llm into smaller crates (the protocols crate). protocols cannot leave dynamo-llm while it imports modules that will stay above it. This PR cuts those imports in place: nothing moves out of the crate, no crate is created, and no Cargo.toml or lock file changes. Each edge family is one of the seven commits and builds on its own, so a single resolution can be dropped or redone without touching the others.

This PR is layer 3 of the 16-layer split stack (main <- #211 <- #209 <- #212 <- #213 <- #218 <- #219 <- #220 <- #221 <- #222 <- #223 <- #224 <- #225 <- #226 <- #227 <- #228 <- #229) and is based on #209.

Behaviour is unchanged: no status code, message, serialized field, validation rule, metric or log line differs. Moved code is cut-and-paste; the only new code is one four-line trait and its four-line impl (the kv_router row).

Edge out of protocols Files before Resolution Files after
engines::ValidateRequest 5 (+9 test-only) The trait every request type implements moves to protocols.rs. engines keeps the engine plumbing (echo engines, ValidateEngine, EngineDispatcher, adapters) and imports the trait downward. 5 lines moved, 24 path updates. 0
lora::runtime::RuntimeLoraSourceUri 1 Imported from dynamo-lora, the crate that owns it, instead of through the crate::lora facade. 1 line. 0
tokenizers::traits::* (crate-root alias of dynamo-tokenizers) 1 Imported from dynamo-tokenizers directly. 8 lines. 0
kv_router::RequestAttemptRelease 1 Inversion. MigrationState (frontend-only, #[serde(skip)] on PreprocessedRequest) parks a Box<dyn PrefillBookingRelease>; the one-method trait is defined beside it and implemented by RequestAttemptRelease in kv_router. 19 lines added, 6 removed. 0
admission_gate::{AdmissionRequest, AdmissionResponse, ResponseOutput} 2 impl AdmissionRequest for PreprocessedRequest, impl AdmissionResponse for LLMEngineOutput and their 2 tests move next to the traits in admission_gate. 126 lines moved. 0
preprocessor::structural_tag::is_beam_parser 1 Predicate over parser names moves to protocols::openai::chat_completions::tool_parser_v2. 13 lines moved. 0
preprocessor::tool_choice::convert_tools 1 (test-only) Pure ChatCompletionTool to ToolDefinition mapping moves to protocols::openai::tools. 11 lines moved. 0
preprocessor::prompt::MediaRequestExt 1 impl MediaRequestExt for UnifiedRequest moves to preprocessor::prompt, beside the trait and its other impls. 6 lines moved. 0
preprocessor::OpenAIPreprocessor (and model_card::ModelDeploymentCard in the same tests) 3 (test-only) The 3 tests that drive the preprocessor move to the preprocessor; bodies unchanged. About 155 lines moved. 0
OAIChatLikeRequest impls for NvCreateChatCompletionRequest and NvCreateCompletionRequest (no import: protocols::unified and the Responses tests call impls that lived in preprocessor::prompt) 1 (+3 test-only) Both impl blocks move to protocols::openai::renderer. 143 lines moved. 0
preprocessor::media::RdmaMediaDataDescriptor 1 Left alone. MultimodalData::Decoded in protocols::common::preprocessor still names the descriptor defined in preprocessor::media. Moving the descriptor types into protocols would make a protocols crate depend on dynamo-memory, so where they live is decided separately. 1
http::service::metrics::WORKER_LAST_* gauges 0 Already cut by #209 below: protocols::common::timing imports the gauges from dynamo-http-server. 0
local_model::runtime_config::*, frontend_config::DISABLE_OPENAI_STRICT_SCHEMA_VALIDATION 2 (+2 test-only), 2 Left alone. These modules move below protocols with the model vocabulary. unchanged

Imports leaving {protocols, types} after this PR: local_model::runtime_config (2 files, 2 test-only), frontend_config (2 files), preprocessor::media::RdmaMediaDataDescriptor (1 file), and the test-only crate::test_utils path include of lib/runtime/src/test_utils.rs (2 files).

Moved tests keep their leaf names. Path changes:

  • protocols::common::preprocessor::tests::worker_ingress_reads_routing_dp_rank_from_the_body -> admission_gate::tests::
  • protocols::common::llm_backend::tests::engine_outputs_classify_their_tokens_and_completion_independently -> admission_gate::tests::
  • protocols::anthropic::types::tests::test_tool_choice_conversion_preserves_request_level_parser_policy -> preprocessor::tool_choice::tests::
  • protocols::anthropic::stream_converter::tests::test_stream_stop_sequence_survives_tool_parser_split -> preprocessor::tests::
  • protocols::openai::responses::tests::native_tools::named_tool_rendering_policy_matches_the_production_preprocessor -> preprocessor::tests::

Validation

The base is #209 at aa261fc (on #211, on upstream main 8e7d29e). Before and after-runs used the same remote 32 vCPU builder.

The numbers below were measured on that base. The stack has since been rebased onto upstream main d20e045; after the rebase this layer's head passed cargo check --workspace --all-targets, cargo check --locked in lib/bindings/python, cargo metadata --locked for the four lock files, cargo fmt --all -- --check and the Anthropic admission tests, and the top of the stack passed the whole-workspace cargo test --locked run (10,022 passed, 0 failed, 87 ignored), clippy, and the bindings, machete, pre-commit, codeowners and OpenAPI checks.

  • cargo test -p dynamo-llm -p dynamo-http-server -p dynamo-sglang-generate -p dynamo-vllm-generate -p dynamo-generate-format-catalog -p dynamo-global-frontend --no-fail-fast plus cargo test -p dynamo-error --all-features: 65 test binaries, 4,770 passed, 0 failed, 33 ignored, before and after. All 65 per-binary result lines are identical (dynamo-llm: 47 binaries, 4,552 passed, lib 3,928).
  • cargo test -p dynamo-llm --lib -- --list: 3,943 names before and after. The only differences are the five path changes above; no name disappeared or appeared in any binary.
  • cargo check --workspace --all-targets and cargo check in lib/bindings/python pass at each of the 7 commits.
  • cargo clippy --no-deps --workspace --all-targets -- -D warnings: exit 0.
  • cargo check --locked in lib/bindings/python passes; cargo metadata --locked passes for the root, lib/bindings/python, lib/bindings/kvbm and lib/runtime/examples.
  • cargo machete in ., lib/bindings/python and lib/runtime/examples: no unused dependencies.
  • cargo fmt --all -- --check is clean; pre-commit run --from-ref aa261fc96a --to-ref HEAD passes.

Closes DYNO-218

…cols

Every request type in protocols implements ValidateRequest, so protocols
imported engines only for this trait. The trait now lives in protocols.rs
next to ContentProvider; engines keeps the engine plumbing (echo engines,
ValidateEngine, EngineDispatcher, adapters) and imports the trait downward.
No behaviour change.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
RoutingHints carries the request-time LoRA source URI, a type dynamo-lora
owns. protocols reached it through the crate::lora facade, which also holds
frontend discovery, metrics and event-transport glue. Import it from the
producer crate instead so protocols no longer depends on the lora module.
No behaviour change.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…ocols

crate::tokenizers is an alias for the dynamo-tokenizers crate declared in
lib.rs. The completions delta generator names the crate directly so
protocols does not reach through the dynamo-llm crate root. No behaviour
change.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
MigrationState, the frontend-only state carried on PreprocessedRequest,
parked the router's RequestAttemptRelease so a committed retry could free
the previous attempt's remote prefill booking. That made protocols import
kv_router. MigrationState now holds a boxed PrefillBookingRelease, a
one-method trait defined beside it and implemented by RequestAttemptRelease
in kv_router; the prefill router boxes the handle when it parks it and the
migration retry path calls release() as before. No behaviour change.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…dmission_gate

PreprocessedRequest implemented AdmissionRequest and LLMEngineOutput
implemented AdmissionResponse inside protocols, so the wire types imported
the worker admission gate. The two impls and their two tests move, unchanged,
next to the traits in admission_gate, which already sits above protocols.

Moved tests (leaf names unchanged):
- protocols::common::preprocessor::tests::worker_ingress_reads_routing_dp_rank_from_the_body
  -> admission_gate::tests::worker_ingress_reads_routing_dp_rank_from_the_body
- protocols::common::llm_backend::tests::engine_outputs_classify_their_tokens_and_completion_independently
  -> admission_gate::tests::engine_outputs_classify_their_tokens_and_completion_independently

No behaviour change.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…s and tests

Four imports from protocols into preprocessor are cut, each at its owner:

- is_beam_parser, a predicate over parser names used by the parser-version
  check, moves from preprocessor::structural_tag to
  protocols::openai::chat_completions::tool_parser_v2 beside the unified
  parser families; the structural-tag and tool-choice policies import it
  from there.
- convert_tools, a pure ChatCompletionTool to ToolDefinition mapping, moves
  from preprocessor::tool_choice to protocols::openai::tools.
- impl MediaRequestExt for UnifiedRequest moves to preprocessor::prompt,
  beside the trait and its other two impls.
- Three tests that drive OpenAIPreprocessor move to the preprocessor. Their
  bodies are unchanged; the fixtures they share with tests that stay in
  protocols become pub(crate) under cfg(test).

Moved tests (leaf names unchanged):
- protocols::anthropic::types::tests::test_tool_choice_conversion_preserves_request_level_parser_policy
  -> preprocessor::tool_choice::tests::test_tool_choice_conversion_preserves_request_level_parser_policy
- protocols::anthropic::stream_converter::tests::test_stream_stop_sequence_survives_tool_parser_split
  -> preprocessor::tests::test_stream_stop_sequence_survives_tool_parser_split
- protocols::openai::responses::tests::native_tools::named_tool_rendering_policy_matches_the_production_preprocessor
  -> preprocessor::tests::named_tool_rendering_policy_matches_the_production_preprocessor

No behaviour change.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…in protocols

The dynamo_renderer::OAIChatLikeRequest impls for
NvCreateChatCompletionRequest and NvCreateCompletionRequest lived in
preprocessor::prompt, while protocols::unified::UnifiedRequest delegates to
them and the Responses tests in protocols render requests through them. That
is a dependency from protocols on preprocessor that no import shows, and
once protocols is its own crate the impls of a foreign trait for its types
can only live there. Both impl blocks move, unchanged, to
protocols::openai::renderer. MediaRequestExt and its impls stay in
preprocessor::prompt. No behaviour change.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani
ishandhanani force-pushed the idhanani/dyn-llm-split-http-server branch from 8ddd6a1 to 5ec41d2 Compare November 16, 2026 20:34
@ishandhanani
ishandhanani force-pushed the idhanani/dyn-llm-split-protocols-edges branch from b30390b to 181d430 Compare November 16, 2026 20:34

This branch had an error being deployed

1 failed deployment
external_collaborator — 181d430f Deployed Nov 16, 2026 by ishandhanani via ok-to-test #751
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant