Repository navigation
refactor(llm): stop protocols importing preprocessor, admission, router and lora modules - #212
Draft
ishandhanani wants to merge 7 commits into
Draft
ishandhanani wants to merge 7 commits into
ishandhanani wants to merge 7 commits into
Conversation
ishandhanani
had a problem deploying
to
external_collaborator
November 12, 2026 08:37 — with
GitHub Actions
Failure
ishandhanani
force-pushed
the
idhanani/dyn-llm-split-protocols-edges
branch
from
November 15, 2026 10:57
48094d4 to
3920aaf
Compare
ishandhanani
had a problem deploying
to
external_collaborator
November 15, 2026 10:57 — with
GitHub Actions
Failure
ishandhanani
changed the base branch from
main
to
idhanani/dyn-llm-split-http-server
November 15, 2026 10:57
This was referenced Nov 15, 2026
ishandhanani
force-pushed
the
idhanani/dyn-llm-split-protocols-edges
branch
from
November 16, 2026 07:07
3920aaf to
b30390b
Compare
ishandhanani
had a problem deploying
to
external_collaborator
November 16, 2026 07:07 — with
GitHub Actions
Failure
…cols Every request type in protocols implements ValidateRequest, so protocols imported engines only for this trait. The trait now lives in protocols.rs next to ContentProvider; engines keeps the engine plumbing (echo engines, ValidateEngine, EngineDispatcher, adapters) and imports the trait downward. No behaviour change. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
RoutingHints carries the request-time LoRA source URI, a type dynamo-lora owns. protocols reached it through the crate::lora facade, which also holds frontend discovery, metrics and event-transport glue. Import it from the producer crate instead so protocols no longer depends on the lora module. No behaviour change. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…ocols crate::tokenizers is an alias for the dynamo-tokenizers crate declared in lib.rs. The completions delta generator names the crate directly so protocols does not reach through the dynamo-llm crate root. No behaviour change. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
MigrationState, the frontend-only state carried on PreprocessedRequest, parked the router's RequestAttemptRelease so a committed retry could free the previous attempt's remote prefill booking. That made protocols import kv_router. MigrationState now holds a boxed PrefillBookingRelease, a one-method trait defined beside it and implemented by RequestAttemptRelease in kv_router; the prefill router boxes the handle when it parks it and the migration retry path calls release() as before. No behaviour change. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…dmission_gate PreprocessedRequest implemented AdmissionRequest and LLMEngineOutput implemented AdmissionResponse inside protocols, so the wire types imported the worker admission gate. The two impls and their two tests move, unchanged, next to the traits in admission_gate, which already sits above protocols. Moved tests (leaf names unchanged): - protocols::common::preprocessor::tests::worker_ingress_reads_routing_dp_rank_from_the_body -> admission_gate::tests::worker_ingress_reads_routing_dp_rank_from_the_body - protocols::common::llm_backend::tests::engine_outputs_classify_their_tokens_and_completion_independently -> admission_gate::tests::engine_outputs_classify_their_tokens_and_completion_independently No behaviour change. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…s and tests Four imports from protocols into preprocessor are cut, each at its owner: - is_beam_parser, a predicate over parser names used by the parser-version check, moves from preprocessor::structural_tag to protocols::openai::chat_completions::tool_parser_v2 beside the unified parser families; the structural-tag and tool-choice policies import it from there. - convert_tools, a pure ChatCompletionTool to ToolDefinition mapping, moves from preprocessor::tool_choice to protocols::openai::tools. - impl MediaRequestExt for UnifiedRequest moves to preprocessor::prompt, beside the trait and its other two impls. - Three tests that drive OpenAIPreprocessor move to the preprocessor. Their bodies are unchanged; the fixtures they share with tests that stay in protocols become pub(crate) under cfg(test). Moved tests (leaf names unchanged): - protocols::anthropic::types::tests::test_tool_choice_conversion_preserves_request_level_parser_policy -> preprocessor::tool_choice::tests::test_tool_choice_conversion_preserves_request_level_parser_policy - protocols::anthropic::stream_converter::tests::test_stream_stop_sequence_survives_tool_parser_split -> preprocessor::tests::test_stream_stop_sequence_survives_tool_parser_split - protocols::openai::responses::tests::native_tools::named_tool_rendering_policy_matches_the_production_preprocessor -> preprocessor::tests::named_tool_rendering_policy_matches_the_production_preprocessor No behaviour change. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…in protocols The dynamo_renderer::OAIChatLikeRequest impls for NvCreateChatCompletionRequest and NvCreateCompletionRequest lived in preprocessor::prompt, while protocols::unified::UnifiedRequest delegates to them and the Responses tests in protocols render requests through them. That is a dependency from protocols on preprocessor that no import shows, and once protocols is its own crate the impls of a foreign trait for its types can only live there. Both impl blocks move, unchanged, to protocols::openai::renderer. MediaRequestExt and its impls stay in preprocessor::prompt. No behaviour change. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
ishandhanani
force-pushed
the
idhanani/dyn-llm-split-http-server
branch
from
November 16, 2026 20:34
8ddd6a1 to
5ec41d2
Compare
ishandhanani
force-pushed
the
idhanani/dyn-llm-split-protocols-edges
branch
from
November 16, 2026 20:34
b30390b to
181d430
Compare
ishandhanani
had a problem deploying
to
external_collaborator
November 16, 2026 20:34 — with
GitHub Actions
Failure
ishandhanani
had a problem deploying
to
external_collaborator
November 16, 2026 20:34 — with
GitHub Actions
Failure
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Prepares step 4 of splitting
lib/llminto smaller crates (theprotocolscrate).protocolscannot leavedynamo-llmwhile it imports modules that will stay above it. This PR cuts those imports in place: nothing moves out of the crate, no crate is created, and noCargo.tomlor lock file changes. Each edge family is one of the seven commits and builds on its own, so a single resolution can be dropped or redone without touching the others.This PR is layer 3 of the 16-layer split stack (
main<- #211 <- #209 <- #212 <- #213 <- #218 <- #219 <- #220 <- #221 <- #222 <- #223 <- #224 <- #225 <- #226 <- #227 <- #228 <- #229) and is based on #209.Behaviour is unchanged: no status code, message, serialized field, validation rule, metric or log line differs. Moved code is cut-and-paste; the only new code is one four-line trait and its four-line impl (the
kv_routerrow).protocolsengines::ValidateRequestprotocols.rs.engineskeeps the engine plumbing (echo engines,ValidateEngine,EngineDispatcher, adapters) and imports the trait downward. 5 lines moved, 24 path updates.lora::runtime::RuntimeLoraSourceUridynamo-lora, the crate that owns it, instead of through thecrate::lorafacade. 1 line.tokenizers::traits::*(crate-root alias ofdynamo-tokenizers)dynamo-tokenizersdirectly. 8 lines.kv_router::RequestAttemptReleaseMigrationState(frontend-only,#[serde(skip)]onPreprocessedRequest) parks aBox<dyn PrefillBookingRelease>; the one-method trait is defined beside it and implemented byRequestAttemptReleaseinkv_router. 19 lines added, 6 removed.admission_gate::{AdmissionRequest, AdmissionResponse, ResponseOutput}impl AdmissionRequest for PreprocessedRequest,impl AdmissionResponse for LLMEngineOutputand their 2 tests move next to the traits inadmission_gate. 126 lines moved.preprocessor::structural_tag::is_beam_parserprotocols::openai::chat_completions::tool_parser_v2. 13 lines moved.preprocessor::tool_choice::convert_toolsChatCompletionTooltoToolDefinitionmapping moves toprotocols::openai::tools. 11 lines moved.preprocessor::prompt::MediaRequestExtimpl MediaRequestExt for UnifiedRequestmoves topreprocessor::prompt, beside the trait and its other impls. 6 lines moved.preprocessor::OpenAIPreprocessor(andmodel_card::ModelDeploymentCardin the same tests)OAIChatLikeRequestimpls forNvCreateChatCompletionRequestandNvCreateCompletionRequest(no import:protocols::unifiedand the Responses tests call impls that lived inpreprocessor::prompt)protocols::openai::renderer. 143 lines moved.preprocessor::media::RdmaMediaDataDescriptorMultimodalData::Decodedinprotocols::common::preprocessorstill names the descriptor defined inpreprocessor::media. Moving the descriptor types intoprotocolswould make a protocols crate depend ondynamo-memory, so where they live is decided separately.http::service::metrics::WORKER_LAST_*gaugesprotocols::common::timingimports the gauges fromdynamo-http-server.local_model::runtime_config::*,frontend_config::DISABLE_OPENAI_STRICT_SCHEMA_VALIDATIONprotocolswith the model vocabulary.Imports leaving
{protocols, types}after this PR:local_model::runtime_config(2 files, 2 test-only),frontend_config(2 files),preprocessor::media::RdmaMediaDataDescriptor(1 file), and the test-onlycrate::test_utilspath include oflib/runtime/src/test_utils.rs(2 files).Moved tests keep their leaf names. Path changes:
protocols::common::preprocessor::tests::worker_ingress_reads_routing_dp_rank_from_the_body->admission_gate::tests::protocols::common::llm_backend::tests::engine_outputs_classify_their_tokens_and_completion_independently->admission_gate::tests::protocols::anthropic::types::tests::test_tool_choice_conversion_preserves_request_level_parser_policy->preprocessor::tool_choice::tests::protocols::anthropic::stream_converter::tests::test_stream_stop_sequence_survives_tool_parser_split->preprocessor::tests::protocols::openai::responses::tests::native_tools::named_tool_rendering_policy_matches_the_production_preprocessor->preprocessor::tests::Validation
The base is #209 at aa261fc (on #211, on upstream
main8e7d29e). Before and after-runs used the same remote 32 vCPU builder.The numbers below were measured on that base. The stack has since been rebased onto upstream
maind20e045; after the rebase this layer's head passedcargo check --workspace --all-targets,cargo check --lockedinlib/bindings/python,cargo metadata --lockedfor the four lock files,cargo fmt --all -- --checkand the Anthropic admission tests, and the top of the stack passed the whole-workspacecargo test --lockedrun (10,022 passed, 0 failed, 87 ignored), clippy, and the bindings, machete, pre-commit, codeowners and OpenAPI checks.cargo test -p dynamo-llm -p dynamo-http-server -p dynamo-sglang-generate -p dynamo-vllm-generate -p dynamo-generate-format-catalog -p dynamo-global-frontend --no-fail-fastpluscargo test -p dynamo-error --all-features: 65 test binaries, 4,770 passed, 0 failed, 33 ignored, before and after. All 65 per-binary result lines are identical (dynamo-llm: 47 binaries, 4,552 passed, lib 3,928).cargo test -p dynamo-llm --lib -- --list: 3,943 names before and after. The only differences are the five path changes above; no name disappeared or appeared in any binary.cargo check --workspace --all-targetsandcargo checkinlib/bindings/pythonpass at each of the 7 commits.cargo clippy --no-deps --workspace --all-targets -- -D warnings: exit 0.cargo check --lockedinlib/bindings/pythonpasses;cargo metadata --lockedpasses for the root,lib/bindings/python,lib/bindings/kvbmandlib/runtime/examples.cargo machetein.,lib/bindings/pythonandlib/runtime/examples: no unused dependencies.cargo fmt --all -- --checkis clean;pre-commit run --from-ref aa261fc96a --to-ref HEADpasses.Closes DYNO-218