Repository navigation
refactor(llm): move protocols into dynamo-frontend-protocols - #218
Draft
ishandhanani wants to merge 6 commits into
Draft
ishandhanani wants to merge 6 commits into
ishandhanani wants to merge 6 commits into
Conversation
ishandhanani
had a problem deploying
to
external_collaborator
November 15, 2026 13:30 — with
GitHub Actions
Failure
ishandhanani
force-pushed
the
idhanani/dyn-llm-split-model-vocabulary
branch
from
November 16, 2026 07:07
73e7ddd to
d7f7c4d
Compare
ishandhanani
force-pushed
the
idhanani/dyno-220-frontend-protocols
branch
from
November 16, 2026 07:07
95f76a9 to
5b3b831
Compare
ishandhanani
had a problem deploying
to
external_collaborator
November 16, 2026 07:07 — with
GitHub Actions
Failure
ishandhanani
had a problem deploying
to
external_collaborator
November 16, 2026 07:07 — with
GitHub Actions
Failure
This was referenced Nov 16, 2026
Step 1 of DYNO-220, which moves lib/llm/src/protocols into its own crate. `MultimodalData::Decoded` held the media loader's `RdmaMediaDataDescriptor`, so protocols imported `preprocessor::media` and, through it, dynamo-memory (NIXL), the `image` crate and the decoders. protocols now owns `common::decoded_media::DecodedMediaDescriptor`, a wire struct of serde and primitive types with the same field names, order and serde attributes. The NIXL memory type is mirrored as an enum (the match in the conversion is exhaustive), the `image` crate's `#[non_exhaustive]` format and colour enums cross as their serde variant names, and the registered storage is kept alive through a type-erased `Arc<dyn Any + Send + Sync>` that is never serialized. The media loader keeps building and caching its typed descriptor; the preprocessor converts it with `TryFrom` when it fills the request slot. Nothing reads the keep-alive back: migration clones it into the first-response guard, and the routing code that reads the registered buffer does so on the typed descriptor before the conversion. A test in the media loader serializes the typed and the wire form of the same values (every NIXL memory type, every `image` format and colour type, with and without metadata and content hash) with JSON and both MessagePack encodings and compares the bytes, and reads each form back from the other's bytes. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…gRPC service Step 2 of DYNO-220. The gRPC tensor service held an inherent `impl tensor::Tensor` with the four functions that fill a tensor from KServe request contents. Once the tensor protocol lives in another crate, Rust's orphan rule forbids an inherent impl here. The functions are the gRPC service's behaviour (they take `InferTensorContents` and return `tonic::Status`), so they stay in the service as a private `KserveTensorContents` extension trait implemented for `Tensor`. Bodies and call sites are unchanged. The `TryFrom<ModelInferRequest>` impls for protocol types in this module and in `grpc/service/openai.rs` already satisfy the orphan rule, because the request type is local. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…xports Step 3 of DYNO-220. `preprocessor` re-exported `PreprocessedRequest`, `BackendOutput`, `PreprocessedEmbeddingRequest`, `LLMMetricAnnotation` and the two annotation names from protocols, and `discovery` re-exported the `WORKER_TYPE_PREFILL` / `WORKER_TYPE_DECODE` label values. After protocols moves to its own crate these would be re-exports of another crate's items, which the split does not keep. Callers now import the items from `protocols::common` (`preprocessor`, `llm_backend`, `metrics`, `timing`): the routing host, session affinity, the HTTP service and metrics, the entrypoint, the integration tests, the Python bindings and the EPP. `preprocessor` keeps private imports for its own use. No item moves and no behaviour changes. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Step 4 of DYNO-220. About 190 items in protocols are `pub(crate)`. The preprocessor, the HTTP and gRPC services, the router, migration and request tracing use 81 of them, which stops working once protocols is a separate crate. Those 81 become `pub`, and nothing else: the list comes from compiling the moved crate and widening only what a caller outside it failed to reach. It covers the `unified`, `delta_common`, `custom_tools`, `tool_call_merge` and `unified_parser` modules, the parser selection and tool validation helpers the preprocessor calls, the migration and request lifecycle state on `PreprocessedRequest` (`MigrationState`, `PrefillBookingRelease`, the `migration_state`, `staged_kv_cleanup` and `jail_seed` fields), and a few accessors. Inside dynamo-llm the change has no effect. The preprocessor names `ParserVersion` through dynamo-runtime, which defines it, instead of through the crate-private re-export in `tool_parser_v2`. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…ules Step 5 of DYNO-220. The preprocessor's tests reuse helpers defined inside protocols' `#[cfg(test)] mod tests` modules: the unified parser's eight-family tool-call inputs, the Anthropic stream converter's `text_chunk` and `sse_values`, and the Responses template fixtures. Another crate cannot see into a `cfg(test)` module, so after the move they need one place that a `testing` cargo feature can export. Each set moves, unchanged, out of its `mod tests` into a sibling `test_fixtures` module (`unified_parser`, `anthropic::stream_converter`, `openai::responses`), and the test modules are private again. Three test-only parser entry points the preprocessor's tests call (`ChoiceState::new`, `parse_complete_with_policy`, `apply_stream_with_constraint`) become `pub` under the same `cfg(test)`. Here everything stays `cfg(test)`; the move commit switches `pub` items under `cfg(test)` to `cfg(any(test, feature = "testing"))`. No test is added, removed or renamed. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Step 6 of DYNO-220 and layer 5 of splitting lib/llm into smaller crates. With the upward edges cut by the preceding commits, the request and response protocols move into a new crate below dynamo-llm: - lib/llm/src/protocols.rs becomes the crate root and the modules under lib/llm/src/protocols (openai, anthropic, common, tensor, codec, unified, agents, systemone, response_merge) become its top-level modules, so `dynamo_llm::protocols::openai::X` is now `dynamo_frontend_protocols::openai::X` - lib/llm/src/types.rs moves with them as `dynamo_frontend_protocols::types`: it holds only the engine type aliases over protocol messages and the runtime's engine traits. engines.rs (the echo, validating and dispatching engines) stays in dynamo-llm The 86 files move unchanged apart from import paths (`crate::protocols::` becomes `crate::`), test data paths, which now point at lib/llm/tests/data, the two test paths the isolated-process tests select themselves by, and the `cfg` of the shared test fixtures, which a new `testing` feature exports to dynamo-llm's tests. dynamo-llm keeps no re-exports: every caller in dynamo-llm, its tests and benches, the Python bindings, backend-common, the sidecars, the generate-format crates, global-frontend and the EPP imports dynamo_frontend_protocols directly. dynamo-llm forwards its media-ffmpeg feature to the new crate, which gates the video variant of the decoded-media metadata, and drops percent-encoding, which only the moved code used. The new crate depends on dynamo-runtime, dynamo-model-card, dynamo-http-server and the router protocol crates. It does not depend on dynamo-llm, dynamo-memory, NIXL or CUDA. Codeowners routes lib/frontend-protocols to the areas that own the files today (types.rs keeps its runtime owners), and the docs, CI path filters, scripts and comments that cite the moved paths are updated. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
ishandhanani
force-pushed
the
idhanani/dyn-llm-split-model-vocabulary
branch
from
November 16, 2026 20:34
d7f7c4d to
41892d5
Compare
ishandhanani
force-pushed
the
idhanani/dyno-220-frontend-protocols
branch
from
November 16, 2026 20:34
5b3b831 to
216f947
Compare
ishandhanani
had a problem deploying
to
external_collaborator
November 16, 2026 20:34 — with
GitHub Actions
Failure
ishandhanani
had a problem deploying
to
external_collaborator
November 16, 2026 20:34 — with
GitHub Actions
Failure
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Layer 5 of the lib/llm split stack (DYNO-220). Base: #213 (model vocabulary), so this PR shows only this layer. It moves
lib/llm/src/protocolsandlib/llm/src/types.rsout ofdynamo-llminto a new workspace crate,dynamo-frontend-protocols, atlib/frontend-protocols.What moved
protocols.rsbecomes the crate root;openai,anthropic,common,tensor,codec,unified,agents,systemoneandresponse_mergebecome its top-level modules.dynamo_llm::protocols::openai::Xis nowdynamo_frontend_protocols::openai::X, and items that lived atdynamo_llm::protocols::Xlive at the crate root.types.rstravels asdynamo_frontend_protocols::types: it holds only the engine type aliases over protocol messages and the runtime's engine traits.engines.rs(echo, validating and dispatching engines) is engine plumbing and stays indynamo-llm.git mv. Inside the moved files only import paths, test data paths, two isolated-test path strings and thecfgof the shared test fixtures change (+282/-297 lines).lib/llm/srcgoes from 341,189 to 271,733 lines of Rust.dynamo-llmhas nopub use dynamo_frontend_protocols.Commits
Five in-place cuts inside
dynamo-llm, each building and passing on its own, then one move commit.MultimodalData::Decodedcarried the media loader'sRdmaMediaDataDescriptor, which pulleddynamo-memory(NIXL, CUDA),imageand the decoders into protocolscommon::decoded_media::DecodedMediaDescriptor, a wire struct of serde and primitive types with the same field names, order and serde attributes. The NIXL memory type is mirrored as an enum, theimagecrate's#[non_exhaustive]format and colour enums cross as their serde variant names, and the registered storage is held asOption<Arc<dyn Any + Send + Sync>>(never serialized). The loader keeps its typed descriptor and the preprocessor converts withTryFromwhen it fills the request slotdynamo-memoryandimage); adynamo-memorydependency behind a featureimpl tensor::Tensorin the gRPC tensor serviceKserveTensorContentsextension trait in the service; bodies and call sites unchanged. It was the only such impl: theTryFrom<ModelInferRequest>impls are legal because the request type is local, and the other impls on protocol types use traitsdynamo-llmownstonic::Statusconversions into protocolspreprocessoranddiscoveryre-exported protocol items (PreprocessedRequest,BackendOutput,PreprocessedEmbeddingRequest,LLMMetricAnnotation, the annotation names,WORKER_TYPE_*)protocols::commonpub(crate)occurrences in protocols, some used by higher modulespub: exactly the ones a caller outside the moved crate failed to reach when compiled. The preprocessor namesParserVersionthroughdynamo-runtimerather than through a widened re-exportpubcfg(test)modules that the preprocessor's tests reuse;crate::test_utils; fixture paths intolib/llm/tests/datatest_fixturesmodules (unified_parser,anthropic::stream_converter,openai::responses), and three test-only parser entry points becomepub; the new crate exports them behind atestingfeature that onlydynamo-llm's dev-dependencies enable. The new crate includeslib/runtime/src/test_utils.rsby#[path], asdynamo-llmdoes. Test data stays inlib/llm/tests/dataand the moved tests point at it, asdynamo-model-carddoesmedia-ffmpeg(gates the video variant of the decoded-media metadata, as the same feature gates it in the loader;dynamo-llm/media-ffmpegforwards it) andtesting. Every existingdynamo-llm/<feature>line keeps workingCallers
lib/llm(85 insrc, the rest tests and benches) and 32 outside it now importdynamo_frontend_protocols: the Python bindings, backend-common, the vLLM, SGLang and TensorRT-LLM sidecars and their testkit, both generate-format crates and the catalog tests, global-frontend and the EPP. Twelve manifests gain the dependency (dynamo-llmincluded);dynamo-llmdropspercent-encoding, which only the moved code used.labeler.yml,protocol-assessment.yml, the media-protocols pre-commit hook,.gitattributes) and comments that cite the moved paths are updated. The protocol field handling page cites no moved path and is unchanged.areas.yamlrouteslib/frontend-protocols/to the frontend area with the same shared entries as before (agents, multimodal, diffusion),types.rskeeps its runtime owners, andCODEOWNERSis regenerated.lib/frontend-protocols/AGENTS.md(with itsCLAUDE.mdtwin) states what the crate owns, what it must not depend on, and how to check it.Dependency graph
cargo tree -p dynamo-frontend-protocols | grep -c dynamo-llmis 0.dynamo-memory,nixl-sysandcudarcdo not appear in its tree.imageappears only transitively, through the publishedopenai-harmonythatdynamo-parsersalready brings in.dynamo-runtime,dynamo-model-card,dynamo-http-server,dynamo-admission,dynamo-scheduler,dynamo-router-protocol,dynamo-kv-hashing,dynamo-kv-hints,dynamo-lora, and the publisheddynamo-protocols,dynamo-parsers,dynamo-parsers-v2,dynamo-renderer,dynamo-tokenizers.Behaviour
imageformat and colour type, with and without metadata and content hash, in JSON and both MessagePack encodings, and reads each form from the other's bytes.imagecrate does not know. TheDebugoutput of the storage keep-alive is nowAny { .. }.Validation
Cargo commands ran on a remote builder with the repository toolchain (1.96.1).
cargo test -p dynamo-llm -p dynamo-model-card -p dynamo-http-server -p dynamo-error --no-fail-fast: 53 binaries, 4658 passed, 0 failed, 33 ignored.-p dynamo-frontend-protocols: 55 binaries, 4663 passed, 0 failed, 33 ignored. The two extra binaries are the new crate's unit and doc tests. The five extra tests are the wire compatibility test and the fourtest_utilsself-tests the new crate now also includes.cargo test -- --list: all 4691 baseline names exist at head; 3543 keep binary and path, 1148 moved fromdynamo-llm'sprotocols::to the same path indynamo-frontend-protocolswithout the prefix; none is missing.dynamo-llmunit tests go from 3771 passed and 15 ignored to 2634 and 5; the new crate has 1142 and 10.cargo run -p dynamo-llm --bin generate-frontend-openapiwritesdocs/frontends/openapi.jsonwith SHA-256706f15d951891703fa911b9c659ac54992fa7144365ce4cd6b2a179a7630ede5(148,295 bytes) at baseline and at head.cargo check -p dynamo-llm --all-targetsand the tests of the touched modules pass (273, 71, 991 and 1530 tests for the first four; the fifth ran the wholedynamo-llmsuite, workspace check and clippy).cargo check --workspace --all-targets,cargo clippy --no-deps --workspace --all-targets -- -D warnings,cargo fmt --all -- --check, andcargo check --lockedinlib/bindings/pythonpass.cargo check -p dynamo-frontend-protocols --all-targetspasses alone and with--features media-ffmpeg,testing(also under clippy), andcargo test -p dynamo-frontend-protocolspasses alone.dynamo-backend-common,dynamo-sglang-generate,dynamo-vllm-generate,dynamo-generate-format-catalog,dynamo-global-frontend,dynamo-ext-proc, the three sidecars anddynamo-sidecar-testkit): 38 test binaries, 952 passed, 0 failed, 12 ignored. Eleven testkit tests need thedynamo-lpx-sidecarbinary and pass once it is built.cargo macheteat the root,lib/bindings/pythonandlib/runtime/examplesreports no unused dependency.cargo metadata --lockedpasses for the four lock files (root,lib/bindings/python,lib/bindings/kvbm,lib/runtime/examples).pre-commit run --from-ref idhanani/dyn-llm-split-model-vocabulary --to-ref HEADpasses, andbuild_codeowners.py --strictpasses withCODEOWNERSmatchingemit_codeowners.py.cargo check -p dynamo-llm --features mm-routing --all-targetspasses.--features media-ffmpeg,mm-routingcould not be built: the builder has no FFmpeg libraries (the baseline fails the same way), so themedia-ffmpegarm of the descriptor conversion indynamo-llmis reviewed but not compiled. The new crate's ownmedia-ffmpegcode compiles.Closes DYNO-220