Skip to content

refactor(llm): move protocols into dynamo-frontend-protocols - #218

Draft
ishandhanani wants to merge 6 commits into
idhanani/dyn-llm-split-model-vocabularyfrom
idhanani/dyno-220-frontend-protocols
Draft

ishandhanani wants to merge 6 commits into
idhanani/dyn-llm-split-model-vocabularyfrom
idhanani/dyno-220-frontend-protocols

Conversation

@ishandhanani

@ishandhanani ishandhanani commented Nov 15, 2026 •

Copy link
Copy Markdown
Owner

Summary

Layer 5 of the lib/llm split stack (DYNO-220). Base: #213 (model vocabulary), so this PR shows only this layer. It moves lib/llm/src/protocols and lib/llm/src/types.rs out of dynamo-llm into a new workspace crate, dynamo-frontend-protocols, at lib/frontend-protocols.

What moved

  • protocols.rs becomes the crate root; openai, anthropic, common, tensor, codec, unified, agents, systemone and response_merge become its top-level modules. dynamo_llm::protocols::openai::X is now dynamo_frontend_protocols::openai::X, and items that lived at dynamo_llm::protocols::X live at the crate root.
  • types.rs travels as dynamo_frontend_protocols::types: it holds only the engine type aliases over protocol messages and the runtime's engine traits. engines.rs (echo, validating and dispatching engines) is engine plumbing and stays in dynamo-llm.
  • 86 files, 71,237 lines (69,872 lines of Rust in 74 files, plus the media fixtures and schemas), moved with git mv. Inside the moved files only import paths, test data paths, two isolated-test path strings and the cfg of the shared test fixtures change (+282/-297 lines). lib/llm/src goes from 341,189 to 271,733 lines of Rust.
  • No compatibility re-exports: dynamo-llm has no pub use dynamo_frontend_protocols.

Commits

Five in-place cuts inside dynamo-llm, each building and passing on its own, then one move commit.

Obstacle Resolution Alternative rejected
MultimodalData::Decoded carried the media loader's RdmaMediaDataDescriptor, which pulled dynamo-memory (NIXL, CUDA), image and the decoders into protocols protocols owns common::decoded_media::DecodedMediaDescriptor, a wire struct of serde and primitive types with the same field names, order and serde attributes. The NIXL memory type is mirrored as an enum, the image crate's #[non_exhaustive] format and colour enums cross as their serde variant names, and the registered storage is held as Option<Arc<dyn Any + Send + Sync>> (never serialized). The loader keeps its typed descriptor and the preprocessor converts with TryFrom when it fills the request slot Moving the descriptor and its nested types into protocols (protocols would depend on dynamo-memory and image); a dynamo-memory dependency behind a feature
Orphan rule: an inherent impl tensor::Tensor in the gRPC tensor service A private KserveTensorContents extension trait in the service; bodies and call sites unchanged. It was the only such impl: the TryFrom<ModelInferRequest> impls are legal because the request type is local, and the other impls on protocol types use traits dynamo-llm owns Moving KServe and tonic::Status conversions into protocols
preprocessor and discovery re-exported protocol items (PreprocessedRequest, BackendOutput, PreprocessedEmbeddingRequest, LLMMetricAnnotation, the annotation names, WORKER_TYPE_*) Removed; 27 files import the items from protocols::common Keeping them as cross-crate re-exports
188 pub(crate) occurrences in protocols, some used by higher modules 81 become pub: exactly the ones a caller outside the moved crate failed to reach when compiled. The preprocessor names ParserVersion through dynamo-runtime rather than through a widened re-export Making everything pub
Test fixtures inside cfg(test) modules that the preprocessor's tests reuse; crate::test_utils; fixture paths into lib/llm/tests/data The shared fixtures move, unchanged, into test_fixtures modules (unified_parser, anthropic::stream_converter, openai::responses), and three test-only parser entry points become pub; the new crate exports them behind a testing feature that only dynamo-llm's dev-dependencies enable. The new crate includes lib/runtime/src/test_utils.rs by #[path], as dynamo-llm does. Test data stays in lib/llm/tests/data and the moved tests point at it, as dynamo-model-card does Copying fixtures; moving the test data
Cargo features No existing feature gated code in protocols. The new crate gets media-ffmpeg (gates the video variant of the decoded-media metadata, as the same feature gates it in the loader; dynamo-llm/media-ffmpeg forwards it) and testing. Every existing dynamo-llm/<feature> line keeps working An ungated video variant, which would accept a payload a build without the feature rejects today

Callers

  • 124 Rust files in lib/llm (85 in src, the rest tests and benches) and 32 outside it now import dynamo_frontend_protocols: the Python bindings, backend-common, the vLLM, SGLang and TensorRT-LLM sidecars and their testkit, both generate-format crates and the catalog tests, global-frontend and the EPP. Twelve manifests gain the dependency (dynamo-llm included); dynamo-llm drops percent-encoding, which only the moved code used.
  • Docs, scripts, CI path filters (labeler.yml, protocol-assessment.yml, the media-protocols pre-commit hook, .gitattributes) and comments that cite the moved paths are updated. The protocol field handling page cites no moved path and is unchanged. areas.yaml routes lib/frontend-protocols/ to the frontend area with the same shared entries as before (agents, multimodal, diffusion), types.rs keeps its runtime owners, and CODEOWNERS is regenerated.
  • lib/frontend-protocols/AGENTS.md (with its CLAUDE.md twin) states what the crate owns, what it must not depend on, and how to check it.

Dependency graph

  • cargo tree -p dynamo-frontend-protocols | grep -c dynamo-llm is 0. dynamo-memory, nixl-sys and cudarc do not appear in its tree. image appears only transitively, through the published openai-harmony that dynamo-parsers already brings in.
  • Direct Dynamo dependencies: dynamo-runtime, dynamo-model-card, dynamo-http-server, dynamo-admission, dynamo-scheduler, dynamo-router-protocol, dynamo-kv-hashing, dynamo-kv-hints, dynamo-lora, and the published dynamo-protocols, dynamo-parsers, dynamo-parsers-v2, dynamo-renderer, dynamo-tokenizers.

Behaviour

  • No status code, message, serialized field, validation rule, metric or log line changes. The wire struct serializes to the same bytes as the descriptor it replaces: a test compares both forms for every NIXL memory type, every image format and colour type, with and without metadata and content hash, in JSON and both MessagePack encodings, and reads each form from the other's bytes.
  • Two differences that are not on the wire. The wire struct reads any string for the image format and colour type, where the typed descriptor rejected names the image crate does not know. The Debug output of the storage keep-alive is now Any { .. }.

Validation

Cargo commands ran on a remote builder with the repository toolchain (1.96.1).

  • Baseline at 73e7ddd, cargo test -p dynamo-llm -p dynamo-model-card -p dynamo-http-server -p dynamo-error --no-fail-fast: 53 binaries, 4658 passed, 0 failed, 33 ignored.
  • Head, same command plus -p dynamo-frontend-protocols: 55 binaries, 4663 passed, 0 failed, 33 ignored. The two extra binaries are the new crate's unit and doc tests. The five extra tests are the wire compatibility test and the four test_utils self-tests the new crate now also includes.
  • Test accounting from cargo test -- --list: all 4691 baseline names exist at head; 3543 keep binary and path, 1148 moved from dynamo-llm's protocols:: to the same path in dynamo-frontend-protocols without the prefix; none is missing. dynamo-llm unit tests go from 3771 passed and 15 ignored to 2634 and 5; the new crate has 1142 and 10.
  • OpenAPI: cargo run -p dynamo-llm --bin generate-frontend-openapi writes docs/frontends/openapi.json with SHA-256 706f15d951891703fa911b9c659ac54992fa7144365ce4cd6b2a179a7630ede5 (148,295 bytes) at baseline and at head.
  • After each cut commit: cargo check -p dynamo-llm --all-targets and the tests of the touched modules pass (273, 71, 991 and 1530 tests for the first four; the fifth ran the whole dynamo-llm suite, workspace check and clippy).
  • Head: cargo check --workspace --all-targets, cargo clippy --no-deps --workspace --all-targets -- -D warnings, cargo fmt --all -- --check, and cargo check --locked in lib/bindings/python pass. cargo check -p dynamo-frontend-protocols --all-targets passes alone and with --features media-ffmpeg,testing (also under clippy), and cargo test -p dynamo-frontend-protocols passes alone.
  • Other crates with edited sources (dynamo-backend-common, dynamo-sglang-generate, dynamo-vllm-generate, dynamo-generate-format-catalog, dynamo-global-frontend, dynamo-ext-proc, the three sidecars and dynamo-sidecar-testkit): 38 test binaries, 952 passed, 0 failed, 12 ignored. Eleven testkit tests need the dynamo-lpx-sidecar binary and pass once it is built.
  • cargo machete at the root, lib/bindings/python and lib/runtime/examples reports no unused dependency. cargo metadata --locked passes for the four lock files (root, lib/bindings/python, lib/bindings/kvbm, lib/runtime/examples).
  • pre-commit run --from-ref idhanani/dyn-llm-split-model-vocabulary --to-ref HEAD passes, and build_codeowners.py --strict passes with CODEOWNERS matching emit_codeowners.py.
  • cargo check -p dynamo-llm --features mm-routing --all-targets passes. --features media-ffmpeg,mm-routing could not be built: the builder has no FFmpeg libraries (the baseline fails the same way), so the media-ffmpeg arm of the descriptor conversion in dynamo-llm is reviewed but not compiled. The new crate's own media-ffmpeg code compiles.
  • The move commit is scripted, and re-running the script on the move commit's parent reproduces the commit's tree exactly.

Closes DYNO-220

Step 1 of DYNO-220, which moves lib/llm/src/protocols into its own crate. `MultimodalData::Decoded` held the media loader's `RdmaMediaDataDescriptor`, so protocols imported `preprocessor::media` and, through it, dynamo-memory (NIXL), the `image` crate and the decoders.

protocols now owns `common::decoded_media::DecodedMediaDescriptor`, a wire struct of serde and primitive types with the same field names, order and serde attributes. The NIXL memory type is mirrored as an enum (the match in the conversion is exhaustive), the `image` crate's `#[non_exhaustive]` format and colour enums cross as their serde variant names, and the registered storage is kept alive through a type-erased `Arc<dyn Any + Send + Sync>` that is never serialized. The media loader keeps building and caching its typed descriptor; the preprocessor converts it with `TryFrom` when it fills the request slot. Nothing reads the keep-alive back: migration clones it into the first-response guard, and the routing code that reads the registered buffer does so on the typed descriptor before the conversion.

A test in the media loader serializes the typed and the wire form of the same values (every NIXL memory type, every `image` format and colour type, with and without metadata and content hash) with JSON and both MessagePack encodings and compares the bytes, and reads each form back from the other's bytes.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…gRPC service

Step 2 of DYNO-220. The gRPC tensor service held an inherent `impl tensor::Tensor` with the four functions that fill a tensor from KServe request contents. Once the tensor protocol lives in another crate, Rust's orphan rule forbids an inherent impl here.

The functions are the gRPC service's behaviour (they take `InferTensorContents` and return `tonic::Status`), so they stay in the service as a private `KserveTensorContents` extension trait implemented for `Tensor`. Bodies and call sites are unchanged. The `TryFrom<ModelInferRequest>` impls for protocol types in this module and in `grpc/service/openai.rs` already satisfy the orphan rule, because the request type is local.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…xports

Step 3 of DYNO-220. `preprocessor` re-exported `PreprocessedRequest`, `BackendOutput`, `PreprocessedEmbeddingRequest`, `LLMMetricAnnotation` and the two annotation names from protocols, and `discovery` re-exported the `WORKER_TYPE_PREFILL` / `WORKER_TYPE_DECODE` label values. After protocols moves to its own crate these would be re-exports of another crate's items, which the split does not keep.

Callers now import the items from `protocols::common` (`preprocessor`, `llm_backend`, `metrics`, `timing`): the routing host, session affinity, the HTTP service and metrics, the entrypoint, the integration tests, the Python bindings and the EPP. `preprocessor` keeps private imports for its own use. No item moves and no behaviour changes.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Step 4 of DYNO-220. About 190 items in protocols are `pub(crate)`. The preprocessor, the HTTP and gRPC services, the router, migration and request tracing use 81 of them, which stops working once protocols is a separate crate.

Those 81 become `pub`, and nothing else: the list comes from compiling the moved crate and widening only what a caller outside it failed to reach. It covers the `unified`, `delta_common`, `custom_tools`, `tool_call_merge` and `unified_parser` modules, the parser selection and tool validation helpers the preprocessor calls, the migration and request lifecycle state on `PreprocessedRequest` (`MigrationState`, `PrefillBookingRelease`, the `migration_state`, `staged_kv_cleanup` and `jail_seed` fields), and a few accessors. Inside dynamo-llm the change has no effect.

The preprocessor names `ParserVersion` through dynamo-runtime, which defines it, instead of through the crate-private re-export in `tool_parser_v2`.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…ules

Step 5 of DYNO-220. The preprocessor's tests reuse helpers defined inside protocols' `#[cfg(test)] mod tests` modules: the unified parser's eight-family tool-call inputs, the Anthropic stream converter's `text_chunk` and `sse_values`, and the Responses template fixtures. Another crate cannot see into a `cfg(test)` module, so after the move they need one place that a `testing` cargo feature can export.

Each set moves, unchanged, out of its `mod tests` into a sibling `test_fixtures` module (`unified_parser`, `anthropic::stream_converter`, `openai::responses`), and the test modules are private again. Three test-only parser entry points the preprocessor's tests call (`ChoiceState::new`, `parse_complete_with_policy`, `apply_stream_with_constraint`) become `pub` under the same `cfg(test)`. Here everything stays `cfg(test)`; the move commit switches `pub` items under `cfg(test)` to `cfg(any(test, feature = "testing"))`.

No test is added, removed or renamed.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Step 6 of DYNO-220 and layer 5 of splitting lib/llm into smaller crates. With the upward edges cut by the preceding commits, the request and response protocols move into a new crate below dynamo-llm:

- lib/llm/src/protocols.rs becomes the crate root and the modules under lib/llm/src/protocols (openai, anthropic, common, tensor, codec, unified, agents, systemone, response_merge) become its top-level modules, so `dynamo_llm::protocols::openai::X` is now `dynamo_frontend_protocols::openai::X`
- lib/llm/src/types.rs moves with them as `dynamo_frontend_protocols::types`: it holds only the engine type aliases over protocol messages and the runtime's engine traits. engines.rs (the echo, validating and dispatching engines) stays in dynamo-llm

The 86 files move unchanged apart from import paths (`crate::protocols::` becomes `crate::`), test data paths, which now point at lib/llm/tests/data, the two test paths the isolated-process tests select themselves by, and the `cfg` of the shared test fixtures, which a new `testing` feature exports to dynamo-llm's tests. dynamo-llm keeps no re-exports: every caller in dynamo-llm, its tests and benches, the Python bindings, backend-common, the sidecars, the generate-format crates, global-frontend and the EPP imports dynamo_frontend_protocols directly. dynamo-llm forwards its media-ffmpeg feature to the new crate, which gates the video variant of the decoded-media metadata, and drops percent-encoding, which only the moved code used.

The new crate depends on dynamo-runtime, dynamo-model-card, dynamo-http-server and the router protocol crates. It does not depend on dynamo-llm, dynamo-memory, NIXL or CUDA.

Codeowners routes lib/frontend-protocols to the areas that own the files today (types.rs keeps its runtime owners), and the docs, CI path filters, scripts and comments that cite the moved paths are updated.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani
ishandhanani force-pushed the idhanani/dyn-llm-split-model-vocabulary branch from d7f7c4d to 41892d5 Compare November 16, 2026 20:34
@ishandhanani
ishandhanani force-pushed the idhanani/dyno-220-frontend-protocols branch from 5b3b831 to 216f947 Compare November 16, 2026 20:34

This branch had an error being deployed

1 failed deployment
external_collaborator — 216f9478 Deployed Nov 16, 2026 by ishandhanani via ok-to-test #749
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant