Skip to content

refactor(llm): serve HTTP through dynamo-http-server - #209

Draft
ishandhanani wants to merge 2 commits into
idhanani/dyn-llm-split-error-statusfrom
idhanani/dyn-llm-split-http-server
Draft

ishandhanani wants to merge 2 commits into
idhanani/dyn-llm-split-error-statusfrom
idhanani/dyn-llm-split-http-server

Conversation

@ishandhanani

@ishandhanani ishandhanani commented Nov 12, 2026 •

Copy link
Copy Markdown
Owner

Summary

Step 1 of splitting lib/llm into smaller crates. dynamo-llm stops carrying its own copy of the HTTP server machinery and depends on dynamo-http-server for it, so the HTTP metrics and connection types that other dynamo-llm modules import now live below the crate instead of inside its largest import cycle.

This PR is layer 2 of the 16-layer split stack (main <- #211 <- #209 <- #212 <- #213 <- #218 <- #219 <- #220 <- #221 <- #222 <- #223 <- #224 <- #225 <- #226 <- #227 <- #228 <- #229) and is based on #211, which moves the error-class to HTTP status mapping into dynamo-error.

Validation

The base is #211 at 701a175 (on upstream main 8e7d29e). Before and after-runs used the same remote 32 vCPU builder.

The numbers below were measured on that base. The stack has since been rebased onto upstream main d20e045; after the rebase this layer's head passed cargo check --workspace --all-targets, cargo check --locked in lib/bindings/python, cargo metadata --locked for the four lock files, cargo fmt --all -- --check and the Anthropic admission tests, and the top of the stack passed the whole-workspace cargo test --locked run (10,022 passed, 0 failed, 87 ignored), clippy, and the bindings, machete, pre-commit, codeowners and OpenAPI checks.

  • cargo test -p dynamo-llm -p dynamo-http-server -p dynamo-sglang-generate -p dynamo-vllm-generate -p dynamo-generate-format-catalog -p dynamo-global-frontend --no-fail-fast plus cargo test -p dynamo-error --all-features: before 65 test binaries, 4,826 passed, 0 failed, 33 ignored; after 65 test binaries, 4,770 passed, 0 failed, 33 ignored.
  • Per crate, the only counts that change are dynamo-llm 4,610 to 4,552 (lib 3,986 to 3,928) and dynamo-http-server 56 to 58. The 46 dynamo-llm integration and doc-test binaries report the same counts; the only edits to those test files are import paths and one call in tests/http_metrics.rs (compiled only with --features integration), which names the free function http::service::metrics::update_metrics_from_mdc now that the method is gone; cargo check -p dynamo-llm --features integration --all-targets passes.
  • Unit-test accounting: cargo test -p dynamo-llm --lib -- --list goes from 4,001 to 3,943 names. 60 names left; 58 of them exist in dynamo-http-server with the same leaf name (55 already ran in both crates, 3 are new there: systemone_metrics_use_configured_prefix_and_bounded_labels, test_output_tokens_counter_flushes_at_threshold_and_drop, inflight_body_preserves_trailers_and_holds_admission), and 2 stayed in dynamo-llm under a new module path (http::service::write_timeout::tests to http::service::write_timeout_tests). One name left dynamo-http-server: test_output_tokens_counter_increments, which the frontend had already replaced with test_output_tokens_counter_flushes_at_threshold_and_drop.
  • Item-level comparison of the frontend files on main (metrics.rs, disconnect.rs, service_v2.rs, openai.rs, server.rs, write_timeout.rs, frontend_load.rs) against lib/http-server/src: 276 shared functions, types and tests are identical after normalizing visibility and paths. 14 are reported as different: 12 differ only by the intended adaptation (ModelLoadView in place of CommittedModelView, the serving-load collector split, set_worker_info_from_parts, visibility, the "unknown_model" literal) and 2 are name collisions with the HttpServer::serve and HttpServer::tls_acceptor methods.
  • cargo check --workspace --all-targets and cargo check in lib/bindings/python: exit 0 at both commits.
  • cargo clippy --no-deps --workspace --all-targets -- -D warnings: exit 0.
  • cargo check --locked in lib/bindings/python: exit 0. cargo metadata --locked passes for the root, lib/bindings/python, lib/bindings/kvbm and lib/runtime/examples lock files.
  • cargo machete in ., lib/bindings/python and lib/runtime/examples: no unused dependencies.
  • cargo fmt --all -- --check and pre-commit run --from-ref 701a1753a6 --to-ref HEAD: clean.
  • In-crate imports of http from other dynamo-llm modules: 25 files in 11 modules before, 10 files in 6 modules after; discovery, migration, frontend_load, generate_format and kv_router no longer import it.

Closes DYNO-217

@ishandhanani ishandhanani reopened this Nov 12, 2026
@ishandhanani
ishandhanani force-pushed the idhanani/dyn-llm-split-http-server branch from 6c86a91 to 81e5e7a Compare November 12, 2026 07:04
@ishandhanani
ishandhanani force-pushed the idhanani/dyn-llm-split-http-server branch from 81e5e7a to aa261fc Compare November 15, 2026 10:57
@ishandhanani
ishandhanani changed the base branch from main to idhanani/dyn-llm-split-error-status November 15, 2026 10:57
@ishandhanani
ishandhanani force-pushed the idhanani/dyn-llm-split-error-status branch from 701a175 to 5ef13fd Compare November 16, 2026 07:07
@ishandhanani
ishandhanani force-pushed the idhanani/dyn-llm-split-http-server branch from aa261fc to 8ddd6a1 Compare November 16, 2026 07:07
dynamo-http-server was copied from lib/llm/src/http/service on 2026-10-23
and the frontend copy kept changing. Port those changes so the crate can
replace the frontend copy:

- metrics: batched output-token counter with flush on finish and drop,
  System One histograms and phase timer, Transcriptions, SystemOne and
  Decisions endpoints, UnsupportedModality error type, sample-count based
  ITL flush.
- metrics: own the Prometheus scrape route (scrape_router) so a caller
  hands over a registry and gets back the path and router.
- server: expose bind_listener, write_idle_timeout and
  HttpServer::tls_acceptor for a second listener that shares the TLS
  settings (the frontend admin listener).
- lifecycle: track_inflight_response holds the permit in a body wrapper
  that forwards frames, is_end_stream and size_hint, so the response keeps
  its length and trailers (the frontend change in ai-dynamo#18177), with its test.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
dynamo-llm kept its own copy of the server loop, write deadlines,
connection monitor, lifecycle observer and the HTTP metrics family after
dynamo-http-server was created from them. Depend on dynamo-http-server
and delete the copies: server.rs, write_timeout.rs, the generic half of
disconnect.rs and metrics.rs, ServiceObserver/ServiceStage/InflightPermit,
get_body_limit and is_json_content_type.
dynamo-llm drops its hyper-util and http-body dependencies, which only
the deleted code used.

The frontend keeps what needs the model catalog: the model-ready and
start-time collectors, LoRA metrics, the serving-load collector adapter,
MDC and runtime-config metric updates, response metric processing and
RouteDoc.

dynamo-llm does not re-export the moved items. Callers in the repo
(the dynamo-llm tests, ext-proc, global-frontend and the generate-format
crates) import them from dynamo-http-server.

Step 1 of splitting lib/llm into smaller crates: the HTTP metrics and
connection types that other dynamo-llm modules import now live below the
crate.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani
ishandhanani force-pushed the idhanani/dyn-llm-split-error-status branch from 5ef13fd to fc1f85b Compare November 16, 2026 20:34
@ishandhanani
ishandhanani force-pushed the idhanani/dyn-llm-split-http-server branch from 8ddd6a1 to 5ec41d2 Compare November 16, 2026 20:34

This branch had an error being deployed

1 failed deployment
external_collaborator — 5ec41d2e Deployed Nov 16, 2026 by ishandhanani via ok-to-test #727
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant