Repository navigation
refactor(llm): serve HTTP through dynamo-http-server - #209
Draft
ishandhanani wants to merge 2 commits into
Draft
ishandhanani wants to merge 2 commits into
ishandhanani wants to merge 2 commits into
Conversation
ishandhanani
had a problem deploying
to
external_collaborator
November 12, 2026 05:39 — with
GitHub Actions
Failure
ishandhanani
had a problem deploying
to
external_collaborator
November 12, 2026 05:41 — with
GitHub Actions
Failure
ishandhanani
force-pushed
the
idhanani/dyn-llm-split-http-server
branch
from
November 12, 2026 07:04
6c86a91 to
81e5e7a
Compare
ishandhanani
had a problem deploying
to
external_collaborator
November 12, 2026 07:04 — with
GitHub Actions
Failure
ishandhanani
force-pushed
the
idhanani/dyn-llm-split-http-server
branch
from
November 15, 2026 10:57
81e5e7a to
aa261fc
Compare
ishandhanani
had a problem deploying
to
external_collaborator
November 15, 2026 10:57 — with
GitHub Actions
Failure
ishandhanani
changed the base branch from
main
to
idhanani/dyn-llm-split-error-status
November 15, 2026 10:57
ishandhanani
force-pushed
the
idhanani/dyn-llm-split-error-status
branch
from
November 16, 2026 07:07
701a175 to
5ef13fd
Compare
ishandhanani
force-pushed
the
idhanani/dyn-llm-split-http-server
branch
from
November 16, 2026 07:07
aa261fc to
8ddd6a1
Compare
ishandhanani
had a problem deploying
to
external_collaborator
November 16, 2026 07:07 — with
GitHub Actions
Failure
ishandhanani
had a problem deploying
to
external_collaborator
November 16, 2026 07:07 — with
GitHub Actions
Failure
This was referenced Nov 16, 2026
dynamo-http-server was copied from lib/llm/src/http/service on 2026-10-23 and the frontend copy kept changing. Port those changes so the crate can replace the frontend copy: - metrics: batched output-token counter with flush on finish and drop, System One histograms and phase timer, Transcriptions, SystemOne and Decisions endpoints, UnsupportedModality error type, sample-count based ITL flush. - metrics: own the Prometheus scrape route (scrape_router) so a caller hands over a registry and gets back the path and router. - server: expose bind_listener, write_idle_timeout and HttpServer::tls_acceptor for a second listener that shares the TLS settings (the frontend admin listener). - lifecycle: track_inflight_response holds the permit in a body wrapper that forwards frames, is_end_stream and size_hint, so the response keeps its length and trailers (the frontend change in ai-dynamo#18177), with its test. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
dynamo-llm kept its own copy of the server loop, write deadlines, connection monitor, lifecycle observer and the HTTP metrics family after dynamo-http-server was created from them. Depend on dynamo-http-server and delete the copies: server.rs, write_timeout.rs, the generic half of disconnect.rs and metrics.rs, ServiceObserver/ServiceStage/InflightPermit, get_body_limit and is_json_content_type. dynamo-llm drops its hyper-util and http-body dependencies, which only the deleted code used. The frontend keeps what needs the model catalog: the model-ready and start-time collectors, LoRA metrics, the serving-load collector adapter, MDC and runtime-config metric updates, response metric processing and RouteDoc. dynamo-llm does not re-export the moved items. Callers in the repo (the dynamo-llm tests, ext-proc, global-frontend and the generate-format crates) import them from dynamo-http-server. Step 1 of splitting lib/llm into smaller crates: the HTTP metrics and connection types that other dynamo-llm modules import now live below the crate. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
ishandhanani
force-pushed
the
idhanani/dyn-llm-split-error-status
branch
from
November 16, 2026 20:34
5ef13fd to
fc1f85b
Compare
ishandhanani
force-pushed
the
idhanani/dyn-llm-split-http-server
branch
from
November 16, 2026 20:34
8ddd6a1 to
5ec41d2
Compare
ishandhanani
had a problem deploying
to
external_collaborator
November 16, 2026 20:34 — with
GitHub Actions
Failure
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Step 1 of splitting
lib/llminto smaller crates.dynamo-llmstops carrying its own copy of the HTTP server machinery and depends ondynamo-http-serverfor it, so the HTTP metrics and connection types that otherdynamo-llmmodules import now live below the crate instead of inside its largest import cycle.This PR is layer 2 of the 16-layer split stack (
main<- #211 <- #209 <- #212 <- #213 <- #218 <- #219 <- #220 <- #221 <- #222 <- #223 <- #224 <- #225 <- #226 <- #227 <- #228 <- #229) and is based on #211, which moves the error-class to HTTP status mapping intodynamo-error.dynamo-llm:http/service/server.rs,http/service/write_timeout.rs, the generic half ofdisconnect.rs(connection handle and monitor) and ofmetrics.rs(theMetricsfamily, guards, response collector, error predicates),ServiceObserver/ServiceStage/InflightPermit, the response-body wrapper that holds the inflight permit, the request-id echo middleware,get_body_limit,is_json_content_type, and the frontend-load frame types.http/service/metrics.rsgoes from 6,381 to 2,069 lines.dynamo-llmdrops itshyper-utilandhttp-bodydependencies, which only the deleted code used.dynamo-llm: everything that needs the model catalog or protocol types: model-ready and start-time collectors, LoRA metrics, the serving-load collector adapter, MDC and runtime-config metric updates, response metric processing, the frontend disconnect stream monitors, andRouteDoc(it carries the frontend's OpenAPI route kinds).dynamo-http-server: the crate was copied on 2026-10-23 and eleven commits changed the frontend copy afterwards (perf(llm): batch the per-token output-token counter per request ai-dynamo/dynamo#17067, feat: add Muse Spark Open support and shared Chat/Responses tool APIs ai-dynamo/dynamo#17068, feat(decisions): coordinate bounded scoring with pinned worker placement ai-dynamo/dynamo#15853, feat(decisions): expose compatible decision and systemone endpoints ai-dynamo/dynamo#15854, refactor(frontend): unify native generate handlers behind format plugins ai-dynamo/dynamo#16983, fix(anthropic): align Messages framing, errors, and validation ai-dynamo/dynamo#17367, feat(vllm): add audio transcription serving ai-dynamo/dynamo#16208, feat(planner): collect batch scheduling observations ai-dynamo/dynamo#14196, feat(frontend): refuse media before routing for text-only models ai-dynamo/dynamo#17874, fix(beam): finalize constrained tool calls at confirmed model EOS ai-dynamo/dynamo#17625, fix(frontend): preserve response framing while tracking inflight requests ai-dynamo/dynamo#18177). Their changes to the shared code are ported in the first commit: batched output-token counter with flush on finish and drop, System One histograms, the Transcriptions / SystemOne / Decisions endpoints, the UnsupportedModality error type, sample-count ITL flush, andtrack_inflight_responseholding the permit in a body wrapper that keeps the response length and trailers (with its test;dynamo-http-servergainshttp-body, andhttp-body-utilas a dev-dependency).dynamo-http-server:metrics::scrape_router(the Prometheus scrape route; the frontend wraps it to add itsRouteDoc), andserver::{bind_listener, write_idle_timeout}plusHttpServer::tls_acceptor, which the frontend admin listener needs to share the main listener's TLS settings.dynamo-llmdoes not re-export the moved items. In-repo callers import them fromdynamo-http-server: seven test files underlib/llm/tests, ext-proc, global-frontend, the generate-format catalog test, and the SGLang and vLLM generate-format crates. The Python bindings need no source change.Validation
The base is #211 at 701a175 (on upstream
main8e7d29e). Before and after-runs used the same remote 32 vCPU builder.The numbers below were measured on that base. The stack has since been rebased onto upstream
maind20e045; after the rebase this layer's head passedcargo check --workspace --all-targets,cargo check --lockedinlib/bindings/python,cargo metadata --lockedfor the four lock files,cargo fmt --all -- --checkand the Anthropic admission tests, and the top of the stack passed the whole-workspacecargo test --lockedrun (10,022 passed, 0 failed, 87 ignored), clippy, and the bindings, machete, pre-commit, codeowners and OpenAPI checks.cargo test -p dynamo-llm -p dynamo-http-server -p dynamo-sglang-generate -p dynamo-vllm-generate -p dynamo-generate-format-catalog -p dynamo-global-frontend --no-fail-fastpluscargo test -p dynamo-error --all-features: before 65 test binaries, 4,826 passed, 0 failed, 33 ignored; after 65 test binaries, 4,770 passed, 0 failed, 33 ignored.dynamo-llm4,610 to 4,552 (lib 3,986 to 3,928) anddynamo-http-server56 to 58. The 46dynamo-llmintegration and doc-test binaries report the same counts; the only edits to those test files are import paths and one call intests/http_metrics.rs(compiled only with--features integration), which names the free functionhttp::service::metrics::update_metrics_from_mdcnow that the method is gone;cargo check -p dynamo-llm --features integration --all-targetspasses.cargo test -p dynamo-llm --lib -- --listgoes from 4,001 to 3,943 names. 60 names left; 58 of them exist indynamo-http-serverwith the same leaf name (55 already ran in both crates, 3 are new there:systemone_metrics_use_configured_prefix_and_bounded_labels,test_output_tokens_counter_flushes_at_threshold_and_drop,inflight_body_preserves_trailers_and_holds_admission), and 2 stayed indynamo-llmunder a new module path (http::service::write_timeout::teststohttp::service::write_timeout_tests). One name leftdynamo-http-server:test_output_tokens_counter_increments, which the frontend had already replaced withtest_output_tokens_counter_flushes_at_threshold_and_drop.main(metrics.rs,disconnect.rs,service_v2.rs,openai.rs,server.rs,write_timeout.rs,frontend_load.rs) againstlib/http-server/src: 276 shared functions, types and tests are identical after normalizing visibility and paths. 14 are reported as different: 12 differ only by the intended adaptation (ModelLoadViewin place ofCommittedModelView, the serving-load collector split,set_worker_info_from_parts, visibility, the"unknown_model"literal) and 2 are name collisions with theHttpServer::serveandHttpServer::tls_acceptormethods.cargo check --workspace --all-targetsandcargo checkinlib/bindings/python: exit 0 at both commits.cargo clippy --no-deps --workspace --all-targets -- -D warnings: exit 0.cargo check --lockedinlib/bindings/python: exit 0.cargo metadata --lockedpasses for the root,lib/bindings/python,lib/bindings/kvbmandlib/runtime/exampleslock files.cargo machetein.,lib/bindings/pythonandlib/runtime/examples: no unused dependencies.cargo fmt --all -- --checkandpre-commit run --from-ref 701a1753a6 --to-ref HEAD: clean.httpfrom otherdynamo-llmmodules: 25 files in 11 modules before, 10 files in 6 modules after;discovery,migration,frontend_load,generate_formatandkv_routerno longer import it.Closes DYNO-217