Skip to content

[Feat] http_server: Make the multi-worker ASGI app path configurable - #44583

Merged
cctry merged 1 commit into
mainfrom
oss-sync-6c7c6b203e
Oct 16, 2026
Merged

cctry merged 1 commit into
mainfrom
oss-sync-6c7c6b203e

Conversation

@metamergebot

@metamergebot metamergebot commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

With more than one tokenizer worker (--tokenizer-worker-num > 1), both the Uvicorn and the Granian launch paths hand the server a fixed import string, sglang.srt.entrypoints.http_server:app, and every worker process re-imports that module-level app. A package that embeds SGLang and registers extra routes on the app in the parent process never sees those routes in the workers, because each worker loads the bare SGLang app. There is currently no way to point the workers at an extended app without forking the launch code.

Modifications

  • Add SGLANG_HTTP_SERVER_APP_PATH to the Envs registry (HTTP server section), defaulting to sglang.srt.entrypoints.http_server:app.
  • Read it at both multi-worker launch sites in http_server.py: the Granian target and the uvicorn.run(...) import string.
  • Document it next to the other HTTP-server variables in docs/docs/references/environment_variables.mdx.
  • The single-worker path is unchanged and still passes the in-process app object; with the variable unset, behaviour is byte-for-byte the same as before.

Example: a package my_pkg.server that does from sglang.srt.entrypoints.http_server import app and adds its own routes can now run SGLANG_HTTP_SERVER_APP_PATH=my_pkg.server:app with multiple tokenizer workers, and every worker serves the extended app.

Accuracy Tests

Not applicable; no model or kernel code is touched.

Speed Tests and Profiling

Not applicable; the change only selects which ASGI app string the multi-worker server loads.

Checklist

  • Format your code according to the Format code with pre-commit. (ruff format/ruff check/isort/codespell at the pinned versions pass on both files.)
  • Add unit tests according to the Run and add unit tests. (Verified by hand that the default, an environment override and envs...override() all reach envs.SGLANG_HTTP_SERVER_APP_PATH.get(); no new test, since the only testable contract is the pass-through of a string into uvicorn.run/Granian.)
  • Update documentation according to Write documentations. (Row added to the HTTP & gRPC Server table in environment_variables.mdx.)
  • Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed. (Not applicable, see above.)
  • Follow the SGLang code style guidance.

CI States

Latest PR Test (Base): ✅ Run #38811398923
Latest PR Test (Extra): ❌ Run #38811398768
Latest PR Test (AMD ROCm 10): ❌ Run #38811398850

With more than one tokenizer worker, Uvicorn and Granian spawn worker
processes that re-import the ASGI app from a fixed import string,
"sglang.srt.entrypoints.http_server:app". A package that embeds SGLang
and registers extra routes on that app never sees them in the workers,
because each worker loads the bare module-level app.

Add SGLANG_HTTP_SERVER_APP_PATH to the Envs registry, read it at both
multi-worker launch sites, and document it with the other HTTP-server
variables, so an embedding package can point the workers at a module
that extends the SGLang app. The default is unchanged, and the
single-worker path still passes the in-process app object.

Co-authored-by: joshuadeng <joshuadeng@users.noreply.github.com>
@mintlify

mintlify Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
sglang-doc 🟢 Ready View Preview Oct 15, 2026, 9:42 PM

💡 Tip: Enable Automations to automatically generate PRs for you.

@metamergebot metamergebot added the run-ci CI: run the baseline test suite on this PR label Oct 15, 2026
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Oct 15, 2026
@metamergebot

Copy link
Copy Markdown
Collaborator Author

This port clears our bar on ce987a0acf307f14a3cbf799f1377b1201950fee: every base-a and base-b job is green.
base-c and the accelerator suites are reported but not gated, because they run
on scarce hardware and upstream merges through them routinely.

gate✓ lint✓ build✓ A✓ B✓ C✓ other✓ · accel ✗amd,npu

@fei-xx @merrymercy @joshuadeng — could one of you take a look and merge it if you are happy? Nothing else is waiting on our side.

@cctry cctry left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, OSS sync of an internal change.

@cctry
cctry merged commit 63a9b7a into main Oct 16, 2026
915 of 1008 checks passed
@cctry
cctry deleted the oss-sync-6c7c6b203e branch October 16, 2026 17:36

This branch was successfully deployed

1 active deployment
staging - docs — ce987a0a Deployed Oct 15, 2026 by mintlify[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants