Skip to content

feat(vllm): wire runtime LoRA into worker requests - #15032

Open
biswapanda wants to merge 23 commits into
bis/dis-2937-lora-coordinatorfrom
bis/dis-2937-python-runtime-lora-plugin
Open

biswapanda wants to merge 23 commits into
bis/dis-2937-lora-coordinatorfrom
bis/dis-2937-python-runtime-lora-plugin

Conversation

@biswapanda

@biswapanda biswapanda commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Connect runtime LoRA resolution to the vLLM worker request path. Requests carrying the private runtime metadata acquire a resolved adapter and an admission lease before generation; cleanup releases the lease after the generator closes.

Publish eligible worker capabilities at registration, initialize resident capacity from validated settings, and drain resolver tasks before engine shutdown. Preserve legacy adapter requests and the primary shutdown error; Omni retains its separate no-op asynchronous handler cleanup.

Dependencies

Stacked on resolver coordination/residency #15842, which combines independently reviewed lifecycle #15839, snapshot validation #15840, and protocol/settings #15841. Common resolver #15835 is included through protocol/settings.

Rust routing #15031 is a separate functional prerequisite for HTTP request-time LoRA routing. Aggregated dynamo.vllm is the supported execution path.

Validation

  • Complete affected common/vLLM suite: 655 passed, 5 existing skips, with Allure 2.15.3 enabled.
  • Combined resolver/lifecycle module statement coverage: 89%; every individual new module exceeds 80%.
  • Mypy: 342 source files passed.
  • Changed-file pre-commit checks passed.
  • Review confirmed original production functions/classes and all original LoRA test cases remain present across the split.
  • Fresh local GPU run with vLLM0.31.0 and Rust feat(lora): add request-time routing protocol #15031 bindings: seven cases passed—base chat, explicit base plus opaque source, bare opaque source, cached completions, streaming, unsupported scheme400, and unknown base404. The resolver ran once across adapter requests.

The GPU smoke used an offline W&B-shaped resolver and an immutable local adapter snapshot. W&B network authentication/downloads, sidecar, Kubernetes, and exact container-image parity were not exercised.

Experimental follow-ups

A maintained GPU E2E workflow using the branch-pinned vLLM image and dedicated runtime-LoRA metrics remain follow-ups before graduating the feature from experimental status.

Relates to #14547 and DEP #15018.

@github-actions github-actions Bot added feat backend::vllm Relates to the vllm backend labels Sep 18, 2026
@biswapanda
biswapanda force-pushed the bis/dis-2937-python-runtime-lora-plugin branch from 4478343 to 70ad616 Compare September 18, 2026 14:25
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Sep 18, 2026
@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

@biswapanda
biswapanda force-pushed the bis/dis-2937-python-runtime-lora-plugin branch from d9cdae0 to 4f6ba6d Compare September 18, 2026 16:35
@rmccorm4 rmccorm4 mentioned this pull request Sep 27, 2026
@biswapanda
biswapanda force-pushed the bis/dis-2937-python-runtime-lora-plugin branch from 4f6ba6d to 619687a Compare October 7, 2026 21:26
@biswapanda
biswapanda force-pushed the bis/dis-2937-python-runtime-lora-plugin branch from 619687a to aa2daa7 Compare October 7, 2026 22:02
@github-actions github-actions Bot added deployment::k8s Relates to dynamo deployment in kubernetes backend::trtllm Relates to the trtllm backend actions labels Oct 7, 2026
@biswapanda biswapanda changed the title feat(lora): add Python resolver plugins for vLLM feat(vllm): add request-time LoRA plugin integration Oct 7, 2026
@biswapanda
biswapanda changed the base branch from bis/dis-2937-implement-pluggable-request-time-lora-resolution to bis/dis-2937-common-lora-resolver October 7, 2026 22:03
@biswapanda
biswapanda force-pushed the bis/dis-2937-python-runtime-lora-plugin branch from aa2daa7 to 27dbb17 Compare October 7, 2026 22:47
@biswapanda biswapanda changed the title feat(vllm): add request-time LoRA plugin integration feat(vllm): wire runtime LoRA into worker requests Oct 7, 2026
@biswapanda
biswapanda changed the base branch from bis/dis-2937-common-lora-resolver to bis/dis-2937-lora-coordinator October 7, 2026 22:47
@devin-ai-integration

devin-ai-integration Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

❌ Dynamo PR CI failed — run 41727616684 (attempt 1) on 69401d1a87

Gate checks: ❌ backend-status-check · ✅ deploy-status-check · ❌ dynamo-status-check

Framework 1-GPU arm64
TRT-LLM ❌ 1
Other Jobs
dynamo-runtime ❌ 1 ✅ 6

Failure details

2 jobs failed: mypy reports 4 type errors in components/src/dynamo/vllm/handlers.py, and the TRT-LLM arm64 test job passed its tests but failed uploading the results artifact (network reset, infrastructure).

❌ dynamo-runtime / mypy: 4 errors in vllm/handlers.py (AsyncIterator has no aclose)

Job: dynamo-runtime / mypy · Failed step: Run mypy (components) · Logs: gh run view --job 126274437927 -R ai-dynamo/dynamo --log-failed

components/src/dynamo/vllm/handlers.py:3897: error: "AsyncIterator[Any]" has no attribute "aclose"  [attr-defined]
components/src/dynamo/vllm/handlers.py:3972: error: Value of type variable "_SupportsAcloseT" of "aclosing" cannot be "AsyncIterator[Any]"  [type-var]
components/src/dynamo/vllm/handlers.py:4695: error: Value of type variable "_SupportsAcloseT" of "aclosing" cannot be "AsyncIterator[Any]"  [type-var]
components/src/dynamo/vllm/handlers.py:4850: error: Value of type variable "_SupportsAcloseT" of "aclosing" cannot be "AsyncIterator[Any]"  [type-var]
Found 4 errors in 1 file (checked 1015 source files)

mypy fails on components/src/dynamo/vllm/handlers.py: a generator typed as AsyncIterator[Any] has .aclose() called on it (line 3897) and is passed to contextlib.aclosing (lines 3972, 4695, 4850), which needs an object with aclose. All 4 errors have the same cause: AsyncIterator does not declare aclose.

❌ trtllm-runtime / Test cuda13.1, arm64: Failed to CreateArtifact: ECONNRESET

Job: trtllm-runtime / Test cuda13.1, arm64 · Failed step: Upload Test Results · Logs: gh run view --job 126275828898 -R ai-dynamo/dynamo --log-failed

======================= 278 passed, 27 skipped in 56.66s =======================
With the provided path, there will be 1 file uploaded
Artifact name is valid!
Root directory input is valid!
##[error]Failed to CreateArtifact: Unable to make request: ECONNRESET
you are using self-hosted runners, please make sure your runner has access to all GitHub endpoints: https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners#communication-between-self-hosted-runners-and-github
##[error]Error: failed to run script step: command terminated with non-zero exit code: error executing command [sh -e /__w/_temp/d09f2c00-d7de-11f1-a2dc-b716f30505d6.sh], exit code 1

The pytest stage passed (278 passed, 27 skipped). The job failed only because the self-hosted runner lost its connection to GitHub while uploading the test-results artifact (ECONNRESET). This is an infrastructure failure; rerunning the failed jobs is likely enough.

For agents
{"pr": 15032, "run_id": 41727616684, "run_attempt": 1, "head_sha": "69401d1a8793d8ba0da2554b86a5637ddc72174b", "failures": [{"job": "dynamo-runtime / mypy", "job_id": 126274437927, "failed_step": "Run mypy (components)", "signature": "mypy: AsyncIterator[Any] has no attribute aclose / aclosing type-var (4 errors in components/src/dynamo/vllm/handlers.py)", "tests": [], "log_cmd": "gh run view --job 126274437927 -R ai-dynamo/dynamo --log-failed"}, {"job": "trtllm-runtime / Test cuda13.1, arm64", "job_id": 126275828898, "failed_step": "Upload Test Results", "signature": "infra: Failed to CreateArtifact: ECONNRESET (tests passed)", "tests": [], "log_cmd": "gh run view --job 126275828898 -R ai-dynamo/dynamo --log-failed"}]}

Posted automatically by Devin for run 41727616684. Updated on every full-CI run of this PR.

@biswapanda
biswapanda force-pushed the bis/dis-2937-python-runtime-lora-plugin branch from b1df4a4 to 4aae8aa Compare October 26, 2026 15:08
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
@biswapanda
biswapanda force-pushed the bis/dis-2937-python-runtime-lora-plugin branch from f0cca24 to 5379027 Compare October 28, 2026 03:54
@biswapanda
biswapanda added this pull request to stack #17485 October 28, 2026 04:10
@biswapanda
biswapanda marked this pull request as ready for review October 28, 2026 04:10
@biswapanda
biswapanda requested review from a team as code owners October 28, 2026 04:10
@biswapanda biswapanda self-assigned this Oct 28, 2026
…-runtime-lora-plugin

Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
biswapanda

This comment was marked as outdated.

…-runtime-lora-plugin

Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

actions backend::trtllm Relates to the trtllm backend backend::vllm Relates to the vllm backend deployment::k8s Relates to dynamo deployment in kubernetes documentation Improvements or additions to documentation feat multimodal size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant