Repository navigation
feat(vllm): wire runtime LoRA into worker requests - #15032
biswapanda wants to merge 23 commits into
Conversation
4478343 to
70ad616
Compare
d9cdae0 to
4f6ba6d
Compare
4f6ba6d to
619687a
Compare
619687a to
aa2daa7
Compare
aa2daa7 to
27dbb17
Compare
❌ Dynamo PR CI failed — run 41727616684 (attempt 1) on
|
| Framework | 1-GPU arm64 |
|---|---|
| TRT-LLM | ❌ 1 |
| Other | Jobs |
|---|---|
| dynamo-runtime | ❌ 1 ✅ 6 |
Failure details
2 jobs failed: mypy reports 4 type errors in components/src/dynamo/vllm/handlers.py, and the TRT-LLM arm64 test job passed its tests but failed uploading the results artifact (network reset, infrastructure).
❌ dynamo-runtime / mypy: 4 errors in vllm/handlers.py (AsyncIterator has no aclose)
Job: dynamo-runtime / mypy · Failed step: Run mypy (components) · Logs: gh run view --job 126274437927 -R ai-dynamo/dynamo --log-failed
components/src/dynamo/vllm/handlers.py:3897: error: "AsyncIterator[Any]" has no attribute "aclose" [attr-defined]
components/src/dynamo/vllm/handlers.py:3972: error: Value of type variable "_SupportsAcloseT" of "aclosing" cannot be "AsyncIterator[Any]" [type-var]
components/src/dynamo/vllm/handlers.py:4695: error: Value of type variable "_SupportsAcloseT" of "aclosing" cannot be "AsyncIterator[Any]" [type-var]
components/src/dynamo/vllm/handlers.py:4850: error: Value of type variable "_SupportsAcloseT" of "aclosing" cannot be "AsyncIterator[Any]" [type-var]
Found 4 errors in 1 file (checked 1015 source files)
mypy fails on components/src/dynamo/vllm/handlers.py: a generator typed as AsyncIterator[Any] has .aclose() called on it (line 3897) and is passed to contextlib.aclosing (lines 3972, 4695, 4850), which needs an object with aclose. All 4 errors have the same cause: AsyncIterator does not declare aclose.
❌ trtllm-runtime / Test cuda13.1, arm64: Failed to CreateArtifact: ECONNRESET
Job: trtllm-runtime / Test cuda13.1, arm64 · Failed step: Upload Test Results · Logs: gh run view --job 126275828898 -R ai-dynamo/dynamo --log-failed
======================= 278 passed, 27 skipped in 56.66s =======================
With the provided path, there will be 1 file uploaded
Artifact name is valid!
Root directory input is valid!
##[error]Failed to CreateArtifact: Unable to make request: ECONNRESET
you are using self-hosted runners, please make sure your runner has access to all GitHub endpoints: https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners#communication-between-self-hosted-runners-and-github
##[error]Error: failed to run script step: command terminated with non-zero exit code: error executing command [sh -e /__w/_temp/d09f2c00-d7de-11f1-a2dc-b716f30505d6.sh], exit code 1
The pytest stage passed (278 passed, 27 skipped). The job failed only because the self-hosted runner lost its connection to GitHub while uploading the test-results artifact (ECONNRESET). This is an infrastructure failure; rerunning the failed jobs is likely enough.
For agents
{"pr": 15032, "run_id": 41727616684, "run_attempt": 1, "head_sha": "69401d1a8793d8ba0da2554b86a5637ddc72174b", "failures": [{"job": "dynamo-runtime / mypy", "job_id": 126274437927, "failed_step": "Run mypy (components)", "signature": "mypy: AsyncIterator[Any] has no attribute aclose / aclosing type-var (4 errors in components/src/dynamo/vllm/handlers.py)", "tests": [], "log_cmd": "gh run view --job 126274437927 -R ai-dynamo/dynamo --log-failed"}, {"job": "trtllm-runtime / Test cuda13.1, arm64", "job_id": 126275828898, "failed_step": "Upload Test Results", "signature": "infra: Failed to CreateArtifact: ECONNRESET (tests passed)", "tests": [], "log_cmd": "gh run view --job 126275828898 -R ai-dynamo/dynamo --log-failed"}]}Posted automatically by Devin for run 41727616684. Updated on every full-CI run of this PR.
b1df4a4 to
4aae8aa
Compare
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
f0cca24 to
5379027
Compare
…-runtime-lora-plugin Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
…-runtime-lora-plugin Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Summary
Connect runtime LoRA resolution to the vLLM worker request path. Requests carrying the private runtime metadata acquire a resolved adapter and an admission lease before generation; cleanup releases the lease after the generator closes.
Publish eligible worker capabilities at registration, initialize resident capacity from validated settings, and drain resolver tasks before engine shutdown. Preserve legacy adapter requests and the primary shutdown error; Omni retains its separate no-op asynchronous handler cleanup.
Dependencies
Stacked on resolver coordination/residency #15842, which combines independently reviewed lifecycle #15839, snapshot validation #15840, and protocol/settings #15841. Common resolver #15835 is included through protocol/settings.
Rust routing #15031 is a separate functional prerequisite for HTTP request-time LoRA routing. Aggregated dynamo.vllm is the supported execution path.
Validation
The GPU smoke used an offline W&B-shaped resolver and an immutable local adapter snapshot. W&B network authentication/downloads, sidecar, Kubernetes, and exact container-image parity were not exercised.
Experimental follow-ups
A maintained GPU E2E workflow using the branch-pinned vLLM image and dedicated runtime-LoRA metrics remain follow-ups before graduating the feature from experimental status.
Relates to #14547 and DEP #15018.