[khluu]

Overview

Re-enable the KV connector CI jobs disabled in #63609 (follow-up to #63608 and #64010). Instead of installing NIXL, LMCache, Mooncake and CuPy at job time, the CUDA 13 CI image now bakes them from a hash-pinned lock.

Claims

• The CI test image installs the KV connector packages in the test-deps stage with uv pip install --require-hashes --no-deps from requirements/test/kv_connectors_cuda.txt. That lock is compiled against requirements/cuda.txt and requirements/test/cuda.txt and regenerated by a new pip-compile-kv-connectors pre-commit hook; a check-kv-connector-lock hook fails if it stops satisfying requirements/kv_connectors.txt. At build time it adds 16 packages and changes none already in the image.
• install-kv-connectors.sh becomes check-kv-connectors.sh. It only verifies the packages are present, so it can no longer resolve packages at job time.
• ROCm needs no image change. kv_connectors_rocm.txt (tblib, lm_eval[api]) is fully covered by requirements/test/rocm.txt, so the runtime installs were no-ops. They and the file are removed.
• 26 job blocks re-enabled. The CUDA and AMD jobs are listed in the build below.
• One missed runtime install removed: "Mooncake EC TCP E2E" ran uv pip install mooncake-transfer-engine>=0.3.12 into a venv. It now uses the image's Mooncake through --system-site-packages.
• 6 tethering-allowlist entries removed.

Validation

• pre-commit run on all changed files passed: pip-compile-kv-connectors, shellcheck, step keys, test tethering (--all reports nothing stale) and Dockerfile graph.
• All Buildkite YAML parses. Each restored block matches its pre-[CI] Disable jobs with unpinned runtime installs #63609 definition except for the script rename and the dropped ROCm install line.
• Cooldown: every newly introduced package was published at least 7 days ago except nixl/nixl-cu13 1.5.2 (PyPI Oct 29). Main already pins it in requirements/kv_connectors.txt since [NIXL][CI/Build] Bump NIXL to 1.5.2 and drop source build for Rubin #64585. Please don't merge before Nov 5 if we hold to the 7-day rule.
• CI: a selected-steps build of the image plus the re-enabled jobs is linked in a comment below.

Details

• The image layer installs only the lock entries the image doesn't already have, with --require-hashes --no-deps. It fails the build if the install changed any preinstalled package. After the vLLM and EP kernel wheels are installed, tools/kv_connector_lock.py verify fails the build if any locked package differs from its locked version or has unmet requirements. Its exceptions are named in the script: cuda-pathfinder, which the vLLM install upgrades; nixl → nixl-cu12; and torch → the image-pinned NCCL. check-kv-connectors.sh reruns the same verify at job start. This guard caught cuda-python/nvtx drift: the runtime base installs versions that requirements/test/cuda.txt doesn't pin. NCCL is excluded because the image pins it with its own NCCL_VERSION/UV_OVERRIDE.
• Scope: only the stable x86_64 CUDA 13 image. Torch-nightly and arm64 images don't get the packages, and check-kv-connectors.sh fails fast there with a clear message. The KV jobs only run on x86 L4/H100/H200/B200 pools.
• Still disabled:
  • A100 "NixlConnector PD + Spec Decode acceptance": the A100 queue was retired in [CI] Remove the retired A100 queue's steps; keep their live AMD mirrors standalone #64602. Its MI355 mirror needs a standalone conversion.
  • The two Intel jobs that pip install -r requirements/kv_connectors.txt on the XPU image. They need an Intel image change.
• The release images (INSTALL_KV_CONNECTORS=true) are unchanged.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

🤖 Generated with Claude Code

[claude]

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

_{Tip: disable this comment in your organization's Code Review settings.}

[khluu]

Claude Code here (AI assistant working with @khluu). Selected-steps CI run for this PR: the image build plus the 20 re-enabled CUDA KV connector jobs and the Mooncake EC job: https://buildkite.com/vllm/ci/builds/99153

[khluu]

Claude Code here (AI assistant working with @khluu). Build 99153 failed at image build: the image pins NCCL 2.31.2 through a global UV_OVERRIDE, which collided with the lock's nvidia-nccl-cu13==2.30.7 under --require-hashes. Fixed in ca77a3f: NCCL is no longer emitted in the lock (the image owns it), and the install step now fails if it changes any package that was already in the image. Re-run: https://buildkite.com/vllm/ci/builds/99164

[khluu]

Claude Code here (AI assistant working with @khluu). Status of selected-steps build https://buildkite.com/vllm/ci/builds/99176:

• Image build passes. The KV connector layer installs 16 hash-verified packages and changes nothing already in the image.
• V1 KV Connectors shard 2/3 fails on test_nixl_connector.py::test_register_packed_dsv4_mla_cache_as_single_region[v41-aliased-groups]. kv_caches_base_addr holds the packed raw base twice ([raw, raw, ring] instead of [raw, ring]); the other 894 tests pass. This is a latent failure on main, not caused by this PR: the job has been disabled since [CI] Disable jobs with unpinned runtime installs #63609 (Oct 25), and both [Bugfix][NIXL] Dedup transfer regions by (base_addr, block_len) #57662 (dedup regions by (base_addr, block_len), Oct 29) and [KV Connector][NIXL] Pair KV regions by layer for every transfer #63416 (pair KV regions by layer, which edited this test, Oct 31) merged while it couldn't run. Which side is right, the dedup behaviour or the assertion, is a NIXL owners' call, so I haven't changed it here.

[khluu]

Claude Code here (AI assistant working with @khluu). Full triage of https://buildkite.com/vllm/ci/builds/99176. None of the failures is install-related; all of them landed on main while these jobs were disabled (Oct 25 to now):

• Kimi K3 Prefix Cache (3/4 shards): the 4 dcp2-*-dspark cases now report XPASS(strict), because the FlashInfer MLA DCP fix ([Bugfix][Attention] Fix FlashInfer MLA DCP decode on ragged spec-decode batches #59392) merged on Oct 27. I dropped the stale xfail in cc47b9e; re-run: https://buildkite.com/vllm/ci/builds/99200
• V1 KV Connectors shard 2/3: the NIXL test_register_packed_dsv4_mla_cache_as_single_region[v41-aliased-groups] failure described above.
• V1 KV Connectors shard 3/3 (6 of 1107 failed):
  • offloading_connector/test_scheduler.py: test_partial_tail_store_uses_attention_and_recurrent_cow_sources and both test_eagle_partial_tail_round_trip_serves_mamba_checkpoint variants. These follow [KV Offload] Emit self-describing events for recurrent state groups #62613, [Bugfix][KV Offload] Preserve stored event prefix order #63249, [Bugfix][KV Offload] Deduplicate in-flight partial-tail loads #64400 and [Perf][KV Offload] Keep offload event metadata out of the cyclic GC #63927.
  • test_mooncake_connector.py: test_register_worker_retries_on_read_timeout and test_register_worker_raises_after_max_attempts. The mocked timeouts aren't reached, so the code hits real ConnectErrors. These follow [KV Connector][Mooncake] Renew prefill KV leases from the decoder #64193, [Perf][KV Connector][Mooncake] Vectorize transfer descriptor build #62478 and [KVConnector][Mooncake] Coalesce descending block ids to reduce descriptor number for Mooncake TE #63142.
  • test_mooncake_stats.py::test_expired_request_bumps_counter: MooncakeConnectorWorker lacks _kv_lease_duration. The test builds the worker without the new lease setup from [KV Connector][Mooncake] Renew prefill KV leases from the decoder #64193.

These need the respective owners to decide whether the code or the test is right. I haven't touched them here.

[orozery]

• offloading_connector/test_scheduler.py: test_partial_tail_store_uses_attention_and_recurrent_cow_sources and both test_eagle_partial_tail_round_trip_serves_mamba_checkpoint variants. These follow [KV Offload] Emit self-describing events for recurrent state groups #62613, [Bugfix][KV Offload] Preserve stored event prefix order #63249, [Bugfix][KV Offload] Deduplicate in-flight partial-tail loads #64400 and [Perf][KV Offload] Keep offload event metadata out of the cyclic GC #63927.

Fixed in #64993.

[khluu]

Claude Code here (on behalf of @khluu). Thanks @orozery. I merged main, which now includes #64993, into this branch (e962258) and am re-running V1 KV Connectors in https://buildkite.com/vllm/ci/builds/99534 to confirm the three offloading-connector scheduler tests pass. The Mooncake and NIXL failures listed above are separate and still unresolved.

🤖 Addressed by Claude Code

[khluu]

Claude Code here (on behalf of @khluu). V1 KV Connectors results after merging main, from https://buildkite.com/vllm/ci/builds/99534:

• Shard 3/3: the three offloading_connector/test_scheduler.py tests now pass, so [Test][KV Offload] Update stale partial-tail scheduler expectations #64993 fixed them. Thanks @orozery. The 3 failures left are all Mooncake and also fail on main:
  • test_register_worker_retries_on_read_timeout and test_register_worker_raises_after_max_attempts: the mocked httpx timeouts never fire, so the tests hit real errors. [CI/Build][KV Connector] Mooncake tests: patch the httpx module the connector uses #63122 looks like the fix.
  • test_mooncake_stats.py::test_expired_request_bumps_counter: MooncakeConnectorWorker has no _kv_lease_duration. The test builds the worker without the lease setup from [KV Connector][Mooncake] Renew prefill KV leases from the decoder #64193. I found no open fix for this one.
• Shard 2/3: the only failure (1 of 896) is the NIXL test_register_packed_dsv4_mla_cache_as_single_region[v41-aliased-groups], also failing on main. [Bugfix][NIXL] Preserve packed-row aliases during region dedup #64661 (preserve packed-row aliases during region dedup) addresses it.

None of these are caused by this PR, so I haven't patched them here.

🤖 Addressed by Claude Code

[khluu]

Claude Code here (on behalf of @khluu). I addressed all three review points in 3fa8954 and 697d171.

1. Name-only filtering. The awk filter is replaced by tools/kv_connector_lock.py:

   • plan installs only the lock entries that are missing. The KV layer runs before the vLLM and EP kernel wheels, and the vLLM install still moves cuda-python (13.4.4 → 13.0.3) and cuda-pathfinder (1.3.3 → 1.8.5). So plan reports preinstalled mismatches without failing on them.
   • verify runs as the last install step of the test stage (697d171) and again at job start in check-kv-connectors.sh. It requires every locked package at its locked version, with one exception: cuda-pathfinder, which the vLLM wheel install upgrades for cuda-core and nccl4py. It also checks every locked package's own requirements against the installed environment, so the exception is still checked for compatibility. Two dependencies are allowed to be unmet, both named in the script: nixl → nixl-cu12 (this image ships only nixl-cu13) and torch → nvidia-nccl-cu13 (the image pins NCCL itself).

   Running the old lock against the current CI image (vllm-ci-test-repo:e9622584…) found one real mismatch: nvtx. The lock had 0.2.16, but vllm pins 0.2.15 in requirements/cuda.txt, and the name filter skipped it. The lock is now also compiled with -c requirements/cuda.txt, which brings nvtx to 0.2.15; no other version changed. Tested in that image:

   • With the old lock, verify fails on nvtx.
   • With the new lock, plan emits nothing to install and verify passes.
   • Downgrading cuda-pathfinder to 1.2.0 makes verify fail on cupy's >=1.3.3.
   • I also replayed the build order in that image: test-deps versions with the connectors uninstalled, then plan, the hash-checked install, and the vLLM install. plan reported cuda-python and installed the 4 missing packages without changing any preinstalled one. verify failed before the vLLM install step and passed after it.

   cryptography looked like a mismatch at first, but only because Ubuntu's apt copy, 41.0.7, sits in /usr/lib/python3/dist-packages. The check reads the version that Python actually imports, 50.0.1, which matches the lock.

2. Drift from requirements/kv_connectors.txt. A new pre-commit hook, check-kv-connector-lock (kv_connector_lock.py align), runs when requirements/kv_connectors.txt, the .in file, or the lock changes. It fails if the CI lock stops satisfying a release requirement, mapping mooncake-transfer-engine to its -cuda13 wheel. It also fails if the .in file lists a package the release file doesn't; nixl-cu13 is the only allowed extra. I tested it by bumping nixl, raising the lmcache floor, and adding a new package; all three were flagged.

3. Duplicate MoRI mirror. I removed the mirror and kept the standalone MI355 job, which also runs test_expert_reload.py.

I also removed test_epd_correctness.py from the test-tethering allowlist, since the re-enabled EC connector job now runs it.

🤖 Addressed by Claude Code

[khluu]

Claude Code here (on behalf of @khluu). Selected-steps run to test the rebuilt image at 697d171: the image build, which now runs kv_connector_lock.py verify on the finished test image, plus the 20 CUDA jobs this PR re-enables, including V1 KV Connectors, NIXL/Mooncake P/D, Torchrun + Examples and Fault Tolerance: https://buildkite.com/vllm/ci/builds/100048

🤖 Generated with Claude Code

[khluu]

Claude Code here (on behalf of @khluu). Results so far from https://buildkite.com/vllm/ci/builds/100048 at 697d171:

• Image build passed with the new checks. The KV layer installed the 16 missing lock entries without changing any preinstalled package. It reported that cuda-python was at 13.4.4 against the lock's 13.0.3 instead of failing. The vLLM wheel install then resolved it, and kv_connector_lock.py verify passed on the finished image ([test 8/11]).
• Passed so far: NixlConnector PD edge case, MultiConnector PD edge cases, Hybrid SSM NixlConnector PD prefix cache, DP EP Distributed NixlConnector PD accuracy, Distributed MooncakeConnector PD accuracy (B200), Mooncake EC TCP E2E, and Kimi-Linear Disaggregated. The rest are still running.
• V1 KV Connectors shards 2/3 and 3/3 failed, all on tests that also fail on main. That job is disabled on main, so none of these were caught there:
  • NIXL test_register_packed_dsv4_mla_cache_as_single_region[v41-aliased-groups], addressed by [Bugfix][NIXL] Preserve packed-row aliases during region dedup #64661.
  • Mooncake test_register_worker_retries_on_read_timeout and test_register_worker_raises_after_max_attempts: likely fixed by [CI/Build][KV Connector] Mooncake tests: patch the httpx module the connector uses #63122.
  • Mooncake test_expired_request_bumps_counter: no open fix yet.
  • New: test_hisparse_connector.py::test_draft_layer_rows_mirrored_after_drafter[FULL|NONE] raises AttributeError: 'types.SimpleNamespace' object has no attribute 'max_swap_rows'. [Perf][HiSparse] Prefetch speculative groups with bounded copy workers #63801, merged Nov 3, made start_load_kv read self.runtime.max_swap_rows, but the test's stub runtime doesn't define it. Because the job was disabled, [Perf][HiSparse] Prefetch speculative groups with bounded copy workers #63801's CI never ran this test. cc @zhenwenqi2024

None of these is caused by this PR, so I haven't patched them here.

🤖 Generated with Claude Code

[AndreasKaratzas]

Should we modify a bit rocm docker file too? Or did I miss the mods there?

[khluu]

Claude Code here (on behalf of @khluu). @AndreasKaratzas The ROCm Dockerfile doesn't need changes. It already does what this PR adds on the CUDA side:

• The connectors are already built into the ROCm image from pinned sources: NIXL from commit 1683cf3b… (NIXL_BRANCH), LMCache from 140819c9… (LMCACHE_REF), and MoRI at v1.2.3.post1 in Dockerfile.rocm_base.
• On ROCm, the old install-kv-connectors.sh installed only requirements/kv_connectors_rocm.txt, which held just tblib and lm_eval[api]. Both are already in the hash-locked requirements/test/rocm.txt, which the test-deps stage installs in full (Dockerfile.rocm L379–383). That stage also fails if a test dependency changes a shipped package. So this PR deletes kv_connectors_rocm.txt.
• check-kv-connectors.sh checks that tblib and lm-eval are present on ROCm and installs nothing at run time.

One caveat: none of the re-enabled AMD jobs have run on this PR yet. They are MI355 V1 KV Connectors, MoRIIO unit, MoRIIO WRITE E2E and the Torchrun mirror. I'll start a selected-steps run for them to confirm.

🤖 Addressed by Claude Code

[khluu]

Claude Code here (on behalf of @khluu). AMD results from https://buildkite.com/vllm/ci/builds/100125 at 697d171:

• Passed: the ROCm test image build, check-kv-connectors.sh on ROCm (tblib 3.1.0, lm-eval 0.4.12 from the hash-locked test lock), MI355 Distributed Torchrun + Examples, MoRIIO Connector Unit Tests, and MoRIIO WRITE E2E shards 1, 3, 4 and 5. Still running: E2E shards 2 and 6, Heterogeneous TP, and Kimi K3 Prefix Cache. All three V1 KV Connectors shards are on their automatic retry.
• First attempts of V1 KV Connectors (MI355 DPX) failed only on tests that also fail on main. The AMD job has been disabled since [CI] Disable jobs with unpinned runtime installs #63609 (Oct 25), so these went unnoticed:
  • Same as CUDA: NIXL test_register_packed_dsv4_mla_cache_as_single_region[v41-aliased-groups] ([Bugfix][NIXL] Preserve packed-row aliases during region dedup #64661), Mooncake test_register_worker_retries_on_read_timeout / test_register_worker_raises_after_max_attempts ([CI/Build][KV Connector] Mooncake tests: patch the httpx module the connector uses #63122), and Mooncake test_expired_request_bumps_counter (_kv_lease_duration).
  • test_hisparse_connector.py::test_draft_layer_rows_mirrored_after_drafter[FULL|NONE] fails on ROCm with RuntimeError: The current platform does not support HiSparse. This is the error [Bugfix][HiSparse] Inject backend in connector unit test #64240 fixes by injecting the backend. On CUDA the same tests fail later on max_swap_rows ([Perf][HiSparse] Prefetch speculative groups with bounded copy workers #63801).
  • AMD only: test_mooncake_store_hma_e2e.py::test_store_values_round_trip_every_layout[mla-BLHNC|mla-BLNHC] fail the block_len == block_stride assertion (L533). [KV Connector] Mooncake store: skip padding in stored KV blocks #63513 added that assertion on Oct 29, after the AMD job was disabled. cc @wzhao18

None of these involve the image or dependency changes in this PR.

🤖 Generated with Claude Code

[khluu]

Claude Code here (on behalf of @khluu). Pushed 0c3e9b1 so V1 KV Connectors can be re-enabled with the rest of this PR. It marks the 9 test IDs that broke on main while the job was disabled as xfail(strict=True). Each marker's reason names its fix, and strict means the fix PR fails until it removes the marker.

Test	Fails on	Fix
test_nixl_connector.py::test_register_packed_dsv4_mla_cache_as_single_region[v41-aliased-groups]	CUDA, ROCm	#64661
test_mooncake_connector.py::test_register_worker_retries_on_read_timeout, …raises_after_max_attempts	CUDA, ROCm	#63122
test_mooncake_stats.py::test_expired_request_bumps_counter (no _kv_lease_duration)	CUDA, ROCm	none yet
test_hisparse_connector.py::test_draft_layer_rows_mirrored_after_drafter[FULL/NONE]	CUDA (stub lacks max_swap_rows), ROCm (platform check, #64240)	partial
test_mooncake_store_hma_e2e.py::test_store_values_round_trip_every_layout[mla-BLHNC/BLNHC]	ROCm only	none yet

The rest of the job is about 2,100 tests on CUDA and 2,400 on ROCm, all passing in builds 100048 and 100125. Selected-steps re-run of V1 KV Connectors on both platforms: https://buildkite.com/vllm/ci/builds/100485