Repository navigation
fix(vllm): retry MessageQueue remote bind so node-spanning engines start - #3355
Merged
Merged
Conversation
vLLM 0.25's MessageQueue picks the port for its remote (TCP) socket with
get_open_port(), which binds a probe socket, releases it, and returns the
number; ZMQ binds it for real several statements later. That window is a
TOCTOU race, and on a non-driver node it is lost reliably rather than
occasionally.
Every RayWorkerProc on a non-driver node takes n_local_reader=0
(ray_executor_v2.py::_init_message_queues), so every one of them needs a
real TCP port and they all scan from the same VLLM_PORT -- 7000 for a
node-spanning engine. _init_message_queues runs immediately after
init_device(), whose process-group setup is a collective barrier, so all
workers on the node reach the probe within microseconds of each other,
all see the same port free, and all but one die with:
zmq.error.ZMQError: Address already in use (addr='tcp://10.65.1.9:7000')
Workers on the driver node take n_local_reader=1 and use an ipc:// socket,
which is why only engines spanning >= 2 nodes are affected -- and why no
nightly test catches it, since none runs an engine whose
tensor_parallel_size * pipeline_parallel_size exceeds cluster.gpus_per_node.
Fix the race at the bind rather than the probe: retry, advancing past the
port that was lost. This terminates because a port a peer already holds
with ZMQ is visible to the next probe. Ports stay anchored at VLLM_PORT
rather than falling back to kernel-ephemeral ones, which is the contention
the reserved sub-ephemeral band exists to prevent (#2380, #3103). Patching
the bind rather than handing each worker a private start port also covers
every other MessageQueue with a remote reader, including the executor's own
rpc_broadcast_mq.
The code is byte-identical on vLLM main; this is an upstream bug, so the
same change is worth proposing there.
Verified against the real vLLM 0.25.1 MessageQueue with concurrent
processes: unpatched, 6/8 and 30/32 workers die (3/3 runs each); patched,
8/8 and 32/32 bind distinct ports inside the reserved band.
Signed-off-by: Terry Kong <terryk@nvidia.com>
This was referenced Jul 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes RL-1111. Stacked on #3280 (the vLLM 0.25.1 bump) — base is
terryk/bump-vllm-0.25.1, notmain.Follow-on to #3350 (RL-1104). That fix was a prerequisite for seeing this one: with the TCPStore collision gone, engine startup gets one step further and then hits this.
The bug
Every non-driver-node worker of a node-spanning engine dies at startup:
Root cause
MessageQueue.__init__picks the port for its remote (TCP) socket by probing and releasing it, then binds it with ZMQ several statements later:That gap is a TOCTOU race, and on a non-driver node it is lost reliably rather than occasionally:
RayWorkerProc._init_message_queuesgives workers on the driver noden_local_reader=1→ anipc://socket, no TCP port. Workers on every other node getn_local_reader=0→ a TCP port.$VLLM_PORT(7000 for a node-spanning engine, viaconfigure_worker's node-local slot 0)._init_message_queuesruns immediately afterinit_device(), whose process-group setup is a collective barrier — so every worker on the node reaches the probe within microseconds of the others, all see 7000 free, and all but one die.Only engines with
tensor_parallel_size * pipeline_parallel_size > cluster.gpus_per_nodeare affected, which is exactly why nothing caught it: zero of the 140 nightly tests run a node-spanning engine.The fix
Retry at the bind rather than trusting the probe, advancing past the port that was lost.
Two properties make this small and safe, both verified empirically rather than assumed:
_get_open_portprobe (a plainbind(("", port))on it failsEADDRINUSE), so each retry makes forward progress and the loop terminates.Ports stay anchored at
VLLM_PORTinstead of falling back tobind(("", 0))— kernel-ephemeral ports are precisely the contention the reserved sub-ephemeral band exists to prevent (#2380, #3103).Patching the bind rather than handing each worker a private start port also covers every other
MessageQueuewith a remote reader — including the executor's ownrpc_broadcast_mq, the queue involved in RL-1104 — instead of only the call site failing today.Validation
The mechanism is pure port arithmetic, so it reproduces with plain processes against the real vLLM 0.25.1
MessageQueue— no GPU, no Ray, no second node. Amultiprocessing.Barrierstands in for theinit_device()collective.Failure signature matches CI exactly:
ZMQError: Address already in use (addr='tcp://…:7000')raised from the same line.Checked in as
tests/unit/models/generation/test_vllm_message_queue_port.py(6 tests, 6s, no GPU), giving node-spanning engine startup its first automated coverage.test_unpatched_message_queue_loses_the_port_racepasses rather than skips in CI-like conditions, so the patched test is not vacuous. The neighbouring RL-1104 suite still passes (12/12 together).Does the patch reach non-driver nodes?
Yes — this was flagged as a genuine unknown in RL-1111, since #3350 only proved the mechanism reaches the executor process. Resolved by reading the code:
patches.pyrewrites vLLM on disk, in the node-local venv._apply_vllm_patchesruns inVllmGenerationWorker.__init__on every worker, including the non-model-owner secondaries that exist on every node the engine spans.RayWorkerGroupblocks onray.getuntil all worker__init__s return, so every node's venv is patched before_load_model/_post_initcreates the engine that spawns theRayWorkerProcactors.py_executableruntime_env, so they import the patched venv.Not covered here
embed_tokens.weight too large for buffer), which also fails onmainwith vLLM 0.20 — so a green DSv3 run is not the success criterion for this PR.main, so this is an upstream bug. An upstream PR is worth filing; this patch is the local fix regardless.VLLM_PORT+32..+63) only past 32 workers on one node. They cannot collide: the TCPStore binds on the driver node, and these response queues only exist on non-driver nodes.🤖 Generated with Claude Code