Skip to content

fix(vllm): retry MessageQueue remote bind so node-spanning engines start - #3355

Merged
terrykong merged 1 commit into
terryk/bump-vllm-0.25.1from
terryk/rl-1111-mq-port-race
Jul 27, 2026
Merged

terrykong merged 1 commit into
terryk/bump-vllm-0.25.1from
terryk/rl-1111-mq-port-race

Conversation

@terrykong

Copy link
Copy Markdown
Collaborator

Fixes RL-1111. Stacked on #3280 (the vLLM 0.25.1 bump) — base is terryk/bump-vllm-0.25.1, not main.

Follow-on to #3350 (RL-1104). That fix was a prerequisite for seeing this one: with the TCPStore collision gone, engine startup gets one step further and then hits this.

The bug

Every non-driver-node worker of a node-spanning engine dies at startup:

zmq.error.ZMQError: Address already in use (addr='tcp://10.65.1.9:7000')
     [repeated 563x across cluster]

Root cause

MessageQueue.__init__ picks the port for its remote (TCP) socket by probing and releasing it, then binds it with ZMQ several statements later:

remote_subscribe_port = get_open_port()   # binds ("", port), CLOSES it, returns the number
...
self.remote_socket.bind(f"tcp://{connect_ip}:{remote_subscribe_port}")   # binds for real

That gap is a TOCTOU race, and on a non-driver node it is lost reliably rather than occasionally:

  • RayWorkerProc._init_message_queues gives workers on the driver node n_local_reader=1 → an ipc:// socket, no TCP port. Workers on every other node get n_local_reader=0 → a TCP port.
  • All of them scan upward from the same $VLLM_PORT (7000 for a node-spanning engine, via configure_worker's node-local slot 0).
  • _init_message_queues runs immediately after init_device(), whose process-group setup is a collective barrier — so every worker on the node reaches the probe within microseconds of the others, all see 7000 free, and all but one die.

Only engines with tensor_parallel_size * pipeline_parallel_size > cluster.gpus_per_node are affected, which is exactly why nothing caught it: zero of the 140 nightly tests run a node-spanning engine.

The fix

Retry at the bind rather than trusting the probe, advancing past the port that was lost.

Two properties make this small and safe, both verified empirically rather than assumed:

  1. A port a peer already holds with ZMQ is visible to the next _get_open_port probe (a plain bind(("", port)) on it fails EADDRINUSE), so each retry makes forward progress and the loop terminates.
  2. A ZMQ socket is re-bindable after a failed bind, so no fresh socket per attempt is needed.

Ports stay anchored at VLLM_PORT instead of falling back to bind(("", 0)) — kernel-ephemeral ports are precisely the contention the reserved sub-ephemeral band exists to prevent (#2380, #3103).

Patching the bind rather than handing each worker a private start port also covers every other MessageQueue with a remote reader — including the executor's own rpc_broadcast_mq, the queue involved in RL-1104 — instead of only the call site failing today.

Validation

The mechanism is pure port arithmetic, so it reproduces with plain processes against the real vLLM 0.25.1 MessageQueue — no GPU, no Ray, no second node. A multiprocessing.Barrier stands in for the init_device() collective.

racers unpatched patched
8 6/8 die, 3/3 runs 8/8 bind, ports 7000–7007, 16/16 runs
32 30/32 die, 3/3 runs 32/32 bind, ports 7000–7031, 3/3 runs

Failure signature matches CI exactly: ZMQError: Address already in use (addr='tcp://…:7000') raised from the same line.

Checked in as tests/unit/models/generation/test_vllm_message_queue_port.py (6 tests, 6s, no GPU), giving node-spanning engine startup its first automated coverage. test_unpatched_message_queue_loses_the_port_race passes rather than skips in CI-like conditions, so the patched test is not vacuous. The neighbouring RL-1104 suite still passes (12/12 together).

Does the patch reach non-driver nodes?

Yes — this was flagged as a genuine unknown in RL-1111, since #3350 only proved the mechanism reaches the executor process. Resolved by reading the code:

  • patches.py rewrites vLLM on disk, in the node-local venv.
  • _apply_vllm_patches runs in VllmGenerationWorker.__init__ on every worker, including the non-model-owner secondaries that exist on every node the engine spans.
  • RayWorkerGroup blocks on ray.get until all worker __init__s return, so every node's venv is patched before _load_model/_post_init creates the engine that spawns the RayWorkerProc actors.
  • Those actors inherit the parent's py_executable runtime_env, so they import the patched venv.

Not covered here

  • Not yet run on a real node-spanning recipe. Expected next failure there is the pre-existing refit-buffer assertion (embed_tokens.weight too large for buffer), which also fails on main with vLLM 0.20 — so a green DSv3 run is not the success criterion for this PR.
  • The code is byte-identical on vLLM main, so this is an upstream bug. An upstream PR is worth filing; this patch is the local fix regardless.
  • Ports walk into the TCPStore window (VLLM_PORT+32..+63) only past 32 workers on one node. They cannot collide: the TCPStore binds on the driver node, and these response queues only exist on non-driver nodes.

🤖 Generated with Claude Code

vLLM 0.25's MessageQueue picks the port for its remote (TCP) socket with
get_open_port(), which binds a probe socket, releases it, and returns the
number; ZMQ binds it for real several statements later. That window is a
TOCTOU race, and on a non-driver node it is lost reliably rather than
occasionally.

Every RayWorkerProc on a non-driver node takes n_local_reader=0
(ray_executor_v2.py::_init_message_queues), so every one of them needs a
real TCP port and they all scan from the same VLLM_PORT -- 7000 for a
node-spanning engine. _init_message_queues runs immediately after
init_device(), whose process-group setup is a collective barrier, so all
workers on the node reach the probe within microseconds of each other,
all see the same port free, and all but one die with:

    zmq.error.ZMQError: Address already in use (addr='tcp://10.65.1.9:7000')

Workers on the driver node take n_local_reader=1 and use an ipc:// socket,
which is why only engines spanning >= 2 nodes are affected -- and why no
nightly test catches it, since none runs an engine whose
tensor_parallel_size * pipeline_parallel_size exceeds cluster.gpus_per_node.

Fix the race at the bind rather than the probe: retry, advancing past the
port that was lost. This terminates because a port a peer already holds
with ZMQ is visible to the next probe. Ports stay anchored at VLLM_PORT
rather than falling back to kernel-ephemeral ones, which is the contention
the reserved sub-ephemeral band exists to prevent (#2380, #3103). Patching
the bind rather than handing each worker a private start port also covers
every other MessageQueue with a remote reader, including the executor's own
rpc_broadcast_mq.

The code is byte-identical on vLLM main; this is an upstream bug, so the
same change is worth proposing there.

Verified against the real vLLM 0.25.1 MessageQueue with concurrent
processes: unpatched, 6/8 and 30/32 workers die (3/3 runs each); patched,
8/8 and 32/32 bind distinct ports inside the reserved band.

Signed-off-by: Terry Kong <terryk@nvidia.com>
@terrykong
terrykong requested review from a team as code owners July 26, 2026 23:55
@copy-pr-bot

copy-pr-bot Bot commented Jul 26, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@terrykong
terrykong merged commit 7d3e09b into terryk/bump-vllm-0.25.1 Jul 27, 2026
10 checks passed
@terrykong
terrykong deleted the terryk/rl-1111-mq-port-race branch July 27, 2026 00:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant