You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ray ports exhausted when using async vllm on some clusters #951
With the ray port range below, we can run well w/ both vllm sync or async engine on our cluster, but will sometimes fail w/ vllm async engine in some clusters.
We have a workaround #950 that expand the range from 54001 ~ 54257 to 54001 ~ 54513 for now.
Some investigation
From one run, before starting uv run python examples/run_grpo_math.py, all the ports in 54001 to 54257 are free, and after async vllm init, the ports are almost exhausted. then after a while ray complains no port available.
Error info core_worker.cc:513: Failed to register worker to Raylet: Invalid: Invalid: No available ports. Please specify a wider port range using --min-worker-port and --max-worker-port.
Activity
terrykong commentedon May 27, 2026
related to https://github.com/NVIDIA-NeMo/RL/pull/2380/changes. i believe the issue is we need to move the port ranges out of the ephemeral range for the workers. that should come from a pr @macandro96 and @yfw will prepare soon