Skip to content

ray ports exhausted when using async vllm on some clusters #951

Open

Description

@yuki-97

With the ray port range below, we can run well w/ both vllm sync or async engine on our cluster, but will sometimes fail w/ vllm async engine in some clusters.

MIN_WORKER_PORT=${MIN_WORKER_PORT:-54001}
MAX_WORKER_PORT=${MAX_WORKER_PORT:-54257}

We have a workaround #950 that expand the range from 54001 ~ 54257 to 54001 ~ 54513 for now.

Some investigation
From one run, before starting uv run python examples/run_grpo_math.py, all the ports in 54001 to 54257 are free, and after async vllm init, the ports are almost exhausted. then after a while ray complains no port available.

Error info
core_worker.cc:513: Failed to register worker to Raylet: Invalid: Invalid: No available ports. Please specify a wider port range using --min-worker-port and --max-worker-port.

Activity

  1. terrykong commented on May 27, 2026

    @terrykong
    Collaborator

    related to https://github.com/NVIDIA-NeMo/RL/pull/2380/changes. i believe the issue is we need to move the port ranges out of the ephemeral range for the workers. that should come from a pr @macandro96 and @yfw will prepare soon

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions