Skip to content

refactor(sglang): allocate ports from reserved ranges - #2997

Merged
terrykong merged 1 commit into
NVIDIA-NeMo:mainfrom
xiuhu17:port_migrate
Jul 14, 2026
Merged

terrykong merged 1 commit into
NVIDIA-NeMo:mainfrom
xiuhu17:port_migrate

Conversation

@xiuhu17

@xiuhu17 xiuhu17 commented Jun 29, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The SGLang backend allocated ports with its own helpers (find_available_port,
is_port_available, get_free_port) that probe OS-ephemeral ports, instead of
the shared range-based primitives in virtual_cluster.py already used by vLLM
and NeMo-Gym. This re-introduced the TOCTOU-prone allocation that #2380 removed,
just in a different place.

This PR rebases SGLang onto the shared helpers and reserved port map, while
keeping the SGLang-specific bits the shared helpers don't cover (contiguous
dp_attention port blocks, a per-node cursor, IPv6 bracketing, non-Ray host lookup).

Closes #2874

What changed

  • virtual_cluster.py: added _get_free_consecutive_ports_local() (returns a
    contiguous block of bindable ports in a reserved range — the one capability the
    existing helpers lacked) and a dedicated router/Prometheus band (28001–30000).
  • ray_utils.py: removed find_available_port / is_port_available /
    get_free_port; get_current_node_ip now delegates to _get_node_ip_local().
    get_host_info and the NOSET env list are unchanged.
  • ip_port_utils.py / sglang_worker.py: allocate through the shared helper
    with port_range_low/high + a per-node cursor instead of a hard-coded base_port=15000.
  • sglang_generation.py: reads port_range_low/high from the generation config.
  • sglang_router.py: router/Prometheus ports from _get_free_port_local over their band.
  • Tests: added TestGetFreeConsecutivePortsLocal; updated SGLang tests/helpers
    that used the removed functions.
  • patch: This patch rebases _patch_sglang_custom_all_reduce_v2_tms_cudagraph onto the final form of sgl-project/sglang#27948 (tracking radixark/miles#1176): instead of the earlier revision's set_cuda_graph_register_inputs toggle threaded through five files, it applies the upstream Python-only fix to custom_all_reduce_v2.py — adding the envs import and self.tms_cudagraph flag, and gating both set_cuda_graph_capture(True) calls on not self.tms_cudagraph — so the kernel's capture flag stays off under TMS and we no longer touch the fragile jit_kernel C++/.cuh sources.

Port layout

Port class Band
HTTP server + NCCL/dist_init 11001–15000 (per-node cursor)
Router / Prometheus 28001–29000 / 29000–30000

Testing

  • pytest tests/unit/distributed/test_virtual_cluster.py::TestGetFreeConsecutivePortsLocal
  • pytest -m sglang tests/unit/models/generation/sglang/

@xiuhu17
xiuhu17 requested review from a team as code owners June 29, 2026 23:20
@copy-pr-bot

copy-pr-bot Bot commented Jun 29, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@yuki-97 yuki-97 left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for update this @xiuhu17 ! left some comments. you also need a rebase since the branch is a bit outdated and has conflict.

Comment thread nemo_rl/distributed/virtual_cluster.py Resolved
@svcnvidia-nemo-ci svcnvidia-nemo-ci added waiting-on-customer Waiting on the original author to respond waiting-on-maintainers Waiting on maintainers to respond and removed waiting-on-customer Waiting on the original author to respond labels Jun 30, 2026

@yuki-97 yuki-97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @xiuhu17 , LGTM and just some nits. also could you help to rebase? looks there's a conflict with main branch.

@terrykong could you take a review as well?

Comment thread nemo_rl/distributed/virtual_cluster.py Resolved
Comment thread nemo_rl/distributed/virtual_cluster.py Resolved
@yuki-97
yuki-97 requested a review from terrykong July 8, 2026 08:02
@svcnvidia-nemo-ci svcnvidia-nemo-ci added waiting-on-maintainers Waiting on maintainers to respond waiting-on-customer Waiting on the original author to respond and removed waiting-on-maintainers Waiting on maintainers to respond labels Jul 8, 2026

@terrykong terrykong left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary

Thanks @xiuhu17 — nice cleanup consolidating SGLang port allocation onto the shared reserved-band helpers in virtual_cluster.py; this closes the loop on #2874 and removes the TOCTOU-prone ephemeral-port probing. Overall the change looks correct: we traced the cursor arithmetic, checked for dangling references to the removed helpers, import cycles, and the router/Prometheus band layout, and found no live correctness issues.

Main blocker (echoing @yuki-97): the branch still has merge conflicts with main — a rebase is needed before this can go in.

Verified locally on this head (1457738):

  • tests/unit/distributed/test_virtual_cluster.py::TestGetFreeConsecutivePortsLocal — 6/6 pass; full file 37/38 (the one failure, test_mcore_py_executable, is pre-existing and unrelated to this PR)
  • sglang unit tests: test_utils_smoke.py 4/4, test_sglang_router.py 10/10 pass
  • pre-commit run --all-files passes

Also verified that 7 of the 9 existing review threads are addressed at the current head, so those can likely be resolved.

3 inline comments below: one doc fix on the port-layout comment, one latent mutable-default-argument footgun, and one optional set of boundary tests (pre-verified to pass).

Generated by Claude Code

Comment thread nemo_rl/distributed/virtual_cluster.py Resolved
@terrykong

Copy link
Copy Markdown
Collaborator

left an aireview @xiuhu17 . thanks for the contribution

there's another related PR i think we should sequence in first #3103

I think that one is on track to merge very soon, and this PR touches some stuff that pr is trying to delete.

@saumishr could you review this one in light of your PR?

@terrykong
terrykong requested a review from saumishr July 10, 2026 20:01
@terrykong terrykong added the CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) label Jul 10, 2026
@xiuhu17
xiuhu17 force-pushed the port_migrate branch 2 times, most recently from fa3cab0 to 7a78998 Compare July 14, 2026 02:23
Replace the TOCTOU-prone ephemeral-port probing in the SGLang backend
(find_available_port / get_free_port random walks) with the shared
reserved-band helpers in virtual_cluster.py:

- new _get_free_consecutive_ports_local() scans a bounded band and
  threads a per-node cursor so blocks never overlap on a node
- engine server/NCCL/dist_init ports come from the generation band
  (policy.generation.port_range_low/high)
- router and Prometheus ports get dedicated hard-coded bands
- workers expose _get_current_free_port/_get_current_node_ip instead of
  _get_current_node_ip_and_free_port
- port_cursors/node_port_cursor use None sentinels (no mutable defaults)
- boundary tests for the new helper

Closes NVIDIA-NeMo#2874

Signed-off-by: zhihaow6 <zhihaow6@illinois.edu>
@nvpohanh

Copy link
Copy Markdown

@terrykong #3103 has been merged. Could you review this PR again? thanks

@terrykong terrykong left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great work @xiuhu17 !

thanks for addressing review. lgtm

@terrykong
terrykong enabled auto-merge (squash) July 14, 2026 17:00
@terrykong

Copy link
Copy Markdown
Collaborator

/ok to test 08b3e91

@svcnvidia-nemo-ci svcnvidia-nemo-ci removed the waiting-on-customer Waiting on the original author to respond label Jul 14, 2026
@terrykong
terrykong merged commit 5a7e877 into NVIDIA-NeMo:main Jul 14, 2026
73 of 75 checks passed
@xiuhu17

xiuhu17 commented Jul 14, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @terrykong !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) community-request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use port range in SGLang generation backend

5 participants