Repository navigation
refactor(sglang): allocate ports from reserved ranges - #2997
Conversation
yuki-97
left a comment
There was a problem hiding this comment.
thanks @xiuhu17 , LGTM and just some nits. also could you help to rebase? looks there's a conflict with main branch.
@terrykong could you take a review as well?
terrykong
left a comment
There was a problem hiding this comment.
Review summary
Thanks @xiuhu17 — nice cleanup consolidating SGLang port allocation onto the shared reserved-band helpers in virtual_cluster.py; this closes the loop on #2874 and removes the TOCTOU-prone ephemeral-port probing. Overall the change looks correct: we traced the cursor arithmetic, checked for dangling references to the removed helpers, import cycles, and the router/Prometheus band layout, and found no live correctness issues.
Main blocker (echoing @yuki-97): the branch still has merge conflicts with main — a rebase is needed before this can go in.
Verified locally on this head (1457738):
tests/unit/distributed/test_virtual_cluster.py::TestGetFreeConsecutivePortsLocal— 6/6 pass; full file 37/38 (the one failure,test_mcore_py_executable, is pre-existing and unrelated to this PR)- sglang unit tests:
test_utils_smoke.py4/4,test_sglang_router.py10/10 pass pre-commit run --all-filespasses
Also verified that 7 of the 9 existing review threads are addressed at the current head, so those can likely be resolved.
3 inline comments below: one doc fix on the port-layout comment, one latent mutable-default-argument footgun, and one optional set of boundary tests (pre-verified to pass).
Generated by Claude Code
fa3cab0 to
7a78998
Compare
Replace the TOCTOU-prone ephemeral-port probing in the SGLang backend (find_available_port / get_free_port random walks) with the shared reserved-band helpers in virtual_cluster.py: - new _get_free_consecutive_ports_local() scans a bounded band and threads a per-node cursor so blocks never overlap on a node - engine server/NCCL/dist_init ports come from the generation band (policy.generation.port_range_low/high) - router and Prometheus ports get dedicated hard-coded bands - workers expose _get_current_free_port/_get_current_node_ip instead of _get_current_node_ip_and_free_port - port_cursors/node_port_cursor use None sentinels (no mutable defaults) - boundary tests for the new helper Closes NVIDIA-NeMo#2874 Signed-off-by: zhihaow6 <zhihaow6@illinois.edu>
|
@terrykong #3103 has been merged. Could you review this PR again? thanks |
|
/ok to test 08b3e91 |
|
Thanks @terrykong ! |
Summary
The SGLang backend allocated ports with its own helpers (
find_available_port,is_port_available,get_free_port) that probe OS-ephemeral ports, instead ofthe shared range-based primitives in
virtual_cluster.pyalready used by vLLMand NeMo-Gym. This re-introduced the TOCTOU-prone allocation that #2380 removed,
just in a different place.
This PR rebases SGLang onto the shared helpers and reserved port map, while
keeping the SGLang-specific bits the shared helpers don't cover (contiguous
dp_attention port blocks, a per-node cursor, IPv6 bracketing, non-Ray host lookup).
Closes #2874
What changed
virtual_cluster.py: added_get_free_consecutive_ports_local()(returns acontiguous block of bindable ports in a reserved range — the one capability the
existing helpers lacked) and a dedicated router/Prometheus band (28001–30000).
ray_utils.py: removedfind_available_port/is_port_available/get_free_port;get_current_node_ipnow delegates to_get_node_ip_local().get_host_infoand the NOSET env list are unchanged.ip_port_utils.py/sglang_worker.py: allocate through the shared helperwith
port_range_low/high+ a per-node cursor instead of a hard-codedbase_port=15000.sglang_generation.py: readsport_range_low/highfrom the generation config.sglang_router.py: router/Prometheus ports from_get_free_port_localover their band.TestGetFreeConsecutivePortsLocal; updated SGLang tests/helpersthat used the removed functions.
_patch_sglang_custom_all_reduce_v2_tms_cudagraphonto the final form of sgl-project/sglang#27948 (tracking radixark/miles#1176): instead of the earlier revision'sset_cuda_graph_register_inputstoggle threaded through five files, it applies the upstream Python-only fix tocustom_all_reduce_v2.py— adding theenvsimport andself.tms_cudagraphflag, and gating bothset_cuda_graph_capture(True)calls onnot self.tms_cudagraph— so the kernel's capture flag stays off under TMS and we no longer touch the fragile jit_kernel C++/.cuhsources.Port layout
Testing
pytest tests/unit/distributed/test_virtual_cluster.py::TestGetFreeConsecutivePortsLocalpytest -m sglang tests/unit/models/generation/sglang/