Repository navigation
feat(runtime): support explicit transport host overrides - #11870
Conversation
9c930e0 to
9a30d4a
Compare
WalkthroughThe change adds ChangesEvent plane networking
Estimated code review effort: 3 (Moderate) | ~25 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Full details: Description checkExplanation The description explains the main changes and validation, but it omits the required Related Issues section and does not identify specific files for reviewer focus. It also uses Summary and Validation headings instead of the template's Overview and Details headings. Resolution Add the required Related Issues section and select either a linked issue, such as
Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
lib/runtime/src/transports/event_plane/mod.rs (1)
431-456: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winBind the ZMQ listener in the advertised address family. Direct mode always binds
tcp://0.0.0.0:0, which is IPv4-only, butDYN_EVENT_PLANE_HOSTcan advertise[IPv6]:port. Subscribers will discover an unreachable IPv6 endpoint. The underlying bind implementation preserves that IPv4 endpoint inlib/runtime/src/transports/event_plane/zmq_transport.rs, lines 96–125.
lib/runtime/src/transports/event_plane/mod.rs#L431-L456: select an IPv6 wildcard bind endpoint (for example,tcp://[::]:0) when advertising an IPv6 address, or reject IPv6 configuration.docs/design-docs/event-plane.md#L93-L96: retain the IPv6 claim only after matching-family binding works; otherwise document IPv4-only support.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/runtime/src/transports/event_plane/mod.rs` around lines 431 - 456, Update the direct-mode binding in event-plane initialization around event_plane_host_from_env and ZmqPubTransport::bind to choose an IPv6 wildcard endpoint when advertised_host is IPv6, or explicitly reject IPv6 configuration if matching-family binding is unsupported; preserve IPv4 behavior otherwise. In docs/design-docs/event-plane.md lines 93-96, retain the IPv6 support claim only if the implementation binds IPv6 correctly; otherwise document that direct mode supports IPv4 only.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@lib/runtime/src/transports/event_plane/mod.rs`:
- Around line 59-68: Update the host validation around the parsed `IpAddr` to
reject `ip.is_unspecified()` before formatting the advertised address, covering
both `0.0.0.0` and `::` with an appropriate validation error. Preserve the
existing IPv4 formatting, bracketed IPv6 formatting, and invalid-address
handling.
---
Outside diff comments:
In `@lib/runtime/src/transports/event_plane/mod.rs`:
- Around line 431-456: Update the direct-mode binding in event-plane
initialization around event_plane_host_from_env and ZmqPubTransport::bind to
choose an IPv6 wildcard endpoint when advertised_host is IPv6, or explicitly
reject IPv6 configuration if matching-family binding is unsupported; preserve
IPv4 behavior otherwise. In docs/design-docs/event-plane.md lines 93-96, retain
the IPv6 support claim only if the implementation binds IPv6 correctly;
otherwise document that direct mode supports IPv4 only.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: e5816421-9f16-4a76-9534-a6ead1ac1be4
📒 Files selected for processing (6)
components/src/dynamo/common/snapshot/constants.pycomponents/src/dynamo/common/tests/test_snapshot_restore_context.pydocs/design-docs/event-plane.mdlib/runtime/src/config/environment_names.rslib/runtime/src/pipeline/network/tcp/server.rslib/runtime/src/transports/event_plane/mod.rs
jthomson04
left a comment
There was a problem hiding this comment.
Reviewed the transport host overrides. The change is clean and well-tested, and the String → IpAddr/SocketAddr refactor is a solid improvement (IPv6 bracketing stays consistent with the auto-detect path, and validation fails fast before any bind).
One thing I'd like to resolve before merge, plus a couple of optional notes — left inline:
- (please address)
DYN_TCP_RESPONSE_STREAM_HOSTisn't preserved across snapshot restore, unlike its new siblingDYN_EVENT_PLANE_HOST. If a deployment pins the TCP host to a node-specific IP literal, that value freezes at snapshot time and the restored process will try to bind the old node's IP. See comment onconstants.py. - (optional) API asymmetry: the event-plane host accepts IP literals only, while the TCP host also accepts interface names (IPv4-only resolution). See comment on
event_plane/mod.rs. - (optional) The
DYN_TCP_RESPONSE_STREAM_HOSTinterface-name behavior change isn't documented. See comment onevent-plane.md.
Minor cosmetic (no inline): after the IpAddr change, tracing::debug!("tcp transport service on {local_ip}:{local_port}") in tcp/server.rs prints an unbracketed IPv6 (…::1:8080) — ambiguous in logs; consider logging the SocketAddr.
|
@coderabbitai full review |
✅ Action performedFull review finished. |
jthomson04
left a comment
There was a problem hiding this comment.
Code review
Clean, well-tested change. Comments below are mostly minor.
Strengths
- Good refactor — moving
IpResolver/DefaultIpResolverintoip_resolver.rs(server re-exports them) removes the priorip_resolver → serverimport cycle and centralizes resolution. The default trait method forlist_afinet_netifaskeeps the existingserver.rstest mock compiling untouched. SocketAddris a real correctness fix, not just cosmetic — the oldformat!("{}:{}", local_ip, port)produced malformed::1:portfor the IPv6 auto-detect fallback;SocketAddr::to_string()correctly brackets it ([::1]:port).- Deterministic interface selection — the old path collapsed interfaces into a
HashMap<String, IpAddr>, silently discarding all but one arbitrary address per name. Sorting + lowest-IPv4 is a genuine improvement. - Strong test coverage — literal / interface / IPv6-reject / unspecified-reject / no-usable-IPv4 / not-found, plus valid+invalid publisher integration tests and the snapshot apply-and-clear update. Validation runs before the ZMQ bind, so an invalid host fails fast.
Issues & suggestions
1. Default event-plane path changed for IPv6-only hosts (low, likely intended). The direct-ZMQ default moved from local_ip_for_advertise() (returns bracketed IPv6 when no IPv4 exists) to local_ipv4_for_advertise(), which falls back to 127.0.0.1. Since direct ZMQ always binds the IPv4 wildcard (tcp://0.0.0.0:0), the old IPv6 advertisement was already unreachable — so this is arguably a fix, but worth a line in the PR description: an IPv6-only host now advertises loopback instead of a scoped-but-broken IPv6.
2. "Lowest usable IPv4" only excludes 0.0.0.0 (low). resolve_advertised_ipv4 filters unspecified addresses but not loopback (127.0.0.0/8) or link-local (169.254.0.0/16). If an interface carries such an address alongside a routable one and it sorts lower (e.g. 169.254.x < 172.16.x), the non-routable address wins. Won't bite the documented IB/IPoIB case, but "lowest" ≠ "lowest routable" — consider filtering loopback/link-local, or noting the assumption in the docs.
3. IP literals aren't verified against local interfaces (low). Interface names are validated against list_afinet_netifas(); an IP literal is only checked for "IPv4 and not unspecified." A typo'd/foreign IP silently becomes the advertised address while the socket binds elsewhere. Matches the "explicit override = operator's responsibility" model, so defensible — just flagging the asymmetry.
4. Snapshot constant grouping is slightly off (nit). In constants.py, DYN_EVENT_PLANE_HOST sits under "Parsed Python runtime config that must also refresh the in-memory config passed to create_runtime()", but it's read only on the Rust side at publisher creation. It fits better next to DYN_TCP_RESPONSE_STREAM_HOST/PORT under "infrastructure env read after restore." RESTORE_RUNTIME_ENV_NAMES is a set, so this is comment accuracy only, not behavior.
5. ServerOptions.interface is now a misnomer (nit). The field accepts an IP literal or an interface name; the name interface (and the Interface not found error text) under-describes it. Rename is out of scope, but a doc comment noting it accepts either would help future readers.
6. Observation (no action): the two transports intentionally differ — the TCP response-stream server binds to the resolved IP, while direct ZMQ binds the wildcard and only changes the advertised address. Both preserve prior per-transport behavior, so correct; just worth being explicit that DYN_TCP_RESPONSE_STREAM_HOST restricts the listen address whereas DYN_EVENT_PLANE_HOST does not.
Verdict
Approve with minor nits. The only ones I'd act on before merge are #1 (one line in the PR description) and #4 (move the constant under the right comment); the rest are optional polish.
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
49a224b to
fdb106f
Compare
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Interface-name resolution picked the last enumerated address of the matching interface without filtering, so an IPv6 link-local address (fe80::/10) could win and TcpListener::bind then fails with EINVAL on Linux. Build on ai-dynamo#11870: resolve_host_or_interface now skips link-local, multicast, broadcast, and unspecified interface addresses and uses the last usable unicast address of the interface. Explicit IP literals are unaffected, and loopback interfaces like `lo` remain usable. This also fixes DYN_EVENT_PLANE_HOST, which shares the same resolution path. Signed-off-by: smb2love <smb2love@users.noreply.github.com>
The Dynamo fork pin (upstream 2026-08-07) cannot parse an interface-name DYN_TCP_RPC_HOST, which the batch3 GB300 nodes export; upstream resolves it since ai-dynamo/dynamo#11870 and v1.6.1 also contains #12162. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…disagg Dynamo 1.4.0.dev20260730 parses DYN_TCP_RPC_HOST as a raw socket address, so the gb300-nv node value enP22s22f0np0 (an interface name) fails with 'Invalid TCP bind address'. Dynamo resolves interface names since ai-dynamo/dynamo#11870 (2026-09-05); 1.6.0.dev20261010 matches the green gb300-nv dsv41flash Dynamo vLLM recipes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Summary
DYN_EVENT_PLANE_HOSTto override the address advertised by direct ZMQ event publishers.DYN_EVENT_PLANE_HOSTandDYN_TCP_RESPONSE_STREAM_HOSTone contract: each accepts an IPv4 or IPv6 literal, or an exact network interface name.SocketAddr, preserve the transport host settings andDYN_TCP_RPC_HOSTacross snapshot restore, and document the supported configuration.Direct ZMQ continues to bind
0.0.0.0; this PR changes its advertised discovery address only. Full IPv6 binding and address-selection policy remain in #13126. NATS and ZMQ broker mode are unchanged.Validation
cargo fmt --all -- --checkcargo test -p dynamo-runtime utils::ip_resolver -- --test-threads=1— 10 passedcargo test -p dynamo-runtime event_plane -- --test-threads=1— 43 passedcargo test -p dynamo-runtime --lib pipeline::network::tcp::server::tests -- --test-threads=1— 39 passedcargo check -p dynamo-runtime --all-targetscargo clippy -p dynamo-runtime --lib --tests -- -D warningsPYTHONPATH=components/src uv run --no-project --with pytest --with pytest-asyncio python -m pytest -c /dev/null -p no:cacheprovider --asyncio-mode=auto components/src/dynamo/common/tests/test_snapshot_restore_context.py— 5 passedfern checkfern docs broken-links