feat: add `vmnet_network` support on macOS 26+ by norio-nomura · Pull Request #4394 · lima-vm/lima

norio-nomura · 2025-11-22T12:55:34Z

vmnet_network support on macOS 26+

On vmType: vz, it uses VZVmnetNetworkDeviceAttachment
On vmType: qemu|krunkit, it uses vmnet_interface_start_with_network

Depends on Code-Hex/vz#205

About what can be done and the restrictions with this API:

Supports using vmnet without root privilege
Supports VMNET_SHARED_MODE, and VMNET_HOST_MODE
Like vzNAT, the "bridge10x" interface is created on the host
Not support VMNET_BRIDGED_MODE
- documented in vmnet_network_configuration_create
To share vmnet_network_ref with multiple processes, it must be the same executable file.
- documented in - initWithNetwork:
- This restriction seems to be imposed by VZVmnetNetworkDeviceAttachment instead of vmnet_network_create.
- ~~CDHash is probably used to determine "same executable file". TN3126: Inside Code Signing: Hashes~~
  - ~~Redoing codesign with "Developer ID", where the hash of the executable file changes, does not interfere with the sharing of vmnet_network_ref.~~
  - ~~CDHash does not change before and after redoing codesign with "Developer ID".~~
  - ~~Even if it is codesign using the same "Developer ID", if CDHash is changed due to source code changes, etc., vmnet_network_ref cannot be shared.~~
- On macOS 26.2, "same executable file" restriction seems to be relaxed; The external vz driver can connect to subnet that created by limactl process.
The network created based on vmnet_network_ref is owned by the process that created vmnet_network_ref, not the process that started the network based on the shared vmnet_network_ref.
- When the owner process terminates, it also terminates the owned network, regardless of the presence or absence of connected VMs.
The serialization of vmnet_network_ref is done in the form of xpc_object_t.
- It is possible to share byte sequences included in the serialized xpc_object_t outside the XPC API, but to avoid relying on private information, use the XPC API for sharing.

This PR allows VMNET_SHARED_MODE and VMNET_HOST_MODE to be used under these conditions.

Functions to be added

`networks.yaml`

vmnet:
  shared:
    mode: shared
    dhcp: true
    dnsProxy: true
    mtu: 1500
    nat44: true
    nat66: true
    routerAdvertisement: true
    subnet: 192.168.107.0/24
  host:
    mode: host
    dhcp: true
    dnsProxy: true
    mtu: 1500
    nat44: true
    nat66: true
    # host mode ignores routerAdvertisement setting
    routerAdvertisement: false
    subnet: 192.168.108.0/24

`lima.yaml`

# The vmnet network interfaces "shared" and "host" can be used with `vmType: vz`, `vmType: qemu`, or `vmType: krunkit`.
# Requires macOS 26.0 or later.
- vmnet: shared
- vmnet: host

`limactl vmnet`

$ limactl vmnet
Run vmnet

Usage:
  limactl vmnet [flags]

Flags:
  -h, --help                      help for vmnet
      --unregister-mach-service   Unregister Mach service

Global Flags:
      --debug               Debug mode
      --log-format string   Set the logging format [text, json] (default "text")
      --log-level string    Set the logging level [trace, debug, info, warn, error]
      --tty                 Enable TUI interactions such as opening an editor. Defaults to true when stdout is a terminal. Set to false for automation. (default true)
  -y, --yes                 Alias of --tty=false

Registration as a Mach service is automatically done by hostagent when the instance using - vmnet: * is started, so only unregistering is visible from the user.
The registered Mach service is displayed as "io.lima-vm.vmnet.sh" in macOS System Settings (General > Login Items & Extensions).
As a Mach service process, it is started by launchd as appropriate at the request of the supported drivers.
~~The Mach service responds to requests only if the executable file of the VZ driver process that sent the request has the same CDHash as itself. It uses xpc_peer_requirement.~~ This restriction is removed on macOS 26.2+.
The Mach service process creates and retains vmnet_network_ref and provides serialization data to the VZ driver.
If QEMU driver or krunkit driver request file descriptor to passing qemu or krunkit, the Mach service process creates vzvmnet.*FileAdaptorForInterface, and provides file descriptor to communicating packets with the adaptor.
When the Mach service detects the termination of the network started by the supported driver based on the serialization data, it discards vmnet_network_ref.
When all the vmnet_network_ref to be retained is gone, the Mach service process will terminate.

Additional changes

Because shutdown takes longer on using VmnetNetwork:
- Extend VZ driver's shutdown timeout from 5 seconds to 15 seconds
- Add ExitTimeOut key with 20 seconds to autostart io.lima-vm.autostart.INSTANCE.plist

Benchmark: `iperf3` between VMs on the same network

MacBook Pro 14 inch, 2023
CPU: Apple M2 Pro
Memory: 16GB
macOS Tahoe 26.2

$ ssh lima-debug ip a show scope global dev lima1|grep -E "^ +inet "
    inet 192.168.105.11/24 metric 100 brd 192.168.105.255 scope global dynamic lima1
$ ssh lima-debug ip a show scope global dev lima3|grep -E "^ +inet "
    inet 192.168.107.8/24 metric 100 brd 192.168.107.255 scope global dynamic lima3
$ ssh lima-docker ip a show scope global dev lima1|grep -E "^ +inet "
    inet 192.168.105.10/24 metric 100 brd 192.168.105.255 scope global dynamic lima1
$ ssh lima-docker ip a show scope global dev lima3|grep -E "^ +inet "
    inet 192.168.107.7/24 metric 100 brd 192.168.107.255 scope global dynamic lima3

socket_vmnet

$ ssh lima-docker iperf3 -c 192.168.105.11
Connecting to host 192.168.105.11, port 5201
[  5] local 192.168.105.10 port 47434 connected to 192.168.105.11 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   148 MBytes  1.24 Gbits/sec    0   3.97 MBytes       
[  5]   1.00-2.00   sec   147 MBytes  1.24 Gbits/sec    0   3.97 MBytes       
[  5]   2.00-3.00   sec   148 MBytes  1.25 Gbits/sec    0   3.97 MBytes       
[  5]   3.00-4.00   sec   149 MBytes  1.25 Gbits/sec    0   3.97 MBytes       
[  5]   4.00-5.00   sec   147 MBytes  1.23 Gbits/sec    0   3.97 MBytes       
[  5]   5.00-6.00   sec   149 MBytes  1.25 Gbits/sec    0   3.97 MBytes       
[  5]   6.00-7.00   sec   148 MBytes  1.24 Gbits/sec    0   3.97 MBytes       
[  5]   7.00-8.00   sec   150 MBytes  1.26 Gbits/sec    0   3.97 MBytes       
[  5]   8.00-9.00   sec   148 MBytes  1.25 Gbits/sec    0   3.97 MBytes       
[  5]   9.00-10.00  sec   147 MBytes  1.23 Gbits/sec    0   3.97 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.45 GBytes  1.24 Gbits/sec    0            sender
[  5]   0.00-10.02  sec  1.45 GBytes  1.24 Gbits/sec                  receiver

iperf Done.

vmnet: shared

$ ssh lima-docker iperf3 -c 192.168.107.8
Connecting to host 192.168.107.8, port 5201
[  5] local 192.168.107.7 port 54642 connected to 192.168.107.8 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.01   sec  3.38 GBytes  28.8 Gbits/sec    0   4.07 MBytes       
[  5]   1.01-2.00   sec  3.47 GBytes  29.9 Gbits/sec    0   4.07 MBytes       
[  5]   2.00-3.00   sec  3.42 GBytes  29.4 Gbits/sec    0   4.07 MBytes       
[  5]   3.00-4.00   sec  3.45 GBytes  29.7 Gbits/sec    0   4.07 MBytes       
[  5]   4.00-5.00   sec  3.38 GBytes  29.0 Gbits/sec    0   4.07 MBytes       
[  5]   5.00-6.00   sec  3.42 GBytes  29.4 Gbits/sec    0   4.07 MBytes       
[  5]   6.00-7.00   sec  3.37 GBytes  28.9 Gbits/sec    0   4.07 MBytes       
[  5]   7.00-8.00   sec  3.29 GBytes  28.3 Gbits/sec    0   4.07 MBytes       
[  5]   8.00-9.00   sec  3.24 GBytes  27.9 Gbits/sec    0   4.07 MBytes       
[  5]   9.00-10.00  sec  3.45 GBytes  29.6 Gbits/sec    0   4.07 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  33.9 GBytes  29.1 Gbits/sec    0            sender
[  5]   0.00-10.00  sec  33.9 GBytes  29.1 Gbits/sec                  receiver

iperf Done.

templates/default.yaml

AkihiroSuda · 2025-11-22T18:00:33Z

Currently, both vzShared and vzHost have almost the same functions as vzNAT.

Any advantage using them?

norio-nomura · 2025-11-22T23:21:54Z

Currently, both vzShared and vzHost have almost the same functions as vzNAT.

Any advantage using them?

As far as the API is concerned, customization that is not supported by vzNAT should be possible.
https://github.com/Code-Hex/vz/pull/205/files#diff-d7ec6a7a97f55e264883085a4676c2b0a5466a657e0e87312accd40caa67295dR80-R106
I still don't know what can be done with those APIs (including unimplemented).

norio-nomura · 2025-12-17T04:16:25Z

Updated PR description.

norio-nomura · 2025-12-17T04:52:25Z

Most of the functions I wanted at the start of this PR have been realized.

go.mod

norio-nomura · 2026-01-15T01:55:37Z

Updated PR description.

Updated krunkit performance log:

$ iperf3 -c 192.168.107.10 --bidir
Connecting to host 192.168.107.10, port 5201
[  5] local 192.168.107.1 port 54776 connected to 192.168.107.10 port 5201
[  7] local 192.168.107.1 port 54777 connected to 192.168.107.10 port 5201
[ ID][Role] Interval           Transfer     Bitrate
[  5][TX-C]   0.00-1.00   sec  2.41 GBytes  20.7 Gbits/sec                  
[  7][RX-C]   0.00-1.00   sec  1.70 GBytes  14.6 Gbits/sec                  
[  5][TX-C]   1.00-2.00   sec  2.27 GBytes  19.5 Gbits/sec                  
[  7][RX-C]   1.00-2.00   sec  1.60 GBytes  13.7 Gbits/sec                  
[  5][TX-C]   2.00-3.00   sec  2.26 GBytes  19.5 Gbits/sec                  
[  7][RX-C]   2.00-3.00   sec  1.62 GBytes  13.9 Gbits/sec                  
[  5][TX-C]   3.00-4.00   sec  2.31 GBytes  19.8 Gbits/sec                  
[  7][RX-C]   3.00-4.00   sec  1.63 GBytes  14.0 Gbits/sec                  
[  5][TX-C]   4.00-5.00   sec  2.36 GBytes  20.3 Gbits/sec                  
[  7][RX-C]   4.00-5.00   sec  1.69 GBytes  14.6 Gbits/sec                  
[  5][TX-C]   5.00-6.00   sec  2.26 GBytes  19.4 Gbits/sec                  
[  7][RX-C]   5.00-6.00   sec  1.60 GBytes  13.7 Gbits/sec                  
[  5][TX-C]   6.00-7.00   sec  2.26 GBytes  19.4 Gbits/sec                  
[  7][RX-C]   6.00-7.00   sec  1.58 GBytes  13.5 Gbits/sec                  
[  5][TX-C]   7.00-8.00   sec  2.33 GBytes  20.0 Gbits/sec                  
[  7][RX-C]   7.00-8.00   sec  1.66 GBytes  14.3 Gbits/sec                  
[  5][TX-C]   8.00-9.00   sec  2.39 GBytes  20.5 Gbits/sec                  
[  7][RX-C]   8.00-9.00   sec  1.69 GBytes  14.6 Gbits/sec                  
[  5][TX-C]   9.00-10.00  sec  2.30 GBytes  19.7 Gbits/sec                  
[  7][RX-C]   9.00-10.00  sec  1.64 GBytes  14.1 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-10.00  sec  23.2 GBytes  19.9 Gbits/sec                  sender
[  5][TX-C]   0.00-10.00  sec  23.2 GBytes  19.9 Gbits/sec                  receiver
[  7][RX-C]   0.00-10.00  sec  16.4 GBytes  14.1 Gbits/sec    0            sender
[  7][RX-C]   0.00-10.00  sec  16.4 GBytes  14.1 Gbits/sec                  receiver

iperf Done.

There are times when TX performance deteriorates, like #4394 (comment), but the cause is not clear.

Document the XPC protocol for developers who want to build their own client library or understand how the broker communicates with clients.

norio-nomura · 2026-01-16T01:32:56Z

There are times when TX performance deteriorates, like #4394 (comment), but the cause is not clear.

I found that krunkit's TX performance changes depending on how the subnet is started.

Update: TX performance degradation occurs when an interface that has not enabled TSO is connected to the subnet. It does not depend on the interface that started the subnet.
QEMU does not support TSO, but it works even if it is enabled, so enabling TSO when creating an interface for QEMU will prevent krunkit's TX performance degradation.

Based on `VMNET_SHARED_MODE`, and `VMNET_HOST_MODE` ```yaml networks: - vzShared: true - vzHost: true ``` But, to sharing network between multiple VMs, `VZVmnetNetworkDeviceAttachment` requires VMs are launched by same process. It depends on Code-Hex/vz#205 Signed-off-by: Norio Nomura <norio.nomura@gmail.com> # Conflicts: # go.sum # Conflicts: # go.sum

It provides `vzvmnet.Network` serialization to VMs. `limactl vmnet` takes flags: - `--unregister-mach-service`: unregister Mach service from `launchd` - There is no need to register manually because the VZ driver registers as appropriate. - `--mach-service=<service name>`: launched as Mach server by `launchd` via `io.lima-vm.vmnet.plist` - Launched on demand to connection from clients by `launchd`. - Receives a request payload from clients with fields: - `Network`: name of the network ("shared", "host", etc) - `Configuration`: `[]bytes@ representing `VzNetworkConfig` in JSON. - Validates clients are the same executable (cdhash) by using xpc_peer_requirement API. - Create `vzvmnet.Network` from provided `Configuration` if cached one does not exist. - Replies to clients with fields: - `Configuration`: If `vzvmnet.Network` is cached, it may be created by different configuration. - `Serialization`: newly created or cached. - Monitors changes of networks - When the interface created by `vzvmnet.Network` disappears from host, remove them from cache. - If all `vzvmnet.Network` are removed, `limactl vmnet` exits. Clients (hostagent) does: - Read `.vmnet` VmnetConfig from `networks.yaml` - Use them on `- vmnet: <network>` fields; "shared" and "host" network are predefined. - Register `limactl vmnet` to `launchd` if not registered. - Request serialization to the Mach service "io.lima-vm.vmnet". - Create `vzvmnet.Network` by provided serialization, then use them. Additional changes: - Because shutdown takes longer on using `vzvmnet.Network`: - Extend VZ driver's shutdown timeout from 5 seconds to 15 seconds - Add `ExitTimeOut` key with 20 seconds to autostart `io.lima-vm.autostart.INSTANCE.plist` - `lima.yaml`: `- vzShared` and `- vzHost` are renamed to `- vmnet: shared` and `- vmnet: host` Signed-off-by: Norio Nomura <norio.nomura@gmail.com>

## In QEMU, there are six options for `-netdev` that uses vmnet. It can be select by `_LIMA_QEMU_VMNET_BACKEND` environment variable. - `datagram`, `dgram`: use `-netdev dgram,...` with `vzvmnet.DatagramFileAdaptorForInterface` (recvmsg/sendmsg) - `datagram_next`, `dgram_next`: use `-netdev dgram,...` with `vzvmnet.DatagramNextFileAdaptorForInterface` (recvmsg_x/sendmsg_x) - `socket`: use `-netdev socket,...` with `vzvmnet.StreamFileAdaptorForInterface` - `stream`: use `-netdev stream,...` with `vzvmnet.StreamFileAdaptorForInterface` - `tap`: use `-netdev tap,...` with `vzvmnet.DatagramFileAdaptorForInterface` (recvmsg/sendmsg) - `tap_next`: use `-netdev tap,...` with `vzvmnet.DatagramNextFileAdaptorForInterface` (recvmsg_x/sendmsg_x) The default is `tap` and the best performance with iperf3 is selected (although not a big difference). ## In krunkit, there are three options for `--device virtio-net` that uses vmnet. It can be selected by `_LIMA_KRUNKIT_VMNET_BACKEND` environment variable. - `datagram`, `dgram`: use `type=unixgram,...` with `vzvmnet.DatagramFileAdaptorForInterface` (recvmsg/sendmsg) - `datagram_next`, `dgram_next`: use `type=unixgram,...` with `vzvmnet.DatagramNextFileAdaptorForInterface` (recvmsg_x/sendmsg_x) - `stream`: use `type=unixstream,...` with `vzvmnet.StreamFileAdaptorForInterface` The default is `stream` because: - In the simple iperf3 test, `dgram` performs the best (than vz), but when `iperf3` runs with the `-P 3` option, `krunkit` will stop the network device after outputting the `ENOBUFS` error: > [2026-01-21T05:02:28Z ERROR devices::virtio::net::worker] Failed to process rx: Backend(Internal(ENOBUFS)) (triggered by backend socket readable) - `stream` records retry in `iperf3` but does not stop. Signed-off-by: Norio Nomura <norio.nomura@gmail.com>

Signed-off-by: Norio Nomura <norio.nomura@gmail.com>

AkihiroSuda · 2026-03-08T17:34:57Z

Let me remove this from v2.1 milestone
Hopefully we can merge this in v2.2

This comment was marked as outdated.

Sign in to view

AkihiroSuda reviewed Nov 22, 2025

View reviewed changes

templates/default.yaml Outdated Show resolved Hide resolved

norio-nomura force-pushed the use-vz-vmnet-network-device-attachment branch from 07eebec to 270556e Compare December 3, 2025 07:59

This comment was marked as outdated.

Sign in to view

norio-nomura force-pushed the use-vz-vmnet-network-device-attachment branch 2 times, most recently from 81d4cc6 to 57553f8 Compare December 4, 2025 01:54

norio-nomura force-pushed the use-vz-vmnet-network-device-attachment branch 4 times, most recently from 433432d to adf5456 Compare December 15, 2025 05:32

This comment was marked as resolved.

Sign in to view

norio-nomura force-pushed the use-vz-vmnet-network-device-attachment branch 9 times, most recently from 4cd78ff to 8cc5f26 Compare December 17, 2025 04:07

norio-nomura marked this pull request as ready for review December 17, 2025 04:52

AkihiroSuda reviewed Dec 17, 2025

View reviewed changes

go.mod Outdated Show resolved Hide resolved

AkihiroSuda added this to the v2.1.0 (?) milestone Dec 17, 2025

AkihiroSuda added the component/vz label Dec 17, 2025

AkihiroSuda marked this pull request as draft December 17, 2025 05:09

norio-nomura marked this pull request as ready for review December 17, 2025 05:10

norio-nomura force-pushed the use-vz-vmnet-network-device-attachment branch from 5f7ec7b to b7c39e0 Compare January 14, 2026 07:17

This comment was marked as outdated.

Sign in to view

norio-nomura force-pushed the use-vz-vmnet-network-device-attachment branch 4 times, most recently from 6e02bc4 to a5d58f3 Compare January 15, 2026 00:46

norio-nomura changed the title ~~feat: add VZVmnetNetworkDeviceAttachment support on macOS 26+~~ feat: add vmnet_network support on macOS 26+ Jan 15, 2026

norio-nomura force-pushed the use-vz-vmnet-network-device-attachment branch from a5d58f3 to c078397 Compare January 15, 2026 01:52

norio-nomura force-pushed the use-vz-vmnet-network-device-attachment branch from c078397 to 024f35d Compare January 15, 2026 02:40

norio-nomura referenced this pull request in nirs/vmnet-broker Jan 15, 2026

Add protocol documentation

53d9453

Document the XPC protocol for developers who want to build their own client library or understand how the broker communicates with clients.

norio-nomura force-pushed the use-vz-vmnet-network-device-attachment branch 2 times, most recently from c0d1289 to a5a0f2b Compare January 15, 2026 03:49

This comment was marked as outdated.

Sign in to view

norio-nomura force-pushed the use-vz-vmnet-network-device-attachment branch 8 times, most recently from be97162 to e406ddd Compare January 21, 2026 05:03

norio-nomura added 4 commits February 10, 2026 00:08

WIP

a35b0f2

Signed-off-by: Norio Nomura <norio.nomura@gmail.com>

norio-nomura force-pushed the use-vz-vmnet-network-device-attachment branch from e406ddd to a35b0f2 Compare February 10, 2026 03:36

AkihiroSuda removed this from the v2.1.0 milestone Mar 8, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add `vmnet_network` support on macOS 26+#4394

feat: add `vmnet_network` support on macOS 26+#4394
norio-nomura wants to merge 4 commits intolima-vm:masterfrom
norio-nomura:use-vz-vmnet-network-device-attachment

norio-nomura commented Nov 22, 2025 •

edited

Loading

Uh oh!

This comment was marked as outdated.

Uh oh!

AkihiroSuda commented Nov 22, 2025

Uh oh!

norio-nomura commented Nov 22, 2025

Uh oh!

This comment was marked as outdated.

This comment was marked as resolved.

norio-nomura commented Dec 17, 2025

Uh oh!

norio-nomura commented Dec 17, 2025

Uh oh!

Uh oh!

This comment was marked as outdated.

norio-nomura commented Jan 15, 2026

Uh oh!

This comment was marked as outdated.

norio-nomura commented Jan 16, 2026 •

edited

Loading

Uh oh!

AkihiroSuda commented Mar 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

norio-nomura commented Nov 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

vmnet_network support on macOS 26+

About what can be done and the restrictions with this API:

Functions to be added

networks.yaml

lima.yaml

limactl vmnet

Additional changes

Benchmark: iperf3 between VMs on the same network

socket_vmnet

vmnet: shared

Uh oh!

This comment was marked as outdated.

Uh oh!

AkihiroSuda commented Nov 22, 2025

Uh oh!

norio-nomura commented Nov 22, 2025

Uh oh!

This comment was marked as outdated.

This comment was marked as resolved.

norio-nomura commented Dec 17, 2025

Uh oh!

norio-nomura commented Dec 17, 2025

Uh oh!

Uh oh!

This comment was marked as outdated.

norio-nomura commented Jan 15, 2026

Uh oh!

This comment was marked as outdated.

norio-nomura commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AkihiroSuda commented Mar 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

norio-nomura commented Nov 22, 2025 •

edited

Loading

`networks.yaml`

`lima.yaml`

`limactl vmnet`

Benchmark: `iperf3` between VMs on the same network

norio-nomura commented Jan 16, 2026 •

edited

Loading