Migrate virtio-net from semu - #748
Charlie-Tsai1123 wants to merge 14 commits into
Conversation
There was a problem hiding this comment.
Benchmarks
Details
| Benchmark suite | Current: e8dd4eb | Previous: c3d9c03 | Ratio |
|---|---|---|---|
Dhrystone |
1551.333 DMIPS |
1575 DMIPS |
1.02 |
CoreMark |
1111.051 iterations/sec |
1109.809 iterations/sec |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
7658d0a to
b775142
Compare
|
I’m not the original author of the virtio-net device. Please check the commit history via |
|
Thanks for the clarification, and sorry for the incorrect attribution. |
b775142 to
56a5c0b
Compare
Remove "This implementation is based on semu's virtio-net device, originally introduced by Jserv" as you already append "Co-authored-by: Jim Huang". |
6dd32bc to
198ea5c
Compare
jserv
left a comment
There was a problem hiding this comment.
Rework semu's network infrastructure, as described in networking.md:
- Linux: TAP (kernel-level) and user-mode (SLIRP) networking
- macOS: vmnet.framework (kernel-level NAT; bridge mode planned) and user-mode (SLIRP) networking
For this pull request, both TAP and SLIRP should be landed.
122d1dc to
50532e7
Compare
c48ad2a to
103be25
Compare
|
This update mainly covers two parts:
I moved virtio-net refresh and interrupt propagation into the per-step execution path. After queue refresh, the virtio-net interrupt state is pushed to the PLIC. This avoids delaying completed RX/TX work until a later interrupt update point. I also updated virtio_net_try_rx() and virtio_net_try_tx() so they only raise used-ring interrupts when the used ring index actually advances. This avoids repeated interrupts when no virtqueue progress was made, and fixes the observed TAP ping latency spike. Before this change, TAP ping could show delayed replies such as: After the change, replies are delivered promptly:
I added a minislirp-based user-mode backend for virtio-net, adapted from semu. The backend connects the virtio-net RX/TX paths with libslirp through non-blocking socketpairs, so guest networking can work without TAP, root rivileges, or host network setup. The current backend support is:
Emscripten is handled separately because rv32emu supports emcc builds, unlike semu's original networking setup. For emcc, the networking backends are disabled and unsupported vnet backends are rejected during argument parsing. CI coverage is arranged as follows:
Future PR may work:
|
|
Hi @Charlie-Tsai1123 , could you document how to test the user mode vnet backend as I only see the |
There was a problem hiding this comment.
All reported issues were addressed across 14 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
d1cbd60 to
1a52828
Compare
7036964 to
848c8da
Compare
This commit migrates virtio-net support from semu with the following modifications: 1. Implement virtio-net device model The virtio-net implementation follows the VirtIO-MMIO flow used by virtio-blk, including feature negotiation, queue setup, QueueNotify handling, used ring update, interrupt status, and device status reset. The device currently supports a TAP-backend network interface and handles basic RX/TX virtqueue processing for guest network packets. 2. Add TAP backend helper Introduce netdev.c and netdev.h to provide host-side TAP device access. Future work may support other host-side backend. 3. Handle virtio-net header processing For guest TX, the device skips the virtio-net header before writing the Ethernet frame to the TAP backend. For guest RX, the device prepends a virtio-net header before copying the received Ethernet frame into the guest-provided RX buffer. 4. Implement MMIO_VIRTIONET Add MMIO routing for virtio-net and connect the device interrupt status to the PLIC, following the existing virtio-blk and virtio-rng interrupt update model. 5. Introduce new argument '-x vnet:<tap>' When virtio-net is enabled, rv32emu dynamically creates a virtio-mmio node in the generated device tree and assigns an MMIO base address and IRQ for the device. 6. Support coexistence with virtio-blk and virtio-rng Update the dynamic virtio-mmio device tree allocation path so virtio-net can coexist with existing virtio-blk and virtio-rng devices without reusing MMIO base addresses or IRQs. 7. Use virtio-net state Unlike semu's device integration model, rv32emu stores the virtio-net state in vm_attr_t so MMIO routing, interrupt routing, and device cleanup can access the same device instance. The emulator should be run with sudo when using the virtio-net TAP backend. Co-authored-by: Jim Huang <jserv@biilabs.io>
The host-arm64 CI job may fail before running any build or test when apt cannot update package indexes due to transient network issues on the Ubuntu ports mirror. Add retry logic and force IPv4 when updating the apt cache in install-llvm.sh to make LLVM repository setup more robust on GitHub-hosted ARM runners. The host-arm64 dependency installation also has a fallback path for partially failed apt installs. However, the fallback only attempted to install make, curl, and wget. As a result, the later Linux boot test could fail with "expect: command not found" even though expect is a required dependency for .ci/boot-linux.sh.
virtio-net queue refresh can complete RX/TX descriptors and set the device interrupt status, but PLIC line was not updated immediately after refresh. This could leave the guest waiting until later interrupt update point before observing completed network work, causing TAP ping latency spikes of around one second. Update the virtio-net interrupt after queue refresh. This delivers completed network packets to the guest promptly. Also make virtio_net_try_rx() and virtio_net_try_tx() raise used-ring interrupts only when used ring index actually advances. TAP is often reported writable by poll(), so trying TX without completing any descriptor must not set VIRTIO_INT_USED_RING. Otherwise guest may see repeated interrupts without corresponding used-ring updates. Before this change, TAP ping could show delayed replies such as: PING 192.168.100.1 (192.168.100.1): 56 data bytes 64 bytes from 192.168.100.1: seq=0 ttl=64 time=3.831 ms 64 bytes from 192.168.100.1: seq=1 ttl=64 time=1001.088 ms 64 bytes from 192.168.100.1: seq=2 ttl=64 time=1.389 ms After the change, replies are delivered promptly: PING 192.168.100.1 (192.168.100.1): 56 data bytes 64 bytes from 192.168.100.1: seq=0 ttl=64 time=1.181 ms 64 bytes from 192.168.100.1: seq=1 ttl=64 time=0.546 ms 64 bytes from 192.168.100.1: seq=2 ttl=64 time=0.565 ms
Migrate semu's user-mode virtio-net networking support to rv32emu. Add a minislirp backend that allows virtio-net to operate without TAP, root privileges, or host network configuration. Connect the guest RX/TX paths to libslirp through non-blocking socketpairs. Adapt semu's timer and event integration to rv32emu by using a CLOCK_MONOTONIC-based timer wrapper and driving SLIRP progress from the existing virtio-net refresh path. Support the standard SLIRP guest network configuration with 10.0.2.15/24 as the guest address and 10.0.2.2 as the gateway. Co-authored-by: Jim Huang <jserv@biilabs.io>
Add Kconfig options for the virtio-net device, Linux TAP backend, and user-mode SLIRP backend. Exclude unused network objects at build time, build minislirp only when the user backend is enabled, and guard the related CLI, runtime, MMIO, DTB, and interrupt integration. Also list only compiled backends in the CLI help and consolidate backend initialization through a shared helper (netdev_setup in src/devices/netdev.c).
Ensure virtio-net.o waits for the SoftFloat dependency before compilation. This fixes a clean parallel build race where virtio-net could be compiled before the SoftFloat submodule was initialized.
Reuse the existing virtio-net device during a cold reboot and reset its guest-visible state instead of allocating a new device. Preserve the host networking backend across reboots.
Compute the number of pending descriptors in uint16_t before comparing it with the queue size. Direct subtraction promotes the 16-bit queue indices to int, so a wrapped index can become a negative value and bypass validation. For example, last_avail = 100 and new_avail = 50 evaluates to -50 instead of the intended modulo-65536 result 65486. This can make the processing loop walk tens of thousands of descriptors that were never posted by the guest. Apply the same fix to virtio-blk, virtio-rng, and virtio-net.
Distinguish retryable transmit failures from permanent backend errors. EAGAIN, EWOULDBLOCK, and EINTR leave the current descriptor pending so the packet can be retried later. Previously, every writev failure stopped TX processing without advancing last_avail. A permanent error such as EMSGSIZE therefore caused the same descriptor to be rebuilt and submitted again on every queue refresh, repeatedly logging the same error and preventing later TX descriptors from making progress. Drop packets that fail with a permanent error and complete their descriptors so the guest can continue processing the transmit queue.
Advertise the MAC address, link status, and MTU features provided by
the virtio-net device, and preserve byte offsets when accessing the
device-specific configuration space.
Previously, only VIRTIO_F_VERSION_1 was advertised, so Linux ignored
the configured MAC address and generated a random address instead.
After advertising VIRTIO_NET_F_MAC, VIRTIO_NET_F_STATUS, and
VIRTIO_NET_F_MTU, the existing config access exposed another issue:
virtio_net_read shifted every MMIO address by two bits before handling
the device-specific configuration.
This discarded the low address bits used for byte-sized config fields.
For example, accesses to offsets 0, 1, 2, and 3 of the MAC address all
mapped to the same 32-bit index. As a result, the configured address
52:54:00:12:34:56
was observed by the guest as
52:52:52:52:34:34
Handle the device-specific configuration using its original byte
address and only convert addresses to 32-bit register indices for the
VirtIO MMIO transport registers. This also allows the guest to read the
configured link status and MTU correctly.
Poll the virtio-net backend only after several thousand guest cycles instead of after every rv_step invocation. The default execution loop advances about 100 guest instructions per rv_step. Previously, virtio_net_refresh_queue was therefore called after roughly every 100 instructions. The TAP backend issued a poll system call on every refresh, while the user-mode backend performed multiple zero-timeout polls and SLIRP pollfd walks. Use the guest cycle counter to refresh the backend every 5000 cycles. This substantially reduces host polling overhead while keeping the network responsive.
Keep every timer created by minislirp instead of storing only the most recent timer in net_user_options_t. Link all timers into a list, unlink them when timer_free is invoked, and scan the list for expired timers. Restart the scan after each timer callback because minislirp may modify the timer list while servicing an expiration. Clean up any timers that remain after the SLIRP instance is destroyed.
Integrate virtio-net validation into the existing Linux boot test flow. Exercise both TAP and user-mode SLIRP backends on Linux, and run the user-mode SLIRP backend on macOS. Reuse the common boot setup and cleanup path instead of maintaining standalone network boot steps. Keep network-specific guest setup and connectivity checks in netdev.sh while invoking them from the modular Linux boot test suite.
1957430 to
c3d9c03
Compare
Migrate the vmnet.framework network backend from semu and integrate it with the existing virtio-net implementation on macOS. Implement shared, host-only, and bridged vmnet initialization paths, while exposing shared mode through the existing vnet backend selection. Factor the common interface setup, MAC address handling, and packet callback registration into shared helpers to avoid duplicating the three mode-specific initialization paths. Bridge vmnet's asynchronous receive callbacks into rv32emu's polling model through a non-blocking pipe while preserving Ethernet packet boundaries. Forward guest TX descriptor chains directly to vmnet_write() and translate vmnet failures into errno values so the existing virtio-net retry and drop policy remains applicable. Use the MAC address assigned by vmnet.framework as the guest-visible VirtIO network MAC address. Add build-time configuration for the vmnet backend on macOS Clang, including Blocks support and vmnet.framework linking. Keep the backend disabled for GCC builds so unsupported Blocks-based vmnet code is not compiled. Extend the existing network boot tests to exercise vmnet shared mode on macOS while retaining the existing TAP and user-mode SLIRP tests. Co-authored-by: Jim Huang <jserv@biilabs.io>
c3d9c03 to
e8dd4eb
Compare
| (. "${SCRIPT_DIR}/virtio-blk.sh") | ||
| RET=$((${RET} + $?)) | ||
|
|
||
| # Virtio-net user-mode backend test |
There was a problem hiding this comment.
The vmnet block below checks .config before running, but the user and tap blocks do not. With CONFIG_VIRTIO_NET_USER=n (or CONFIG_VIRTIO_NET_TAP=n), virtio_net_backend_supported() rejects -x vnet:user during argument parsing, so rv32emu exits immediately and the whole boot suite fails instead of skipping the test. Guard all three the same way, for example with grep -q '^CONFIG_VIRTIO_NET_USER=y$' .config.
| register_cleanup cleanup_emulator | ||
|
|
||
| TIMEOUT=${NETDEV_BOOT_TIMEOUT:-${TIMEOUT}} | ||
| MESSAGES+=("${COLOR_R}Fail to ping gateway") |
There was a problem hiding this comment.
The appended entry covers exit code 4, but the expect block also exits 2 when readlink /sys/bus/virtio/devices/virtio0/driver does not report virtio_net, and that indexes the inherited Fail to login message. A guest that boots and logs in fine but never binds the driver gets reported as a login failure. Give that case its own message, or move the exits past the end of the inherited array.
| */ | ||
| uint32_t pkt_len = (uint32_t) len; | ||
|
|
||
| if (write(state->pipe_fds[1], &pkt_len, sizeof(pkt_len)) != |
There was a problem hiding this comment.
The length prefix and the payload are two separate writes, and only pipe_fds[0] is set non-blocking, so the write side blocks: once the guest stops posting RX buffers the pipe fills and this callback stalls the serial vmnet queue while holding state->lock. If the length write lands and the payload write is short or fails, the code only logs and returns, leaving the stream permanently desynchronized so net_vmnet_read consumes payload bytes as the next length. Use a non-blocking SOCK_DGRAM socketpair so each frame is one atomic message and a full buffer drops the packet instead of blocking.
| mode_name, state->mac[0], state->mac[1], state->mac[2], | ||
| state->mac[3], state->mac[4], state->mac[5]); | ||
|
|
||
| vmnet_register_packet_callback(state, iface); |
There was a problem hiding this comment.
iface is read here inside the completion block, but it is only assigned when vmnet_start_interface returns on the calling thread. The block runs on state->queue with no ordering against that assignment, so the packet callback can be registered on a NULL interface and no packet ever arrives. Register it after dispatch_semaphore_wait returns and state->iface has been set.
| dispatch_semaphore_signal((dispatch_semaphore_t) state->sem); | ||
| }); | ||
|
|
||
| dispatch_semaphore_wait((dispatch_semaphore_t) state->sem, |
There was a problem hiding this comment.
vmnet_start_interface returns NULL when it fails outright, and in that case the completion handler that signals state->sem never runs, so this wait blocks forever and rv32emu hangs at startup with no diagnostic. Check the return value before waiting, and give the wait a finite deadline so a framework that accepts the request but never calls back still unwinds.
| state->running = false; | ||
|
|
||
| if (state->iface) { | ||
| vmnet_stop_interface( |
There was a problem hiding this comment.
vmnet_stop_interface only schedules its completion handler on state->queue, yet the code below releases that queue, closes pipe_fds, and destroys state->lock right away, and netdev_delete then frees state. A packet callback already queued or still running will touch a destroyed mutex, a closed fd, and freed memory on every shutdown and cold reboot. Clear the event callback with vmnet_interface_set_event_callback(iface, VMNET_INTERFACE_PACKETS_AVAILABLE, NULL, NULL), wait for the stop completion, then drain the queue before releasing anything.
| { | ||
| net_user_options_t *usr = (net_user_options_t *) opaque; | ||
|
|
||
| if (!usr || usr->guest_to_host_channel[SLIRP_WRITE_SIDE] < 0) |
There was a problem hiding this comment.
The guard tests guest_to_host_channel[SLIRP_WRITE_SIDE], but the write below uses host_to_guest_channel[SLIRP_WRITE_SIDE]. The descriptor actually written is never validated; the check only holds today because net_slirp_init creates and net_slirp_cleanup closes both pairs together.
| if (!usr || usr->guest_to_host_channel[SLIRP_WRITE_SIDE] < 0) | |
| if (!usr || usr->host_to_guest_channel[SLIRP_WRITE_SIDE] < 0) |
| if (!vnet_check_word_range(vnet, desc_addr, 4)) | ||
| return false; | ||
|
|
||
| const struct virtq_desc *desc = |
There was a problem hiding this comment.
vnet_preprocess only rejects addresses with the low two bits set, so queue_desc is guaranteed 4-byte alignment while struct virtq_desc needs 8 for its uint64_t addr. A guest that programs QueueDescLow to an address that is 4-byte but not 8-byte aligned makes every field access through this pointer undefined, which faults on strict-alignment hosts. virtio_blk_handle_request copies the entry with memcpy instead of casting for exactly this reason, and carries a comment saying so; do the same here.
jserv
left a comment
There was a problem hiding this comment.
Rebase latest master branch and resolve conflicts.


Summary
This PR migrates virtio-net support from semu and follows the existing virtio-blk and virtio-rng integration in rv32emu.
The implementation adds a TAP-backed virtio-net device model for system emulation mode, including MMIO register handling, queue setup, feature negotiation, RX/TX virtqueue handling, virtio-net header processing, interrupt delivery through the PLIC, dynamic DTB node creation, and a runtime option for enabling the device.
Implementation notes
virtio,mmioDTB node for virtio-net.sudoor equivalent permissions when using the TAP backend.Test
If don't have build/linux-image/Image
Build:
After run rv32emu with
-x vnet:taprv32emu would build TAP, so host linux doesn't need to build TAP again.Host TAP setup:
Guest device verification:
readlink /sys/bus/virtio/devices/virtio0/driver ip link set eth0 up ip addr add 192.168.100.2/24 dev eth0 ip addr show eth0 ping -c 3 192.168.100.1Expected Result:
Summary by cubic
Migrates virtio-net support from semu, adding a VirtIO-MMIO network device with build-time selectable host backends and runtime selection via
-x vnet:tap,-x vnet:user, or-x vnet:vmnet. The device gets a DTB node and is covered by Linux and macOS boot tests.New Features
tapis Linux-only and needs host privileges;userneeds no host setup on Linux and macOS;vmnetis macOS/Clang-only in shared mode; Emscripten compiles networking out; only onevnetdevice is allowed and CLI validation lists only compiled backends.CONFIG_VIRTIO_NET,CONFIG_VIRTIO_NET_TAP,CONFIG_VIRTIO_NET_USER, andCONFIG_VIRTIO_NET_VMNET; unused net objects are excluded;src/minislirpbuilds only foruser; MMIO/DTB/interrupt wiring compiles only when enabled; macOS links-lresolv..ci/netdev.shfrom.ci/boot-linux.sh, add Linuxuser/tapand macOSuser/conditionalvmnetboot jobs, make ARM64 apt-get retry with IPv4 and installexpect, and document usage indocs/networking.md.Bug Fixes
virtio-net.owait for the SoftFloat submodule to avoid a clean parallel build race.Written for commit e8dd4eb. Summary will update on new commits.