Skip to content

Support for lima usernet network #1383

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 4, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .cirrus.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ task:
EXAMPLE: fedora.yaml
EXAMPLE: archlinux.yaml
EXAMPLE: opensuse.yaml
EXAMPLE: experimental/net-user-v2.yaml
info_script:
- uname -a
- df -T
Expand Down
1 change: 1 addition & 0 deletions cmd/limactl/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,7 @@ func newApp() *cobra.Command {
newEditCommand(),
newFactoryResetCommand(),
newDiskCommand(),
newUsernetCommand(),
)
return rootCmd
}
Expand Down
71 changes: 71 additions & 0 deletions cmd/limactl/usernet.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
package main

import (
"errors"
"fmt"
"os"
"strconv"

"github.com/lima-vm/lima/pkg/networks/usernet"
"github.com/spf13/cobra"
)

func newUsernetCommand() *cobra.Command {
var hostagentCommand = &cobra.Command{
Use: "usernet",
Short: "run usernet",
Args: cobra.ExactArgs(0),
RunE: usernetAction,
Hidden: true,
}
hostagentCommand.Flags().StringP("pidfile", "p", "", "write pid to file")
hostagentCommand.Flags().StringP("endpoint", "e", "", "exposes usernet api(s) on this endpoint")
hostagentCommand.Flags().String("listen-qemu", "", "listen for qemu connections")
hostagentCommand.Flags().String("listen", "", "listen on a Unix socket and receive Bess-compatible FDs as SCM_RIGHTS messages")
hostagentCommand.Flags().Int("mtu", 1500, "mtu")
return hostagentCommand
}

func usernetAction(cmd *cobra.Command, args []string) error {
pidfile, err := cmd.Flags().GetString("pidfile")
if err != nil {
return err
}
if pidfile != "" {
if _, err := os.Stat(pidfile); !errors.Is(err, os.ErrNotExist) {
return fmt.Errorf("pidfile %q already exists", pidfile)
}
if err := os.WriteFile(pidfile, []byte(strconv.Itoa(os.Getpid())+"\n"), 0644); err != nil {
return err
}
defer os.RemoveAll(pidfile)
}
endpoint, err := cmd.Flags().GetString("endpoint")
if err != nil {
return err
}
qemuSocket, err := cmd.Flags().GetString("listen-qemu")
if err != nil {
return err
}
fdSocket, err := cmd.Flags().GetString("listen")
if err != nil {
return err
}

mtu, err := cmd.Flags().GetInt("mtu")
if err != nil {
return err
}

os.RemoveAll(endpoint)
os.RemoveAll(qemuSocket)
os.RemoveAll(fdSocket)

return usernet.StartGVisorNetstack(cmd.Context(), &usernet.GVisorNetstackOpts{
MTU: mtu,
Endpoint: endpoint,
QemuSocket: qemuSocket,
FdSocket: fdSocket,
})
}
1 change: 1 addition & 0 deletions docs/experimental.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ The following features are experimental and subject to change:
- `vmType: vz` and relevant configurations (`mountType: virtiofs`, `rosetta`, `[]networks.vzNAT`)
- `arch: riscv64`
- `video.display: vnc` and relevant configuration (`video.vnc.display`)
- `mode: user-v2` in `networks.yml` and relevant configuration in `lima.yaml`

The following flags are experimental and subject to change:

Expand Down
30 changes: 30 additions & 0 deletions docs/network.md
Original file line number Diff line number Diff line change
Expand Up @@ -193,3 +193,33 @@ networks:
The range of the IP address is not specifiable.

The "vzNAT" network does not need the `socket_vmnet` binary and the `sudoers` file.

## Lima user-v2 network

user-v2 network provides a user-mode networking similar to the [default user-mode network](#user-mode-network--1921685024-) and also provides support for `vm -> vm` communication.

> **Warning**
> This network mode is experimental

To enable this network mode, define a network with `mode: user-v2` in networks.yaml

```yaml
...
networks:
example-user-v2:
mode: user-v2
...
```

Instances can then reference these networks from their `lima.yaml` file:

```yaml
networks:
- lima: example-user-v2
```

_Note_

- Enabling this network will disable the [default user-mode network](#user-mode-network--1921685024-)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are drawbacks?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only drawback i see is,
Running high number of vm's under this network might add bottleneck from gvisor-tap-vsock.

- Subnet used for this network is 192.168.5.0/24 with 192.168.5.2 used for host connection and 192.168.5.3 used for DNS resolution
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this configurable or not?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not configurable as of now. I will provide this support in a follow-up PR.
This needs some changes related to DNS resolving as well


13 changes: 13 additions & 0 deletions examples/experimental/net-user-v2.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Example to run lima instance with experimental user-v2 network enabled
images:
- location: "https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-amd64.img"
arch: "x86_64"
- location: "https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-arm64.img"
arch: "aarch64"

mounts:
- location: "~"
- location: "/tmp/lima"
writable: true
networks:
- lima: user-v2
4 changes: 4 additions & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ require (
github.com/a8m/envsubst v1.4.2 // indirect
github.com/alecthomas/participle/v2 v2.0.0 // indirect
github.com/apparentlymart/go-cidr v1.1.0 // indirect
github.com/balajiv113/fd v0.0.0-20230330094840-143eec500f3e // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/digitalocean/go-libvirt v0.0.0-20201209184759-e2a69bcd5bd1 // indirect
github.com/dimchansky/utfbom v1.1.1 // indirect
Expand Down Expand Up @@ -80,10 +81,13 @@ require (
github.com/json-iterator/go v1.1.12 // indirect
github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 // indirect
github.com/kr/fs v0.1.0 // indirect
github.com/linuxkit/virtsock v0.0.0-20220523201153-1a23e78aa7a2 // indirect
github.com/magiconair/properties v1.8.7 // indirect
github.com/mailru/easyjson v0.7.6 // indirect
github.com/mattn/go-colorable v0.1.13 // indirect
github.com/mattn/go-runewidth v0.0.12 // indirect
github.com/mdlayher/socket v0.4.0 // indirect
github.com/mdlayher/vsock v1.2.0 // indirect
github.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
Expand Down
12 changes: 12 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,12 @@ github.com/apparentlymart/go-cidr v1.1.0 h1:2mAhrMoF+nhXqxTzSZMUzDHkLjmIHC+Zzn4t
github.com/apparentlymart/go-cidr v1.1.0/go.mod h1:EBcsNrHc3zQeuaeCeCtQruQm+n9/YjEn/vI25Lg7Gwc=
github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8=
github.com/armon/go-proxyproto v0.0.0-20210323213023-7e956b284f0a/go.mod h1:QmP9hvJ91BbJmGVGSbutW19IC0Q9phDCLGaomwTJbgU=
github.com/balajiv113/fd v0.0.0-20150925145434-c6d800382fff h1:igcsiQjkP5E7AfbHNG73RNYQq+MZt0MezdYyW/zqOcg=
github.com/balajiv113/fd v0.0.0-20150925145434-c6d800382fff/go.mod h1:aXGMJsd3XrnUFTuyf/pTGg5jG6CY8JMZ5juywvShjgQ=
github.com/balajiv113/fd v0.0.0-20230330061141-18b1d0daf6e7 h1:6Y2nNPVsEoUA3fmWCouMVLlezu++LP6y9CSX1J8VtVg=
github.com/balajiv113/fd v0.0.0-20230330061141-18b1d0daf6e7/go.mod h1:aXGMJsd3XrnUFTuyf/pTGg5jG6CY8JMZ5juywvShjgQ=
github.com/balajiv113/fd v0.0.0-20230330094840-143eec500f3e h1:IdMhFPEfTZQU971tIHx3UhY4l+yCeynprnINrDTSrOc=
github.com/balajiv113/fd v0.0.0-20230330094840-143eec500f3e/go.mod h1:aXGMJsd3XrnUFTuyf/pTGg5jG6CY8JMZ5juywvShjgQ=
github.com/cenkalti/backoff v1.1.1-0.20190506075156-2146c9339422/go.mod h1:b6Nc7NRH5C4aCISLry0tLnTjcuTEvoiqcWDdsU0sOGM=
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/cheggaaa/pb/v3 v3.1.2 h1:FIxT3ZjOj9XJl0U4o2XbEhjFfZl7jCVCDOGq1ZAB7wQ=
Expand Down Expand Up @@ -362,6 +368,8 @@ github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/leodido/go-urn v1.2.0 h1:hpXL4XnriNwQ/ABnpepYM/1vCLWNDfUNts8dX3xTG6Y=
github.com/lima-vm/sshocker v0.3.2 h1:o0WqVzcpt6mzVCuqtS3N3O8kwTx6X4SLr4h7YaRISuE=
github.com/lima-vm/sshocker v0.3.2/go.mod h1:9SWN6wob210VM6oJkkzvWQOlHSp/rQLB+0fSEc92zig=
github.com/linuxkit/virtsock v0.0.0-20220523201153-1a23e78aa7a2 h1:DZMFueDbfz6PNc1GwDRA8+6lBx1TB9UnxDQliCqR73Y=
github.com/linuxkit/virtsock v0.0.0-20220523201153-1a23e78aa7a2/go.mod h1:SWzULI85WerrFt3u+nIm5F9l7EvxZTKQvd0InF3nmgM=
github.com/lithammer/dedent v1.1.0 h1:VNzHMVCBNG1j0fh3OrsFRkVUwStdDArbgBWoPAffktY=
github.com/lithammer/dedent v1.1.0/go.mod h1:jrXYCQtgg0nJiN+StA2KgR7w6CiQNv9Fd/Z9BP0jIOc=
github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
Expand Down Expand Up @@ -394,6 +402,10 @@ github.com/mdlayher/netlink v1.1.1/go.mod h1:WTYpFb/WTvlRJAyKhZL5/uy69TDDpHHu2VZ
github.com/mdlayher/raw v0.0.0-20190606142536-fef19f00fc18/go.mod h1:7EpbotpCmVZcu+KCX4g9WaRNuu11uyhiW7+Le1dKawg=
github.com/mdlayher/raw v0.0.0-20191009151244-50f2db8cc065 h1:aFkJ6lx4FPip+S+Uw4aTegFMct9shDvP+79PsSxpm3w=
github.com/mdlayher/raw v0.0.0-20191009151244-50f2db8cc065/go.mod h1:7EpbotpCmVZcu+KCX4g9WaRNuu11uyhiW7+Le1dKawg=
github.com/mdlayher/socket v0.4.0 h1:280wsy40IC9M9q1uPGcLBwXpcTQDtoGwVt+BNoITxIw=
github.com/mdlayher/socket v0.4.0/go.mod h1:xxFqz5GRCUN3UEOm9CZqEJsAbe1C8OwSK46NlmWuVoc=
github.com/mdlayher/vsock v1.2.0 h1:klRY9lndjmg6k/QWbX/ucQ3e2JFRm1M7vfG9hijbQ0A=
github.com/mdlayher/vsock v1.2.0/go.mod h1:w4kdSTQB9p1l/WwGmAs0V62qQ869qRYoongwgN+Y1HE=
github.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b h1:j7+1HpAFS1zy5+Q4qx1fWh90gTKwiN4QCGoY9TWyyO4=
github.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b/go.mod h1:01TrycV0kFyexm33Z7vhZRXopbI8J3TDReVlkTgMUxE=
github.com/miekg/dns v1.1.25/go.mod h1:bPDLeHnStXmXAq1m/Ch/hvfNHr14JKNPMBo3VZKjuso=
Expand Down
27 changes: 27 additions & 0 deletions hack/test-example.sh
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ declare -A CHECKS=(
["port-forwards"]="1"
["vmnet"]=""
["disk"]=""
["user-v2"]=""
)

case "$NAME" in
Expand All @@ -49,6 +50,10 @@ case "$NAME" in
"test-misc")
CHECKS["disk"]=1
;;
"net-user-v2")
CHECKS["port-forwards"]=""
CHECKS["user-v2"]=1
;;
esac

if limactl ls -q | grep -q "$NAME"; then
Expand Down Expand Up @@ -303,6 +308,28 @@ if [[ -n ${CHECKS["restart"]} ]]; then
fi
fi

if [[ -n ${CHECKS["user-v2"]} ]]; then
INFO "Testing user-v2 network"
secondvm="$NAME-1"
limactl start "$FILE" --name "$secondvm" --tty=false
guestNewip="$(limactl shell "$secondvm" ip -4 -j addr show dev eth0 | jq -r '.[0].addr_info[0].local')"
INFO "IP of $secondvm is $guestNewip"
set -x
if ! limactl shell "$NAME" ping -c 1 "$guestNewip"; then
ERROR "Failed to do vm->vm communication via user-v2"
INFO "Stopping \"$secondvm\""
limactl stop "$secondvm"
INFO "Deleting \"$secondvm\""
limactl delete "$secondvm"
exit 1
fi
INFO "Stopping \"$secondvm\""
limactl stop "$secondvm"
INFO "Deleting \"$secondvm\""
limactl delete "$secondvm"
set +x
fi

INFO "Stopping \"$NAME\""
limactl stop "$NAME"
sleep 3
Expand Down
6 changes: 5 additions & 1 deletion pkg/cidata/cidata.go
Original file line number Diff line number Diff line change
Expand Up @@ -211,7 +211,11 @@ func GenerateISO9660(instDir, name string, y *limayaml.LimaYAML, udpDNSLocalPort

slirpMACAddress := limayaml.MACAddress(instDir)
args.Networks = append(args.Networks, Network{MACAddress: slirpMACAddress, Interface: networks.SlirpNICName})
for _, nw := range y.Networks {
firstUsernetIndex := limayaml.FirstUsernetIndex(y)
for i, nw := range y.Networks {
if i == firstUsernetIndex {
continue
}
args.Networks = append(args.Networks, Network{MACAddress: nw.MACAddress, Interface: nw.Interface})
}

Expand Down
14 changes: 14 additions & 0 deletions pkg/limayaml/defaults.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ import (
"strconv"
"text/template"

"github.com/lima-vm/lima/pkg/networks"

"github.com/lima-vm/lima/pkg/guestagent/api"
"github.com/lima-vm/lima/pkg/osutil"
"github.com/lima-vm/lima/pkg/store/dirnames"
Expand Down Expand Up @@ -49,6 +51,18 @@ func defaultContainerdArchives() []File {
}
}

// FirstUsernetIndex gets the index of first usernet network under l.Network[]. Returns -1 if no usernet network found
func FirstUsernetIndex(l *LimaYAML) int {
for i := range l.Networks {
nwName := l.Networks[i].Lima
isUsernet, _ := networks.Usernet(nwName)
if isUsernet {
return i
}
}
return -1
}

func MACAddress(uniqueID string) string {
sha := sha256.Sum256([]byte(osutil.MachineID() + uniqueID))
// "5" is the magic number in the Lima ecosystem.
Expand Down
20 changes: 12 additions & 8 deletions pkg/limayaml/validate.go
Original file line number Diff line number Diff line change
Expand Up @@ -281,7 +281,18 @@ func validateNetwork(y LimaYAML, warn bool) error {
for i, nw := range y.Networks {
field := fmt.Sprintf("networks[%d]", i)
if nw.Lima != "" {
if runtime.GOOS != "darwin" {
config, err := networks.Config()
if err != nil {
return err
}
if config.Check(nw.Lima) != nil {
return fmt.Errorf("field `%s.lima` references network %q which is not defined in networks.yaml", field, nw.Lima)
}
usernet, err := config.Usernet(nw.Lima)
if err != nil {
return err
}
if !usernet && runtime.GOOS != "darwin" {
return fmt.Errorf("field `%s.lima` is only supported on macOS right now", field)
}
if nw.Socket != "" {
Expand All @@ -296,13 +307,6 @@ func validateNetwork(y LimaYAML, warn bool) error {
if nw.SwitchPortDeprecated != 0 {
return fmt.Errorf("field `%s.switchPort` cannot be used with field `%s.lima`", field, field)
}
config, err := networks.Config()
if err != nil {
return err
}
if config.Check(nw.Lima) != nil {
return fmt.Errorf("field `%s.lima` references network %q which is not defined in networks.yaml", field, nw.Lima)
}
} else if nw.Socket != "" {
if nw.VZNAT != nil && *nw.VZNAT {
return fmt.Errorf("field `%s.socket` and field `%s.vzNAT` are mutually exclusive", field, field)
Expand Down
8 changes: 8 additions & 0 deletions pkg/networks/commands.go
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,14 @@ func (config *YAML) Check(name string) error {
return fmt.Errorf("network %q is not defined", name)
}

// Usernet Returns true if the mode of given network is ModeUserV2
func (config *YAML) Usernet(name string) (bool, error) {
if nw, ok := config.Networks[name]; ok {
return nw.Mode == ModeUserV2, nil
}
return false, fmt.Errorf("network %q is not defined", name)
}

// DaemonPath returns the daemon path.
func (config *YAML) DaemonPath(daemon string) (string, error) {
switch daemon {
Expand Down
13 changes: 9 additions & 4 deletions pkg/networks/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@ import (
"os"
"os/exec"
"path/filepath"
"runtime"
"sync"

"github.com/goccy/go-yaml"
Expand Down Expand Up @@ -122,9 +121,6 @@ func loadCache() {

// Config returns the network config from the _config/networks.yaml file.
func Config() (YAML, error) {
if runtime.GOOS != "darwin" {
return YAML{}, errors.New("networks.yaml configuration is only supported on macOS right now")
}
loadCache()
return cache.config, cache.err
}
Expand All @@ -144,6 +140,15 @@ func Sock(name string) (string, error) {
return cache.config.Sock(name), nil
}

// Usernet Returns true if the given network name is usernet network
func Usernet(name string) (bool, error) {
loadCache()
if cache.err != nil {
return false, cache.err
}
return cache.config.Usernet(name)
}

// VDESock returns a vde socket.
//
// Deprecated. Use Sock.
Expand Down
Loading