USB passthrough rabbit hole

[retpolanne]

Description

QEMU usb passthrough rabbit hole

Okay, so I went down this rabbit hole to see how doable it is to implement USB passthrough.

Let's go step by step:

1. Lima talks to QEMU through QMP (QEMU Machine Protocol). It uses a lib made by DigitalOcean, go-qemu, which has a function DeviceAdd [1] that seems to be autogenerated. From the qemu qmp ref docs [2] we can see that it accepts device, bus and id. However, according to [3], it should accept other kinds of args as well. In our case, device would be usb-host, and then we would require vendorid and productid.

2. Trying on the qemu monitor, I see this error:

(qemu) device_add usb-host,vendorid=0x1235,productid=0x8211
libusb_detach_kernel_driver: -3 [ACCESS]
libusb_detach_kernel_driver: -3 [ACCESS]
libusb_detach_kernel_driver: -3 [ACCESS]
libusb_detach_kernel_driver: -3 [ACCESS]
libusb_kernel_driver_active: -5 [NOT_FOUND]
libusb_kernel_driver_active: -5 [NOT_FOUND]
libusb_kernel_driver_active: -5 [NOT_FOUND]
libusb_kernel_driver_active: -5 [NOT_FOUND]
libusb_kernel_driver_active: -5 [NOT_FOUND]
libusb_kernel_driver_active: -5 [NOT_FOUND]
libusb_kernel_driver_active: -5 [NOT_FOUND]

4. The whole bus could be connected, according to [4]

5. From my tests, if the device_add syntax is wrong or device is inexistent, it freezes the whole instance, seems to be a qemu bug.

More info at [5] and [6]

References

[1] https://pkg.go.dev/github.com/digitalocean/[email protected]/qmp/raw#Monitor.DeviceAdd

[2] https://www.qemu.org/docs/master/interop/qemu-qmp-ref.html#qapidoc-2506

[3] https://qemu-project.gitlab.io/qemu/system/devices/usb.html

[4] #1317

[5] https://gitlab.com/qemu-project/qemu/-/issues/1951

[6] libusb/libusb#908

[retpolanne]

I wonder if libusb_get_device_descriptor is not returning, so qemu gets stuck.

[ 0.074566] [002c9212] libusb: debug [libusb_get_device_descriptor]  
0   libusb-1.0.0.dylib                  0x0000000102884cf8 libusb_get_device_descriptor + 88
1   qemu-system-arm                     0x0000000100ab3268 usb_host_auto_check + 200
2   qemu-system-arm                     0x0000000100ab074c usb_host_realize + 536
3   qemu-system-arm                     0x00000001009c4b48 usb_qdev_realize + 244
4   qemu-system-arm                     0x0000000100cb40d8 device_set_realized + 384
5   qemu-system-arm                     0x0000000100cbc28c property_set_bool + 100
6   qemu-system-arm                     0x0000000100cba1f8 object_property_set + 136
7   qemu-system-arm                     0x0000000100cbe5ec object_property_set_qobject + 60
8   qemu-system-arm                     0x0000000100cba6e0 object_property_set_bool + 60
9   qemu-system-arm                     0x0000000100a41b6c qdev_device_add_from_qdict + 2304
10  qemu-system-arm                     0x0000000100a42400 qmp_device_add + 104
11  qemu-system-arm                     0x0000000100e3aea4 do_qmp_dispatch_bh + 56
12  qemu-system-arm                     0x0000000100e5a1fc aio_bh_poll + 192
13  qemu-system-arm                     0x0000000100e45230 aio_dispatch + 40
14  qemu-system-arm                     0x0000000100e5afb8 aio_ctx_dispatch + 16
15  libglib-2.0.0.dylib                 0x000000010351db30 g_main_context_dispatch_unlocked + 236
16  libglib-2.0.0.dylib                 0x000000010351da34 g_main_context_dispatch + 44
17  qemu-system-arm                     0x0000000100e5b6bc main_loop_wait + 412
18  qemu-system-arm                     0x0000000100a477a8 qemu_main_loop + 104
19  qemu-system-arm                     0x0000000100cb0484 qemu_default_main + 16
20  dyld                                0x0000000186e750e0 start + 2360

This happened after I called:

╰→ telnet localhost 4444
Trying ::1...
Connected to localhost.
Escape character is '^]'.
{"QMP": {"version": {"qemu": {"micro": 50, "minor": 2, "major": 8}, "package": "v8.2.0-1767-g91e3bf2e92-dirty"}, "capabilities": ["oob"]}}
{ "execute": "qmp_capabilities" }
{"return": {}}
{ "execute": "device_add", "arguments": {"driver": "usb-host", "vendorid": "0x1235", "productid": "0x8211"} }

In other words

1. if we change go-qemu's raw monitor DeviceAdd call to receive any kind of args, then they'll be passed to QEMU and the device_add call will work!
2. since qemu + libusb are broken, then it fails, but that is due to qemu freezing.

In fact, I see errors from usb after I kill qemu!

{ "execute": "device_add", "arguments": {"driver": "usb-host", "vendorid": "0x13fe", "productid": "0x3e00ffff"} }

{"timestamp": {"seconds": 1708801281, "microseconds": 231261}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-signal"}}
{"error": {"class": "GenericError", "desc": "productid out of range"}}
Connection closed by foreign host.

[retpolanne]

In the end, it seems to come down to this [1],

This is the correct payload that QEMU supports

{ "execute": "device_add", "arguments": {"driver": "usb-host", "vendorid": "0x13fe", "productid": "0x3e00", "id": "usb-disk", "hostbus": "1", "hostaddr": "26"} }

in order to actually connect the USB device, we need to unload the kexts that talk to the device. If that doesn't happen, libusb_kernel_driver_active will fail.

[1] https://github.com/libusb/libusb/wiki/FAQ#how-can-i-run-libusb-applications-under-mac-os-x-if-there-is-already-a-kernel-extension-installed-for-the-device-and-claim-exclusive-access

[paschun]

fyi, podman 4.x has support for this on macos, which I think comes down to a couple qemu flags. From podman-machine-default.json:

  "-device",
  "qemu-xhci",
  "-device",
  "usb-host,vendorid=1234,productid=54321",

And looks like lima already uses the flag -device qemu-xhci,id=usb-bus

[AkihiroSuda]

fyi, podman 4.x has support for this on macos

Does it need the root on the host?

[savvn001]

Following. Would love to see this implemented 😃

[mschoenlaub]

I got an implementation working, based on https://github.com/SPICEorg/usbredir.

It does run the usbredirect daemon as root, one per device and this kinda works.

I just wanted to know about the expections for the config file structure.
I'm going to assume that:

1. You want to redirect/pass through one port/device to at maximum one running instance.
2. You want only max one instance running with some flag like --auto-add-usb.

How would you expect to configure this in the YAML structure?

What are the expections if you try to start an instance where the usb port/device is already in use by another instance?

Note: It does not use SPICE, but rather qemu's usb-redir function. A similar approach could be used with WSL.

Personally I feel like the implementation would be "driver specific", with VZ not offering any support currently.

[mschoenlaub]

Puh... working on a PR, but I'd need some guidance around where want to go with the "daemons". It feels like there's a ton of code tied into the networks package, that could be very well used for generic daemons.
But then, there's lots of slices only containg SocketVMNet related values. Mostly because the VDE stuff was removed.

i'm mainly writing this, because if there's another PR working in this area, rather large conflicts in code with low test coverage will arise.

[ballo]

If it's the same passthrough method as UTM, it's broken and I don't think anyone is working on fixing it

[mschoenlaub]

@ballo

if you refer to https://docs.getutm.app/guest-support/sharing/usb/

Due to the way macOS/iOS handles USB capturing (without custom kernel drivers), it is not possible to get a proper hardware reset on the connected device. The device will always be configured on the host before a software reset is sent to the device and it is seen by the virtual machine. This means that many devices will not work properly when captured by the virtual machine.

Yes, it is "broken", at least your mileage may vary. And yes, after a lot of research it looks like all solutions out there use usbredir to some degree - and therefor suffer from the same limitations.

E.g. even for QEMU people complain about some devices not working all the time 🤷

As I use MacOS only for work (as do most people I guess) and there's no real business case for this feature (for most companies) I suppose this won't be fixed anytime soon 🤷

FYI: I was only interested in this, because MacOS doesn't allow for USB sniffing without deactivating System Integrity Protection and I wanted a workaround :D
Turned out it is just way easier to not use MacOS :D

[ballo]

Yes, it is "broken", at least your mileage may vary. And yes, after a lot of research it looks like all solutions out there use usbredir to some degree - and therefor suffer from the same limitations.

Parallels works, qemu doesn't

[mschoenlaub]

Parallels works, qemu doesn't.

Possible. In case you refer to "Parallels Desktop", that's a closed-source, commercial solution. 🤷
I found some stuff in their docs that would indicate that use a k(ernel) ext(ension) at least for some parts of their functionality. That's where you are sometimes out of luck with open source.

Most people don't have the time (nor the nerve) to cater for the needs of people who complain about how bad open source is, want stuff for free and then compare solutions with industry standard paid products (all while overlooking that nothing in the Apple world is actually meant to incentivize open source community contributions :D)

[ballo]

A very weird response my dude. I'm merely pointing out that it is possible. This isn't a criticism of open source, so please park your zealotry. I'm also pointing out that a solution exists for those who need it.

As for kernel extensions, they can also be open source and the API is well documented. This isn't Linux (thank God). A long ass time ago (2001?), I even wrote a driver for OSX for an obscure USB device.

Also, it's unlikely a kernel extension is used let alone necessary. DriverKit runs in userspace. Even the first OS X release had USB drivers run in userspace. Again, this isn't Linux (thank God).