[Feature Request] Support changing the vsock CID on snapshot restore

[simonis]

Feature Request

I'm aware of the vsock device limitations described in snapshot-support.md.

However there's no mentioning of how to support multiple clones of a snapshot with a configured vsock on the same host (they'll all share the same CID). For network interfaces, Network Connectivity for Clones describes pretty nicely how multiple clones can be run on the same host by using network namespaces. But to my knowledge, there's no network namespace support available for vsock sockets (I only found "vsock: support network namespace" which doesn't seem to have been finished yet).

I tried to configure a different vsock CID/path before restoring a snapshot, but in that case restoring fails with:

2022-12-30T20:46:38.507150111 [anonymous-instance:fc_api:ERROR:src/api_server/src/parsed_request.rs:190] Received Error. Status code: 400 Bad Request. Message: Loading a microVM snapshot not allowed after configuring boot-specific resources.

Describe the desired solution

The vsock man page mentions the following about "live migrations":

   Sockets are affected by live migration of virtual machines.
   Connected SOCK_STREAM sockets become disconnected when the
   virtual machine migrates to a new host.  Applications must
   reconnect when this happens.

   The local CID may change across live migration if the old CID is
   not available on the new host.  Bound sockets are automatically
   updated to the new CID.

I wonder if it wouldn't be possible to implement the vsock support in Firecracker such that the CID/socket path can be changed for a snapshot at restore time?

In addition to allowing multiple clones with configured vsockets, such a change would also make it possible to use the vsock device for obtaining a simple SysGenID (see #2546).

Please let me know if there's anything obvious that I'm missing?

Checks

• Have you searched the Firecracker Issues database for similar requests?
• Have you read all the existing relevant Firecracker documentation?
• Have you read and understood Firecracker's core tenets?

[mattschlebusch]

Hi @simonis,

Thanks for logging the request. This is something we will need to look into and seems reasonable at first glance.

As a side thought, you made mention of SysGenID which seems like a particularly specific application between guest<->host communication. I'm not sure what use-case you are working with more specifically, but using MMDS may be worth considering as well.

[Swchexit]

Disclaimer: I might be missing some details, but here’s my understanding.

We likely don’t need to override the CID at restore time—just allowing a new uds_path should be sufficient for supporting snapshot clones.

In Firecracker, vsock is identified by both a guest CID and a host-side UDS path. While the CID is used to construct guest-side packet headers, it’s the Unix domain socket that actually routes traffic on the host. Since each VM connects to its own UDS socket, multiple VMs can safely share the same CID as long as they use different socket paths. So, enabling uds_path override during snapshot restore could be a clean solution to the clone issue, without the need to change CID.

I have attempted a draft PR that allows overriding uds_path on snapshot restore. I'll continue to work on tests and docs if you think this is the right direction.

[bchalios]

Hey @Swchexit, you are right! Since we don't use the host kernel's infra for implementing vsock and we rely on per-microVM UDS, multiple microVMs can have the share the same CID without issues.

Please note that when using the jailer to launch microVMs, which is the suggested (and only supported) way to launch microVMs in a production environments, we don't need to do anything on the snapshot resume path. The UDS will live in the jailed environment in a different chroot per microVM. So there is not a vsock problem with creating multiple microVMs from a single snapshot.

If using the jailer is not an option, your PR @Swchexit is the way to go, so I will take a look at it as we recently merged a similar change for network devices. However, I would like to emphasize again that we strongly advise against launching Firecracker microVMs without a jailer as this removes an important security barrier in production deployments.

[sawradip]

any plan to handle this??
@bchalios

[xmarcalx]

Hi @sawradip ,

sorry for the late reply.
We are not actively working at the moment at this feature but if you are interested and you need it, please feel free to contribute and we can guide you in the implementation.

[JamesC1305]

Support for vsock UDS path renaming has now landed in main with PR #5323, and will be included in our next release!