Regression updating from 0.3.2 to 0.3.3

[ludfjig]

After Hyperlight updated to mshv@0.3.3 we experienced sporadic "131 errors". A repro can be found here. Make sure to run it in release mode. The error sometimes doesn't show up depending on the number of iterations, for example if I replace 10_000 with 1000, it runs fine. Not sure why...

use std::{
    hint::black_box,
    sync::Arc,
    thread::{JoinHandle, spawn},
};

use mshv_bindings::{
    hv_partition_property_code_HV_PARTITION_PROPERTY_SYNTHETIC_PROC_FEATURES,
    hv_partition_synthetic_processor_features,
};
use mshv_ioctls::{Mshv, VmFd};

fn main() {
    let mshv = Arc::new(Mshv::new().expect("Failed to open mshv device"));
    let handles: Vec<JoinHandle<()>> = (0..10_000)
        .map(|_| {
            let mshv = Arc::clone(&mshv);
            spawn(move || {
                black_box(create_vm(&mshv));
            })
        })
        .collect();

    for handle in handles {
        handle.join().expect("Thread panicked");
    }
}

fn create_vm(mshv: &Mshv) -> VmFd {
    let vm = mshv
        .create_vm_with_args(&Default::default())
        .expect("Failed to create VM");
    let features: hv_partition_synthetic_processor_features = Default::default();
    vm.hvcall_set_partition_property(
        hv_partition_property_code_HV_PARTITION_PROPERTY_SYNTHETIC_PROC_FEATURES,
        unsafe { features.as_uint64[0] },
    )
    .expect("Failed to set vm property");
    vm.initialize().expect("Failed to initialize vm");
    vm.create_vcpu(0).expect("Failed to create vcpu");
    vm
}

On 0.3.2, this works fine, but on 0.3.3 I get errors such as

...
thread '<unnamed>' panicked at src/main.rs:32:10:
Failed to create VM: Errno(Error(131))

thread '<unnamed>' panicked at src/main.rs:39:21:
Failed to initialize vm: Errno(Error(131))
...

My system:

[ ~/mshv3-tester ]$ uname -a
Linux aure-linux3-dev-vm 6.6.57.mshv4-1000.g59e1545b.lv3 #1 SMP Tue Nov 19 14:35:16 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
[ ~/mshv3-tester ]$ rustc --version
rustc 1.85.0 (4d91de4e4 2025-02-17)
[ ~/mshv3-tester ]$ 

[russell-islam]

What do you see in the dmesg specific the mshv driver?

[ludfjig]

I can see a lot of the following:

[ 9602.838529] hv_call_create_partition: HV_STATUS_NO_RESOURCES
[ 9602.838870] hv_call_create_partition: HV_STATUS_NO_RESOURCES
[ 9602.840102] hv_call_initialize_partition: HV_STATUS_NO_RESOURCES
[ 9602.840767] hv_call_create_partition: HV_STATUS_NO_RESOURCES
[ 9602.842370] hv_call_create_partition: HV_STATUS_NO_RESOURCES
[ 9602.844064] hv_call_create_partition: HV_STATUS_NO_RESOURCES
[ 9602.846489] hv_call_initialize_partition: HV_STATUS_NO_RESOURCES
[ 9602.847559] hv_call_initialize_partition: HV_STATUS_NO_RESOURCES
[ 9602.854566] hv_call_initialize_partition: HV_STATUS_NO_RESOURCES

[russell-islam]

@NunoDasNeves Any idea?

[jinankjain]

I am able to repro this on 0.3.2 as well:

[russell-islam]

@ludfjig How many succeeded out of 10_000?

[NunoDasNeves]

I'm not at all sure why you're only seeing this in v0.3.3.
The only significant change I can see since v0.3.2 is that we use the vp register page now: #183 .
So, maybe using the vp register page (not just mapping it, which we were already doing in the kernel) is causing some extra resource usage in the hypervisor and the limit is getting hit sooner due to that? That's all I can think of, for now...

[russell-islam]

Creating each VCPU maps the Register page to root. Look at the code that creates one VCPU per vm

[ludfjig]

@ludfjig How many succeeded out of 10_000?

For me, the first 2040 are initialized successfully, but the 2041th fails. After that, all of the rest fails. This 2040 number is consistent across runs (same for both --release and debug profile).

I also realized that the error still occurs on the 2041st iteration when running sequentially, with no spawned threads at all:

use std::{
    hint::black_box,
    sync::Arc,
    thread::{JoinHandle, spawn},
};

use mshv_bindings::{
    hv_partition_property_code_HV_PARTITION_PROPERTY_SYNTHETIC_PROC_FEATURES,
    hv_partition_synthetic_processor_features,
};
use mshv_ioctls::{Mshv, VmFd};

fn main() {
    let mshv = Arc::new(Mshv::new().expect("Failed to open mshv device"));
    let _: Vec<i32> = (0..3_000)
        .map(|i| {
            let mshv = Arc::clone(&mshv);
            create_vm(&mshv);
            i
        })
        .collect();
}

fn create_vm(mshv: &Mshv) -> VmFd {
    let vm = mshv
        .create_vm_with_args(&Default::default())
        .expect("Failed to create VM");
    let features: hv_partition_synthetic_processor_features = Default::default();
    vm.hvcall_set_partition_property(
        hv_partition_property_code_HV_PARTITION_PROPERTY_SYNTHETIC_PROC_FEATURES,
        unsafe { features.as_uint64[0] },
    )
    .expect("Failed to set vm property");
    vm.initialize().expect("Failed to initialize vm");
    vm.create_vcpu(0).expect("Failed to create vcpu");
    println!("pass!");
    vm
}

Run with cargo run --release &> log.txt and search log.txt for pass! and you'll find exactly 2040 occurrences

[russell-islam]

@ludfjig Is this something just part of your test case (Creating thousands of VM)? Or there is any use case?

[ludfjig]

@russell-islam I discovered in part of a test case of ours, but it is indeed a use-case we'd want to support.

Aside: In the repro above, why aren't the VM memory cleaned up after each iteration? I was under impression that dropping the vm_fd cleans up the memory.