tdx vTPM: Source of device firmware/ ROM files passed to guest

[starka-work]

We are working on attesting preview TDX CVMs on Azure that use OpenHCL.
The vTPM PCR2 contains 38 measurements of type "EV_EFI_BOOT_SERVICES_DRIVER".
These measurements are not present on the non-OpenHCL version.

It is unclear where the firmware for these comes from, as they do not seem to be part of the openHCL project directly.
Our best guess is that they are either from mu_msvm or Hyper-V(vmfirmwarepcat.dll, vmfirmware.dll)?

Is it documented somewhere what ROM files openHCL forwards to the OS?
Are these embedded into the OpenHCL binary, or are they passed through from the hypervisor?

[maheeraeron]

@starka-work Are you able to share the attestation report or where you found these measurements?

The current suspicion I have might revolve around different UEFI firmwares.

The non-OpenHCL version you refer to-- I think-- is referring to our legacy HCL, which has an older UEFI firmware. It's bundled into vmfirmwarehcl.dll.

Now that you guys are using OpenHCL, which also bundles a UEFI firmware into its IGVM file, it's likely this UEFI firmware is different compared to the aforementioned older UEFI firmware. The binary for this is vmfirmwareigvm.dll

Namely, the newer UEFI firmware differs in changes that are part of some submodules in upstream code.

[starka-work]

I attached the decoded attestation report for both hcl (hcl_pcrs-parsed.json) and openhcl (openhcl_pcrs-parsed.json).

The raw files with the quote and eventlog. hcl_raw.zip openhcl_raw.zip

[maheeraeron]

@starka-work, I took a look at the parsed json files. In summary, I see the following:

• Some PCR0 uefiplatformfirmwareblob measurement changed in its hash. The blobbase and bloblength values also got flipped
• A bunch of PCR2 measurements to some boot services drivers that got added
• A TPM_PCR_INIT value on PCR2 got removed

The legacy HCL used the AH2024 UEFI firmware. These used the release/202302 branches in the following submodules:

[submodule "MU_BASECORE"]
    path = MU_BASECORE
    url = https://dev.azure.com/microsoft/OS/_git/hyperv.uefi.mu_basecore
    branch = release/202302
[submodule "Common/MU"]
    path = Common/MU
    url = https://github.com/Microsoft/mu_plus.git
    branch = release/202302
[submodule "Common/MU_TIANO"]
    path = Common/MU_TIANO
    url = https://dev.azure.com/microsoft/OS/_git/hyperv.uefi.mu_tiano_plus
    branch = release/202302
[submodule "Silicon/ARM/MU_TIANO"]
    path = Silicon/ARM/MU_TIANO
    url = https://github.com/Microsoft/mu_silicon_arm_tiano.git
    branch = release/202302

@starka-work, which OpenHCL branch/version are you using specifically? I can track which UEFI submodules we used here. It is very likely those use newer submodules of course (either release/202405 or release/202502).

To my understanding, the newer submodule branches upstream could contain changes to PCR2 measurements. But I'm working with the core UEFI team to get an idea of what specifically changed and why.

To answer some of your other questions:

Is it documented somewhere what ROM files openHCL forwards to the OS?

By ROM files, do you mean what binary OpenHCL uses to load the UEFI firmware? See the building instructions for mu_msvm here: https://github.com/microsoft/mu_msvm/blob/main/MsvmPkg/Docs/BUILDING.md

NOTE: Release flavors of OpenHCL come from our closed source repository, which get built on its ADO pipelines. The build process is pretty similar of course.

The public OpenHCL consumes the UEFI firmware through a github release on mu_msvm. However, the release OpenHCL that we ship gets consumed as a nuget packaged, which is generated from the ADO closed source hyperv-uefi repository pipelines.

The legacyHCL, however, consumes the UEFI firmware from a published vpack generated by the ADO closed source hyperv-uefi repository pipelines.

Are these embedded into the OpenHCL binary, or are they passed through from the hypervisor?

You can see how the open-source OpenHCL consumes the UEFI firmware from github releases from mu_msvm here: https://github.com/microsoft/openvmm/blob/30fc386344fdd8145dbf1e2cce5cbcf75a57b99b/flowey/flowey_lib_hvlite/src/_jobs/cfg_versions.rs#L25C1-L25C36

The UEFI firmware gets embedded to the openhcl.bin file. If using an official OpenHCL that's in-box with Windows, that openhcl.bin file then gets embedded as part of vmfirmwareigvm.dll

Similarly, for the legacy HCL, the UEFI firmware it consumes is part of vmfirmwarehcl.dll

[smo4201]

Thanks @maheeraeron for the detailed reply, we are closer now (replying for @starka-work).

We downloaded MSVM.fd from here, specifically versions v25.0.2 (release/2505) and v24.0.4 (release/2411), as these were the only two release branches. You mentioned release/202405 and release/202502, but we could not find these branches/tags in this repository.

We then inspected the MSVM.fd and observed that it contains PE Images with UUIDs matching all of the parsed UUIDs from the TPM PCR2 event log. So from my understanding, this should be the UEFI boot services drivers that get measured into PCR2.

Therefore, we extracted and hashed these PE images as OVMF does. However, using the downloaded versions, we were not able to match the hashes (we verified our measuring code against our own VMs using OVMF, but did not yet verify, whether mu_msvm does contain patches altering the edk2/OVMF measuring code).

@starka-work, which OpenHCL branch/version are you using specifically? I can track which UEFI submodules we used here. It is very likely those use newer submodules of course (either release/202405 or release/202502).

We are using the official "Azure confidential VMs DCesv6 and ECesv6 preview with Intel TDX" on Azure. We were not able to determine the openhcl version from inside our Azure VMs, if you can tell us how to determine the version, we can do so.

However, I assume this:

The public OpenHCL consumes the UEFI firmware through a github release on mu_msvm. However, the release OpenHCL that we ship gets consumed as a nuget packaged, which is generated from the ADO closed source hyperv-uefi repository pipelines.

means that we cannot reproduce the hashes of the github release MSVM.fd from mu_msvm, as there is closed source code involved? So this, and potentially mismatched mu_msvm versions are probably the reason, we cannot reproduce the PCR2 hashes.

Furthermore, to fully establish trust in the TDX CVM, we would need to build mu_msvm from source (thanks for providing the build instructions). I guess, we will also not be able reproduce the MSVM.fd, as running on Azure CVMs, because of the closed source pipelines?

Are you planning to provide a reproducible build for mu_msvm, so that the hashes can be reproduced and trust in TDX CVMs can be established purely based on the open source firmware?