Skip to content

tests: run every cold cell and the public VM lifecycle on real VMs (SL5.1b) - #1496

Draft
Pedro Henrique Penna (ppenna) wants to merge 3 commits into
feature/nvx-refactorfrom
agents/slim-sl5-1b-cold-smoke
Draft

tests: run every cold cell and the public VM lifecycle on real VMs (SL5.1b)#1496
Pedro Henrique Penna (ppenna) wants to merge 3 commits into
feature/nvx-refactorfrom
agents/slim-sl5-1b-cold-smoke

Conversation

@ppenna

@ppenna Pedro Henrique Penna (ppenna) commented Nov 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Part of #1118 (SL5.1b, "Cold boots work end to end on real VMs"). Parent #1086, umbrella #1076.

What and why

NVX deletes its runtime adapter in SL5.2 (#1119), once the tests that replace it pass. Until now, only the adapter's VM tests booted a GPT root. Only the adapter took a cold VM through create, spawn, reattach and stop using nothing but the public API on a real hypervisor. This PR adds both checks to the existing real-VM lane, scripts/nvx.py test-microvm, which CI's guest and sandbox job runs on each backend. It also records in the case matrix that the four public cold-boot scenarios are kept. It changes no production code.

  • The twelve cold cells. The CLI has twelve cold cells: an EROFS layer or a GPT disk, over a scratch upper or a RAM upper, on KVM, MSHV or WHP (nvx-rs: final acceptance for the image-backed runtime #624). The EROFS cells already ran:

    • EROFS over scratch: CI's layer smoke step runs it with sandbox run, and sandbox-lifecycle and managed-exec-config run it with provision and start.
    • EROFS over RAM: sandbox-ram-upper runs it with run and with provision and start.

    The new public sandbox-gpt-layer scenario covers the GPT cells. It writes a GPT disk with the UEFI layout: a protective MBR, both headers and both entry arrays with their CRCs, and 2048-sector alignment. The disk has a first partition of a few bytes and a layer partition of at least 256 MiB. A guest of the control initramfs formats the layer partition as ext4 and copies the Ubuntu EROFS layer's tree into it.

    The scenario then runs the disk over a scratch upper and over a RAM upper. For each upper, it runs the smoke probe with sandbox run, and takes a managed sandbox through provision, start, exec, stop and deprovision. Each VM must:

    • boot the disk over that upper alone, in a VM that nvx-init manages;
    • have only the disk's GUID and that upper's token on its kernel command line;
    • pass /sbin/nvx-sandbox-smoke's checks;
    • show the guest the layer partition as the root's lower directory;
    • stop with a successful outcome and a complete teardown.

    The run must attach the disk read-only, and no cell may change the disk's digest. Each backend's CI job runs the cells of its own backend. Together, the jobs cover the fixed product of two formats, two uppers and three backends, without a matrix abstraction.

  • The public API's lifecycle. nvx-rs/tests/vm_lifecycle.rs holds one ignored test, a_cold_vm_is_reattached_from_saved_values_and_stopped, which the new public vm-lifecycle scenario runs with Cargo. The test:

    1. Creates the VM with vm::create: nvx-init over the Ubuntu EROFS layer and a scratch upper, as sandbox start configures one. The creation must start nothing.
    2. Starts it with vm::spawn. The caller keeps only SavedVm's values and the VmOutputs, and drops the launch.
    3. Reattaches with vm::reattach from those saved values alone, twice. Each time, it opens a new authenticated connection, which the guest answers.
    4. Stops the VM with vm::stop, through its guest. The guest must exit without a forced termination and report success and a complete teardown.

    If any step fails, the test ends the VM from the saved values alone. The scenario passes the test the runtime's files with their digests. Afterwards, it checks the VM's outcome report and that no OpenVMM process of the VM remains. Its Cargo command now has the same form as managed-lifecycle's; both use ignored_nvx_rs_test_command.

  • CI. Both "Run public managed sandbox acceptance" steps of run-nvx-microvm-tests.yml add --scenario sandbox-gpt-layer --scenario vm-lifecycle. Each backend's guest and sandbox job now runs these two beside the four existing public scenarios.

  • The case matrix. doc/acceptance-case-matrix.md gains a "Cold boots" section. For each of the cells, it names the scenario or CI step that boots the cell through run, and the one that boots it through provision and start. It also says what vm-lifecycle checks. The four current scenarios are kept:

    • managed-exec-config and sandbox-lifecycle boot EROFS over scratch.
    • sandbox-proxy-only covers the network that it checks.
    • sandbox-ram-upper boots EROFS over RAM.

    The adapter's erofs_over_* and gpt_over_* scenarios, and its restart and reattachment scenarios, now name the tests that replace them.

  • Docs. doc/run.md, doc/ci.md, doc/usage.md, doc/contribute.md, doc/project-structure.md and nvx-rs/README.md describe the two scenarios and the test.

Size

Measured against feature/nvx-refactor 3c3dfce1:

The second counted file comes from adding two public scenarios to the harness, instead of folding the new steps into the existing scenarios. This way, the four current scenarios stay unchanged and CI names each new scenario.

Commits

  1. tests: run the CLI's cold cells of a GPT disk (co-author Enrique Saurez (@esaurez))
  2. tests: reattach to and stop a VM through the public API (co-author Enrique Saurez (@esaurez))
  3. doc: map the cold cells and the VM lifecycle in the case matrix

Enrique Saurez (@esaurez)'s first implementation is esaurez/private-sl51b-0325 (e30ce117, from the checkpoint on #1118). Its own commits, 79e96e14, 4bbe3952 and e30ce117, added the GPT cells and the API smoke test to managed-exec-config. This branch reworks them onto the current base as two public scenarios that use the CLI's current configuration format and probe arguments, and credits him as co-author.

Not in this PR

  • Snapshot capture and restore, which SL5.1c adds.
  • Deleting the adapter, its runner or its gates, which SL5.2 (nvx-rs slimming 4.1.2: the test adapter is gone #1119) does.
  • A new harness, runner pool or test framework.
  • More formats, uppers or hypervisors.
  • Changes to the production API or CLI. The public API snapshots don't change.

vm_lifecycle.rs uses NVXC's public protocol::encode_app and decode_app, so a later change that makes NVXC private must rewrite it.

Validation

Real VMs, on bd342380, whose three commits are identical to this head's (b7b8aff3, per git range-diff). OpenVMM and the guest artifacts came from CI run 43075697333. That run built feature/nvx-refactor 3340bbd7 with the same openvmm pin, 14dc4116. The only guest-side change since then is #1488's unit test.

Backend Host test-microvm: managed-exec-config, sandbox-lifecycle, sandbox-proxy-only, sandbox-ram-upper, sandbox-gpt-layer, vm-lifecycle CI's layer smoke step (sandbox run, EROFS over scratch)
Linux/KVM nvx-kvm-test passed passed
Linux/MSHV nvx-mshv-test passed passed
Windows/WHP local passed passed

Together, these runs passed all twelve cold cells and the public API's lifecycle on each backend. The same runs also passed on two earlier heads: ac8a4c0e and db9c6fd2. The latter was on SL3.4b's earlier revision. On every backend, the GPT layer's tree takes about 100 MB of its 256 MiB partition.

The runs predate the rebase onto 3c3dfce1. That base adds #1488 itself, a capture unit test fix, and an edit to an example, none of which changes the VMs that these scenarios boot.

Gates (from this branch's check-nvx-rs, check-quality and validate-nvx actions; Rust in nvx-rs/ with NVX_BUNDLE=skip):

Check Windows (local) Linux (nvx-kvm-test)
cargo fmt --all --check passed passed
The six cargo clippy --locked ... -- -D warnings runs of the Lint step passed, all six --all-targets --all-features passed
cargo test --locked --all-features, cargo test --locked, and cargo build/test --locked --lib --no-default-features passed not run: this host fails 28 guest:: unit tests at the base
cargo +1.89 check --locked --all-targets --all-features passed
cargo test --locked --test vm_lifecycle, which builds the ignored test passed
ruff check and ruff format --check of scripts .github/specula; pyright --pythonplatform Linux and Windows passed
validate-nvx's Python suite 1847 tests, OK (73 skipped) test_microvm_tests and test_nvx_tools passed
test_adversarial.py, test_hosts.py, and the nvx.py ... --help commands passed
check_markdown_links.py, check_test_layout.py, nvx.py check-kernel-config passed check_test_layout.py passed

The Rust gates ran on ac8a4c0e; my commits have changed only type annotations in the Python harness and its tests since then, which pyright had flagged. On this head, b7b8aff3, the pyright, ruff, Python-suite, link and test-layout checks were rerun on Windows, and all passed.

Left to CI:

  • shellcheck and shfmt (this PR changes no shell scripts);
  • Specula's tests (Linux only; unchanged here);
  • the public API check (no change);
  • the macOS check (the new test builds only on Linux and Windows);
  • the guest binaries;
  • the full nvx-rs suites on Linux;
  • the runtime adapter;
  • the microVM lanes.

Changes to existing tests

python3 scripts/check_test_changes.py --base origin/feature/nvx-refactor lists 6 changes to existing tests and 65 added items. This PR therefore has the tests-modified label:

  • CiConfigurationTests.test_ci_runs_the_public_sandbox_acceptance_on_every_backend, in scripts/test_nvx_tools.py: it also requires --scenario sandbox-gpt-layer and --scenario vm-lifecycle in both public acceptance steps.
  • MICROVM_TEST_SCENARIOS, SANDBOX_CONTROL_SCENARIOS and PUBLIC_SANDBOX_SCENARIOS, in scripts/nvx_tools/microvm_tests.py: they now list the two new scenarios. As a result, --scenario accepts them, guests without sandbox control skip them, and CI runs them in the guest and sandbox jobs instead of in the default suite's shards.
  • managed_lifecycle_test_command: it returns the same command as before, now through the new ignored_nvx_rs_test_command, which vm-lifecycle also uses.
  • _run_scenarios: it now dispatches the two new scenarios.

None of these changes an existing test's checks, and no test is removed.

@ppenna Pedro Henrique Penna (ppenna) added tests-modified Changes existing tests, test data, or test wiring feat-nvx-refactor labels Nov 11, 2026
@ppenna Enrique Saurez
tests: run the CLI's cold cells of a GPT disk
42a7e52 
The CLI's twelve cold cells are an EROFS layer and a GPT disk, each
over a scratch upper and a RAM upper, on KVM, MSHV and WHP (#624). The
public sandbox scenarios boot the EROFS layer over both uppers on each
backend, but no test booted a GPT disk through the CLI: only the
runtime adapter's VM tests assembled GPT roots, and NVX deletes the
adapter once the tests that replace it pass.

The public sandbox-gpt-layer scenario of test-microvm now runs the GPT
cells of its backend. It writes a GPT disk as UEFI lays one out, with a
first partition of a few bytes and an empty layer partition, which a
guest of the control initramfs formats as ext4 and fills with a copy of
the tree of the Ubuntu EROFS layer. Then, over a scratch upper and over
a RAM upper, a sandbox run of the smoke probe, and a managed sandbox's
provision, start, exec, stop and deprovision, must boot the disk over
the upper alone in a VM that nvx-init manages, with the disk's GUID and
the upper's token alone on the kernel command line, pass the probe's
checks, show the guest the disk's layer partition as the lower
directory of its root, and stop with a successful outcome and a
complete teardown, while the disk keeps its digest. CI's guest and
sandbox job of each backend runs it beside the other public sandbox
scenarios.

Evidence: the scenario passes on Linux/KVM (nvx-kvm-test), Linux/MSHV
(nvx-mshv-test) and Windows/WHP (local), with OpenVMM and the guest
artifacts of CI run 43075697333; the layer's tree takes about 100 MB
of the partition's 256 MiB.

Tests: a fake of the CLI drives the scenario through both uppers and
the regressions that it must catch, and the disk's tables are checked
as UEFI specifies them.

Part of #1118 (SL5.1b).

Co-authored-by: Enrique Saurez <esaurez@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ppenna Enrique Saurez
tests: reattach to and stop a VM through the public API
74f64fa 
nvx's public API creates and spawns a cold VM, and reattaches to it and
stops it from the values that its caller saved, with no store of its
own. Its unit tests check each step with stand-ins for OpenVMM, and the
CLI's sandbox scenarios drive the steps through nvx-rs, but only the
runtime adapter ran them through the public API alone on a real VM,
and NVX deletes the adapter once the tests that replace it pass.

nvx-rs/tests/vm_lifecycle.rs now does, in an ignored test that the new
public vm-lifecycle scenario of test-microvm runs with Cargo. The test
creates a cold launch of nvx-init over the Ubuntu EROFS layer and a
scratch upper, as nvx-rs sandbox start configures one, and checks that
the creation started nothing; spawns it; saves the values of SavedVm
and the VM's outputs, and lets the launch go; reattaches from those
values alone, twice, each time with a new authenticated connection
that the guest answers; and stops the VM through its guest, which must
exit without a forced termination and report its success and a
complete teardown. A failure ends the VM from the saved values alone.
The capability stays in the test's memory. The scenario gives the test
the runtime's files and their digests, then checks the outcome report
that it leaves, and that no OpenVMM of its VM remains. Its Cargo
command takes the form of managed-lifecycle's, which both now share.
CI's guest and sandbox job of each backend runs it.

Evidence: the scenario passes on Linux/KVM (nvx-kvm-test), Linux/MSHV
(nvx-mshv-test) and Windows/WHP (local), with OpenVMM and the guest
artifacts of CI run 43075697333.

Tests: a stand-in for Cargo's run of the test checks the inputs that
the scenario gives it, its evidence, and the failures that the
scenario reports: a failed test, a failed stop, a VM that outlives the
test, and a missing Cargo.

Part of #1118 (SL5.1b).

Co-authored-by: Enrique Saurez <esaurez@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ppenna
doc: map the cold cells and the VM lifecycle in the case matrix
b7b8aff 
The case matrix left it to SL5.1b to keep the four public sandbox
scenarios of a cold boot running or to give them their fates, and
pointed the adapter's cold-boot and reattachment scenarios at SL5.1b's
tests, which didn't exist yet.

The matrix now lists the CLI's twelve cold cells in a Cold boots
section: for each image and upper, the scenario or CI step that boots
it through sandbox run, and through a managed sandbox's provision and
start, on each backend, and what vm-lifecycle checks through the
public API. managed-exec-config, sandbox-lifecycle, sandbox-proxy-only
and sandbox-ram-upper are kept, each with the cell that it boots or,
for sandbox-proxy-only, the network that it checks. The adapter's
erofs_over_* and gpt_over_* scenarios, and its restart and
reattachment scenarios, name the tests that cover them now, and the CI
rows of the public acceptance and the layer smoke test say what they
run.

Part of #1118 (SL5.1b).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

github-actions Bot commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

This pull request changes existing tests

It modifies or removes 6 existing test asset(s), listed below. Review each with care: a change to a test can hide a regression that the test caught. See doc/contribute.md#changes-to-existing-tests.

Test code

Change What Before After
modified test def CiConfigurationTests.test_ci_runs_the_public_sandbox_acceptance_on_every_backend in scripts/test_nvx_tools.py scripts/test_nvx_tools.py:12856 scripts/test_nvx_tools.py:12856

End-to-end tests

Change What Before After
modified helper assignment MICROVM_TEST_SCENARIOS in scripts/nvx_tools/microvm_tests.py scripts/nvx_tools/microvm_tests.py:99 scripts/nvx_tools/microvm_tests.py:128
modified helper assignment SANDBOX_CONTROL_SCENARIOS in scripts/nvx_tools/microvm_tests.py scripts/nvx_tools/microvm_tests.py:153 scripts/nvx_tools/microvm_tests.py:184
modified helper assignment PUBLIC_SANDBOX_SCENARIOS in scripts/nvx_tools/microvm_tests.py scripts/nvx_tools/microvm_tests.py:175 scripts/nvx_tools/microvm_tests.py:209
modified helper def managed_lifecycle_test_command in scripts/nvx_tools/microvm_tests.py scripts/nvx_tools/microvm_tests.py:1328 scripts/nvx_tools/microvm_tests.py:1381
modified helper def _run_scenarios in scripts/nvx_tools/microvm_tests.py scripts/nvx_tools/microvm_tests.py:10894 scripts/nvx_tools/microvm_tests.py:11441

Relocated unchanged: 0; added: 65. Compared the merge base 3c3dfce1c4c2 with b7b8aff3d7dc.

@ppenna
Pedro Henrique Penna (ppenna) requested a balanced review from Copilot and removed request for Copilot November 11, 2026 22:23
Copilot AI reviewed Nov 11, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain with the expanded cold-boot and public API lifecycle test coverage.

No open issues 🎉

What changed in this PR

Adds real-VM cold-boot and public API lifecycle coverage for SL5.1b (#1118), preparing NVX to remove its runtime adapter. The changes extend the existing test lane without changing production behavior.

Changes:

  • Adds GPT-over-scratch and GPT-over-RAM scenarios, completing coverage of the twelve cold-boot combinations across KVM, MSHV, and WHP.
  • Adds a public API test covering VM creation, spawning, reattachment from saved values, and graceful stopping.
  • Wires both scenarios into CI and documents their coverage while retaining the existing public sandbox scenarios.
File Description
scripts/​test_nvx_tools.py Requires both new scenarios in CI wiring checks.
scripts/​test_microvm_tests.py Tests GPT layout, scenario checks, cleanup, evidence, and dispatch.
scripts/​nvx_tools/​microvm_tests.py Implements and registers both scenarios and shares the Cargo command helper.
nvx-rs/​tests/​vm_lifecycle.rs Adds the ignored real-VM public API lifecycle test.
nvx-rs/​README.md Documents the lifecycle test and how to run it.
doc/​usage.md Updates scenario descriptions and Cargo prerequisites.
doc/​run.md Describes GPT acceptance checks and collected evidence.
doc/​project-structure.md Adds the public API lifecycle test to the repository layout.
doc/​contribute.md Explains how the new scenarios fit the existing harness.
doc/​ci.md Documents expanded cold-boot and lifecycle coverage in CI.
doc/​acceptance-case-matrix.md Maps cold-boot combinations and adapter cases to replacement tests.
.github/​workflows/​run-nvx-microvm-tests.yml Runs both scenarios in Linux and Windows public acceptance steps.

Review effort: Balanced

This was referenced Nov 11, 2026
Draft
Draft
Draft

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees

@ppenna Pedro Henrique Penna (ppenna)

Labels

feat-nvx-refactor tests-modified Changes existing tests, test data, or test wiring

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ppenna