Repository navigation
openvmm: restore one-shot LAPIC timers without drift or a downtime refire - #1505
Open
Pedro Henrique Penna (ppenna) wants to merge 3 commits into
Open
Pedro Henrique Penna (ppenna) wants to merge 3 commits into
Pedro Henrique Penna (ppenna) wants to merge 3 commits into
Conversation
On KVM, a time ABI restore could give a guest one extra LAPIC timer interrupt, and fired a running one-shot timer early (#1366). KVM_SET_LAPIC fires a one-shot timer whose current count is 0 again, so a timer that expired during the downtime, whose vector the restore queues in IRR, reached a VP restored runnable with interrupts enabled twice. And the capture saved each VP's timer count after its anchor, while the restore let KVM run the saved timer from the processor restore, before the restore anchor, and then subtracted the whole downtime, so a restored timer fired early by both windows, milliseconds in the VM worker. MSHV and WHP freeze partition time over both windows. Pin 024ecee62d7c9b70456b63a5b4f94a6488f835cd, the head of microsoft/openvmm#5053, three commits on 90f87319d, the head of feature/microvm/dev: - 2a0401c43 virt_kvm: share the guest RAM of the real-mode KVM tests - 21a76f878 virt_kvm: do not fire an expired LAPIC timer whose interrupt is pending - 024ecee62 vmm_core, virt: run a restored LAPIC timer from the time ABI anchors The KVM backend now gives KVM an initial count of 0 for an expired one-shot timer whose unmasked LVT entry delivers a fixed interrupt to a valid vector that is pending in IRR, so that it fires once. A time ABI restore takes each VP's timer out of its saved APIC state and sets it only after the synchronized TSC set, from its count at the capture anchor, which the saved TSC dates, advanced by the downtime and by the ticks of the TSC past the set's target, so that it runs as the TSC does, as the previous pin's VM time does: from the restore anchor on KVM, and on MSHV and WHP from the resumption of partition time that ends the set. No saved-state, manifest, or time ABI format changes; a snapshot taken before this pin restores with the same correction. A one-shot timer that expired before the capture, and whose interrupt the guest took, still fires once more on KVM. KVM saves it with the same LAPIC state as a timer that expired while its vCPU was stopped and whose interrupt KVM has not injected yet, which is the common state at the capture, since the snapshot boundary stops the VPs before the quiesce. Giving KVM an initial count of 0 for such a page loses that interrupt, so the restore keeps KVM's re-fire, and only a KVM change that tells the two apart can remove the spurious one (#1390). Evidence: KVM-backed tests run a real-mode guest that counts its timer interrupts through the restore sequence, on nvx-kvm-test (Linux 6.6, nested, without APICv or the VMX preemption timer). A timer armed for 1 s and restored after a 2 s downtime with interrupts enabled fires twice without the KVM backend's initial count of 0, and once at the pin. A timer that expired while its VP was stopped is saved expired, with its vector neither pending nor in service, the same page as a timer delivered before the capture, and fires once at the pin, while an initial count of 0 for that page gives it no interrupt at all. A 100 ms timer, with 20 ms from the capture anchor to the save and 20 ms from the processor restore to the restore anchor, fires 29.3 to 29.5 ms early with the earlier restore sequence, and 161 to 333 us late at the pin in 15 runs under a load average of 9, against 176 to 234 us in 13 of 15 runs of the same timer without a snapshot, and 429 us and 3.3 ms in the others. On MSHV, whose frozen write restarts the TSCs 45 to 134 us after the restore anchor in the scenarios' restores, the LAPIC advance runs after that restart and counts each timer from its VP's TSC, so the timers run from where the previous pin runs VM time. At the pin, the smp-lapic, smp-snapshot, restore-processors, snapshot-core, snapshot-rollback, and time-abi-conformance microVM scenarios pass on KVM (nvx-kvm-test) and MSHV (nvx-mshv-test), with OpenVMM built from the pin. The restore-downtime scenario, which now simulates a year, needs guest artifacts with the 365-day bound, and is left to CI with WHP. NVX needs no other coordinated change: the control-session vectors, the source excerpts, the contract versions, and the CPU profiles that nvx-rs copies from OpenVMM match the revision's. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The time ABI's timer rule (3.2.5) advanced a counting one-shot LAPIC timer by the downtime's ticks without saying from which count, or from when the restored timer runs, and the restore protocol (5.5) restored every processor with its timer running before the restore anchor. On KVM, whose timers count while the VPs are stopped, a restored timer therefore fired early, and a timer that the downtime expired fired twice (#1366). 3.2.5 now says that the count that advances is the timer's count at the capture anchor, which the saved TSC dates, since a save reads each processor's TSC after its LAPIC state, and that the restored timer runs as the TSC does: the restore runs no timer until it advances the count by the downtime's ticks and by the ticks for which the TSC has run from the target that the restore wrote to it, from the restore anchor, or, on a backend that writes the TSCs with partition time frozen, from the resumption of partition time (3.2.8). It also states what KVM's LAPIC state leaves open: a one-shot timer that expired before the capture keeps its counts and fires once more on KVM, which is due if KVM had not delivered its interrupt and spurious if the guest had taken it (#1390), and on KVM an expired timer whose unmasked LVT entry delivers a fixed interrupt to a valid vector that is pending in IRR reads an initial count of zero after the restore, so that KVM does not fire it again. Steps 14 and 18 of the restore protocol now restore every processor with its LAPIC timer stopped, and set the timers again after the TSC readback, beside step 18's VM time, which runs from the instant from which the TSCs run. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Maps NVX's new pin, 024ecee62, the head of microsoft/openvmm#5053, at position 258, which with positions 256 and 257, 2a0401c43 and 21a76f878, follows position 255, 90f87319d, the head of feature/microvm/dev, which microsoft/openvmm#5052 merged. The pin is 127 commits past the mapped tip. They change how a restore runs a one-shot LAPIC timer and change nothing that a mapped commit adds, so they stay unassigned. Position 256 moves the guest RAM of the real-mode ExtINT tests of virt_kvm, which upstream main has, into a test module that position 257's tests share. Position 257 changes the KVM backend's set_apic, which upstream main has, and the Guide's timekeeping reference, which position 255 also changes. Position 258 changes the time ABI restore of the LAPIC timers that position 139 adds, which positions 146, 147, 155, 158, and 255 also change, and sets each timer again after the synchronized TSC set, so that it runs as the TSCs do, as position 255's VM time does. Replaying position 256 on upstream main, ba105dd7f, conflicts only in virt_kvm's x86-64 module list, beside the modules of positions 133 and 138. Position 257 applies there without conflicts, but its KVM tests use position 256's guest RAM, position 139's timer advance, the TSC helpers of positions 132 and 160, and the kvm crate's TSC accessors of PR10's position 026, so it follows them upstream. Position 258 conflicts in all 12 of its files, seven of which upstream main lacks, so it follows the time ABI series upstream. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
Cross-backend interrupt and timer restoration changes still lack exact-pin WHP and long-downtime validation, with required CI checks and the prerequisite OpenVMM #5053 fast-forward merge outstanding.
No open issues 🎉
What changed in this PR
This PR documents the OpenVMM LAPIC timer restore fixes for #1366: preventing duplicate interrupts for timers that expire during downtime and correcting timer drift. It also records the remaining KVM limitation for timers whose interrupts were delivered before capture, tracked by #1390.
Changes:
- Specifies timer advancement relative to the capture anchor and restored TSC.
- Updates the restore sequence to hold LAPIC timers stopped until after TSC restoration.
- Records OpenVMM pin
024ecee62and maps its three commits at roadmap positions 256–258.
| File | Description |
|---|---|
| doc/openvmm-upstream-roadmap.md | Records the new pin, commit mappings, and upstream dependencies. |
| doc/design/5.5-restore-protocol.md | Documents deferred timer restoration and anchor-based advancement. |
| doc/design/3.2-clock-and-time.md | Defines timer timing semantics and the remaining KVM re-fire limitation. |
Review effort: Balanced
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #1366
What and why
On KVM, under the NVX time ABI, a restored one-shot LAPIC timer could fire an extra time or fire early (#1366). This pull request fixes two of the issue's three cases and documents the third as a KVM limit:
KVM_SET_LAPICthen fires the expired one-shot timer again, so a VP restored runnable with interrupts enabled took the interrupt twice.KVM_GET_LAPICsaves it with the same page as a timer whose expiry KVM has not injected yet, which is the common state at OpenVMM's capture point, since the snapshot boundary stops the VPs before the quiesce. An initial count of 0 for such a page would lose that interrupt, so the restore keeps KVM's re-fire. openvmm: KVM_SET_LAPIC fires an expired, already delivered one-shot timer again #1390, the KVM half of this issue, addresses it, and its fix is stacked on this pull request.Part of #1366, notFixes, because case 1 is part of the issue's text and stays open until #1390's fix lands.The pin, microsoft/openvmm#5053 at
024ecee62d7c9b70456b63a5b4f94a6488f835cd, adds three commits tofeature/microvm/dev:2a0401c43virt_kvm: share the guest RAM of the real-mode KVM tests. Test-only: the ExtINT tests' guest RAM moves to a module that the new timer tests share.21a76f878virt_kvm: do not fire an expired LAPIC timer whose interrupt is pending. For an expired one-shot timer whose unmasked LVT entry delivers a fixed interrupt to a valid vector that is pending in IRR, the KVM backend gives KVM an initial count of 0. KVM starts no such timer, and IRR already holds the interrupt that the expiry owes, so the timer fires once. The guest reads 0 from the initial count register of such a timer. MSHV and WHP do not re-fire, and are unchanged.024ecee62vmm_core, virt: run a restored LAPIC timer from the time ABI anchors.dev, runs VM time.The NVX commits:
9eb1cb2901dopenvmm: pin LAPIC timer restores without drift or a downtime refire. It changes only the gitlink, from90f87319dto024ecee62.b6cf1642f20doc: specify how a restore runs a one-shot LAPIC timer. Section 3.2.5 specifies the timer's restore and the KVM limit, and restore steps 14 and 18 hold the timers until after the TSC readback, beside step 18's VM time.a9ada35d79bdoc: map the LAPIC timer restore fixes in the OpenVMM roadmap. Positions 256 to 258.No saved state, manifest, time ABI, contract, or format version changes. A snapshot taken before this change restores with the same correction.
Validation
Local checks at this head (Windows):
python scripts/openvmm_pin.py check-branch --topic: passed (3 signed commits, no coordinated files).python scripts/openvmm_pin.py check-roadmap --worktree: passed (258 positions, 267 references).python scripts/openvmm_pin.py source 024ecee62d7c9b70456b63a5b4f94a6488f835cd --pr 5053: good signature, and the revision is the pull request's head.python scripts/nvx.py verify: passed.python scripts/check_test_changes.py --base origin/dev: 0 changes to existing tests.python scripts/check_test_layout.py: passed.python scripts/check_markdown_links.py: passed.git diff --check origin/dev HEAD: clean.cargo test --locked --manifest-path nvx-rs/Cargo.toml --no-default-features --test openvmm_vectors -- --ignored: 4 passed.cargo test --locked --manifest-path nvx-rs/Cargo.toml --no-default-features --test openvmm_cpu_profiles -- --ignored: 1 passed.nvx-cligate,python -m unittest discoveroverscripts/: not run locally. This change does not touchscripts/, and the gate takes over two hours on this shared machine. It is left to CI'sNVX CLI / LinuxandNVX CLI / Windowsjobs, which must pass on this head.OpenVMM package checks at
024ecee62, forvirt,vmm_core,openvmm_core, andvirt_kvm:cargo clippy --all-targetsandcargo doc --no-depsproduced no warnings from the packages;cargo nextest run --profile agent: 282 passed, 1 skipped;cargo xtask fmt --fix: clean.virt_kvmpassed.Remote runs at this pin, with OpenVMM built from
024ecee62over thev0.1.0-dev.c52fc3f2fd33artifacts (the binary's hash changed on both hosts). The KVM measurements ran under a load average of 9, and "before" reverts the fix in the test's restore sequence:smp-lapic(1, 2, 4, 8),smp-snapshot,restore-processors,snapshot-core,snapshot-rollback,time-abi-conformancefb4f34126, before the rebase: those scenarios,restore-downtime, andsnapshot-tiersLeft to CI:
openvmm-ci: OpenVMM's required jobs for [DO NOT REVIEW] vmm_core, virt_kvm: restore one-shot LAPIC timers without drift or a downtime refire openvmm#5053.NVX CLI / LinuxandNVX CLI / Windows.restore-downtime. That scenario now simulates a year, and needs the guest artifacts with the 365-day bound, which no test host had.No existing test changes, so no
tests-modifiedlabel.024ecee62d7c9b70456b63a5b4f94a6488f835cd.90f87319d, the head offeature/microvm/devafter [DO NOT REVIEW] openvmm: run restored VM time from the instant the TSCs resume openvmm#5052 (openvmm: after a microVM restore, the TSC leads VM time by the restore's length #1371). There are no other prerequisites.fb4f34126:2a0401c43and21a76f878are patch-identical todba7fe25band7207732c8(git range-diff).024ecee62differs frome61a57018only in comments, the snapshot guide's wording, and its message. After openvmm: after a microVM restore, the TSC leads VM time by the restore's length #1371, "from the restore anchor" became "as the TSC does" or "past the target", since the TSCs on MSHV and WHP restart only when partition time resumes.feature/microvm/devfrom90f87319dto exactly024ecee62d7c9b70456b63a5b4f94a6488f835cd. OpenVMM's merge button would squash the commits and leave the pin off the branch. This pull request then merges through the queue.