Track Performance

[ludfjig]

Hyperlight Performance Plan

Core Benchmarks

Microbenchmarks (Time per Task)

Measure round-trip time (RTT) and lifecycle operations:

• Guest call RTT (empty, no workload)
• Guest call RTT (large param/return, no workload)
• Sandbox create, evolve, and drop

Test across:

• Hypervisors: Windows Server 2025/2022 Hyper-V, Linux KVM (5.x/6.x), MSHV (5.x/6.x)
• CPU vendors: Intel, AMD

Native vs Hyperlight

Compare native host code vs Hyperlight (x86_64-unknown-none) on compute-heavy tasks to evaluate performance overhead due to limited CPU feature use (e.g. SSE/AVX).

HyperlightJS & HyperlightWASM

Load Testing

Benchmark lightweight compute workloads (per-request tasks):

• Metrics: HTTP throughput, latency, memory use
• Workloads: header rewrite, JWT validation, TBD CPU-intensive task

Comparisons

Evaluate:

• HyperlightJS
• HyperlightWASM (with JS component)
• Wasmtime (with JS component, baseline)

Test across:

• Hypervisors: Windows Server 2025/2022 Hyper-V, Linux KVM (5.x/6.x), MSHV (5.x/6.x)
• CPU vendors: Intel, AMD

Performance Monitoring

Track metrics over time to catch regressions and improvements:

• Run benchmarks on every PR merge and release
• Allow local runs during dev
• Visualize data on public dashboard

Notes

• Benchmarks run in nested virtualization on CI (Standard_D4(a)ds_v5 VMs, premium SSD)
• Expect some noise, but useful for trends and comparisons

[dblnz]

Two important points that have been brought up in the last community meeting:

• When we compare measurements between commits we should run the benchmarks on the same machine (maybe keep a separate machine only to run benchmarks). Ideally we want to see the most accurate measurements without outside factors' interference.
• We should measure performance with different sandbox size(1MB, 10MB, 100MB ..), parameters size and other similar variations

[simongdavies]

I also think we need to measure creating, using and dropping sandboxes in parallel with various degrees of parallelism