Skip to content

benchmark.md

Latest commit

 

History

History
95 lines (78 loc) · 4.45 KB

benchmark.md

File metadata and controls

95 lines (78 loc) · 4.45 KB
Raw

Benchmarks

The benchmarks are classified based on their characteristics and cover various aspects of system performance. Most are derived from the industry-standard BYTEmark (nbench) suite, supplemented by cryptographic and system benchmarks:

Benchmark Description
numeric sort Focuses on sorting integer arrays using various algorithms
string sort Evaluates string sorting capabilities
bitfield Tests bitwise operations and integer arithmetic on data words
emfloat Focuses on emulating floating-point calculations using integer arithmetic
assignment Tests solving resource allocation problems (e.g., assignment algorithm)
idea Assesses encryption and decryption using the International Data Encryption Algorithm (IDEA)
huffman Measures performance in data compression using Huffman coding
dhrystone Assesses general integer performance with a mix of string processing and control operations
primes Measures efficiency in computing prime numbers using algorithms like the Sieve of Eratosthenes
sha512 Tests cryptographic hash computations

These benchmarks were performed by rv32emu (with tiered JIT enabled) and QEMU v9.0.0 on an Intel Core i7-11700 CPU running at 2.5 GHz with Ubuntu Linux 22.04.1 LTS. The toolchain used was GCC v14.2.0 with RV32IM extensions.

The figure below illustrates the speedup (normalized reciprocal of average elapsed time over 200 iterations) of rv32emu with tiered JIT compilation compared to QEMU. Higher values indicate better performance.

Performance summary:

  • rv32emu with tiered JIT compilation outperforms QEMU v9.0.0 across all benchmarks
  • Significant performance gains in compute-intensive workloads (primes, sha512, emfloat)
  • Strong performance in optimization and cryptography benchmarks (assignment, idea, huffman)
  • Consistent advantages in sorting operations (numeric sort, string sort)
  • The tiered JIT approach effectively balances compilation overhead with code optimization quality

Interpreter comparison

tests/interpreter_bench.py measures the workloads listed above using externally timed process lifetimes. It compares interpreter-only rv32emu with libriscv while explicitly disabling libriscv translation:

python3 tests/interpreter_bench.py \
    --libriscv <path-to-compatible-libriscv-rv32-runner> \
    --libriscv-build-flags '-DRISCV_BRK_MEMORY_SIZE=33554432' \
    --runs 3 --warmup 1 --json build/interpreter-bench.json

Use --libriscv-args when a compatible runner needs runtime options. For example, the ARM64 32 MiB runner is invoked with --libriscv-args '-m 32'.

By default the runner skips bitfield and idea, whose single interpreter runs take many minutes. Pass --workloads all to include them, or a comma-separated list of workload names to select a subset.

The runner verifies an interpreter-only effective configuration, interleaves the two engines, retains raw samples and build provenance in the JSON report, and returns failure unless rv32emu wins every workload by the configured statistical margin (2% by default).

The libriscv runner must support these static newlib guests and accept -n to disable translation. The runner preflight compares normalized guest output and refuses to time a workload if either emulator fails or produces a different result. primes needs more than libriscv's stock 16 MiB brk cap, so build the RV32 newlib runner with -DRISCV_BRK_MEMORY_SIZE=33554432.

Continuous benchmarking

Continuous benchmarking is integrated into GitHub Actions, allowing the committer and reviewer to examine the comment on benchmark comparisons between the pull request commit(s) and the latest commit on the master branch within the conversation. This comment is generated by the benchmark CI and provides an opportunity for discussion before merging.

The results of the benchmark will be rendered on a GitHub page. Check benchmark-action/github-action-benchmark for the reference of benchmark CI workflow.

Modifications to any of the following paths trigger the benchmark CI (see .github/workflows/benchmark.yml for the authoritative list):

  • src/riscv.c
  • src/decode.c
  • src/emulate.c
  • src/rv32_template.c
  • src/rv32_constopt.c
  • src/cache.c
  • src/io.c
  • src/jit.c
  • mk/kconfig.mk
  • mk/toolchain.mk
  • tools/detect-env.py