How to reconstruct your benchmarks?

[Robbepop]

Hi, I am one of the authors of wasmi and got curious about your s1vm which claims to be significantly (approx 2.2 times) faster than wasmi for some fibonacci testcase.
I got seriously interested in your approach because of this claim and tried to reproduce your benchmarks.

My results so far with fib(25):

• s1vm: 9.668629ms
• wasmi_v0: 21.343 ms ~120% slower
• wasmi_v1: 12.966 ms ~30% slower

Note that wasmi_v0 uses an old engine whereas wasmi_v1 is the most recent engine. Both use stack machine based bytecode which naturally is inferior with respect to performance to register machine bytecode used in both Wasm3 and s1vm. This paper indicates roughly 50%-150% performance improvement by switching over to a register machine base bytecode. So if wami_v1 were to use register machine based bytecode it would probably not be ~30% slower but ~35% faster than s1vm. In fact there is an issue open for wasmi to implement register machine based bytecode, however, we are not yet sure which instruction dispatching technique to use. For this we currently research into the most effective dispatching techniques and closure-based approaches so far do not seem to be very promising unfortunately.

To reconstruct my own benchmarks you can checkout wasmi using this branch. And a fork of your s1vm using this link.

For wasmi simply use cargo bench fib_recursive and for the s1vm fork use cargo run --release -- fib.wasm fib_recursive 30.

[Neopallium]

I think my results are from about 2 years ago. I will try to make some time this week to re-run the benchmarks. It is possible that wasmi has improved a lot since then.

My main goal with s1vm was to just try out some ideas for writing a VM in Rust. I have experience with VMs written in C. Since Rust doesn't guarantee tail-calls, the method used by wasm3 can't be used in Rust.

s1vm does more than just using closures for dispatch. If I remember correctly switching to closures didn't make a big improve on their own, but they do allow for some interesting optimizations/specializations.

1. During the bytecode -> closure compiling step, track inputs to each opcode (local, constants, stack, etc..).
2. Most stack values can be traced back to the instruction that output it. Most of these stack writes/reads can be eliminated. This decreases usage of the wasm runtime value stack.

Looking at your closure based tests they don't use the same nesting that s1vm does:

1. They still pop inputs and push results (using the wasm runtime stack).
2. Still increments the PC counter (even for the nested closures which shouldn't need it?).

A virtual stack is built during the compile phase to track the inputs/outputs for each opcode:

enum Input {
  Local(u32), // Normally a function argument.
  Const(StackValue), // Constance value from wasm bytecode.
  Op(OpFunc), // Closure that produces a value (the opcode pushes a stack value).
}

This is code cut from the bytecode -> closure compiler here

      // For binops (add, sub, etc....), get the two `Input` values from the virtual stack (compile time)
      let right = state.pop()?;
      let left = state.pop()?;
      match left {
        /* .... CUT OTHER LEFT matches: local/constant. */
        Input::Op(left_closure) => {
          match right {
            /* ... CUT OTHER RIGHT matches: local/constant. */
            Input::Op(right_closure) => {
              // Push the new closure onto the virtual stack (compile time).
              state.push(Input::Op(Box::new(
             ///////////////// BINOP closure: 
               // Two parameters `state` is the immutable WASM state (functions, code).  `store` is the mutable WASM instance state (memory, runtime stack).
             move |state: &vm::State, store: &mut Store| -> Trap<StackValue> {
                let left = left_closure(state, store)?; // <---------------------- call the closure that produces the left value.
                let right = right_closure(state, store)?; // <------------------- call the closure that produces the right value.

                let res = (left.0 as $type).$op(right.0 as $type); // <------ the opcode's code.

                Ok(StackValue(res as _)) // <------ return the results (native stack, not wasm runtime stack).
              }
           ////////////////////
            )));
              return Ok(());
            },
          }
        },
        /* .... CUT OTHER LEFT matches */
   }

[Robbepop]

Since Rust doesn't guarantee tail-calls, the method used by wasm3 can't be used in Rust.

While Rust does not support guaranteed tail-calls we can still use them when compiling for --release since then Rust will usually tail call optimize code if code is written in a particular way. I did that in the linked research repo and found that the performance of tail-call dispatch is pretty much the same as the simple switch-dispatch. Maybe there will be some gains when proper tail-calls via become keyword will be implemented in Rust but so far it doesn't look like it.

You are right that my closure based approaches for instruction dispatch are not entirely similar to yours, however, the closure-block dispatch still has plenty of nested closure calls very similar to your approach which should be comparable. It is also by far the slowest dispatch. However, it could very well be that I did some mistakes in the implementation.

I will probably try out a fully nested closure based dispatch soon.

Looking at your closure based tests they don't use the same nesting that s1vm does:

• They still pop inputs and push results (using the wasm runtime stack).
• Still increments the PC counter (even for the nested closures which shouldn't need it?).

All the dispatching approaches in the research repo use a register machine as the backbone so no push or pop operations. I wonder what you mean. Maybe there is some confusion.
The tail-call based dispatches do use a pc counter since they are still dispatching over a slice of instructions, i.e. goto based branching.

The main question in the room to me is: Are closures actually worth it for dispatching?
Again: In my research the two fastest dispatching techniques (by far) were the simple switch dispatch and the switch dispatch using tali calls. The closure based dispatch using tail calls (that compares a lot with your approach) was super slow in comparison.
I see what you are trying to do with them and it would certainly be challenging to model those semantics using an enum representing all the different combinations (which I btw also tried out in the research repo under the fused/ct files).

You could go even further with this approach because inputs and outputs can also be global variables or even linear memory accesses. So you could for example define an add operation that has a global variable as its input and output or even some linear memory region, so you'd combine the linear memory store or load into the instruction. I tried this out and it did not bear fruits in my case. The reason is probably that with more and more fused instructions (and their combinations) you spill the instruction cache of the interpreter.

When you benchmark the new v1 engine of wasmi make sure to checkout the master branch and use --package wasmi_v1 since otherwise you refer to the old engine. We do have fibonacci benchmarks in wasmi so you could as well just do: cargo bench fib_recursive/v1.

[Neopallium]

Since Rust doesn't guarantee tail-calls, the method used by wasm3 can't be used in Rust.

While Rust does not support guaranteed tail-calls we can still use them when compiling for --release since then Rust will usually tail call optimize code if code is written in a particular way. I did that in the linked research repo and found that the performance of tail-call dispatch is pretty much the same as the simple switch-dispatch. Maybe there will be some gains when proper tail-calls via become keyword will be implemented in Rust but so far it doesn't look like it.

I am looking forward to Rust adding support for proper tail-calls. The main reason I don't try using tail-calls in Rust is because they only work in release mode, but also because they will not work with async fn as far as I know.

Another reason I had started s1vm was to try building a VM that could support async (suspend/resume) calling. Haven't gotten to that yet with s1vm.

You are right that my closure based approaches for instruction dispatch are not entirely similar to yours, however, the closure-block dispatch still has plenty of nested closure calls very similar to your approach which should be comparable. It is also by far the slowest dispatch. However, it could very well be that I did some mistakes in the implementation.

It also looks like the test code for the different designs are not using the same set of instructions:
https://github.com/Robbepop/interpreter-dispatch-research/blob/dc4c31f1462b4a8ef2061a7661231513dd705e81/src/closure_tail.rs#L93
vs:
https://github.com/Robbepop/interpreter-dispatch-research/blob/dc4c31f1462b4a8ef2061a7661231513dd705e81/src/switch.rs#L131

The closure designs only have a sub in the loop, the switch designs have a mul and two sub instructions.

I will probably try out a fully nested closure based dispatch soon.

Looking at your closure based tests they don't use the same nesting that s1vm does:

• They still pop inputs and push results (using the wasm runtime stack).
• Still increments the PC counter (even for the nested closures which shouldn't need it?).

All the dispatching approaches in the research repo use a register machine as the backbone so no push or pop operations. I wonder what you mean. Maybe there is some confusion. The tail-call based dispatches do use a pc counter since they are still dispatching over a slice of instructions, i.e. goto based branching.

I should have said read/write instead of push/pop. Even with the unsafe code in context.set_reg and context.get_reg those are still memory accesses that the Rust compiler can't eliminate (convert to native registers). While developing s1vm I profiled memory/cache accesses using cachegrind. The closure design used in s1vm is able to eliminate a lot of memory accesses and that is what makes it fast. Also note that there is no unsafe code in s1vm.

It looked like the Instruction handlers are still doing a PC increment here:
https://github.com/Robbepop/interpreter-dispatch-research/blob/dc4c31f1462b4a8ef2061a7661231513dd705e81/src/lib.rs#L71

The main question in the room to me is: Are closures actually worth it for dispatching? Again: In my research the two fastest dispatching techniques (by far) were the simple switch dispatch and the switch dispatch using tali calls. The closure based dispatch using tail calls (that compares a lot with your approach) was super slow in comparison. I see what you are trying to do with them and it would certainly be challenging to model those semantics using an enum representing all the different combinations (which I btw also tried out in the research repo under the fused/ct files).

I has been a few years, but I think that when I had created the first "closure" version it was either the same speed or slower than a loop+switch design. It wasn't until I started to add optimizations/specializations to eliminate the read/writes to the wasm runtime stack that it started to be faster. At one point I had both VM designs in the repo to make it easier to compare.

You could go even further with this approach because inputs and outputs can also be global variables or even linear memory accesses. So you could for example define an add operation that has a global variable as its input and output or even some linear memory region, so you'd combine the linear memory store or load into the instruction. I tried this out and it did not bear fruits in my case. The reason is probably that with more and more fused instructions (and their combinations) you spill the instruction cache of the interpreter.

If you mean the fused folder, it seems to still do memory accesses for the registers.

I think the closure based design will allow for more optimizations:

1. Detect a loop with a counter (conditional branching) and use a closure that handles the loop logic in Rust. JIT compilers do this to make compiled loops fast.
2. Merge common patterns of opcodes (add + mul, sub + mul, etc....) into one closure.

[Neopallium]

I just tried out the master branch of wasmi and tested both v0 and v1.

My original benchmarks were just using the time command for total run time. For wasmi I used the wasmi/examples/invoke.rs example to run the fib.wasm file.

Here is a quickly modified invoke.rs to get it to work with wasmi_v1:

extern crate parity_wasm;
extern crate wasmi;

use std::env::args;

use wasmi::RuntimeValue;
use wasmi_v1 as v1;

fn main() {
    let args: Vec<_> = args().collect();
    if args.len() < 3 {
        println!("Usage: {} <wasm file> <exported func> [<arg>...]", args[0]);
        return;
    }
    let (_, program_args) = args.split_at(3);

    let module = load_module(&args[1]);
    let mut linker = <v1::Linker<()>>::default();
    let mut store = v1::Store::new(module.engine(), ());
    let instance = linker.instantiate(&mut store, &module)
        .unwrap()
        .start(&mut store)
        .unwrap();

    let fib_func = instance.get_export(&store, "fib")
        .and_then(v1::Extern::into_func)
        .unwrap();
    let mut result = [RuntimeValue::I32(0)];

    let args = [
       RuntimeValue::I32(
           program_args[0]
               .parse::<i32>()
               .unwrap_or_else(|_| panic!("Can't parse arg #{} as i32", program_args[0])),
       ),
    ];
    fib_func.call(&mut store, &args, &mut result).expect;
    println!("{:?}", result[0]);
}

fn load_module(file: &str) -> v1::Module {
    let buf = std::fs::read(file).expect("Read file");
    let engine = v1::Engine::default();
    v1::Module::new(&engine, &buf[..]).expect("Failed to load")
}

Run on my AMD Ryzen 9 5900HX laptop (not the same as last time).

wasmi v0:

time ./target/release/examples/invoke ./fib.wasm fib 40
Result: Some(I32(165580141))

real    0m19.313s
user    0m19.302s
sys     0m0.001s

wasmi v1:

time ./target/release/examples/invoke_fib ./fib.wasm fib 40
I32(165580141)

real    0m29.395s
user    0m29.380s
sys     0m0.001s

s1vm:

time ./target/release/main fib.wasm fib 40
165580141

real    0m8.873s
user    0m8.863s
sys     0m0.005s

Cachegrind results. Just the "I refs" (CPU instructions executed) and "D refs" (memory accesses).
wasmi v0:

valgrind --tool=cachegrind  ./target/release/examples/invoke ./fib.wasm fib 30
Some(I32(1346269))
==746407==
==746407== I   refs:      2,169,232,440
==746407==
==746407== D   refs:      1,112,018,749  (704,242,633 rd   + 407,776,116 wr)

wasmi v1:

valgrind --tool=cachegrind ./target/release/examples/invoke_fib ./fib.wasm fib 30
I32(1346269)
==746185==
==746185== I   refs:      2,819,702,274
==746185==
==746185== D   refs:      1,590,154,734  (947,924,143 rd   + 642,230,591 wr)

s1vm:

valgrind --tool=cachegrind ./target/release/main fib.wasm fib 30
1346269
==746282==
==746282== I   refs:      808,427,721
==746282==
==746282== D   refs:      468,791,439  (293,623,787 rd   + 175,167,652 wr)

[Neopallium]

Also this is the fib.wasm file I used: https://github.com/Neopallium/s1vm/blob/master/fib.wasm
Forgot to add it before.

[Neopallium]

Not sure why wasmi v1 was slower then v0 for me. I didn't have the time to fully port the invoke.rs example from v0 to v1, so maybe I made a mistake in how to call into the wasm module.

[Robbepop]

Not sure why wasmi v1 was slower then v0 for me. I didn't have the time to fully port the invoke.rs example from v0 to v1, so maybe I made a mistake in how to call into the wasm module.

It seems you were not using the following Cargo profile for compiling your invoke binary:

[profile.release]
lto = "fat"
codegen-units = 1

In experiments we saw regressions of up to 400% for wasmi_v1 when this profile was not set.
Note thought that with your way of benchmarking you distort the results since you do not only benchmark the execution time but also the entire setup, module compilation, validation, instantiation etc. that is much more complex on wasmi side which is why I was using the cargo bench command on wasmi due to the nature of it not being a POC. :)

For a fair comparison of the actual execution time (which we are mainly talking about) you should wrap the execution only:

let before = std::time::Instant::now();
execute(fib);
let elapsed = before.elapsed();
println!("time = {:?}", elapsed);

[Robbepop]

I am looking forward to Rust adding support for proper tail-calls. The main reason I don't try using tail-calls in Rust is because they only work in release mode, but also because they will not work with async fn as far as I know.

Oh yes! I too wait for tail call support in Rust. However, it is stated by the devs to have low priority. So I guess if we ever want to see it soon(TM) we should become active ourselves and support the development actively. There seem to be an underwhelming amount of people who appreciate the importance of proper tail call support in the Rust community atm.

Another reason I had started s1vm was to try building a VM that could support async (suspend/resume) calling. Haven't gotten to that yet with s1vm.

This sounds interesting. Can you elaborate why your closure based approach is well suited for this? You mean without performance pentalties in the non-resume execution? The old wasmi engine supports resuming execution, too, however, for the new engine we did not implement it for performance reasons.

It also looks like the test code for the different designs are not using the same set of instructions:

Hehe, you are comparing two different test cases there. ;) I just benchmarked the counter_loop tests which is not available for every tested architecture due to me being lazy and I deemed it to be not worth the effort always.

I should have said read/write instead of push/pop. Even with the unsafe code in context.set_reg and context.get_reg those are still memory accesses that the Rust compiler can't eliminate (convert to native registers). While developing s1vm I profiled memory/cache accesses using cachegrind. The closure design used in s1vm is able to eliminate a lot of memory accesses and that is what makes it fast. Also note that there is no unsafe code in s1vm.

This is interesting. I have heard a lot about "moving stuff to registers" by the Wasm3 authors which is probably what you are saying there but I have to admit that I was not able to fully grasp what they meant in their artificial descriptions of how they achieve it. I will have another look at your code again. Will definitely update my research repo once understood.

If you mean the fused folder, it seems to still do memory accesses for the registers.

Yeah sorry for the bad description what I mean by "fused": With fused I mean to not only allow instructions such as add to operate on registers and constants but also on global variables directly or even linear memory regions. So instead of

local.get 0
i32.load 0
i32.const 1
i32.add
local.get 0
i32.store 0

You'd be able to fuse all those instructions into a single one:

add (mem (local 0) 0) (mem (local 0) 0) (i32.const 1)

Which does the entire loading, adding and storing in a single instruction without calling any other function or dispatching into a closure for any of the sub procedures. This is especially possible with closures. We simply not only regard local.get and local.set as well as ty.const as value providers but anything that actually provides a value such as linear memory loads and global variable loads. And everything that updates values such as local.set, global.set or ty.store follows the same pattern.

[Neopallium]

It seems you were not using the following Cargo profile for compiling your invoke binary:

[profile.release]
lto = "fat"
codegen-units = 1

Correct. That helped a lot for the wasmi v1. I re-ran my simple benchmark with all 3 compiled with those options.

wasmi v0:

time  ./target/release/examples/invoke ./fib.wasm fib 40
Result: Some(I32(165580141))

real    0m18.467s
user    0m18.452s
sys     0m0.006s

valgrind --tool=cachegrind  ./target/release/examples/invoke ./fib.wasm fib 30
Some(I32(1346269))
==788933==
==788933== I   refs:      1,918,812,388
==788933==
==788933== D   refs:        916,803,710  (550,763,865 rd   + 366,039,845 wr)

wasmi v1

time ./target/release/examples/invoke_fib ./fib.wasm fib 40
I32(165580141)

real    0m15.008s
user    0m14.998s
sys     0m0.002s

valgrind --tool=cachegrind ./target/release/examples/invoke_fib ./fib.wasm fib 30
I32(1346269)
==788893==
==788893== I   refs:      1,575,732,238
==788893==
==788893== D   refs:        856,430,067  (471,339,573 rd   + 385,090,494 wr)

No improvement for s1vm:

time ./target/release/main fib.wasm fib 40
165580141

real    0m8.969s
user    0m8.959s
sys     0m0.003s

valgrind --tool=cachegrind ./target/release/main fib.wasm fib 30
1346269
==788852==
==788852== I   refs:      811,101,388
==788852==
==788852== D   refs:      466,091,055  (288,233,468 rd   + 177,857,587 wr)

In experiments we saw regressions of up to 400% for wasmi_v1 when this profile was not set. Note thought that with your way of benchmarking you distort the results since you do not only benchmark the execution time but also the entire setup, module compilation, validation, instantiation etc. that is much more complex on wasmi side which is why I was using the cargo bench command on wasmi due to the nature of it not being a POC. :)

For a fair comparison of the actual execution time (which we are mainly talking about) you should wrap the execution only:

let before = std::time::Instant::now();
execute(fib);
let elapsed = before.elapsed();
println!("time = {:?}", elapsed);

For benchmarking VMs I like to include the setup time too. Since some VMs (Java) can have high startup times. Also s1vm should have a more costly setup time (compiling wasm bytecode to closures and related optimizations), then VMs that mostly just load the wasm bytecode and execute that or translate those simple opcodes.

It could be useful to split the start & execution times. One thing that I haven't checked is the memory cost of using closures in s1vm for for complex wasm modules. I think it will be the main disadvantage of the design in s1vm.

[Robbepop]

Thank you for the updated benchmarks! Yet, wasmi_v1 still looks a bit slow imo but at least these results are more realistic.

It could be useful to split the start & execution times. One thing that I haven't checked is the memory cost of using closures in s1vm for for complex wasm modules. I think it will be the main disadvantage of the design in s1vm.

If this ever becomes a problem you can use Bumpalo or another alternative ultra fast bump allocator. Another advantage of bump allocation is that your closure data then resides closely to each other and won't be spilled anywhere.

For benchmarking VMs I like to include the setup time too. Since some VMs (Java) can have high startup times. Also s1vm should have a more costly setup time (compiling wasm bytecode to closures and related optimizations), then VMs that mostly just load the wasm bytecode and execute that or translate those simple opcodes.

I generally agree, but having both separate (setup and execution) would be a very useful information here. Also as I have mentioned earlier it is not entirely fair to compare a full fledged implementation with a POC that lacks most of the functionality (of Wasm).

[Neopallium]

I am looking forward to Rust adding support for proper tail-calls. The main reason I don't try using tail-calls in Rust is because they only work in release mode, but also because they will not work with async fn as far as I know.

Oh yes! I too wait for tail call support in Rust. However, it is stated by the devs to have low priority. So I guess if we ever want to see it soon(TM) we should become active ourselves and support the development actively. There seem to be an underwhelming amount of people who appreciate the importance of proper tail call support in the Rust community atm.

Is there an open issue or RFC about the become keyword?

Another reason I had started s1vm was to try building a VM that could support async (suspend/resume) calling. Haven't gotten to that yet with s1vm.

This sounds interesting. Can you elaborate why your closure based approach is well suited for this? You mean without performance pentalties in the non-resume execution? The old wasmi engine supports resuming execution, too, however, for the new engine we did not implement it for performance reasons.

The closure design might make it a little bit more difficult. A loop+switch design is much easier to suspend and resume.

To support async with the closure based compiler, I think it would be best to add "yield" points to instruction blocks. This would allow most of the closures to be normal non-async functions. Only opcodes that call other functions (internal or external) would need to support async and the compiler could do some static analysis to avoid adding overhead to simple functions that don't have loops or complex branching. There is a wasm module translator that can instrument a wasm module with yield calls (external calls).

https://emscripten.org/docs/porting/asyncify.html
https://web.dev/asyncify/

I should have said read/write instead of push/pop. Even with the unsafe code in context.set_reg and context.get_reg those are still memory accesses that the Rust compiler can't eliminate (convert to native registers). While developing s1vm I profiled memory/cache accesses using cachegrind. The closure design used in s1vm is able to eliminate a lot of memory accesses and that is what makes it fast. Also note that there is no unsafe code in s1vm.

This is interesting. I have heard a lot about "moving stuff to registers" by the Wasm3 authors which is probably what you are saying there but I have to admit that I was not able to fully grasp what they meant in their artificial descriptions of how they achieve it. I will have another look at your code again. Will definitely update my research repo once understood.

It has been a few years since I read about the wasm3 VM. I think they used a fixed set of parameters for each opcode function which allows them to jmp from one instruction directly to the next using tail-calls. This allows the VM to avoid using the native stack and the parameters stay in the same native registers (C ABI calling convention, the first X parameters use native registers, not the native stack). When I started s1vm I wanted to see if I could find a way to emulate that same design in Rust. Maybe with become support it would be possible.

If you mean the fused folder, it seems to still do memory accesses for the registers.

Yeah sorry for the bad description what I mean by "fused": With fused I mean to not only allow instructions such as add to operate on registers and constants but also on global variables directly or even linear memory regions. So instead of

local.get 0
i32.load 0
i32.const 1
i32.add
local.get 0
i32.store 0

You'd be able to fuse all those instructions into a single one:

add (mem (local 0) 0) (mem (local 0) 0) (i32.const 1)

Which does the entire loading, adding and storing in a single instruction without calling any other function or dispatching into a closure for any of the sub procedures. This is especially possible with closures. We simply not only regard local.get and local.set as well as ty.const as value providers but anything that actually provides a value such as linear memory loads and global variable loads. And everything that updates values such as local.set, global.set or ty.store follows the same pattern.

It would be easy to add Memory/Global support to the Input enum in s1vm, but that would only help with the input parameter case. local.set, global.set and store are still compiled to a closure that gets the value from the compiler's virtual stack of Inputs.

[Robbepop]

Is there an open issue or RFC about the become keyword?

The most recent activities are

• Reviving tail-call elimination rust-lang/rfcs#2691
• Proper tail calls rust-lang/rfcs#1888

The closure design might make it a little bit more difficult. A loop+switch design is much easier to suspend and resume.

I have not yet figured out a way that would not introduce additional overhead for cases where you do not need to resume execution. For wasmi resumable execution is a nice-to-have but not a must-have feature, therefore we should not regress performance of the main path of execution.

I think they used a fixed set of parameters for each opcode function which allows them to jmp from one instruction directly to the next using tail-calls. This allows the VM to avoid using the native stack and the parameters stay in the same native registers (C ABI calling convention, the first X parameters use native registers, not the native stack).

These parameters are exactly what I never could wrap my head around in how they are used. Unfortunately Wasm3 implementation is rather cryptic to me with all of its macros.

It would be easy to add Memory/Global support to the Input enum in s1vm, but that would only help with the input parameter case. local.set, global.set and store are still compiled to a closure that gets the value from the compiler's virtual stack of Inputs.

Yeah exactly, this is where the closure approach actually could shine. Just look at this research for an example of how NOT to do it. :D

[Neopallium]

Thank you for the updated benchmarks! Yet, wasmi_v1 still looks a bit slow imo but at least these results are more realistic.

My "simple" benchmarks are not high-quality. I used a high input value for fib to try getting the results to mostly cover the execution time and to help with warm-up.

I think using criterion while developing the VM would help a lot. 2 years ago I didn't have as much experience with Rust, so didn't know about all the tooling that was available.

It could be useful to split the start & execution times. One thing that I haven't checked is the memory cost of using closures in s1vm for for complex wasm modules. I think it will be the main disadvantage of the design in s1vm.

If this ever becomes a problem you can use Bumpalo or another alternative ultra fast bump allocator. Another advantage of bump allocation is that your closure data then resides closely to each other and won't be spilled anywhere.

I agree that keeping the closure data close together should improve things for larger wasm module, but I would wait until there is a test case that shows high memory usage or high cache misses before trying to improve that.

For benchmarking VMs I like to include the setup time too. Since some VMs (Java) can have high startup times. Also s1vm should have a more costly setup time (compiling wasm bytecode to closures and related optimizations), then VMs that mostly just load the wasm bytecode and execute that or translate those simple opcodes.

I generally agree, but having both separate (setup and execution) would be a very useful information here. Also as I have mentioned earlier it is not entirely fair to compare a full fledged implementation with a POC that lacks most of the functionality (of Wasm).

I agree that benchmarking s1vm against a full wasm VM is not fair. Part of the reason I haven't really tried publishing the results, other then just releasing the code. I only made s1vm public as a showcase for my Rust experience.

[Robbepop]

I agree that benchmarking s1vm against a full wasm VM is not fair. Part of the reason I haven't really tried publishing the results, other then just releasing the code. I only made s1vm public as a showcase for my Rust experience.

I am super thankful for you publishing s1vm so I was able to take a look! :)
A goal of mine for wasmi is to somehow manage to pull together a Wasm interpreter that is actually competetive with Wasm3 performance wise. If your proposed interpreter architecture will do the best possible job I will happily implement it. There is just more research needed since reimplementing the internals of wasmi won't be done over a weekend or two.

Some questions that arise from the top of my head about the POC:

• How will multi-value Wasm proposal support look like?

  • Looking at your OpFunc type definition that defines closures that optionally return a single StackValue I wonder how this could be efficiently extended to work with multi-value Wasm proposal that allows a function, block, loop or any kind of instruction to take and return multiple values (which is what we support in wasmi). Returning a Vec<StackValue> will pretty much not scale very well here. Maybe an output parameter will do?

• How will the different control flow constructs work like? E.g. loop and if/else? Currently your POC shows that for basic blocks.

• How does performance compare to a switch/loop interpreter architecture when we use the same access patterns (register machine etc.)

• How much does the performance regress once this architecture supports more than just a handful of instructions? High chance for performance regression due to bloated instruction cache dependency.

• Will tail calls in Rust impact performance positively once they exist?