Skip to content
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 46 additions & 3 deletions src/engine.rs
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ pub fn run(script_path: PathBuf, input_path: PathBuf) -> Result<FunctionRunResul
let error_stream = wasi_common::pipe::WritePipe::new_in_memory();

let runtime: Duration;
let memory_usage: u64;

{
let mut linker = Linker::new(&engine);
Expand All @@ -43,11 +44,17 @@ pub fn run(script_path: PathBuf, input_path: PathBuf) -> Result<FunctionRunResul
let start = Instant::now();

let instance = linker.instantiate(&mut store, &module)?;

let memory = instance
.get_memory(&mut store, "memory")
.ok_or(anyhow::format_err!("failed to find `memory` export"))?;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a standard? Do we need to document this somewhere?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

@DuncanUszkay1 DuncanUszkay1 May 27, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be clear, I'm talking about the memory string, not the method of measuring the associated memory

I don't see any use of the magic memory name in those docs 🤔 In fact the first link uses mem in some of their examples

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems unlikely the function developer will have seen that before using this tool. We'll either need to

  • (Ideal) Sum up all exported memory stores
  • Document that they need to use the special memory name somewhere

Copy link
Contributor

@DuncanUszkay1 DuncanUszkay1 May 30, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summing up everything they export seems like a reasonable solution, given that we have access to a list of memory export names. I'd rather not document a naming limitation if we could remove the limitation instead.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not a Rust standard, is a standard across compiler infrastructures (LLVM, Binaryen). As Kevin mentioned, you could post-process your Wasm and change the name of the exported memory, not sure what you'd gain by doing so though. Once multiple memories are officially supported by the compiler toolchains I'm pretty sure we'd need to update this. For now this seems a good, standard approach.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also multiple memories is going to introduce some complexities to measure resource usage. I suspect that with multiple memories we'd need to implement the Resource Limiter because not all memories declared in a Wasm module are expected to be exported. So the current approach won't be accurate.

Copy link
Contributor

@DuncanUszkay1 DuncanUszkay1 May 30, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the current approach won't be accurate.

True, but that's true regardless of whether or not we adopt the "memory" name standard since that's a name we're using to query exports. If that's an issue, let's open a separate separate thread about the approach of the PR in general.

On the topic of the "memory" name: In the latest commit Dominic added an approach that sums up all exported memories, which avoids this naming limitation. Is there a reason to re-add the limitation? It seems like a reasonable limitation if it's standard for LLVM and Binaryen, but it seems better to simply eliminate the limitation from my perspective.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, the approach looks good. With the recent changes I don't see the need to reintroduce the "memory" naming standard. I emphasize on multi-memory because when we support it, for example for module linking, the approach here will need to differ considerably. Just want to make sure that this is on your radar.


let module_result = instance
.get_typed_func::<(), (), _>(&mut store, "_start")?
.call(&mut store, ());

runtime = start.elapsed();
memory_usage = memory.size(&store);

match module_result {
Ok(_) => {}
Expand All @@ -71,7 +78,8 @@ pub fn run(script_path: PathBuf, input_path: PathBuf) -> Result<FunctionRunResul
let output: serde_json::Value = serde_json::from_slice(output.as_slice())
.map_err(|e| anyhow!("Couldn't decode Script Output: {}", e))?;

let function_run_result = FunctionRunResult::new(runtime, output, logs.to_string());
let function_run_result =
FunctionRunResult::new(runtime, memory_usage, output, logs.to_string());

Ok(function_run_result)
}
Expand All @@ -81,6 +89,9 @@ mod tests {
use super::*;
use std::path::Path;

// Arbitrary, used to verify that the runner works as expected.
const RUNTIME_THRESHOLD: Duration = Duration::from_millis(5);

#[test]
fn test_runtime_under_threshold() {
let function_run_result = run(
Expand All @@ -89,7 +100,7 @@ mod tests {
)
.unwrap();

assert!(function_run_result.runtime <= Duration::from_millis(5));
assert!(function_run_result.runtime <= RUNTIME_THRESHOLD);
}

#[test]
Expand All @@ -100,6 +111,38 @@ mod tests {
)
.unwrap();

assert!(function_run_result.runtime > Duration::from_millis(5));
assert!(function_run_result.runtime > RUNTIME_THRESHOLD);
}

#[test]
fn test_memory_usage_under_threshold() {
let function_run_result = run(
Path::new("tests/benchmarks/hello_world.wasm").to_path_buf(),
Path::new("tests/benchmarks/hello_world.json").to_path_buf(),
)
.unwrap();

assert_eq!(function_run_result.memory_usage, 17);
}

#[test]
fn test_memory_usage_over_threshold() {
let function_run_result = run(
Path::new("tests/benchmarks/hello_42_pages.wasm").to_path_buf(),
Path::new("tests/benchmarks/hello_42_pages.json").to_path_buf(),
)
.unwrap();

assert_eq!(function_run_result.memory_usage, 42);
}

#[test]
#[should_panic]
fn test_panic() {
run(
Path::new("tests/benchmarks/stack_overflow.wasm").to_path_buf(),
Path::new("tests/benchmarks/stack_overflow.json").to_path_buf(),
)
.unwrap();
}
}
12 changes: 10 additions & 2 deletions src/function_run_result.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,21 @@ use std::{fmt, time::Duration};

pub struct FunctionRunResult {
pub runtime: Duration,
pub memory_usage: u64,
pub logs: String,
pub output: serde_json::Value,
}

impl FunctionRunResult {
pub fn new(runtime: Duration, output: serde_json::Value, logs: String) -> Self {
pub fn new(
runtime: Duration,
memory_usage: u64,
output: serde_json::Value,
logs: String,
) -> Self {
FunctionRunResult {
runtime,
memory_usage,
output,
logs,
}
Expand All @@ -22,7 +29,8 @@ impl fmt::Display for FunctionRunResult {
let title = " Benchmark Results ".black().on_bright_green();
write!(f, "{}\n\n", title)?;

writeln!(f, "Runtime: {:?}\n", self.runtime)?;
writeln!(f, "Runtime: {:?}", self.runtime)?;
writeln!(f, "Memory Usage: {}KB\n", self.memory_usage * 64)?;

writeln!(
f,
Expand Down
11 changes: 11 additions & 0 deletions tests/benchmarks/hello_42_pages.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{
"input": {
"context": {
"suffix": "From Core!"
}
},
"configuration": {
"message": "From Core!"
},
"extension_point": "hello_world"
}
Binary file added tests/benchmarks/hello_42_pages.wasm
Binary file not shown.
11 changes: 11 additions & 0 deletions tests/benchmarks/stack_overflow.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{
"input": {
"context": {
"suffix": "From Core!"
}
},
"configuration": {
"message": "From Core!"
},
"extension_point": "hello_world"
}
Binary file added tests/benchmarks/stack_overflow.wasm
Binary file not shown.