Add benchmarks.

[glennpratt]

I've been creating benchmarks for some exercises in the benches dir and testing with cargo bench.

Using test::Bencher, they would only work on nightly, but that could be documented easily enough and cargo test would still work fine on stable.

This Go exercise is a pretty compelling reason to have that. Is your parallel implementation even faster than a simple sequential implementation? Find out:

https://github.com/exercism/xgo/blob/master/exercises/parallel-letter-frequency/parallel_letter_frequency_test.go#L48-L58

Would a PR adding a benchmark to that example to start be accepted?

[ijanos]

Hi, thank you for bringing up this topic!

I did not play with the bencher yet, mainly due to the nightly requirement but I am in favour of this. I think we would accept a PR if the purpose of the bench is clearly documented as not part of the main exercise and if it does not cause any trouble for people using stable rust.

I am also curious how hard would be to modify the CI to compile the bench code only on nightly. I would not like to commit any code that is not checked by the CI framework.

[petertseng]

I am curious to see the result. I don't see a reason why not.

Trying to see what we can do to make Travis run it. Worse case we can rustc --version | grep nightly. We may be able to make use of some variable at https://docs.travis-ci.com/user/environment-variables/ though. I wonder what the value of TRAVIS_RUST_VERSION is for us.

[glennpratt]

Thanks! I'll try to give this a shot tonight. I'm pretty familiar with Travis and imagine we can make it work.

[glennpratt]

This is the rabbit hole that led me to create this issue. It's hard to create a parallel solution that's faster than for loops (even on large inputs). The example benches are the example from this repo.

$ cargo bench
   Compiling fnv v1.0.5
   Compiling parallel-letter-frequency v0.0.0 (file:///Users/glenn.pratt/Code/exercism/rust/parallel-letter-frequency)
    Finished release [optimized] target(s) in 5.42 secs
     Running target/release/deps/bench-6c0440791c71869e

running 8 tests
test bench_1                      ... bench:     130,838 ns/iter (+/- 8,825)
test bench_3                      ... bench:     118,764 ns/iter (+/- 32,890)
test bench_example_1              ... bench:      78,249 ns/iter (+/- 4,946)
test bench_example_3              ... bench:      78,486 ns/iter (+/- 4,229)
test bench_serial                 ... bench:      37,390 ns/iter (+/- 7,545)
test bench_serial_fnv             ... bench:      36,986 ns/iter (+/- 6,878)
test bench_serial_fnv_no_to_lower ... bench:      16,505 ns/iter (+/- 3,325)
test bench_serial_no_to_lower     ... bench:      17,111 ns/iter (+/- 3,405)

test result: ok. 0 passed; 0 failed; 0 ignored; 8 measured

     Running target/release/deps/parallel_letter_frequency-43484563747266a8

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured

[glennpratt]

The Go concurrent example is actually faster than sequential (though slower than serial Rust; it's not filtering for alpha and normalizing like the Rust exercise tests require so not apples to apples):

$ go test -bench .
BenchmarkSequentialFrequency-8   	   30000	     45975 ns/op
BenchmarkConcurrentFrequency-8   	   50000	     36453 ns/op
PASS
ok  	_/Users/glenn.pratt/Code/exercism/go/parallel-letter-frequency	9.050s

[glennpratt]

I opened #245 with a benchmark for parallel-letter-frequency.