jmh code might be unreproducible

[ittaiz]

One of the travis builds failed on the reproducibility test without apparent relation to the change (I think). Makes me think maybe it just surfaced there:
https://travis-ci.org/bazelbuild/rules_scala/jobs/497218553

< 47c3d415c9e5dbe424c573d08f40da1e  bazel-bin/test/jmh/test_benchmark_compiled_benchmark_lib.jar
< 47c3d415c9e5dbe424c573d08f40da1e  bazel-bin/test/jmh/test_benchmark.runfiles/io_bazel_rules_scala/test/jmh/test_benchmark_compiled_benchmark_lib.jar
1501a1500
> 4f5fa74aa94da701176f042a1e47c841  bazel-bin/test/jmh/test_benchmark_codegen_resources.jar
5393a5393,5394
> d9a26e038170d9e26c593555af311b79  bazel-bin/test/jmh/test_benchmark_compiled_benchmark_lib.jar
> d9a26e038170d9e26c593555af311b79  bazel-bin/test/jmh/test_benchmark.runfiles/io_bazel_rules_scala/test/jmh/test_benchmark_compiled_benchmark_lib.jar
7596d7596
< ef6ee9a44b47c199f47ee962eaf89e7f  bazel-bin/test/jmh/test_benchmark_codegen_resources.jar

@johnynek wdyt?

[ittaiz]

mainly since the PR passed and then I pushed two commits and it broke:
b7979ba
a530f8b
They seem really unrelated

[ittaiz]

indeed passed after rerun without code changing

[johnynek]

yeah, it looks like it is unreproducible. Fortunately, we never depend on a benchmark, we only ever run them, so I guess it is not totally critical to fix, but it would be nice to.

Surprising this pops up now years later.

[ittaiz]

yeah, very surprising.
I agree about the lack of urgency.
maybe we should exclude it from the reproducibility test for now?

[johnynek]

I just noticed that the shas are always the same:

< 47c3d415c9e5dbe424c573d08f40da1e  bazel-bin/test/jmh/test_benchmark_compiled_benchmark_lib.jar
< 47c3d415c9e5dbe424c573d08f40da1e  bazel-bin/test/jmh/test_benchmark.runfiles/io_bazel_rules_scala/test/jmh/test_benchmark_compiled_benchmark_lib.jar
1620a1619
> 4f5fa74aa94da701176f042a1e47c841  bazel-bin/test/jmh/test_benchmark_codegen_resources.jar
5836a5836,5837
> d9a26e038170d9e26c593555af311b79  bazel-bin/test/jmh/test_benchmark_compiled_benchmark_lib.jar
> d9a26e038170d9e26c593555af311b79  bazel-bin/test/jmh/test_benchmark.runfiles/io_bazel_rules_scala/test/jmh/test_benchmark_compiled_benchmark_lib.jar
6954d6954
< ef6ee9a44b47c199f47ee962eaf89e7f  bazel-bin/test/jmh/test_benchmark_codegen_resources.jar

that's the linux build. Note, they are reproducibly wrong. The first hash above matches the one in the original issue, as does the second.

This suggests it should be debuggable.

[benjaminp]

This seems to be because the JMH generator holds the benchmarks list in a hashmap, which results in the benchmark set being emitted in a non-deterministic order.

[johnynek]

Nice find. Would be interesting to see if they would take a PR to make it a sorted set or a list. Did you find a line number somewhere to look at?