Implement non-deterministc mode #683

Aaron1011 · 2019-04-07T22:47:22Z

Part of #653

This allows us to properly implement getrandom(),
which unlocks the default HashMap type (e.g. HashMap<K, V>)
with RandomState)

This commit adds a new '-Zmiri-seed=' option. When present,
this option takes a 64-bit hex value, which is used as the seed
to an internal PRNG. This PRNG is used to implement the 'getrandom()'
syscall.

When '-Zmiri-seed' is not passed, 'getrandom()' will be disabled.

Part of rust-lang#653 This allows us to properly implement getrandom(), which unlocks the default HashMap type (e.g. HashMap<K, V>) with RandomState) This commit adds a new '-Zmiri-seed=<seed>' option. When present, this option takes a 64-bit hex value, which is used as the seed to an internal PRNG. This PRNG is used to implement the 'getrandom()' syscall. When '-Zmiri-seed' is not passed, 'getrandom()' will be disabled.

oli-obk

Just some nits. This is awesome!

src/fn_call.rs

tests/compile-fail/getrandom.rs

src/bin/miri.rs

src/fn_call.rs

src/lib.rs

tests/run-pass/hashmap.rs

RalfJung · 2019-04-08T08:40:42Z

This is pretty cool!

To make the test suite pass with your changes to the HashMap tests, you'd have to also implement this for macOS and Windows.

Windows says

error[E0080]: constant evaluation error: can't call foreign function: SystemFunction036
 --> C:/Users/appveyor/.rustup/toolchains/master/lib/rustlib/src/rust/src/libstd/sys/windows/rand.rs:8:9
  |
8 | /         c::RtlGenRandom(&mut v as *mut _ as *mut u8,
9 | |                         mem::size_of_val(&v) as c::ULONG)
  | |_________________________________________________________^ can't call foreign function: SystemFunction036
  |
  = note: inside call to `std::sys::windows::rand::hashmap_random_keys` at C:/Users/appveyor/.rustup/toolchains/master/lib/rustlib/src/rust/src/libstd/collections/hash/map.rs:3181:23

and macOS says

error[E0080]: constant evaluation error: can't call foreign function: open
   --> /Users/travis/.rustup/toolchains/master/lib/rustlib/src/rust/src/libstd/sys/unix/fs.rs:467:13
    |
467 |             open64(path.as_ptr(), flags, opts.mode as c_int)
    |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ can't call foreign function: open
    |
    = note: inside call to closure at /Users/travis/.rustup/toolchains/master/lib/rustlib/src/rust/src/libstd/sys/unix/mod.rs:143:19

Alternatively, you could make the affected tests linux-only. Then we should open two issues to track the missing support for Windows and macOS.

Implementing random number generation on OS X will require special-casing the 'openat' system call to special-case reading from /dev/urandom

Aaron1011 · 2019-04-09T02:45:19Z

I've implemented random number generation for Windows.

For now, I've left the test disabled for OS X. Implementing random number generation on OS X will be trickier, since we'll need to handle attempted reads from /dev/urandom (also, I don't have a Mac to test on).

Apparently 'ignore-' doesn't work with compiletest_rs

Aaron1011 · 2019-04-09T05:42:22Z

@oli-obk @RalfJung I've made the changes you requested

RalfJung · 2019-04-09T08:11:39Z

tests/run-pass/hashmap.rs

+        let map_normal: HashMap<i32, i32> = HashMap::new();
+        test_map(map_normal);
+    } else {
+        test_map(_map);


Can you move the let _map into this conditional branch to avoid the unused variable on other platforms? Then you can also avoid the _ in the name.

I wanted to test creation of both types of maps on all platforms, even though only one has the full set of tests run on it.

TBH I don't see the point, this code just looks weird. Why test creation on both but not the rest of the test?

The purpose of this is mostly to test Miri, not to test the HashMap. For that, HashBrown actually runs its test suite in Miri. I don't see a good reason to use multiple different hash functions here.

My intention was to ensure that Miri's test coverage didn't regress (e.g. ensure that we catch any regressions with creating a HashMap with a custom hasher).

I'll clean it up.

src/fn_call.rs

Aaron1011 · 2019-04-09T14:20:03Z

@RalfJung: I've addressed your comments

src/fn_call.rs

oli-obk · 2019-04-09T17:12:22Z

just some nits then this looks ready to go to me

Aaron1011 · 2019-04-09T19:49:55Z

@oli-obk Fixed

src/fn_call.rs

Co-Authored-By: Aaron1011 <[email protected]>

Aaron1011 · 2019-04-10T03:37:07Z

@RalfJung Fixed

RalfJung · 2019-04-10T07:20:29Z

Awesome, thanks for bearing with our nitpicking :)

Aaron1011 force-pushed the feature/random branch from cb1a3cc to 2e28498 Compare April 7, 2019 22:58

Aaron1011 force-pushed the feature/random branch from 2e28498 to 7323957 Compare April 7, 2019 23:26

Aaron1011 added 3 commits April 7, 2019 19:30

Fix benchmark

312f938

Use getrandom() syscall number from libc

dddeda7

Use raw syscall numbers

808b149

oli-obk requested changes Apr 8, 2019

View reviewed changes

src/fn_call.rs Show resolved Hide resolved

tests/compile-fail/getrandom.rs Outdated Show resolved Hide resolved

RalfJung reviewed Apr 8, 2019

View reviewed changes

src/bin/miri.rs Outdated Show resolved Hide resolved

RalfJung reviewed Apr 8, 2019

View reviewed changes

src/fn_call.rs Outdated Show resolved Hide resolved

RalfJung reviewed Apr 8, 2019

View reviewed changes

src/lib.rs Outdated Show resolved Hide resolved

RalfJung reviewed Apr 8, 2019

View reviewed changes

tests/run-pass/hashmap.rs Outdated Show resolved Hide resolved

Aaron1011 added 6 commits April 8, 2019 21:36

Simplify cast using as_mut_ptr()

5530d29

Cleanup argument parsing

6b0440e

Only run test with default hasher

b120e8b

Refactor random number generation

6d3e93c

Disable normal HashMap test on OS X

858e82b

Implementing random number generation on OS X will require special-casing the 'openat' system call to special-case reading from /dev/urandom

Interpret system call numbers relative to target architecture

5f99764

Aaron1011 added 4 commits April 8, 2019 22:48

OS X is "macos", not "darwin"

1748744

Only run 'getrandom' test on Linux

5e07ff6

Fix 'RtlGenRandom' argument slots

6b4c5b8

Use 'ignore-' instead of 'only-'

9243680

Apparently 'ignore-' doesn't work with compiletest_rs

oli-obk approved these changes Apr 9, 2019

View reviewed changes

RalfJung mentioned this pull request Apr 9, 2019

Can't call foreign function: clock_gettime #641

Closed

RalfJung reviewed Apr 9, 2019

View reviewed changes

src/fn_call.rs Outdated Show resolved Hide resolved

RalfJung reviewed Apr 9, 2019

View reviewed changes

src/fn_call.rs Outdated Show resolved Hide resolved

Improve deterministic mode error message

22044c8

Fix compile-fail error message

ae8e7f6

Retrieve SYS_getrandom from libc using const-eval

c6e0d09

oli-obk reviewed Apr 9, 2019

View reviewed changes

src/fn_call.rs Outdated Show resolved Hide resolved

oli-obk reviewed Apr 9, 2019

View reviewed changes

src/fn_call.rs Outdated Show resolved Hide resolved

oli-obk reviewed Apr 9, 2019

View reviewed changes

src/fn_call.rs Outdated Show resolved Hide resolved

Some final cleanup

0837d63

Merge branch 'master' into feature/random

2eb2c8d

RalfJung reviewed Apr 9, 2019

View reviewed changes

src/fn_call.rs Outdated Show resolved Hide resolved

RalfJung and others added 2 commits April 9, 2019 23:33

Fix typo

48b22b8

Co-Authored-By: Aaron1011 <[email protected]>

Don't create HashMap<K, V> when not needed

a8763f3

RalfJung merged commit 2dc6e8b into rust-lang:master Apr 10, 2019

Aaron1011 mentioned this pull request Apr 11, 2019

Implement libstd HashMap seeding on OS X #686

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement non-deterministc mode #683

Implement non-deterministc mode #683

Aaron1011 commented Apr 7, 2019

oli-obk left a comment

RalfJung commented Apr 8, 2019 •

edited

Loading

Aaron1011 commented Apr 9, 2019

Aaron1011 commented Apr 9, 2019

RalfJung Apr 9, 2019

Aaron1011 Apr 9, 2019

RalfJung Apr 9, 2019

Aaron1011 Apr 10, 2019

Aaron1011 commented Apr 9, 2019

oli-obk commented Apr 9, 2019

Aaron1011 commented Apr 9, 2019

Aaron1011 commented Apr 10, 2019

RalfJung commented Apr 10, 2019

Implement non-deterministc mode #683

Implement non-deterministc mode #683

Conversation

Aaron1011 commented Apr 7, 2019

oli-obk left a comment

Choose a reason for hiding this comment

RalfJung commented Apr 8, 2019 • edited Loading

Aaron1011 commented Apr 9, 2019

Aaron1011 commented Apr 9, 2019

RalfJung Apr 9, 2019

Choose a reason for hiding this comment

Aaron1011 Apr 9, 2019

Choose a reason for hiding this comment

RalfJung Apr 9, 2019

Choose a reason for hiding this comment

Aaron1011 Apr 10, 2019

Choose a reason for hiding this comment

Aaron1011 commented Apr 9, 2019

oli-obk commented Apr 9, 2019

Aaron1011 commented Apr 9, 2019

Aaron1011 commented Apr 10, 2019

RalfJung commented Apr 10, 2019

RalfJung commented Apr 8, 2019 •

edited

Loading