.pop_front() causing memory corruption (macOS)

[aldanor]

This took me a long while to figure out, but I narrowed it down to just a single line:

println!("{:?}", queue);
queue.pop_front();
println!("{:?}", queue);

The elements are simple Clone/Copy structs; the first println outputs the queue in its normal state, whereas in the second all values are like 4294999990 or 123145302343606. This only happens in a quickcheck-like test suite where the queue is used super intensively, and pushes/pop are done thousands of times (it constantly fails at the same spot though).

I could try running it through valgrind if it helps, not sure how else I could help. Is this a known issue perhaps?

[gnzlbg]

Hi @aldanor thank you for the report and sorry for the slow reply (i'm travelling). Could you share a full snippet that reproduces the issue ? It's ok if it depends on quicktest or one of your libraries, I can use that as a starting point to investigate further.

Also it would be nice to know whether this happens with the latest version of the library or some older version (could you try with master ?). IIUC, this happens on MacOSX right ? Could you try enabling the unix_sysv cargo feature and see if you also reproduce it with that ?

[aldanor]

@gnzlbg I've managed to isolate this. Here's an example that always fails:

    #[test]
    fn test_deque() {
        #[derive(Clone, Copy, Debug, PartialEq)]
        pub struct Foo {
            a: i64,
            b: Option<(bool, i64)>,
        }

        use slice_deque::SliceDeque;
        use rand::{StdRng, SeedableRng, distributions::Uniform};

        let mut rng: StdRng = SeedableRng::seed_from_u64(0);
        let mut deque = SliceDeque::new();

        loop {
            let n = rng.sample(Uniform::new_inclusive(0, 1000));
            for i in 0..n {
                deque.push_front(Foo { a: 42, b: None });
            }
            let n = rng.sample(Uniform::new_inclusive(1, deque.len()));
            for i in 0..n {
                assert_eq!(deque.pop_front(), Some(Foo { a: 42, b: None }));
                if !deque.is_empty() {
                    // this assertion fails (becomes corrupt after pop_front())
                    assert_eq!(unsafe { *deque.get_unchecked(deque.len() - 1) },
                               Foo { a: 42, b: None });
                }
            }
        }
    }

fails like so:

thread 'test_deque' panicked at 'assertion failed: `(left == right)`
  left: `Foo { a: 1048576, b: Some((true, 2)) }`,
 right: `Foo { a: 42, b: None }`', test.rs:251:21

Note that the same example if you replace a struct with int works just fine. Could it have anything to do with alignment?

(Haven't tried master/unix_sysv yet, will do next.)

[aldanor]

@gnzlbg Just checked the master branch and unix_sysv enabled, same outcome.

[gnzlbg]

This is indeed a bug somewhere in SliceDeque, minimal working example:

const C: [i16; 3] = [42; 3];

let mut deque = SliceDeque::new();
for _ in 0..918 {
    deque.push_front(C);
}

for _ in 0..237 {
    assert_eq!(deque.pop_front(), Some(C));
    assert!(!deque.is_empty());
    assert_eq!(*deque.back().unwrap(), C); // fails B != C
}

[aldanor]

Indeed, your example is even more minimal.

Does this only occur on macOS?

[gnzlbg]

Does this only occur on macOS?

I have only tested this on macOS, working on a fix. I suspect the bug is platform independent, but I can't say for sure yet.

[aldanor]

Just checked on 64-bit Linux:

thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `Some([0, 42, 42])`,
 right: `Some([42, 42, 42])`', src/main.rs:14:9

[aldanor]

Another thing to note re: your example, if you replace 3 with any power of two (2/4/8), everything works. If you use an odd number though, like 3/5/7, it fails, deterministically. So something to do with alignment.

[gnzlbg]

It appears that this was caused by a "by 1 off"-error: 70c87a4#diff-b4aea3e418ccdb71239b96952d9cddb6R751

[gnzlbg]

Could you test if that branch solves the problem for you?

[aldanor]

It appears that this was caused by a "by 1 off"-error

That's usually the nastiest type of errors

Could you test if that branch solves the problem for you?

I've checked the branch on 64-bit Linux, seems to work fine so far, I think that fixed it.

[gnzlbg]

I thought about alignment too at first, but at the end the problem had nothing to do with that.

When the SliceDeque had all elements in the second mirrored region except for one element in the first one, a pop_front would remove that element, such that all elements are in the second mirrored region.

For simplicity, the whole implementation assumes that if all elements fit in a single mirrored region, they are always in the first one.

The job of the code with the bug is to make sure that this is the case. It worked in many cases, but it was missing all cases in which the elements of the second memory region lied exactly on the boundary between both regions.

This introduced a memory error that allows safe Rust code to read uninitialized memory if the deque was put in the state described above.