Tests panick with `misaligned pointer dereference: address must be a multiple of ... but is ...`

[qmonnet]

Tests consistently fail with recent versions of the toolchain, with messages such as:

thread 'test_jit_mbuff' panicked at 'misaligned pointer dereference: address must be a multiple of 0x4 but is 0x7f8038001011', src/jit.rs:100:5

Full error message

     Running tests/misc.rs (target/debug/deps/misc-6f56f75d4e6d43c9)

running 20 tests
thread 'test_jit_mbuff' panicked at 'misaligned pointer dereference: address must be a multiple of 0x4 but is 0x7f8038001011', src/jit.rs:100:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'test_jit_mbuff' panicked at 'panic in a function that cannot unwind', library/core/src/panicking.rs:126:5
stack backtrace:
   0:     0x562d63bf4351 - std::backtrace_rs::backtrace::libunwind::trace::h28494931c73179b2
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:     0x562d63bf4351 - std::backtrace_rs::backtrace::trace_unsynchronized::h9032c52edccf7bd1
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x562d63bf4351 - std::sys_common::backtrace::_print_fmt::hd90562e967f4e4e1
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/std/src/sys_common/backtrace.rs:65:5
   3:     0x562d63bf4351 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h113657117676131e
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x562d63c1995f - core::fmt::rt::Argument::fmt::hd56cdfa11c364505
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/core/src/fmt/rt.rs:138:9
   5:     0x562d63c1995f - core::fmt::write::h24c20284e5d6be9e
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/core/src/fmt/mod.rs:1094:21
   6:     0x562d63bf1b41 - std::io::Write::write_fmt::hbf02c94f0e7342d1
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/std/src/io/mod.rs:1712:15
   7:     0x562d63bf4165 - std::sys_common::backtrace::_print::he85212e2c716c859
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/std/src/sys_common/backtrace.rs:47:5
   8:     0x562d63bf4165 - std::sys_common::backtrace::print::h888aaf3ad10f084e
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/std/src/sys_common/backtrace.rs:34:9
   9:     0x562d63bf5cb7 - std::panicking::default_hook::{{closure}}::hba0edb58dc223add
  10:     0x562d63bf5aa4 - std::panicking::default_hook::h1555b8bada2010d7
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/std/src/panicking.rs:288:9
  11:     0x562d63b6cdd4 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h52e6cd440f597cb6
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/alloc/src/boxed.rs:1999:9
  12:     0x562d63b6cdd4 - test::test_main::{{closure}}::hdcda637653172e31
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/test/src/lib.rs:134:21
  13:     0x562d63bf62c7 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h07438796673f3d04
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/alloc/src/boxed.rs:1999:9
  14:     0x562d63bf62c7 - std::panicking::rust_panic_with_hook::h72a06453beb2cbcb
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/std/src/panicking.rs:695:13
  15:     0x562d63bf6001 - std::panicking::begin_panic_handler::{{closure}}::h0281d6cc05cfd2a4
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/std/src/panicking.rs:580:13
  16:     0x562d63bf4796 - std::sys_common::backtrace::__rust_end_short_backtrace::h1c79565770be27d9
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/std/src/sys_common/backtrace.rs:150:18
  17:     0x562d63bf5db2 - rust_begin_unwind
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/std/src/panicking.rs:578:5
  18:     0x562d63b32b33 - core::panicking::panic_nounwind_fmt::hdb51e63c7c599e80
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/core/src/panicking.rs:96:14
  19:     0x562d63b32bd7 - core::panicking::panic_nounwind::hc4511f17cef1f7da
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/core/src/panicking.rs:126:5
  20:     0x562d63b32d63 - core::panicking::panic_cannot_unwind::hbac7ba3f5f929c6c
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/core/src/panicking.rs:188:5
  21:     0x562d63b8f4e4 - rbpf::jit::emit4::haff8aacbfffbe8b6
                               at /home/runner/work/rbpf/rbpf/src/jit.rs:99:1
  22:     0x562d63b8ff07 - rbpf::jit::emit_alu64_imm32::hd930f0d06f862872
                               at /home/runner/work/rbpf/rbpf/src/jit.rs:211:5
  23:     0x562d63b911af - rbpf::jit::JitMemory::jit_compile::h0a631e7ea59867f2
                               at /home/runner/work/rbpf/rbpf/src/jit.rs:544:9
  24:     0x562d63b93722 - rbpf::jit::compile::hb7b83a17bcef25db
                               at /home/runner/work/rbpf/rbpf/src/jit.rs:1014:5
  25:     0x562d63b842d4 - rbpf::EbpfVmMbuff::jit_compile::h1c675e69584d7361
                               at /home/runner/work/rbpf/rbpf/src/lib.rs:643:25
  26:     0x562d63b34ed0 - misc::test_jit_mbuff::h8eb4bceb4b166566
                               at /home/runner/work/rbpf/rbpf/tests/misc.rs:336:9
  27:     0x562d63b37b57 - misc::test_jit_mbuff::{{closure}}::h55d6770297b8bea5
                               at /home/runner/work/rbpf/rbpf/tests/misc.rs:314:21
  28:     0x562d63b377e5 - core::ops::function::FnOnce::call_once::hafeaeb2bdb95188a
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/core/src/ops/function.rs:250:5
  29:     0x562d63b7232f - core::ops::function::FnOnce::call_once::heba63a2808b93cd2
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/core/src/ops/function.rs:250:5
  30:     0x562d63b7232f - test::__rust_begin_short_backtrace::he9a5c6c59e1b0086
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/test/src/lib.rs:655:18
  31:     0x562d63b3e22c - test::run_test::{{closure}}::h1b8c22c7d438b531
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/test/src/lib.rs:646:30
  32:     0x562d63b3e22c - core::ops::function::FnOnce::call_once{{vtable.shim}}::h36a9083506ac2a12
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/core/src/ops/function.rs:250:5
  33:     0x562d63b711f7 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h11e5ff78080e1397
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/alloc/src/boxed.rs:1985:9
  34:     0x562d63b711f7 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::hfbfbda226f7a7c27
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/core/src/panic/unwind_safe.rs:271:9
  35:     0x562d63b711f7 - std::panicking::try::do_call::h42ac7b4bd35c5b5d
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/std/src/panicking.rs:485:40
  36:     0x562d63b711f7 - std::panicking::try::hc6ad089a257e422e
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/std/src/panicking.rs:449:19
  37:     0x562d63b711f7 - std::panic::catch_unwind::hdf4784c0c06ac0ba
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/std/src/panic.rs:140:14
  38:     0x562d63b711f7 - test::run_test_in_process::hbe86cacc7510eb8b
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/test/src/lib.rs:678:27
  39:     0x562d63b711f7 - test::run_test::run_test_inner::{{closure}}::hbaa8a391d0f22860
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/test/src/lib.rs:572:39
  40:     0x562d63b38c28 - test::run_test::run_test_inner::{{closure}}::hb4232323322f922d
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/test/src/lib.rs:599:37
  41:     0x562d63b38c28 - std::sys_common::backtrace::__rust_begin_short_backtrace::hca81c15946d6f154
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/std/src/sys_common/backtrace.rs:134:18
  42:     0x562d63b3e45b - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::h97d990ba33daf685
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/std/src/thread/mod.rs:529:17
  43:     0x562d63b3e45b - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::he0d612b643c049a4
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/core/src/panic/unwind_safe.rs:271:9
  44:     0x562d63b3e45b - std::panicking::try::do_call::haf308dcb076117ac
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/std/src/panicking.rs:485:40
  45:     0x562d63b3e45b - std::panicking::try::h51fd60d29cf6feee
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/std/src/panicking.rs:449:19
  46:     0x562d63b3e45b - std::panic::catch_unwind::h8e7f9d8c4a32bc2f
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/std/src/panic.rs:140:14
  47:     0x562d63b3e45b - std::thread::Builder::spawn_unchecked_::{{closure}}::h07454567f5413f25
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/std/src/thread/mod.rs:528:30
  48:     0x562d63b3e45b - core::ops::function::FnOnce::call_once{{vtable.shim}}::hfb80b7cf501e16ff
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/core/src/ops/function.rs:250:5
  49:     0x562d63bfa815 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h71c821b130855373
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/alloc/src/boxed.rs:1985:9
  50:     0x562d63bfa815 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h1701cec9acb1061c
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/alloc/src/boxed.rs:1985:9
  51:     0x562d63bfa815 - std::sys::unix::thread::Thread::new::thread_start::h6dd5ef62dde103f1
                               at /rustc/2f6bc5d259e7ab25ddfdd33de53b892770218918/library/std/src/sys/unix/thread.rs:108:17
  52:     0x7f803c6e9b43 - <unknown>
  53:     0x7f803c77ba00 - <unknown>
  54:                0x0 - <unknown>
thread caused non-unwinding panic. aborting.
error: test failed, to rerun pass `--test misc`

This seems to be a consequence of rust-lang/rust#98112.

As an easy “fix”, we can disable debug assertions for the tests, but we risk introducing more bugs. Not sure how to fix it otherwise, given that we do support unaligned memory access in rbpf at this time.

[qmonnet]

Fixed with a339ce8 from #78, but it would be great to find a solution to re-enable the asserts. Leaving the issue open in the meantime.

[saethlin]

It looks to me like the offending code is this:

rbpf/src/jit.rs

Lines 81 to 82 in 196e537

	let mut ptr = $jit.contents.as_ptr().add($jit.offset) as *mut $t;
	*ptr = $data;

This could do ptr.write_unaligned($data);

[saethlin]

I'm not sure what you mean by

we do support unaligned memory access in rbpf at this time

The checks are inserted as part of compiling the Rust code that is the JIT itself, so I don't think the semantics of BPF here are relevant, right? I feel like I must be misunderstanding something here.

[qmonnet]

Hi Ben, thanks a lot for helping!

I feel like I must be misunderstanding something here.

This would be entirely possible, my experience with coding with Rust is limited to this project :)

In this case, let me explain more: in rbpf, we have two ways to execute eBPF code, the interpreter (contained in src/lib.rs) and the JIT-compiler (srs/jit.rs). I tested your suggestion to use ptr.write_unaligned($data); for the JIT-compiler, and it works well indeed for taming the asserts that appear when compiling the JIT itself, and fixes a number of tests 🎉. I'll push a PR with this change [edit: #80, now merged].

Then I've also observed error messages referring to the interpreter, and related to the tests using the interpreter for the eBPF programs, in particular the instructions for loading multiple bytes of memory into registers: ebpf::LD_H_REG/ebpf::LD_W_REG/ebpf::LD_DW_REG, around this line. This is what I was referring to earlier. Maybe there's a similar trick to use here with write_unaligned() or something else to tell Rust that these are possibly not aligned, I need to look more into it.

[qmonnet]

List of failing test (running on the interpreter), after fixing the JIT, for the record:

• README.md:

  • Block starting at line 393

• test/misc.rs:

  • test_vm_block_port
  • test_vm_jit_ldabsh
  • test_vm_jit_ldabsw
  • test_vm_jit_ldabsdw
  • test_vm_jit_ldindw
  • test_vm_jit_ldinddw

• test/ubpf_vm.rs

  • test_vm_ldxdw
  • test_vm_ldxw
  • test_vm_stdw
  • test_vm_stw
  • test_vm_stxdw
  • test_vm_stxw