Panic when `cass_future_error_code()` is called from a future callback

[wprzytula]

Minimal reproducer:

    #[test]
    fn reproducer() {
        unsafe extern "C" fn callback(
            future: CassBorrowedSharedPtr<CassFuture, CMut>,
            _data: *mut std::ffi::c_void,
        ) {
            unsafe {
                cass_future_error_code(future.borrow());
            }
        }

        unsafe {
            let cluster_raw = cass_cluster_new();
            let session_raw = cass_session_new();
            let connect_fut =
                cass_session_connect(session_raw.borrow(), cluster_raw.borrow().into_c_const());
            cass_future_set_callback(connect_fut.borrow(), Some(callback), std::ptr::null_mut());

            cass_future_free(connect_fut);
            cass_session_free(session_raw);
            cass_cluster_free(cluster_raw);
        }
    }

Error message:

Tl;dr: it says that the error originates from RUNTIME.block_on() at CassFuture::with_waited_state(), which is called by cass_future_error_message(), in turn called by the tokio executor thread that has finished execution of the CassFuture.

The lengthy message:

thread 'tokio-runtime-worker' panicked at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/runtime/scheduler/multi_thread/mod.rs:86:9:
Cannot start a runtime from within a runtime. This happens because a function (like `block_on`) attempted to block the current thread while the thread is being used to drive asynchronous tasks.
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

thread 'tokio-runtime-worker' panicked at library/core/src/panicking.rs:218:5:
panic in a function that cannot unwind
stack backtrace:
   0:     0x55c135b3c5b2 - std::backtrace_rs::backtrace::libunwind::trace::hc20b48b31ee52608
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9
   1:     0x55c135b3c5b2 - std::backtrace_rs::backtrace::trace_unsynchronized::h5d207cd20f193d88
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14
   2:     0x55c135b3c5b2 - std::sys::backtrace::_print_fmt::h46402c32145e026c
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/sys/backtrace.rs:66:9
   3:     0x55c135b3c5b2 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::hc04c8f544ab24d66
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/sys/backtrace.rs:39:26
   4:     0x55c135b697b3 - core::fmt::rt::Argument::fmt::h2bcd587ea50ac9a4
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/fmt/rt.rs:184:76
   5:     0x55c135b697b3 - core::fmt::write::hfe57b7174b7d8eab
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/fmt/mod.rs:1481:21
   6:     0x55c135b38b53 - std::io::default_write_fmt::h20c7e2d50577e95e
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/io/mod.rs:639:11
   7:     0x55c135b38b53 - std::io::Write::write_fmt::h154385efa8565236
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/io/mod.rs:1914:13
   8:     0x55c135b3c402 - std::sys::backtrace::BacktraceLock::print::h0c8f24e22f5873a8
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/sys/backtrace.rs:42:9
   9:     0x55c135b3d99c - std::panicking::default_hook::{{closure}}::hd07d57e6a602c8e4
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:300:22
  10:     0x55c135b3d79f - std::panicking::default_hook::h63d12f7d95bd91ed
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:327:9
  11:     0x55c1354dc034 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::hc6f711c24942eef9
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/alloc/src/boxed.rs:1980:9
  12:     0x55c1354dc034 - test::test_main::{{closure}}::h5a8d52ff3997df93
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/test/src/lib.rs:136:21
  13:     0x55c135b3e413 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::hfa00b43f9ae13ab5
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/alloc/src/boxed.rs:1980:9
  14:     0x55c135b3e413 - std::panicking::rust_panic_with_hook::h33b18b24045abff4
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:841:13
  15:     0x55c135b3e0b6 - std::panicking::begin_panic_handler::{{closure}}::hf8313cc2fd0126bc
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:699:13
  16:     0x55c135b3cab9 - std::sys::backtrace::__rust_end_short_backtrace::h57fe07c8aea5c98a
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/sys/backtrace.rs:168:18
  17:     0x55c135b3dd7d - __rustc[95feac21a9532783]::rust_begin_unwind
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:697:5
  18:     0x55c13502d9dd - core::panicking::panic_nounwind_fmt::runtime::hb3c0b642eb140891
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/panicking.rs:117:22
  19:     0x55c13502d9dd - core::panicking::panic_nounwind_fmt::h5b1f12ac720fc3e4
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/intrinsics/mod.rs:3241:9
  20:     0x55c13502da72 - core::panicking::panic_nounwind::h9fba2247abed021c
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/panicking.rs:218:5
  21:     0x55c13502dc66 - core::panicking::panic_cannot_unwind::hf5212e3afc7ccaed
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/panicking.rs:323:5
  22:     0x55c135395011 - cass_future_error_code
                               at /home/wojciech/Scylla/cpp-rust-driver/scylla-rust-wrapper/src/future.rs:392:1
  23:     0x55c1352163ae - scylla_cpp_driver::session::tests::reproducer::callback::h4ddb2954157fffa8
                               at /home/wojciech/Scylla/cpp-rust-driver/scylla-rust-wrapper/src/session.rs:1840:17
  24:     0x55c135393d0f - scylla_cpp_driver::future::BoundCallback::invoke::h1eecbcfaea8ced6e
                               at /home/wojciech/Scylla/cpp-rust-driver/scylla-rust-wrapper/src/future.rs:47:13
  25:     0x55c1351fbbec - scylla_cpp_driver::future::CassFuture::new_from_future::{{closure}}::h264f38411518d371
                               at /home/wojciech/Scylla/cpp-rust-driver/scylla-rust-wrapper/src/future.rs:124:17
  26:     0x55c135278e7d - <core::pin::Pin<P> as core::future::future::Future>::poll::h79275ad02e0f5206
                               at /home/wojciech/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/future.rs:124:9
  27:     0x55c13532826c - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::h27ed5ebe0c79a2e6
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/runtime/task/core.rs:331:17
  28:     0x55c135321a7d - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::hf3df54d4160e7d58
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/loom/std/unsafe_cell.rs:16:9
  29:     0x55c135321a7d - tokio::runtime::task::core::Core<T,S>::poll::h14ef2e6935bc525a
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/runtime/task/core.rs:320:13
  30:     0x55c135158c8a - tokio::runtime::task::harness::poll_future::{{closure}}::h3c070f15de53521a
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/runtime/task/harness.rs:532:19
  31:     0x55c13507d6c0 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h9aa27450330fbd33
                               at /home/wojciech/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:272:9
  32:     0x55c135288624 - std::panicking::try::do_call::h78bb6ec3983dc586
                               at /home/wojciech/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:589:40
  33:     0x55c1353d91ab - __rust_try
  34:     0x55c1353c4ce6 - std::panicking::try::hdd4094ceaeaf1b29
                               at /home/wojciech/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:552:19
  35:     0x55c1353c4ce6 - std::panic::catch_unwind::h0d57ce17671a431c
                               at /home/wojciech/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:359:14
  36:     0x55c13514df4f - tokio::runtime::task::harness::poll_future::h643b32cb86a34128
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/runtime/task/harness.rs:520:18
  37:     0x55c135162671 - tokio::runtime::task::harness::Harness<T,S>::poll_inner::h9049694adbdec9c8
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/runtime/task/harness.rs:209:27
  38:     0x55c135173d83 - tokio::runtime::task::harness::Harness<T,S>::poll::h742b23ea56f9f165
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/runtime/task/harness.rs:154:15
  39:     0x55c135348ecb - tokio::runtime::task::raw::poll::hbc1f3a63ff4c56b4
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/runtime/task/raw.rs:271:5
  40:     0x55c135aab097 - tokio::runtime::task::raw::RawTask::poll::h0ed8f9763ee03f45
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/runtime/task/raw.rs:201:18
  41:     0x55c135a86e42 - tokio::runtime::task::LocalNotified<S>::run::h1542d3ab75ffacdc
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/runtime/task/mod.rs:463:9
  42:     0x55c135a7d58d - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}::hb11d82eb82f87645
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/runtime/scheduler/multi_thread/worker.rs:600:13
  43:     0x55c135a7d3e7 - tokio::task::coop::with_budget::h0b32893ba2e2a6f2
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/task/coop/mod.rs:167:5
  44:     0x55c135a7d3e7 - tokio::task::coop::budget::h0c21b749d230d9dc
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/task/coop/mod.rs:133:5
  45:     0x55c135a7d3e7 - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::h0043b63ea423ff9f
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/runtime/scheduler/multi_thread/worker.rs:594:9
  46:     0x55c135a7c956 - tokio::runtime::scheduler::multi_thread::worker::Context::run::h254bd4870b613ac7
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/runtime/scheduler/multi_thread/worker.rs:542:24
  47:     0x55c135a7c499 - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}::h750444e80949d9e4
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/runtime/scheduler/multi_thread/worker.rs:507:21
  48:     0x55c135ac2a90 - tokio::runtime::context::scoped::Scoped<T>::set::h8e6b1655b49ff632
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/runtime/context/scoped.rs:40:9
  49:     0x55c135ab082b - tokio::runtime::context::set_scheduler::{{closure}}::h97fda4c6b4cf3228
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/runtime/context.rs:180:26
  50:     0x55c135a7838b - std::thread::local::LocalKey<T>::try_with::h323cd2c4e3444f9f
                               at /home/wojciech/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:311:12
  51:     0x55c135a77d5f - std::thread::local::LocalKey<T>::with::hcbef6e5000ff9ba8
                               at /home/wojciech/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:275:15
  52:     0x55c135ab0764 - tokio::runtime::context::set_scheduler::h2f16417e02e40426
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/runtime/context.rs:180:9
  53:     0x55c135a7c3a4 - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::h0c16bfe0b864db9b
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/runtime/scheduler/multi_thread/worker.rs:502:9
  54:     0x55c135a76d92 - tokio::runtime::context::runtime::enter_runtime::h4ca386321a8fd1d8
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/runtime/context/runtime.rs:65:16
  55:     0x55c135a7c157 - tokio::runtime::scheduler::multi_thread::worker::run::h9b9337f0f9318cd9
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/runtime/scheduler/multi_thread/worker.rs:494:5
  56:     0x55c135a7be8b - tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}::h559b2e2b7759495b
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/runtime/scheduler/multi_thread/worker.rs:460:45
  57:     0x55c135ab468e - <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll::hd91748c31541bbd2
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/runtime/blocking/task.rs:42:21
  58:     0x55c135ad2d80 - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::h99a282ad574e4279
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/runtime/task/core.rs:331:17
  59:     0x55c135ad25c1 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::hf88ee42eb2d6dc23
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/loom/std/unsafe_cell.rs:16:9
  60:     0x55c135ad25c1 - tokio::runtime::task::core::Core<T,S>::poll::h4b92631a74a770e2
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/runtime/task/core.rs:320:13
  61:     0x55c135a5103e - tokio::runtime::task::harness::poll_future::{{closure}}::h193e7da5b0854c6a
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/runtime/task/harness.rs:532:19
  62:     0x55c135a6ecd1 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::hd071528b03ad710b
                               at /home/wojciech/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:272:9
  63:     0x55c135aa6095 - std::panicking::try::do_call::h44b9bb142b13407c
                               at /home/wojciech/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:589:40
  64:     0x55c135a9701b - __rust_try
  65:     0x55c135a95bd6 - std::panicking::try::hf5bd7c14756c28b9
                               at /home/wojciech/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:552:19
  66:     0x55c135a95bd6 - std::panic::catch_unwind::h788ed0f718c3a8bf
                               at /home/wojciech/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:359:14
  67:     0x55c135a50500 - tokio::runtime::task::harness::poll_future::h65902e919797adc0
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/runtime/task/harness.rs:520:18
  68:     0x55c135a4ccc8 - tokio::runtime::task::harness::Harness<T,S>::poll_inner::h5207ce43414b3d6d
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/runtime/task/harness.rs:209:27
  69:     0x55c135a4c8e7 - tokio::runtime::task::harness::Harness<T,S>::poll::h247dbc191a3f3114
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/runtime/task/harness.rs:154:15
  70:     0x55c135aab40d - tokio::runtime::task::raw::poll::hd25cf8a2d1eb0b38
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/runtime/task/raw.rs:271:5
  71:     0x55c135aab097 - tokio::runtime::task::raw::RawTask::poll::h0ed8f9763ee03f45
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/runtime/task/raw.rs:201:18
  72:     0x55c135a86f07 - tokio::runtime::task::UnownedTask<S>::run::h267ed752e7b884bf
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/runtime/task/mod.rs:500:9
  73:     0x55c135a63357 - tokio::runtime::blocking::pool::Task::run::hd95501ac7fdfb5f0
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/runtime/blocking/pool.rs:161:9
  74:     0x55c135a672bd - tokio::runtime::blocking::pool::Inner::run::h48415e8fc8df3fe8
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/runtime/blocking/pool.rs:511:17
  75:     0x55c135a66fe4 - tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}::h7331fc96cafcad6d
                               at /home/wojciech/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.44.2/src/runtime/blocking/pool.rs:469:13
  76:     0x55c135a94e96 - std::sys::backtrace::__rust_begin_short_backtrace::h4fb9e8fac85c4970
                               at /home/wojciech/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/backtrace.rs:152:18
  77:     0x55c135a53302 - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::hce4e6c0615a7c0b7
                               at /home/wojciech/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:559:17
  78:     0x55c135a6ebe1 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::ha45a1f77e9debf1d
                               at /home/wojciech/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:272:9
  79:     0x55c135aa6050 - std::panicking::try::do_call::h40882ab6a135badc
                               at /home/wojciech/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:589:40
  80:     0x55c135a5c67b - __rust_try
  81:     0x55c135a52ef4 - std::panicking::try::h9c3335aa033c580e
                               at /home/wojciech/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:552:19
  82:     0x55c135a52ef4 - std::panic::catch_unwind::hdcaee9ac179fa91f
                               at /home/wojciech/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:359:14
  83:     0x55c135a52ef4 - std::thread::Builder::spawn_unchecked_::{{closure}}::hf6549cb5a8603dc2
                               at /home/wojciech/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:557:30
  84:     0x55c135a8969f - core::ops::function::FnOnce::call_once{{vtable.shim}}::hbe56cafcd747424d
                               at /home/wojciech/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:250:5
  85:     0x55c135b402db - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hc031c115ad99256b
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/alloc/src/boxed.rs:1966:9
  86:     0x55c135b402db - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::ha14f846e5cecd1a7
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/alloc/src/boxed.rs:1966:9
  87:     0x55c135b402db - std::sys::pal::unix::thread::Thread::new::thread_start::h2717d4f9aca577e6
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/sys/pal/unix/thread.rs:109:17
  88:     0x7fdea78ac907 - start_thread
  89:     0x7fdea7933adc - __GI___clone3
  90:                0x0 - <unknown>
thread caused non-unwinding panic. aborting.

Impact

The dedicated example fails

The callbacks.c example calls cass_future_error_code() in the callback. The example therefore panics. We haven't noticed it, because we haven't been running examples in the CI yet (#305 is still not done; this is a sign that we should prioritize it).

An interesting thing that requires further investigation is that the callbacks.c example is marked as working in the driver's progress spreadsheet. I'm going to run git bisect and see if, and, if so, when, the behaviour got broken.

A larger problem

This uncovers the broader problem of callbacks: at the moment they are invoked directly by the tokio runtime executor threads. This is fundamentally flawed: if the callback performs any kind of blocking, the executor thread will get blocked! There are two categories of blocking that the user can cause:

1. blocking by driver's functions, such as cass_session_execute(), cass_session_free(), or the aforementioned cass_future_error_code().
2. arbitrary external blocking, such as a blocking syscall - read() for example.
   Even if we somehow managed to make driver's API functions detect that they are called from the async context and behave correctly even then, we have no chance to prevent blocking of the second kind. This leads me to a conclusion that callbacks should not be run on tokio executor threads, which means that callbacks should be run on a separate thread.
   The simplest way to do this, and perhaps a good enough one, could be to employ RUNTIME.spawn_blocking() to run callbacks. As tokio keeps a pool of worker threads dedicated to blocking tasks, this could have an acceptable, moderate overhead.

WDYT @Lorak-mmk?

Further investigation results

By running git bisect, I've found that a change I commited made the example fail. The explanation is as follows:

• before the change, a thread that awaits completion of a future would just sleep on a std::sync::Condvar. What's more, it would fall asleep if and only if the result of the future were not yet available. The second part is important, because this is precisely why the example would not fail even on runtimes with 1 worker thread: the thread would not fall asleep upon calling cass_future_error_code() from the future callback, because at that point of time the future has already been completed. Note that if any other not-yet-resolved future were awaited from a callback assuming 1-worker-thread-pool, execution would deadlock.
• after the change, the first thread that awaits completion of a future takes the tokio::task::JoinHandle and calls RUNTIME.block_on(). This was done to support the current_thread runtime (see the detailed description in 793eaaf). Since an executor thread is the first thread to await the future (by calling cass_future_error_code() from the callback context, it attempts to run RUNTIME.block_on(), which is forbidden by tokio and results in the panic.

Conclusion

I believe the proper solution to the problem is to forbid tokio executor threads call driver API functions.
As the first idea to ensure this, I suggest making callbacks always on the dedicated tokio blocking tasks threads, by employing RUNTIME::spawn_blocking(callback).

[Lorak-mmk]

WDYT @Lorak-mmk?

note that if any other not-yet-resolved future were awaited from a callback assuming 1-worker-thread-pool, execution would deadlock.

User of the driver must not use blocking code in callbacks, so this is a non-issue for me.

Supporting cass_future_error_code and other future functions is important, since those are guaranteed to not block when used on a future from callback.
Using spawn_blocking will have an overhead, especially for highly-concurrent app. Remember that this is something on a hot path. What other solutions do we have here?

[wprzytula]

Supporting cass_future_error_code and other future functions is important, since those are guaranteed to not block when used on a future from callback.

You mean that they are guaranteed from the abstract point of view? You're talking about a guarantee that we must give, but at the moment it's not the case?

[Lorak-mmk]

You mean that they are guaranteed from the abstract point of view?

I am saying that cass_future_error_code and other future-awaiting functions should be guaranteed (by us) to be safely callable for already resolved futures in a context that forbids blocking code (such as callback)

[wprzytula]

The approach you are suggesting, namely allowing callbacks to be executed on tokio executor threads, opens a Pandora box of analysing which functions should be supported to be called from callbacks. Now we have to:

1. take a look at virtually any API function,
2. determine if it should be executable from a callback,
3. if so, work hard on allowing this.

As an example, consider cass_session_execute(). Should it be possible to run it from a callback? It seems to be a reasonable use case that someone wants to execute a chain of statements using callbacks. But with the current state of #328, cass_session_execute() calls RUNTIME.block_on() on the session read lock, so the panic occurs. This by the way means that if we want to support such functions from callbacks, we must use a different synchronization method to keep session opened until the requests finish.

Alternatively, we can take an optimistic stance that users won't call API functions from callbacks, except those on already resolved futures (which we surely need to fix due to an explicit use in callbacks.c example), and only fix issues as they arise from real world usage of the driver by users.

[wprzytula]

The approach you are suggesting, namely allowing callbacks to be executed on tokio executor threads, opens a Pandora box of analysing which functions should be supported to be called from callbacks. Now we have to:

1. take a look at virtually any API function,

2. determine if it should be executable from a callback,

3. if so, work hard on allowing this.

As an example, consider cass_session_execute(). Should it be possible to run it from a callback? It seems to be a reasonable use case that someone wants to execute a chain of statements using callbacks. But with the current state of #328, cass_session_execute() calls RUNTIME.block_on() on the session read lock, so the panic occurs. This by the way means that if we want to support such functions from callbacks, we must use a different synchronization method to keep session opened until the requests finish.

Alternatively, we can take an optimistic stance that users won't call API functions from callbacks, except those on already resolved futures (which we surely need to fix due to an explitic use in callbacks.c example), and only fix issues as they arise from real world usage of the driver by users.

@Lorak-mmk I'd appreciate your thoughts on this.