"Atomics.wait cannot be called in this context" when using sync API

[aran]

Describe the bug

I have an object "State".

It is imported from another crate via pub use automerge::sync::State;

I have:

#[frb(external)]
impl State {
    pub fn new() -> Self {}
}

In my flutter_rust_bridge.yaml, I have full_dep: false and default_dart_async: false.

My Dart code calls State() to build a new instance of this type, in flutter web, not using wasm.

Intermittently, this causes:

══╡ EXCEPTION CAUGHT BY MAIN ZONE ╞═════════════════════════════════════════════════════════════════
js_primitives.dart:28 The following PanicException was thrown:
js_primitives.dart:28 PanicException(EXECUTE_SYNC_ABORT RuntimeError: Atomics.wait cannot be called in this context
js_primitives.dart:28 pkg/rust_lib_bg.wasm 1:2006906                                                     1412
js_primitives.dart:28 pkg/rust_lib_bg.wasm 1:2255675                                                     2494
js_primitives.dart:28 pkg/rust_lib.js 230:26
js_primitives.dart:28 frb_pde_ffi_dispatcher_sync
js_primitives.dart:28 packages/flutter_rust_bridge/src/generalized_frb_rust_binding/_web.dart 37:12
js_primitives.dart:28 pdeFfiDispatcherSync
js_primitives.dart:28 packages/flutter_rust_bridge/src/codec/pde.dart 24:37                              pdeCallFfi
js_primitives.dart:28 packages/plans/src/rust/frb_generated.dart 1964:16                                 <fn>
js_primitives.dart:28 packages/flutter_rust_bridge/src/main_components/handler.dart 25:24                executeSync
...

Debug version:

══╡ EXCEPTION CAUGHT BY MAIN ZONE ╞═════════════════════════════════════════════════════════════════
js_primitives.dart:28 The following PanicException was thrown:
js_primitives.dart:28 PanicException(EXECUTE_SYNC_ABORT RuntimeError: Atomics.wait cannot be called in this context
js_primitives.dart:28 pkg/rust_lib_bg.wasm 1:11453490
js_primitives.dart:28 core::core_arch::wasm32::atomic::memory_atomic_wait32::h7ba7f7f3997c71f5
js_primitives.dart:28 pkg/rust_lib_bg.wasm 1:9062326
js_primitives.dart:28 std::sys::pal::wasm::futex::futex_wait::h5915d6aa9b442a50
js_primitives.dart:28 pkg/rust_lib_bg.wasm 1:4834251
js_primitives.dart:28 std::sys::sync::rwlock::futex::RwLock::write_contended::hcd3dbec2984b26da
js_primitives.dart:28 pkg/rust_lib_bg.wasm 1:8555821
js_primitives.dart:28 std::sys::sync::rwlock::futex::RwLock::write::hc6fd5c80124621de
js_primitives.dart:28 pkg/rust_lib_bg.wasm 1:11160872
js_primitives.dart:28 std::sync::poison::rwlock::RwLock<T>::write::h016e6494d2e7974c
js_primitives.dart:28 pkg/rust_lib_bg.wasm 1:5795377
js_primitives.dart:28 <rust_lib::frb_generated::MoiArc<T> as
js_primitives.dart:28 flutter_rust_bridge::generalized_arc::base_arc::BaseArc<T>>::new::h3e45ab7c5185fe80
js_primitives.dart:28 pkg/rust_lib_bg.wasm 1:10415075
js_primitives.dart:28 flutter_rust_bridge::rust_opaque::utils::<impl
js_primitives.dart:28 flutter_rust_bridge::rust_opaque::RustOpaqueBase<T,A>>::new::h0dcbf3cdb299c485
js_primitives.dart:28 pkg/rust_lib_bg.wasm 1:9476927
js_primitives.dart:28 flutter_rust_bridge::rust_auto_opaque::dart2rust_implicit::rust_auto_opaque_encode::hf6cf0430e2846431
js_primitives.dart:28 pkg/rust_lib_bg.wasm 1:10886083
js_primitives.dart:28 <automerge::sync::state::State as rust_lib::frb_generated::SseEncode>::sse_encode::h3f237cb69b691d39
js_primitives.dart:28 pkg/rust_lib_bg.wasm 1:10239673
js_primitives.dart:28 rust_lib::frb_generated::transform_result_sse::{{closure}}::h5cc1b8f474961fad
js_primitives.dart:28 pkg/rust_lib_bg.wasm 1:5907148
js_primitives.dart:28 flutter_rust_bridge::codec::sse::SseCodec::encode::h66f91c1fa466ce4e
js_primitives.dart:28 pkg/rust_lib_bg.wasm 1:6534170
js_primitives.dart:28 rust_lib::frb_generated::transform_result_sse::h3ebd1684a310bc3b
js_primitives.dart:28 pkg/rust_lib_bg.wasm 1:8464227
js_primitives.dart:28 rust_lib::frb_generated::wire__crate__api__plans__State_new_impl::{{closure}}::he770dcd6ddf5274f

This feels like the revenge of #1910. We had ported the app at that time to entirely use #[frb(sync)] to avoid that issue. I don't know why we avoided problems for so long, or what's changed to bring this back.

When I see this error, I also see other errors:

panicked at front/getplans_io/rust/src/frb_generated.rs:35:1:
called `Option::unwrap()` on a `None` value

Stack:

Error
    at http://localhost:8000/pkg/rust_lib.js:917:25
    at logError (http://localhost:8000/pkg/rust_lib.js:12:22)
    at imports.wbg.__wbg_new_8a6f238a6ece86ea (http://localhost:8000/pkg/rust_lib.js:916:70)
    at rust_lib.wasm.__wbg_new_8a6f238a6ece86ea externref shim (http://localhost:8000/pkg/rust_lib_bg.wasm:wasm-function[47720]:0xb1fbd9)
    at rust_lib.wasm.console_error_panic_hook::Error::new::h2de395ce7e285184 (http://localhost:8000/pkg/rust_lib_bg.wasm:wasm-function[28294]:0xa06a2d)
    at rust_lib.wasm.console_error_panic_hook::hook_impl::h50c46261fc523de5 (http://localhost:8000/pkg/rust_lib_bg.wasm:wasm-function[6118]:0x648ecd)
    at rust_lib.wasm.console_error_panic_hook::hook::h750e8fbbdd35ac89 (http://localhost:8000/pkg/rust_lib_bg.wasm:wasm-function[41643]:0xae3119)
    at rust_lib.wasm.core::ops::function::Fn::call::hff2d0fa6f6660664 (http://localhost:8000/pkg/rust_lib_bg.wasm:wasm-function[34747]:0xa8048e)
    at rust_lib.wasm.<alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h4a354516481437c9 (http://localhost:8000/pkg/rust_lib_bg.wasm:wasm-function[27992]:0xa00259)
    at rust_lib.wasm.flutter_rust_bridge::misc::panic_backtrace::PanicBacktrace::setup::{{closure}}::h09ff7aae7a3c6794 (http://localhost:8000/pkg/rust_lib_bg.wasm:wasm-function[10614]:0x792926)
    at rust_lib.wasm.<alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h4a354516481437c9 (http://localhost:8000/pkg/rust_lib_bg.wasm:wasm-function[27992]:0xa00259)
    at rust_lib.wasm.std::panicking::rust_panic_with_hook::hd0b636188372b74a (http://localhost:8000/pkg/rust_lib_bg.wasm:wasm-function[391]:0x1bc608)
    at rust_lib.wasm.std::panicking::begin_panic_handler::{{closure}}::hd07ba03491f10a21 (http://localhost:8000/pkg/rust_lib_bg.wasm:wasm-function[5741]:0x623030)
    at rust_lib.wasm.std::sys::backtrace::__rust_end_short_backtrace::h8d41ea3a4e160e56 (http://localhost:8000/pkg/rust_lib_bg.wasm:wasm-function[46119]:0xb14f8f)
    at rust_lib.wasm.rust_begin_unwind (http://localhost:8000/pkg/rust_lib_bg.wasm:wasm-function[20108]:0x932bae)
    at rust_lib.wasm.core::panicking::panic_fmt::h5a860a59faaaca34 (http://localhost:8000/pkg/rust_lib_bg.wasm:wasm-function[22028]:0x96beac)
    at rust_lib.wasm.core::panicking::panic::h0ea6a96cb14afb0b (http://localhost:8000/pkg/rust_lib_bg.wasm:wasm-function[22339]:0x974eed)
    at rust_lib.wasm.core::option::unwrap_failed::h3b44683865251404 (http://localhost:8000/pkg/rust_lib_bg.wasm:wasm-function[47454]:0xb1e48c)
    at rust_lib.wasm.rust_lib::frb_generated::MoiArc<T>::decrement_strong_count_raw::h829ea9bdf7fa6034 (http://localhost:8000/pkg/rust_lib_bg.wasm:wasm-function[7797]:0x6d6738)
    at rust_lib.wasm.rust_lib::frb_generated::MoiArc<T>::decrement_strong_count::hb36fbf9ec82a7ebd (http://localhost:8000/pkg/rust_lib_bg.wasm:wasm-function[8318]:0x6fd85b)
    at rust_lib.wasm.rust_lib::frb_generated::web::rust_arc_decrement_strong_count_RustOpaque_flutter_rust_bridgefor_generatedRustAutoOpaqueInnerText::h6abe39308beeaa5e (http://localhost:8000/pkg/rust_lib_bg.wasm:wasm-function[40816]:0xad8617)
    at rust_lib.wasm.rust_arc_decrement_strong_count_RustOpaque_flutter_rust_bridgefor_generatedRustAutoOpaqueInnerText (http://localhost:8000/pkg/rust_lib_bg.wasm:wasm-function[29503]:0xa1fd9c)
    at __exports.rust_arc_decrement_strong_count_RustOpaque_flutter_rust_bridgefor_generatedRustAutoOpaqueInnerText (http://localhost:8000/pkg/rust_lib.js:582:14)
    at frb_generated$46web.RustLibWire.fromExternalLibrary.rust_arc_decrement_strong_count_RustOpaque_flutter_rust_bridgefor_generatedRustAutoOpaqueInnerText (http://localhost:8000/packages/plans/src/rust/frb_generated.web.dart.lib.js:1534:145)

Maybe it's a clue, but it's worth mentioning that my project does not build with full_dep: true. I'm unable to find any docs about full_dep, but it appears this switches the codec to CST/DCO and in that environment many of my objects aren't handled automatically. I am not sure what I might be doing wrong that would result in CST/DCO errors, and if that is a clue for why things break at runtime under full_dep: false.

Steps to reproduce

No minimal repro yet.

Logs

No logs yet.

Expected behavior

No response

Generated binding code

OS

No response

Version of flutter_rust_bridge_codegen

No response

Flutter info

Version of clang++

No response

Additional context

No response

[fzyzcjy]

Pretty weird...

We had ported the app at that time to entirely use #[frb(sync)] to avoid that issue.

Hmm, is it possible there are some new non-frb-sync code?

[aran]

I don't think so. Is there a way that non-sync code can sneak in through default_dart_async: false?

From what I can tell about the stack trace, it appears to be that my object is being wrapped as a <Arc<RwLock<...>> and either this was always happening and unsafe and by good luck now it goes boom, or something changed to make it unsafe.

The usage is:

pub use automerge::sync::State;
...

#[frb(external)]
impl State {
    pub fn new() -> Self {}
}

...

pub fn generate_sync_message(doc: &mut Automerge, sync_state: &mut State) -> Option<Vec<u8>> {
    doc.generate_sync_message(sync_state)
        .map(|msg| msg.encode())
}

pub fn receive_sync_message(
    doc: &mut Automerge,
    sync_state: &mut State,
    msg: Vec<u8>,
) -> Result<()> {
    let msg = Message::decode(&msg)?;
    doc.receive_sync_message(sync_state, msg)?;
    Ok(())
}

(Both of those functions are sync, by the global configuration setting.)

Is there some way I can tell FRB to avoid the RwLock to avoid this, taking advantage of the fact that all my methods are sync?

[fzyzcjy]

my object is being wrapped as a <Arc<RwLock<...>>

opaque objects are implemented like that

The usage is:

Oh do you mean this always gives an error? or does it happen only once in a while? (if it always error then we should be able to debug it more easily)

If so, could you please make a minimal reproducible sample? Then I would be happy to debug it.

Is there some way I can tell FRB to avoid the RwLock to avoid this, taking advantage of the fact that all my methods are sync?

Not yet indeed... But anyway, since everything is sync, I am wondering why this error can happen...

Quickly checked the error stack, a quick guess: when calling new, is there another function that is in progress?

Btw do you have #[frb(...)], or pub async fn etc

[aran]

Thanks for your help. The error is not consistent. I don't know how there could be another function in progress, because every FRB function is sync, we do not use any 'async fn' on Rust side and afaik in Flutter web the entire event loop runs on one main browser thread. I'm going to be working on shrinking the reproducible case today.

[aran]

(I am still trying to get a repro smaller than refreshing the page while running the whole app.)

Is there any way I could configure my own RustAutoOpaque implementation? Or, is there any way to avoid RustAutoOpaque? For example, I might try Rc<RefCell<T>>. From first principles, it seems dangerous to use Arc<RwLock<T>> when compiling to web, because that code uses atomic waits in its implementation.

I haven't identified what could cause rust_arc_decrement_strong_count to unwrap a None. If not a bug in FRB, what kind of mistake in my app code could cause that?

[aran]

Also managed to hit a "memory access out of bounds", in case that gives any clue about what I should look for in my app code / repro attempt.

PanicException(EXECUTE_SYNC_ABORT RuntimeError: memory access out of bounds pkg/rust_lib_bg.wasm 1:2517166                                                  2822
pkg/rust_lib.js 248:14                                                          rust_arc_increment_strong_count_RustOpaque_flutter_rust_bridgefor_generatedRustAutoOpaqueInnerAutomerge
packages/plans/src/rust/frb_generated.web.dart 2223:12                          rust_arc_increment_strong_count_RustOpaque_flutter_rust_bridgefor_generatedRustAutoOpaqueInnerAutomerge
packages/flutter_rust_bridge/src/rust_arc/_common.dart 32:38                    clone
packages/flutter_rust_bridge/src/misc/rust_opaque.dart 61:52                    frbInternalCstEncode
packages/flutter_rust_bridge/src/misc/rust_opaque.dart 68:19                    frbInternalSseEncode
packages/plans/src/rust/frb_generated.dart 7770:14                              sse_encode_Auto_RefMut_RustOpaque_flutter_rust_bridgefor_generatedRustAutoOpaqueInnerAutomerge
packages/plans/src/rust/frb_generated.dart 3039:9                               <fn>
packages/flutter_rust_bridge/src/main_components/handler.dart 25:24             executeSync
...

[fzyzcjy]

Hmm really weird...

Another brainstorm: https://github.com/rust-lang/rust/blob/30f168ef811aec63124eac677e14699baa9395bd/library/std/src/sys/sync/rwlock/futex.rs#L173

Is it possible that, even though it is single thread, RwLock can still go to the write_contented branch, due to compare_exchange_weak somehow fails? (Not checked the details of that compare-exchange operation) Then we should avoid RwLock.

Btw there is a thing that may be useful for this brainstorm:

flutter_rust_bridge/frb_rust/src/dart_opaque/boxes/thread_box.rs

Line 9 in 4c91787

pub struct ThreadBox<T: Debug>(GuardedBox<T, GuardedBoxContextThread>);

. The ThreadBox is a box that can be Send and Sync through threads, but you can only open the box when in the thread that creates it. If we use this to replace RwLock, then we are (1) no longer touch any real locks (2) no unsafe code, i.e. if someone happens to add some code that wants to use data on another thread, then instead of allowing so and cause rust undefined behavior etc, we just error.

Another brainstorm: Is it possible we use other RwLock implementations which is safe on web?

(Too busy recently so cannot sit down and think more about this issue... but I will do so in the next batch once having time. Feel free to firstly have a quick try to replace those errored RwLock by ThreadBox and recompile and see whether it works!)

Another brainstorm: Is it possible we polyfill the Atomics.wait and provide a naive implementation if we detect it is on main thread, e.g. just spinning? Say, window.Atomics.wait = function() { if(main thread) ... else original_atomics_wait }.

[aran]

Thank you again for the help. I will try out that suggestion as soon as I can. Another clue I've discovered is that I have not managed to repro on Flutter 3.27.4 - so far only on 3.29.

[fzyzcjy]

You are welcome! Ah, then maybe another workaround before I have time digging more is to use 3.27.4, since it is quite recent version as well

[aran]

It seemingly bisects to flutter/flutter@f89546c, i.e. almost certainly flutter/engine#56900, although I don't understand why that would have this effect.

[fzyzcjy]

Interesting... So I guess it triggers some low-level internal changes for the compiled system.

Anyway, I hope to find time to work on it within a few days, e.g. completely remove the Atomics.wait needs for such scenario.