Option unwrap() on None when using AsyncWriteExt

[msftsecurityteam]

Version
1.41.0 and 1.40.0

Platform
Linux rddlr 6.1.0-26-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.112-1 (2024-09-30) x86_64 GNU/Linux

Description
I am writing async code in which I am passing around a lazy_static!'ed Arc<Option<File>>. I am a new Rust developer so maybe some of what I am doing is incorrect, but this issue always happens at the end of the runtime of my code. My code does a lot of async HTTP requests (each of which is tokio::spawn()'ed and then each return value is worked on via a second set of tokio::spawn() threads. This is all done via a stream::iter(chunks).map(|| { ... }).buffer_unordered(N).for_each(|res| async { ... }).await, where chunks is a collection of Strings.

I have attached the full backtrace here:

thread 'tokio-runtime-worker' panicked at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/fs/file.rs:732:51:
called `Option::unwrap()` on a `None` value
stack backtrace:
   0:     0x56055c7aa4e5 - std::backtrace_rs::backtrace::libunwind::trace::h649ab3318d3445c5
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5
   1:     0x56055c7aa4e5 - std::backtrace_rs::backtrace::trace_unsynchronized::hf4bb60c3387150c3
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x56055c7aa4e5 - std::sys::backtrace::_print_fmt::hd9186c800e44bd00
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs:65:5
   3:     0x56055c7aa4e5 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h1b9dad2a88e955ff
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs:40:26
   4:     0x56055c7cfc9b - core::fmt::rt::Argument::fmt::h351a7824f737a6a0
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/fmt/rt.rs:173:76
   5:     0x56055c7cfc9b - core::fmt::write::h4b5a1270214bc4a7
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/fmt/mod.rs:1182:21
   6:     0x56055c7a790f - std::io::Write::write_fmt::hd04af345a50c312d
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/io/mod.rs:1827:15
   7:     0x56055c7ab7e1 - std::sys::backtrace::BacktraceLock::print::h68d41b51481bce5c
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs:43:9
   8:     0x56055c7ab7e1 - std::panicking::default_hook::{{closure}}::h96ab15e9936be7ed
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:269:22
   9:     0x56055c7ab4bc - std::panicking::default_hook::h3cacb9c27561ad33
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:296:9
  10:     0x56055c7abe41 - std::panicking::rust_panic_with_hook::hfe205f6954b2c97b
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:800:13
  11:     0x56055c7abc73 - std::panicking::begin_panic_handler::{{closure}}::h6cb44b3a50f28c44
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:667:13
  12:     0x56055c7aa9a9 - std::sys::backtrace::__rust_end_short_backtrace::hf1c1f2a92799bb0e
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs:168:18
  13:     0x56055c7ab934 - rust_begin_unwind
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:665:5
  14:     0x56055c04b2d3 - core::panicking::panic_fmt::h3d8fc78294164da7
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panicking.rs:74:14
  15:     0x56055c04b35c - core::panicking::panic::hec978767ec2d35ff
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panicking.rs:148:5
  16:     0x56055c04b239 - core::option::unwrap_failed::hba6b08832f9ce30b
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/option.rs:2020:5
  17:     0x56055c6400af - core::option::Option<T>::unwrap::h2c408f65545b6fc1
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/option.rs:970:21
  18:     0x56055c6400af - <tokio::fs::file::File as tokio::io::async_write::AsyncWrite>::poll_write::hf4e8f8d18a0ce1bd
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/fs/file.rs:732:35
  19:     0x56055c062251 - <&mut T as tokio::io::async_write::AsyncWrite>::poll_write::hbad151a078d0bfd2
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/io/async_write.rs:192:13
  20:     0x56055c076255 - <tokio::io::util::write_all::WriteAll<W> as core::future::future::Future>::poll::h6d02d0ad4bf03094
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/io/util/write_all.rs:43:28
  21:     0x56055c08caca - rddlr::write_to_file::{{closure}}::h6c52804999bcbc84
                               at /home/m/riddler/rddlr/src/lib.rs:68:48
  22:     0x56055c08f4d4 - rddlr::update_db::{{closure}}::{{closure}}::hba4e3babc34d81bc
                               at /home/m/riddler/rddlr/src/lib.rs:96:42
  23:     0x56055c0abd3f - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::hfcbe4afeae601794
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/core.rs:331:17
  24:     0x56055c0aab8d - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h4648bddbd7bef5bf
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/loom/std/unsafe_cell.rs:16:9
  25:     0x56055c0aab8d - tokio::runtime::task::core::Core<T,S>::poll::hccc119ba2c2a2f14
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/core.rs:320:13
  26:     0x56055c05170a - tokio::runtime::task::harness::poll_future::{{closure}}::h9b9cbc0f8693b90a
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/harness.rs:499:19
  27:     0x56055c09d140 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::hb3303775436e64fc
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panic/unwind_safe.rs:272:9
  28:     0x56055c0958d8 - std::panicking::try::do_call::h241a365f20480bad
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:557:40
  29:     0x56055c0a4edb - __rust_try
  30:     0x56055c0a2316 - std::panicking::try::heca9eac5abfb8096
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:521:19
  31:     0x56055c0a2316 - std::panic::catch_unwind::h151c0d8f1eb7e988
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panic.rs:350:14
  32:     0x56055c04f56f - tokio::runtime::task::harness::poll_future::h43a6182ab4b73fee
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/harness.rs:487:18
  33:     0x56055c0520e7 - tokio::runtime::task::harness::Harness<T,S>::poll_inner::h08431fd45c9a495f
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/harness.rs:209:27
  34:     0x56055c054dd3 - tokio::runtime::task::harness::Harness<T,S>::poll::h946a01863d743bf1
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/harness.rs:154:15
  35:     0x56055c07c3db - tokio::runtime::task::raw::poll::h737a55a06e0d483c
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/raw.rs:271:5
  36:     0x56055c6246c7 - tokio::runtime::task::raw::RawTask::poll::h5d15c4892734d6b0
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/raw.rs:201:18
  37:     0x56055c63ac52 - tokio::runtime::task::LocalNotified<S>::run::h6074739368ab5b61
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/mod.rs:435:9
  38:     0x56055c693be7 - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}::h5b2b845e8538efc4
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/scheduler/multi_thread/worker.rs:659:17
  39:     0x56055c693404 - tokio::runtime::coop::with_budget::h6880d8e13f1bc2ab
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/coop.rs:107:5
  40:     0x56055c693404 - tokio::runtime::coop::budget::h8a6346b062152998
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/coop.rs:73:5
  41:     0x56055c693404 - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::h48cc5e988d5ab844
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/scheduler/multi_thread/worker.rs:595:9
  42:     0x56055c692ad3 - tokio::runtime::scheduler::multi_thread::worker::Context::run::h1845136b12163b49
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/scheduler/multi_thread/worker.rs:546:24
  43:     0x56055c692739 - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}::h6458dd2a1c080743
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/scheduler/multi_thread/worker.rs:511:21
  44:     0x56055c641060 - tokio::runtime::context::scoped::Scoped<T>::set::h35ff1f539ee75ab1
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/context/scoped.rs:40:9
  45:     0x56055c62e73b - tokio::runtime::context::set_scheduler::{{closure}}::h3b2fc7981eb184c7
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/context.rs:180:26
  46:     0x56055c6508d0 - std::thread::local::LocalKey<T>::try_with::h916dab96c048bcec
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/thread/local.rs:283:12
  47:     0x56055c64f8eb - std::thread::local::LocalKey<T>::with::h4d1a4e5c73044a4e
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/thread/local.rs:260:9
  48:     0x56055c62e674 - tokio::runtime::context::set_scheduler::h1c36c4fce850766c
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/context.rs:180:9
  49:     0x56055c692644 - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::h57fd5603045879b8
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/scheduler/multi_thread/worker.rs:506:9
  50:     0x56055c69db20 - tokio::runtime::context::runtime::enter_runtime::hc10b0594d99eb384
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/context/runtime.rs:65:16
  51:     0x56055c6923e6 - tokio::runtime::scheduler::multi_thread::worker::run::hd8c2dae097f14897
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/scheduler/multi_thread/worker.rs:498:5
  52:     0x56055c69210b - tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}::hbf53b781f9169222
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/scheduler/multi_thread/worker.rs:464:45
  53:     0x56055c6624fe - <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll::h13ce29aa647a1f74
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/blocking/task.rs:42:21
  54:     0x56055c6885d3 - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::ha564cfb16c89726a
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/core.rs:331:17
  55:     0x56055c688081 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::hc267351fc8f9d3f6
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/loom/std/unsafe_cell.rs:16:9
  56:     0x56055c688081 - tokio::runtime::task::core::Core<T,S>::poll::h4b0161cbdca6d7ae
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/core.rs:320:13
  57:     0x56055c6171ae - tokio::runtime::task::harness::poll_future::{{closure}}::hab31ce16ab6fb02d
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/harness.rs:499:19
  58:     0x56055c61db21 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h88b5b9f00ad79926
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panic/unwind_safe.rs:272:9
  59:     0x56055c663749 - std::panicking::try::do_call::hed8ab69602e68b3c
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:557:40
  60:     0x56055c69e4bb - __rust_try
  61:     0x56055c69c5f6 - std::panicking::try::h7c9620815e3036a6
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:521:19
  62:     0x56055c69c5f6 - std::panic::catch_unwind::h2f21e58816f72ad5
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panic.rs:350:14
  63:     0x56055c616880 - tokio::runtime::task::harness::poll_future::h973c2f04a8e94a8e
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/harness.rs:487:18
  64:     0x56055c61474e - tokio::runtime::task::harness::Harness<T,S>::poll_inner::h53116de9c9d8687d
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/harness.rs:209:27
  65:     0x56055c6143b7 - tokio::runtime::task::harness::Harness<T,S>::poll::hd2795e66e6a9ecd7
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/harness.rs:154:15
  66:     0x56055c62492d - tokio::runtime::task::raw::poll::h035ede83177f58ed
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/raw.rs:271:5
  67:     0x56055c6246c7 - tokio::runtime::task::raw::RawTask::poll::h5d15c4892734d6b0
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/raw.rs:201:18
  68:     0x56055c63ad17 - tokio::runtime::task::UnownedTask<S>::run::hccc8ffbfb8d3c54e
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/task/mod.rs:472:9
  69:     0x56055c6970f7 - tokio::runtime::blocking::pool::Task::run::h33d4ebb4b885b2c4
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/blocking/pool.rs:161:9
  70:     0x56055c699ede - tokio::runtime::blocking::pool::Inner::run::h392fb86cd584a4d3
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/blocking/pool.rs:511:17
  71:     0x56055c699bf4 - tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}::hfdeff38519ea2830
                               at /home/m/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.0/src/runtime/blocking/pool.rs:469:13
  72:     0x56055c672116 - std::sys::backtrace::__rust_begin_short_backtrace::h8c793e533b1c36fd
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs:152:18
  73:     0x56055c673ad2 - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::h58c065ad7e75361e
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/thread/mod.rs:538:17
  74:     0x56055c61da22 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h54f71e4f943f91c8
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panic/unwind_safe.rs:272:9
  75:     0x56055c663093 - std::panicking::try::do_call::h20b94d30cc8ce131
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:557:40
  76:     0x56055c67721b - __rust_try
  77:     0x56055c673716 - std::panicking::try::h71d8ca21fec86411
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:521:19
  78:     0x56055c673716 - std::panic::catch_unwind::h399adad2b13170a4
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panic.rs:350:14
  79:     0x56055c673716 - std::thread::Builder::spawn_unchecked_::{{closure}}::h22242bfe68727c34
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/thread/mod.rs:537:30
  80:     0x56055c66794f - core::ops::function::FnOnce::call_once{{vtable.shim}}::h2e60a4cbf4c7a1d1
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/ops/function.rs:250:5
  81:     0x56055c7aea5b - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::ha1963004222e7822
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/alloc/src/boxed.rs:2070:9
  82:     0x56055c7aea5b - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h1086ced1f7c494c2
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/alloc/src/boxed.rs:2070:9
  83:     0x56055c7aea5b - std::sys::pal::unix::thread::Thread::new::thread_start::ha8af9c992ef0b208
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/pal/unix/thread.rs:108:17
  84:     0x7efdcf8a8144 - start_thread
                               at ./nptl/pthread_create.c:442:8
  85:     0x7efdcf9287dc - __GI___clone3
                               at ./misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
  86:                0x0 - <unknown>

[msftsecurityteam]

Towards the end of my process_chunk() logic (this is the second of the two tokio::spawn()'s that I mentioned), I write to the shared filehandle that is guarded by a mutex, via another tokio::spawn thread. Adding this and I see ~25% performance boost, and it seems to work just fine because all of the data is processed, but this is when I begin to see these unwrap() on None errors at process shutdown. If I remove this third tokio::spawn(), I take a slight performance hit but it doesn't seem to be causing any unwrap() on None issues anymore.

[Darksonn]

This is happening during runtime shutdown?

[msftsecurityteam]

I don't know if there is an official time for "runtime shutdown" but yes, it happens at the very end of my code after all of the data has been processed. I get like, 8 or 16 or whatever the number of threads is saying Option unwrap() on None panic

[msftsecurityteam]

I am updating the comment. I believe the second tokio::spawn() in the .for_each() was unnecessary. I pulled it out, keeping the tokio::spawn() just for the very end for the file I/O write_all() call, and I am still getting these unwrap() errors.

Code is more or less:

let _ = stream::iter(chunks).map(|chunk| { tokio::spawn(async move { fetch_with_retry(&chunk).await })}) .buffer_unordered(NUM_REQUESTS).for_each(|fetch_result| async { match fetch_result { Ok(Some(data)) => { if let Err(e) = process_chunk(&tag).await { eprintln!("process_chunk failed: {}", e); } }, Ok(None) => { eprintln!("fetch_with_retry returned no data"); }, Err(e) => { panic!("match fetch_with_retry fail: {:?}", e) } } }) .await;

Inside process_chunk it is a bunch of JSON parsing, with a tokio::spawn at the end to async write_all the results.

[Darksonn]

Runtime shutdown starts once you return from the #[tokio::main] function.

Are you able to share a small complete example that triggers the error so I can try it myself?