Is it safe to use block_on at the top level of a spawn_blocking call?

[stuhood]

Hey folks!

Is it safe to use block_on at the top level of a spawn_blocking call? We have code that works a majority of the time while doing this, but which seems to be subject to a hang given enough concurrency (which might imply that we're deadlocking when we hit https://docs.rs/tokio/1.5.0/tokio/runtime/struct.Builder.html#method.max_blocking_threads), and we'd like to know whether this usecase is supported.

Thanks!

[Darksonn]

In general, yes, using block_on at the top level of a spawn_blocking is supported, but if you make use of this feature, there are certain types of behavior that can deadlock (this applies to spawn_blocking in general, whether or not you use block_on inside it). One rule you can follow which guarantees that you have no deadlocks due to the spawn_blocking pool capacity is that any task running on it (using block_on or not) should eventually exit on its own even if all other threads were paused for the full duration of the spawn_blocking call.

So for example, if your spawn_blocking task cannot complete until some other spawn_blocking task completes, then this can cause a deadlock given enough concurrency. Another example is if the task cannot complete until some message is sent or received on a channel. This can sometimes cause a deadlock depending on the nature of the other end of the channel.

It's worth noting that there are some cases that don't follow this rule, but still cannot deadlock, however, the rule can be followed in most cases. One case that doesn't follow the rule but is still ok is when the spawn_blocking task spawns a new thread with std::thread::spawn and awaits its completion. This is ok because nothing can prevent that new thread from running indefinitely due to how it is not subject to any pool capacity. It could still fail due to an upper limit on OS threads, but in this case the spawn would fail immediately rather than block forever, so this would not be a deadlock.

The article Async: What is blocking? discusses this issue a bit in the section on dedicated threads. It is also discussed here.

[stuhood]

Thanks, that makes sense.

One rule you can follow which guarantees that you have no deadlocks due to the spawn_blocking pool capacity is that any task running on it (using block_on or not) should eventually exit on its own even if all other threads were paused for the full duration of the spawn_blocking call.

I believe that we began violating this principle at some point during a refactoring by moving from block_oning on a library that used dedicated threads to accomplish some work, to a library that used tokio.

---

To make this code safe in the face of further refactors, it seems like giving the code a dedicated unbounded threadpool might work. Would it be possible/ergonomic to have spawn_blocking and spawn_blocking_unbounded, where the latter would prevent deadlocks even in this case?

[Darksonn]

You can use std::thread::spawn as an spawn_blocking_unbounded method. Or really any thread pool library in the Rust ecosystem.