Skip to content

Conversation

@bikeshedder
Copy link
Contributor

@bikeshedder bikeshedder commented Dec 5, 2019

It is possible to leave a mutex in a locked state even if no future exists holding the lock.

  • create Mutex
  • lock Mutex
  • try to lock mutex in another future that is aborted (e.g. via timeout)
  • unlock Mutex
  • locking the mutex again fails

I think this should be considered a critical issue as it can be exploited as shown in issue #1898 when using hyper and aborting requests.

@bikeshedder bikeshedder force-pushed the feature/mutex-aborted-future branch from 519fd4c to c13853c Compare December 5, 2019 09:42
@bikeshedder bikeshedder changed the title Add tests for mutex issue #1898 Fix issue #1898 - possible deadlock when future waiting for a semaphore is aborted Dec 5, 2019
@bikeshedder
Copy link
Contributor Author

Permit::release must be called regardless if it's in the state Acquired or Waiting. If a Waiting Permit is not released its waiter will stay in the queue blocking any future Permit to be acquired.

The original code did the following:

  • release semaphore if it is aquired
  • panic via unreachable! if the semaphore is idle or waiting unless the code is already panicking

Since a Permit must always be released unless it is in the Idle state I wonder if that unreachable! is even needed. If needed the following code should be added at the beginning of MutexGuard::drop:

if self.permit.state == semaphore::PermitState::Idle && !::std::thread::panicking() {
    unreachable!("Permit was idle when MutexGuard was dropped")
}

@bikeshedder bikeshedder force-pushed the feature/mutex-aborted-future branch from 48de2b3 to 472c87b Compare December 5, 2019 11:34
Permit::release() must also be called when it is in the waiting state.
@bikeshedder bikeshedder force-pushed the feature/mutex-aborted-future branch from 472c87b to 257a904 Compare December 5, 2019 11:37
@bikeshedder bikeshedder changed the title Fix issue #1898 - possible deadlock when future waiting for a semaphore is aborted Fix issue #1898 - Permit::release() must be called when Mutex::lock() is aborted even if it is in the Waiting state Dec 5, 2019
Copy link
Member

@hawkw hawkw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me, but it would be good to hear from @carllerche as well. I commented on some minor nits.

Comment on lines +97 to +98
iv.tick().await;
iv.tick().await;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The use of the interval here is a little un-obvious, it could be good if there was a comment explaining what it's doing here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought that the comment Try to lock mutex in a future that is aborted prematurely was telling enough. tbh. That test case is not really needed. I just added it when debugging the code and trying to reproduce the locking behaviour. This test just makes sure Aquired locks are returned when their future is aborted.

Is there a better way to let a future delay its execution than creating an interval and waiting twice? tick().await is called twice as the first call of it returns instantly.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a better way to let a future delay its execution than creating an interval and waiting twice? tick().await is called twice as the first call of it returns instantly.

You could possibly just use

task::yield_now().await;

if all you want is to yield to the scheduler so that another task will be polled? But, what you're doing now does make sense, I just thought it would be helpful if there was a comment explaining what it was for.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm going to move forward merging & shipping. We can tweak the test in a follow up PR.

#[tokio::main]
#[test]
/// Ensure a mutex is unlocked if a future holding the lock
/// is aborted prematurely.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: It would be good to reference the GitHub issue that these tests reproduce.

@carllerche carllerche requested a review from jonhoo December 5, 2019 18:37
Copy link
Member

@carllerche carllerche left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for debugging & fixing 👍

@carllerche carllerche merged commit c632337 into tokio-rs:master Dec 6, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants