debugging live lock-ish issues between services using futures

[nikomatsakis]

Brief Summary

One challenge would be how to debug live lock-ish issues between services using futures. On fuchsia pretty much everything is asynchronous. So when you have some client interacting with some other service, they both have executors driving the futures. So:

• client starts an executor
• client adds a future to the executor
• The future makes a call to some service.
• the executor suspends the future and waits on a kernel object for any channel to receive a message.
• For whatever reason the server does not reply, nor does it close the channel.

While fuchsia has some tools to help debug issues like this, the main technology we have to triage live-ish locks like this to set timeouts on our calls, and annotat most call sites with logs. Inevitably we forget some locations, and it becomes difficult to understand what happened.

My dream to try to debug issues like this would if there was some mechanism where we could ask the executor to wake up every pending future, then dump a backtrace of the future stack. Something like that would make it easy to detect when’s future is stuck waiting for something to happen.

Originally posted by @erickt in #69 (comment)

[nikomatsakis]

My dream to try to debug issues like this would if there was some mechanism where we could ask the executor to wake up every pending future, then dump a backtrace of the future stack. Something like that would make it easy to detect when’s future is stuck waiting for something to happen.

This is shiny future. Leave that out for now. But the rest is a perfect status quo story.

[sdroege]

To make debugging async code as nice as sync code, I believe it would be great to be able to have debugger integration for async tasks.

Consider for example the simple scenario where you have two async mutexes A and B, one task taking them both in that order and another in the other order. If these were threads then this would be completely obvious from the debugger: one thread will show up as waiting for one mutex, the other for the other and by looking at the surrounding code it's all easy. However with async tasks there is no indication at all when looking at the program state in the debugger. All threads are happily doing their jobs in the executors or have the executor wait for more work, no indication which tasks currently exist and what/where they're waiting.

More generally, threads have an advantage over async tasks whenever your problem is that some code is (a)waiting for something that "never" happens.

The only way to debug this right now is by adding logging/tracing to the application code to get a log of what exactly happened in the application and from there go back to the code. Actively inspecting the program state in a debugger is not going to be helpful.

For me personally this is the biggest problem I have when debugging async code.

[rylev]

@nikomatsakis do we think that #106 and #114 have covered this sufficiently? If not, what's missing?

[nikomatsakis]

I do. I'm going to close the issue.