Engine RunAsync should run agent, driver and framework asyncronously

[CharliePoole]

Currently, ProcessRunner.RunAsync() creates a task which calls the synchronous Run method in the agent. TestAgentRunner.RunAsync, which is used in the agents does the same thing in calling the driver.

We discussed this problem in issue #1187 and decided we should be using asynchronous runs "all the way down" if RunAsync is used at the top level.

[CharliePoole]

Current Status in version4 branch:

1. The console runner calls MasterTestRunner.Run. The RunAsync method is only ever called by 3rd party runners.
2. MasterTestRunner.RunAsync calls RunAsync on its top-level ITestEngineRunner.
3. Each of the top-level engine runners eventually delegate to ProcessRunner.
4. ProcessRunner.RunAsync calls RunAsync on the remote runner through TestAgentTcpProxy.
5. The remote runner's TestAgentRunner.RunAsync creates a task, which calls Run, The async call is not passed to the driver at all.

The above is the full sequence as present in the .NET Framework 4.6.2 runner. The .NET 8.0 runner simply skips steps 3 and 4 and uses `TestAgentRunner directly.

If TestAgentRunner.RunAsync were called, the sequence would continue as follows:
5. TestAgent.RunAsync would call the driver's RunAsync method.
6. The driver's RunAsync would call the framework's RunAsnyc,
7. The framework would start a thread to allow Run to be called asynchronously.

So, for the engine stack to be asynchronous "all the way down" we only need to correct step 5 and add tests to verify that it all works. :-)

UPDATE: We also have to add RunAsync to IFrameworkDriver and implement it in each driver! IIRC, we once had that method but eliminated it in both V3 and V4 at some point

[CharliePoole]

@nunit/core-team @nunit/engine-team

Now that I've had some time to play in the code, I'd like to establish some goals for the net beta and final 4.0 release.

For beta 2:

• Get the current signature working correctly all the way down...

  AsyncTestEngineResult RunAsync(ITestEventListener, TestFilter);

• Experimentally, see if we can run the console using RunAsync rather than Run.

Before the final release:

• Replace RunAsync with an actual awaitable async method.
• Can we add a CancellationToken to the signature and honor it? Depends on framework I guess.
• Use the async method in the console runner to execute tests
• Drop the Run method and rename RunAsync to Run, i.e.

  async Task<TestEngineResult> Run(ITestEventListener, TestFilter, CancellationToken);

As I say, these are goals and we don't know if we can actually reach them but I'd like to see how far we can get.

Regarding the naming, and not having two methods, my thinking is that

• Yes, MS guidelines say (or at least once said) that "Async" should be appended to async methods, but Run is logically an asynchronous operation and the signature tells the user that's what it is.
• If the user wants to make it synchronous, they can write a method that awaits ours.
• If the user doesn't want a cancellation token they can add CancellationToken.None at their risk.

@jnm Given proposals you've made in the past, I'd love to see your comments on this! :-)

[CharliePoole]

Well, here's something about the framework I had forgotten. It doesn't actually expose RunAsync for external use. The FrameworkController.RunAsync method is private. Its only public method for running tests is RunTests, which calls RunAsync and waits for it to complete.

UPDATE: Thinking about it, this makes sense... it was intended to prevent a second run of the same tests starting while the first one was still in process. It probably has to stay this way.

However, this is still somewhat useful in the engine and is now a bit simpler. We should make sure that we only transition from async to sync once in the engine. In that way each agent process will complete in its own time and there will be no extra waits on the engine side of things. I've edited my comment above to reflect this.

[CharliePoole]

This issue has been resolved in version 4.0.0-beta.2

The release is available on:
GitHub.
NuGet packages are also available NuGet.org and
Chocolatey Packages may be found at Chocolatey.org