Skip to content

Implement Akka.MultiNodeTestRunner.VisualStudio package #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 48 commits into from
Feb 7, 2020

Conversation

IgorFedchenko
Copy link
Contributor

@IgorFedchenko IgorFedchenko commented Jan 31, 2020

This is work on #7 , and port of akkadotnet/akka.net#4177

The goal here is to implement new Akka.MultiNodeTestRunner.VisualStudio package and make dotnet test to run multi-node tests.

As a good bonus, we may try run multi-node tests in separate app domains instead of OS processes, making it possible to debug each node's execution. This is something that xunit is already doing with parallel tests running, so maybe I will be able to reuse their implementation.

Some more details about what should be done can be found here: akkadotnet/akka.net#4177 (comment)

Main reference is here: https://github.com/Microsoft/vstest-docs/blob/master/RFCs/0004-Adapter-Extensibility.md

@IgorFedchenko
Copy link
Contributor Author

Have working sample test adapter, now need to integrate it with existing MNTR code.

Also, since there are more projects to come, had to rename projects and namespaces in more convenient way, like Akka.MultiNode.Shared and Akka.MultiNode.TestRunner, AkkaMultiNode.TestAdapter etc.

@IgorFedchenko
Copy link
Contributor Author

dotnet test seems to be working (at least for basic scenarios), but running old-style Akka.MultiNode.TestRunner.exe ..options... under .net core fails

@IgorFedchenko
Copy link
Contributor Author

IgorFedchenko commented Feb 5, 2020

All right, that's because when .net core project references another exe project (like our TestRunner references NodeRunner app), only exe and dll files are copied (it is treated as referenced class library, but not referenced console app - see here). But having *.runtimeconfig.json is also required to run console app. So need to include those from NodeRunner to TestRunner build output.

@IgorFedchenko
Copy link
Contributor Author

Now it is working both as a standalone util (just like before) and from dotnet test.

Need to add more tests to verify that specs failure/skipping works too.

@IgorFedchenko IgorFedchenko marked this pull request as ready for review February 5, 2020 16:00
@IgorFedchenko
Copy link
Contributor Author

IgorFedchenko commented Feb 7, 2020

All right, so one of the issues was that dotnet test may run multiple test runners in parallel - and MNTR was using same default port all the time.

Now we can specify listen-port to 0 to let it pick random available TCP port - and that will be the default for dotnet test.

@IgorFedchenko
Copy link
Contributor Author

IgorFedchenko commented Feb 7, 2020

Another issue I get from dotnet test is:

Tests discovery from C:\Users\igor\AppData\Local\Temp\94b60cf3-51fe-4f2e-846b-ae4fe46af07f\94b60cf3-51fe-4f2e-846b-ae4fe46af07f\assembly\dl3\e12c956d\fd425e84_a1ddd501\Akka.MultiNode.SampleMultiNodeTests.dll failed: System.InvalidOperationException: Unknown test framework: could not find xunit.dll (v1) or xunit.execution.*.dll (v2) in C:\Users\igor\AppData\Local\Temp\94b60cf3-51fe-4f2e-846b-ae4fe46af07f\94b60cf3-51fe-4f2e-846b-ae4fe46af07f\assembly\dl3\e12c956d\fd425e84_a1ddd501
   в Xunit.XunitFrontController.CreateInnerController() в C:\Dev\xunit\xunit\src\xunit.runner.utility\Frameworks\XunitFrontController.cs:строка 142
   в Xunit.XunitFrontController.EnsureInitialized() в C:\Dev\xunit\xunit\src\xunit.runner.utility\Frameworks\XunitFrontController.cs:строка 163
   в Xunit.XunitFrontController.Find(Boolean includeSourceInformation, IMessageSink messageSink, ITestFrameworkDiscoveryOptions discoveryOptions) в C:\Dev\xunit\xunit\src\xunit.runner.utility\Frameworks\XunitFrontController.cs:строка 173
   в Akka.MultiNode.TestRunner.Shared.MultiNodeTestRunner.DiscoverSpecs(String assemblyPath)

The funny part is that Akka.MultiNode.SampleMultiNodeTests.dll is actually already finished execution from Debug folder.
This error is only under net472, net core does not have this issues.

@IgorFedchenko
Copy link
Contributor Author

Looks like dotnet test under full .net framework handles referenced assemblies different from what vstest does from VS, or how .net core works.
So just changed a way how to locate tests assembly.

@IgorFedchenko
Copy link
Contributor Author

Ok, now dotnet test, VS test runner and standalone MNTR runs without errors and issues on my side.

But looks like CI is still not happy due to some nuget packaging issues. Will fix it then.

Next TODO is:

  • Instead of logging to console, use some simple logger that would log to console (for standalone run) or to test adapter output channel
  • Add more mntr specs and verify that passed/skipped/failed specs are reported correctly.

@IgorFedchenko
Copy link
Contributor Author

Fixed nuget, but have issues under linux with free port detection

@IgorFedchenko
Copy link
Contributor Author

Cool, finally CI loves my code.

So, now ideally duplicate console logging to test adapter output, and add specs for failing and skipped mntr tests.
Maybe will move logging improvement to separate issue, and merge this PR once additional specs pass.

@Aaronontheweb Aaronontheweb merged commit b7c5923 into akkadotnet:dev Feb 7, 2020
@IgorFedchenko IgorFedchenko deleted the add-vs-runner branch February 8, 2020 07:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants