The test runner sometimes emits browser log events and test finished in the wrong order

[ysabih2]

We use Karma test runner in live coding service to run Angular solutions with tests. We use a long running ng process with spec reporter plugin for test output. We found that the test output for the same solution can be inconsistent in some rare cases. This was reported to the reporter plugin team in this issue. Please check the linked issue for more details on the issue. The reporter plugin team confirmed that the plugin is unlikely to be the cause of the issue because it only handles emitted events from the test runner as they are received. Please note that the issue happens quite rarely, especially when the live coding service instance runs for 2 weeks without crashing.

The reporter plugin team's explanation was confirmed when we wrote a custom plugin to extract the browser logs printed by executed tests. Please find attached the plugin file and the karma config, we currently use Karma 6.4:

karma-console-logger.js
karma.conf.js

To give you a clear idea of the issue. We run the solution + tests, the normal output file looks like this:

Output from Test1
-------- Test finished: Test1 Example case
-------- Test finished: Test2 No output
-------- Test finished: Test3 No output either
Output from Console Output test.
-------- Test finished: Console Output

In some rare cases, the output is wrong:

Output from Test1
Output from Console Output test.
-------- Test finished: Test1 Example case
-------- Test finished: Test2 No output
-------- Test finished: Test3 Various items in the array
-------- Test finished: Console Output`

The problem here is that we rely on the browser log events and test finished events to be emitted in the correct order to extract the output of specific tests correctly. We suspect that there is some race condition behind these rare instances of incorrect events order.

Thanks in advance for looking into this!