Test count in start-run report is always 0

[cyanite]

This is because the MasterTestRunner is never "loaded" (i.e. the Load method called) before the tests are run, which then makes CountTests return 0. Curiously, no code ever calls Load except some unit tests, which thus seem to test behavior not actually encountered when running the program.

There is also a bug in Load which makes it crash for aggregated test runs.

I don't know if there is a good reason to not always call Load. It seems to work fine for me on a large-ish .nunit test suite with 1 nunit3 test assembly and 50 nunit2.

I will create a pull request which "fixes" the bug in Load (makes it return null where it otherwise crash; no one uses the result anyway), and adds an option to nunit-console, --preload, to load the MasterTestRunner before running the tests. This fixes the bug.

[mikkelbu]

Some googling in the repository gave #146 and in particular the following #146 (comment) about why the count of test cases for the <start-run> event is currently zero. I've not looked into the code or PR in more detail.

Note: we possibly have the same issue in the netstandard version, src/NUnitEngine/nunit.engine.netstandard/Runners/MasterTestRunner.cs, but perhaps this will be solved by #388.

[cyanite]

Well, that makes sense. However @CharliePoole 's comment:

However, it's only in the console that we limit the number of agents and the console runner doesn't need to have a count in advance.

Is only true by default. For example, there is a teamcity addin shipped with NUnit which produces special output while running tests, and I have a similar addin whose purpose is to provide progress information to a tool that's running nunit.

For our particular test workloads we won't actually end up starting a lot of processes, but I certainly appreciate that it can be a problem in general. My PR adds an command line option.

[cyanite]

Just to be clear on the above: I need the test count in order to display progress.

[CharliePoole]

@cyanite Your point is a good one. If an extension wants a valid count then we should try to give it. A command-line option feels like a hack though. Can we allow an extension to indicate what it needs or directly ask for it instead?

[cyanite]

@CharliePoole I agree it's a bit hack-ish. Having the extension activate the load would be ideal, but I don't see how it's possible at the moment. The extension interfaces don't have a lot of methods. The one I am using is the ITestEventListener with just one method.

Also, the call to Load (in my "hack") is all the way in the console-runner executable, which otherwise just calls Run directly. I feel a command line option, while not ideal, would still be useful. Maybe a longer, less ambiguous name.

Otherwise we perhaps need something like an optional IConfiguration-something interface that extensions can implement.

[rprouse]

@cyanite thanks for the pull request, but I agree with @CharliePoole on this one, a command line option isn't the right way to go, so I closed that PR. An IConfiguration type interface might be the right way to go, but that will take a bit of design, so I am going to label this issue as such.

[cyanite]

@rprouse, alright, then. I'll have to keep the changes in my local build since we need progress reports on our large suites. Also note that the PR fixes another crash issue (in Load).

[cyanite]

@rprouse Of course it would be easy to add a method/property "ShouldLoad" to ITestEventListener but that would break all existing extensions. Alternatively, a new callback-style interface could be optionally implemented, which would have a method where the test runner would be passed. But which test runner, then? Any ITestEventListeners are added at the MasterTestRunner level, but other extension points are added in different places.

[rprouse]

Or we could just provide a way for extensions to get the test count that would ensure that they tests were loaded?

[CharliePoole]

Least intrusive approach is probably declarative. Either a new attribute or a property of Extension attribute could pass info to the engine about what an extension requires. I imagine a flags enum.

[cyanite]

@CharliePoole Yes... although the problem with that is that my extension only activated when a certain environment variable is present, so as not to mess with regular console runs. With an attribute it would activate all the time. Compare to the teamcity plugin which is normally disabled (and enabled somehow, I don't know). Things like that seem to be needed since there is no way (I know of) to enable and disable plugins on the command line, is there?

Alternatively: With a new single-purpose interface, it could decide whether or not to activate dynamically:
interface ITestRunnerConfiguration { void ConfigureRunner(ITestRunner runner); } or something, with whatever names. "Configure" might be a too strong word. In my case, I would just call runner.Load. Maybe "BeforeTestRun" is better. (this alternative is moot if you can indeed enable and disable extensions on the command line, in which case I also vote for the declarative approach).

Edit: An even simpler interface, in line with what @rprouse talked about, ITestCountSomething { void SetCount(int count); }.

Edit edit: If there were a general way to enable/disable extensions on the command line (like the hardcoded one for --teamcity), @CharliePoole's declarative approach would be fine also. Don't we want such a feature? Shouldn't be hard to do, I think.

[CharliePoole]

@cyanite TL;DR I agree with your last suggestion. 😄

Enabling / Disabling extensions is purely an engine feature, which may be accessed by a runner. Conceptually, the extensions themselves are imagined to be always active, if they are called by the engine. Disabling one just means it won't be called.

Of course the engine can't possibly know anything about whether an extension should be disabled, since that's a function of the runner (nunit3-console in this case). However, an extension author can decide whether an extension should be initially disabled or enabled, indicating the choice via ExtensionAttribute. The runner can use whatever logic it likes to change that setting, but whatever it does is completely hidden from the extension. IOW, with the current design, an extension can't do different things based on being enabled or disabled. In fact, that makes no sense in the case of the disabled state since it will never be called. An extension that needs to do some initialization can do so on first call.

If we were to allow an extension to disable itself, it would need to have a way to tell the engine that it was disabled. We could do that through a callback interface to the engine or by having the extension implement an interface to be monitored by the engine. That would allow the logic for enablement / disablement to be placed in the extension as you have done.

A command-line approach is appropriate IME if the user may or may not want to use the extension on a run-by-run basis. That's exactly the case with the teamcity extension: we only want it to be active when the console is being executed by teamcity. Unfortunately, that forces us to put special logic in the console runner to understand the purpose of the teamcity extension and when to enable it. I didn't much like doing that, but there seemed to be no choice at the time.

A more general approach would be to allow the console runner to support extensions as well and to package console and engine extensions together in one assembly. That was the initial design, in fact, but it was never implemented.

I would be in favor of two shorter term fixes to the overall problem:

1. Create a more general console option to enable / disable extensions by their extension path.

2. Provide an optional interface to be implemented by any extension that wants to provide information to the engine, such as "treat me as disabled" or "I need to have a valid count of tests on the start of the run". Note that the last example doesn't try to tell the engine how to get such a count, it might do it in different ways depending on the other command-line settings, especially in case one of those settings asked for the number of active agents to be limited.