Simplify locating of addins

[nvborisenko]

Currently there is only one way how user is able to say where nunit-console should find extensions. This is via *.addins file. What if user will be able to install extension via nuget into tests project and that's all what is required from user.

I propose to extend nunit-console and try to search extensions in the directory where user's tests dll is located. It would be default behavior.

[CharliePoole]

You can currently do this by either modifying the delivered addins file or (better) adding a new one pointing to that directory.

That may not totally do the job for you but if you try it I think we'll learn better what us needed.

[ChrisMaddock]

Where is your console coming from at the moment? If you install both console and addins by NuGet, the engine should find the addins automatically already.

[nvborisenko]

Imaging nunit-console came from unknown location, for instance it's pre-installed on build machine (appveyour, etc).

And I do

nunit3-console /full/path/to/my/tests.dll

Actually I don't know where exe is located, hence I cannot use *.addins file.

Generally if you explore some example of the extension https://github.com/reportportal/agent-net-nunit/blob/master/README.md, you might see that the hardest part of the installation is to say nunit-console how to find the extension.

This issue is about the following case:

1. User developed tests
2. Installed extension via nuget, built/publish project
3. Copied all artifacts to some another machine
4. Executed tests

[CharliePoole]

It's probably the biggest issue with extensions that there's no easy way to install them manually.

If you use either nuget or (my favorite) chocolatey, they are all findable very easily. But if you copy artifacts, you are basically down to a manual process, which is easy to get wrong.

As you can see from the docs, the process of locating extensions starts wherever the nunit.engine.dll is located. If it is located in your project, because you installed via nuget.org, then any extensions that are also installed by nuget.org are found automatically.

When I use chocolatey to install a copy of the console runner centrally, then the starting point is inside a chocolatey directory. Extensions that are also installed via chocolatey are found, but no others.

When the msi is installed centrally, there is no way to add new extensions except to add new addin files in the the engine directory.

This overall separation of different install methods is actually by design. If you install the engine in your project, it should not be picking up extensions from some central location. However, IMO at least, it would make sense to do as you are asking and have the central installs also look into the project that is being executed for extensions.

[ChrisMaddock]

Hey @nvborisenko - you've raised a good point. I agree that in your situation there should be an easier way to install extensions. I'm less convinced that searching the test directory for them is the right solution.

As far as I can see, there's a few drawbacks to that approach:

1. The engine will now have to examine every assembly in the test directory to look for extensions. In the case of big projects, that's a lot of assemblies, and may be a significant time cost - especially in the case of running a single test.
2. It feels like a bit of a semantic oddity to me. Why does something I have as a dependency of my test project affect how the test runner is working?
3. What happens if I pass two assemblies into the console runner - with an extension in the test directory of only one of these assemblies? Do all extensions apply to all assemblies? Are the extensions enabled and disabled depending which assembly they were found besides? (Would the latter even make technical sense?)

---

Would a console command line parameter be a better solution? We already have --list-extensions - how about a --load-extensions? This could take a file path to either a directory, extension assembly, or addins file, to keep it consistent with current extension loading functionality.

Tagging in @nunit/engine-team for their thoughts - let's see what others think about this before going any further. 🙂

[jnm2]

It feels like a bit of a semantic oddity to me. Why does something I have as a dependency of my test project affect how the test runner is working?

I've always felt this way about referencing the VSTest adapter from a csproj. :D

--extensions-dir maybe? I like the concept and I agree that we don't want to be scanning the project output being tested.

[ChrisMaddock]

I've always felt this way about referencing the VSTest adapter from a csproj. :D

Yes - same!

---

--extensions-dir maybe? I like the concept and I agree that we don't want to be scanning the project output being tested.

And limit it just to take a directory? It seems like an overhead to need to create a separate directory specifically to house a single extension assembly. I think it would be good to also support the .addins file, to be consistent current behaviour.

[jnm2]

Oh, true. --load-extensions is better for that. Would it ever be beneficial to disable the default loading search logic? I can't think of a reason.

[CharliePoole]

The search for extensions starts in a root directory, which is the directory containing the engine itself.
The idea was that this might be expanded to search in other directories as well, using exactly the same search pattern. One possibility is a common directory used by all engine extensions. Another is some directory specific to the test project being loaded.

However, if you are loading multiple assemblies, there is still only one engine with extensions (i.e. the primary engine) It would take a fair bit of effort, I think, to load an extension only for one assembly, possibly a full restructuring of how we load assemblies. For that reason I agree with an approach where the user simply tells the user where to look for extensions. The command-line makes sense for this purpose. All the console would need to do would be to add the new setting to the TestPackage and the engine should do the rest.

[ChrisMaddock]

Great! @nvborisenko - does the command line option still work for you, and are you still interested in implementing it?

I suggest --load-extensions can take an extension assembly, a directory, or a .addins file. I think it should be possible to use --load-extensions multiple times, for future-proofing. The load-extensions options will need to go into a new EnginePackageSetting, which can be passed into the engine, and the ExtensionService can then load any relevant extensions.

[CharliePoole]

@ChrisMaddock That flexibility (dll/addins/directory) is probably useful for the user even if it takes a bit more work to reorganize the service initialization code. It seems doable.

You haven't specified what a directory means. I suggest that it should mean the same thing as if you added the directory path to the addins file. That is: process any .addins files if present OR process *.dll in that directory. This is slightly different from how we process the original engine directory, where we ignore any dlls, but it seems less surprising.

[ChrisMaddock]

Would it ever be beneficial to disable the default loading search logic? I can't think of a reason.

Me neither. Let's cross that bridge when we come it it. 🙂 There's workarounds, like adding an empty. .addins file.