V4 Codebase Should be Null-Enabled

[CharliePoole]

I'm treating this as a prerequisite for re-booting work on version 4. I observed that null-enabling the framework code brought up many issues, so it may be desirable to do this one assembly at a time, in separate PRs.

This depends on updating the language level, which I'm doing in a separate issue.

[manfred-brands]

@CharliePoole Converting existing code to nullable is not simple if done right.

It heavily depends on the code at hand.
If a class has various fields that are set in certain methods and used in others it could result in a lot of tests on null.

Removing the need for this might mean splitting functionality in separate classes.

How far do you want to take this?

I can have a look and do an initial investigation.

What project do you think I should start with?

[mikkelbu]

I was considering starting with determining how the different project relates and then start with the "bottom" (the project that only has no/external dependencies).

I think we should just start with the quick solutions - e.g. marking certain areas/methods as ignored via nullable disable and similar and then improve on this later on. Otherwise, I fear that this could take a very long time.

I had originally considered volunteering for giving a hand on this, but I lack the time at the moment (or at least proper time chunks and not just 15-25 minute blocks now and then).

[CharliePoole]

@manfred-brands Thanks! I'm glad to have you working on this. I'll help by asking naive questions. :-)

Seriously, I've experimented only a little with nullable types, so I prefer not to do this myself in the engine. One bit of general guidance: I suggest not trying to get each project "done" all at once. If there are particularly troublesome areas, @mikkelbu 's suggestion of ignoring them for a time makes sense to me. The only risk is that we forget to return and fix it, so let's not do that... we can leave behind some cleanup issues as we go or scan the entire repo and make a list at the end.

The dependencies will change as development continues, so I'll show both the current ones and those planned.

Current Dependencies

nunit4-console
  => nunit.engine.api
  => nunit.engine (1)

nunit.engine
  => nunit.engine.api
  => nunit.engine.core  =>  nunit.engine.api

nunit-agents (2)
  => nunit.engine.api
  => nunit.engine.core

Planned Dependencies

nunit4-console
  => nunit.engine.api

nunit.engine
  => nunit.engine.api
  => nunit.extensibility (3)

pluggable agents
  => nunit.engine.api
  => nunit.agent.core (4)
  => nunit.extensibility (3)

Notes

1. At the moment, the runner depends on the engine itself as well as the api. This dependency crept in as a workaround to resolve a problem but I'm going to try to remove it. [See Eliminate reference to nunit.engine by nunit-console runner #1576]
2. There are currently five separately named agents, all with the same dependencies. Eventually, all agents will be pluggable engine extensions each in it's own repo.
3. See Separate assembly for NUnit extensibility #1049
4. The plan is to no longer have any refs common to both the engine and the agents except for nunit.engine.api. Types currently in nunit.engine.core will be moved to either nunit.agent.core or nunit.engine itself. I need to create an actual issue for this! [See Isolate agents from the engine itself #1578]

---

As for where to start, nunit.engine.api seems sensible. It mostly has interfaces, with two reference types used as arguments. So most of the work will be in the classes that implement the interfaces and those in other projects that reference them. The TestPackage class has the particular difficulty that it uses a null Name property to signify that it's the unnamed top-level package, i.e. the one created from the command-line. This is something that may require refactoring but it would be good to get it out of the way at the start.

How does that sound?

[manfred-brands]

@CharliePoole Apologies for my list of comments, I know a lot of this code has been there a long time.

I started on the api/core assemblies and came to a few observations/questions:

1. TestEngineActivator has different CreateInstance signature and implementation for .netstandard2.0 vs .net 462/
   a. For the first the overload to specify a minimum version doesn't exist.
   b. For the second, even though it already has the Assembly loaded, it calls a method to load it again by name. I don't see any benefit to the CreateInstanceFromAndUnwrap method on the current domain vs Activator.CreateInstance especially as the first calls the second

2. NUnitEngineUnloadException a user might have to check multiple properties to find the underlying exception. The standard way would be to pass the list of exceptions to AggregatedException and then pass that as the inner exception. But that is a breaking change.

3. In ExtensionManager.FindExtensionsInAssembly has a local assemblyTargetFramework which is ever only set to null, but its properties are used in exception and log messages. Note this is also the case in the main branch. I see several changes to this code part, but not sure the intention. There was an TargetFramework property on IExtensionAssembly but that is commented out as the definition of RuntimeFramework is in nunit.engine and hence not available in nunit.engine.api.
   The result of this is that ExtensionNode.TargetFramework is always null.

4. TestCentric.Metadata could do with a nullable enabled version. So that GetNamedArgument is marked as object?.
   I started on that, but with that project everything could be null or is set indirectly. So that will take a while.

5. ExtensionNode.Path says it must be unique, but as it is not set by a factory it can be null for all of them or multiple instances can have the same path. There is no code to check for uniqueness. The only way to guarantee that is by creating a factory that has a dictionary of all paths and make an internal constructor that also expects the path and remove the public setter. But as the class is public that would be breaking.

6. ExtensionNode.CreateExtensionObject can return null, but the consuming code doesn't expect that.
   This also has two different way to create an instance as mentioned in 1.

7. TestPackage.FullName is only set for sub-packages. But as it is there is only one type. Having a separate types would be nicer.

8. Several places open a RegistryKey without disposing it.

9. The TestAgentRunner checks there is a single package, but in the Load it does another selection on !subPackage() and a for loop. If the runner only supports a single package then the loop in Load is not needed.

10. TestAgentRunner.TestDomain is a property used in the base class, but expected to be set in a derived class. Ideally this should be passed in a constructor as we now have no way of guaranteeing it is set by a (3rd party) derived class.
    I added a Guard.OperationValid to the Load method.

11. InternalTraceWriter.Write(string? value) calls base.Write instead of _writer.Write. Eventually that will call the Write(char) overload but the same could be said for the WriteLine overload. Seems inconsistent.

12. In TestAgentRemotingTransport ITestEngineRunner implemenation the user must call CreateRunner before calling any of the interface methods. There is no way to force that is actually done unless we return a wrapper class with a non-null runner. Again a breaking change. For now I added a (new) ShouldNotBeNull Guard.
    Another thing is that if users call CreateRunner twice we might have a "leak". I added a conditional Dispose before creating a new one.

13. TcpChannelUtils.CreateTcpChannel had a limit parameter who's usage is commented out. Should it be removed or re-instated?

14. NotRunnableFrameworkDriver relies on derived classes to set runstate, result and label. A better solution would have been virtual properties for these field with implementations in the derived classes. But again a breaking change.

15. Same for NotRunnableTestRunner but here the derived class doesn't set them.

16. IFrameworkDriver depends on users calling Load first. A design where Load return a callable object would fit better with nullability.

17. NUnit3FrameworkDriver passes in a CallbackHandler to the actions and depends on that method calling this once. Could these method not return a string instead of calling a callback? We have now no guarantee that the api will call that handler. Whilst the NUnitNetCore31Driver calls the method that return values. Why the difference? Can the framework driver not be the same?

18. Path.GetDirectoryName is very annoying, so is DirectoryInfo.Parent.

19. nunit.engine.core uses space and nunit.engine.core.test uses tabs.

20. There are various places that test a variable for null then call log.Error but then continue accessing that null variable.
    I cannot find that Logger.Error aborts.

if (method == null)
    log.Error($"Method {methodName} was not found in {_frameworkControllerType.Name}");
log.Debug($"Executing {method.DeclaringType}.{method.Name}");