Engine should not crash on encountering assembly that cannot be examined

[ChrisMaddock]

See #794.

Currently, if the engine is passed an assembly that it cannot examine, the engine throws an exception. This was the correct behaviour at the time of implementation, however our handling of invalid assemblies has since improved.

We should instead suppress the crash, and create some form of NotRunnableFrameworkDriver for the assembly. That will allow the engine and various runners to produce more user-friendly error messages for these cases.

[CharliePoole]

Interesting. Our DriverService already does what you suggest in the face of a BadImageFormatException.

But this exception comes in the code that initially examines the assembly in order to select a runner and the DriverService isn't in play until we have a runner that needs to create a driver.

I'm pretty sure this worked in the past but we have added more pre-checks to the code before we actually try to load the assembly and probably that's what broke it.

What I'm not getting is why this doesn't happen for Run as well as Explore.

[ChrisMaddock]

I think I does happen for run as well. 🙂

[CharliePoole]

Ah. That would make sense. I think we should go through all the "pre-discovery" we do with Cecil and try to handle errors in a more friendly way.

In terms of the sequence of things, I think we would want to implement #718 first, if you decide to do it. It's still sitting as an "idea" here although I have a vague memory you said something about going ahead with it somewhere else.

[ChrisMaddock]

Yes, I think we must have had that conversation elsewhere. Please do go ahead with it, if you're still willing! 😀

[CharliePoole]

I'll try to work on it over the weekend. Considering that I'm not employed and sheltering in place at home, it's amazing how many things I have going on. 😄

[CharliePoole]

This is pretty old but still a problem. Current behavior is as follows...

--	Command	Exception	Message
1	nunit-console junk.txt	InvalidOperationException	TestPackage is not an assembly package
2	nunit-console --explore junk.txt	ArgumentException	Empty path name is not legal
3	nunit-console bad.dll	NUnitEngineException	Unsupported Target Framework: Unmanaged,Version=0.0
4	`nunit-console --explore bad.dll	NUnitEngineException	Unsupported Target Framework: Unmanaged,Version=0.0

There are two kinds of errors here.

First, the type of exception and message is only correct for line 1. Neither of the two files are actually assemblies. Lines 2, 3 and 4 should probably have the same message as 1.

Second, the exceptions should not be thrown to the runner but reported as an error in the test run or the explore. This is consistent with how we handle errors such as file not found and allows us to report in a reasonable manner when more than one assembly is specified on the command-line.

I'm moving this out of the beta1 milestone so we have more time to work on it.