No errors reported for adapters that are not loaded because of 'copied from internet' bit on adapter binary

[shyamnamboodiripad]

Description

While debugging a Live Unit Testing bug, I ran into a problem where LUT would discover 0 tests in the solution that customer had shared.

After much head scratching about why LUT discovery works fine for simpler local solutions but doesn't work for the customer's solution (using the exact same version of adapter, target framework etc.), it turns out that test platform will not load adapters that have the 'copied from internet' bit set.

However, no errors / messages are reported via ITestMessageEventHandler.HandleLogMessage or ITestMessageEventHandler.HandleRawMessage. Also, even though LUT discovery fails, the regular Test Explorer window is successfully able to discover tests for the same solution which leads to further confusion.

If I check the below 'Unblock' checkbox on the adapter dll file and click 'OK' then discovery starts working without errors.

• 

Steps to reproduce

1. Create a simple unit test project with a few unit tests and turn on LUT. Make sure LUT works and you see the glyphs next to the unit test methods.
2. Zip and copy the solution (including the 'packages' folder next to the .sln file that contains the test adapter package) to the internet (OneDrive). I don't know if this matters, but in my case the adapter in question was MSTest (v1.1.17).
3. Copy down and unzip the solution from internet to a different machine.
4. Open solution and turn on verbose logging for LUT (Tools -> Options -> LUT -> Logging level).
5. Start LUT via Test -> Live Unit Testing -> Reset Clean.
6. Notice messages in the verbose log that indicate that LUT discovered 0 tests.
7. Notice how Test Explorer window will still discover and display 'nit yet run' entries for the test.

I ended up creating a quick standalone tool to test discovery (copied the code that LUT uses for discovery). You could also repro this much easier by copying just the test adapter dll to the internet and then copying it back under the packages folder - running the below tool with the original test adapter dll will work but running it with the one copied from the internet will result in discovering 0 tests with no errors / messages reported via ITestMessageEventHandler.HandleLogMessage or ITestMessageEventHandler.HandleRawMessage. Compile the below code and run the tool with 0 args to see usage.

// ------------------------------------------------------------------------------------------------------------------------------
// NOTE: Add references to following nuget packages. The versions are the same ones that are used by LUT.
//   <package id="Microsoft.TestPlatform.ObjectModel" version="15.3.0-preview-20170609-03" targetFramework="net461" />
//   <package id = "Microsoft.TestPlatform.TranslationLayer" version="15.3.0-preview-20170609-03" targetFramework="net461" />
// ------------------------------------------------------------------------------------------------------------------------------

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using Microsoft.TestPlatform.VsTestConsole.TranslationLayer;
using Microsoft.VisualStudio.TestPlatform.ObjectModel;
using Microsoft.VisualStudio.TestPlatform.ObjectModel.Client;
using Microsoft.VisualStudio.TestPlatform.ObjectModel.Logging;

internal class Program
{
    private const string DefaultDiscoveryRunSettings = @"
<RunSettings>
    <RunConfiguration>
        <DisableAppDomain>false</DisableAppDomain>
        <TargetPlatform>x86</TargetPlatform>
        <TargetFrameworkVersion>.NETFramework,Version=v4.6.1</TargetFrameworkVersion>
    </RunConfiguration>
</RunSettings>";

    private readonly static string Usage = $@"usage: TestDiscoverer.exe <vstest.console.exe path> <path to test adapter> [<path to discovery run settings xml>] <path to test assembly 1> [<path to test assembly 2> ...]

Default discovery run settings xml (if not provided):
{DefaultDiscoveryRunSettings}";

    static void Main(string[] args)
    {
        try
        {
            if (args.Length < 3)
            {
                Console.WriteLine(Usage);
                return;
            }

            var vsTestConsolePath = Path.GetFullPath(args[0]);
            var vsTestAdapterPath = Path.GetFullPath(args[1]);
            var testAssemblyPaths = args.Skip(2).Select(arg => Path.GetFullPath(arg)).TakeWhile(arg => File.Exists(arg) && Path.GetExtension(arg).ToLowerInvariant() != ".xml");
            var runSettingsXmlPaths = args.Skip(2).Select(arg => Path.GetFullPath(arg)).TakeWhile(arg => File.Exists(arg) && Path.GetExtension(arg).ToLowerInvariant() == ".xml");

            if (!File.Exists(vsTestConsolePath) ||
                Path.GetFileName(vsTestConsolePath).ToLowerInvariant() != "vstest.console.exe" ||
                !File.Exists(vsTestAdapterPath) ||
                !testAssemblyPaths.Any() ||
                runSettingsXmlPaths.Count() > 1)
            {
                Console.WriteLine(Usage);
                return;
            }

            var runSettingsXml = string.Empty;
            if (runSettingsXmlPaths.Any())
            {
                runSettingsXml = File.ReadAllText(runSettingsXmlPaths.Single());
            }
            else
            {
                runSettingsXml = DefaultDiscoveryRunSettings;
            }

            var discoveryEventsHandler = new TestDiscoveryEventsHandler();
            var console = CreateVsTestConsoleWrapper(vsTestConsolePath, new[] { vsTestAdapterPath });
            console.DiscoverTests(testAssemblyPaths, runSettingsXml, discoveryEventsHandler);
        }
        catch (Exception ex)
        {
            Console.WriteLine(ex.ToString());
            Console.WriteLine();
            Console.WriteLine(Usage);
        }
    }

    private static VsTestConsoleWrapper CreateVsTestConsoleWrapper(string vsTestConsolePath, IEnumerable<string> vsTestAdapterPaths)
    {
        VsTestConsoleWrapper vsTestConsoleWrapper;
        vsTestConsoleWrapper = new VsTestConsoleWrapper(vsTestConsolePath);
        vsTestConsoleWrapper.StartSession();
        vsTestConsoleWrapper.InitializeExtensions(vsTestAdapterPaths);
        return vsTestConsoleWrapper;
    }
    
    private sealed class TestDiscoveryEventsHandler : ITestDiscoveryEventsHandler
    {
        public List<TestCase> DiscoveredTestCases { get; } = new List<TestCase>();

        void ITestDiscoveryEventsHandler.HandleDiscoveredTests(IEnumerable<TestCase> discoveredTestCases)
        {
            HandleDiscoveredTests(discoveredTestCases);
        }

        private void HandleDiscoveredTests(IEnumerable<TestCase> discoveredTestCases)
        {
            foreach (var testCase in discoveredTestCases)
            {
                Console.WriteLine($"Discovered test {testCase.FullyQualifiedName}");
                DiscoveredTestCases.Add(testCase);
            }
        }

        void ITestDiscoveryEventsHandler.HandleDiscoveryComplete(long totalTests, IEnumerable<TestCase> lastChunk, bool isAborted)
        {
            if (lastChunk != null)
            {
                HandleDiscoveredTests(lastChunk);
            }

            Console.WriteLine($"Discovery {(isAborted ? "aborted" : "completed")} - {DiscoveredTestCases.Count} tests discovered");
        }

        void ITestMessageEventHandler.HandleLogMessage(TestMessageLevel level, string message)
        {
            Console.WriteLine($"[{level.ToString()}] {message}");
        }

        void ITestMessageEventHandler.HandleRawMessage(string rawMessage)
        {
            Console.WriteLine(rawMessage);
        }
    }
}

Expected behavior

• Discovery APIs should report errors / messages via ITestMessageEventHandler.HandleLogMessage or ITestMessageEventHandler.HandleRawMessage if they are rejecting supplied adapter dlls because those adapters have the 'copied from internet' bit set.
• Ideally, if the TP v2 APIs that LUT uses will reject adapter dlls that have the 'copied from internet' bit set due to security reasons, then the Test Explorer window should have identical behavior (i.e. test explorer should also discover 0 tests).

Actual behavior

• No errors / messages reported via ITestMessageEventHandler.HandleLogMessage or ITestMessageEventHandler.HandleRawMessage when adapter dll is rejected because it has 'copied from internet' bit set.
• LUT fails to discover tests but Test Explorer window succeeds.

Environment

• VS 2017 15.3 Preview 4

[shyamnamboodiripad]

@pvlakshm @Faizan2304 any updates on when this will be fixed?

I just ran into this again when debugging a different Live Unit Testing bug report on VS 15.6 Preview. Discovery was not working on the repro solution and I ended up spending a bunch of time trying to isolate the problem by switching between different xunit versions etc. before remembering about this issue and realizing that the problem was caused because of the same 'copied from internet' flag.

It would be great to fix this so that this doesn't trip me (and others) up again six months later when this issue is no longer fresh in my mind 😄

[pvlakshm]

Contributions welcome. This is marked up-for-grabs.