add a 'Run Flaky Tests' step that does not fail the build

[analogrelay]

This adds a step to all the builds, immediately following the main build step, which runs all the tests marked "Flaky". If this step fails, it does not fail the build. However, the TRX files and logs should be collected and published, so the Tests view should show the failing flaky tests.

Once this is working, I can fairly easily port it to EF which is using Arcade. AspNetCore will need something a little different.

Closes #1224

[analogrelay]

Let's see how/if this works :)

[dougbu]

Looks good as long as the validation checks are passing 😺

[dougbu]

Aren't -trait "Flaky:AzP:All=true" and -notrait "Flaky:AzP:All=true" unnecessary in these two lines?

[analogrelay]

Not quite. The logic isn't as simple as just setting the OS-based flag. The Flaky attribute is trying to be a little agnostic as to the "filters" in use (since we may add other pivots like architectures, etc.), so if you say it's flaky on Darwin, like so:

[Flaky("...", AzurePipelines.macOS)]

Then we set the xUnit trait: Flaky:AzP:OS:Darwin=true

If you say it's flaky everywhere, like so:

[Flaky("...", AzurePipelines.All)]

Then we set a different xUnit trait: Flaky:AzP:All=true since we don't want the Flaky attribute logic to have to enumerate every possible pivot and filter.

[analogrelay]

I did it this way so that the complexity could live in the build script rather than having to keep updating the Flaky attribute whenever we add new pivots or new pivot pieces. If we wanted to simplify this logic, we'd need Flaky to inject traits representing all possible pivots for which the test is flaky which makes things like AzurePipelines.All tricky.

[analogrelay]

Looks like it's somewhat working. The main AzP summary page does what I'd hoped:

The tests page shows all tests passing, I need to check the scripts that collect test results.

[analogrelay]

Ahh, they didn't run at all:

The application to execute does not exist: '/home/vsts/.nuget/packages/xunit.runner.console/2.4.1/tools/netcoreapp2.0/xunit.console.dll'
=== COMMAND LINE ===
"/home/vsts/work/1/s/.dotnet/dotnet" exec --depsfile "/home/vsts/work/1/s/artifacts/bin/Microsoft.Extensions.Configuration.Tests/Debug/netcoreapp3.0/Microsoft.Extensions.Configuration.Tests.deps.json" --runtimeconfig "/home/vsts/work/1/s/artifacts/bin/Microsoft.Extensions.Configuration.Tests/Debug/netcoreapp3.0/Microsoft.Extensions.Configuration.Tests.runtimeconfig.json"  "/home/vsts/.nuget/packages/xunit.runner.console/2.4.1/tools/netcoreapp2.0/xunit.console.dll" "/home/vsts/work/1/s/artifacts/bin/Microsoft.Extensions.Configuration.Tests/Debug/netcoreapp3.0/Microsoft.Extensions.Configuration.Tests.dll" -noautoreporters -xml "/home/vsts/work/1/s/artifacts/TestResults/Debug/Flaky/Microsoft.Extensions.Configuration.Tests_netcoreapp3.0_x64.xml" -html "/home/vsts/work/1/s/artifacts/TestResults/Debug/Flaky/Microsoft.Extensions.Configuration.Tests_netcoreapp3.0_x64.html" -trait "Flaky:All=true" -trait "Flaky:AzP:All=true" -trait "Flaky:AzP:OS:Linux=true" > "/home/vsts/work/1/s/artifacts/log/Debug/Flaky/Microsoft.Extensions.Configuration.Tests_netcoreapp3.0_x64.log" 2>&1

[analogrelay]

GitHub Checks:

AzP Tests Page:

Unix builds are failing due to a permissions error (grumble...). I'll fix it

Looks good to me as long as Unix follows along after the permissions are fixed.

I'm going to leave the FlakyAttributeTests in place for now. They do mean that Extensions will always have failing flaky tests, because they are in fact tests that test the FlakyAttribute.

[ryanbrandenburg]

RE Tests that test FlakyAttributeTests: RAAS will have to start having a list of tests which it won't fail for (because they're intended to fail.).

[analogrelay]

Ah, good point. Ideally it should just be ignoring all tests from the flaky test pass, but I don't think you get that much detail when querying the APIs you're querying.

[ryanbrandenburg]

I don't follow why we would want RaaS to ignore tests from the Flaky pass. RaaS should triage them like normal and update the affected issues. Isn't that the whole point of the Flaky attribute, to still get stats about how often these fail, without blocking?

[analogrelay]

Assuming we think it will work well, that should be fine. My concern is that a flaky test will already have an issue tracking it (the attribute requires that an issue be filled in). Will RaaS be able to find and update that issue? Or will we end up with two issues?

[ryanbrandenburg]

https://github.com/aspnet/AspNetCore-Internal/issues/1867

[analogrelay]

Merging this now.

@Eilon - Be aware that RaaS may file issues in AspNetCore-Internal for the FlakyAttributeTest tests because it doesn't know they are expected to fail, and RaaS looks at all failing tests, which includes these. Of course, RaaS only runs on failing builds in master right now and those are much rarer in Extensions so they may never come up, but I wanted to bring it up.

@ryanbrandenburg is planning to fix that but we don't want to block this PR on that change. We can just disregard those issues when they appear.