[WIP] Run helix tests on master #6728

HaoK · 2019-01-15T23:40:17Z

17 Skipped projects:

/src/Azure/AzureAD/test/FunctionalTests/Microsoft.AspNetCore.Authentication.AzureAD.FunctionalTests.csproj
/src/Components/test/E2ETest/Microsoft.AspNetCore.Components.E2ETests.csproj
/src/DefaultBuilder/test/Microsoft.AspNetCore.FunctionalTests/Microsoft.AspNetCore.FunctionalTests.csproj
/src/Hosting/test/FunctionalTests/Microsoft.AspNetCore.Hosting.FunctionalTests.csproj
/src/Identity/test/Identity.FunctionalTests/Microsoft.AspNetCore.Identity.FunctionalTests.csproj
/src/Middleware/CORS/test/FunctionalTests/FunctionalTests.csproj
/src/MusicStore/test/MusicStore.E2ETests/MusicStore.E2ETests.csproj
/src/Mvc/test/Microsoft.AspNetCore.Mvc.FunctionalTests/Microsoft.AspNetCore.Mvc.FunctionalTests.csproj
/src/Mvc/test/Mvc.Analyzers.Test/Mvc.Analyzers.Test.csproj
/src/Mvc/test/Mvc.Api.Analyzers.Test/Mvc.Api.Analyzers.Test.csproj
/src/ProjectTemplates/test/ProjectTemplates.Tests.csproj
/src/Security/test/AuthSamples.FunctionalTests/AuthSamples.FunctionalTests.csproj
/src/Servers/IIS/IIS/test/IISExpress.FunctionalTests/IISExpress.FunctionalTests.csproj
/src/Servers/Kestrel/test/InMemory.FunctionalTests/InMemory.FunctionalTests.csproj
/src/Servers/test/FunctionalTests/ServerComparison.FunctionalTests.csproj
/src/SignalR/clients/ts/FunctionalTests/FunctionalTests.csproj
/src/Tools/dotnet-watch/test/dotnet-watch.Tests.csproj

13 Individually Skipped tests

/src/Components/Blazor/Build/test/RuntimeDependenciesResolverTest.cs
(2) /src/DataProtection/Extensions/test/DataProtectionProviderTests.cs
(3) /src/Identity/ApiAuthorization.IdentityServer/test/Configuration/ConfigureSigningCredentialsTests.cs
(3) /src/Identity/ApiAuthorization.IdentityServer/test/Configuration/SigningKeysLoaderTests.cs
(2) /src/Identity/test/Identity.Test/IdentityUIScriptsTest.cs
/src/Servers/Kestrel/Kestrel/test/GeneratedCodeTests.cs
/src/Tools/FirstRunCertGenerator/test/CertificateManagerTests.cs

HaoK · 2019-01-17T06:24:07Z

Yay arcade fix unblocked running helix the real way again, I had to remove the mvc functional tests and the signalr client ts functional tests though since they failed with build errors as part of the helix gather, not sure what's special about those two, I skipped the from building helix payloads for now

src/Components/Blazor/Build/test/RuntimeDependenciesResolverTest.cs

src/Hosting/test/FunctionalTests/Microsoft.AspNetCore.Hosting.FunctionalTests.csproj

test/helix.proj

test/vstest/runtests.sh

BrennanConroy · 2019-01-24T16:48:18Z

I had to remove the mvc functional tests and the signalr client ts functional tests though since they failed with build errors as part of the helix gather

Need any help from SignalR folks to figure this out?

HaoK · 2019-01-24T19:52:41Z

Sure @BrennanConroy that'd be great, you just need to turn BuildHelixPayload back on for the test project in this branch to investigate

HaoK · 2019-01-29T23:48:20Z

Helix tests are finally green including localdb tests as well. Functional tests and tests that rely on the sln are the biggest category of SkipOnHelix tests left

HaoK · 2019-01-30T06:09:03Z

17 test projects are skipped completely on helix (14 functional + 2 mvc analyzers + dotnet watch) + 13 individually skipped tests (mostly looking for sln + cert tests on OSX). Rest of the tests look good now.

natemcmaster

It looks like this is really close. I've left a few nit-picks and questions. The CI is failing normal builds with this change (obviously something to fix first), but other than that, I'd say this is ready to go.

src/SignalR/clients/ts/FunctionalTests/FunctionalTests.csproj

...gnalR/server/Specification.Tests/src/Microsoft.AspNetCore.SignalR.Specification.Tests.csproj

test/xunit/SkipOnHelixAttribute.cs

test/helix.proj

src/Identity/Specification.Tests/src/Microsoft.AspNetCore.Identity.Specification.Tests.csproj

Directory.Build.props

natemcmaster · 2019-01-31T00:22:03Z

Another question - any idea why we don't see the test failures under 'Tests' in AzDO? I thought the Helix SDK was going to report test failures to the UI.

HaoK · 2019-01-31T01:52:49Z

@MattGal do you know what I need to do to get helix test failures to get reported to the AzDo UI that nate is mentioning?

HaoK · 2019-01-31T02:14:59Z

@natemcmaster should we move the helix job to a separate checkbox similar to how pr-fastvalidation used to be instead of running it as part of normal ci to start?

natemcmaster · 2019-01-31T16:36:07Z

should we move the helix job to a separate checkbox similar to how pr-fastvalidation used to be instead of running it as part of normal ci to start?

Yeah, that's probably a good idea while we stabilize Helix. I expect we'll find more flaky tests an account of how many times each test will run. If you move the YML changes to .azure/pipelines/helix-test.yml, I can setup a new AzDO definition.

MattGal · 2019-01-31T17:44:59Z

@natemcmaster I believe to get this you should only need to set EnableAzurePipelinesReporter to true.

Relevent line in props file

HaoK · 2019-01-31T19:55:47Z

@natemcmaster moved the helix job to its own yml file, and also turned on the azure reporting flag. Running time is also approx 20 min per queue, so running all 10 queues takes a while. Matt identified some obvious perf bottlenecks a few weeks ago, I'll see if I can do some of the low hanging ones to speed things up now that things are working

.azure/pipelines/ci.yml

natemcmaster · 2019-01-31T20:14:32Z

Thanks @MattGal!

@HaoK this looks awesome! Obviously there is a handful of work necessary to get all of our tests running in Helix, but I think we should get this merged soon. I'm about to merge #7137 which may have a few conflicts on ci.yml, but otherwise I think this should be good to merge today.

HaoK · 2019-01-31T21:22:42Z

Yeah I'll rebase against master again now, I'm not sure why the normal ci would be failing after I do that since the helix stuff shouldn't be affecting it at all anymore

natemcmaster · 2019-02-01T00:05:29Z

My changes are in master now. Looks like the merge conflict is on FunctionalTests.csproj now.

HaoK · 2019-02-06T09:26:49Z

Almost there, helix tests are all passing after skipping a few new failing tests, @natemcmaster can you add the system access token to the pipeline that helix is trying to access? Hopefully things will finally go green again once it has that.

2019-02-06T08:43:00.3652531Z D:\a\1\s\.nuget\packages\microsoft.dotnet.helix.sdk\2.0.0-beta.19073.6\tools\azure-pipelines\AzurePipelines.props(27,5): 
error : Azure Pipelines SYSTEM_ACCESSTOKEN variable not found. Either the build is not running in Azure Pipelines, or the current step is not configured correctly. 
Please configure this step with the System.AccessToken variable, see: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=vsts&tabs=yaml,batch#systemaccesstoken [D:\a\1\s\test\helix.proj]

pakrym · 2019-02-07T06:02:24Z

Green! 👍

HaoK · 2019-02-07T06:40:21Z

Yeah, guess the azure error reporting thing is causing that new failure, I turned that flag off for now. retriggering the normal ci since that failed now :(

natemcmaster

Awesome, let's do this. Let's open a bug to follow up on re-enabling AzDO reporting.

Last suggestions are nits about code organization, but I don't want to block the PR on this since this has been in progress for so long.

Nit 1 - I'd like to avoid adding a new top-level folder to this repo. We are using the eng/ folder for targets/tasks/scripts etc. eng/helix/ or eng/scripts/ seems like a better place for some of this code.

Nit 2 - put shared source code (SkipOnHelixAttribute.cs) in src/Shared/

HaoK · 2019-02-07T17:18:59Z

Cool, I'll merge and start another PR with the tweaks, and it looks like there's a few flaky tests still too since the helix job still fails like half the time.

BrennanConroy · 2019-02-07T21:06:02Z

Now that this is merged and running on peoples PR's, an email to the team describing what it is, how it works, and how to see failures would be nice.

HaoK · 2019-02-07T22:18:28Z

Good idea I'll send something out this afternoon

MattGal · 2019-02-07T23:02:48Z

Awesome to see this merged you guys, you came a long way in a (relatively) short period of time on this.

HaoK requested a review from natemcmaster as a code owner January 15, 2019 23:40

HaoK requested a review from Tratcher as a code owner January 16, 2019 07:35

HaoK requested review from jkotalik and SteveSandersonMS as code owners January 17, 2019 21:09

HaoK force-pushed the haok/master/helix branch from 2f3ee15 to e8f2c85 Compare January 18, 2019 19:02

natemcmaster reviewed Jan 18, 2019

View reviewed changes

src/Components/Blazor/Build/test/RuntimeDependenciesResolverTest.cs Outdated Show resolved Hide resolved

natemcmaster reviewed Jan 18, 2019

View reviewed changes

src/Hosting/test/FunctionalTests/Microsoft.AspNetCore.Hosting.FunctionalTests.csproj Show resolved Hide resolved

natemcmaster reviewed Jan 18, 2019

View reviewed changes

test/helix.proj Outdated Show resolved Hide resolved

natemcmaster reviewed Jan 18, 2019

View reviewed changes

test/vstest/runtests.sh Outdated Show resolved Hide resolved

HaoK force-pushed the haok/master/helix branch from 89e7c8c to ab7ae5f Compare January 18, 2019 22:48

HaoK force-pushed the haok/master/helix branch from ff07e13 to 2e4316b Compare January 24, 2019 20:45

Eilon added the area-infrastructure Includes: MSBuild projects/targets, build scripts, CI, Installers and shared framework label Jan 25, 2019

HaoK mentioned this pull request Jan 29, 2019

[WIP] Try running tests on helix #6037

Closed

natemcmaster suggested changes Jan 31, 2019

View reviewed changes

natemcmaster reviewed Jan 31, 2019

View reviewed changes

.azure/pipelines/ci.yml Outdated Show resolved Hide resolved

HaoK force-pushed the haok/master/helix branch from 0dd670e to ec6b459 Compare January 31, 2019 21:41

HaoK added 6 commits February 5, 2019 15:13

Update WebHostTests.cs

b1fe864

Update Microsoft.AspNetCore.Hosting.Tests.csproj

5b7b51a

Update WebHostTests.cs

b12f872

Update helix-test.yml

90b7966

Update WebHostTests.cs

4a10294

Update H2SpecTests.cs

e7a3355

HaoK added 11 commits February 6, 2019 01:31

Move xunit runner to csharp targets

4289acc

Update Directory.Build.targets

60de015

Update CSharp.Common.targets

1df0f7e

Update Directory.Build.targets

3649310

Update Directory.Build.targets

fb29366

Update CSharp.Common.targets

04c3571

Update CSharp.Common.props

299e382

Update CSharp.Common.props

9c7d63a

Add xunit ref back

07d28de

Update WebHostTests.cs

b71bd37

Update helix.proj

4c2f588

HaoK added 2 commits February 6, 2019 22:41

Update CertificateManagerTests.cs

d141271

Update CSharp.Common.props

1654d00

natemcmaster approved these changes Feb 7, 2019

View reviewed changes

HaoK merged commit c9499e1 into master Feb 7, 2019

BrennanConroy deleted the haok/master/helix branch February 7, 2019 21:06

pakrym mentioned this pull request Feb 12, 2019

Admin permissions are required to run ANCM tests #7484

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Run helix tests on master #6728

[WIP] Run helix tests on master #6728

HaoK commented Jan 15, 2019 •

edited

Loading

HaoK commented Jan 17, 2019

BrennanConroy commented Jan 24, 2019

HaoK commented Jan 24, 2019

HaoK commented Jan 29, 2019

HaoK commented Jan 30, 2019 •

edited

Loading

natemcmaster left a comment

natemcmaster commented Jan 31, 2019 •

edited

Loading

HaoK commented Jan 31, 2019

HaoK commented Jan 31, 2019

natemcmaster commented Jan 31, 2019

MattGal commented Jan 31, 2019 •

edited

Loading

HaoK commented Jan 31, 2019

natemcmaster commented Jan 31, 2019

HaoK commented Jan 31, 2019

natemcmaster commented Feb 1, 2019

HaoK commented Feb 6, 2019

pakrym commented Feb 7, 2019

HaoK commented Feb 7, 2019

natemcmaster left a comment

HaoK commented Feb 7, 2019

BrennanConroy commented Feb 7, 2019

HaoK commented Feb 7, 2019

MattGal commented Feb 7, 2019

[WIP] Run helix tests on master #6728

[WIP] Run helix tests on master #6728

Conversation

HaoK commented Jan 15, 2019 • edited Loading

HaoK commented Jan 17, 2019

BrennanConroy commented Jan 24, 2019

HaoK commented Jan 24, 2019

HaoK commented Jan 29, 2019

HaoK commented Jan 30, 2019 • edited Loading

natemcmaster left a comment

Choose a reason for hiding this comment

natemcmaster commented Jan 31, 2019 • edited Loading

HaoK commented Jan 31, 2019

HaoK commented Jan 31, 2019

natemcmaster commented Jan 31, 2019

MattGal commented Jan 31, 2019 • edited Loading

HaoK commented Jan 31, 2019

natemcmaster commented Jan 31, 2019

HaoK commented Jan 31, 2019

natemcmaster commented Feb 1, 2019

HaoK commented Feb 6, 2019

pakrym commented Feb 7, 2019

HaoK commented Feb 7, 2019

natemcmaster left a comment

Choose a reason for hiding this comment

HaoK commented Feb 7, 2019

BrennanConroy commented Feb 7, 2019

HaoK commented Feb 7, 2019

MattGal commented Feb 7, 2019

HaoK commented Jan 15, 2019 •

edited

Loading

HaoK commented Jan 30, 2019 •

edited

Loading

natemcmaster commented Jan 31, 2019 •

edited

Loading

MattGal commented Jan 31, 2019 •

edited

Loading