Run automated tests on debug builds

[halter73]

My understanding is that our CI servers now only run automated tests against release builds. If that's the case, this means that the many Debug.Asserts throughout the AspNetCore repo are pretty worthless since we can only ever see an assertion failures by running tests or samples locally even the Debug.Asserts in the test projects themselves.

I think the best solution is to run our automated tests against debug builds. I'm not asking for all our testing to be done against debug builds, but nightly tests of debug builds would probably be a good idea.

@davidfowl suggested using critical logs to replace Debug.Asserts. That was definitely the right approach for #11624, but in a lot of places we assert currently we don't have a logger. There's also the problem that not all our tests will fail given a critical log, but a lot will.

Changing Debug.Asserts to Trace.Asserts seems like a somewhat reasonable idea, but not only does that create performance concerns, it could also potentially introduce new DoS vectors as @benaadams rightfully pointed out.

[benaadams]

Run automated tests on debug builds

I'd go for this option; at least one (e.g. Windows); preferably two (e.g. Windows and Linux); while keeping the current release testing that runs.

[analogrelay]

I think the best solution is to run our automated tests against debug builds.

This is also relatively easy to do in parallel by simply adding jobs to the YAML of DOOM. It theoretically would magnify existing flakiness slightly by giving flaky tests and extra couple chances to fail, but we need to drive those down anyway.

We should also make sure we know the behavior and diagnosability of Debug.Assert in our test code. In the past it's failed in hard-to-understand ways ("Test Run Aborted" errors, etc.).

suggested using critical logs to replace Debug.Asserts.

Why not both? We could have a helper that writes a critical log (for Release) and does a Debug.Assert (for Debug).

[BrennanConroy]

We run on release and debug in Extensions so there is prior art
https://github.com/aspnet/Extensions/blob/8c6d69570722ee4cd6583cbb4af2a9c74c8c10dc/azure-pipelines.yml#L148

[halter73]

@anurse Can we define tests that run nightly instead of on every PR/checking using the YAML of DOOM?

[halter73]

I'm looking at signalr-daily-tests.yml, but I don't see the trigger.

[halter73]

I see the comment about scheduled triggers, but I don't see any "schedules:" in any yaml files. Do I need an AzDo admin account?

https://github.com/aspnet/AspNetCore/blob/master/.azure/pipelines/signalr-daily-tests.yml#L1-L2

[analogrelay]

Schedules in YAML didn't exist when we made the SignalR daily tests.

[dougbu]

Might be worthwhile but sounds more like a DoI issue

[dougbu]

Adding @JunTaoLuo as an assignee. Suggest focusing on daily aspnetcore-ci and aspnetcore-helix-matrix runs that override the Configuration value just in that scheduled build.

[dougbu]

Note obvious answer of adding a variable override to a schedules entry doesn't work: https://docs.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&tabs=schema%2Cparameter-schema#scheduled-trigger