Enterprise Testing System

[davidsh]

This is the first of several PRs that add Enterprise Scenarios Testing capability to
the repo. This PR focusses on Linux which allows for docker containers to be used
in an enterprise network configuration.

I focussed on 2 workflows: 1) The 'dev' workflow, and 2) The PR/CI workflow. The dev
workflow works well since it's using containers in a docker-compose environment along
with volume mounting your current dev's repo enlistment. The PR/CI workflow gives us
an Azure DevOps pipeline to automate verification.

I still need to work with the infra team to add a real pipeline that will run. I can't
do that until this is merged. In the meantime, I have my own DevOps pipeline that verified this PR.

See: https://dev.azure.com/systemnetncl/Enterprise%20Testing/_build/results?buildId=141

I will be linking a follow-up GitHub issue describing the roadmap for building on this system
including adding Windows environments, NTLM protocol, proxies, and other libraries such as
System.Net.Mail and System.Data.SqlClient. Those libraries also use Negotiate/Kerberos/NTLM
enterprise-oriented protocols.

Contributes to:
https://github.com/dotnet/corefx/issues/41652
https://github.com/dotnet/corefx/issues/41489
https://github.com/dotnet/corefx/issues/36896
https://github.com/dotnet/corefx/issues/30150
https://github.com/dotnet/corefx/issues/24707
https://github.com/dotnet/corefx/issues/10041
https://github.com/dotnet/corefx/issues/6606
https://github.com/dotnet/corefx/issues/6161

[davidsh]

cc: @danmosemsft @karelz

[davidsh]

@Tratcher

[davidsh]

fyi @mconnew

[scalablecory]

I'm not familiar enough with Docker to approve those scripts, but the C# looks fine to me.

[wfurt]

LGTM.

[Tratcher]

Can you summarize the network architecture so far? Is this all set up on a single container? Or do you have a separate container for the KDC?
How much more do you think you'd need to do to get a test like this one working? https://github.com/dotnet/runtime/blob/master/src/libraries/System.Net.Security/tests/FunctionalTests/NegotiateStreamKerberosTest.cs#L169

[Tratcher]

TLS/SSL extensively used.

Really? TLS always seemed to be a secondary concern for these systems because of their intranet network topology. They also rely on the design of NTLM, Kerberos, and Negotiate to avoid credential theft without the additional overhead of TLS.

[davidsh]

Thanks for the feedback. This document was written to help explain the overall concept of 'enterprise scenarios' to help developers new to the repo and these scenarios. I'll try to revise the text per your feedback.

[davidsh]

@Tratcher

Is this all set up on a single container? Or do you have a separate container for the KDC?

Currently this PR sets up a docker-compose network of 3 containers: kdc, apachewebserver, linuxclient. That is sufficient to run some tests. Strictly speaking the kdc could be merged together with the webserver but I kept things separate. But the workflow is that the tests run on the "client" machine. This is explained in the README.md file.

How much more do you think you'd need to do to get a test like this one working? https://github.com/dotnet/runtime/blob/master/src/libraries/System.Net.Security/tests/FunctionalTests/NegotiateStreamKerberosTest.cs#L169

For that particular test (which tests the server-side of NegotiateStream), I am already testing that server-side NegotiateStream scenario exactly as part of the new NegotiateStreamLoopbackTest.cs (this PR). It's just that it is doing it using the VirtualNetwork instead of a real TCP socket. I plan to add more NegotiateStream tests in a later PR to demonstrate more end-to-end testing.

I also have implemented another linux docker container that has ASP.NET Core running as a web server. And both Negotiate and NTLM (via Negotiate's Kerberos to NTLM fallback) is working on that now. So, I will add more tests to the HttpClient tests that verify that scenario (client talking to authenticating ASP.NET Core web server). I will add that docker container as part of this "enterprise network" of docker containers.

Can you summarize the network architecture so far?

I currently have the README.md to explain how the dev workflow and CI workflow works. But I plan to write up soon a larger document to explain the architectures.

This PR establishes the current multiple docker container architecture which works well for Linux only. Also, for Linux, this docker-compose set of containers can be run on a dev box. This is useful for tight innerloop development and debugging.

But Windows requires real VMs for Active Directory scenarios. For that, we are now hosting a multi-VM cluster of machines running Active Directory. This runs on a dedicated and funded Azure subscription for network testing. We will install Azure DevOps agents on the client machine on these VMs which will then run these tests automatically in CI for Windows and Windows/Linux scenarios. That is not fully implemented yet.

[Tratcher]

Cool, thanks. I have some cross-machine tests using HttpClient and Kestrel that still need an environment to run in. You may be able to poach from there.
https://github.com/aspnet/AspNetCore/blob/master/src/Security/Authentication/Negotiate/test/Negotiate.FunctionalTest/CrossMachineReadMe.md

[davidsh]

Cool, thanks. I have some cross-machine tests using HttpClient and Kestrel that still need an environment to run in. You may be able to poach from there.

Thanks for permission for "poaching" them. I was planning that. :)

[eiriktsarpalis]

I still need to work with the infra team to add a real pipeline that will run. I can't
do that until this is merged.

You can debug your pipeline files by pushing your branch to the dotnet repo. It's frowned upon in general, but tolerated as long as it addresses infra concerns. It will certainly save you from having to post new PRs if a pipeline issue is discovered.

[eiriktsarpalis]

@davidsh I like your choice of folder structure. I'm wondering if src/libraries/System/Net/Stress might be an appropriate home for stress testing as well.

[eiriktsarpalis]

A few comments, but generally a very good addition. Would recommend testing the pipelines before merging in though.

[davidsh]

You can debug your pipeline files by pushing your branch to the dotnet repo. It's frowned upon in general, but tolerated as long as it addresses infra concerns. It will certainly save you from having to post new PRs if a pipeline issue is discovered.

Thanks for the reminder about branches on the main repo. I chose not to do that for now. But I will do that for some subsequent PRs as I tune the *.yml code for the pipeline. Right now, there is no actual dotnet/runtime pipeline for this *.yml code to run against.

[davidsh]

@davidsh I like your choice of folder structure. I'm wondering if src/libraries/System/Net/Stress might be an appropriate home for stress testing as well.

I'm not sure I understand what you are asking. I don't have anything directly off the repo root of 'src/libraries' paths.

I have the actual tests in a folder called 'EnterpriseTests' that is parallel to 'FunctionalTests' for a particular library. And the setup code for the docker-compose machines/network is in 'src/libraries/Common/tests/System/Net/EnterpriseTests' folder.

Can you clarify what new paths you are proposing for the 'stress' testing?

[eiriktsarpalis]

I have the actual tests in a folder called 'EnterpriseTests' that is parallel to 'FunctionalTests' for a particular library. And the setup code for the docker-compose machines/network is in 'src/libraries/Common/tests/System/Net/EnterpriseTests' folder.

I mistyped, I meant to say the src/libraries/Common/tests/System/Net/EnterpriseTests folder.

[davidsh]

@eiriktsarpalis

I mistyped, I meant to say the src/libraries/Common/tests/System/Net/EnterpriseTests folder.

If you need common configuration/setup for stress tests, then putting them in new, parallel, folder like 'src/libraries/Common/tests/System/Net/StressTests' would be fine.

[safern]

Note that there is a lot of changes going on with our CI and build logic so initially once: #494 is merged you will need to change the way you build stuff in order to include a locally built coreclr, so this build will be broken since we're removing the ability to build from a coreclr package.

There are other changes that we're doing in order to achieve that, i.e: #530, #526

Of course we can sync up and work together to do the required updates in order to achieve that.