.NET Core 2.1 SocketsHttpHandler does not use Negotiate / SPNego

[CJHarmath]

Overview

While testing PowerShell Core 6.1 I ran into an issue with not being able to authenticate to a Kerberized REST API running on Linux unless I disable SocketsHttpHandler.
PowerShell/PowerShell#7801

It seems like that when the server responds with both Negotiate and NTLM, the SocketsHttpHandler picks NTLM which in my case results in a 401 as the service in question is really expecting Negotiate / SPNego and is not working with NTLM.

As requested by @karelz in https://github.com/dotnet/corefx/issues/30166
I've reproduced it on the daily builds without PowerShell Core involved and same results, so submitting a new issue for this.

Expected result

When server sends multiple auth schemes like Negotiate and NTLM, pick the strongest one which in this case is Negotiate.

Dotnet Info

C:\dev\test\httpclient-spnego\test2>dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   2.1.403-servicing-009270
 Commit:    def6c5f48d

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.15063
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\2.1.403-servicing-009270\

Host (useful for support):
  Version: 2.1.5-servicing-26911-03
  Commit:  efdba896f7

.NET Core SDKs installed:
  2.1.201-preview-007614 [C:\Program Files\dotnet\sdk]
  2.1.202 [C:\Program Files\dotnet\sdk]
  2.1.400 [C:\Program Files\dotnet\sdk]
  2.1.402 [C:\Program Files\dotnet\sdk]
  2.1.403-servicing-009270 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.5-rtm-31008 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.5-rtm-31008 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.5-servicing-26911-03 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

Example repro

var handler = new HttpClientHandler
{
    UseDefaultCredentials = true,
    AllowAutoRedirect = true,
};
using (var client = new HttpClient(handler))
{
    var res = client.SendAsync(new HttpRequestMessage(HttpMethod.Get, uri)).GetAwaiter().GetResult();
    System.Console.WriteLine(res);
}

Result: 401

HTTP traffic from packet capture

GET / HTTP/1.1
Host: mykerberossite.lab.local

HTTP/1.1 401 Unauthorized
Date: Mon, 17 Sep 2018 21:31:42 GMT
Server: Apache-Coyote/1.1
WWW-Authenticate: Negotiate
WWW-Authenticate: NTLM
Content-Length: 0

GET / HTTP/1.1
Authorization: Negotiate ****
Host: mykerberossite.lab.local

HTTP/1.1 401 Unauthorized
Date: Mon, 17 Sep 2018 21:31:42 GMT
Server: Apache-Coyote/1.1
WWW-Authenticate: NTLM
Content-Length: 0

Workaround is to disable SocketsHttpHandler

AppContext.SetSwitch("System.Net.Http.UseSocketsHttpHandler", false);
var handler = new HttpClientHandler
{
    UseDefaultCredentials = true,
    AllowAutoRedirect = true,

};
using (var client = new HttpClient(handler))
{
    var res = client.SendAsync(new HttpRequestMessage(HttpMethod.Get, uri)).GetAwaiter().GetResult();
    System.Console.WriteLine(res);
}

result: 200

HTTP Traffic

GET / HTTP/1.1
Connection: Keep-Alive
Host: mykerberosite.lab.local

HTTP/1.1 401 Unauthorized
Date: Mon, 17 Sep 2018 21:30:27 GMT
Server: Apache-Coyote/1.1
WWW-Authenticate: Negotiate
WWW-Authenticate: NTLM
Content-Length: 0
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive

GET / HTTP/1.1
Connection: Keep-Alive
Host: mykerberossite.lab.local
Authorization: Negotiate ***

HTTP/1.1 200 OK
Date: Mon, 17 Sep 2018 21:30:27 GMT
Server: Apache-Coyote/1.1
WWW-Authenticate: Negotiate ***
Cache-Control: no-cache
Expires: -1
Content-Type: text/plain;charset=UTF-8
Content-Length: 103
Keep-Alive: timeout=5, max=99
Connection: Keep-Alive

[davidsh]

It seems like that when the server responds with both Negotiate and NTLM, the SocketsHttpHandler picks NTLM which in my case results in a 401 as the service in question is really expecting Negotiate / SPNego and is not working with NTLM.

Your attached output from the wire trace doesn't show that. The client-side HTTP is picking Negotiate.

GET / HTTP/1.1
Authorization: Negotiate ****
Host: mykerberossite.lab.local

However, the server is still not authenticating and returning back a 401.

[CJHarmath]

Your attached output from the wire trace doesn't show that. The client-side HTTP is picking Negotiate.

Hmm, could be a copy paste error. Will re-test on Monday and update.

[CJHarmath]

Hey, sorry for the slow response...

So I've checked again and it's indeed using NTLM with that Negotiate response ( NTLM_NEGOTIATE).
It was just not obvious with my *-ed out HTTP traffic as it didn't show the size of that blob.
NTLM is way smaller than Kerberos so you can just eye ball it.

Wireshark can actually dissect and show that it's NTLMSSP_NEGOTIATE as show below.

And here is the snip when I disable SocketsHttpHandler

[davidsh]

So I've checked again and it's indeed using NTLM with that Negotiate response ( NTLM_NEGOTIATE).

So, what you're really saying is that the HTTP stack correctly responded with Negotiate scheme. But Negotiate ended up "negotiating" NTLM instead of Kerberos.

That happens a lot when the requirements for a valid Kerberos infrastructure don't exist. For example, if the client machine is not joined to the Windows Active Directory (or Linux Kerberos) domain of the server, or timestamps aren't matching etc.

In your example, the server is a Linux machine using Kerberos. I'm assuming that the Linux machine is also the Kerberos ticketing server, or is there a separate machine for that?

Usually, this kind of problem is a configuration problem with the machines and not a problem with the client-side HTTP stack since it is picking Negotiate scheme correctly.

cc: @wfurt @geoffkizer @karelz

[CJHarmath]

TLDR: Unlike the rest of the Windows web clients (browsers, .NET full, etc) SocketsHTTPHandler is not canonicalizing the given host when trying to request the SPN which breaks Kerberos if the Url has a CNAME and the SPN is only on the DNS A record of the host.

Details
While writing a pretty long reply linking to docs on MSDN about how Kerberos is implemented on windows in terms of SPNego's selection of auth mechanisms, etc etc and preparing the example traces I just found the issue why it's not working.

phew.

This is what's happening in very high level when it's working as expected ( after purged krb tickets and flushed DNS cache - which actually helped me nailing it down - always purge and always flush!)

• DNS query of HOST
• HTTP GET
• HTTP Response with 401 Negotiate header
• DNS lookup of the host to get the A record
• KRB5 TGS_REQ to AD DC for HTTP/DNS_A
• KRB5 TGS-REP with referral to the Linux KDC realm
• DNS lookup for SRV _kerberos....
• KRB5 TGS_REQ to Linux KDC for HTTP/DNS A
• KRB5 TGS_REP with the service ticket
• HTTP GET with Negotiate with that ticket
• HTTP Response with 200

And this is what's happening with SocketsHttpHandler

• DNS query of HOST
• HTTP GET
• HTTP Response with 401 Negotiate header
• KRB5 TGS_REQ to AD DC for HTTP/HOSTNAME_AS_IS ( which can be a CNAME )
• KRB5 ERROR - principal unknown
• HTTP GET NTLMSSP_NEGOTIATE
• HTTP RESPONSE 401

So the reason SocketsHTTPHandler is not working is because it's trying to find the SPN for the CNAME instead of canonicalizing it with a forward lookup.
While Windows SSPI is not canonicalizing, all applications are. IE, Firefox, Chrome, .NET web clients, etc are all canonicalizing.
i.e.: coming from .NET Full this is a breaking change.

So SocketsHTTPHandler would need to stand in line and stay consistent.
Perhaps allow a flag to not canonicalize but have the default to do whatever IE,Chrome, etc .NET full web is doing.

Linux
On Linux this is configured via krb5.conf so SocketsHTTPHandler should honor that setting.
rdns = true|false

[davidsh]

TLDR: Unlike the rest of the Windows web clients (browsers, .NET full, etc) SocketsHTTPHandler is not canonicalizing the given host when trying to request the SPN which breaks Kerberos if the Url has a CNAME and the SPN is only on the DNS A record of the host.

Thanks for the additional details.

SocketsHttpHandler on Windows uses the Windows SSPI libraries for doing Negotiate and NTLM protocols. I do not think it is something that is directly controllable by SocketsHttpHandler. It would need to be investigated further.

We have tested Linux clients against a Windows server/ActiveDirectory domain. We've demonstrated that Negotiate scheme will use Kerberos if all the machines are configured properly. See: #26418.

But we have not extensively tested a Windows client using Negotiate (Kerberos) into a Linux server environment.

[mattpwhite]

In Linux, canonicalization is controlled by configuration knobs in krb5.conf. Applications are explicitly not supposed to do any kind of canonicalization there and rely on the GSSAPI implementation and its configuration.

In Windows, SSPI does not canonicalize, and AFAIK, there are no global knobs to turn. But many applications, including all of the major browsers, WinHTTP (and thus legacy .NET HTTP clients), and most (but not all) OS components do forward canonicalize CNAMEs prior to calling into SSPI. People/things now generally expect that behavior, witness the outcry when Chrome accidentally stopped pre-canonicalizing recently: https://bugs.chromium.org/p/chromium/issues/detail?id=872665

[CJHarmath]

But we have not extensively tested a Windows client using Negotiate (Kerberos) into a Linux server environment.

My guess is that this will reproduce on windows->windows as well ( haven't tested it yet).
i.e.: Setup IIS with negotiate:kerberos as the only windows auth provider. Configure the Restrict NTLM GPO, then have an SPN for for the A and use a CNAMe to access it.
If configured correctly SockesHttpHandler will break as it will try the wrong SPN then can't use NTLM while the legacy .NET client will work as is.

[davidsh]

But many applications, including all of the major browsers, WinHTTP (and thus legacy .NET HTTP clients), and most (but not all) OS components do forward canonicalize CNAMEs prior to calling into SSPI.

So, this scenario does work using WinHttpHandler (WinHTTP) on the client-side? So, did you turn off SocketsHttpHandler (via AppContext switch for example) and demonstrate that the scenario works?

See: https://github.com/dotnet/core/blob/master/release-notes/2.1/2.1.0.md

Networking Performance
You can use one of the following mechanisms to configure a process to use the older HttpClientHandler:
From code, use the AppContext class:
AppContext.SetSwitch("System.Net.Http.UseSocketsHttpHandler", false);
The AppContext switch can also be set by config file.
The same can be achieved via the environment variable DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER. To opt out, set the value to either false or 0.

If this works with WinHttpHandler, then it should be possible to fix it for SocketsHttpHandler. WinHttpHandler uses native WinHTTP which uses the same Windows SSPI libraries as SocketsHttpHandler for doing Negotiate and NTLM.

[davidsh]

@stephentoub

[CJHarmath]

So, this scenario does work using WinHttpHandler (WinHTTP) on the client-side? So, did you turn off SocketsHttpHandler (via AppContext switch for example) and demonstrate that the scenario works?

Yep. see my very first post with the below workaround to make this work
AppContext.SetSwitch("System.Net.Http.UseSocketsHttpHandler", false);

[geoffkizer]

I think we just need to use the canonical host name here (as returned by Dns.GetHostEntry) instead of the hostname in the uri. Correct?

https://github.com/dotnet/corefx/blob/2c8be59ed4405dbdd91777f79763cb2f1384729e/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/AuthenticationHelper.NtAuth.cs#L80

[mattpwhite]

Almost. That would handle CNAMEs and partially qualified names of As (that become fully qualified when the OS resolver appends one of the configured search suffixes). But...

• If you pass an IP to GetHostEntry(), it will reverse it with a PTR lookup. I believe this behavior would be unexpected. Would probably want to be conditional on a failing IPAddress.TryParse.
• GetHostEntry() will potentially use legacy and broadcast based name resolution protocols on Windows. These are not useful for canonicalization and slow down negative responses. A potential optimization would be to pass the relevant flags to the underlying Win32 APIs to not consider LLMNR, NetBIOS.

[davidsh]

@mattpwhite Thank you for the added details regarding CNAMEs and the issues with LLMNR, NETBIOS, etc.

Since you observe the correct behavior in .NET Framework, we will look at that implementation to see where it differs from .NET Core SocketsHttpHandler. That will give us more insight into the correct implementation.