Need test for mandatory client cert auth from router for reencrypt case

[richm]

For the router reencrypt test, client certs/keys are allowed, but do not appear to be used.
https://github.com/openshift-qe/v3-testfiles/blob/master/routing/reencrypt/route_reencrypt-path.json#L14

This sets up the route with cert and key. However, the /etc/Caddyfile in the caddy-docker pod looks like this:

# cat /etc/Caddyfile 
:8443 {
 tls /srv/certs/example_wildcard_chain.pem /srv/certs/example_wildcard.key
 root /srv/publics
 browse /test
}
:8080 {
 root /srv/public
 browse /test
}

The Caddy documentation shows this:

https://caddyserver.com/docs/tls

Advanced users may open a settings block for more control, optionally specifying their own certificate and key:

tls [cert key] {
    protocols min max
    ciphers   ciphers...
    clients   [request|require|verify_if_given] clientcas...
    load      dir
    max_certs limit
    key_type  type
    dns       provider
}

I believe clients should be configured like this to require client certs:

   clients require /srv/certs/example_ca.pem

or wherever the example_ca cert is stored.

I think the lack of testing is masking a bug in the router code in openshift, which may have been fixed in 1.4. In 1.3 /var/lib/haproxy/conf/haproxy.config looks like this:

# Secure backend which requires re-encryption

backend be_secure_logging_route-reencrypt
  mode http
  option redispatch

  balance leastconn

  timeout check 5000ms
  http-request set-header X-Forwarded-Host %[req.hdr(host)]
  http-request set-header X-Forwarded-Port %[dst_port]
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header Forwarded for=%[src];host=%[req.hdr(host)];proto=%[req.hdr(X-Forwarded-Proto)]
  cookie 6c8e522e725165e51ee43f0d2843ad5d insert indirect nocache httponly secure

    server 60e1ea32b4066dfcf5a22142f3087dfa 10.1.0.6:8443 ssl check inter 5000ms verify required ca-file /var/lib/haproxy/router/cacerts/logging_route-reencrypt.pem cookie 60e1ea32b4066dfcf5a22142f3087dfa weight 100

Note the server line. I'm not sure which haproxy documentation is applicable to the haproxy version in OSE 3.3. But I did find this: https://www.haproxy.com/doc/aloha/7.0/haproxy/tls.html#configuring-haproxy-for-ssl-tls

crt <pem-file> :

    Designates the file containing a client certificate and its associated private key. HAProxy will use it if the server asks for a client certificate.

I do not see a "crt" directive in the "server" line above. So I don't know how openshift is supposed to use the cert and key specified in the reencrypt route configuration.

[bmeng]

@richm Thanks for your feedback.
I have updated the caddy-docker image to make it require clientca, the Caddyfile looks like below:

:8443 {
 tls /srv/certs/example_wildcard.crt /srv/certs/example_wildcard.key {
  clients require /srv/certs/ca.pem
 }
 root /srv/publics
 browse /test
}
:8080 {
 root /srv/public
 browse /test
}

And after I create reencrypt route with the image above. The haproxy.config shows as the same what you have pasted.

backend be_secure_u1p1_route-reencrypt
  mode http
  option redispatch  
  balance leastconn  

  timeout check 5000ms
  http-request set-header X-Forwarded-Host %[req.hdr(host)]
  http-request set-header X-Forwarded-Port %[dst_port]
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header Forwarded for=%[src];host=%[req.hdr(host)];proto=%[req.hdr(X-Forwarded-Proto)]

  cookie d2414779495cf6fbcf38bab93d1053a9 insert indirect nocache httponly secure

    server 7b6079c65fe6fc12f9d8213b323cceb2 10.129.0.196:8443 ssl check inter 5000ms verify required ca-file /var/lib/haproxy/router/cacerts/u1p1_route-reencrypt.pem cookie 7b6079c65fe6fc12f9d8213b323cceb2 weight 100

And when I try to access the backend via the route, I got failed.

$ curl --resolve sticky-reen.example.com:443:10.66.140.165 https://sticky-reen.example.com --cacert ca.pem 
curl: (51) Unable to communicate securely with peer: requested domain name does not match the server's certificate.
$ curl --resolve sticky-reen.example.com:443:10.66.140.165 https://sticky-reen.example.com --cacert ca.pem --cert example_wildcard.pem --key example_wildcard.key 
curl: (51) Unable to communicate securely with peer: requested domain name does not match the server's certificate.

When I test the image on local, it works.

$ curl https://www.example.com:8443/ --cacert certs/ca.pem --cert certs/example_wildcard.pem --key certs/example_wildcard.key 
Hello-OpenShift-1 https-8443

Looks like the --cert and --key parameter is required when the clients require /srv/certs/ca.pem added to the caddyfile.

I am not sure if this is the bug you mentioned in the router code.

Here is the definition of the reencrypt route.
https://docs.openshift.org/latest/architecture/core_concepts/routes.html#secured-routes

If I got your point correctly, I think I can file bug or contact the devel for help.

[richm]

$ curl --resolve sticky-reen.example.com:443:10.66.140.165 https://sticky-reen.example.com --cacert ca.pem
curl: (51) Unable to communicate securely with peer: requested domain name does not match the server's certificate.

What is the sticky-reen.example.com server cert subject DN? subjectAltName?

Looks like the --cert and --key parameter is required when the clients require /srv/certs/ca.pem added to the caddyfile.

I think it was really my misunderstanding of how to make the re-encrypt route work when the server requires client cert. I could not figure out how to configure the route to use client cert auth when re-encrypting the connection from the router to the server. That should be an RFE for OpenShift and/or Kubernetes.

[bmeng]

Here is the server cert info:

$ openssl x509 -in example_wildcard.pem -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 14 (0xe)
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, ST=SC, L=Default City, O=Default Company Ltd, OU=Test CA, CN=www.exampleca.com/emailAddress=example@example.com
        Validity
            Not Before: May 25 08:12:58 2016 GMT
            Not After : May 23 08:12:58 2026 GMT
        Subject: CN=*.example.com, ST=BJ, C=CN/emailAddress=bmeng@redhat.com, O=RH, OU=OS
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (1024 bit)
                Modulus:
                    00:a0:11:cd:c3:b6:d2:57:20:fa:a0:df:7b:dc:40:
                    11:d7:b6:75:b5:22:8f:d2:d6:72:55:ff:55:f1:a9:
                    d1:78:72:77:37:ff:38:39:8b:e5:cf:c8:cb:f7:31:
                    80:ab:e8:a3:b0:d9:4e:e3:d8:b1:7a:d2:ef:3f:6e:
                    b8:b2:38:4b:8b:2d:aa:3f:e7:db:52:ab:60:8c:03:
                    74:85:ef:68:e7:3a:09:34:d0:c6:a5:04:db:42:d9:
                    d3:c2:e4:47:c0:4b:ad:ae:0b:6e:7e:e5:86:54:60:
                    6b:23:8c:0a:57:dd:50:be:3b:e0:2b:f4:4b:83:a5:
                    0d:2d:3b:89:d6:c3:d6:45:f3
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
    Signature Algorithm: sha1WithRSAEncryption
         c2:11:81:ef:1d:5d:6c:99:26:cb:de:d5:ac:9c:9e:20:71:77:
         25:94:32:83:3d:b4:f0:43:bd:f6:40:94:b2:c3:5a:e6:ee:a1:
         a1:17:2e:88:f2:df:01:87:1b:30:c6:39:6a:ef:60:7c:26:3d:
         f5:83:35:ff:88:96:bc:5e:b6:22:0e:89:7c:4a:7c:6b:53:b0:
         eb:aa:74:90:9c:6f:3d:f2:ef:45:a2:20:2c:e2:5f:99:bb:76:
         a7:a8:19:f9:c6:85:fe:98:b3:af:cb:d3:f8:5e:70:72:f1:c3:
         ce:91:96:19:49:9c:7e:d6:d1:e0:77:a9:e4:fb:bd:ce:f2:b4:
         3c:5c:44:59:6f:84:5c:60:00:3d:ce:17:51:fc:5c:a2:56:b9:
         a8:21:ce:eb:34:ac:33:e5:42:f8:84:3d:89:18:7e:65:01:30:
         d2:e5:fd:d6:d6:11:53:55:df:00:9a:91:a6:8c:a1:88:35:db:
         2e:29:d5:17:78:a6:05:b7:0f:05:d6:61:b9:c0:5f:92:b5:4d:
         bb:6e:00:b5:af:9a:23:88:70:66:ed:2d:3c:0a:a8:8e:a9:74:
         f2:71:e1:ba:3e:cd:bc:0e:ba:8f:d0:f6:56:e3:45:c6:46:00:
         4a:db:0c:a0:38:39:fa:53:0a:ba:6e:fe:a0:03:5d:f9:c9:54:
         60:ae:59:4a

[richm]

I think there is a problem with the cert.

CN=*.example.com, ST=BJ, C=CN/emailAddress=bmeng@redhat.com, O=RH, OU=OS

It looks as though this is in the wrong order. For one, the order is different than the Issuer, which has CN as the righmost component. For another, here is what a working server cert looks like:

C=US, ST=NM, L=ABQ, O=CO, CN=f13x8664.testdomain.com

It is the CN value that the client side uses to check the server cert for MITM. This CN value should be the rightmost value in the Subject in order for the client to be able to use it for checking.

[akostadinov]

@richm how do you obtain info about your cert? It seems that openssl text output does not necessarily match actual order. It can change using different options. For example:

$ openssl x509 -in example_wildcard.pem -text -noout -nameopt RFC2253
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 14 (0xe)
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: emailAddress=example@example.com,CN=www.exampleca.com,OU=Test CA,O=Default Company Ltd,L=Default City,ST=SC,C=US
        Validity
            Not Before: May 25 08:12:58 2016 GMT
            Not After : May 23 08:12:58 2026 GMT
        Subject: OU=OS,O=RH,emailAddress=bmeng@redhat.com,C=CN,ST=BJ,CN=*.example.com
<...>

I Believe @jianlinliu has generated the certificates by OpenSSL and I doubt format is invalid. What I would suspect is that the route is not using the desired certificate for some reason in the example above.

One can see actual certificate returned by server in two easy ways:

$ curl -v https://google.com
<...>
* SSL connection using TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* 	subject: CN=*.google.com,O=Google Inc,L=Mountain View,ST=California,C=US
* 	start date: Mar 01 14:47:44 2017 GMT
<...>

btw you can see the google cert also appears with CN in the beginning but I believe this is only output formatting. One has to look lower level at cert to check actual order.

Also full server chain can be extracted via openssl:

# retrieve cert chain
openssl s_client -showcerts -connect www.example.com:443 -prexit </dev/null
  # for SNI:
openssl s_client -showcerts -servername www.example.com -connect www.example.com:443 -prexit </dev/null

I think first thing is to check whether the route is returning the expected certificate to client. And only then check whether client cert validation is performed.

[richm]

I used openssl x509 -in servercert.pem -text - in my case, both Issuer and Subject are printed with the CN being the rightmost part of the DN. So I guess that isn't the issue here.