crypto/x509: error parsing large ASN.1 identifiers

[danielraq]

Describe the bug
For a vault container in version> 1.8.5 a the start of the server on openshift 4.9 i get message:

Error initializing service_registration of type kubernetes: error reading certs: x509: invalid certificate policies

It's correlated with: golang/go#49678

Expected behavior
Start of the vault server

Could you please compile vault with the new go version ?

[heatherezell]

We don't have any plans to change the Go version used in 1.8. However, when 1.9.4 is released it will use Go version 1.17.7, which I believe fixes this issue.
We'll have some internal discussions about better ways of potentially handling these sorts of situations in the future. Thanks for the heads-up! :)

[heatherezell]

Hi @danielraq - it looks like this issue doesn't affect Go 1.16 - would you be willing to confirm what version you're on? And also, can you provide more repro steps? Thanks much!

[danielraq]

at the moment i work on vault in 1.8.5 ocp: 4.5 and corporate certificate. Unfortunately i cannot give you our cert

repro steps:

1. start empty vault container on the ocp
2. get issue :)

[cipherboy]

@danielraq could you share your OID and in which certificate field you placed it? See details below for my failed reproduction:

Using the information from the linked ticket

I did:

$ source devvault
++ vault server -dev -log-level=trace -config=/home/cipherboy/devvault/config.hcl -dev-listen-address=0.0.0.0:8200 -dev-root-token-id=devroot
$ vault verison
Vault v1.8.5 (647eccfe0bd5817bdd8628f3c3171402dfc8a8fc)
$ vault secrets enable pki
$ vault write pki/root/generate/internal common_name=example.com
$ vault write pki/roles/example-com no_store=true generate_lease=true policy_identifiers="1.3.6.1.4.1.311.21.8.3719450.11115469.11946914.3506198.8878958.75.1492336001.1138714952"
Key                 Value
---                 -----
certificate         -----BEGIN CERTIFICATE-----
MIIDeDCCAmCgAwIBAgIUaZRK2Xecq22/RJuH1UnHgSDd5AQwDQYJKoZIhvcNAQEL
BQAwFjEUMBIGA1UEAxMLZXhhbXBsZS5jb20wHhcNMjIwMzAxMTM1MjU1WhcNMjIw
MzAxMTQwMzI1WjAUMRIwEAYDVQQDEwlsb2NhbGhvc3QwggEiMA0GCSqGSIb3DQEB
AQUAA4IBDwAwggEKAoIBAQDi619iRuU37mgL1M/aExRc6MOnjI4nGM7YQGBCJywb
NHtVroi1ZThd4N16KM2NBGSGPbvaQ1y1Ji0DiTwbkrcb/ElGaAn/GjSWrgMIX+eS
FB8sS48MaWQKWxNJlDucpSXdiDWi+dVfgeDMr70QP0wtKuJkZGcG0zEt7O9Wcr2C
nV/rg47qZA5BvRMb1Qc4lGq2Z719DRSVUcmSGfVQn3iCUR9T2Qz/6dzG+qW498VZ
1jG9pf5WFbZH0uPOy+OapO4f7oOfeSXX0eNrTLWGc2wlTvpozcMr/XD6Vi8k7KgN
0Rs0dQzmY605G+1cv7ntKhSkWcrqVG1azZ3wJe0MeejNAgMBAAGjgb8wgbwwDgYD
VR0PAQH/BAQDAgOoMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAdBgNV
HQ4EFgQUbojuq+/q7SQntkT4i+sJnHtS3vkwHwYDVR0jBBgwFoAU/6IA5Zaqnu/t
dB4s9/w/Bj/PcJowFAYDVR0RBA0wC4IJbG9jYWxob3N0MDUGA1UdIAQuMCwwKgYo
KwYBBAGCNxUIgeOCGoWmt02F2ZcigdaAFoSd9m5LhcfM+wGEnv3SSDANBgkqhkiG
9w0BAQsFAAOCAQEAqI5+iH55rVEH68MX55RVY77Be3YhJ6lSG9k+En2Jgx8NovEu
jYvGPeZUGvcVl9T4nzcgyYAsfldZZlRUg0JAdA8XpHBAIwxEKudGb3WBgEUvnwRF
qSz0nB+nzeEYPTw6nhbJNT4IWvNPQPg1ObjtouwOGDpDGcFv73Bl8Q1T2g+S5BVG
McdwJ8GGxLLbul/NehZpnl8Oc+DalGbVYyqDSTGcqLvCYBJ+cPviFwL20uI1lCpk
HZHjPPeVrZbbrImzXm9F//Dp98p3hdW41Q1PptzfPWzhnNLXNSSdy5k3Pd6QxgHI
eiL2GrsS4qsyWla25Km3KOPFT3xrUkfgLgqikQ==
-----END CERTIFICATE-----
expiration          1646143405
issuing_ca          -----BEGIN CERTIFICATE-----
MIIDNTCCAh2gAwIBAgIUQF8fd2zGeM0+0QZj85hQY2+Xm6IwDQYJKoZIhvcNAQEL
BQAwFjEUMBIGA1UEAxMLZXhhbXBsZS5jb20wHhcNMjIwMzAxMTM1MjA1WhcNMjIw
NDAyMTM1MjM1WjAWMRQwEgYDVQQDEwtleGFtcGxlLmNvbTCCASIwDQYJKoZIhvcN
AQEBBQADggEPADCCAQoCggEBAOFagmz3FREDih/7H28NU6measM5ZN/FShfXjtUD
5rVZvxoVBPxB0JJ9GXb58S8TJqgMhWfFOA20fuLjuyonfe+9EKyH9eemDyVFV899
DtbvNLtG2X8ToTKZZHLRuggAwvEPnysqjW3waa9BLbSfHwTCjotEy5krrKcqOv+S
VujaxKH9b+sC+pFQS/ijGDnbgXdunfDKUAUh+Vz/0YPg/EilBfc+GYmLMzS4HjjP
94h6eHYALl0qwll7O9aFIDtC4HlFNjpexMyx0uFQPqdk6R0LtMlDaG3YXCH7KMP+
sMHRIFZ/dYQ0A/lJs5H4ijrZhGu7jn1RH+WL3i/jGM4vIcMCAwEAAaN7MHkwDgYD
VR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFP+iAOWWqp7v
7XQeLPf8PwY/z3CaMB8GA1UdIwQYMBaAFP+iAOWWqp7v7XQeLPf8PwY/z3CaMBYG
A1UdEQQPMA2CC2V4YW1wbGUuY29tMA0GCSqGSIb3DQEBCwUAA4IBAQDVxMJKYU+0
gCNm9Igzvzh/Lg8XguMIXbOtmf3vWcMcY4u85S0p5ChTpYq6bOrFL5xga1BlbD5A
Op8AONeOKewlfaPMV0qDMO6dQZ5hfu6qB1nssalqUCN1812g62cZh1ChVXkzfGkE
QZ1nyiWR2mGJax94jloInpkUhiA3GLOCC1dkA29guhKWmwhac0skY/dmfe9lKwGT
+pIdgvxP5IwkFg5a5Nsi4gVC/eQ4O/PWvysu1pwt0SU0QatNxT38GNadSPya8N90
n1+VnJkkWusYgGV4p5F4pY9X6TveAQeMC2T/c9tXBX+8OTviwxmamhGz3NWGLSMw
mIrg3TMPXxVz
-----END CERTIFICATE-----
private_key         -----BEGIN RSA PRIVATE KEY-----
MIIEpQIBAAKCAQEA4utfYkblN+5oC9TP2hMUXOjDp4yOJxjO2EBgQicsGzR7Va6I
tWU4XeDdeijNjQRkhj272kNctSYtA4k8G5K3G/xJRmgJ/xo0lq4DCF/nkhQfLEuP
DGlkClsTSZQ7nKUl3Yg1ovnVX4HgzK+9ED9MLSriZGRnBtMxLezvVnK9gp1f64OO
6mQOQb0TG9UHOJRqtme9fQ0UlVHJkhn1UJ94glEfU9kM/+ncxvqluPfFWdYxvaX+
VhW2R9LjzsvjmqTuH+6Dn3kl19Hja0y1hnNsJU76aM3DK/1w+lYvJOyoDdEbNHUM
5mOtORvtXL+57SoUpFnK6lRtWs2d8CXtDHnozQIDAQABAoIBACBqYzvUyNXOlspp
4i9S3mvPTHPY3k2ze4jk2VtZ4TEZtCyvPBQ9m2m9v6W9wm9Mm1eZ5gXt7YZrJ68D
obSGpuW2inkmT5dDlROZVBGUMa5qdXGMcajiPUpFbGHsbP+Zj+kvlgnsCpEUKprS
1/1ELYuePqz8ssME1FetKxt8VJodwD6jMTzMIhPuLjDHto37JGDC14xuAiZZbLsP
m9qfc+c0MzU7QwFNgBdmumpC0xOEheG5RFigXuQ0KCV3kfDWTd+i/wmuU/ujDzPm
IWORdDjcpT76Iox6zjeLSRmSZ1Bp6euyslFHwGSvujHRwp2jgCGWalWli8vnNhtm
NQ3QDwECgYEA9tA9D2Q2JZGDnuZmrcy68se91ZJZ8FBlT1Ct9lGbIkUWjRVwc4W1
e5al44yWgIxiXzJk2lvnktSZSCUfOTWl/T9LC9ekkT4Cob71j7PDz0KulDJRO+jp
2BGpvxnCQf7u/5aTAbywOQx7V+wyB77WD7rCxh2h/SWlCJae+X8wFK0CgYEA612S
9P8rQop7WeCAXDgDL77G/fn4XMDcbJHqP5PmWlKqBDtHSY+hAkdHM1pW81xrvq9f
YblonUgn7xL1eRvCp256fL0g74zD27TV4cxDqYKDkzt65pGEuup97ywoIdahEsSI
BlnTWLVYzrYsjmBBmGEgIGp6gasmc0L8k0xliKECgYEApz66cYiNbG2rS+yCXXki
oUsaZvNyAHbp/5IDH6tyvQkl29pQReev75xIsuXcQiVer3Vdg93NEOEBMTL+cmzZ
Gr9d4htxB36GmAXnLkY5BGQCGqfyBe+xLdniNX2F9eSDHSxyVzX+D9pWZmDGEhuN
wEDkpQ0FbJ2qtCQke5plAU0CgYEA55oPpueEwRiV4s4IYTZ45D2dgIO+gj8BaBB5
ahB2ZsDbLgjK0Db2X9CV3pFx6E00dNILUJVqvQIo0ZddV8waUjljbGWLCE4C2Kps
oKEon5+fK30nJcVhz7A5NeXEo++YVmhv1ZgEYVEpz1byJzPOuEwymdNNM1WgI2Fy
Otu9OaECgYEAopbM44xKXXbplA/NO96jBnxdTpMhIbTFRK/iqEe3cBjKXbdsL3qk
zUT8m15PJIJlcKO+YfMBWhZ7CQ1+pmBIR1ZujFY1Acesh6qmjoT1maNyuaoT/wLn
m+wiUO/B7TaPH98/b4I1EVAnm5MKPfR5c3MCnlYoVeufE1Ph3b240hw=
-----END RSA PRIVATE KEY-----
private_key_type    rsa
serial_number       69:94:4a:d9:77:9c:ab:6d:bf:44:9b:87:d5:49:c7:81:20:dd:e4:04

Decoding the issued certificate shows that the OID got specified in full (towards the end of the dump).

I then setup a tls-enabled devvault:

$ cat /home/cipherboy/devvault/config.hcl
storage "inmem" {}
disable_mlock = true

listener "tcp" {
  address       = "0.0.0.0:8200"
  disable_clustering = true
  tls_cert_file = "/home/cipherboy/devvault/chain.pem"
  tls_key_file  = "/home/cipherboy/devvault/service-key.pem"
  tls_client_ca_file = "/home/cipherboy/devvault/ca.crt"
}

$ vault server -log-level=trace -config=/home/cipherboy/devvault/config.hcl

Opened, init'd, whole nine yards:

{
  "unseal_keys_b64": [
    "EI54NH/tHppegFRyjKvIJprmruzG/QWVNfYH8uiOWxGJ",
    "J2W/m1tk+JofHeFT3Tq81WeXdpCjQUNYNV5KLrrG/9tO",
    "AtRdsxqEm5pG0HbSoOxJUnKi63e7DVMhNT8IuodEheus"
  ],
  "unseal_keys_hex": [
    "108e78347fed1e9a5e8054728cabc8269ae6aeecc6fd059535f607f2e88e5b1189",
    "2765bf9b5b64f89a1f1de153dd3abcd567977690a3414358355e4a2ebac6ffdb4e",
    "02d45db31a849b9a46d076d2a0ec495272a2eb77bb0d5321353f08ba874485ebac"
  ],
  "unseal_shares": 3,
  "unseal_threshold": 2,
  "recovery_keys_b64": [],
  "recovery_keys_hex": [],
  "recovery_keys_shares": 5,
  "recovery_keys_threshold": 3,
  "root_token": "s.VGiFWv5C05RlUYPvgUdURhHy"
}

$ vault status
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       3
Threshold          2
Unseal Progress    1/2
Unseal Nonce       71b12d5f-12bd-1afb-efd2-1fa8c4fc8178
Version            1.8.5
Storage Type       inmem
HA Enabled         false


$ openssl s_client -connect localhost:8200
CONNECTED(00000003)
Can't use SSL_get_servername
depth=1 CN = example.com
verify error:num=19:self signed certificate in certificate chain
verify return:1
depth=1 CN = example.com
verify return:1
depth=0 CN = localhost
verify return:1
---
Certificate chain
 0 s:CN = localhost
   i:CN = example.com
 1 s:CN = example.com
   i:CN = example.com
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDeDCCAmCgAwIBAgIUaZRK2Xecq22/RJuH1UnHgSDd5AQwDQYJKoZIhvcNAQEL
BQAwFjEUMBIGA1UEAxMLZXhhbXBsZS5jb20wHhcNMjIwMzAxMTM1MjU1WhcNMjIw
MzAxMTQwMzI1WjAUMRIwEAYDVQQDEwlsb2NhbGhvc3QwggEiMA0GCSqGSIb3DQEB
AQUAA4IBDwAwggEKAoIBAQDi619iRuU37mgL1M/aExRc6MOnjI4nGM7YQGBCJywb
NHtVroi1ZThd4N16KM2NBGSGPbvaQ1y1Ji0DiTwbkrcb/ElGaAn/GjSWrgMIX+eS
FB8sS48MaWQKWxNJlDucpSXdiDWi+dVfgeDMr70QP0wtKuJkZGcG0zEt7O9Wcr2C
nV/rg47qZA5BvRMb1Qc4lGq2Z719DRSVUcmSGfVQn3iCUR9T2Qz/6dzG+qW498VZ
1jG9pf5WFbZH0uPOy+OapO4f7oOfeSXX0eNrTLWGc2wlTvpozcMr/XD6Vi8k7KgN
0Rs0dQzmY605G+1cv7ntKhSkWcrqVG1azZ3wJe0MeejNAgMBAAGjgb8wgbwwDgYD
VR0PAQH/BAQDAgOoMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAdBgNV
HQ4EFgQUbojuq+/q7SQntkT4i+sJnHtS3vkwHwYDVR0jBBgwFoAU/6IA5Zaqnu/t
dB4s9/w/Bj/PcJowFAYDVR0RBA0wC4IJbG9jYWxob3N0MDUGA1UdIAQuMCwwKgYo
KwYBBAGCNxUIgeOCGoWmt02F2ZcigdaAFoSd9m5LhcfM+wGEnv3SSDANBgkqhkiG
9w0BAQsFAAOCAQEAqI5+iH55rVEH68MX55RVY77Be3YhJ6lSG9k+En2Jgx8NovEu
jYvGPeZUGvcVl9T4nzcgyYAsfldZZlRUg0JAdA8XpHBAIwxEKudGb3WBgEUvnwRF
qSz0nB+nzeEYPTw6nhbJNT4IWvNPQPg1ObjtouwOGDpDGcFv73Bl8Q1T2g+S5BVG
McdwJ8GGxLLbul/NehZpnl8Oc+DalGbVYyqDSTGcqLvCYBJ+cPviFwL20uI1lCpk
HZHjPPeVrZbbrImzXm9F//Dp98p3hdW41Q1PptzfPWzhnNLXNSSdy5k3Pd6QxgHI
eiL2GrsS4qsyWla25Km3KOPFT3xrUkfgLgqikQ==
-----END CERTIFICATE-----
subject=CN = localhost

issuer=CN = example.com

---
No client certificate CA names sent
Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2329 bytes and written 379 bytes
Verification error: self signed certificate in certificate chain
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 19 (self signed certificate in certificate chain)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_128_GCM_SHA256
    Session-ID: 0CCB904DF38345068A6940851F28B2633F40094FC0BB46D7F6D922B05BD41DCE
    Session-ID-ctx: 
    Resumption PSK: 3FE8944B1728CA98275AEE76FE1DB669157F43BB5775207CFDDA85756EC732DF
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 604800 (seconds)
    TLS session ticket:
    0000 - 9d c2 14 28 4b 27 ee 77-30 f1 68 8a 36 f9 2a 24   ...(K'.w0.h.6.*$
    0010 - 2f c3 ea 7a 5f a2 45 fc-ed c0 21 c9 b1 6a 77 03   /..z_.E...!..jw.
    0020 - 94 42 3f 51 2d 4a 4c 79-12 ff 07 0d 57 ec 22 eb   .B?Q-JLy....W.".
    0030 - 4b 65 44 ee bf 22 50 1a-07 7f d7 32 c7 c4 8a 53   KeD.."P....2...S
    0040 - 16 82 91 96 8c 31 e2 b6-9e 06 02 ee c5 ea 49 99   .....1........I.
    0050 - f8 4c d4 65 7f d7 eb 9a-7e ad 45 49 31 53 31 aa   .L.e....~.EI1S1.
    0060 - b2 25 49 0e c1 bd 6f 85-54 ab 5c 69 ff 4d 9c 69   .%I...o.T.\i.M.i
    0070 - 9e                                                .

    Start Time: 1646143236
    Timeout   : 7200 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
DONE

which you can confirm the s_client cert matches the cert above. So at least on 1.8.5 I'm unable to reproduce this.

I confirmed also on 1.8.8 that this isn't reproducible with the above cert either. 1.8.5 used Go 1.16.9 and 1.8.8 used Go 1.16.12.

Note that the above cert is only valid until 9:03AM ET, so you may need to adjust your clock to use it in testing. :-)

I think it'd be best if you were able to generate a new certificate (even a self-signed one -- unless the problematic OID is on a parent cert, in which case you'd need to create a root CA and issue a leaf) that had the same structure as the existing certificate chain, but with different key material and only valid for localhost. You could then share both the private key and certificate with us and we'd hopefully be able to reproduce the issue.

[umlublin]

@danielraq and me are from same organization with "strange" certificates.
There is no problem with 1.8.* versions (dont burn time to reproduce this), but 1.9.* because of it is compiled with go 1.17.2 or 1.17.3
I've make fix that eliminate this certificated related issues in 1.17.6 and up
When Vault 1.9.4 (with go 1.17.7 - i think) will be released ?

[cipherboy]

Ahhh ok, good to hear :)

1.9.4 release will be released soon if all goes well. And yes, 1.17.7. Do you want me to leave this open until the release or close it, given that it appears it was a Go toolchain problem and we've pulled in a patched version?

[danielraq]

Yes, leave it open till 1.9.4. If all will be ok, ill close the ticket.

[sgmiller]

@danielraq : We've released 1.9.4 today. Give it a shot!

[danielraq]

@sgmiller it's alive :) thank you for your time. with 1.9.4 I don't see any certificate issues