Skip to content

Cannot get sources from https://crates.io as curl fails the certificate check #976

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Siosm opened this issue Nov 25, 2014 · 32 comments
Closed

Comments

@Siosm
Copy link

Siosm commented Nov 25, 2014

Building any project with dependencies on crates.io, with the latest nightly, on Arch Linux, ends with this error:

$ cargo build --verbose
    Updating registry `https://github.com/rust-lang/crates.io-index`
 Downloading pkg-config v0.0.1
Unable to get packages from source

Caused by:
  Failed to download package `pkg-config v0.0.1` from https://crates.io/api/v1/crates/pkg-config/0.0.1/download

Caused by:
  Peer certificate cannot be authenticated with given CA certificates

Sample Cargo.toml reproducing the issue:

$ cat Cargo.toml
[package]

name = "nickel-demo"
version = "0.0.1"
authors = ["Timothée Ravier <[email protected]>"]

[dependencies.nickel]

git = "https://github.com/nickel-org/nickel.rs.git"

And version info:

$ rustc --version=verbose
rustc 0.13.0-nightly (395901393 2014-11-24 00:46:30 +0000)
binary: rustc
commit-hash: 395901393c369c910a8ae36ace71402cb206d217
commit-date: 2014-11-24 00:46:30 +0000
host: x86_64-unknown-linux-gnu
release: 0.13.0-nightly
$ cargo --version
cargo 0.0.1-pre-nightly (1bdfb25 2014-11-20 19:29:22 +0000)

Using curl manually gives the following output:

$ curl -v https://crates.io/api/v1/crates/pkg-config/0.0.1/download
* Hostname was NOT found in DNS cache
*   Trying 107.20.211.76...
* Connected to crates.io (107.20.211.76) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* Server certificate:
*        subject: OU=GT87339749; OU=See www.rapidssl.com/resources/cps (c)14; OU=Domain Control Validated - RapidSSL(R); CN=crates.io
*        start date: 2014-11-06 11:11:23 GMT
*        expire date: 2015-11-09 14:30:14 GMT
*        subjectAltName: crates.io matched
*        issuer: C=US; O=GeoTrust Inc.; CN=RapidSSL SHA256 CA - G3
*        SSL certificate verify ok.
> GET /api/v1/crates/pkg-config/0.0.1/download HTTP/1.1
> User-Agent: curl/7.39.0
> Host: crates.io
> Accept: */*
> 
< HTTP/1.1 302 Found
< Connection: keep-alive
< Server: nginx
< Date: Tue, 25 Nov 2014 13:46:14 GMT
< Transfer-Encoding: chunked
< Set-Cookie: cargo_session=--3c156d7a3629aa08a1a0d538832f99ea30835cc8; HttpOnly; Secure; Path=/
< Location: https://crates-io.s3-us-west-1.amazonaws.com/crates/pkg-config/pkg-config-0.0.1.crate
< Strict-Transport-Security: max-age=31536000
< Via: 1.1 vegur
< 
* Connection #0 to host crates.io left intact

And tracing cargo build with strace gives:

$ strace -f -e trace=file cargo build --verbose
(...)
 Downloading pkg-config v0.0.1
stat("/home/tim/.cargo/registry/cache/github.1485827954.workers.dev-1ecc6299db9ec823", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
getcwd("/home/tim/projects/perso/rust/nickel-demo", 2048) = 42
stat("/home/tim/projects/perso/rust/nickel-demo/.cargo/config", 0x7fffb1eba210) = -1 ENOENT (No such file or directory)
stat("/home/tim/projects/perso/rust/.cargo/config", 0x7fffb1eba210) = -1 ENOENT (No such file or directory)
stat("/home/tim/projects/perso/.cargo/config", 0x7fffb1eba210) = -1 ENOENT (No such file or directory)
stat("/home/tim/projects/.cargo/config", 0x7fffb1eba210) = -1 ENOENT (No such file or directory)
stat("/home/tim/.cargo/config", 0x7fffb1eba210) = -1 ENOENT (No such file or directory)
stat("/home/.cargo/config", 0x7fffb1eba210) = -1 ENOENT (No such file or directory)
stat("/.cargo/config", 0x7fffb1eba210)  = -1 ENOENT (No such file or directory)
access("/home/tim/.gitconfig", F_OK)    = 0
stat("/home/tim/.gitconfig", {st_mode=S_IFREG|0644, st_size=1541, ...}) = 0
stat("/home/tim/.gitconfig", {st_mode=S_IFREG|0644, st_size=1541, ...}) = 0
open("/home/tim/.gitconfig", O_RDONLY)  = 3
access("/home/tim/.config/git/config", F_OK) = -1 ENOENT (No such file or directory)
access("/etc/gitconfig", F_OK)          = -1 ENOENT (No such file or directory)
stat("/home/tim/.gitconfig", {st_mode=S_IFREG|0644, st_size=1541, ...}) = 0
open("/home/rustbuild/root64/ssl/openssl.cnf", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/var/ssl", 0x7fffb1eba7a0)        = -1 ENOENT (No such file or directory)
stat("/usr/share/ssl", 0x7fffb1eba7a0)  = -1 ENOENT (No such file or directory)
stat("/usr/local/ssl", 0x7fffb1eba7a0)  = -1 ENOENT (No such file or directory)
stat("/usr/local/openssl", 0x7fffb1eba7a0) = -1 ENOENT (No such file or directory)
stat("/usr/local/share", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/usr/lib/ssl", 0x7fffb1eba7a0)    = -1 ENOENT (No such file or directory)
stat("/usr/ssl", 0x7fffb1eba7a0)        = -1 ENOENT (No such file or directory)
stat("/etc/openssl", 0x7fffb1eba7a0)    = -1 ENOENT (No such file or directory)
stat("/etc/pki/tls", 0x7fffb1eba7a0)    = -1 ENOENT (No such file or directory)
stat("/etc/ssl", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/etc/hosts", O_RDONLY|O_CLOEXEC)  = 3
open("/etc/gai.conf", O_RDONLY|O_CLOEXEC) = 3
stat("/dev/urandom", {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 9), ...}) = 0
open("/dev/urandom", O_RDONLY)          = 4
open("/etc/ssl/cert.pem", O_RDONLY)     = 4
open("/etc/hosts", O_RDONLY|O_CLOEXEC)  = 4
open("/etc/ssl/cert.pem", O_RDONLY)     = 5
stat("/etc/ssl/certs/415660c1.0", 0x7fffb1eba1e0) = -1 ENOENT (No such file or directory)
Unable to get packages from source

Caused by:
  Failed to download package `pkg-config v0.0.1` from https://crates.io/api/v1/crates/pkg-config/0.0.1/download

Caused by:
  Peer certificate cannot be authenticated with given CA certificates
+++ exited with 101 +++

So it looks like the curl module is not looking at the correct place for certificate checking and so far my attempts to use environment variables such as CURL_CA_BUNDLE have been unsuccessful with cargo.

This may be a rust-curl bug but I can't confirm this because trying to build a minimal example with the curl-rust crate using cargo fails with the same error :(.

@alexcrichton
Copy link
Member

Interesting! It's always a fun time linking openssl :)

The code in question for this sort of probe is in the openssl-sys crate which we then leverage in curl-rust by default. So, in light of this information, I have a few questions:

  1. What system are you running on? Looks like it's linux-based, but I'm curious what distribution you're using.
  2. Do you know where your ca certificates are located by default? This list shows a bunch of common locations, but we may just need to add to that list.
  3. Could you run a strace of curl itself downloading a file from crates.io? I find it useful to compare the two outputs.

Thanks for opening an issue!

@Siosm
Copy link
Author

Siosm commented Nov 25, 2014

  1. I'm running an Arch Linux x86_64 system. I'm investigating if I can reproduce the issue on a CentOS 7 and a Fedora 20 system.
  2. Arch Linux recently moved to an "everything in a one file" scheme for certificate storage, just like Fedora did a couple of releases ago (http://fedoraproject.org/wiki/Features/SharedSystemCertificates & https://lists.archlinux.org/pipermail/arch-dev-public/2014-August/026586.html):
$ tree -p /etc/ca-certificates /etc/ssl
/etc/ca-certificates
├── [drwxr-xr-x]  extracted
│   ├── [drwxr-xr-x]  java
│   │   └── [lrwxrwxrwx]  cacerts -> ../../../ssl/certs/java/cacerts
│   ├── [drwxr-xr-x]  openssl
│   │   └── [-r--r--r--]  ca-bundle.trust.crt
│   └── [drwxr-xr-x]  pem
│       ├── [-r--r--r--]  email-ca-bundle.pem
│       ├── [-r--r--r--]  objsign-ca-bundle.pem
│       └── [lrwxrwxrwx]  tls-ca-bundle.pem -> ../../../ssl/certs/ca-certificates.crt
└── [drwxr-xr-x]  trust-source
    ├── [drwxr-xr-x]  anchors
    └── [drwxr-xr-x]  blacklist
/etc/ssl
├── [lrwxrwxrwx]  cert.pem -> ../ca-certificates/extracted/pem/tls-ca-bundle.pem
├── [drwxr-xr-x]  certs
│   ├── [lrwxrwxrwx]  ca-bundle.trust.crt -> ../../ca-certificates/extracted/openssl/ca-bundle.trust.crt
│   ├── [-r--r--r--]  ca-certificates.crt
│   └── [drwxr-xr-x]  java
│       └── [-r--r--r--]  cacerts
├── [drwxr-xr-x]  misc
│   ├── [-rwxr-xr-x]  CA.pl
│   ├── [-rwxr-xr-x]  CA.sh
│   ├── [-rwxr-xr-x]  c_hash
│   ├── [-rwxr-xr-x]  c_info
│   ├── [-rwxr-xr-x]  c_issuer
│   ├── [-rwxr-xr-x]  c_name
│   └── [-rwxr-xr-x]  tsget
├── [-rw-r--r--]  openssl.cnf
└── [drwxr-xr-x]  private

11 directories, 17 files

So opening /etc/ssl/cert.pem should be OK. But instead, it looks like cargo is going for /etc/ssl/certs/415660c1.0, which does not exist.

  1. strace of curl:
$ strace -e trace=file curl https://crates.io/api/v1/crates/pkg-config/0.0.1/download
execve("/usr/bin/curl", ["curl", "https://crates.io/api/v1/crates/"...], [/* 70 vars */]) = 0
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/home/tim/.rust/lib/tls/x86_64/libcurl.so.4", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/home/tim/.rust/lib/tls/x86_64", 0x7fffdc3383e0) = -1 ENOENT (No such file or directory)
open("/home/tim/.rust/lib/tls/libcurl.so.4", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/home/tim/.rust/lib/tls", 0x7fffdc3383e0) = -1 ENOENT (No such file or directory)
open("/home/tim/.rust/lib/x86_64/libcurl.so.4", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/home/tim/.rust/lib/x86_64", 0x7fffdc3383e0) = -1 ENOENT (No such file or directory)
open("/home/tim/.rust/lib/libcurl.so.4", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/home/tim/.rust/lib", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib/libcurl.so.4", O_RDONLY|O_CLOEXEC) = 3
open("/home/tim/.rust/lib/libz.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libz.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/home/tim/.rust/lib/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
open("/home/tim/.rust/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
open("/home/tim/.rust/lib/libidn.so.11", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libidn.so.11", O_RDONLY|O_CLOEXEC) = 3
open("/home/tim/.rust/lib/libssh2.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libssh2.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/home/tim/.rust/lib/libssl.so.1.0.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libssl.so.1.0.0", O_RDONLY|O_CLOEXEC) = 3
open("/home/tim/.rust/lib/libcrypto.so.1.0.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libcrypto.so.1.0.0", O_RDONLY|O_CLOEXEC) = 3
open("/home/tim/.rust/lib/libgssapi_krb5.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libgssapi_krb5.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/home/tim/.rust/lib/libkrb5.so.3", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libkrb5.so.3", O_RDONLY|O_CLOEXEC) = 3
open("/home/tim/.rust/lib/libk5crypto.so.3", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libk5crypto.so.3", O_RDONLY|O_CLOEXEC) = 3
open("/home/tim/.rust/lib/libcom_err.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libcom_err.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/home/tim/.rust/lib/libdl.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/home/tim/.rust/lib/libkrb5support.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libkrb5support.so.0", O_RDONLY|O_CLOEXEC) = 3
open("/home/tim/.rust/lib/libkeyutils.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libkeyutils.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/home/tim/.rust/lib/libresolv.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libresolv.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/etc/ssl/openssl.cnf", O_RDONLY)  = 3
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
open("/home/tim/.curlrc", O_RDONLY)     = -1 ENOENT (No such file or directory)
stat("/dev/urandom", {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 9), ...}) = 0
open("/dev/urandom", O_RDONLY)          = 4
open("/dev/urandom", O_RDONLY|O_NOCTTY|O_NONBLOCK) = 4
open("/etc/ssl/certs/ca-certificates.crt", O_RDONLY) = 4
open("/proc/meminfo", O_RDONLY|O_CLOEXEC) = 4
open("/etc/localtime", O_RDONLY|O_CLOEXEC) = 4
+++ exited with 0 +++

Here it opens /etc/ssl/certs/ca-certificates.crt directly as this path was given directly at configure/compile time: https://projects.archlinux.org/svntogit/packages.git/tree/trunk/PKGBUILD?h=packages/curl#n40

@Siosm
Copy link
Author

Siosm commented Nov 25, 2014

I cannot reproduce the issue on CentOS 7 nor on Fedora 20, so this is an Arch Linux specific issue. Here is the relevant strace output for CentOS 7 (Fedora 20 is similar):

 Downloading pkg-config v0.0.1
stat("/usr/local/share", {st_mode=S_IFDIR|0755, st_size=46, ...}) = 0
stat("/etc/pki/tls", {st_mode=S_IFDIR|0755, st_size=76, ...}) = 0
stat("/etc/ssl", {st_mode=S_IFDIR|0755, st_size=18, ...}) = 0
open("/etc/hosts", O_RDONLY|O_CLOEXEC)  = 3
stat("/dev/urandom", {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 9), ...}) = 0
open("/dev/urandom", O_RDONLY)          = 4
open("/etc/pki/tls/cert.pem", O_RDONLY) = 4
open("/etc/hosts", O_RDONLY|O_CLOEXEC)  = 4
open("/etc/pki/tls/cert.pem", O_RDONLY) = 5
Process 2668 attached
Process 2669 attached
[pid  2668] open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
[pid  2668] open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
[pid  2668] open("/lib64/libgcc_s.so.1", O_RDONLY|O_CLOEXEC) = 3
[pid  2668] open("/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
[pid  2668] open("/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
[pid  2668] open("/lib64/librt.so.1", O_RDONLY|O_CLOEXEC) = 3
[pid  2668] open("/lib64/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
[pid  2668] open("/sys/devices/system/cpu/online", O_RDONLY|O_CLOEXEC) = 3
[pid  2668] open("/proc/self/maps", O_RDONLY|O_CLOEXEC) = 3
Process 2671 attached
[pid  2671] syscall_318(0x7f7a6f3f7330, 0, 0, 0x7f7a7649f490, 0x63, 0x500000000) = -1 (errno 38)
[pid  2671] open("/dev/urandom", O_RDONLY) = 3
[pid  2671] +++ exited with 0 +++
[pid  2668] +++ exited with 0 +++
[pid  2670] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2668, si_status=0, si_utime=2, si_stime=4} ---
[pid  2670] +++ exited with 0 +++
[pid  2669] +++ exited with 0 +++
Process 2672 attached
Process 2673 attached
[pid  2672] open("/dev/urandom", O_RDONLY) = 5
   Compiling http v0.1.0-pre (https://github.com/nickel-org/rust-http.git#5ec61527)

@Siosm
Copy link
Author

Siosm commented Nov 25, 2014

Is Rust recursively following symbolic links? Here we have:

$ ll /etc/ssl/cert.pem
lrwxrwxrwx 1 root root 50 Nov 14 10:58 /etc/ssl/cert.pem -> ../ca-certificates/extracted/pem/tls-ca-bundle.pem
$ ll /etc/ssl/../ca-certificates/extracted/pem/tls-ca-bundle.pem
lrwxrwxrwx 1 root root 38 Nov 14 10:58 /etc/ssl/../ca-certificates/extracted/pem/tls-ca-bundle.pem -> ../../../ssl/certs/ca-certificates.crt
$ ll /etc/ssl/../ca-certificates/extracted/pem/../../../ssl/certs/ca-certificates.crt
-r--r--r-- 1 root root 242K Nov 16 14:23 /etc/ssl/../ca-certificates/extracted/pem/../../../ssl/certs/ca-certificates.crt

@alexcrichton
Copy link
Member

Would you be able to build cargo from source to test out a change? I would be curious if this diff applied to openssl-sys would work for you:

diff --git a/openssl-sys/src/probe.rs b/openssl-sys/src/probe.rs    
index fc16202..10ad899 100644                                       
--- a/openssl-sys/src/probe.rs                                      
+++ b/openssl-sys/src/probe.rs                                      
@@ -66,7 +66,7 @@ pub fn probe() -> ProbeResult {                   
 }                                                                  

 fn try(dst: &mut Option<Path>, val: Path) {                        
-    if dst.is_none() && val.exists() {                             
+    if val.exists() {                                              
         *dst = Some(val);                                          
     }                                                              
 }                                                                  

You could build cargo with that patch via:

git clone https://github.com/rust-lang/cargo
git clone https://github.com/sfackler/rust-openssl
cd rust-openssl
// apply the patch
cd ../cargo
// create .cargo/config with `paths = ["../rust-openssl/openssl-sys"]
./configure
make

@Siosm
Copy link
Author

Siosm commented Nov 26, 2014

Unfortunately, I followed your instructions and I ran into the same error while trying to build cargo :(.
But I've tried something else which seems to solve the issue for me:

sudo ln -s /etc/ca-certificates/extracted/pem/email-ca-bundle.pem /etc/pki/tls/cert.pem

So this is really strange and looks like an Arch Linux bug. I'll report this asap and will report back here.

I'm now hitting another bug (which still prevents me from trying your patch) but I'll make another bug report tomorrow.

@alexcrichton
Copy link
Member

I'll note that we're doing somewhat sketchy things with openssl (we're linking it statically in the nightly distribution) which may not be kosher on all platforms, so it may not necessarily be a bug with Arch.

I also just realized, can you try running SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt cargo fetch ? (using nightlies) Either that or the SSL_CERT_DIR environment variable may help cargo find this location, although we should definitely find it by default as well.

@Siosm
Copy link
Author

Siosm commented Nov 26, 2014

Well, I've tried every combination I could think of and the only one working is: SSL_CERT_FILE=/etc/ca-certificates/extracted/pem/email-ca-bundle.pem cargo build ...
This is really strange when the following commands both work:

$ SSL_CERT_FILE=/etc/ca-certificates/extracted/pem/tls-ca-bundle.pem curl https://crates.io/api/v1/crates/pkg-config/0.0.1/download -v
* Hostname was NOT found in DNS cache
*   Trying 107.20.211.76...
* Connected to crates.io (107.20.211.76) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: /etc/ca-certificates/extracted/pem/tls-ca-bundle.pem
  CApath: none
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* Server certificate:
*        subject: OU=GT87339749; OU=See www.rapidssl.com/resources/cps (c)14; OU=Domain Control Validated - RapidSSL(R); CN=crates.io
*        start date: 2014-11-06 11:11:23 GMT
*        expire date: 2015-11-09 14:30:14 GMT
*        subjectAltName: crates.io matched
*        issuer: C=US; O=GeoTrust Inc.; CN=RapidSSL SHA256 CA - G3
*        SSL certificate verify ok.
> GET /api/v1/crates/pkg-config/0.0.1/download HTTP/1.1
> User-Agent: curl/7.39.0
> Host: crates.io
> Accept: */*
> 
< HTTP/1.1 302 Found
< Connection: keep-alive
< Server: nginx
< Date: Wed, 26 Nov 2014 09:05:51 GMT
< Transfer-Encoding: chunked
< Set-Cookie: cargo_session=--3c156d7a3629aa08a1a0d538832f99ea30835cc8; HttpOnly; Secure; Path=/
< Location: https://crates-io.s3-us-west-1.amazonaws.com/crates/pkg-config/pkg-config-0.0.1.crate
< Strict-Transport-Security: max-age=31536000
< Via: 1.1 vegur
< 
* Connection #0 to host crates.io left intact
$ SSL_CERT_FILE=/etc/ca-certificates/extracted/pem/email-ca-bundle.pem curl https://crates.io/api/v1/crates/pkg-config/0.0.1/download -v 
* Hostname was NOT found in DNS cache
*   Trying 107.20.211.76...
* Connected to crates.io (107.20.211.76) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: /etc/ca-certificates/extracted/pem/email-ca-bundle.pem
  CApath: none
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* Server certificate:
*        subject: OU=GT87339749; OU=See www.rapidssl.com/resources/cps (c)14; OU=Domain Control Validated - RapidSSL(R); CN=crates.io
*        start date: 2014-11-06 11:11:23 GMT
*        expire date: 2015-11-09 14:30:14 GMT
*        subjectAltName: crates.io matched
*        issuer: C=US; O=GeoTrust Inc.; CN=RapidSSL SHA256 CA - G3
*        SSL certificate verify ok.
> GET /api/v1/crates/pkg-config/0.0.1/download HTTP/1.1
> User-Agent: curl/7.39.0
> Host: crates.io
> Accept: */*
> 
< HTTP/1.1 302 Found
< Connection: keep-alive
< Server: nginx
< Date: Wed, 26 Nov 2014 09:06:11 GMT
< Transfer-Encoding: chunked
< Location: https://crates-io.s3-us-west-1.amazonaws.com/crates/pkg-config/pkg-config-0.0.1.crate
< Set-Cookie: cargo_session=--3c156d7a3629aa08a1a0d538832f99ea30835cc8; HttpOnly; Secure; Path=/
< Strict-Transport-Security: max-age=31536000
< Via: 1.1 vegur
< 
* Connection #0 to host crates.io left intact

@Siosm
Copy link
Author

Siosm commented Nov 26, 2014

My other bug is rust-lang/rust#16402, which is unrelated.
I've tried using cargo with your patch but it doesn't seem to work.

@alexcrichton
Copy link
Member

Hm interesting! We may just be missing some option to libcurl, do you know if that email-ca-bundle.pem file is symlinked anywhere else on the system?

@Siosm
Copy link
Author

Siosm commented Dec 1, 2014

It does not seem that email-ca-bundle.pem is linked from somewhere else.

Anyway, the issue is now resolved for me as the updated ca-certificates-utils-20140923-7.1 package from Arch Linux creates all the symlinks expected by openssl/curl in cargo, and thus fixes this bug.
So, as far as I understand, this was a temporary Arch Linux specific issue.

@Siosm Siosm closed this as completed Dec 1, 2014
@alexcrichton
Copy link
Member

Ok, thanks for looking into this @Siosm!

@Siosm
Copy link
Author

Siosm commented Jun 15, 2015

I would like to reopen this issue because I think that I have found the underlying issue.

It looks like cargo is affected by this openssl bug: https://bugzilla.redhat.com/show_bug.cgi?id=1144808, which has only been recently been fixed in openssl 1.0.2b (see discussions and similar issues at https://gist.github.com/grawity/15eabf67191e17080241, https://bugs.archlinux.org/task/42921).

Both the stable (OpenSSL 1.0.1i 6 Aug 2014) and latest nightly (OpenSSL 1.0.2a 19 Mar 2015) cargo builds are affected by this.

As the openssl issue has been fixed in Arch Linux, the old certificates were removed (https://projects.archlinux.org/svntogit/packages.git/commit/trunk?h=packages/nss&id=d0ccc2d596f2414eb277aead14233a5cf06cbef5) thus I had this issue again using cargo.

The issue #1689 might also be a duplicate of this one.

A "simple" fix would be to rebuild cargo against openssl 1.0.2b or superior. In the mean time, manually re-adding the old certificates to the trust store will work as a workaround.

Another way to fix this would be to remove the old certificate from the certificate list sent by the server hosting crates.io, but this may prevent some clients on old systems without the updated certificate from connecting to it.

@Siosm
Copy link
Author

Siosm commented Jun 15, 2015

Note: the ca-certificate-mozilla package update is currently in the testing repository on Arch Linux.

Workaround example on Arch Linux:

  1. With ca-certificates-mozilla-3.19.1-1-x86_64.pkg.tar.xz installed:
    sudo cp /etc/ca-certificates/extracted/cadir/Verisign_Class_3_Public_Primary_Certification_Authority.pem /etc/ca-certificates/trust-source/anchors/
  2. Run: sudo update-ca-trust or update the system to the latest ca-certificates-mozilla package (currently in testing).

@steveklabnik steveklabnik reopened this Jun 15, 2015
@alexcrichton
Copy link
Member

Wow, thanks for the thorough investigation @Siosm! I'll try to update OpenSSL on the bots soon.

@alexcrichton
Copy link
Member

Deployed! The next nightlies should have the updated version of OpenSSL.

@sjmackenzie
Copy link

NixOS (nixos.org) is also affected by this issue.
I'm using rustcMaster (https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/all-packages.nix#L4578) which points to this file https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/compilers/rustc/head.nix

Shall wait for the update or do it myself, @Siosm great job on the recon.

@globin
Copy link
Contributor

globin commented Jun 21, 2015

@sjmackenzie using the cargoSnapshot binary works if you patch it to a newer version:
mayflower/nixpkgs@14c8a8e

If you find a real fix please give a notice. I can't seem to fix it by building cargo with the newest openssl.

@sjmackenzie
Copy link

@globin Thanks for the info! If you couldn't get it solved after trying the newest openssl, I've very curious to see if this issue is solved in other distros. btw I'm using it w.r.t. building regex_syntax as part of https://github.com/killercup/trpl-ebook.

It throws this error:

$ cargo run --verbose
 Downloading regex-syntax v0.1.0
Unable to get packages from source

Caused by:
  Failed to download package `regex-syntax v0.1.0` from https://crates.io/api/v1/crates/regex-syntax/0.1.0/download

Caused by:
  Peer certificate cannot be authenticated with given CA certificates

@globin
Copy link
Contributor

globin commented Jun 22, 2015

It fails for any crate because of certificate failure with Amazon S3, curl does, too:

curl https://crates-io.s3-us-west-1.amazonaws.com/crates/cookie/cookie-0.1.21.crate -v
*   Trying 54.231.235.8...
* Connected to crates-io.s3-us-west-1.amazonaws.com (54.231.235.8) port 443 (#0)
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.2, TLS handshake, Client hello (1):
* TLSv1.2, TLS handshake, Server hello (2):
* TLSv1.2, TLS handshake, CERT (11):
* TLSv1.2, TLS alert, Server hello (2):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: http://curl.haxx.se/docs/sslcerts.html

@sjmackenzie
Copy link

@globin NixOS/nixpkgs#8431

@southerngs
Copy link

I'm experiencing the same problem with a Ubuntu 14.04.1 system. I'm trying to figure out if there's a problem with my local configuration or if the problem is with Amazon S3 (or somewhere else).

@daniel-ferguson
Copy link

Just to add, working for me now on Arch Linux x86_64 (4 hours fresh install):

> cargo -V 
cargo 0.3.0-nightly (a2dd2ac 2015-06-24) (built 2015-06-23) 

Version below did not work:

> cargo -V
cargo 0.2.0-nightly (efb482d 2015-04-30) (built 2015-04-30)

@southerngs
Copy link

I tried on two other systems this evening (one Ubuntu 12.04 and one Arch Linux) and was not able to reproduce the problem. The system that I have the problem with is behind a firewall and requires use of a proxy server (the ones that work are not). I thought I would add this information in case anyone else sees this issue.

Also in all cases I'm testing with the release 1.0 version of Rust and included Cargo (cargo 0.2.0-nightly (efb482d 2015-04-30) (built 2015-04-30)).

@azerupi
Copy link
Contributor

azerupi commented Jun 26, 2015

I am also on Arch Linux and I have the same issue on stable that uses cargo 0.2.0-nightly.

multirust: stable revision:

rustc 1.1.0 (35ceea399 2015-06-19)
cargo 0.2.0-nightly (a483581 2015-05-14) (built 2015-05-15)

But it's gone on the beta version that uses cargo 0.3.0-nightly

multirust: beta revision:

rustc 1.2.0-beta.1 (2ad26e850 2015-06-23)
cargo 0.3.0-nightly (a2dd2ac 2015-06-24) (built 2015-06-23)

So I guess I will have to stick to the beta version for the next couple of weeks until it moves to stable

@Keats
Copy link

Keats commented Jun 26, 2015

On Archlinux as well, I didn't use Rust for about a month when it was working and i am now running into the issue, same after running a rustup to get 1.1:

cargo 0.2.0-nightly (a483581 2015-05-14) (built 2015-05-15)

I'll try @Siosm workaround when I have time

@daniel-ferguson
Copy link

Just to note cargo 0.3.0-nightly can be installed for older rusts (I'm on 1.0.0), I assume without ill consequence.
The newer version runs on my Arch box fine.

Binaries are available here:
https://crates.io/install

@Siosm
Copy link
Author

Siosm commented Jun 26, 2015

As far as I'm concerned, this issue is only affecting the stable 1.0 and 1.1 releases. The 1.2 beta and nightly releases are working for me.

@tksfz
Copy link

tksfz commented Jul 5, 2015

Same here (works on 1.2 beta and 1.3 nightly; breaks on stable 1.1).

@sjmackenzie
Copy link

So the NixOS community have updated rust and it's without problem. I'm running rustMaster and cargoSnapshot.

$ rustc --version
rustc 1.1.0-dev (built 2015-06-05)
$ cargo --version
cargo 0.0.1-pre-nightly (84d6d2c 2015-03-31) (built 2015-04-01)

@Keats
Copy link

Keats commented Jul 7, 2015

It seems to now work for me, only did a pacman -Syu, nothing related to rust

telotortium added a commit to telotortium/nixpkgs that referenced this issue Jul 8, 2015
Update snapshot to avoid rust-lang/cargo#976, which otherwise breaks the
build.

Also move the `cargoSnapshot` derivation inside a set in
pkgs/top-level/all-packages.nix in order to hide the `cargo-snapshot`
packages from `nix-env -qa`, since it's only used to build the `cargo`
package.
telotortium added a commit to telotortium/nixpkgs that referenced this issue Jul 10, 2015
Update snapshot to avoid rust-lang/cargo#976, which otherwise breaks the
build.

Also move the `cargoSnapshot` derivation inside a set in
pkgs/top-level/all-packages.nix in order to hide the `cargo-snapshot`
packages from `nix-env -qa`, since it's only used to build the `cargo`
package.
@kendall
Copy link

kendall commented Jul 13, 2015

I get this error trying to work through the guessing game toy program in the docs:

~/work/rust/gg • rustc --version -v 
rustc 1.2.0-beta.2 (c8bab9d06 2015-07-09)
binary: rustc
commit-hash: c8bab9d06a179028a0d5129aa62f09d694d9cc49
commit-date: 2015-07-09
host: x86_64-apple-darwin
release: 1.2.0-beta.2

I'm only using a nightly because I had the problem initially with Rust 1.1 and I switched to a nightly, thinking it might be fixed there...

~/work/rust/gg • openssl version
OpenSSL 1.0.2d 9 Jul 2015

Fetching libc by hand with curl seems fine:

~/work/rust/gg • curl -v https://crates.io/api/v1/crates/libc/0.1.9/download          
* Hostname was NOT found in DNS cache
*   Trying 54.243.176.56...
* Connected to crates.io (54.243.176.56) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
* Server certificate: crates.io
* Server certificate: RapidSSL SHA256 CA - G3
* Server certificate: GeoTrust Global CA
> GET /api/v1/crates/libc/0.1.9/download HTTP/1.1
> User-Agent: curl/7.37.1
> Host: crates.io
> Accept: */*
>
< HTTP/1.1 302 Found
< Connection: keep-alive
* Server nginx is not blacklisted
< Server: nginx
< Date: Mon, 13 Jul 2015 14:44:00 GMT
< Transfer-Encoding: chunked
< Location: https://crates-io.s3-us-west-1.amazonaws.com/crates/libc/libc-0.1.9.crate
< Set-Cookie: cargo_session=--3c156d7a3629aa08a1a0d538832f99ea30835cc8; HttpOnly; Secure; Path=/
< Strict-Transport-Security: max-age=31536000
< Via: 1.1 vegur
<
* Connection #0 to host crates.io left intact

wizeman pushed a commit to wizeman/nixpkgs that referenced this issue Oct 28, 2015
Update snapshot to avoid rust-lang/cargo#976, which otherwise breaks the
build.

Also move the `cargoSnapshot` derivation inside a set in
pkgs/top-level/all-packages.nix in order to hide the `cargo-snapshot`
packages from `nix-env -qa`, since it's only used to build the `cargo`
package.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests