Implement LTO

This commit implements LTO for rust leveraging LLVM's passes. What this means
is:

* When compiling an rlib, in addition to insdering foo.o into the archive, also
  insert foo.bc (the LLVM bytecode) of the optimized module.

* When the compiler detects the -Z lto option, it will attempt to perform LTO on
  a staticlib or binary output. The compiler will emit an error if a dylib or
  rlib output is being generated.

* The actual act of performing LTO is as follows:

    1. Force all upstream libraries to have an rlib version available.
    2. Load the bytecode of each upstream library from the rlib.
    3. Link all this bytecode into the current LLVM module (just using llvm
       apis)
    4. Run an internalization pass which internalizes all symbols except those
       found reachable for the local crate of compilation.
    5. Run the LLVM LTO pass manager over this entire module

    6a. If assembling an archive, then add all upstream rlibs into the output
        archive. This ignores all of the object/bitcode/metadata files rust
        generated and placed inside the rlibs.
    6b. If linking a binary, create copies of all upstream rlibs, remove the
        rust-generated object-file, and then link everything as usual.

As I have explained in #10741, this process is excruciatingly slow, so this is
*not* turned on by default, and it is also why I have decided to hide it behind
a -Z flag for now. The good news is that the binary sizes are about as small as
they can be as a result of LTO, so it's definitely working.

Closes #10741
Closes #10740

Author: alexcrichton

src/librustc/back/archive.rs

@@ -42,7 +42,8 @@ fn run_ar(sess: Session, args: &str, cwd: Option<&Path>,
     }
     let o = Process::new(ar, args.as_slice(), opts).finish_with_output();
     if !o.status.success() {
-        sess.err(format!("{} failed with: {}", ar, o.status));
+        sess.err(format!("{} {} failed with: {}", ar, args.connect(" "),
+                         o.status));
         sess.note(format!("stdout ---\n{}", str::from_utf8(o.output)));
         sess.note(format!("stderr ---\n{}", str::from_utf8(o.error)));
         sess.abort_if_errors();
@@ -88,16 +89,34 @@ impl Archive {
 
     /// Adds all of the contents of the rlib at the specified path to this
     /// archive.
-    pub fn add_rlib(&mut self, rlib: &Path) {
-        let name = rlib.filename_str().unwrap().split('-').next().unwrap();
-        self.add_archive(rlib, name, [METADATA_FILENAME]);
+    ///
+    /// This ignores adding the bytecode from the rlib, and if LTO is enabled
+    /// then the object file also isn't added.
+    pub fn add_rlib(&mut self, rlib: &Path, name: &str, lto: bool) {
+        let object = format!("{}.o", name);
+        let bytecode = format!("{}.bc", name);
+        let mut ignore = ~[METADATA_FILENAME, bytecode.as_slice()];
+        if lto {
+            ignore.push(object.as_slice());
+        }
+        self.add_archive(rlib, name, ignore);
     }
 
     /// Adds an arbitrary file to this archive
     pub fn add_file(&mut self, file: &Path) {
         run_ar(self.sess, "r", None, [&self.dst, file]);
     }
 
+    /// Removes a file from this archive
+    pub fn remove_file(&mut self, file: &str) {
+        run_ar(self.sess, "d", None, [&self.dst, &Path::new(file)]);
+    }
+
+    pub fn files(&self) -> ~[~str] {
+        let output = run_ar(self.sess, "t", None, [&self.dst]);
+        str::from_utf8(output.output).lines().map(|s| s.to_owned()).collect()
+    }
+
     fn add_archive(&mut self, archive: &Path, name: &str, skip: &[&str]) {
         let loc = TempDir::new("rsar").unwrap();
 
@@ -109,11 +128,17 @@ impl Archive {
         // The reason for this is that archives are keyed off the name of the
         // files, so if two files have the same name they will override one
         // another in the archive (bad).
+        //
+        // We skip any files explicitly desired for skipping, and we also skip
+        // all SYMDEF files as these are just magical placeholders which get
+        // re-created when we make a new archive anyway.
         let files = fs::readdir(loc.path());
         let mut inputs = ~[];
         for file in files.iter() {
             let filename = file.filename_str().unwrap();
             if skip.iter().any(|s| *s == filename) { continue }
+            if filename.contains(".SYMDEF") { continue }
+
             let filename = format!("r-{}-{}", name, filename);
             let new_filename = file.with_filename(filename);
             fs::rename(file, &new_filename);