Include envoy in (SmartOS) pkgsrc

[siepkes]

Envoy describes itself as "an opensource edge and service proxy designed for cloud native applications". Envoy usage is becoming quite wide spread (some go as far as to say the defacto standard). It was originally developed at Lyft but is now part of the Cloud Native Computing Foundation.

Long story short; It's a cloud native haproxy and NGINX alternative which can be used with for example HTTP/2 and is also popular for use with gRPC (Google's OSS Protobuf + HTTP/2 RPC implementation).

Aside from Lyft's own success story examples of it's popularity are Google and IBM creating Istio which uses Envoy to create a service mesh, a bunch of other success stories and online tutorials being produced by organisations like RedHat.

I've attempted to get Envoy to build on Illumos / SmartOS but ran in to issues. It was quite some time ago so I don't remember the exact issue. I'll give it a shot again and report my findings here. Envoy uses Bezel for it's build. Bezel is Java based, being primarily a Java dev Bezel shouldn't be too much of an issue for me. However for some of the subtleties of creating patches for Envoy I will require assistance.

[mamash]

Well, bazel is the main problem really. It only works on sanctioned and explicitly listed platforms, and - not surprisingly - SunOS is not one of them.

There's probably no significant reason why it's so, but upstream is pretty clear on not wanting the job, and I'm not sure spending much time on fixing yet another crazy build system is really worth it. Remember, Bazel source tarball comes with close to 10.000 files.

[mamash]

I should add I'm interested in hearing how you ever got close to running bazel on SmartOS.

[siepkes]

I think I'm actually pretty close to getting Bazel to work. I've looked at FreeBSD and Darwin support (which were both shoe-horned in later on). For some reason Bazel still complains about not detecting the correct OS but I suspect thats just a minor error I made somewhere. Since I have reviewed FreeBSD and Darwin support and I don't see any additional changes they made that I haven't.

Got sidetracked by a bunch of other work but I hope to get back to it in a couple of days.

[mamash]

Cool, let me know if you need a second pair of eyes for anything.

[siepkes]

I definitely underestimated the complexity of Bazel. It seems a lot of functionality was never part of the original design and were more or less duct-tapped and shoehorned in later.

For example 90% the time Linux is called K8 (probably from Bazel's Kubernetes build tool roots) and other OS'es, like FreeBSD, are regarded as a "CPU". Oddly enough "arm" is the only real cpu regarded as a "CPU" by Bazel. I'm guessing thats actually Linux on ARM. Oddly enough for Windows there is an explicit x64 prefix in it's CPU name ("x64_windows") while FreeBSD x64 is just called "freebsd" (so what about FreeBSD arm support? Or ARM64?). Also Linux (K8) is kind of the default which is implied everywhere and all other things are then explicitly mentioned (Darwin, Windows, FreeBSD, etc.).

I got as far as this diff, unfortunely Bazel doesn't compile with it (by running compile.sh). The patch can be applied to bazel-0.11.1-dist.zip. I might take a look at it in the future but for now I'm going to let this rest.

--- bazel-orig/scripts/bootstrap/buildenv.sh    2018-04-01 14:52:36.150686900 +0000
+++ bazel/scripts/bootstrap/buildenv.sh 2018-04-01 14:53:02.524832025 +0000
@@ -83,6 +83,10 @@
   JAVA_HOME="${JAVA_HOME:-/usr/local/openjdk8}"
   ;;
 
+solaris)
+  JAVA_HOME="${JAVA_HOME:-/opt/local/openjdk8}"
+  ;;
+
 darwin)
   if [[ -z "$JAVA_HOME" ]]; then
     JAVA_HOME="$(/usr/libexec/java_home -v ${JAVA_VERSION}+ 2> /dev/null)" \
diff -uar bazel-orig/scripts/packages/BUILD bazel/scripts/packages/BUILD
--- bazel-orig/scripts/packages/BUILD   2018-04-01 14:52:36.152851396 +0000
+++ bazel/scripts/packages/BUILD        2018-04-01 14:53:02.525363493 +0000
@@ -20,6 +20,7 @@
             ":zip-bazel-exe_with_jdk",
         ],
         "//src/conditions:freebsd": [],
+        "//src/conditions:solaris": [],
         "//src/conditions:darwin": [
             ":with-jdk/install.sh",
             ":without-jdk/install.sh",
diff -uar bazel-orig/src/conditions/BUILD bazel/src/conditions/BUILD
--- bazel-orig/src/conditions/BUILD     2018-04-01 14:52:36.170223015 +0000
+++ bazel/src/conditions/BUILD  2018-04-01 14:53:02.525807667 +0000
@@ -41,6 +41,12 @@
 )
 
 config_setting(
+    name = "solaris",
+    values = {"cpu": "solaris"},
+    visibility = ["//visibility:public"],
+)
+
+config_setting(
     name = "windows",
     values = {"cpu": "x64_windows"},
     visibility = ["//visibility:public"],
diff -uar bazel-orig/src/conditions/BUILD.tools bazel/src/conditions/BUILD.tools
--- bazel-orig/src/conditions/BUILD.tools       2018-04-01 14:52:36.170282808 +0000
+++ bazel/src/conditions/BUILD.tools    2018-04-01 14:53:02.526128210 +0000
@@ -5,6 +5,12 @@
 )
 
 config_setting(
+    name = "solaris",
+    values = {"cpu": "solaris"},
+    visibility = ["//visibility:public"],
+)
+
+config_setting(
     name = "darwin",
     values = {"cpu": "darwin"},
     visibility = ["//visibility:public"],
diff -uar bazel-orig/src/main/java/com/google/devtools/build/lib/analysis/config/AutoCpuConverter.java bazel/src/main/java/com/google/devtools/build/lib/analysis/config/AutoCpuConverter.java
--- bazel-orig/src/main/java/com/google/devtools/build/lib/analysis/config/AutoCpuConverter.java        2018-04-01 14:52:36.216707875 +0000
+++ bazel/src/main/java/com/google/devtools/build/lib/analysis/config/AutoCpuConverter.java     2018-04-01 14:53:02.526694805 +0000
@@ -37,6 +37,8 @@
           return "darwin";
         case FREEBSD:
           return "freebsd";
+        case SOLARIS:
+          return "solaris";
         case WINDOWS:
           switch (CPU.getCurrent()) {
             case X86_64:
@@ -79,6 +81,8 @@
       return Pair.of(CPU.getCurrent(), OS.DARWIN);
     } else if (input.startsWith("freebsd")) {
       return Pair.of(CPU.getCurrent(), OS.FREEBSD);
+    } else if (input.startsWith("solaris")) {
+      return Pair.of(CPU.getCurrent(), OS.SOLARIS);
     } else if (input.startsWith("x64_windows")) {
       return Pair.of(CPU.getCurrent(), OS.WINDOWS);
     }
diff -uar bazel-orig/src/main/java/com/google/devtools/build/lib/rules/cpp/CppActionConfigs.java bazel/src/main/java/com/google/devtools/build/lib/rules/cpp/CppActionConfigs.java
--- bazel-orig/src/main/java/com/google/devtools/build/lib/rules/cpp/CppActionConfigs.java      2018-04-01 14:52:36.280856404 +0000
+++ bazel/src/main/java/com/google/devtools/build/lib/rules/cpp/CppActionConfigs.java   2018-04-01 14:53:02.527808835 +0000
@@ -1064,7 +1064,7 @@
   }
 
   private static String ifLinux(CppPlatform platform, String... lines) {
-    // Platform `LINUX` also includes FreeBSD.
+    // Platform `LINUX` also includes FreeBSD and Solaris.
     return ifTrue(platform == CppPlatform.LINUX, lines);
   }
 
diff -uar bazel-orig/src/main/java/com/google/devtools/build/lib/rules/cpp/CppConfiguration.java bazel/src/main/java/com/google/devtools/build/lib/rules/cpp/CppConfiguration.java
--- bazel-orig/src/main/java/com/google/devtools/build/lib/rules/cpp/CppConfiguration.java      2018-04-01 14:52:36.281580812 +0000
+++ bazel/src/main/java/com/google/devtools/build/lib/rules/cpp/CppConfiguration.java   2018-04-01 14:53:02.528615420 +0000
@@ -214,6 +214,7 @@
 
   static CppConfiguration create(CppConfigurationParameters params)
       throws InvalidConfigurationException {
+    System.out.println("Target CPU:" + params.toolchain.getTargetCpu());
     CrosstoolConfig.CToolchain toolchain = params.toolchain;
     CppOptions cppOptions = params.cppOptions;
     PathFragment crosstoolTopPathFragment =
diff -uar bazel-orig/src/main/java/com/google/devtools/build/lib/rules/cpp/CppConfigurationLoader.java bazel/src/main/java/com/google/devtools/build/lib/rules/cpp/CppConfigurationLoader.java
--- bazel-orig/src/main/java/com/google/devtools/build/lib/rules/cpp/CppConfigurationLoader.java        2018-04-01 14:52:36.281668789 +0000
+++ bazel/src/main/java/com/google/devtools/build/lib/rules/cpp/CppConfigurationLoader.java     2018-04-01 15:04:50.380820354 +0000
@@ -143,8 +143,10 @@
     CrosstoolConfigurationLoader.CrosstoolFile file =
         CrosstoolConfigurationLoader.readCrosstool(env, crosstoolTopLabel);
     if (file == null) {
+      System.out.println("No Crosstool file");
       return null;
     }
+    System.out.println("Crosstool file: '" + file.getProto().toString() + "'.");
     CrosstoolConfig.CToolchain toolchain =
         CrosstoolConfigurationLoader.selectToolchain(
             file.getProto(), options, cpuTransformer.getTransformer());
diff -uar bazel-orig/src/main/java/com/google/devtools/build/lib/rules/cpp/CrosstoolConfigurationIdentifier.java bazel/src/main/java/com/google/devtools/build/lib/rules/cpp/CrosstoolConfigurationIdentifier.java
--- bazel-orig/src/main/java/com/google/devtools/build/lib/rules/cpp/CrosstoolConfigurationIdentifier.java      2018-04-01 14:52:36.283483130 +0000
+++ bazel/src/main/java/com/google/devtools/build/lib/rules/cpp/CrosstoolConfigurationIdentifier.java   2018-04-01 14:53:02.529455917 +0000
@@ -40,6 +40,12 @@
 
   /** Creates a new {@link CrosstoolConfigurationIdentifier} with the given parameters. */
   CrosstoolConfigurationIdentifier(String cpu, String compiler, String libc) {
+    System.out.println("cpu: '" + cpu + "' compiler: '" + compiler + "' libc: '" + libc + "'");
+
+    for (StackTraceElement ste : Thread.currentThread().getStackTrace()) {
+        System.out.println(ste);
+    }
+
     this.cpu = Preconditions.checkNotNull(cpu);
     this.compiler = compiler;
     this.libc = libc;
diff -uar bazel-orig/src/main/java/com/google/devtools/build/lib/rules/cpp/CrosstoolConfigurationLoader.java bazel/src/main/java/com/google/devtools/build/lib/rules/cpp/CrosstoolConfigurationLoader.java
--- bazel-orig/src/main/java/com/google/devtools/build/lib/rules/cpp/CrosstoolConfigurationLoader.java  2018-04-01 14:52:36.283562611 +0000
+++ bazel/src/main/java/com/google/devtools/build/lib/rules/cpp/CrosstoolConfigurationLoader.java       2018-04-01 14:53:02.530014889 +0000
@@ -324,6 +324,7 @@
       boolean convertLipoToThinLto,
       Function<String, String> cpuTransformer)
       throws InvalidConfigurationException {
+    System.out.println("config compiler:" + config.getCompiler() + "config libc: " + config.getLibc());
     if ((config.getCompiler() != null) || (config.getLibc() != null)) {
       ArrayList<CrosstoolConfig.CToolchain> candidateToolchains = new ArrayList<>();
       for (CrosstoolConfig.CToolchain toolchain : release.getToolchainList()) {
@@ -356,6 +357,7 @@
     // We use fake CPU values to allow cross-platform builds for other languages that use the
     // C++ toolchain. Translate to the actual target architecture.
     String desiredCpu = cpuTransformer.apply(config.getCpu());
+    System.out.println("CPU: '" + config.getCpu() + "' desired CPU: '" + desiredCpu + "'");
     boolean needsLipo = lipoMode != LipoMode.OFF && !convertLipoToThinLto;
     for (CrosstoolConfig.DefaultCpuToolchain selector : release.getDefaultToolchainList()) {
       if (needsLipo && !selector.getSupportsLipo()) {
@@ -367,6 +369,7 @@
       }
     }
 
+    System.out.println("selectedIdentifier: '" + selectedIdentifier + "'");
     if (selectedIdentifier == null) {
       StringBuilder cpuBuilder = new StringBuilder();
       for (CrosstoolConfig.DefaultCpuToolchain selector : release.getDefaultToolchainList()) {
diff -uar bazel-orig/src/main/java/com/google/devtools/build/lib/util/OS.java bazel/src/main/java/com/google/devtools/build/lib/util/OS.java
--- bazel-orig/src/main/java/com/google/devtools/build/lib/util/OS.java 2018-04-01 14:52:36.327916523 +0000
+++ bazel/src/main/java/com/google/devtools/build/lib/util/OS.java      2018-04-01 14:53:02.530468640 +0000
@@ -21,11 +21,12 @@
 public enum OS {
   DARWIN("osx", "Mac OS X"),
   FREEBSD("freebsd", "FreeBSD"),
+  SOLARIS("solaris","SunOS"),
   LINUX("linux", "Linux"),
   WINDOWS("windows", "Windows"),
   UNKNOWN("unknown", "");
 
-  private static final EnumSet<OS> POSIX_COMPATIBLE = EnumSet.of(DARWIN, FREEBSD, LINUX);
+  private static final EnumSet<OS> POSIX_COMPATIBLE = EnumSet.of(DARWIN, FREEBSD, SOLARIS, LINUX);
 
   private final String canonicalName;
   private final String detectionName;
diff -uar bazel-orig/src/main/java/com/google/devtools/build/lib/vfs/LocalPath.java bazel/src/main/java/com/google/devtools/build/lib/vfs/LocalPath.java
--- bazel-orig/src/main/java/com/google/devtools/build/lib/vfs/LocalPath.java   2018-04-01 14:52:36.363892385 +0000
+++ bazel/src/main/java/com/google/devtools/build/lib/vfs/LocalPath.java        2018-04-01 14:53:02.531108306 +0000
@@ -667,6 +667,7 @@
     switch (OS.getCurrent()) {
       case LINUX:
       case FREEBSD:
+      case SOLARIS:
       case UNKNOWN:
         return new UnixOsPathPolicy();
       case DARWIN:
diff -uar bazel-orig/src/main/tools/jdk.BUILD bazel/src/main/tools/jdk.BUILD
--- bazel-orig/src/main/tools/jdk.BUILD 2018-04-01 14:52:36.380117028 +0000
+++ bazel/src/main/tools/jdk.BUILD      2018-04-01 14:53:02.531508432 +0000
@@ -21,6 +21,11 @@
 )
 
 filegroup(
+    name = "jni_md_header-solaris",
+    srcs = ["include/solaris/jni_md.h"],
+)
+
+filegroup(
     name = "jni_md_header-windows",
     srcs = ["include/win32/jni_md.h"],
 )
diff -uar bazel-orig/third_party/BUILD bazel/third_party/BUILD
--- bazel-orig/third_party/BUILD        2018-04-01 14:52:36.555267787 +0000
+++ bazel/third_party/BUILD     2018-04-01 14:53:02.532119695 +0000
@@ -662,6 +662,11 @@
 )
 
 config_setting(
+    name = "solaris",
+    values = {"host_cpu": "solaris"},
+)
+
+config_setting(
     name = "s390x",
     values = {"host_cpu": "s390x"},
 )
diff -uar bazel-orig/tools/cpp/BUILD bazel/tools/cpp/BUILD
--- bazel-orig/tools/cpp/BUILD  2018-04-01 14:52:36.549549685 +0000
+++ bazel/tools/cpp/BUILD       2018-04-01 14:53:02.532556059 +0000
@@ -40,6 +40,7 @@
         "armeabi-v7a|compiler": ":cc-compiler-armeabi-v7a",
         "darwin|compiler": ":cc-compiler-darwin",
         "freebsd|compiler": ":cc-compiler-freebsd",
+        "solaris|compiler": ":cc-compiler-solaris",
         "local|compiler": ":cc-compiler-local",
         "x64_windows|compiler": ":cc-compiler-x64_windows",
         "x64_windows_msvc|compiler": ":cc-compiler-x64_windows_msvc",
@@ -123,6 +124,20 @@
     all_files = ":empty",
     compiler_files = ":empty",
     cpu = "local",
+    dwp_files = ":empty",
+    dynamic_runtime_libs = [":empty"],
+    linker_files = ":empty",
+    objcopy_files = ":empty",
+    static_runtime_libs = [":empty"],
+    strip_files = ":empty",
+    supports_param_files = 0,
+)
+
+cc_toolchain(
+    name = "cc-compiler-solaris",
+    all_files = ":empty",
+    compiler_files = ":empty",
+    cpu = "local",
     dwp_files = ":empty",
     dynamic_runtime_libs = [":empty"],
     linker_files = ":empty",
diff -uar bazel-orig/tools/cpp/BUILD.static bazel/tools/cpp/BUILD.static
--- bazel-orig/tools/cpp/BUILD.static   2018-04-01 14:52:36.549606668 +0000
+++ bazel/tools/cpp/BUILD.static        2018-04-01 14:53:02.532895942 +0000
@@ -19,6 +19,7 @@
     toolchains = {
         "armeabi-v7a|compiler": ":cc-compiler-armeabi-v7a",
         "freebsd|compiler": ":cc-compiler-freebsd",
+        "solaris|compiler": ":cc-compiler-solaris",
         "x64_windows|msvc-cl": ":cc-compiler-x64_windows",
         "x64_windows|msys-gcc": ":cc-compiler-x64_windows_msys",
         "x64_windows|mingw-gcc": ":cc-compiler-x64_windows_mingw",
@@ -45,6 +46,20 @@
     all_files = ":empty",
     compiler_files = ":empty",
     cpu = "local",
+    dwp_files = ":empty",
+    dynamic_runtime_libs = [":empty"],
+    linker_files = ":empty",
+    objcopy_files = ":empty",
+    static_runtime_libs = [":empty"],
+    strip_files = ":empty",
+    supports_param_files = 0,
+)
+
+cc_toolchain(
+    name = "cc-compiler-solaris",
+    all_files = ":empty",
+    compiler_files = ":empty",
+    cpu = "local",
     dwp_files = ":empty",
     dynamic_runtime_libs = [":empty"],
     linker_files = ":empty",
diff -uar bazel-orig/tools/cpp/CROSSTOOL bazel/tools/cpp/CROSSTOOL
--- bazel-orig/tools/cpp/CROSSTOOL      2018-04-01 14:52:36.549774320 +0000
+++ bazel/tools/cpp/CROSSTOOL   2018-04-01 14:53:02.533700016 +0000
@@ -19,6 +19,10 @@
   toolchain_identifier: "local_freebsd"
 }
 default_toolchain {
+  cpu: "solaris"
+  toolchain_identifier: "local_solaris"
+}
+default_toolchain {
   cpu: "armeabi-v7a"
   toolchain_identifier: "stub_armeabi-v7a"
 }
@@ -388,6 +392,126 @@
   # Gold linker only? Can we enable this by default?
   # linker_flag: "-Wl,--warn-execstack"
   # linker_flag: "-Wl,--detect-odr-violations"
+
+  compilation_mode_flags {
+    mode: DBG
+    # Enable debug symbols.
+    compiler_flag: "-g"
+  }
+  compilation_mode_flags {
+    mode: OPT
+
+    # No debug symbols.
+    # Maybe we should enable https://gcc.gnu.org/wiki/DebugFission for opt or
+    # even generally? However, that can't happen here, as it requires special
+    # handling in Bazel.
+    compiler_flag: "-g0"
+
+    # Conservative choice for -O
+    # -O3 can increase binary size and even slow down the resulting binaries.
+    # Profile first and / or use FDO if you need better performance than this.
+    compiler_flag: "-O2"
+
+    # Disable assertions
+    compiler_flag: "-DNDEBUG"
+
+    # Removal of unused code and data at link time (can this increase binary size in some cases?).
+    compiler_flag: "-ffunction-sections"
+    compiler_flag: "-fdata-sections"
+    linker_flag: "-Wl,--gc-sections"
+  }
+  linking_mode_flags { mode: DYNAMIC }
+}
+
+toolchain {
+  abi_version: "local"
+  abi_libc_version: "local"
+  builtin_sysroot: ""
+  compiler: "compiler"
+  host_system_name: "local"
+  needsPic: true
+  supports_gold_linker: false
+  supports_incremental_linker: false
+  supports_fission: false
+  supports_interface_shared_objects: false
+  supports_normalizing_ar: false
+  supports_start_end_lib: false
+  target_libc: "local"
+  target_cpu: "solaris"
+  target_system_name: "local"
+  toolchain_identifier: "local_solaris"
+
+  tool_path { name: "ar" path: "/opt/local/bin/ar" }
+  tool_path { name: "compat-ld" path: "/usr/bin/ld" }
+  tool_path { name: "cpp" path: "/opt/local/gcc7/bin/cpp" }
+  tool_path { name: "dwp" path: "/usr/bin/dwp" }
+  tool_path { name: "gcc" path: "/opt/local/gcc7/bin/gcc" }
+  cxx_flag: "-std=c++0x"
+  linker_flag: "-lstdc++"
+  linker_flag: "-B/usr/bin/"
+
+  # TODO(bazel-team): In theory, the path here ought to exactly match the path
+  # used by gcc. That works because bazel currently doesn't track files at
+  # absolute locations and has no remote execution, yet. However, this will need
+  # to be fixed, maybe with auto-detection?
+  cxx_builtin_include_directory: "/opt/local/gcc7/lib"
+  cxx_builtin_include_directory: "/opt/local/include"
+  cxx_builtin_include_directory: "/usr/include"
+  tool_path { name: "gcov" path: "/opt/local/gcc7/bin/gcov" }
+
+  # C(++) compiles invoke the compiler (as that is the one knowing where
+  # to find libraries), but we provide LD so other rules can invoke the linker.
+  tool_path { name: "ld" path: "/usr/bin/ld" }
+
+  tool_path { name: "nm" path: "/opt/local/bin/nm" }
+  tool_path { name: "objcopy" path: "/usr/bin/objcopy" }
+  objcopy_embed_flag: "-I"
+  objcopy_embed_flag: "binary"
+  tool_path { name: "objdump" path: "/usr/bin/objdump" }
+  tool_path { name: "strip" path: "/usr/bin/strip" }
+
+  # Anticipated future default.
+  unfiltered_cxx_flag: "-no-canonical-prefixes"
+  unfiltered_cxx_flag: "-fno-canonical-system-headers"
+
+  # Make C++ compilation deterministic. Use linkstamping instead of these
+  # compiler symbols.
+  unfiltered_cxx_flag: "-Wno-builtin-macro-redefined"
+  unfiltered_cxx_flag: "-D__DATE__=\"redacted\""
+  unfiltered_cxx_flag: "-D__TIMESTAMP__=\"redacted\""
+  unfiltered_cxx_flag: "-D__TIME__=\"redacted\""
+
+  # Security hardening on by default.
+  # Conservative choice; -D_FORTIFY_SOURCE=2 may be unsafe in some cases.
+  # We need to undef it before redefining it as some distributions now have
+  # it enabled by default.
+  compiler_flag: "-U_FORTIFY_SOURCE"
+  compiler_flag: "-D_FORTIFY_SOURCE=1"
+  compiler_flag: "-fstack-protector"
+  linker_flag: "-Wl,-z,relro,-z,now"
+
+  # Enable coloring even if there's no attached terminal. Bazel removes the
+  # escape sequences if --nocolor is specified. This isn't supported by gcc
+  # on Ubuntu 14.04.
+  # compiler_flag: "-fcolor-diagnostics"
+
+  # All warnings are enabled. Maybe enable -Werror as well?
+  compiler_flag: "-Wall"
+  # Enable a few more warnings that aren't part of -Wall.
+  compiler_flag: "-Wunused-but-set-parameter"
+  # But disable some that are problematic.
+  compiler_flag: "-Wno-free-nonheap-object" # has false positives
+
+  # Keep stack frames for debugging, even in opt mode.
+  compiler_flag: "-fno-omit-frame-pointer"
+
+  # Anticipated future default.
+  linker_flag: "-no-canonical-prefixes"
+  # Have gcc return the exit code from ld.
+  linker_flag: "-pass-exit-codes"
+  # Gold linker only? Can we enable this by default?
+  # linker_flag: "-Wl,--warn-execstack"
+  # linker_flag: "-Wl,--detect-odr-violations"
 
   compilation_mode_flags {
     mode: DBG
diff -uar bazel-orig/tools/cpp/cc_configure.bzl bazel/tools/cpp/cc_configure.bzl
--- bazel-orig/tools/cpp/cc_configure.bzl       2018-04-01 14:52:36.550063824 +0000
+++ bazel/tools/cpp/cc_configure.bzl    2018-04-01 14:53:02.534036692 +0000
@@ -24,7 +24,7 @@
       Label("@bazel_tools//tools/cpp:dummy_toolchain.bzl"), "dummy_toolchain.bzl")
   env = repository_ctx.os.environ
   cpu_value = get_cpu_value(repository_ctx)
-  if cpu_value == "freebsd":
+  if (cpu_value == "freebsd" or cpu_value == "solaris"):
     # This is defaulting to the static crosstool, we should eventually
     # autoconfigure this platform too.  Theorically, FreeBSD should be
     # straightforward to add but we cannot run it in a docker container so
diff -uar bazel-orig/tools/jdk/BUILD bazel/tools/jdk/BUILD
--- bazel-orig/tools/jdk/BUILD  2018-04-01 14:52:36.548335741 +0000
+++ bazel/tools/jdk/BUILD       2018-04-01 14:53:02.534423986 +0000
@@ -77,6 +77,11 @@
 )
 
 alias(
+    name = "jni_md_header-solaris",
+    actual = "@local_jdk//:jni_md_header-solaris",
+)
+
+alias(
     name = "java",
     actual = "@local_jdk//:java",
 )
diff -uar bazel-orig/tools/platforms/BUILD bazel/tools/platforms/BUILD
--- bazel-orig/tools/platforms/BUILD    2018-04-01 14:52:36.549193491 +0000
+++ bazel/tools/platforms/BUILD 2018-04-01 14:53:02.534848733 +0000
@@ -56,6 +56,11 @@
 )
 
 constraint_value(
+    name = "solaris",
+    constraint_setting = ":os",
+)
+
+constraint_value(
     name = "linux",
     constraint_setting = ":os",
 )
@@ -83,6 +88,7 @@
     os_constraints = [
         ":osx",
         ":freebsd",
+        ":solaris",
         ":linux",
         ":windows",
     ],
@@ -100,6 +106,7 @@
     os_constraints = [
         ":osx",
         ":freebsd",
+        ":solaris",
         ":linux",
         ":windows",
     ],
diff -uar bazel-orig/tools/platforms/platforms.BUILD bazel/tools/platforms/platforms.BUILD
--- bazel-orig/tools/platforms/platforms.BUILD  2018-04-01 14:52:36.549245186 +0000
+++ bazel/tools/platforms/platforms.BUILD       2018-04-01 14:53:02.535174740 +0000
@@ -46,6 +46,11 @@
 )
 
 constraint_value(
+    name = "solaris",
+    constraint_setting = ":os",
+)
+
+constraint_value(
     name = "linux",
     constraint_setting = ":os",
 )
@@ -73,6 +78,7 @@
     os_constraints = [
         ":osx",
         ":freebsd",
+        ":solaris",
         ":linux",
         ":windows",
     ],
@@ -90,6 +96,7 @@
     os_constraints = [
         ":osx",
         ":freebsd",
+        ":solaris",
         ":linux",
         ":windows",
     ],

[siepkes]

So turns out I couldn't let this rest ;-) I've gotten past the point I was stuck and I've created bazelbuild/bazel#4969 and bazelbuild/bazel#4967 to fix some POSIX incompatibilities in Bazel to test the waters to see if they're open to such PR's. Other communities like NetBSD might also be interested in those changes.

[mamash]

You truly are determined! :)

[siepkes]

I came across this gem in the Bazel Wiki:

Let's say that for some reason we need to configure our C++ toolchain in Bazel. This is currently quite a frustrating task. To quote one of many:

This is a fabulously difficult project that causes hardened engineers to stare blankly at screens in defeat.

But after a whole bunch of changes I'm getting somewhere! However I could use your help with the following error I'm bumping in to (see below). gcc in pkgsrc is configured (compile time) to use the SmartOS (aka Solaris, aka Illumos) linker instead of the GNU linker. The SmartOS linker does not know how how to find the GCC shared libs (-lgcc). How should I deal with this? Is this something that shouldn't have occurred in the first place (I don't specify the use of -lgcc anywhere so I'm assuming GCC does it)? I'm assuming using the SmartOS linker is preferred (since gcc is compiled that way in pkgsrc)?

ERROR: /root/bazel/third_party/protobuf/3.4.0/BUILD:399:1: Linking of rule '//third_party/protobuf/3.4.0:js_embed' failed (Exit 1): gcc failed: error executing command 
  (cd /tmp/bazel_NJvKzmPd/out/execroot/io_bazel && \
  exec env - \
    PATH=/usr/local/sbin:/usr/local/bin:/opt/local/sbin:/opt/local/bin:/usr/sbin:/usr/bin:/sbin:/data/pkgbuild/scripts \
    PWD=/proc/self/cwd \
  /opt/local/gcc49/bin/gcc -o bazel-out/host/bin/third_party/protobuf/3.4.0/js_embed bazel-out/host/bin/third_party/protobuf/3.4.0/_objs/js_embed/third_party/protobuf/3.4.0/src/google/protobuf/compiler/js/embed.o -lstdc++ -B/usr/bin/ -no-canonical-prefixes -pass-exit-codes -Wl,-S)
ld: fatal: dlopen() of support library (-lgcc) failed with error: ld.so.1: ld: fatal: -lgcc: open failed: No such file or directory
collect2: error: ld returned 1 exit status
Target //src:bazel failed to build
ERROR: /root/bazel/src/main/cpp/BUILD:74:1 Linking of rule '//third_party/protobuf/3.4.0:js_embed' failed (Exit 1): gcc failed: error executing command 
  (cd /tmp/bazel_NJvKzmPd/out/execroot/io_bazel && \
  exec env - \
    PATH=/usr/local/sbin:/usr/local/bin:/opt/local/sbin:/opt/local/bin:/usr/sbin:/usr/bin:/sbin:/data/pkgbuild/scripts \
    PWD=/proc/self/cwd \
  /opt/local/gcc49/bin/gcc -o bazel-out/host/bin/third_party/protobuf/3.4.0/js_embed bazel-out/host/bin/third_party/protobuf/3.4.0/_objs/js_embed/third_party/protobuf/3.4.0/src/google/protobuf/compiler/js/embed.o -lstdc++ -B/usr/bin/ -no-canonical-prefixes -pass-exit-codes -Wl,-S)
INFO: Elapsed time: 546.197s, Critical Path: 405.28s
FAILED: Build did NOT complete successfully

ERROR: Could not build Bazel

[jperkin]

You shouldn't need to, GCC figures out by itself where to find libgcc, unless it is told not to look in the default locations. All of the -lstdc++ -B/usr/bin/ -no-canonical-prefixes -pass-exit-codes -Wl,-S arguments do not look correct, I'd start by trying to remove them all (-lstdc++ is valid but GCC should be able to figure out if when it needs it, and hardcoding will only introduce problems for building with clang).

[siepkes]

@jperkin Thanks for the assist! I'll investigate where those flags are coming from.

I think the issue with Bazel is that it is (unfortunately for us) not just another build system. It is different from other build systems in the regard that Bazel defines the entire toolchain (compiler flags, linker flags, etc.). That's a fundamentally different view of things then for example autoconf which will try and detect the environment it is in and adapt to that. Bazel's root's are with Google which uses it to build all projects in their giant GIT mono-repo. I have currently defined a SmartOS toolchain with GCC 4.9 in Bazel but for clang support a new toolchain will have to be defined inside the Bazel config. So getting it to work is one thing but getting it into pkgsrc in such a way that it will be maintainable is going to be another.

[siepkes]

The paths seem to be correct (see https://gist.github.com/siepkes/956b3bbd17072e8553b4e8f2104025f7). One of the paths is /opt/local/gcc49/x86_64-sun-solaris2.11/lib/amd64 where libgcc_s.so.1 actually lives. Could it be that -lgcc must actually be -lgcc_s for the Solaris linker to work? Or should -lgcc work? I would test it but I can't seem to find how Bazel gets the idea it should use -lgcc...

BTW I tried cheating by creating a symlink called libgcc.so but that didn't fool the linker.

[mamash]

Yes, it's libgcc_s, but one thing is that ideally the linker should not be called directly but rather through GCC. That's where GCC automatically links in any required objects with proper paths. If Bazel tries to call ld directly than things can get ugly.

[siepkes]

As far as I can tell the linker is called via gcc (see: https://gist.github.com/siepkes/956b3bbd17072e8553b4e8f2104025f7 ) or am I missing something?

[mamash]

In that case I'd just patch out any attempts to link in -lgcc and the like. Unless Bazel also "helpfully" uses something like -nostdlib or -nodefaultlibs, the standard GCC libs like libgcc_s or libstdc++ should be linked in automatically.

The entire "-lgcc -lgcc_eh -lc -lgcc -lgcc_eh" part sounds fishy.

[siepkes]

I think I figured out where those flags are coming from. Because gcc in pkgsrc is configured with -static-libgcc (see gcc -v) gcc expands -static-libgcc to -lgcc -lgcc_eh -lc -lgcc -lgcc_eh it seems.

-static-libgcc is not used in Bazel itself but in one of its depdencies, protobuf.