Skip to content

Refactor dockerfile to support Buildkite AND Github Actions #71954

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Nov 16, 2023
Merged

Refactor dockerfile to support Buildkite AND Github Actions #71954

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
348 changes: 255 additions & 93 deletions libcxx/utils/ci/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,29 @@
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#
#===----------------------------------------------------------------------===##

#
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not attached to this line: What testing did you do? Did you build the images and run some of the CI jobs to make sure things won't blow up when we change to these images?

# This Dockerfile describes the base image used to run the various libc++
# build bots. By default, the image runs the Buildkite Agent, however one
# can also just start the image with a shell to debug CI failures.
# This file defines the buildkite and github actions builder images.
# You can build & push both images using:
#
# docker compose build
# docker compose push
#
# Or you can select a single image to build & push using:
#
# docker compose build buildkite-builder
# docker compose push buildkite-builder
#
# The final images can be found at
#
# ghcr.io/libcxx/buildkite-builder
# ghcr.io/libcxx/actions-builder
# ghcr.io/libcxx/android-buildkite-builder
#
# Members of the github.com/libcxx/ organizations have permissions required to push new images.
#
# ===----------------------------------------------------------------------===##
# Running the buildkite image
# ===----------------------------------------------------------------------===##
#
# To start a Buildkite Agent, run it as:
# $ docker run --env-file <secrets> -it $(docker build -q libcxx/utils/ci)
Expand All @@ -21,42 +39,82 @@
#
# If you're only looking to run the Docker image locally for debugging a
# build bot, see the `run-buildbot-container` script located in this directory.
#
# A pre-built version of this image is maintained on Github under the libc++ organization, as ghcr.io/libcxx/libcxx-builder.
# To update the image, rebuild it and push it to github (all members of the libc++ organization should be able to do this).
#
# $ docker compose build
# $ docker compose push
#

FROM ubuntu:jammy

# HACK: We set the base image in the docker-compose file depending on the final target (buildkite vs github actions).
# This means we have a much slower container build, but we can use the same Dockerfile for both targets.
ARG BASE_IMAGE
FROM $BASE_IMAGE AS builder-base

# Make sure apt-get doesn't try to prompt for stuff like our time zone, etc.
ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update && apt-get install -y bash curl
# populated in the docker-compose file
ARG GCC_LATEST_VERSION
ENV GCC_LATEST_VERSION=${GCC_LATEST_VERSION}

# populated in the docker-compose file
ARG LLVM_HEAD_VERSION
ENV LLVM_HEAD_VERSION=${LLVM_HEAD_VERSION}

# HACK: The github actions runner image already has sudo and requires its use. The buildkite base image does not.
# Reconcile this.
RUN <<EOF
apt-get update || true
apt-get install -y sudo || true
echo "ALL ALL = (ALL) NOPASSWD: ALL" | tee /etc/sudoers || true
EOF

RUN sudo apt-get update \
&& sudo apt-get install -y \
python3 \
python3-distutils \
python3-psutil \
git \
gdb \
ccache \
gpg \
wget \
bash \
curl \
python3 \
python3-dev \
libpython3-dev \
uuid-dev \
libncurses5-dev \
swig3.0 \
libxml2-dev \
libedit-dev \
language-pack-en \
language-pack-fr \
language-pack-ja \
language-pack-ru \
language-pack-zh-hans \
lsb-release \
wget \
unzip \
software-properties-common \
&& sudo rm -rf /var/lib/apt/lists/*


# Install various tools used by the build or the test suite
#RUN apt-get update && apt-get install -y ninja-build python3 python3-distutils python3-psutil git gdb ccache
# TODO add ninja-build once 1.11 is available in Ubuntu, also remove the manual installation.
RUN apt-get update && apt-get install -y python3 python3-distutils python3-psutil git gdb ccache
RUN apt-get update && apt-get install -y wget && \
wget -qO /usr/local/bin/ninja.gz https://github.com/ninja-build/ninja/releases/latest/download/ninja-linux.zip && \
gunzip /usr/local/bin/ninja.gz && \
chmod a+x /usr/local/bin/ninja

# Install dependencies required to run the LLDB data formatter tests
RUN apt-get update && apt-get install -y python3 python3-dev libpython3-dev uuid-dev libncurses5-dev swig3.0 libxml2-dev libedit-dev

# Locales for gdb and localization tests
RUN apt-get update && apt-get install -y language-pack-en language-pack-fr \
language-pack-ja language-pack-ru \
language-pack-zh-hans
# These two are not enabled by default so generate them
RUN printf "fr_CA ISO-8859-1\ncs_CZ ISO-8859-2" >> /etc/locale.gen
RUN mkdir /usr/local/share/i1en/
RUN printf "fr_CA ISO-8859-1\ncs_CZ ISO-8859-2" >> /usr/local/share/i1en/SUPPORTED
RUN locale-gen
RUN <<EOF
wget -qO /tmp/ninja.gz https://github.com/ninja-build/ninja/releases/latest/download/ninja-linux.zip
gunzip /tmp/ninja.gz
chmod a+x /tmp/ninja
sudo mv /tmp/ninja /usr/local/bin/ninja
EOF


# These two locales are not enabled by default so generate them
RUN <<EOF
printf "fr_CA ISO-8859-1\ncs_CZ ISO-8859-2" | sudo tee -a /etc/locale.gen
sudo mkdir /usr/local/share/i1en/
printf "fr_CA ISO-8859-1\ncs_CZ ISO-8859-2" | sudo tee -a /usr/local/share/i1en/SUPPORTED
sudo locale-gen
EOF

# Install Clang <latest>, <latest-1> and ToT, which are the ones we support.
# We also install <latest-2> because we need to support the "latest-1" of the
Expand All @@ -65,75 +123,179 @@ RUN locale-gen
# LLVM 15, we still need to have Clang 12 in this Docker image because the LLVM
# 14 release branch CI uses it. The tip-of-trunk CI will never use Clang 12,
# though.
ARG LLVM_HEAD_VERSION # populated in the docker-compose file
ENV LLVM_HEAD_VERSION=${LLVM_HEAD_VERSION}
RUN apt-get update && apt-get install -y lsb-release wget software-properties-common
RUN wget https://apt.llvm.org/llvm.sh -O /tmp/llvm.sh
RUN bash /tmp/llvm.sh $(($LLVM_HEAD_VERSION - 3)) # for CI transitions
RUN bash /tmp/llvm.sh $(($LLVM_HEAD_VERSION - 2)) # previous release
RUN bash /tmp/llvm.sh $(($LLVM_HEAD_VERSION - 1)) # latest release
RUN bash /tmp/llvm.sh $LLVM_HEAD_VERSION # current ToT

# Install clang-scan-deps, which is required to build modules; always all supported versions.
RUN apt-get update && apt-get install -y clang-tools-$(($LLVM_HEAD_VERSION - 3)) \
clang-tools-$(($LLVM_HEAD_VERSION - 2)) \
clang-tools-$(($LLVM_HEAD_VERSION - 1)) \
clang-tools-$LLVM_HEAD_VERSION

# Install clang-format; always use the lastest stable branch.
RUN apt-get update && apt-get install -y clang-format-$(($LLVM_HEAD_VERSION - 2)) clang-format-$(($LLVM_HEAD_VERSION - 1))

# Install clang-tidy
# TODO(LLVM-17) revert D148831 to only install $(($LLVM_HEAD_VERSION - 1)) and $LLVM_HEAD_VERSION
# The usage of the ToT version is needed due to module issues with Clang 16
RUN apt-get update && apt-get install -y clang-tidy-$(($LLVM_HEAD_VERSION - 2)) clang-tidy-$(($LLVM_HEAD_VERSION - 1)) clang-tidy-$LLVM_HEAD_VERSION

# Install llvm-dev and libclang-dev to compile custom clang-tidy checks
# TODO(LLVM-17) revert D148831 to only install $(($LLVM_HEAD_VERSION - 1)) and $LLVM_HEAD_VERSION
# The usage of the ToT version is needed due to module issues with Clang 16
RUN apt-get update && apt-get install -y llvm-$(($LLVM_HEAD_VERSION - 2))-dev llvm-$(($LLVM_HEAD_VERSION - 1))-dev llvm-$LLVM_HEAD_VERSION-dev \
libclang-$(($LLVM_HEAD_VERSION - 2))-dev libclang-$(($LLVM_HEAD_VERSION - 1))-dev libclang-$LLVM_HEAD_VERSION-dev \
libomp5-$LLVM_HEAD_VERSION
RUN <<EOF
sudo apt-get update
wget https://apt.llvm.org/llvm.sh -O /tmp/llvm.sh
chmod +x /tmp/llvm.sh
sudo /tmp/llvm.sh $(($LLVM_HEAD_VERSION - 3)) all # for CI transitions
sudo /tmp/llvm.sh $(($LLVM_HEAD_VERSION - 2)) all # previous release
sudo /tmp/llvm.sh $(($LLVM_HEAD_VERSION - 1)) all # latest release
sudo /tmp/llvm.sh $LLVM_HEAD_VERSION all # current ToT
sudo apt-get install -y libomp5-$LLVM_HEAD_VERSION
sudo rm -rf /var/lib/apt/lists/*
EOF

# Install the most recent GCC, like clang install the previous version as a transition.
RUN add-apt-repository ppa:ubuntu-toolchain-r/test
ARG GCC_LATEST_VERSION # populated in the docker-compose file
ENV GCC_LATEST_VERSION=${GCC_LATEST_VERSION}
RUN apt-get update && apt install -y gcc-$((GCC_LATEST_VERSION - 1)) g++-$((GCC_LATEST_VERSION - 1))
RUN apt-get update && apt install -y gcc-$GCC_LATEST_VERSION g++-$GCC_LATEST_VERSION

# Remove all, no longer needed, apt data
RUN rm -rf /var/lib/apt/lists/*

# Install a recent CMake
RUN wget https://github.com/Kitware/CMake/releases/download/v3.21.1/cmake-3.21.1-linux-x86_64.sh -O /tmp/install-cmake.sh
RUN bash /tmp/install-cmake.sh --prefix=/usr --exclude-subdir --skip-license
RUN rm /tmp/install-cmake.sh

# Install a newer CMake for modules
# TODO Remove the duplicated installation when all runtimes can be build with CMake 3.28.
RUN wget https://github.com/Kitware/CMake/releases/download/v3.27.1/cmake-3.27.1-linux-x86_64.sh -O /tmp/install-cmake.sh
RUN bash /tmp/install-cmake.sh --prefix=/opt --exclude-subdir --skip-license
RUN rm /tmp/install-cmake.sh

RUN wget https://github.com/Kitware/CMake/releases/download/v3.28.0-rc4/cmake-3.28.0-rc4-linux-x86_64.sh -O /tmp/install-cmake.sh
RUN mkdir /opt/cmake-3.28
RUN bash /tmp/install-cmake.sh --prefix=/opt/cmake-3.28 --exclude-subdir --skip-license
RUN rm /tmp/install-cmake.sh

# Change the user to a non-root user, since some of the libc++ tests
# (e.g. filesystem) require running as non-root. Also setup passwordless sudo.
RUN apt-get update && apt-get install -y sudo
RUN echo "ALL ALL = (ALL) NOPASSWD: ALL" >> /etc/sudoers
RUN useradd --create-home libcxx-builder
RUN <<EOF
sudo add-apt-repository ppa:ubuntu-toolchain-r/test
sudo apt-get update
sudo apt-get install -y \
gcc-$((GCC_LATEST_VERSION - 1)) \
g++-$((GCC_LATEST_VERSION - 1)) \
gcc-$GCC_LATEST_VERSION \
g++-$GCC_LATEST_VERSION
sudo rm -rf /var/lib/apt/lists/*
EOF

RUN <<EOF
# Install a recent CMake
wget https://github.com/Kitware/CMake/releases/download/v3.21.1/cmake-3.21.1-linux-x86_64.sh -O /tmp/install-cmake.sh
sudo bash /tmp/install-cmake.sh --prefix=/usr --exclude-subdir --skip-license
rm /tmp/install-cmake.sh

# Install a newer CMake for modules
# TODO Remove the duplicated installation when all runtimes can be build with CMake 3.28.
wget https://github.com/Kitware/CMake/releases/download/v3.27.1/cmake-3.27.1-linux-x86_64.sh -O /tmp/install-cmake.sh
sudo bash /tmp/install-cmake.sh --prefix=/opt --exclude-subdir --skip-license
rm /tmp/install-cmake.sh

wget https://github.com/Kitware/CMake/releases/download/v3.28.0-rc4/cmake-3.28.0-rc4-linux-x86_64.sh -O /tmp/install-cmake.sh
sudo mkdir /opt/cmake-3.28
sudo bash /tmp/install-cmake.sh --prefix=/opt/cmake-3.28 --exclude-subdir --skip-license
rm /tmp/install-cmake.sh
EOF

# ===----------------------------------------------------------------------===##
# Android Buildkite Image
# ===----------------------------------------------------------------------===##

FROM ubuntu:jammy AS android-builder-base

ARG ANDROID_CLANG_VERSION
ARG ANDROID_CLANG_PREBUILTS_COMMIT
ARG ANDROID_SYSROOT_BID

RUN apt-get update && apt-get install -y curl unzip git

# Install the Android platform tools (e.g. adb) into /opt/android/sdk.
RUN <<EOF
mkdir -p /opt/android/sdk
cd /opt/android/sdk
curl -LO https://dl.google.com/android/repository/platform-tools-latest-linux.zip
unzip platform-tools-latest-linux.zip
rm platform-tools-latest-linux.zip
EOF

# Install the current Android compiler. Specify the prebuilts commit to retrieve
# this compiler version even after it's removed from HEAD.

ENV ANDROID_CLANG_VERSION=$ANDROID_CLANG_VERSION
ENV ANDROID_CLANG_PREBUILTS_COMMIT=$ANDROID_CLANG_PREBUILTS_COMMIT
RUN <<EOF
git clone --filter=blob:none --sparse \
https://android.googlesource.com/platform/prebuilts/clang/host/linux-x86 \
/opt/android/clang
git -C /opt/android/clang checkout ${ANDROID_CLANG_PREBUILTS_COMMIT}
git -C /opt/android/clang sparse-checkout add clang-${ANDROID_CLANG_VERSION}
rm -fr /opt/android/clang/.git
ln -sf /opt/android/clang/clang-${ANDROID_CLANG_VERSION} /opt/android/clang/clang-current
# The "git sparse-checkout" and "ln" commands succeed even if nothing was
# checked out, so use this "ls" command to fix that.
ls /opt/android/clang/clang-current/bin/clang
EOF

# Install an Android sysroot. New AOSP sysroots are available at
# https://ci.android.com/builds/branches/aosp-main/grid, the "ndk" target. The
# NDK also makes its sysroot prebuilt available at
# https://android.googlesource.com/platform/prebuilts/ndk/+/refs/heads/dev/platform/sysroot.

ENV ANDROID_SYSROOT_BID=$ANDROID_SYSROOT_BID
RUN <<EOF
cd /opt/android
curl -L -o ndk_platform.tar.bz2 \
https://androidbuildinternal.googleapis.com/android/internal/build/v3/builds/${ANDROID_SYSROOT_BID}/ndk/attempts/latest/artifacts/ndk_platform.tar.bz2/url
tar xf ndk_platform.tar.bz2
rm ndk_platform.tar.bz2
EOF

# Install Docker
RUN <<EOF
curl -fsSL https://get.docker.com -o /tmp/get-docker.sh
sh /tmp/get-docker.sh
rm /tmp/get-docker.sh

# Install Docker. Mark the binary setuid so it can be run without prefixing it
# with sudo. Adding the container user to the docker group doesn't work because
# /var/run/docker.sock is owned by the host's docker GID, not the container's
# docker GID.
chmod u+s /usr/bin/docker
EOF

# ===----------------------------------------------------------------------===##
# Buildkite Builder Image
# ===----------------------------------------------------------------------===##
#
# IMAGE: ghcr.io/libcxx/buildkite-builder.
#
FROM builder-base AS buildkite-builder

# Create the libcxx-builder user, regardless of if we use it or not
RUN sudo useradd --create-home libcxx-builder

USER libcxx-builder
WORKDIR /home/libcxx-builder

# Install the Buildkite agent and dependencies. This must be done as non-root
# for the Buildkite agent to be installed in a path where we can find it.
RUN bash -c "$(curl -sL https://raw.githubusercontent.com/buildkite/agent/main/install.sh)"
RUN <<EOF
cd /home/libcxx-builder
curl -sL https://raw.githubusercontent.com/buildkite/agent/main/install.sh -o /tmp/install-agent.sh
bash /tmp/install-agent.sh
rm /tmp/install-agent.sh
echo "tags=\"queue=libcxx-builders,arch=$(uname -m),os=linux\"" \
>> /home/libcxx-builder/.buildkite-agent/buildkite-agent.cfg
EOF

USER libcxx-builder
WORKDIR /home/libcxx-builder

ENV PATH="${PATH}:/home/libcxx-builder/.buildkite-agent/bin"
RUN echo "tags=\"queue=libcxx-builders,arch=$(uname -m),os=linux\"" >> "/home/libcxx-builder/.buildkite-agent/buildkite-agent.cfg"

# By default, start the Buildkite agent (this requires a token).
CMD ["buildkite-agent", "start"]

# ===----------------------------------------------------------------------===##
# Android Buildkite Builder Image
# ===----------------------------------------------------------------------===##
#
# IMAGE: ghcr.io/libcxx/android-buildkite-builder.
#
FROM buildkite-builder AS android-buildkite-builder

COPY --from=android-builder-base /opt/android /opt/android
COPY ./vendor/android/container-setup.sh /opt/android/container-setup.sh

ENV PATH="/opt/android/sdk/platform-tools:${PATH}"

USER libcxx-builder
WORKDIR /home/libcxx-builder

# Reset the configuration, we pass the configuration via the environment.
RUN cp /home/libcxx-builder/.buildkite-agent/buildkite-agent.dist.cfg \
/home/libcxx-builder/.buildkite-agent/buildkite-agent.cfg

# Modify the Buildkite agent cmdline to do Android setup stuff first.
CMD /opt/android/container-setup.sh && buildkite-agent start

# ===----------------------------------------------------------------------===##
# Github Actions Builder Image
# ===----------------------------------------------------------------------===##
#
# IMAGE: ghcr.io/libcxx/actions-builder.
#
FROM builder-base AS actions-builder

WORKDIR /home/runner
USER runner



Loading