Skip to content

Commit befafd5

Browse files
ryancragunryancragun
authored
[VAULT-35682] build(cgo): Build CGO binaries in a container (#30834)
Ubuntu 20.04 has reached EOL and is no longer a supported runner host distro. Historically we've relied on it for our CGO builds as it contains an old enough version of glibc that we can retain compatibility with all of our supported distros and build on a single host distro. Rather than requiring a new RHEL 8 builder (or some equivalent), we instead build CGO binaries inside an Ubuntu 20.04 container along with its glibc and various C compilers. I've separated out system package changes, the Go toolchain install, and external build tools tools install into different container layers so that the builder container used for each branch is maximally cacheable. On cache misses these changes result in noticeably longer build times for CGO binaries. That is unavoidable with this strategy. Most of the time our builds will get a cache hit on all layers unless they've changed any of the following: - .build/* - .go-version - .github/actions/build-vault - tools/tools.sh - Dockerfile I've tried my best to reduce the cache space used by each layer. Currently our build container takes about 220MB of cache space. About half of that ought to be shared cache between main and release branches. I would expect total new cache used to be in the 500-600MB range, or about 5% of our total space. Some follow-up idea that we might want to consider: - Build everything inside the build container and remove the github actions that set up external tools - Instead of building external tools with `go install`, migrate them into build scripts that install pre-built `linux/amd64` binaries - Migrate external to `go tool` and use it in the builder container. This requires us to be on 1.24 everywhere so ought not be considered until that is a reality. Signed-off-by: Ryan Cragun <[email protected]>
1 parent 8bcd3e7 commit befafd5

File tree

10 files changed

+212
-57
lines changed

10 files changed

+212
-57
lines changed

‎.build/entrypoint.sh

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
#!/bin/bash
2+
# Copyright (c) HashiCorp, Inc.
3+
# SPDX-License-Identifier: BUSL-1.1
4+
5+
set -e
6+
7+
fail() {
8+
echo "$1" 1>&2
9+
exit 1
10+
}
11+
12+
[[ -z "$GOARCH" ]] && fail "A GOARCH has not been defined"
13+
[[ -z "$GITHUB_TOKEN" ]] && fail "A GITHUB_TOKEN has not been defined"
14+
15+
host_arch="$(dpkg --print-architecture)"
16+
host_arch="${host_arch##*-}"
17+
if [[ "$host_arch" != "$GOARCH" ]]; then
18+
# We're building for a different architecture than our target host OS so
19+
# we have to tell the Go compiler to use the correct C cross-compiler for
20+
# our target instead of relying on the host C compiler.
21+
#
22+
# https://packages.ubuntu.com/search?suite=noble&section=all&arch=any&keywords=linux-gnu-gcc&searchon=contents
23+
case "$GOARCH" in
24+
amd64)
25+
export CC=x86_64-linux-gnu-gcc
26+
;;
27+
arm64)
28+
export CC=aarch64-linux-gnu-gcc
29+
;;
30+
s390x)
31+
export CC=s390x-linux-gnu-gcc
32+
;;
33+
*)
34+
fail "Building for $GOARCH has not been implemented"
35+
;;
36+
esac
37+
fi
38+
39+
# Assume that /build is where we've mounted the vault repo.
40+
git config --global --add safe.directory /build
41+
git config --global url."https://${GITHUB_TOKEN}@github.com".insteadOf "https://github.com"
42+
43+
# Exec our command
44+
cd build || exit 1
45+
exec "$@"

‎.build/go.sh

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
#!/bin/bash
2+
# Copyright (c) HashiCorp, Inc.
3+
# SPDX-License-Identifier: BUSL-1.1
4+
set -e
5+
6+
host_arch="$(dpkg --print-architecture)"
7+
host_arch="${host_arch##*-}"
8+
curl -L "https://go.dev/dl/go${GO_VERSION}.linux-${host_arch}.tar.gz" | tar -C /opt -zxv

‎.build/system.sh

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
#!/bin/bash
2+
# Copyright (c) HashiCorp, Inc.
3+
# SPDX-License-Identifier: BUSL-1.1
4+
5+
set -e
6+
7+
export DEBIAN_FRONTEND=noninteractive
8+
9+
install() {
10+
apt-get install -y "$@"
11+
}
12+
13+
# Install our cross building tools
14+
# https://packages.ubuntu.com/search?suite=noble&section=all&arch=any&keywords=crossbuild-essential&searchon=names
15+
16+
apt-get update
17+
apt-get install -y --no-install-recommends build-essential \
18+
gcc-s390x-linux-gnu \
19+
crossbuild-essential-s390x \
20+
ca-certificates \
21+
curl \
22+
git
23+
24+
host_arch="$(dpkg --print-architecture)"
25+
host_arch="${host_arch##*-}"
26+
case "$host_arch" in
27+
amd64)
28+
install crossbuild-essential-arm64 gcc-aarch64-linux-gnu
29+
;;
30+
arm64)
31+
install gcc-x86-64-linux-gnu
32+
;;
33+
*)
34+
echo "Building on $host_arch has not been implemented" 1>&2
35+
exit 1
36+
;;
37+
esac
38+
39+
# Clean up after ourselves for a minimal image
40+
apt-get clean
41+
rm -rf /var/lib/apt/lists/*

‎.github/actions/build-vault/action.yml

Lines changed: 70 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -11,59 +11,44 @@ description: |
1111
1212
inputs:
1313
github-token:
14-
type: string
1514
description: An elevated Github token to access private Go modules if necessary.
1615
default: ""
1716
cgo-enabled:
18-
type: number
1917
description: Enable or disable CGO during the build.
20-
default: 0
18+
default: "0"
2119
create-docker-container:
22-
type: boolean
2320
description: Package the binary into a Docker/AWS container.
24-
default: true
21+
default: "true"
2522
create-redhat-container:
26-
type: boolean
2723
description: Package the binary into a Redhat container.
28-
default: false
24+
default: "false"
2925
create-packages:
30-
type: boolean
3126
description: Package the binaries into deb and rpm formats.
32-
default: true
27+
default: "true"
3328
goos:
34-
type: string
3529
description: The Go GOOS value environment variable to set during the build.
3630
goarch:
37-
type: string
3831
description: The Go GOARCH value environment variable to set during the build.
3932
goarm:
40-
type: string
4133
description: The Go GOARM value environment variable to set during the build.
4234
default: ""
4335
goexperiment:
44-
type: string
4536
description: Which Go experiments to enable.
4637
default: ""
4738
go-tags:
48-
type: string
4939
description: A comma separated list of tags to pass to the Go compiler during build.
5040
default: ""
5141
package-name:
52-
type: string
5342
description: The name to use for the linux packages.
5443
default: ${{ github.event.repository.name }}
5544
vault-binary-name:
56-
type: string
5745
description: The name of the vault binary.
5846
default: vault
5947
vault-edition:
60-
type: string
6148
description: The edition of vault to build.
6249
vault-version:
63-
type: string
6450
description: The version metadata to inject into the build via the linker.
6551
web-ui-cache-key:
66-
type: string
6752
description: The cache key for restoring the pre-built web UI artifact.
6853

6954
outputs:
@@ -74,29 +59,12 @@ outputs:
7459
runs:
7560
using: composite
7661
steps:
77-
- name: Ensure zstd is available for actions/cache
78-
# actions/cache restores based on cache key and "cache version", the former is unique to the
79-
# build job or web UI, the latter is a hash which is based on the runner OS, the paths being
80-
# cached, and the program used to compress it. Most of our workflows will use zstd to compress
81-
# the cached artifact so we have to have it around for our machines to get both a version match
82-
# and to decompress it. Most runners include zstd by default but there are exception like
83-
# our Ubuntu 20.04 compatibility runners which do not.
84-
shell: bash
85-
run: which zstd || (sudo apt update && sudo apt install -y zstd)
86-
- uses: ./.github/actions/set-up-go
62+
- id: set-up-go
63+
uses: ./.github/actions/set-up-go
8764
with:
8865
github-token: ${{ inputs.github-token }}
89-
- uses: ./.github/actions/install-external-tools
90-
- if: inputs.goarch == 's390x' && inputs.vault-edition == 'ent.hsm'
91-
name: Configure CGO compiler for HSM edition on s390x
92-
shell: bash
93-
run: |
94-
sudo apt-get update
95-
sudo apt-get install -y gcc-multilib-s390x-linux-gnu
96-
{
97-
echo "CC=s390x-linux-gnu-gcc"
98-
echo "CC_FOR_TARGET=s390x-linux-gnu-gcc"
99-
} | tee -a "$GITHUB_ENV"
66+
- if: inputs.cgo-enabled == '0'
67+
uses: ./.github/actions/install-external-tools
10068
- if: inputs.vault-edition != 'ce'
10169
name: Configure Git
10270
shell: bash
@@ -126,15 +94,27 @@ runs:
12694
build_step_name='Vault ${{ inputs.goos }} ${{ inputs.goarch }} v${{ inputs.vault-version }}+${{ inputs.vault-edition }}'
12795
package_version='${{ inputs.vault-version }}+ent' # this should always be +ent here regardless of enterprise edition
12896
fi
97+
# Generate a builder cache key that considers anything that might change
98+
# our build container, including:
99+
# - The Go version we're building with
100+
# - External Go build tooling as defined in tools/tools.sh
101+
# - The Dockerfile or .build directory
102+
# - The build-vault Github action
103+
docker_sha=$(git ls-tree HEAD Dockerfile --object-only --abbrev=5)
104+
build_sha=$(git ls-tree HEAD .build --object-only --abbrev=5)
105+
tools_sha=$(git ls-tree HEAD tools/tools.sh --object-only --abbrev=5)
106+
github_sha=$(git ls-tree HEAD .github/actions/build-vault --object-only --abbrev=5)
129107
{
130108
echo "artifact-basename=$(make ci-get-artifact-basename)"
131109
echo "binary-path=dist/${{ inputs.vault-binary-name }}"
132110
echo "build-step-name=${build_step_name}"
111+
echo "vault-builder-cache-key=${docker_sha}-${build_sha}-${tools_sha}-${github_sha}-$(cat .go-version)"
133112
echo "package-version=${package_version}"
134113
} | tee -a "$GITHUB_OUTPUT"
135-
- name: ${{ steps.metadata.outputs.build-step-name }}
114+
- if: inputs.cgo-enabled == '0'
115+
name: ${{ steps.metadata.outputs.build-step-name }}
136116
env:
137-
CGO_ENABLED: ${{ inputs.cgo-enabled }}
117+
CGO_ENABLED: 0
138118
GO_TAGS: ${{ inputs.go-tags }}
139119
GOARCH: ${{ inputs.goarch }}
140120
GOARM: ${{ inputs.goarm }}
@@ -145,6 +125,54 @@ runs:
145125
VERSION_METADATA: ${{ inputs.vault-edition != 'ce' && inputs.vault-edition || '' }}
146126
shell: bash
147127
run: make ci-build
128+
- if: inputs.cgo-enabled == '1'
129+
uses: docker/setup-buildx-action@b5ca514318bd6ebac0fb2aedd5d36ec1b5c232a2 # v3.10.0
130+
with:
131+
driver-opts: network=host # So we can run our own little registry
132+
- if: inputs.cgo-enabled == '1'
133+
shell: bash
134+
run: docker run -d -p 5000:5000 --restart always --name registry registry:2
135+
- if: inputs.cgo-enabled == '1'
136+
name: Build CGO builder image
137+
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0
138+
env:
139+
DOCKER_BUILD_SUMMARY: false
140+
with:
141+
context: .
142+
build-args: |
143+
GO_VERSION=${{ steps.set-up-go.outputs.go-version }}
144+
# Only build a container for the host OS since the same container
145+
# handles cross building.
146+
platforms: linux/amd64
147+
push: true
148+
target: builder
149+
tags: localhost:5000/vault-builder:${{ steps.metadata.outputs.vault-builder-cache-key }}
150+
# Upload the resulting minimal image to actions cache. This could
151+
# be a problem if the resulting images are too big.
152+
cache-from: type=gha,scope=vault-builder-${{ steps.metadata.outputs.vault-builder-cache-key }}
153+
cache-to: type=gha,mode=min,scope=vault-builder-${{ steps.metadata.outputs.vault-builder-cache-key }}
154+
github-token: ${{ inputs.github-token }}
155+
- if: inputs.cgo-enabled == '1'
156+
name: ${{ steps.metadata.outputs.build-step-name }}
157+
shell: bash
158+
run: |
159+
mkdir -p dist
160+
mkdir -p out
161+
docker run \
162+
-v $(pwd):/build \
163+
-v $(go env GOMODCACHE):/go-mod-cache \
164+
--env GITHUB_TOKEN='${{ inputs.github-token }}' \
165+
--env CGO_ENABLED=1 \
166+
--env GO_TAGS='${{ inputs.go-tags }}' \
167+
--env GOARCH='${{ inputs.goarch }}' \
168+
--env GOARM='${{ inputs.goarm }}' \
169+
--env GOEXPERIMENT='${{ inputs.goexperiment }}' \
170+
--env GOMODCACHE=/go-mod-cache \
171+
--env GOOS='${{ inputs.goos }}' \
172+
--env VERSION='${{ inputs.version }}' \
173+
--env VERSION_METADATA='${{ inputs.vault-edition != 'ce' && inputs.vault-edition || '' }}' \
174+
localhost:5000/vault-builder:${{ steps.metadata.outputs.vault-builder-cache-key }} \
175+
make ci-build
148176
- if: inputs.vault-edition != 'ce'
149177
shell: bash
150178
run: make ci-prepare-ent-legal

‎.github/actions/metadata/action.yml

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -24,9 +24,6 @@ outputs:
2424
compute-build:
2525
description: A JSON encoded "runs-on" for App build worfkflows.
2626
value: ${{ steps.workflow-metadata.outputs.compute-build }}
27-
compute-build-compat:
28-
description: A JSON encoded "runs-on" for App build workflows that need an older glibc to link against.
29-
value: ${{ steps.workflow-metadata.outputs.compute-build-compat }}
3027
compute-build-ui:
3128
description: A JSON encoded "runs-on" for web UI build workflows.
3229
value: ${{ steps.workflow-metadata.outputs.compute-build-ui }}
@@ -153,7 +150,6 @@ runs:
153150
if [ "$is_enterprise" = 'true' ]; then
154151
{
155152
echo 'compute-build=["self-hosted","ondemand","os=linux","disk_gb=64","type=c6a.4xlarge"]'
156-
echo 'compute-build-compat=["self-hosted","ubuntu-20.04"]' # for older glibc compatibility, m6a.4xlarge
157153
echo 'compute-build-ui=["self-hosted","ondemand","os=linux", "disk_gb=64", "type=c6a.2xlarge"]'
158154
echo 'compute-test-go=["self-hosted","ondemand","os=linux","disk_gb=64","type=c6a.2xlarge"]'
159155
echo 'compute-test-ui=["self-hosted","ondemand","os=linux","type=m6a.2xlarge"]'
@@ -165,7 +161,6 @@ runs:
165161
else
166162
{
167163
echo 'compute-build="custom-linux-medium-vault-latest"'
168-
echo 'compute-build-compat="custom-linux-medium-vault-latest"'
169164
echo 'compute-build-ui="custom-linux-xl-vault-latest"'
170165
echo 'compute-test-go="custom-linux-medium-vault-latest"'
171166
echo 'compute-test-ui="custom-linux-medium-vault-latest"'

‎.github/workflows/build-artifacts-ce.yml

Lines changed: 0 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -23,10 +23,6 @@ on:
2323
type: string # JSON encoded to support passing arrays
2424
description: A JSON encoded "runs-on" for build worfkflows
2525
required: true
26-
compute-build-compat:
27-
type: string # JSON encoded to support passing arrays
28-
description: A JSON encoded "runs-on" for build workflows that need older glibc
29-
required: true
3026
compute-small:
3127
type: string # JSON encoded to support passing arrays
3228
description: A JSON encoded "runs-on" for non-resource-intensive workflows
@@ -62,10 +58,6 @@ on:
6258
type: string # JSON encoded to support passing arrays
6359
description: A JSON encoded "runs-on" for build worfkflows
6460
required: true
65-
compute-build-compat:
66-
type: string # JSON encoded to support passing arrays
67-
description: A JSON encoded "runs-on" for build workflows that need older glibc
68-
required: true
6961
compute-small:
7062
type: string # JSON encoded to support passing arrays
7163
description: A JSON encoded "runs-on" for non-resource-intensive workflows

‎.github/workflows/build.yml

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -89,7 +89,6 @@ jobs:
8989
changed-files: ${{ steps.changed-files.outputs.changed-files }}
9090
checkout-ref: ${{ steps.checkout.outputs.ref }}
9191
compute-build: ${{ steps.metadata.outputs.compute-build }}
92-
compute-build-compat: ${{ steps.metadata.outputs.compute-build-compat }}
9392
compute-build-ui: ${{ steps.metadata.outputs.compute-build-ui }}
9493
compute-small: ${{ steps.metadata.outputs.compute-small }}
9594
is-draft: ${{ steps.metadata.outputs.is-draft }}
@@ -237,7 +236,6 @@ jobs:
237236
build-date: ${{ needs.setup.outputs.build-date }}
238237
checkout-ref: ${{ needs.setup.outputs.checkout-ref }}
239238
compute-build: ${{ needs.setup.outputs.compute-build }}
240-
compute-build-compat: ${{ needs.setup.outputs.compute-build-compat }}
241239
compute-small: ${{ needs.setup.outputs.compute-small }}
242240
vault-revision: ${{ needs.setup.outputs.vault-revision }}
243241
vault-version: ${{ needs.setup.outputs.vault-version }}

‎Dockerfile

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -177,3 +177,49 @@ FROM ubi AS ubi-fips
177177
FROM ubi AS ubi-hsm
178178

179179
FROM ubi AS ubi-hsm-fips
180+
181+
## Builder:
182+
#
183+
# A build container used to build the Vault binary. We use focal because the
184+
# version of glibc is old enough for all of our supported distros for editions
185+
# that require CGO.
186+
#
187+
# You can build the builder container like so:
188+
# docker build -t builder --build-arg GO_VERSION=$(cat .go-version) .
189+
#
190+
# To can build Vault using the builder container like so:
191+
# docker run -it -v $(pwd):/build -v $(go env GOMODCACHE):/go-mod-cache --env GITHUB_TOKEN=$GITHUB_TOKEN --env GO_TAGS='ui enterprise cgo hsm venthsm' --env GOARCH=s390x --env GOOS=linux --env VERSION=1.20.0-beta1 --env VERSION_METADATA=ent.hsm --env GOMODCACHE=/go-mod-cache --env CGO_ENABLED=1 builder make ci-build
192+
#
193+
# Note that the container is automatically built in CI
194+
FROM ubuntu:focal AS builder
195+
196+
# Pass in the GO_VERSION as a build-arg
197+
ARG GO_VERSION
198+
199+
# Set our environment
200+
ENV PATH="/root/go/bin:/opt/go/bin:$PATH"
201+
ENV GOPRIVATE='github.com/hashicorp/*'
202+
203+
# Install the necessary system tooling to cross compile vault for our various
204+
# CGO targets. Do this separately from branch specific Go and build toolchains
205+
# so our various builder image layers can share cache.
206+
COPY .build/system.sh .
207+
RUN chmod +x system.sh
208+
RUN ./system.sh
209+
210+
# Install the correct Go toolchain
211+
COPY .build/go.sh .
212+
RUN chmod +x go.sh
213+
RUN ./go.sh
214+
215+
# Install the vault build tools. Clean up after ourselves so our layer is
216+
# minimal.
217+
COPY tools/tools.sh .
218+
RUN chmod +x tools.sh
219+
RUN ./tools.sh install-external && rm -rf "$(go env GOCACHE)" && rm -rf "$(go env GOMODCACHE)"
220+
221+
# Run the build
222+
COPY .build/entrypoint.sh .
223+
RUN chmod +x entrypoint.sh
224+
225+
ENTRYPOINT ["/entrypoint.sh"]

0 commit comments

Comments
 (0)