Skip to content

Commit 6c2fe3a

Browse files
huydhnhuydhn
authored
[BE] Add shareable calculate-docker-image GHA (#4397)
This refactors the logic in PyTorch https://github.com/pytorch/pytorch/blob/main/.github/actions/calculate-docker-image/action.yml to a shareable GHA. Notably, the changes include: * Accept a docker build directory input, i.e. `.ci/docker`, with a `build.sh` script entry. The usage is `build.sh "${IMAGE_NAME}" -t "${DOCKER_IMAGE}"` as how it's used in PyTorch atm * Accept a working directory input because Nova Linux job could checkout the repo in a different directory * Simplify the list of configurable options to only 2 `always-rebuild` and `push` (set to true to push the image to ECR) One hidden caveat is that `calculate-docker-image` requires a full checkout to work as it tries to figure out the Docker tag of the base commit. This means that `fetch-depth: 0` is needed to use this action. This action can then be used in the upcoming CI change on Executorch where the repo could build and use its own Docker image(s). ### Testing * Use the GHA on Nova Linux job #4399 * Use the GHA on PyTorch pytorch/pytorch#105372 * Executorch * Publish Docker image: pytorch/executorch#11 * Use Docker image in Linux job: pytorch/executorch#12
1 parent 1369cc7 commit 6c2fe3a

File tree

1 file changed

+164
-0
lines changed

1 file changed

+164
-0
lines changed

‎.github/actions/calculate-docker-image/action.yml

Lines changed: 164 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,164 @@
1+
name: Calculate docker image
2+
3+
description: Determine docker image to pull, building a new one if necessary.
4+
5+
inputs:
6+
docker-image-name:
7+
description: |
8+
The name of a docker image, like `pytorch-linux-focal-linter`. A fullname
9+
with ECR prefix is also supported.
10+
required: true
11+
docker-build-dir:
12+
description: |
13+
The directory containing the build.sh shell script to build the docker image.
14+
The script parameters can be passed to docker build similar to how it is used
15+
in PyTorch, i.e. build.sh "${IMAGE_NAME}" -t "${DOCKER_IMAGE}".
16+
default: .ci/docker
17+
working-directory:
18+
description: The working directory where the repo is checked out.
19+
default: .
20+
docker-registry:
21+
description: The registry to store the image after it is built.
22+
default: 308535385114.dkr.ecr.us-east-1.amazonaws.com
23+
always-rebuild:
24+
description: If set to any value, always build a fresh docker image.
25+
required: false
26+
push:
27+
description: If set to true, push the image to ECR.
28+
required: false
29+
30+
outputs:
31+
docker-image:
32+
description: The docker image to use for the rest of the workflow
33+
value: ${{ steps.calculate-image.outputs.docker-image }}
34+
35+
runs:
36+
using: composite
37+
steps:
38+
- name: Calculate docker image
39+
id: calculate-image
40+
shell: bash
41+
working-directory: ${{ inputs.working-directory }}
42+
env:
43+
REPO_NAME: ${{ github.event.repository.name }}
44+
DOCKER_IMAGE_NAME: ${{ inputs.docker-image-name }}
45+
DOCKER_BUILD_DIR: ${{ inputs.docker-build-dir }}
46+
DOCKER_REGISTRY: ${{ inputs.docker-registry }}
47+
run: |
48+
set -ex
49+
50+
# If the docker build directory or the build script doesn't exist, the action will
51+
# gracefully return the docker image name as it is. Pulling docker image in Linux
52+
# job could then download the pre-built image as usual
53+
if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then
54+
echo "skip=true" >> "${GITHUB_OUTPUT}"
55+
echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
56+
57+
echo "There is no Docker build script in ${REPO_NAME} repo, skipping..."
58+
exit 0
59+
else
60+
echo "skip=false" >> "${GITHUB_OUTPUT}"
61+
fi
62+
63+
if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then
64+
# The docker image name already includes the ECR prefix and tag, so we can just
65+
# use it as it is, but first let's extract the tag
66+
DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}')
67+
echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
68+
echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
69+
else
70+
DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}")
71+
echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
72+
echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
73+
fi
74+
75+
- name: Check if image should be built
76+
id: check-image
77+
shell: bash
78+
working-directory: ${{ inputs.working-directory }}
79+
if: ${{ steps.calculate-image.outputs.skip != 'true' && !inputs.always-rebuild }}
80+
env:
81+
DOCKER_BUILD_DIR: ${{ inputs.docker-build-dir }}
82+
BASE_REVISION: ${{ github.event.pull_request.base.sha || github.sha }}
83+
DOCKER_IMAGE: ${{ steps.calculate-image.outputs.docker-image }}
84+
DOCKER_TAG: ${{ steps.calculate-image.outputs.docker-tag }}
85+
run: |
86+
set +e
87+
set -x
88+
89+
# Check if image already exists, if it does then skip building it
90+
if docker manifest inspect "${DOCKER_IMAGE}"; then
91+
exit 0
92+
fi
93+
94+
# NB: This part requires a full checkout. Otherwise, the merge base will
95+
# be empty. The default action would be to continue rebuild the image
96+
if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then
97+
# if we're on the base branch then use the parent commit
98+
MERGE_BASE=$(git rev-parse HEAD~)
99+
else
100+
# otherwise we're on a PR, so use the most recent base commit
101+
MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION")
102+
fi
103+
104+
if [[ -z "${MERGE_BASE}" ]]; then
105+
echo "rebuild=true" >> "${GITHUB_OUTPUT}"
106+
107+
echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..."
108+
exit 0
109+
fi
110+
111+
if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then
112+
echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit"
113+
exit 1
114+
fi
115+
116+
PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}")
117+
# If no image exists but the hash is the same as the previous hash then we should error out here
118+
if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then
119+
echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch"
120+
echo " Will re-build docker image to store in local cache, TTS may be longer"
121+
fi
122+
123+
echo "rebuild=true" >> "${GITHUB_OUTPUT}"
124+
125+
- name: Build and push docker image
126+
shell: bash
127+
working-directory: ${{ inputs.working-directory }}/${{ inputs.docker-build-dir }}
128+
if: ${{ steps.calculate-image.outputs.skip != 'true' && (inputs.always-rebuild || steps.check-image.outputs.rebuild) }}
129+
env:
130+
REPO_NAME: ${{ github.event.repository.name }}
131+
DOCKER_PUSH: ${{ inputs.push }}
132+
DOCKER_IMAGE: ${{ steps.calculate-image.outputs.docker-image }}
133+
DOCKER_REGISTRY: ${{ inputs.docker-registry }}
134+
run: |
135+
set -ex
136+
137+
login() {
138+
aws ecr get-authorization-token --region us-east-1 --output text --query 'authorizationData[].authorizationToken' |
139+
base64 -d |
140+
cut -d: -f2 |
141+
docker login -u AWS --password-stdin "$1"
142+
}
143+
144+
retry () {
145+
$* || (sleep 1 && $*) || (sleep 2 && $*)
146+
}
147+
148+
# Only run these steps if not on github actions
149+
if [[ -z "${GITHUB_ACTIONS}" ]]; then
150+
retry login "${DOCKER_REGISTRY}"
151+
# Logout on exit
152+
trap "docker logout ${DOCKER_REGISTRY}" EXIT
153+
fi
154+
155+
IMAGE_NAME=$(echo ${DOCKER_IMAGE#"${DOCKER_REGISTRY}/${REPO_NAME}/"} | awk -F '[:,]' '{print $1}')
156+
# Build new image
157+
./build.sh "${IMAGE_NAME}" -t "${DOCKER_IMAGE}"
158+
159+
if [ "${DOCKER_PUSH:-false}" == "true" ]; then
160+
# Only push if docker image doesn't exist already
161+
if ! docker manifest inspect "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
162+
docker push "${DOCKER_IMAGE}"
163+
fi
164+
fi

0 commit comments

Comments
 (0)