git-artifacts (CI): add support for CLANGARM64 #4185
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -14,6 +14,9 @@ on: | |
| repository: | ||
| description: 'Optionally override from where to fetch the specified ref' | ||
| required: false | ||
| build_arm64: | ||
| description: 'Optionally build ARM64 artifacts (requires a self-hosted ARM64 runner to be active in this repo)' | ||
| required: false | ||
|
|
||
| env: | ||
| GPG_OPTIONS: "--batch --yes --no-tty --list-options no-show-photos --verify-options no-show-photos --pinentry-mode loopback" | ||
|
|
@@ -22,6 +25,7 @@ env: | |
| USERPROFILE: "${{github.workspace}}\\home" | ||
| BUILD_ONLY: "${{github.event.inputs.build_only}}" | ||
| REPOSITORY: "${{github.event.inputs.repository}}" | ||
| BUILD_ARM64: "${{github.event.inputs.build_arm64}}" | ||
| REF: "${{github.event.inputs.ref}}" | ||
|
|
||
| jobs: | ||
|
|
@@ -96,26 +100,34 @@ jobs: | |
| name: bundle-artifacts | ||
| path: bundle-artifacts | ||
| pkg: | ||
| # Hack to ensure that the "determine skip" step works if no self-hosted ARM64 runner is available | ||
| runs-on: ${{ ((matrix.arch.name == 'aarch64' && github.event.inputs.build_arm64 == 'true') && fromJSON('["Windows", "ARM64"]')) || 'windows-latest' }} | ||
| needs: bundle-artifacts | ||
| strategy: | ||
| matrix: | ||
| arch: | ||
| - name: x86_64 | ||
| pacman_arch: x86_64 | ||
| bin: /amd64 | ||
| - name: i686 | ||
| pacman_arch: i686 | ||
| bin: '' | ||
| - name: aarch64 | ||
| pacman_arch: clang-aarch64 | ||
| bin: '' | ||
| steps: | ||
| - name: Determine whether this job should be skipped | ||
| shell: bash | ||
| run: | | ||
| if test "${{matrix.arch.name}}" = "aarch64" && test "$BUILD_ARM64" != "true" | ||
| then | ||
| echo "SKIP=true" >>$GITHUB_ENV | ||
| exit 0 | ||
| fi | ||
| for e in ${BUILD_ONLY:-pkg} | ||
| do | ||
| case $e in | ||
| *-${{matrix.arch.name}}) exit 0;; # build this artifact | ||
| *-*) ;; # not this build artifact, keep looking | ||
| *) exit 0;; # build this artifact | ||
| esac | ||
|
|
@@ -135,6 +147,9 @@ jobs: | |
| if: env.SKIP != 'true' | ||
| with: | ||
| flavor: build-installers | ||
| architecture: ${{ (matrix.arch.name == 'aarch64' && 'aarch64') || 'x86_64' }} | ||
| # We only have to clean up on self-hosted runners | ||
| cleanup: ${{ (runner.arch == 'ARM64' && true) || false }} | ||
| - name: Download bundle-artifacts | ||
| if: env.SKIP != 'true' | ||
| uses: actions/download-artifact@v1 | ||
|
|
@@ -184,7 +199,29 @@ jobs: | |
| git config --global user.email "<${info#*<}" | ||
| env: | ||
| GPGKEY: ${{secrets.GPGKEY}} | ||
| # Until there is a Git SDK for arm64, we'll need to install a few packages manually | ||
| # We install prebuilt binaries to save lots of CI time | ||
| - name: Install aarch64 deps | ||
| if: env.SKIP != 'true' && matrix.arch.name == 'aarch64' | ||
| shell: bash | ||
| run: | | ||
| set -x | ||
|
|
||
| package_dir="tmp-aarch64-deps" | ||
| release_url="https://github.com/dennisameling/git/releases/download/v2.39.0.windows.99" | ||
| packages="mingw-w64-clang-aarch64-openssl-1.1.1.s-1-any.pkg.tar.zst mingw-w64-clang-aarch64-curl-7.86.0-1-any.pkg.tar.zst" | ||
|
|
||
| mkdir -p $package_dir && cd $package_dir | ||
|
|
||
| for package in $packages | ||
| do | ||
| curl -LOf $release_url/$package || exit 1 | ||
| done | ||
|
|
||
| pacman -U --noconfirm $packages | ||
|
|
||
| cd ../ | ||
|
Comment on lines
+202
to
+223
There was a problem hiding this comment. Related discussion for an ARM64 SDK and/or Pacman repo: #4186 |
||
| - name: Build mingw-w64-${{matrix.arch.pacman_arch}}-git | ||
| if: env.SKIP != 'true' | ||
| env: | ||
| GPGKEY: "${{secrets.GPGKEY}}" | ||
|
|
@@ -195,11 +232,21 @@ jobs: | |
| # Make sure that there is a `/usr/bin/git` that can be used by `makepkg-mingw` | ||
| printf '#!/bin/sh\n\nexec /mingw64/bin/git.exe "$@"\n' >/usr/bin/git && | ||
|
|
||
| # We don't use `cv2pdb` on aarch64, which we build using Clang instead of GCC | ||
| if test "${{matrix.arch.name}}" != "aarch64" | ||
| then | ||
| # Restrict `PATH` to MSYS2 and to Visual Studio (to let `cv2pdb` find the relevant DLLs) | ||
| PATH="/mingw64/bin:/usr/bin:/c/Program Files/Microsoft Visual Studio/2022/Enterprise/Common7/IDE/:/C/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin${{matrix.arch.bin}}:/C/Windows/system32" | ||
| type -p mspdb140.dll || exit 1 | ||
| else | ||
| # We don't want to build dashed built-ins anymore. Let's do this on aarch64 only to begin with | ||
| export SKIP_DASHED_BUILT_INS=YesPlease | ||
|
Comment on lines
+242
to
+243
There was a problem hiding this comment. @dscho I thought this'd be a good opportunity to activate There was a problem hiding this comment. I'd like to enable it in git-for-windows/git, regardless of the target architecture. But because it is potentially disruptive a change, I would like to wait for a few days because I think we need a new Git for Windows release because of a cURL problem that will supposedly be fixed this Wednesday and because of a Git Credential Manager problem that was also fixed in the meantime. There have been some rumblings about an upstream maintenance release, too, so I definitely need to keep Git for Windows very stable for at least a few more days. After that, I am more than happy to see |
||
| fi && | ||
|
|
||
| # Temp until https://github.com/git-for-windows/build-extra/pull/452 is merged | ||
| /mingw64/bin/curl.exe https://raw.githubusercontent.com/git-for-windows/build-extra/3d6fc6dfe74902644c042500ad80e17abd134bfc/please.sh -o /usr/src/build-extra/please.sh && | ||
|
|
||
| sh -x /usr/src/build-extra/please.sh build-mingw-w64-git --only-${{matrix.arch.name}} --build-src-pkg -o artifacts HEAD && | ||
| cp bundle-artifacts/ver artifacts/ && | ||
| if test -n "$GPGKEY" | ||
| then | ||
|
|
@@ -219,94 +266,16 @@ jobs: | |
| if: always() && env.SKIP != 'true' | ||
| shell: bash | ||
| run: rm -rf home | ||
| - name: Publish mingw-w64-${{matrix.arch.pacman_arch}}-git | ||
| if: env.SKIP != 'true' | ||
| uses: actions/upload-artifact@v1 | ||
| with: | ||
| name: pkg-${{matrix.arch.name}} | ||
| path: artifacts | ||
| artifacts: | ||
| # Hack to ensure that the "determine skip" step works if no self-hosted ARM64 runner is available | ||
| runs-on: ${{ ((matrix.arch.name == 'aarch64' && github.event.inputs.build_arm64 == 'true') && fromJSON('["Windows", "ARM64"]')) || 'windows-latest' }} | ||
| needs: pkg | ||
| strategy: | ||
| matrix: | ||
| artifact: | ||
|
|
@@ -327,29 +296,32 @@ jobs: | |
| fileextension: zip | ||
| arch: | ||
| - name: x86_64 | ||
| pacman_arch: x86_64 | ||
| bitness: 64 | ||
| msystem: MINGW64 | ||
| - name: i686 | ||
| pacman_arch: i686 | ||
| bitness: 32 | ||
| msystem: MINGW32 | ||
| - name: aarch64 | ||
| pacman_arch: clang-aarch64 | ||
| bitness: 64 | ||
| msystem: CLANGARM64 | ||
| fail-fast: false | ||
| env: | ||
| MSYSTEM: ${{matrix.arch.msystem}} | ||
| steps: | ||
| - name: Determine whether this job should be skipped | ||
| shell: bash | ||
| run: | | ||
| if test "${{matrix.arch.name}}" = "aarch64" && test "$BUILD_ARM64" != "true" | ||
| then | ||
| echo "SKIP=true" >>$GITHUB_ENV | ||
| exit 0 | ||
| fi | ||
| case " $BUILD_ONLY " in | ||
| ' ') ;; # not set; build all | ||
| *" ${{matrix.artifact.name}} "*|*" ${{matrix.artifact.name}}-${{matrix.arch.name}}"*) ;; # build this artifact | ||
| *) echo "SKIP=true" >>$GITHUB_ENV;; | ||
| esac | ||
| - name: Download pkg-${{matrix.arch.name}} | ||
|
|
@@ -365,27 +337,43 @@ jobs: | |
| name: bundle-artifacts | ||
| path: bundle-artifacts | ||
| - uses: git-for-windows/setup-git-for-windows-sdk@v1 | ||
| if: env.SKIP != 'true' | ||
| with: | ||
| flavor: build-installers | ||
| architecture: ${{matrix.arch.name}} | ||
| # We only have to clean up on self-hosted runners | ||
| cleanup: ${{ (runner.arch == 'ARM64' && true) || false }} | ||
| # Until there is a Git SDK for arm64, we'll need to install a few packages manually | ||
| # We install prebuilt binaries to save lots of CI time | ||
| - name: Install aarch64 deps | ||
| if: env.SKIP != 'true' && matrix.arch.name == 'aarch64' | ||
| shell: bash | ||
| run: | | ||
| set -x | ||
|
|
||
| package_dir="tmp-aarch64-deps" | ||
| release_url="https://github.com/dennisameling/git/releases/download/v2.39.0.windows.99" | ||
| packages="mingw-w64-clang-aarch64-openssl-1.1.1.s-1-any.pkg.tar.zst | ||
| mingw-w64-clang-aarch64-curl-7.86.0-1-any.pkg.tar.zst | ||
| mingw-w64-clang-aarch64-wintoast-1.0.0.181.9b0663d-1-any.pkg.tar.zst | ||
| mingw-w64-clang-aarch64-xpdf-tools-4.00-1-any.pkg.tar.zst | ||
| mingw-w64-clang-aarch64-git-credential-manager-2.0.886-1-any.pkg.tar.zst | ||
| mingw-w64-clang-aarch64-git-lfs-3.3.0-1-any.pkg.tar.zst | ||
| git-extra-1.1.616.ced335bb5-1-any.pkg.tar.zst" | ||
|
|
||
| mkdir -p $package_dir && cd $package_dir | ||
|
|
||
| for package in $packages | ||
| do | ||
| curl -LOf $release_url/$package || exit 1 | ||
| done | ||
|
|
||
| pacman -U --noconfirm $packages | ||
|
|
||
| # Some additional packages we need for most artifacts | ||
| pacman -S --noconfirm mingw-w64-clang-aarch64-connect mingw-w64-clang-aarch64-antiword mingw-w64-clang-aarch64-odt2txt | ||
|
|
||
| cd ../ | ||
| - name: Clone and update build-extra | ||
| if: env.SKIP != 'true' | ||
| shell: bash | ||
|
|
@@ -410,26 +398,23 @@ jobs: | |
| echo -n "$CODESIGN_P12" | tr % '\n' | base64 -d >home/.sig/codesign.p12 && | ||
| echo -n "$CODESIGN_PASS" >home/.sig/codesign.pass && | ||
| git config --global alias.signtool '!sh "/usr/src/build-extra/signtool.sh"' | ||
| - name: Build ${{matrix.arch.name}} ${{matrix.artifact.name}} | ||
| if: env.SKIP != 'true' | ||
| shell: bash | ||
| run: | | ||
| set -x | ||
|
|
||
| # Temp until https://github.com/git-for-windows/build-extra/pull/452 is merged | ||
| /mingw${{matrix.arch.bitness}}/bin/curl.exe https://raw.githubusercontent.com/git-for-windows/build-extra/3d6fc6dfe74902644c042500ad80e17abd134bfc/please.sh -o /usr/src/build-extra/please.sh && | ||
|
|
||
| eval /usr/src/build-extra/please.sh make_installers_from_mingw_w64_git --version=$(cat pkg-${{matrix.arch.name}}/ver) -o artifacts --${{matrix.artifact.name}} --pkg=pkg-${{matrix.arch.name}}/mingw-w64-${{matrix.arch.pacman_arch}}-git-[0-9]*.tar.xz --pkg=pkg-${{matrix.arch.name}}/mingw-w64-${{matrix.arch.pacman_arch}}-git-doc-html-[0-9]*.tar.xz && | ||
| if test portable = '${{matrix.artifact.name}}' && test -n "$(git config alias.signtool)" | ||
| then | ||
| git signtool artifacts/PortableGit-*.exe | ||
| fi && | ||
| openssl dgst -sha256 artifacts/${{matrix.artifact.fileprefix}}-*.${{matrix.artifact.fileextension}} | sed "s/.* //" >artifacts/sha-256.txt | ||
| - name: Copy package-versions and pdbs | ||
| if: env.SKIP != 'true' && matrix.artifact.name == 'installer' && matrix.arch.name != 'aarch64' | ||
| shell: bash | ||
| run: | | ||
| cp /usr/src/build-extra/installer/package-versions.txt artifacts/ && | ||
|
|
@@ -445,17 +430,11 @@ jobs: | |
| shell: bash | ||
| run: rm -rf home | ||
| - name: Publish ${{matrix.artifact.name}}-${{matrix.arch.name}} | ||
| if: env.SKIP != 'true' | ||
| uses: actions/upload-artifact@v1 | ||
| with: | ||
| name: ${{matrix.artifact.name}}-${{matrix.arch.name}} | ||
| path: artifacts | ||
| nuget: | ||
| runs-on: windows-latest | ||
| needs: pkg | ||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The
fromJSON()part has been a pain to get right (and I'm not the only one), but it works now.In any case, I think it'll be more scalable to set the entire
matrixdynamically (example here). That also allows us to get rid of all thoseif: env.SKIP != 'true'conditions in the pipeline. But let's leave that for a follow-up PR, as I spent a lot of time getting all of this to work 🙏🏼There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We also have a splendid example here: https://github.com/git-for-windows/build-extra/blob/f6041a722c0cd6a36d139537f1253aaae74800e9/.github/workflows/main.yml#L19-L24 (and here: https://github.com/git-for-windows/build-extra/blob/f6041a722c0cd6a36d139537f1253aaae74800e9/.github/workflows/main.yml#L47).