Skip to content
CI

Merge pull request #150 from sysprog21/fix #470

Sign in to view logs

Merge pull request #150 from sysprog21/fix

Merge pull request #150 from sysprog21/fix #470

Workflow file for this run

.github/workflows/main.yml at cae49bb
name: CI
on:
push:
branches: ["main"]
pull_request:
branches: ["main"]
workflow_dispatch:
# Read by default. Only "publish-latest" widens this, and only for itself, so
# no job that executes pull-request-controlled code ever holds a write token.
permissions:
contents: read
env:
CARGO_TERM_COLOR: always
EXTENSION_PACKAGE_DIR: extension-package/zhtw-mcp-extension
EXTENSION_ZIP: zhtw-mcp-extension.zip
concurrency:
group: "ci-${{ github.ref }}"
# Off for main so a run is never killed mid-publish. This is not free:
# pushes to main now queue behind the previous run's whole pipeline. Only the
# in-progress run is protected; a superseded *pending* run is still cancelled,
# which is what keeps the queue to one.
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
jobs:
check:
strategy:
# One platform's failure must not hide another's. The run that made this
# necessary died on a macOS-only ruleset gate and cancelled the Linux and
# Windows legs mid-test, so nothing said whether either of those two also
# had something to report. "build-latest" below has carried this setting
# all along, for the same reason.
fail-fast: false
matrix:
os: [ubuntu-24.04, macos-15, windows-2025]
runs-on: ${{ matrix.os }}
timeout-minutes: 20
steps:
- uses: actions/checkout@v7
with:
# The ruleset provenance check reads the PR base revision. The
# "Resolve the ruleset baseline" step below leans on this covering an
# ordinary push too, and fetches by hand only what a force-push strands.
fetch-depth: 0
- name: Install Rust toolchain
run: rustup toolchain install stable --profile minimal --component rustfmt,clippy
- name: Rust cache
uses: Swatinem/rust-cache@v2
with:
prefix-key: "v1-rust"
shared-key: ${{ matrix.os }}
- name: Set up Python
uses: actions/setup-python@v7
with:
python-version: "3.x"
# "make check" runs black through scripts/indent.sh, and the runner image
# does not carry black. Only this job needs it; the build jobs run the
# table generator, which black never sees.
- name: Install black
run: pip install black
# The shell lanes of "make check". shfmt takes its style from
# .editorconfig, so installing it here is the whole configuration. Linux
# only, and that is deliberate: formatting does not vary by platform, and
# checking it on one runner is what the other two would repeat. Both lanes
# report a skip where the tool is absent, which is what the macOS and
# Windows legs then do.
#
# The runner image carries an apt source for Google Chrome, and
# dl.google.com has served a Packages.gz whose hash disagrees with the
# InRelease indexing it. apt-get update exits 100 when any source fails to
# fetch, so one third-party repository having a bad afternoon takes down a
# step that wants two packages out of the Ubuntu archives. See
# actions/runner-images#14708.
#
# Nothing in this workflow installs Chrome from apt, so drop the source
# rather than teach apt to tolerate one that fails: an "|| true" would also
# swallow the archive outages worth failing on, and package-extension below
# has no apt-get call of its own to pass options to. The glob is open at
# both ends because the file is a one-line ".list" on older images and a
# deb822 ".sources" on newer ones, and upstream removes only the former.
# It names chrome and not google, which would take the image's Google Cloud
# CLI source with it: dropping a working source nobody complained about is
# the same mistake as the "|| true" this is avoiding.
- name: Install the shell checkers
if: matrix.os == 'ubuntu-24.04'
run: |
sudo rm -fv /etc/apt/sources.list.d/*chrome*
sudo apt-get update
sudo apt-get install -y shellcheck shfmt
# commentflow settles comment width, which no other formatter here does:
# rustfmt leaves a short-wrapped comment short and shfmt does not touch
# comment text at all. scripts/indent.sh skips its reflow lane when the
# binary is missing, and ZHTW_REQUIRE_TOOLS below turns that skip into a
# failure, so this step is what the formatting gate stands on.
#
# Built from a pinned revision rather than downloaded from the release
# tagged "latest". That tag is rolling, so the asset behind it moves, and a
# digest read from the same release metadata proves the transfer arrived
# intact without saying anything about which build arrived. A commit is
# immutable and readable; bumping it is an edit to this line, which is
# exactly the review a formatter that rewrites every comment in the tree
# deserves. It also drops the "sudo" and the write into /usr/local/bin that
# the tarball needed: cargo installs into ~/.cargo/bin, which is on PATH.
#
# The cost is about a minute of compile on a cold cache, and rust-cache
# above carries ~/.cargo across runs, so most runs pay nothing.
- name: Install commentflow
if: matrix.os == 'ubuntu-24.04'
env:
COMMENTFLOW_REV: 329ad4178e65e700528ef111cb11a075addb4eba
run: |
set -euo pipefail
# cargo records the source revision it installed from, so a warm
# ~/.cargo from the cache above makes this a no-op rather than a rebuild.
cargo install --git https://github.com/sysprog21/commentflow \
--rev "$COMMENTFLOW_REV" --locked commentflow
commentflow --help > /dev/null
# Every platform, because both check steps below gate on provenance and a
# gate that means one thing on Linux and another on Windows is not a gate.
# Each leg is its own runner with its own checkout, so each resolves the
# baseline for itself. Writing it to GITHUB_ENV rather than repeating the
# expression keeps the policy in one place: a baseline source added here
# reaches both consumers, or neither.
- name: Resolve the ruleset baseline
shell: bash
env:
# The provenance gate compares against the revision before the change.
# workflow_dispatch has neither a PR base nor a push's before SHA and
# leaves this empty, which the Makefile and the Windows leg below both
# read as HEAD.
RULESET_BASELINE: ${{ github.event.pull_request.base.sha || github.event.before }}
run: |
# A push that creates the branch reports all zeros for the revision
# before it. That names no commit for the remote to serve, and it
# means what the empty value means, so default it rather than asking
# upload-pack for a ref nobody has.
if [ "$RULESET_BASELINE" = 0000000000000000000000000000000000000000 ]; then
RULESET_BASELINE=
fi
# The checkout above takes every ref, which covers a pull request base
# and an ordinary push alike. A force-push is the gap it does not
# cover: the before SHA is left on no ref at all, so ask the remote for
# that one object rather than substitute an ancestor of HEAD, which
# would narrow a multi-commit push and let its earlier rules evade
# provenance. Asking git first keeps every other run to a local lookup.
if [ -n "$RULESET_BASELINE" ] &&
! git cat-file -e "$RULESET_BASELINE^{commit}" 2> /dev/null; then
git fetch --no-tags origin "$RULESET_BASELINE"
fi
echo "RULESET_BASELINE=$RULESET_BASELINE" >> "$GITHUB_ENV"
- name: make check
if: matrix.os != 'windows-2025'
env:
ZHTW_REQUIRE_TOOLS: ${{ matrix.os == 'ubuntu-24.04' && '1' || '' }}
run: make check
# "make" is not dependable on the Windows runner (see build-latest's
# "Build and package" step below), so run the same checks "make check"
# would directly instead. Two of them are missing here on purpose:
# scripts/indent.sh and the hook suite drive tar, mktemp, git worktrees and
# four hook wrappers through whatever sh the runner happens to provide, and
# neither a formatting rule nor a commit rule varies by platform. The Linux
# leg is where both run; the direct "cargo fmt --check" and "black --check"
# below are what this leg keeps of the formatter chain. The ruleset lint
# is not one of the two: it takes the baseline the step above resolved, so
# provenance means the same thing here as it does on the other legs.
#
# The rest of the list is what caught #124 and #126 turning
# into recurring Windows-only regressions: those bugs only ever
# surfaced on a contributor's own Windows machine because nothing in CI
# ran the test suite there.
- name: make check (Windows)
if: matrix.os == 'windows-2025'
shell: bash
run: |
python scripts/gen-s2t-tables.py
rustfmt src/engine/s2t_data.rs
cargo metadata --locked --format-version 1 >/dev/null
cargo test
./scripts/clippy-lanes.sh
cargo fmt --check
black --check .
python scripts/check-ruleset.py --lint \
--baseline-ref "${RULESET_BASELINE:-HEAD}"
# The commit-msg hook binds whoever installed it. This binds everyone else: a
# rebase, an amend, "--no-verify", or a subject typed into the GitHub merge box
# all reach the branch unread otherwise. It runs the same script the pre-push
# hook runs, so a contributor with hooks and a contributor without are judged
# by one list.
#
# Pull requests only, and deliberately. "base.sha..HEAD" is the work being
# proposed, which is the thing these rules are about. The push event's range
# is not: a merge to main replays every commit of the merged branch, so a rule
# this log only started following recently would reject history rather than the
# change in front of it.
#
# Its own job rather than a step in "check": commit messages do not vary by
# platform, so folding it into that matrix would run one second of shell three
# times, and this is skipped outside a pull request while "check" is not.
commit-log:
if: github.event_name == 'pull_request'
runs-on: ubuntu-24.04
timeout-minutes: 5
steps:
- uses: actions/checkout@v7
with:
fetch-depth: 0
- name: Check the commit messages
env:
BASE_SHA: ${{ github.event.pull_request.base.sha }}
# The pull request's own tip, not the checkout's HEAD. For a
# pull_request event the checkout is a merge ref that GitHub built to
# test the change, and its commit is authored by nobody and subjected
# to nothing: "Merge <sha> into <sha>" is 92 columns and belongs to no
# author. Ending the range at the head keeps this judging the commits
# the change proposes.
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
run: |
set -euo pipefail
# The width rules are measured by a python3 helper, and the hook treats
# a missing python3 as a lane to skip. That is right on a laptop and
# wrong here: this job is the only thing binding a contributor without
# hooks, so a runner image without python3 has to fail rather than
# quietly stop counting columns.
python3 --version
# An unfetched base is not an empty range. Say which one this is rather
# than reporting "nothing to check" for a check that could not run.
for sha in "$BASE_SHA" "$HEAD_SHA"; do
if ! git cat-file -e "${sha}^{commit}" 2>/dev/null; then
echo "commit $sha is not in this checkout" >&2
exit 1
fi
done
# The checker comes from the base revision rather than the checkout. A
# pull request that edits scripts/check-commit-log.sh or the rules it
# calls would otherwise be graded by its own copy of them, which is not
# a gate. Both scripts land in one directory because the checker finds
# the message hook beside itself.
#
# Falling back to the checkout is not a hole: it only happens when the
# base has no checker at all, and a policy that does not exist yet
# cannot be weakened by the change introducing it.
trusted=$(mktemp -d)
for script in check-commit-log.sh git-commit-msg.sh; do
if git cat-file -e "$BASE_SHA:scripts/$script" 2>/dev/null; then
git show "$BASE_SHA:scripts/$script" > "$trusted/$script"
else
echo "::notice::$script is new in this change; using its own copy"
cp "scripts/$script" "$trusted/$script"
fi
chmod +x "$trusted/$script"
done
# Merges included, matching the pre-push hook: their subjects are exempt
# from the style rules and not from the character ones.
git rev-list "$BASE_SHA..$HEAD_SHA" | "$trusted/check-commit-log.sh"
# Reads Cargo.lock against the RustSec advisory DB. Deliberately has no
# Swatinem/rust-cache step: this job never compiles the workspace, and sharing
# the check job's cache key would let it save an empty target/ tree over the
# real cache.
audit:
runs-on: ubuntu-24.04
timeout-minutes: 10
steps:
- uses: actions/checkout@v7
- name: Install cargo-audit
uses: taiki-e/install-action@cargo-audit
- name: cargo audit
run: cargo audit
# No "needs: check". This measures a binary, it does not publish one, and
# "publish-latest" names "check" itself, so waiting for the slowest leg of a
# three-platform matrix bought no safety and cost a pull request about two
# minutes before it heard its size verdict.
check-size:
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
- uses: actions/checkout@v7
# rustfmt because "make check-size" goes through the generator rule, which
# runs it over the tables it just wrote. Asking for the component is what
# keeps that from borrowing whichever one the runner image happens to ship.
- name: Install Rust toolchain
run: rustup toolchain install stable --profile minimal --component rustfmt
# Its own key: rust-cache never overwrites an existing entry, so a key
# shared with the check job on this runner would keep whichever of the two
# trees finished first, and the release tree this job builds is not the one
# that wins that race.
- name: Rust cache
uses: Swatinem/rust-cache@v2
with:
prefix-key: "v1-rust"
shared-key: "ubuntu-24.04-size"
- name: Set up Python
uses: actions/setup-python@v7
with:
python-version: "3.x"
- name: make check-size
run: make check-size
# Builds the WASM scanner, runs the extension's own tests, and packages the
# unpacked and zipped extension. It lives here rather than in a workflow of
# its own because the zip ships on the same "latest" release as the binaries,
# and one release wants one owner: "publish-latest" below checksums it,
# uploads it, and verifies it alongside everything else.
package-extension:
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
- uses: actions/checkout@v7
- name: Install Rust toolchain
run: |
rustup toolchain install stable --profile minimal --component rustfmt
rustup target add wasm32-unknown-unknown
- name: Rust cache
uses: Swatinem/rust-cache@v2
with:
prefix-key: "v1-extension-rust"
shared-key: "ubuntu-24.04"
- name: Set up Node.js
uses: actions/setup-node@v7
with:
node-version: "22"
- name: Install extension test dependencies
run: npm ci --prefix extension
# Drops the Google Chrome apt source for the reason spelled out in the
# check job's "Install the shell checkers" step above. This step has no
# apt-get of its own to guard: "--with-deps" runs apt-get update as root
# from inside playwright and inherits its exit 100. The browser these
# tests drive is playwright's own Chromium download rather than anything
# out of that repository, so removing the source costs the job nothing.
- name: Install headless Chromium
run: |
sudo rm -fv /etc/apt/sources.list.d/*chrome*
npx --prefix extension playwright install --with-deps chromium
# Prebuilt, like cargo-audit above. "cargo install wasm-pack" builds it
# from source, which only stayed cheap because rust-cache happened to carry
# ~/.cargo/bin across runs.
- name: Install wasm-pack
uses: taiki-e/install-action@wasm-pack
- name: Build scanner WASM
run: sh extension/build-wasm.sh
- name: Test extension
run: npm run test:all --prefix extension
- name: Assemble distributable extension
run: |
rm -rf extension-package "$EXTENSION_ZIP"
mkdir -p "$EXTENSION_PACKAGE_DIR"/{dist,icons}
cp -R \
extension/src \
extension/styles \
extension/manifest.json \
extension/popup.html \
"$EXTENSION_PACKAGE_DIR"/
cp extension/dist/*.js extension/dist/*.wasm "$EXTENSION_PACKAGE_DIR"/dist/
if [ -d extension/dist/snippets ]; then
cp -R extension/dist/snippets "$EXTENSION_PACKAGE_DIR"/dist/
fi
cp extension/icons/icon-{16,32,48,128}.png "$EXTENSION_PACKAGE_DIR"/icons/
(cd extension-package && zip -r "../$EXTENSION_ZIP" zhtw-mcp-extension)
# Deliberately NOT named "zhtw-mcp-*": publish-latest downloads that
# pattern into ./dist, and an unpacked tree there would be checksummed and
# uploaded file by file.
- name: Upload unpacked extension artifact
uses: actions/upload-artifact@v7
with:
name: extension-unpacked
path: ${{ env.EXTENSION_PACKAGE_DIR }}
if-no-files-found: error
- name: Upload zipped extension artifact
uses: actions/upload-artifact@v7
with:
name: zhtw-mcp-extension
path: ${{ env.EXTENSION_ZIP }}
if-no-files-found: error
# Rolling binaries for the "latest" tag. This is the project's release: it
# is not a prerelease, so it carries the "Latest" badge and /releases/latest
# resolves to it. Nothing here needs a version tag to be pushed.
#
# It builds alongside "check" rather than behind it; the release gate is on
# "publish-latest". Queued behind the matrix these four targets waited on its
# slowest leg, which on a push to main was most of the pipeline's wall clock.
# What it costs is four builds thrown away on the pushes where main does fail,
# which is the rarer case.
build-latest:
name: build (${{ matrix.name }})
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
strategy:
fail-fast: false
matrix:
include:
- name: Linux x86_64
os: ubuntu-24.04
target: x86_64-unknown-linux-gnu
- name: Linux arm64
os: ubuntu-24.04-arm
target: aarch64-unknown-linux-gnu
- name: macOS arm64
os: macos-15
target: aarch64-apple-darwin
- name: Windows x86_64
os: windows-2025
target: x86_64-pc-windows-msvc
runs-on: ${{ matrix.os }}
timeout-minutes: 20
steps:
- uses: actions/checkout@v7
- name: Install Rust toolchain
run: rustup toolchain install stable --profile minimal --target ${{ matrix.target }}
# Keyed on the target, not the runner. rust-cache never overwrites an
# existing entry, so a key shared with the check jobs on the same runner
# would save whichever tree finished first and then never refresh, and the
# release tree is not the one that wins that race.
- name: Rust cache
uses: Swatinem/rust-cache@v2
with:
prefix-key: "v1-rust"
shared-key: "${{ matrix.target }}-release"
- name: Set up Python
uses: actions/setup-python@v7
with:
python-version: "3.x"
# Runs the generator directly rather than through make, which is not
# dependable on the Windows runner. Unlike the Makefile's $(S2T_STAMP)
# recipe this skips rustfmt and the stamp: nothing here fmt-checks, and the
# minimal toolchain has no rustfmt. "python" not "python3" is for Windows;
# setup-python provides both elsewhere.
- name: Build and package
shell: bash
run: |
bin=zhtw-mcp${{ runner.os == 'Windows' && '.exe' || '' }}
python scripts/gen-s2t-tables.py
cargo build --locked --release --target ${{ matrix.target }}
mkdir pkg
cp "target/${{ matrix.target }}/release/$bin" LICENSE README.md pkg/
# Prove the binary we are shipping actually starts. Every target here
# builds on its own architecture, so every archive gets run once; the
# check matrix is ubuntu + macos, which makes this the only place the
# Windows and arm64 Linux builds execute at all. 0 is a clean verdict
# and 1 is a prose gate; both mean it worked.
printf '\xe6\xb8\xac\xe8\xa9\xa6\xe6\xaa\x94\xe6\xa1\x88\n' > smoke.txt
rc=0; "pkg/$bin" lint smoke.txt || rc=$?
[ "$rc" -le 1 ] || { echo "smoke test failed with exit $rc" >&2; exit 1; }
tar czf "zhtw-mcp-${{ matrix.target }}.tar.gz" -C pkg "$bin" LICENSE README.md
- uses: actions/upload-artifact@v7
with:
name: zhtw-mcp-${{ matrix.target }}
path: zhtw-mcp-${{ matrix.target }}.tar.gz
if-no-files-found: error
retention-days: 1
# "check" by name rather than by way of another job: "check-size" and
# "build-latest" run beside it now, so this list is the whole of what stands
# between a failing test suite and a published binary.
publish-latest:
needs: [check, audit, check-size, build-latest, package-extension]
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
runs-on: ubuntu-24.04
# Without this the job inherits the 360 minute default, and since main runs
# do not cancel each other a wedged gh call would hold the concurrency group
# (and every later push to main) for six hours.
timeout-minutes: 10
permissions:
contents: write
env:
GH_TOKEN: ${{ github.token }}
GH_REPO: ${{ github.repository }}
steps:
- uses: actions/download-artifact@v8
with:
pattern: zhtw-mcp-*
path: dist
merge-multiple: true
# This never *moves* the "latest" git tag. A download URL keys on the
# release's tag name, not on the commit the ref points at, so force-moving
# the ref on every build bought nothing. "gh release create" does create
# the tag when it is missing, and --target applies only in that case; if the
# tag already exists the release adopts wherever it points, which is why the
# commit a build came from is recorded in the notes instead.
#
# Nothing here deletes the release either. The README hands out these URLs,
# and a delete-then-create cycle 404s them for as long as the upload takes:
# the first run of this workflow left a five and a half minute hole.
- name: Publish the latest release
shell: bash
run: |
cd dist
sha256sum -- zhtw-mcp-* > SHA256SUMS
# The queue already runs pushes in order, so what this actually catches
# is a manual re-run of an older run, plus main moving while this run is
# in flight. Best effort: main can advance again right after the check.
tip=$(gh api "repos/$GH_REPO/commits/main" --jq .sha)
if [ "$tip" != "$GITHUB_SHA" ]; then
echo "::warning::latest not refreshed: main moved to ${tip:0:7}, this run built ${GITHUB_SHA:0:7}"
exit 0
fi
# Notes name a commit, so they go up after the archives built from that
# commit. --clobber deletes every matching asset and re-uploads them
# concurrently, and gh warns that a failed upload loses the originals,
# so the whole set is briefly absent and can stay that way. Hence the
# retry, and the verification below. --draft=false also recovers a
# release left as a draft by a "create" that died mid-upload.
notes="Automated build of $GITHUB_SHA on $(date -u +%Y-%m-%d)."
upload_rc=0
if gh release view latest >/dev/null 2>&1; then
gh release upload latest ./* --clobber ||
gh release upload latest ./* --clobber || upload_rc=$?
# Notes name a commit, so they go up only once the archives from that
# commit are in place. Letting a failed upload through to here would
# leave the notes advertising binaries that are not on the release.
# --prerelease=false is spelled out rather than left off: the flag is
# sticky, and the release was published as a prerelease originally.
if [ "$upload_rc" -eq 0 ]; then
gh release edit latest --notes "$notes" \
--prerelease=false --latest --draft=false
fi
else
gh release create latest ./* --target "$GITHUB_SHA" \
--title latest --notes "$notes" --latest
fi
# Trust the published set, not the exit status. Scratch files go to
# RUNNER_TEMP so they cannot land in the glob being compared. Only a
# missing asset is fatal: an extra one is someone else's upload or a
# leftover from a renamed target, and failing on that would wedge every
# later run.
printf '%s\n' * | sort > "$RUNNER_TEMP/want"
gh release view latest --json assets --jq '.assets[].name' \
| sort > "$RUNNER_TEMP/got"
missing=$(comm -23 "$RUNNER_TEMP/want" "$RUNNER_TEMP/got")
extra=$(comm -13 "$RUNNER_TEMP/want" "$RUNNER_TEMP/got")
[ -z "$extra" ] || echo "::warning::latest carries assets this build did not produce: $(printf %s "$extra" | tr '\n' ' ')"
[ -z "$missing" ] || {
echo "::error::latest is missing assets after publish: $(printf %s "$missing" | tr '\n' ' ')"
exit 1
}
[ "$upload_rc" -eq 0 ] ||
{ echo "::error::asset upload failed even after a retry"; exit 1; }