Skip to content

Commit ec208dc

Browse files
ilitteriilitteri
authored
ci(l1,l2): enrich daily snapsync Slack notifications and link run logs (#6939)
**Motivation** The daily snapsync notifications carry only a bare commit SHA. When a sync fails there is no link to the workflow run, so getting to the logs means manually finding the run in the Actions tab. This mirrors the gaps recently fixed for the workflow-failure alerts in #6850. **Description** - Link the workflow run from the headline (`https://github.com/.../actions/runs/<id>`), so the success/failure message is one click from the logs. - Show the commit title next to the SHA. The subject is read from the checkout the job already has (`git log`), which works for `schedule`/`workflow_dispatch` runs where the event payload has no `head_commit`, and avoids interpolating any untrusted input into the workflow. - Distinguish a cancelled run (`:warning:` "was cancelled") from a real sync failure (`:rotating_light:` "failed"), so a manual stop or runner death does not read as a broken sync. Cancellations still go to the failure channel for visibility. - Route by outcome to the existing channels unchanged (success → `ETHREX_NOTIFICATIONS_SLACK_WEBHOOK`, failure/cancelled → `ETHREX_INTERNO_SLACK_WEBHOOK`), and only require the webhook actually being used. - Hygiene: Slack delivery failures now fail the notify step (with `--retry 3`) instead of being swallowed by `|| echo`; Slack mrkdwn special characters in the commit title are escaped; a re-run shows its attempt number and links the attempt. **Examples** Before — a failed sync: ``` 🚨 Node failed to snapsync Job Commit Duration prysm-sepolia abc12345 00:01:00 ``` After — success, failure, and cancelled: ``` ✅ Snapsync succeeded: lighthouse-hoodi Commit: d66f17c fix(l1): handle <edge> & cases (#9999) Duration: 01:02:05 🚨 Snapsync failed: prysm-sepolia Commit: abc12345 perf(l1): speed up trie Duration: 00:01:01 ⚠️ Snapsync was cancelled: lighthouse-hoodi Commit: abc1234 chore: bump Duration: 00:00:11 ``` The headline links the run, the SHA links the commit, and the commit title gives the version context at a glance. **Checklist** - [ ] Updated `STORE_SCHEMA_VERSION` (crates/storage/lib.rs) if the PR includes breaking changes to the `Store` requiring a re-sync. — N/A (CI only)
1 parent 0045449 commit ec208dc

2 files changed

Lines changed: 98 additions & 68 deletions

File tree

‎.github/scripts/notify_snapsync_run.sh‎

Lines changed: 94 additions & 68 deletions
Original file line numberDiff line numberDiff line change
@@ -3,87 +3,113 @@ set -euo pipefail
33

44
# Usage: notify_snapsync_run.sh
55
# Expects the following env vars (provided by the caller workflow):
6-
# SLACK_WEBHOOK_URL_SUCCESS, SLACK_WEBHOOK_URL_FAILURE, REPO, NAME, OUTCOME, HEAD_SHA, START_TIME
7-
8-
if [[ -z "${SLACK_WEBHOOK_URL_SUCCESS}" ]]; then
9-
echo "Slack webhook URL not provided; skipping notification." >&2
10-
exit 0
11-
fi
12-
13-
if [[ -z "${SLACK_WEBHOOK_URL_FAILURE}" ]]; then
14-
echo "Slack webhook URL not provided; skipping notification." >&2
15-
exit 0
16-
fi
6+
# SLACK_WEBHOOK_URL_SUCCESS, SLACK_WEBHOOK_URL_FAILURE, REPO, NAME, OUTCOME,
7+
# HEAD_SHA, START_TIME, RUN_ID, RUN_ATTEMPT
8+
# Optional:
9+
# COMMIT_MESSAGE (falls back to `git log` of HEAD_SHA when unset)
1710

1811
REPO=${REPO:-}
1912
NAME=${NAME:-}
2013
OUTCOME=${OUTCOME:-}
2114
HEAD_SHA=${HEAD_SHA:-}
2215
START_TIME=${START_TIME:-}
16+
RUN_ID=${RUN_ID:-}
17+
RUN_ATTEMPT=${RUN_ATTEMPT:-1}
18+
COMMIT_MESSAGE=${COMMIT_MESSAGE:-}
2319

24-
DURATION_SECS=$((EPOCHSECONDS - START_TIME))
25-
DURATION=$(date -d@"$DURATION_SECS" -u +%H:%M:%S)
20+
if ! [[ "$RUN_ATTEMPT" =~ ^[0-9]+$ ]]; then
21+
RUN_ATTEMPT=1
22+
fi
2623

27-
SHORT_SHA="${HEAD_SHA:0:8}"
28-
COMMIT_URL="https://github.com/${REPO}/commit/${HEAD_SHA}"
24+
# Outcome decides both the destination channel and how the run is presented.
25+
# A cancellation (manual stop, runner death) is not a sync failure, so it gets
26+
# a distinct, lower-alarm headline while still going to the failure channel for
27+
# visibility.
28+
case "$OUTCOME" in
29+
success)
30+
WEBHOOK=${SLACK_WEBHOOK_URL_SUCCESS:-}
31+
HEADLINE_EMOJI=":white_check_mark:"
32+
HEADLINE_VERB="succeeded"
33+
;;
34+
cancelled)
35+
WEBHOOK=${SLACK_WEBHOOK_URL_FAILURE:-}
36+
HEADLINE_EMOJI=":warning:"
37+
HEADLINE_VERB="was cancelled"
38+
;;
39+
*)
40+
WEBHOOK=${SLACK_WEBHOOK_URL_FAILURE:-}
41+
HEADLINE_EMOJI=":rotating_light:"
42+
HEADLINE_VERB="failed"
43+
;;
44+
esac
2945

30-
# Construct the Slack payload using jq for safe JSON escaping
31-
if [[ "$OUTCOME" == "success" ]]; then
32-
PAYLOAD=$(jq -n \
33-
--arg name "$NAME" \
34-
--arg sha "$SHORT_SHA" \
35-
--arg commit_url "$COMMIT_URL" \
36-
--arg duration "$DURATION" \
37-
'{
38-
blocks: [
39-
{
40-
type: "section",
41-
text: {
42-
type: "mrkdwn",
43-
text: ":white_check_mark: Node snapsynced successfully"
44-
}
45-
},
46-
{
47-
type: "section",
48-
fields: [
49-
{ type: "mrkdwn", text: "*Job*\n\($name)" },
50-
{ type: "mrkdwn", text: "*Commit*\n<\($commit_url)|\($sha)>" },
51-
{ type: "mrkdwn", text: "*Duration*\n\($duration)" }
52-
]
53-
}
54-
]
55-
}')
56-
curl -sS --fail -X POST \
57-
-H 'Content-type: application/json' \
58-
--data "$PAYLOAD" \
59-
"$SLACK_WEBHOOK_URL_SUCCESS" || echo "Failed to send Slack notification" >&2
46+
# A missing webhook (e.g. on forks, where secrets are unavailable) is not an
47+
# error; failing to deliver to a configured webhook is.
48+
if [[ -z "$WEBHOOK" ]]; then
49+
echo "Slack webhook URL not provided for outcome '$OUTCOME'; skipping notification." >&2
50+
exit 0
51+
fi
52+
53+
# Escape the characters that are special in Slack mrkdwn text. Uses sed
54+
# because in bash >= 5.2 an unquoted `&` in a ${var//pat/repl} replacement
55+
# expands to the matched text instead of a literal ampersand.
56+
slack_escape() {
57+
printf '%s' "$1" | sed -e 's/&/\&amp;/g' -e 's/</\&lt;/g' -e 's/>/\&gt;/g'
58+
}
59+
60+
DURATION="unknown"
61+
if [[ "$START_TIME" =~ ^[0-9]+$ ]]; then
62+
DURATION_SECS=$((EPOCHSECONDS - START_TIME))
63+
if (( DURATION_SECS >= 0 )); then
64+
DURATION=$(date -d@"$DURATION_SECS" -u +%H:%M:%S)
65+
fi
66+
fi
67+
68+
RUN_URL="https://github.com/${REPO}/actions/runs/${RUN_ID}"
69+
if (( RUN_ATTEMPT > 1 )); then
70+
RUN_URL="${RUN_URL}/attempts/${RUN_ATTEMPT}"
71+
fi
72+
73+
HEADLINE="Snapsync ${HEADLINE_VERB}: $(slack_escape "$NAME")"
74+
if (( RUN_ATTEMPT > 1 )); then
75+
HEADLINE="${HEADLINE} (attempt ${RUN_ATTEMPT})"
76+
fi
77+
78+
# Prefer an explicitly provided commit subject; otherwise read it from the
79+
# checkout the job already has.
80+
COMMIT_TITLE="$COMMIT_MESSAGE"
81+
if [[ -z "$COMMIT_TITLE" && -n "$HEAD_SHA" ]]; then
82+
COMMIT_TITLE=$(git log -1 --format=%s "$HEAD_SHA" 2>/dev/null || true)
83+
fi
84+
COMMIT_TITLE=$(printf '%s' "$COMMIT_TITLE" | head -n 1)
85+
86+
if [[ -n "$HEAD_SHA" ]]; then
87+
SHORT_SHA="${HEAD_SHA:0:8}"
88+
COMMIT_URL="https://github.com/${REPO}/commit/${HEAD_SHA}"
89+
COMMIT_LINE="*Commit:* <${COMMIT_URL}|${SHORT_SHA}>"
90+
if [[ -n "$COMMIT_TITLE" ]]; then
91+
COMMIT_LINE="${COMMIT_LINE} $(slack_escape "$COMMIT_TITLE")"
92+
fi
6093
else
61-
PAYLOAD=$(jq -n \
62-
--arg name "$NAME" \
63-
--arg sha "$SHORT_SHA" \
64-
--arg commit_url "$COMMIT_URL" \
65-
--arg duration "$DURATION" \
94+
COMMIT_LINE="*Commit:* unknown"
95+
fi
96+
97+
DETAILS="${COMMIT_LINE}"$'\n'"*Duration:* ${DURATION}"
98+
99+
# Construct the Slack payload using jq for safe JSON escaping
100+
PAYLOAD=$(jq -n \
101+
--arg headline "${HEADLINE_EMOJI} *<${RUN_URL}|${HEADLINE}>*" \
102+
--arg details "$DETAILS" \
66103
'{
67104
blocks: [
68-
{
69-
type: "section",
70-
text: {
71-
type: "mrkdwn",
72-
text: ":rotating_light: *Node failed to snapsync*"
73-
}
74-
},
75-
{
76-
type: "section",
77-
fields: [
78-
{ type: "mrkdwn", text: "*Job*\n\($name)" },
79-
{ type: "mrkdwn", text: "*Commit*\n<\($commit_url)|\($sha)>" },
80-
{ type: "mrkdwn", text: "*Duration*\n\($duration)" }
81-
]
82-
}
105+
{ type: "section", text: { type: "mrkdwn", text: $headline } },
106+
{ type: "section", text: { type: "mrkdwn", text: $details } }
83107
]
84108
}')
85-
curl -sS --fail -X POST \
109+
110+
# Let delivery failures fail the job so lost notifications are visible in the
111+
# Actions tab instead of being silently swallowed.
112+
curl -sS --fail --retry 3 -X POST \
86113
-H 'Content-type: application/json' \
87114
--data "$PAYLOAD" \
88-
"$SLACK_WEBHOOK_URL_FAILURE" || echo "Failed to send Slack notification" >&2
89-
fi
115+
"$WEBHOOK"

‎.github/workflows/daily_snapsync.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -128,6 +128,8 @@ jobs:
128128
OUTCOME: ${{ steps.snapsync.outcome }}
129129
HEAD_SHA: ${{ github.sha }}
130130
START_TIME: ${{ steps.start.outputs.timestamp }}
131+
RUN_ID: ${{ github.run_id }}
132+
RUN_ATTEMPT: ${{ github.run_attempt }}
131133
run: |
132134
bash .github/scripts/notify_snapsync_run.sh
133135
@@ -178,5 +180,7 @@ jobs:
178180
OUTCOME: ${{ steps.snapsync.outcome }}
179181
HEAD_SHA: ${{ github.sha }}
180182
START_TIME: ${{ steps.start.outputs.timestamp }}
183+
RUN_ID: ${{ github.run_id }}
184+
RUN_ATTEMPT: ${{ github.run_attempt }}
181185
run: |
182186
bash .github/scripts/notify_snapsync_run.sh

0 commit comments

Comments
 (0)