Simple List Downloader instead of using Blocky Fetching

[huseyinkogo]

I initially done this script because issue #1654.

Snippet of the config.yml

blocking:
  denylists:
    general:
      - /opt/blocky/lists/pro.txt
      - /opt/blocky/lists/fake.txt
      - /opt/blocky/lists/popupads.txt
      - /opt/blocky/lists/tif.txt
      - /opt/blocky/lists/doh-vpn-proxy-bypass.txt
      - /opt/blocky/lists/hoster.txt
      - /opt/blocky/lists/dyndns.txt
      - /opt/blocky/lists/anti.piracy.txt
      - /opt/blocky/lists/gambling.txt
      - /opt/blocky/lists/native.samsung.txt
      - /opt/blocky/lists/native.apple.txt
      - /opt/blocky/lists/native.winoffice.txt
      - /opt/blocky/lists/native.tiktok.txt
      - /opt/blocky/lists/native.amazon.txt
      - /opt/blocky/lists/nrd-phishing-30day_wildcard.txt
      - /opt/blocky/lists/domainswild
  allowlists:
    general:
      - /opt/blocky/lists/allowlist.txt

Bash Script - List Downloader

#!/bin/bash
source /opt/blocky/.env
IP_ADDRESS=$(hostname -i)
target_dir="/opt/blocky/lists"

url_list=(
  "https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro.txt"
  "https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/fake.txt"
  "https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/popupads.txt"
  "https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif.txt"
  "https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/doh-vpn-proxy-bypass.txt"
  "https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/hoster.txt"
  "https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/dyndns.txt"
  "https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/anti.piracy.txt"
  "https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/gambling.txt"
  "https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/native.samsung.txt"
  "https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/native.apple.txt"
  "https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/native.winoffice.txt"
  "https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/native.tiktok.txt"
  "https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/native.amazon.txt"
  "https://raw.githubusercontent.com/xRuffKez/NRD/refs/heads/main/lists/30-day_phishing/wildcard/nrd-phishing-30day_wildcard.txt"
  "https://nsfw.oisd.nl/domainswild"
  "https://git.local.dev/homelab/blocky/raw/branch/main/lists/allowlist.txt"
)

any_failed=false

for url in "${url_list[@]}"; do
  filename=$(basename "$url")
  target_file="$target_dir/$filename"

  curl --remote-header-name -L -s -o "$target_file" -w "%{http_code}" "$url" > http_code.txt

  http_code=$(cat http_code.txt)
  rm http_code.txt

  if [ "$http_code" -eq 200 ] && [ -f "$target_file" ] && [ -s "$target_file" ]; then
    echo "Downloaded $url to $target_file"
  else
    echo "Failed to download $url (HTTP code: $http_code)"
    rm -f "$target_file"
    any_failed=true
  fi

  sleep 5
done

if $any_failed; then
  MESSAGE="One or more blocklist files failed to download or were empty."
  TITLE="Download Failure on Blocky Server List - $IP_ADDRESS"
  TAGS="red_circle"
  PRIORITY=5

  curl -H "Authorization: Bearer $NTFY_EVLAB_TOKEN" \
       -H "Title: $TITLE" \
       -H "Tags: $TAGS" \
       -H "Priority: $PRIORITY" \
       -d "$MESSAGE" \
       "$NTFY_HOST/$NTFY_EVLAB_TOPIC"
fi

[shizunge]

This sounds similar to my downloader https://github.com/shizunge/blocky-lists-updater, which in addition could aggregate multiple lists into a single file, and deduplicate entries.