Is scraping permitted within GitHub Actions usage?

[ananyabchetia]

Why are you starting this discussion?

Question

What GitHub Actions topic or product is this about?

General

Discussion Details

Hi, I am an early career journalist creating a public-interest dashboard.
I run a very small scraper every two weeks using GitHub Actions to update a single CSV in my repo. Specifically, I am looking at a nonprofit which files complaints against U.S universities. I want to regularly update the CSV so users can track if their university is on the list. I scrape every two weeks. I can also make it once a month.
It is light, responsible, and tied to my GitHub Pages project.
Is this permitted within GitHub Actions usage?

[krishparmar22242]

Yes, this is absolutely permitted and is well within the acceptable use policy for GitHub Actions.

Why your project is permitted
GitHub’s Terms of Service generally prohibit using Actions for "mining" or "server-like" tasks that consume excessive resources. However, running a small script every two weeks to update a CSV is considered a low-impact, productive use of the platform. Since your work is for a public-interest dashboard, it aligns perfectly with GitHub's mission of supporting open data and transparency.

Key things to keep in mind (Responsible Use)
Even though it is permitted, you should follow these "best practices" to ensure your workflow stays healthy:

1. Frequency: Running every two weeks (or even once a week) is very light. You only run into trouble if you try to scrape every minute or every hour, as that can be flagged as "resource abuse."
2. The "Scraping" Aspect: GitHub doesn't mind the automation, but the target website might. Since you mentioned your scraper is "light and responsible," ensure you are following the target site’s robots.txt and not overwhelming their servers.
3. Repo Size: Since you are updating a CSV, the Git history will grow over time. As long as the CSV doesn't grow into the gigabyte range, you are perfectly fine.

Best Practices for your Workflow

1. Use GitHub Secrets: If your scraper needs any API keys or credentials, never hard-code them in your script. Store them in Settings > Secrets and variables > Actions.
2. Check for Changes: To keep your Git history clean, you can add a small check in your Action to see if the CSV actually changed. If the data is the same as last time, you can skip the "commit" step so you don't create unnecessary "No change" commits.

Example "Clean Commit" Logic
You can use this snippet in your .yml file to ensure you only commit when the data actually changes:

name: Commit and Push if changed
run: |
git config --global user.name "github-actions[bot]"
git config --global user.email "github-actions[bot]@users.noreply.github.com"
git add data.csv
git diff --quiet && git diff --staged --quiet || (git commit -m "Update university data" && git push)

Your project sounds like a great service to the community. Good luck with your dashboard!

If this answer clarifies your concerns, please consider marking this as the accepted answer so it can help other journalists and researchers find this information easily!

[krishparmar22242]

thank you @ananyabchetia

[mayurcodes01]

Hi, thanks for explaining the context so clearly. Short answer: yes, what you’re describing is generally permitted.

GitHub Actions is intended for CI, automation, and scheduled maintenance tasks, and a small, infrequent scraper that updates a CSV on a fixed schedule fits well within that scope. Running a lightweight job every two weeks, or even monthly, to refresh data for a GitHub Pages project is well within normal and responsible usage.

A few practical points to keep in mind:

Resource usage: Keep the job lightweight and short-running, which it sounds like you already are. Avoid long loops, heavy parallelism, or frequent retries.

Scheduling: A biweekly or monthly cron schedule is very reasonable and unlikely to raise any flags.

Scraping behavior: Make sure you’re respecting the target site’s robots.txt and terms of service, and avoid aggressive request rates.

Transparency: Document in your repo what the Action does and why. This is especially helpful for public-interest projects like yours.

As long as the workflow isn’t being used to provide a commercial scraping service, overwhelm a third-party site, or bypass access controls, it’s squarely within acceptable use.

This sounds like a thoughtful, low-impact use of Actions for a public-interest project. You should be fine.