Setting to disable expensive endpoints for anonymous users

[Victorious3]

Feature Description

Since AI scrapers are terrorizing the web and flooding innocent gitea instances, it would make sense to have an option to only allow expensive endpoints (like /src/commit or /blame) for logged in users.

What I have observed is that crawlers like Claudebot and Bytespider don't respect my robots.txt and decide to crawl every single file from every single commit. For big repositories this can become a massive performance hit since gitea has to run git to be able to serve the requests, which has a lot of overhead. I even enabled a redis cache but since they hit new files all the time it didn't help much.

As a workaround I have configured my reverse proxy nginx to redirect these endpoints to an Anubis instance (https://anubis.techaro.lol/) which seems to kill most of the scrapers or at least wastes their time for long enough to make their DDOS (because that's what it is, really!) less annoying.

However, since this is a solution that works on proxying with nginx, every user sees the Anubis thing before being able to look at commits, even if they are logged in. Therefore it would be preferrable to just have an option to disallow these endpoints. If someone external wants to look at the commits they can just check out the repository and look at the history there.

Screenshots

No response

[qwertfisch]

This feature would be very welcomed. I have the same problems as there are AI scrapers which do not even set an appropriate useragent string, that terrorize my Gitea instance for a small open source project. The vast majority of accesses go to /pulls? and /issues?.

Edit: my proposal to block queries with GET parameters seems pointless as even navigating through the issues needs a page parameter. So it would indeed help disabling some endpoints or (better) giving permission settings for anonymous users, with possible values read/write, read and none.

[lunny]

Or perhaps anonymous users will have limited access or reduced traffic.

[wxiaoguang]

What do you think about this?

-> Add a config option to block "expensive" pages #34024

[delvh]

@wxiaoguang isn't that another solution for #33951 ?
That PR sounds much less intrusive to me.
Instead of outright crippling gitea - the routes you describe are the routes I myself use the most often - we could also use that approach to gracefully degrade only when there is a too large request queue…
That would fix the same problem and probably much more user-friendly.

[wxiaoguang]

These 2 are different.

• Block "expensive" pages (including issues/PRs): no need to spend CPU/Memory resources for the AI crawlers.
• Git content QoS: it only focuses "git content", it still consumes CPU/Memory to render expensive pages.

These 2 PRs could also co-exist and won't conflict.

And since #33951 has no progress in recent days, I think at least we can implement this issue's proposal to help the users under "AI crawler attack" as a quick solution.

[wxiaoguang]

1.23 nightly is ready (it is a stable release and will be 1.23.7 soon)

It has a new config option:

[service]
;; User must sign in to view anything.
;; It could be set to "expensive" to block anonymous users accessing some pages which consume a lot of resources,
;; for example: block anonymous AI crawlers from accessing repo code pages.
;; The "expensive" mode is experimental and subject to change.
;REQUIRE_SIGNIN_VIEW = false

Welcome to try and provide feedbacks.

• https://dl.gitea.com/gitea/1.23-nightly/
• https://hub.docker.com/r/gitea/gitea/tags?name=1.23-nightly

[qwertfisch]

Works wonderful, thank you. It already reduced the load and the number of accesses both.

@wxiaoguang Is it possible to configure what endpoints are considered as “expensive”? I would like to give anonymous users access to the source code, because this is already what one can see on the main repository page, but only the root directory. Also if users are able to checkout the complete repo anonymously, it makes not much sense to restrict the /src paths.

[wxiaoguang]

/src is quite special and indeed it is one of the most expensive endpoints, it is the endpoint which is the direct victim of AI crawlers ........ so exposing it to anonymous users would just bring the problem back.

---

Since the REQUIRE_SIGNIN_VIEW = expensive is experimental and it does work, maybe you could try to fine tune the logic and build your own binary to find the balance. And actually #33951 could do better to have QoS for some endpoints.

[qwertfisch]

Normally I would say this is an expensive endpoint. It’s just that in my case the bots hammered on the /pull endpoint with 98% of all requests, whereas /src was hardly frequented. I will try to have a look in the code and build a custom binary. At least the raid stopped (down from 2m requests per day to 60k) using this measure. And interested real people can still checkout or register, so it’s not too restricted or inconvenient.

[richmahn]

@wxiaoguang Was there any change to the admin config settings page to reflect this? I turned it on the "expensive", but I still get an X. Could we maybe get it to say if the expensive setting took?

[wxiaoguang]

Was there any change to the admin config settings page to reflect this? I turned it on the "expensive", but I still get an X. Could we maybe get it to say if the expensive setting took?

"admin config settings page" is quite out-dated. A lot of config options are not there or incorrectly displayed there.

To confirm, just try to visit a repo's file anonymously.

---

If we'd like to make the "admin config settings page" work correctly, it needs to completely refactor the setting system.

[kolaente]

Just enabled this, good stuff.

My two cents: I think it could be a little more fine-grained though - not being able to view issues strikes me as too much. I also think it should be possible to browse code, but only for the default branch, not for any commit.

And can we add the compare endpoint as well?

[wxiaoguang]

I have a complete plan here: #33951 (comment)

1. Anti-crawler policy: block expensive pages, apply QoS to expensive pages
2. Anti-crawler action: respond 503, redirect to login, show a JS page to do some proof-of-work or use some rate limit token (this one seems to provide best user experience).

Then end users could choose a combination to satisfy their requirements.

---

To answer your question:

1. I think "block expensive pages" + "show a JS page with PoW or rate limit token" should work for you (the "action" is not implemented yet)
2. The compare endpoint is also heavy, but I think at the moment since other pages have been blocked, so the crawlers won't see it, so I guess it won't be a problem?

[wxiaoguang]

not being able to view issues strikes me as too much.

For this one: would you like to build your own binary to allow the "issues" related endpoints? I think they are not that heavy and it might work well in your case.

[kolaente]

I like these ideas!

The compare endpoint is also heavy, but I think at the moment since other pages have been blocked, so the crawlers won't see it, so I guess it won't be a problem?

I was still able to reach it via the releases.

[wxiaoguang]

Would you like to try this one? Rate-limit anonymous accesses to expensive pages when the server is overloaded #34167

It will do the best to serve contents to end users and crawlers, anonymous end users just need to wait for 1 second to see the "expensive" page if the server is overloaded.