Provide a reasonable robots.txt

[Sainan]

Feature Description

There's a lot of views in a Gitea instance like diff/compare and blame which may be useful for a user but are absolute bloat for crawlers, which a reasonable robots.txt (e.g. https://github.com/robots.txt) could filter out. Right now, Gitea gives 404 page not found (https://demo.gitea.com/robots.txt), resulting in frustration all around.

Screenshots

No response

[Sainan]

This seems reasonable to me from a quick glance at some logs, at least it would eliminate some of the bigger issues:

User-agent: *

Disallow: /user/login?redirect_to=*
Disallow: /*/*/compare

Disallow: /*/*/issues?*
Disallow: /*/*/pulls?*

Disallow: /*/*/src/commit/*
Disallow: /*/*/raw/commit/*
Disallow: /*/*/blame/commit/*
Disallow: /*/*/commits/commit/*

Disallow: /*/*/archive/*

I've deployed this on my instance now and will update it as I notice problems.

[a1012112796]

looks gitea hasn't provide default robots.txt, you can add it manually. ref:

gitea/routers/web/misc/misc.go

Lines 36 to 43 in a0f492d

	func RobotsTxt(w http.ResponseWriter, req *http.Request) {
	robotsTxt := util.FilePathJoinAbs(setting.CustomPath, "public/robots.txt")
	if ok, _ := util.IsExist(robotsTxt); !ok {
	robotsTxt = util.FilePathJoinAbs(setting.CustomPath, "robots.txt") // the legacy "robots.txt"
	}
	httpcache.SetCacheControlInHeader(w.Header(), httpcache.CacheControlForPublicStatic())
	http.ServeFile(w, req, robotsTxt)
	}

https://docs.gitea.com/administration/search-engines-indexation#block-search-engines-indexation-using-robotstxt

[silverwind]

Many users configure a custom robots.txt, but I agree we should probably provide a reasonable default.