Do not block the database(sqlite) during archive creation

[matrss]

The database transaction in doArchive is currently held open for the entirety of the archive generation process. For larger repositories this could be a long time. If the database is sqlite3 (others not tested) then this can block other processes from using the database, which e.g. leads to long load times for the web UI route of all repositories on the gitea server, among other things. This patch finishes the transaction right before generating the archive and starts a new one to update the database afterwards.

You can reproduce the issue with the following steps:

1. create a git repository with a large file (e.g. 1GB seems sufficient to see the issue)
2. commit the file and push to a gitea instance using a sqlite database
3. from the web UI start a .tar.gz (or .zip) download
4. try to navigate to the same or another repository
5. observe a long loading time until the archive is created (i.e. the download starts)

---

I am new to Go and therefore not sure if this is the most idiomatic way to fix this. I essentially just copied the way the transaction is created in the beginning to after the archive is created and .Commit and .Closed before that. Specifically I am not sure if I would have to propagate the err from .Close as well, since that is deferred in the beginning which would silence the error as well, right? Is this "double .Close" from the defer and the explicit call OK to do?

[lunny]

How about moving the archive generation before database updates?

[matrss]

That would work if the doArchive function is only ever called when the archive should definitely be created, but the existence of

gitea/services/repository/archiver/archiver.go

Lines 189 to 194 in 170177c

	if archiver != nil {
	// FIXME: If another process are generating it, we think it's not ready and just return
	// Or we should wait until the archive generated.
	if archiver.Status == repo_model.ArchiverGenerating {
	return nil, nil
	}

suggests that this is not the case, i.e. there can be a situation in which the archive shouldn't be created. I am not familiar enough with the codebase to judge if that would be a real issue though, or if it would "just" be unnecessary work.

[lunny]

That would work if the doArchive function is only ever called when the archive should definitely be created, but the existence of

gitea/services/repository/archiver/archiver.go

Lines 189 to 194 in 170177c

	if archiver != nil {
	// FIXME: If another process are generating it, we think it's not ready and just return
	// Or we should wait until the archive generated.
	if archiver.Status == repo_model.ArchiverGenerating {
	return nil, nil
	}

suggests that this is not the case, i.e. there can be a situation in which the archive shouldn't be created. I am not familiar enough with the codebase to judge if that would be a real issue though, or if it would "just" be unnecessary work.

Yes, that's a simpler lock mechanism based on database. And I think this is only affecting sqlite but for other database, it works.

[matrss]

And I think this is only affecting sqlite but for other database, it works.

Yes, I think the underlying issue is sqlite's limitations in regard to concurrent reads/writes. Holding the transaction open for the entire duration of archive creation seems to even block read access to the database.

Other databases do not have these limitations, although keeping the transaction open for a long time might have some effect there as well, like increased resource usage and such to accommodate this situation.

Some sqlite options, like WAL mode, might also have an effect on how this issue manifests itself, but I haven't tested out anything in that regard.

[lunny]

It's better to use the current globallock modules to instead of database operations.

[lunny]

Can you help to confirm that #32186 can resolve the problem?

[lunny]

Replaced by #32186