Initial Git hook syncing causes downtime for large instances

[fnetX]

Description

#21619, #21625 & #21626 seem to do a Git hook sync on first run, because previous app config path is empty.

In our logs, we see

2022/11/07 02:57:24 routers/init.go:90:syncAppConfForGit() [I] CustomConf changed from '' to '/etc/gitea/conf/app.ini'

and for several tens of thousands of repositories, this takes quite a while, and caused unexpected downtime.

We reverted the patch for now in https://codeberg.org/Codeberg/gitea/commit/26b92e3f80961de11a48d3d307d50065aca8922a

Gitea Version

1.17.3 7a2daae

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

No response

Operating System

No response

How are you running Gitea?

Custom build

Database

MySQL

[fnetX]

If at all, this should be done async, after the software is back online IMHO. Or the value for the config should not be empty on first run.

Another solution I proposed a while ago: #18995

[lunny]

If at all, this should be done async, after the software is back online IMHO. Or the value for the config should not be empty on first run.

Another solution I proposed a while ago: #18995

You can also run the sync task on admin panel. And for a large instance, maybe more than 1 instance is better so that you could upgrade one by one.

[fnetX]

My point is: Our app location didn't change, it is where it always was, so we don't even need to update 47k project's hooks. This sync is only triggered because our alleged app location is empty.

[lunny]

It should have been checked if the configuration file path changed. ref https://github.com/go-gitea/gitea/pull/21626/files#diff-3d9812b7e268b55c71e312a7dd154900ab1036c47493f4cb91f608b32a7541fcR91

[fnetX]

Yes, and as you surely read in my initial report, it is considered "empty" for some reason. But the config file wasn't moved.

2022/11/07 02:57:24 routers/init.go:90:syncAppConfForGit() [I] CustomConf changed from '' to '/etc/gitea/conf/app.ini'

[lunny]

Yes, and as you surely read in my initial report, it is considered "empty" for some reason. But the config file wasn't moved.

2022/11/07 02:57:24 routers/init.go:90:syncAppConfForGit() [I] CustomConf changed from '' to '/etc/gitea/conf/app.ini'

Ah, yes. It's a problem for the first running. The new system state item was introduced from that PR, so before it's empty, but next time it should be right value.

[fnetX]

Yes, but in order to reach next time, we need to face a prolonged downtime in order to sync the hooks of many repositories, this is what the bug report is about.

[wxiaoguang]

It helps poor users like #22049

[fnetX]

The implemented feature helps users, yes. The bug that the initial value is empty does not help anyone, but is a bug.

[wxiaoguang]

But what's the proper initial value? Gitea knows nothing about the correct config path at the beginning.

Users are usually affected by such bug when then do upgrade:

• User is running an old Gitea with /etc/gitea/app-1.17.ini
• Then the user decides to upgrade Gitea and use a config /etc/gitea/app-1.18.ini
• When new Gitea runs, Gitea only sees /etc/gitea/app-1.18.ini, if it's stored as the default value, the git hooks are out-of-sync then.

ps: it could be a rare case, but it could happen, it only happens for users who first upgrade to Gitea with config path sync support and they change the config path.

[lunny]

The newly installed big instances will not be affected by this if they doesn't change the configuration file path.

[wxiaoguang]

Close as there is no new proposal.