Unable to access repos after going from 1.15.0 => 1.15.2

[echodreamz]

• Gitea version (or commit ref): 1.15.2
• Git version: 2.33.0
• Operating system: Ubuntu 20.04.2 using systemd and Gitea downloaded from the github releases page (amd64 build)
• Database (use [x]):
  • PostgreSQL
  • [ x] MySQL
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL)
  • [ x] No
• Log gist:
  2021/09/05 09:58:23 ...ules/context/repo.go:343:repoAssignment() [E] GetUserRepoPermission: skipObjectDecoder: expect object or null, error found in #0 byte of ...|&\003d{\003d}\003d|..., bigger context ...|&\003d{\003d}\003d|... 2021/09/05 09:54:32 ...ters/private/serv.go:293:ServCommand() [E] Unable to get permissions for 3:REMOVED with key 8 in 2:ORG/myrepo Error: skipObjectDecoder: expect object or null, error found in #0 byte of ...|&\003d{\003d}\003d|..., bigger context ...|&\003d{\003d}\003d|... 2021/09/05 09:54:33 modules/ssh/ssh.go:140:sessionHandler() [E] SSH: Wait: exit status 1

Description

When logging in, I am able to see all commits on the dashboard to repos I have access to, but the right side repos card is empty and missing all repos that I have access to. When selecting to view a repo, I get a 500 error with the error above logged. Rolling back to 1.15.0 and everything returns to normal.

SSH access also fails with errors as well as shown above.

[NoirPi]

Facing the same Issue... even if i try version 1.15.1 i get the 500 error

needed to roll back to 1.15.0 to get it back working again.

[echodreamz]

Same here, v1.15.1 also has the same issue.

[XPhilipp]

v1.14.7 has the same issue.

[zeripath]

What version of MySQL are you using?

[zeripath]

Also are you restoring from a backup created on v1.14.3-v1.14.6? These look like the bug v1.14.7 and v1.15.1 were expressly created to fix.

[XPhilipp]

Also are you restoring from a backup created on v1.14.3-v1.14.6? These look like the bug v1.14.7 and v1.15.1 were expressly created to fix.

Yes, was restored from 1.14.5 or 1.14.4) And upgraded later to 1.14.7

What version of MySQL are you using?

mariadb-server/stable,now 1:10.5.11-1 all

[zeripath]

Also are you restoring from a backup created on v1.14.3-v1.14.6? These look like the bug v1.14.7 and v1.15.1 were expressly created to fix.

Yes, was restored from 1.14.5 or 1.14.4) And upgraded later to 1.14.7

This is the problem that 1.14.7 and 1.15.1/2 were released for.

The instructions on the 1.15.1/1.14.7 blog post are:

... We have specially released v1.14.7 as it contains an important fix for an issue with gitea dump on versions 1.14.3-1.14.6 and version 1.15.0.

We highly encourage users on versions v1.14.3-v1.14.7 to update to v1.14.7 before creating their backup dump files, and then update to v1.15.1 for some important bug-fixes.

By ignoring this you've restored from broken dump and now we're in trouble because fixing it is not going to be easy - at all.

I specifically made sure we released 1.14.7 to help stop this from happening.

[zeripath]

Please tell me that your 1.14.x db is still in existence outside of the dump?

[zeripath]

v1.14.7 has the same issue.

This implies that your db may already be hosed. ☹️

[XPhilipp]

Please tell me that your 1.14.x db is still in existence outside of the dump?

Yes, but lagging behind

Any way, can i fix db my self?

[zeripath]

It's not easy to fix.

The issue is that there is a bug with the dump command in 1.14.2-1.14.6 that means that instead of writing JSON it's doing the equivalent of fmt.Println(v) . That means instead of a field containing: {"EnableTimetracker":true,"AllowOnlyContributorsToTrackTime":true,"EnableDependencies":true} it ends up containing &{true true true}.

Now...

the table and fields affected by this are:

• Table login_source field cfg
• Table repo_unit field config

both of these are relatively rare to be updated so a good guess for the correct values is going to be the values that are in the previous db before the broken dump.

[echodreamz]

So overall, we need to stop using gitea itself to dump the database, but instead use the dedicated mysqldump tool?

Checked our oldest backup, (we keep 30 days). Unfortunately, it still has the issue within it. What should our next steps be?

Our login_source table is empty, appears to always have been. The repo_unit has about ~300 rows in it, it appears to be storing repo settings?

UPDATE:

After reviewing the repo_unit table, it appeared to be mostly empty JSON data with an &{} in front. We set all rows to {} which allowed the upgrade to 1.15.2 to go without issues, then went to each repo and reset/saved all the settings, which appears to repopulate the repo_unit table with the correct data.

Along with the gitea dump, we are also using mysqldump to make sure the data coming out of MySQL is correct (at least from a format standpoint).

[zeripath]

I'm saying there was a serious regression in 1.14.3-1.14.6 that meant the dumps were broken for these two columns but no one noticed. Certainly having spent so damned long making sure that they worked in the first place I didn't expect that they would suffer a regression like that.

I've fixed that regression and added a test to stop it from happening again, however we clearly need to add a stronger testsuite to the dumping behaviour as this test only covers a very small component.

I'm glad that your repo_units config column appeared so easily fixable but I'm surprised it was empty. Have you been consistently restoring from backups? My suspicion is you've lost some configuration data here. We need to check your older backups - please contact me on discord or on matrix.

It may not be coming across but I'm seriously and very deeply sorry and annoyed about this. It's just not good enough that this has happened. If we're gonna supply a dump command it should bloody well work.

[echodreamz]

No, we've never restored from backups. We've been on Gitea since... mid 2019. This was our first restore. We migrated from an old CentOS-based server to a shiny new Ubuntu-based servers. Did the gitea dump figuring it was good enough. We also had mad issues with the action table, apparently the backup dump done by gitea outputted \n instead of \\n so the data when imported back into MySQL had full line-breaks instead of just the entity, which was then invalid JSON. I was able to fix this pretty easily and everything went OK, though I did not realize issues existed elsewhere. Note that we were on 1.15.0 before we migrated.

Before this, we had no issues with Gitea aside from 1 other report I filed.

I am surprised that gitea dumps its own DB. Especially for MySQL, do a check and see if mysqldump is available (or even allow an option) and use it, why re-invent the wheel when a perfectly good dump tool exists and was created by the people who wrote the database server.

[lunny]

MySQL is not the only database Gitea supports.

For a workaround for your situation, regenerated repo_unit table should fix all the problems. Could you show some records of your repo_unit?