500 error accessing dashboard after upgrade to 1.16.1 from 1.15.10, caused by mismatched SQLite colmun type #18650

nicorac · 2022-02-07T07:44:40Z

Gitea Version

1.16.1

Git Version

2.25.1

Operating System

Ubuntu Server 20.04 LTS

How are you running Gitea?

Running as service with binaries (gitea-1.16.1-linux-amd64).

Database

SQLite

Can you reproduce the bug on the Gitea demo site?

No

Description

After upgrading from 1.15.10 to 1.16.1 I get a 500 error when accessing the dashboard.
Other pages seem to work correctly if I open them directly (repos, issues, ...).

(the same issue happened a week ago when I tried to upgrade from 1.15.10 to 1.16.0 but I haven't opened an issue at that time due to lack of time, sorry)

Log contains only this relevant line:

.../web/feed/profile.go:22:RetrieveFeeds() [E] GetFeeds: Find: converting driver.Value type string to a int64: invalid syntax

Database version before update was 189, after first run it was 209.

I've reverted back to 1.15.10 (both binaries and DB).

The text was updated successfully, but these errors were encountered:

lunny · 2022-02-07T07:54:07Z

Could you paste more logs before this log and after?

lunny · 2022-02-07T07:55:31Z

And could you paste your database table action's structure?

nicorac · 2022-02-07T08:03:31Z

This is the first log line in gitea.log as of today, repeated more times (I hit F5 morte than once).
Then another appeared, maybe related.
The last line is my attempt to get back to 1.15.10 before restoring the DB:

2022/02/07 08:22:41 .../web/feed/profile.go:22:RetrieveFeeds() [E] GetFeeds: Find: converting driver.Value type string to a int64: invalid syntax
2022/02/07 08:22:43 .../web/feed/profile.go:22:RetrieveFeeds() [E] GetFeeds: Find: converting driver.Value type string to a int64: invalid syntax
2022/02/07 08:22:44 .../web/feed/profile.go:22:RetrieveFeeds() [E] GetFeeds: Find: converting driver.Value type string to a int64: invalid syntax
2022/02/07 08:22:54 .../web/feed/profile.go:22:RetrieveFeeds() [E] GetFeeds: Find: converting driver.Value type string to a int64: invalid syntax
2022/02/07 08:23:47 ...s/context/context.go:290:PlainTextBytes() [E] PlainTextBytes: Not found.
2022/02/07 08:23:47 ...s/context/context.go:290:PlainTextBytes() [E] PlainTextBytes: Not found.
2022/02/07 08:25:22 .../web/feed/profile.go:22:RetrieveFeeds() [E] GetFeeds: Find: converting driver.Value type string to a int64: invalid syntax
2022/02/07 08:25:43 .../web/feed/profile.go:22:RetrieveFeeds() [E] GetFeeds: Find: converting driver.Value type string to a int64: invalid syntax
2022/02/07 08:26:48 ...ations/migrations.go:414:Migrate() [F] Downgrading database version from '209' to '189' is not supported and may result in loss of data integrity.
        If you really know what you're doing, execute `UPDATE version SET version=189 WHERE id=1;`

This is the requested table structure (in v1.15.10 DB):

CREATE TABLE [action] (
    id           INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    user_id      INTEGER,
    op_type      INTEGER,
    act_user_id  INTEGER,
    repo_id      INTEGER,
    comment_id   INTEGER,
    is_deleted   INTEGER DEFAULT 0 NOT NULL,
    ref_name     TEXT,
    is_private   INTEGER DEFAULT 0 NOT NULL,
    content      TEXT,
    created_unix INTEGER
);
CREATE INDEX IDX_action_user_id ON [action] (user_id);
CREATE INDEX IDX_action_act_user_id ON [action] (act_user_id);
CREATE INDEX IDX_action_repo_id ON [action] (repo_id);
CREATE INDEX IDX_action_comment_id ON [action] (comment_id);
CREATE INDEX IDX_action_is_deleted ON [action] (is_deleted);
CREATE INDEX IDX_action_is_private ON [action] (is_private);
CREATE INDEX IDX_action_created_unix ON [action] (created_unix);

zeripath · 2022-02-08T00:52:12Z

The error is coming from the sqlite DB itself and it indicates that one or more of the rows of action has a string value in one of its integer columns.

How this has happened I do not know. Gitea shouldn't have done this directly. Have you restored from a cross db backup previously?

The problem is that this cannot be automatically fixed. You're going to have to look at the sqlite DB yourself using sqlitebrowser or sqlite3. My suspicion is going to be that one of the is_ prefixed columns contains "true" and/or "false" when they should be 1 or 0.

wxiaoguang · 2022-02-08T05:59:37Z

Maybe related to

Users and Organizations gone #17748 (comment)

The SQLite database was corrupted

nicorac · 2022-02-08T07:28:16Z

The database worked correctly up to 1.15.10.

Have you restored from a cross db backup previously?

My steps:

stopped 1.15.10
backed up the DB file
downloaded 1.16.1 binary and started it
got 500 error
reverted to 1.15.10 and restored DB
restarted 1.15.10... it still works

My suspicion is going to be that one of the is_ prefixed columns contains "true" and/or "false" when they should be 1 or 0.

This is a good starting point, thanks.
I'll check it out as soon as I can stop the service 👍

zeripath · 2022-02-08T09:59:32Z

There should be no need to stop the service to just read the data - sqlite will not take a lock unless you start writing to the db and once you write the lock will be released.

The issue would come if you start editing data but don't write it immediately as that will lock the table and gitea will hold-up until you release that lock.

nicorac · 2022-02-09T08:59:02Z

I've checked actions table content and none of the is_* columns contain text, only 0 and 1.

The first 5 records in the table (id 118 ->122) have some differences with all the others:

they are the only one with is_private == 0
4 of them they have repo_id == 0
ref_name contains a number instead of branch/tag names like "master", "integration", ...
created_unix column is empty
next id after these 5 rows is 359 (no IDs from 123 to 358...)

id	user_id	op_type	act_user_id	repo_id	comment_id	is_deleted	ref_name	is_private	created_unix
118	2	2	22	0	0	0	0	0
119	0	2	19	0	0	0	0	0
120	2	2	19	0	0	0	0	0
121	9	2	29	0	0	0	2	0
122	7	2	29	17	0	0	0	0
359	4	1	4	11	0	0		1	1549289674
360	1	1	4	11	0	0		1	1549289674
361	4	1	4	12	0	0		1	1549292689
362	1	1	4	12	0	0		1	1549292689
...(other 14000+ similar records, with consecutive IDs)

Could it be that those 5 leading records are causing the issue?

lunny · 2022-02-09T09:27:56Z

You can try to deleting them after you have a backup.

nicorac · 2022-02-09T18:09:35Z

You can try to deleting them after you have a backup.

It worked: after deleting those 5 records I can now access toe dashboard without errors, thanks!

Just wondering what I've lost by deleting them and if there's a way to fix and re-adding them back...

wxiaoguang · 2022-02-10T03:02:47Z

These records are not important, they are used used to show user activities, so you can safely delete them (I even truncated the action table before, Gitea still works correctly).

It seems that empty created_unix causes the bug. SQLite doesn't have data type inside.

Is your database created by a very old Gitea?

nicorac · 2022-02-10T07:29:32Z

These records are not important, they are used used to show user activities, so you can safely delete them

Ok, I've suspected that... can live without them 😉

Is your database created by a very old Gitea?

Yes it is.
I don't remember the version, but it surely was one of the very first ones...

lunny · 2022-02-10T09:11:44Z

But I think the created_unix should be there when action table created.

nicorac · 2022-02-10T09:17:55Z

But I think the created_unix should be there when action table created.

I think so, but it didn't cause any issue till 1.15.10.
I hope this thread will help any other one lucky as me 😄

wxiaoguang · 2022-02-10T09:32:50Z

But I think the created_unix should be there when action table created.

No, I have reviewed the whole history, it changed a lot.

lunny · 2022-02-10T13:50:00Z

But I think the created_unix should be there when action table created.

No, I have reviewed the whole history, it changed a lot.

Even if v1.0.0 the field has been there. https://github.com/go-gitea/gitea/blob/v1.0.0/models/action.go#L86

wxiaoguang · 2022-02-10T14:00:42Z

The previous commit of v1.0.0 doesn't have that field. https://github.com/go-gitea/gitea/blame/0c9a616326ba096a2ff6c058cc96950f68c0fa6e/models/action.go

And I am not sure whether there are something wrong occurs during SQLite migrations since the field type/index all changed a lot. Well, this is just a guess.

@nicorac can you help to check what's the earliest created_unix in your action table?

nicorac · 2022-02-10T14:03:23Z

what's the earliest created_unix in your action table?

It's 1549289674, which means GMT: Monday 4 February 2019 14:14:34

wxiaoguang · 2022-02-10T14:06:56Z

what's the earliest created_unix in your action table?

It's 1549289674, which means GMT: Monday 4 February 2019 14:14:34

Well, not that early 😂 I have no idea

6543 · 2022-05-07T18:53:11Z

should be fixed at #19629 ?

singuliere · 2022-05-08T06:47:04Z

Yes, #19629 will fix the problem of the rows that have a repo_id of 0 but not the one with repo_id = 17 if the root cause of this particular error was indeed the empty created_unix field.

singuliere · 2022-05-08T08:50:04Z

A null created_unix field cannot explain the error

2022/02/07 08:22:41 .../web/feed/profile.go:22:RetrieveFeeds() [E] GetFeeds: Find: converting driver.Value type string to a int64: invalid syntax

Figuring out the root cause would require more information:

A copy of the database to repeat the problem, or
An error message from XORM that provides a copy of the row that caused the error

What is puzzling is that there are no fields in Action that Find would need to convert from string to int64...

wxiaoguang · 2022-05-08T09:11:50Z

should be fixed at #19629 ?

I do not think they are the same problem. The problem in this issue is caused by incorrect database value, it might be caused by incorrect SQLite database operations (eg: incorrect upgrade/downgrade, incorrect dump/restore, etc) or unknown bugs (eg: during migration).

There is a chance that #19629 could bypass the problem, but the data in SQLite database is still incorrect.

I would propose to close this issue since there is no clue. Other databases won't be affected by this problem either.

And this problem can be bypassed manually (easily) by truncating action table or deleting the incorrect data in action table.

singuliere · 2022-05-08T09:17:18Z

Other databases won't be affected by this problem either.

Why ?

wxiaoguang · 2022-05-08T09:20:38Z

Log: converting driver.Value type string to a int64: invalid syntax

singuliere · 2022-05-08T09:28:50Z

SQLite doesn't have data type inside.

I do not understand this sentence. How does that explain why other databases won't be affected by this problem?

wxiaoguang · 2022-05-08T09:32:29Z

SQLite doesn't have data type inside.

I do not understand this sentence. How does that explain why other databases won't be affected by this problem?

In SQLite, you can have created_unix='' or created_unix=null or created_unix=123
In others, you can only have created_unix=null or created_unix=123

created_unix='' will cause the converting driver.Value type string to a int64 error, others won't.

singuliere · 2022-05-08T10:05:44Z

I did not know that, thanks for explaining. Since this problem can still exist for Gitea instances that run on SQLite, I propose this issue is left open.

singuliere · 2022-05-08T16:14:39Z

With these improvements to gitea doctor that replace created_unix='' with created_unix=0 this issue can be closed.

wxiaoguang changed the title ~~500 error accessing dashboard after upgrade to 1.16.1 from 1.15.10~~ 500 error accessing dashboard after upgrade to 1.16.1 from 1.15.10, caused by mismatched SQLite colmun type Apr 6, 2022

6543 added this to the 1.16.8 milestone May 7, 2022

wxiaoguang added the issue/workaround it is or has a workaround label May 8, 2022

wxiaoguang removed this from the 1.16.8 milestone May 8, 2022

singuliere mentioned this issue May 8, 2022

doctor: add check/fix for bogus Action rows #19656

Merged

6543 closed this as completed in #19656 May 10, 2022

go-gitea locked and limited conversation to collaborators May 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

500 error accessing dashboard after upgrade to 1.16.1 from 1.15.10, caused by mismatched SQLite colmun type #18650

500 error accessing dashboard after upgrade to 1.16.1 from 1.15.10, caused by mismatched SQLite colmun type #18650

nicorac commented Feb 7, 2022 •

edited

Loading

lunny commented Feb 7, 2022

lunny commented Feb 7, 2022

nicorac commented Feb 7, 2022 •

edited

Loading

zeripath commented Feb 8, 2022

wxiaoguang commented Feb 8, 2022

nicorac commented Feb 8, 2022

zeripath commented Feb 8, 2022

nicorac commented Feb 9, 2022

lunny commented Feb 9, 2022

nicorac commented Feb 9, 2022

wxiaoguang commented Feb 10, 2022 •

edited

Loading

nicorac commented Feb 10, 2022

lunny commented Feb 10, 2022

nicorac commented Feb 10, 2022 •

edited

Loading

wxiaoguang commented Feb 10, 2022

lunny commented Feb 10, 2022

wxiaoguang commented Feb 10, 2022

nicorac commented Feb 10, 2022

wxiaoguang commented Feb 10, 2022

6543 commented May 7, 2022

singuliere commented May 8, 2022

singuliere commented May 8, 2022

wxiaoguang commented May 8, 2022 •

edited

Loading

singuliere commented May 8, 2022

wxiaoguang commented May 8, 2022

singuliere commented May 8, 2022

wxiaoguang commented May 8, 2022

singuliere commented May 8, 2022

singuliere commented May 8, 2022 •

edited

Loading

500 error accessing dashboard after upgrade to 1.16.1 from 1.15.10, caused by mismatched SQLite colmun type #18650

500 error accessing dashboard after upgrade to 1.16.1 from 1.15.10, caused by mismatched SQLite colmun type #18650

Comments

nicorac commented Feb 7, 2022 • edited Loading

Gitea Version

Git Version

Operating System

How are you running Gitea?

Database

Can you reproduce the bug on the Gitea demo site?

Description

lunny commented Feb 7, 2022

lunny commented Feb 7, 2022

nicorac commented Feb 7, 2022 • edited Loading

zeripath commented Feb 8, 2022

wxiaoguang commented Feb 8, 2022

nicorac commented Feb 8, 2022

zeripath commented Feb 8, 2022

nicorac commented Feb 9, 2022

lunny commented Feb 9, 2022

nicorac commented Feb 9, 2022

wxiaoguang commented Feb 10, 2022 • edited Loading

nicorac commented Feb 10, 2022

lunny commented Feb 10, 2022

nicorac commented Feb 10, 2022 • edited Loading

wxiaoguang commented Feb 10, 2022

lunny commented Feb 10, 2022

wxiaoguang commented Feb 10, 2022

nicorac commented Feb 10, 2022

wxiaoguang commented Feb 10, 2022

6543 commented May 7, 2022

singuliere commented May 8, 2022

singuliere commented May 8, 2022

wxiaoguang commented May 8, 2022 • edited Loading

singuliere commented May 8, 2022

wxiaoguang commented May 8, 2022

singuliere commented May 8, 2022

wxiaoguang commented May 8, 2022

singuliere commented May 8, 2022

singuliere commented May 8, 2022 • edited Loading

nicorac commented Feb 7, 2022 •

edited

Loading

nicorac commented Feb 7, 2022 •

edited

Loading

wxiaoguang commented Feb 10, 2022 •

edited

Loading

nicorac commented Feb 10, 2022 •

edited

Loading

wxiaoguang commented May 8, 2022 •

edited

Loading

singuliere commented May 8, 2022 •

edited

Loading