Test-sqlite sometimes run time out on CI.

[lunny]

@typeless

[typeless]

It could be probably yet another deadlock like those we encountered before. It's easy to spot the culprit by sending the ABRT signal to the hanging process. But I am not sure how to do it in the CI pipeline.

I have not yet encountered it in my local machine.

Edit: otherwise, I have to come up with a debugging strategy like a watchdog to automatically detect the hanging.

[typeless]

From the easiest to the hardest effort (presuming that the hanging is caused by DB deadlock):

1, fixes the hanging case by case as we encounter it by collecting logs.
2. Give a default timeout for the SQLite dialect in xorm. go-sqlite3 does have BeginTx which can pass a context.
3. Expose BeginTx for the xorm sessions API, and let the application specify the deadline context of each transaction.
4. Modify macaron or use another routing package which supports the context.Context-like deadline.

The 'watchdog' approach seems to be not as trivial as I thought since it's not obvious what the watchdog should look at.

But at first, we should try to reproduce it again.

[typeless]

This is the backtrace I found suspicious in https://drone.gitea.io/go-gitea/gitea/3612/10.

goroutine 10 [chan receive, 2 minutes, locked to thread]:
code.gitea.io/gitea/vendor/github.com/mattn/go-sqlite3.unlock_notify_wait(0x7f3d40002a68, 0x0)
	/go/src/code.gitea.io/gitea/vendor/github.com/mattn/go-sqlite3/sqlite3_opt_unlock_notify.go:90 +0x158
code.gitea.io/gitea/vendor/github.com/mattn/go-sqlite3._cgoexpwrap_2dae2c0674b5_unlock_notify_wait(0x7f3d40002a68, 0x7f3d40002a68)
	_cgo_gotypes.go:1371 +0x2b
code.gitea.io/gitea/vendor/github.com/mattn/go-sqlite3._Cfunc__sqlite3_step_internal(0x7f3d40037d48, 0x7f3d00000000)
	_cgo_gotypes.go:352 +0x49
code.gitea.io/gitea/vendor/github.com/mattn/go-sqlite3.(*SQLiteRows).Next.func1(0x7f3d40037d48, 0x134b268)
	/go/src/code.gitea.io/gitea/vendor/github.com/mattn/go-sqlite3/sqlite3.go:1983 +0x56
code.gitea.io/gitea/vendor/github.com/mattn/go-sqlite3.(*SQLiteRows).Next(0xc0031700a0, 0xc003170140, 0x5, 0x5, 0x0, 0x0)
	/go/src/code.gitea.io/gitea/vendor/github.com/mattn/go-sqlite3/sqlite3.go:1983 +0xce
database/sql.(*Rows).nextLocked(0xc001f02180, 0xc001bf0000)
	/usr/local/go/src/database/sql/sql.go:2663 +0xc1
database/sql.(*Rows).Next.func1()
	/usr/local/go/src/database/sql/sql.go:2641 +0x3c
database/sql.withLock(0x146b160, 0xc001f021b0, 0xc0022bb928)
	/usr/local/go/src/database/sql/sql.go:3074 +0x63
database/sql.(*Rows).Next(0xc001f02180, 0xc0022bb9a8)
	/usr/local/go/src/database/sql/sql.go:2640 +0x87
code.gitea.io/gitea/vendor/github.com/go-xorm/xorm.(*Session).rows2Beans(0xc00259c000, 0xc003887710, 0xc0031700f0, 0x5, 0x5, 0xc000682700, 0xc0025a2660, 0xc00259a690, 0x0, 0xc0022bbac0)
	/go/src/code.gitea.io/gitea/vendor/github.com/go-xorm/xorm/session.go:305 +0x56
code.gitea.io/gitea/vendor/github.com/go-xorm/xorm.(*Session).noCacheFind(0xc00259c000, 0xc000682700, 0x111d0e0, 0xc0025a2400, 0x197, 0xc00e3e40c0, 0x5d, 0xc003887570, 0x1, 0x1, ...)
	/go/src/code.gitea.io/gitea/vendor/github.com/go-xorm/xorm/session_find.go:275 +0x6e6
code.gitea.io/gitea/vendor/github.com/go-xorm/xorm.(*Session).find(0xc00259c000, 0x10f7320, 0xc0025a2400, 0x0, 0x0, 0x0, 0x1, 0xc00259c000)
	/go/src/code.gitea.io/gitea/vendor/github.com/go-xorm/xorm/session_find.go:191 +0x5a8
code.gitea.io/gitea/vendor/github.com/go-xorm/xorm.(*Session).Find(0xc00259c000, 0x10f7320, 0xc0025a2400, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/code.gitea.io/gitea/vendor/github.com/go-xorm/xorm/session_find.go:29 +0x7a
code.gitea.io/gitea/models.getUnitsByRepoID(0x147a160, 0xc0000a6000, 0x1, 0xc0022bbdf8, 0x4172cf, 0x203000, 0x203000, 0x30)
	/go/src/code.gitea.io/gitea/models/repo_unit.go:170 +0x10d
code.gitea.io/gitea/models.(*Repository).getUnits(0xc0024d5380, 0x147a160, 0xc0000a6000, 0x4172cf, 0xc003170050)
	/go/src/code.gitea.io/gitea/models/repo.go:318 +0x50
code.gitea.io/gitea/models.(*Repository).getUnitsByUserID(0xc0024d5380, 0x147a160, 0xc0000a6000, 0x1, 0x125c400, 0x1, 0xc003170050)
	/go/src/code.gitea.io/gitea/models/repo.go:346 +0x68
code.gitea.io/gitea/models.(*Repository).CheckUnitUser(0xc0024d5380, 0x1, 0xc002667900, 0x3, 0x0)
	/go/src/code.gitea.io/gitea/models/repo.go:324 +0x5a
code.gitea.io/gitea/models.createOrUpdateIssueNotifications(0x147a200, 0xc002667900, 0xc00216a840, 0x1, 0xc0034e9fb8, 0x1)
	/go/src/code.gitea.io/gitea/models/notification.go:126 +0x419
code.gitea.io/gitea/models.CreateOrUpdateIssueNotifications(0xc00216a840, 0x1, 0x0, 0x0)
	/go/src/code.gitea.io/gitea/models/notification.go:68 +0xb7
code.gitea.io/gitea/modules/notification/ui.(*notificationService).Run(0xc00000e1b0)
	/go/src/code.gitea.io/gitea/modules/notification/ui/ui.go:40 +0x65
created by code.gitea.io/gitea/modules/notification.RegisterNotifier
	/go/src/code.gitea.io/gitea/modules/notification/notification.go:20 +0x4a

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions.

[lunny]

It seems this has been resolved. Please feel free to reopen it.