SQLite Error 5: 'database is locked'.

[PhyxionNL]

Which Umbraco version are you using? (Please write the exact version, example: 10.1.0)

11.x - 17.x

Bug summary

Happens in all releases.

Specifics

No response

Steps to reproduce

1. Open a document type and make a change.
2. Save.
3. Go to 1.

Eventually the entire backend will hang and after a minute or so the following exception is logged:

Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 5: 'database is locked'.
   at Microsoft.Data.Sqlite.SqliteException.ThrowExceptionForRC(Int32 rc, sqlite3 db)
   at Microsoft.Data.Sqlite.SqliteDataReader.NextResult()
   at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader(CommandBehavior behavior)
   at StackExchange.Profiling.Data.ProfiledDbCommand.ExecuteNonQuery() in C:\projects\dotnet\src\MiniProfiler.Shared\Data\ProfiledDbCommand.cs:line 300
   at Umbraco.Cms.Infrastructure.Persistence.FaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func)
   at Umbraco.Cms.Infrastructure.Persistence.FaultHandling.FaultHandlingDbCommand.Execute[T](Func`1 f)
   at Umbraco.Cms.Infrastructure.Persistence.FaultHandling.FaultHandlingDbCommand.ExecuteNonQuery()
   at NPoco.Database.<>c__DisplayClass296_0.<ExecuteNonQueryHelper>b__0()
   at NPoco.Database.ExecutionHook[T](Func`1 action)
   at NPoco.Database.ExecuteNonQueryHelper(DbCommand cmd)
   at NPoco.Database.Execute(String sql, CommandType commandType, Object[] args)
   at NPoco.Database.Execute(Sql Sql)
   at Umbraco.Cms.Infrastructure.Persistence.Repositories.Implement.CacheInstructionRepository.DeleteInstructionsOlderThan(DateTime pruneDate)
   at Umbraco.Cms.Infrastructure.Services.CacheInstructionService.PruneOldInstructions()
   at Umbraco.Cms.Infrastructure.Services.CacheInstructionService.ProcessInstructions(CacheRefresherCollection cacheRefreshers, ServerRole serverRole, CancellationToken cancellationToken, String localIdentity, DateTime lastPruned, Int32 lastId)
   at Umbraco.Cms.Infrastructure.Sync.DatabaseServerMessenger.Sync()
   at Umbraco.Cms.Infrastructure.HostedServices.ServerRegistration.InstructionProcessTask.PerformExecuteAsync(Object state)

Expected result / actual result

No response

[github-actions]

Hi there @PhyxionNL!

Firstly, a big thank you for raising this issue. Every piece of feedback we receive helps us to make Umbraco better.

We really appreciate your patience while we wait for our team to have a look at this but we wanted to let you know that we see this and share with you the plan for what comes next.

• We'll assess whether this issue relates to something that has already been fixed in a later version of the release that it has been raised for.
• If it's a bug, is it related to a release that we are actively supporting or is it related to a release that's in the end-of-life or security-only phase?
• We'll replicate the issue to ensure that the problem is as described.
• We'll decide whether the behavior is an issue or if the behavior is intended.

We wish we could work with everyone directly and assess your issue immediately but we're in the fortunate position of having lots of contributions to work with and only a few humans who are able to do it. We are making progress though and in the meantime, we will keep you in the loop and let you know when we have any questions.

Thanks, from your friendly Umbraco GitHub bot 🤖 🙂

[kjac]

Hi @PhyxionNL,

That sounds super annoying - not to mention really weird. I haven't seen that kind of behaviour at all on any version of Umbraco, nor can I reproduce it.

Could it be that some custom code is executing and causing this error?

If you can reproduce this on a blank install, please get back to us with your system info; you can copy it from the help drawer:

[PhyxionNL]

There's no custom code in the project that should cause this problem. I can also reproduce this on a fresh project. Just add one Test document and edit a data type.

To reproduce this more quickly:

1. Open a new tab in Firefox and open a data type in backend, e.g. List View Content.
2. Open dev tools in Firefox.
3. Click Save (the data type).
4. Go to network tab in dev tools, right-click on the PostSave request, Copy Value > Copy as Fetch.
5. Open console, paste this, and press enter to execute it:

(async function() {
while (true) {
 // Paste the await fetch method you copied in 4 here.
})();

5. Repeat 1-5.
6. Wait and watch it hang after some time. To speed this up even faster repeat 1-5 again.

It usually freezes here within 15-60s. This is the easiest way to reproduce it for me. I can also do this manually but it usually takes more time. I am aware that the above code isn't how you normally use Umbraco, but this also happens randomly when you normally use it. The error is the same though.

Result:

[15:00:06 ERR] Exception (957b6565).
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 5: 'database is locked'.
   at Microsoft.Data.Sqlite.SqliteException.ThrowExceptionForRC(Int32 rc, sqlite3 db)
   at Microsoft.Data.Sqlite.SqliteDataReader.NextResult()
   at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader(CommandBehavior behavior)
   at StackExchange.Profiling.Data.ProfiledDbCommand.ExecuteNonQuery() in C:\projects\dotnet\src\MiniProfiler.Shared\Data\ProfiledDbCommand.cs:line 300
   at Umbraco.Cms.Infrastructure.Persistence.FaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func)
   at NPoco.Database.ExecuteNonQueryHelper(DbCommand cmd)
   at NPoco.Database.Execute(String sql, CommandType commandType, Object[] args)
[15:00:06 INF] Executing ContentResult with HTTP Response ContentType of application/json
[15:00:06 INF] Executed action Umbraco.Cms.Web.BackOffice.Controllers.DataTypeController.PostSave (Umbraco.Web.BackOffice) in 582887.0808ms
[15:00:06 INF] Executed endpoint 'Umbraco.Cms.Web.BackOffice.Controllers.DataTypeController.PostSave (Umbraco.Web.BackOffice)'
[15:00:06 INF] Request finished HTTP/2 POST https://localhost/umbraco/backoffice/umbracoapi/datatype/PostSave application/json;charset=utf-8 1088 - 500 - application/json 582888.4010ms
[15:00:07 INF] Request starting HTTP/2 GET https://localhost/umbraco/backoffice/umbracoapi/authentication/GetRemainingTimeoutSeconds - -
[15:00:07 INF] Executing endpoint 'Umbraco.Cms.Web.BackOffice.Controllers.AuthenticationController.GetRemainingTimeoutSeconds (Umbraco.Web.BackOffice)'
[15:00:07 INF] Route matched with {area = "UmbracoApi", action = "GetRemainingTimeoutSeconds", controller = "Authentication"}. Executing controller action with signature System.Threading.Tasks.Task`1[System.Double] GetRemainingTimeoutSeconds() on controller Umbraco.Cms.Web.BackOffice.Controllers.AuthenticationController (Umbraco.Web.BackOffice).
[15:00:07 INF] Executing action method Umbraco.Cms.Web.BackOffice.Controllers.AuthenticationController.GetRemainingTimeoutSeconds (Umbraco.Web.BackOffice) - Validation state: Valid
[15:00:07 INF] Executed action method Umbraco.Cms.Web.BackOffice.Controllers.AuthenticationController.GetRemainingTimeoutSeconds (Umbraco.Web.BackOffice), returned result Microsoft.AspNetCore.Mvc.ObjectResult in 0.1268ms.
[15:00:07 INF] Executing ObjectResult, writing value of type 'System.Double'.
[15:00:07 INF] Executed action Umbraco.Cms.Web.BackOffice.Controllers.AuthenticationController.GetRemainingTimeoutSeconds (Umbraco.Web.BackOffice) in 13.219ms
[15:00:07 INF] Executed endpoint 'Umbraco.Cms.Web.BackOffice.Controllers.AuthenticationController.GetRemainingTimeoutSeconds (Umbraco.Web.BackOffice)'
[15:00:07 INF] Request finished HTTP/2 GET https://localhost/umbraco/backoffice/umbracoapi/authentication/GetRemainingTimeoutSeconds - - - 200 - application/json;+charset=utf-8 13.9368ms
[15:00:22 ERR] Exception (8b7e58f9).
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 5: 'database is locked'.
   at Microsoft.Data.Sqlite.SqliteException.ThrowExceptionForRC(Int32 rc, sqlite3 db)
   at Microsoft.Data.Sqlite.SqliteDataReader.NextResult()
   at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader(CommandBehavior behavior)
   at StackExchange.Profiling.Data.ProfiledDbCommand.ExecuteNonQuery() in C:\projects\dotnet\src\MiniProfiler.Shared\Data\ProfiledDbCommand.cs:line 300
   at Umbraco.Cms.Infrastructure.Persistence.FaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func)
   at NPoco.Database.ExecuteNonQueryHelper(DbCommand cmd)
   at NPoco.Database.Execute(String sql, CommandType commandType, Object[] args)
   at Umbraco.Cms.Infrastructure.Persistence.Repositories.Implement.CacheInstructionRepository.DeleteInstructionsOlderThan(DateTime pruneDate)
   at Umbraco.Cms.Infrastructure.Services.CacheInstructionService.ProcessInstructions(CacheRefresherCollection cacheRefreshers, ServerRole serverRole, CancellationToken cancellationToken, String localIdentity, DateTime lastPruned, Int32 lastId)
   at Umbraco.Cms.Infrastructure.Sync.DatabaseServerMessenger.Sync()
   at Umbraco.Cms.Infrastructure.HostedServices.ServerRegistration.InstructionProcessTask.PerformExecuteAsync(Object state)
[15:00:22 ERR] Failed (will repeat).
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 5: 'database is locked'.
   at Microsoft.Data.Sqlite.SqliteException.ThrowExceptionForRC(Int32 rc, sqlite3 db)
   at Microsoft.Data.Sqlite.SqliteDataReader.NextResult()
   at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader(CommandBehavior behavior)
   at StackExchange.Profiling.Data.ProfiledDbCommand.ExecuteNonQuery() in C:\projects\dotnet\src\MiniProfiler.Shared\Data\ProfiledDbCommand.cs:line 300
   at Umbraco.Cms.Infrastructure.Persistence.FaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func)
   at NPoco.Database.ExecuteNonQueryHelper(DbCommand cmd)
   at NPoco.Database.Execute(String sql, CommandType commandType, Object[] args)
   at Umbraco.Cms.Infrastructure.Persistence.Repositories.Implement.CacheInstructionRepository.DeleteInstructionsOlderThan(DateTime pruneDate)
   at Umbraco.Cms.Infrastructure.Services.CacheInstructionService.ProcessInstructions(CacheRefresherCollection cacheRefreshers, ServerRole serverRole, CancellationToken cancellationToken, String localIdentity, DateTime lastPruned, Int32 lastId)
   at Umbraco.Cms.Infrastructure.Sync.DatabaseServerMessenger.Sync()
   at Umbraco.Cms.Infrastructure.HostedServices.ServerRegistration.InstructionProcessTask.PerformExecuteAsync(Object state)

Also set the following in config so this reproduces more quickly:

"DatabaseServerMessenger": {"TimeBetweenPruneOperations": "00:00:00", "TimeBetweenSyncOperations": "00:00:00"}

You don't have to load the frontend, the backend is enough.

Category	Data
Server OS	Microsoft Windows 10.0.19045
Server Framework	.NET 7.0.5
Default Language	en-US
Umbraco Version	11.3.0
Current Culture	en-US
Current UI Culture	en-US
Current Webserver	Kestrel
Models Builder Mode	SourceCodeAuto
Runtime Mode	BackofficeDevelopment
Debug Mode	False
Database Provider	System.Data.SQLite
Current Server Role	Single
Browser	Firefox 112.0
Browser OS	Win32

[PhyxionNL]

Still happening in 13.2.2. I hoped the locking improvements from #15854 would've fixed it, but no.

Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 5: 'database is locked'.
   at Microsoft.Data.Sqlite.SqliteException.ThrowExceptionForRC(Int32 rc, sqlite3 db)
   at Microsoft.Data.Sqlite.SqliteDataReader.NextResult()
   at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader(CommandBehavior behavior)
   at StackExchange.Profiling.Data.ProfiledDbCommand.ExecuteNonQuery() in C:\projects\dotnet\src\MiniProfiler.Shared\Data\ProfiledDbCommand.cs:line 261
   at Umbraco.Cms.Infrastructure.Persistence.FaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func)
   at NPoco.Database.ExecuteNonQueryHelper(DbCommand cmd)
   at NPoco.Database.Execute(String sql, CommandType commandType, Object[] args)
   at NPoco.Database.UpdateImpAsync(String tableName, String primaryKeyName, Object poco, Object primaryKeyValue, IEnumerable`1 columns, Boolean sync)
   at NPoco.AsyncHelper.RunSync[T](Task`1 task)
   at NPoco.Database.Update(Object poco)
   at Umbraco.Cms.Infrastructure.Persistence.Repositories.Implement.DataTypeRepository.PersistUpdatedItem(IDataType entity)
   at Umbraco.Cms.Core.Cache.DefaultRepositoryCachePolicy`2.Update(TEntity entity, Action`1 persistUpdated)
   at Umbraco.Cms.Core.Services.Implement.DataTypeService.Save(IDataType dataType, Int32 userId)
   at Umbraco.Cms.Web.BackOffice.Controllers.DataTypeController.PostSave(DataTypeSave dataType)
   at lambda_method14009(Closure, Object, Object[])
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Logged|12_1(ControllerActionInvoker invoker)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextExceptionFilterAsync>g__Awaited|26_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)

[nymwo]

Check if the database is in WAL mode:

sqlite3 "umbraco\Data\Umbraco.sqlite.db" "PRAGMA journal_mode"

should return "wal".

I've also noticed that Umbraco now recommends using "Private" instead of "Shared" cache: https://docs.umbraco.com/umbraco-cms/13.latest/reference/configuration/connectionstringssettings

Using WAL mode and private cache fixed similar issues for me.

[PhyxionNL]

Check if the database is in WAL mode:

sqlite3 "umbraco\Data\Umbraco.sqlite.db" "PRAGMA journal_mode"

should return "wal".

I've also noticed that Umbraco now recommends using "Private" instead of "Shared" cache: https://docs.umbraco.com/umbraco-cms/13.latest/reference/configuration/connectionstringssettings

Using WAL mode and private cache fixed similar issues for me.

Doesn't fix it for me (and these settings were already used).

[nymwo]

Yeah, I'm also still getting this error sporadically on the projects I work on where we use SQLite. Seems strange that the only error I'm getting is from Umbraco.Cms.Infrastructure.Persistence.Repositories.Implement.CacheInstructionRepository.DeleteInstructionsOlderThan. Does that method do something different from how Umbraco usually handles database operations?