Orchestration getting stuck while getting the lock

[vany0114]

Description

The full description with code snippets, screenshots, issue samples, etc is here #2530

Expected behavior

Acquire the lock in seconds at most, not minutes or hours.

Actual behavior

It seems the orchestration is getting stuck while acquiring the locks of the entities intervening in the orchestration.

Known workarounds

Reset the durable storage account and the func app storage account

App Details

• Durable Functions extension version (e.g. v1.8.3): 2.10.0
• Azure Functions runtime version (1.0 or 2.0): 2
• Programming language used: C#

If deployed to Azure

• Timeframe issue observed:
• Function App name: orders-saga
• Azure region: West-US
• Azure storage account name: ordersagav2

[davidmrdavid]

@vany0114: while we start this investigation, can you share with us when this issue first started (rough timestamp in UTC)? Does it correlate with any code changes (i.e partial classes, or trying out the new partition manager)?

[vany0114]

@davidmrdavid It seems the issue started around 2023-08-07 13:34 UTC

[davidmrdavid]

Thanks! Any known code/config changes around that time? @vany0114?

[vany0114]

@davidmrdavid Nope, we haven't touched anything since the last deployment on August 2nd which btw that was the last time the issue showed up and we had to reset everything.

[vany0114]

@davidmrdavid BTW I'll re-deploy the func app today to try to mitigate this issue, but this time it will use Netherite as the storage provider, meaning that the old ordersagav2 storage account is still there for troubleshooting however the func app's storage account will be reset, would that be a problem for you guys while you look into?

[davidmrdavid]

@vany0114: that shouldn't be an issue on our end as our historical telemetry is stored independently of our app's live status, so you're good to go ahead :).

I'm not certain at this point in time that this is an issue with the storage provider itself, so I can't comment on whether changing to Netherite will help, but it probably won't hurt to try it out and may tell us more about the problem. Just an FYI.

[vany0114]

Agree I'm not sure either, just wanna try something different this time, and as you said it may tell us more about the problem.

[vany0114]

@davidmrdavid BTW if you guys can also take a look at this microsoft/durabletask-netherite#294 would be great since that's related to the Netherite implementation.

[davidmrdavid]

A few small updates:
(1) I don't see your app using the new table partition manager. I'm only calling this out because you mentioned this new feature in the discussion thread. Just to confirm: is this expected or did you take an action to try and use the new partition manager? Just triple checking the expectation.

(2) I'm not immediately spotting a partition manager-type issue. In other words, the queues that are hosting your orchestrator and entity messages seem to be receiving new messages just fine. So I'm starting to think the issue here might have to do with the entity message processing, not with receiving messages. Still need to validate this.

[davidmrdavid]

Actually, I'm seeing abnormally high message ages in "control queue 3". Let me drill into that. If that queue is hosting any of the entities you're trying to lock, maybe that's to blame.

[vany0114]

(1) I don't see your app using the new table partition manager. I'm only calling this out because you mentioned this new feature in the discussion thread. Just to confirm: is this expected or did you take an action to try and use the new partition manager? Just triple checking the expectation.

Yeah I rolled back that change because I thought that was the culprit, but it wasn't, sorry I forgot to mention that.

[davidmrdavid]

Actually, I'm seeing abnormally high message ages in "control queue 3". Let me drill into that. If that queue is hosting any of the entities you're trying to lock, maybe that's to blame.

I'm not able to draw a correlation from this observation to the symptoms you experienced, so let's ignore this for now.

Also, I've been focusing on this set of data, from the other thread:

{"storeId":"615c29df-5e7c-458c-9486-ef9480d2bf3b","courseUid":"c77bb860-04f7-46d3-86d9-d8c0e65fe856","checkNumber":"90007"} same, the instance was stuck 21 minutes:

* @ticketnumberkey@615c29df-5e7c-458c-9486-ef9480d2bf3b_2023-08-07_90007-1
* @originorderkey@615c29df-5e7c-458c-9486-ef9480d2bf3b_2023-08-07_90007-1
* @scheduledkey@615c29df-5e7c-458c-9486-ef9480d2bf3b_2023-08-07_90007-1
* @order@{"storeId":"615c29df-5e7c-458c-9486-ef9480d2bf3b","courseUid":"c77bb860-04f7-46d3-86d9-d8c0e65fe856","checkNumber":"90007"}

So that orchestrator is stuck from about ~11:33 to ~11:54, and I see in my telemetry that EntityId @scheduledkey@615c29df-5e7c-458c-9486-ef9480d2bf3b_2023-08-07_90007-1 is the last Entity to respond to that lock request, precisely at about ~11:54, whereas others responded around ~11:33. So the question became: why did it take us so long to acquire the lock on this final entity?

Well, I see that entity is managed by control queue 01. According to our telemetry, control queue 01 was being listened to "VM 19" from ~13:15 to ~13:31. Then, no VM is actively listening to new messages from that queue until ~13:54, when "VM 20" takes over.

To summarize, it appears that no new messages are being processed from that queue from ~13:31 to ~13:54.
So now what I need to answer is the following:
(1) Why does VM19 stop listening to this queue in the first place? Normally, this is because that VM is in the process of shutting down, another worker may steal ownership of its assigned queues, and therefore the VM will stop listening to new messages.
(2) Why does it take so long for VM20 to start listening to queue 01? My guess is that VM 19 took too long to restart, which prevented VM20 from being able to listen to that queue (only one VM can listen to a single queue at a time, otherwise we risk "split brain" issues).

That's what I've been able to find so far. Assuming I understand the problem correctly, this means, to me, that simply restarting your app should suffice to mitigate this situation when using the Azure Storage backend.

I'll continue looking into the logs to answer the (2) open questions above, I'll keep you posted.

[vany0114]

@davidmrdavid I much appreciate the update 🙏

[vany0114]

[...] this means, to me, that simply restarting your app should suffice to mitigate this situation when using the Azure Storage backend.

Unfortunately, it didn't work, I'm gonna reset it again.