App without role assignment to storage silently fails

[mattchenderson]

We used to present storage access errors better, but it seems like at some point there was a regression around this, and it's fairly easy to enter a state where the app is broken but no error information is returned. All you have to do is mess up an identity-based connection for AzureWebJobsStorage, specifically in the creation path, and by ensuring that the app does not have the right permissions.

Investigative information

• Function App version: 4.1038, 4.838

Repro steps

The easiest way to reproduce this is with a current UX experiment for creating a function app with identity-based connections, and providing it an existing user-assigned managed identity that does not already have permissions over the target storage account scope. For team members, you should already see this on the internal portal - please engage me if not.

But in general:

1. Create a .NET 9 app on the isolated worker model, using Windows Consumption, and with an identity-based AzureWebJobsStorage connection against an identity which doesn't have any permissions over the storage account.
2. Confirm that no containers have been created in storage.
3. Check the overview tab in the portal.
4. Observe no errors.
5. Check logs.
6. Observe no errors.

Expected behavior

I would like to see some sort of error captured and available to the user. In the setup described above, host storage interactions will get a 403 returned. This used to be the case. It seems like there was a regression at some point.

Actual behavior

No errors are returned from any of the portal overview calls. You can get an error if you do anything that would, for example, deal with keys, but this will be an opaque 500 without an explanation. No error information is logged. Not in Application Insights. Not in filesystem logs.

The Application Insights behavior seems consistent regardless of if I've set that up with identity (failing similarly) or not.

Interestingly, if you do the same thing for Logic Apps Standard, an error IS at least surfaced in the portal. This seems to be rooted in their extension doing some storage operations at startup. You would see (as a banner in the portal) an error from the Azure SDK like:

Azure.Storage.Blobs: This request is not authorized to perform this operation using this permission. RequestId:4fe11ec3-301e-004e-7739-a08735000000 Time:2025-03-28T23:33:05.9156624Z Status: 403 (This request is not authorized to perform this operation using this permission.) ErrorCode: AuthorizationPermissionMismatch Content: AuthorizationPermissionMismatchThis request is not authorized to perform this operation using this permission. RequestId:4fe11ec3-301e-004e-7739-a08735000000 Time:2025-03-28T23:33:05.9156624Z Headers: Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0 x-ms-request-id: 4fe11ec3-301e-004e-7739-a08735000000 x-ms-client-request-id: 526807d7-d2e7-44ce-b023-00243326927f x-ms-version: 2023-11-03 x-ms-error-code: AuthorizationPermissionMismatch Date: Fri, 28 Mar 2025 23:33:05 GMT Content-Length: 279 Content-Type: application/xml .

Known workarounds

Creating the necessary role assignments as expected addresses the error state, but the concern here is if I don't know that this is the issue in the first place. Nothing about the system tells me that it's necessary action.

[jviau]

@mattchenderson can you clarify if there were errors in Application Insights or not?

[fabiocav]

This should be addressed by the health check work. Moving to a future sprint for validation.

[AartBluestoke]

having done this recently, the raw low level log stream shows an azure blob storage sdk permission denied error, but that is not a particularly helpful error path, as it doesn't show what the operation was, or what the account it was operating on - you can look at the stack trace to get a few hints as to where on the startup path it triggered.

In general storage sdk exceptions are poorly communicated.

[jviau]

The upcoming host version includes a health check for AzureWebJobsStorage. Any issue communicating with it will result in an unhealthy check along with the error code from Azure SDK reported in the check.