Skip to content

[BUG] PersistKeysToAzureBlobStorage: 'Unable to retrieve the decryption key.' #38586

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
peterbomers opened this issue Sep 8, 2023 · 13 comments
Closed

[BUG] PersistKeysToAzureBlobStorage: 'Unable to retrieve the decryption key.' #38586

peterbomers opened this issue Sep 8, 2023 · 13 comments
Assignees
@JoshLove-msft
Labels
Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. Extensions ASP.NET Core extensions needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team question The issue doesn't require a change to the product in order to be resolved. Most issues start as that

Comments

@peterbomers
Copy link

peterbomers commented Sep 8, 2023 •
edited
Loading

Library name and version

Azure.Extensions.AspNetCore.DataProtection.Blobs 1.3.2

Describe the bug

Re-opening #32592 because the issue still occurs.
(it was closed during holiday season)

The example of the corrupted keys file is still valid.
Especially the creation, activation and expiration dates are strange. Below the last 3 entries. (total file size = 67kb)

<?xml version="1.0" encoding="utf-8"?>
<repository>
    <key id="d7720961-removed" version="1">
        <creationDate>2022-07-26T15:58:04.9292928Z</creationDate>
        <activationDate>2022-07-28T15:34:11.4371143Z</activationDate>
        <expirationDate>2022-10-24T15:58:04.907673Z</expirationDate>
        <descriptor deserializerType="Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60">
            <descriptor>
                <encryption algorithm="AES_256_CBC" />
                <validation algorithm="HMACSHA256" />
                <encryptedSecret decryptorType="Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60"
                    xmlns="http://schemas.asp.net/2015/03/dataProtection">
                    <EncryptedData Type="http://www.w3.org/2001/04/xmlenc#Element"
                        xmlns="http://www.w3.org/2001/04/xmlenc#">
                        <EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#aes256-cbc" />
                        <KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#">
                            <EncryptedKey xmlns="http://www.w3.org/2001/04/xmlenc#">
                                <EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#rsa-1_5" />
                                <KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#">
                                    <X509Data>
                                        <X509Certificate>removed</X509Certificate>
                                    </X509Data>
                                </KeyInfo>
                                <CipherData>
                                    <CipherValue>removed</CipherValue>
                                </CipherData>
                            </EncryptedKey>
                        </KeyInfo>
                        <CipherData>
                            <CipherValue>removed</CipherValue>
                        </CipherData>
                    </EncryptedData>
                </encryptedSecret>
            </descriptor>
        </descriptor>
    </key>
    <key id="a04a75a1-removed" version="1">
        <creationDate>2022-10-23T03:32:59.4371134Z</creationDate>
        <activationDate>2022-10-24T15:58:04.907673Z</activationDate>
        <expirationDate>2023-01-21T03:32:59.3917697Z</expirationDate>
        <descriptor deserializerType="Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60">
            <descriptor>
                <encryption algorithm="AES_256_CBC" />
                <validation algorithm="HMACSHA256" />
                <encryptedSecret decryptorType="Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60"
                    xmlns="http://schemas.asp.net/2015/03/dataProtection">
                    <EncryptedData Type="http://www.w3.org/2001/04/xmlenc#Element"
                        xmlns="http://www.w3.org/2001/04/xmlenc#">
                        <EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#aes256-cbc" />
                        <KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#">
                            <EncryptedKey xmlns="http://www.w3.org/2001/04/xmlenc#">
                                <EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#rsa-1_5" />
                                <KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#">
                                    <X509Data>
                                        <X509Certificate>removed</X509Certificate>
                                    </X509Data>
                                </KeyInfo>
                                <CipherData>
                                    <CipherValue>removed</CipherValue>
                                </CipherData>
                            </EncryptedKey>
                        </KeyInfo>
                        <CipherData>
                            <CipherValue>removed</CipherValue>
                        </CipherData>
                    </EncryptedData>
                </encryptedSecret>
            </descriptor>
        </descriptor>
    </key>
    <key id="2de0efd7-removed" version="1">
        <creationDate>2022-11-17T09:35:54.6857467Z</creationDate>
        <activationDate>2022-11-17T09:35:54.3656624Z</activationDate>
        <expirationDate>2023-02-15T09:35:54.3656624Z</expirationDate>
        <descriptor deserializerType="Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60">
            <descriptor>
                <encryption algorithm="AES_256_CBC" />
                <validation algorithm="HMACSHA256" />
                <encryptedSecret decryptorType="Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60"
                    xmlns="http://schemas.asp.net/2015/03/dataProtection">
                    <EncryptedData Type="http://www.w3.org/2001/04/xmlenc#Element"
                        xmlns="http://www.w3.org/2001/04/xmlenc#">
                        <EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#aes256-cbc" />
                        <KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#">
                            <EncryptedKey xmlns="http://www.w3.org/2001/04/xmlenc#">
                                <EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#rsa-1_5" />
                                <KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#">
                                    <X509Data>
                                        <X509Certificate>removed</X509Certificate>
                                    </X509Data>
                                </KeyInfo>
                                <CipherData>
                                    <CipherValue>removed</CipherValue>
                                </CipherData>
                            </EncryptedKey>
                        </KeyInfo>
                        <CipherData>
                            <CipherValue>removed</CipherValue>
                        </CipherData>
                    </EncryptedData>
                </encryptedSecret>
            </descriptor>
        </descriptor>
    </key>
</repository>

Our demand on the Data Protection feature is high and I suspect that the AzureBlobXmlRepository can't handle rotation when the blobstorage produces client errors. (ClientOtherError)

This is a stack trace when the decrypting of an antiforgery token fails.
Other data-protection parts suffer from the same issue.

Details  
Message The antiforgery token could not be decrypted. Unable to retrieve the decryption key.  
Exception type System.Security.Cryptography.CryptographicException  
Failed method System.Security.Cryptography.Xml.EncryptedXml.GetDecryptionKey  
Problem Id System.Security.Cryptography.CryptographicException at System.Security.Cryptography.Xml.EncryptedXml.GetDecryptionKey  
Assembly System.Security.Cryptography.Xml, Version=6.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51  
Innermost assembly System.Security.Cryptography.Xml, Version=6.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51
Innermost Method System.Security.Cryptography.Xml.EncryptedXml.GetDecryptionKey 
Microsoft.AspNetCore.Antiforgery.AntiforgeryValidationException:
   at Microsoft.AspNetCore.Antiforgery.DefaultAntiforgeryTokenSerializer.Deserialize (Microsoft.AspNetCore.Antiforgery, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.Antiforgery.DefaultAntiforgery.GetCookieTokenDoesNotThrow (Microsoft.AspNetCore.Antiforgery, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
Inner exception System.Security.Cryptography.CryptographicException handled at Microsoft.AspNetCore.Antiforgery.DefaultAntiforgeryTokenSerializer.Deserialize:
   at System.Security.Cryptography.Xml.EncryptedXml.GetDecryptionKey (System.Security.Cryptography.Xml, Version=6.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
   at System.Security.Cryptography.Xml.EncryptedXml.DecryptDocument (System.Security.Cryptography.Xml, Version=6.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor.Decrypt (Microsoft.AspNetCore.DataProtection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.XmlEncryptionExtensions.DecryptElement (Microsoft.AspNetCore.DataProtection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager.Microsoft.AspNetCore.DataProtection.KeyManagement.Internal.IInternalXmlKeyManager.DeserializeDescriptorFromKeyElement (Microsoft.AspNetCore.DataProtection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.DeferredKey+<>c__DisplayClass1_0.<GetLazyDescriptorDelegate>b__0 (Microsoft.AspNetCore.DataProtection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Lazy`1.ViaFactory (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Lazy`1.CreateValue (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.CngGcmAuthenticatedEncryptorFactory.CreateEncryptorInstance (Microsoft.AspNetCore.DataProtection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyBase.CreateEncryptor (Microsoft.AspNetCore.DataProtection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRing+KeyHolder.GetEncryptorInstance (Microsoft.AspNetCore.DataProtection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRing.GetAuthenticatedEncryptorByKeyId (Microsoft.AspNetCore.DataProtection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.UnprotectCore (Microsoft.AspNetCore.DataProtection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Unprotect (Microsoft.AspNetCore.DataProtection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.Antiforgery.DefaultAntiforgeryTokenSerializer.Deserialize (Microsoft.AspNetCore.Antiforgery, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)

Expected behavior

Rock solid key rotation with Azure Blob Storage as store.

Actual behavior

Corrupted keys.xml file as described above and data-loss due to missing decryption keys

Reproduction Steps

  • Setup a ASP.NET application with data-protection
services.AddDataProtection()
        .PersistKeysToAzureBlobStorage(options.BlobStorage, "data-protection", "keys.xml")
        .ProtectKeysWithCertificate(keyVault.GetCertificate("data-protection"));
  • Deploy mulitple instances
  • Generate high request demand on the deployed instances
  • Wait for the blob storage to generate client errors during key rotation
  • Inspect the corrupted keys.xml file

Environment

Azure App Service with .NET 6 runtime
@github-actions github-actions bot added customer-reported Issues that are reported by GitHub users external to the Azure organization. needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that labels Sep 8, 2023
@jsquire jsquire added Extensions ASP.NET Core extensions Client This issue points to a problem in the data-plane of the library. needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team and removed needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. labels Sep 8, 2023
@jsquire
Copy link
Member

jsquire commented Sep 8, 2023

Thank you for your feedback. Tagging and routing to the team member best able to assist.

@peterbomers
Copy link
Author

@jsquire is there any update on this issue?
Within a month our keys will rotate again and we're not happy with losing information again.

@JoshLove-msft
Copy link
Member

JoshLove-msft commented Oct 18, 2023 •
edited
Loading

@peterbomers, apologies for the delay. Can you clarify what in the keys.xml file is corrupted? You mentioned the dates seem wrong, but can you elaborate on why they are wrong?

You also mentioned that the blob storage client is producing errors. Do you have a stack trace for that?

@peterbomers
Copy link
Author

@peterbomers, apologies for the delay. Can you clarify what in the keys.xml file is corrupted? You mentioned the dates seem wrong, but can you elaborate on why they are wrong?

The creationDate of the last entry is after the activationDate.

You also mentioned that the blob storage client is producing errors. Do you have a stack trace for that?

The blob storage client isn't throwing exceptions. The ClientOtherError can be found in the Azure portal when visiting the monitoring page of the storage account.

@JoshLove-msft
Copy link
Member

The same issue is being reported here - dotnet/aspnetcore#33071

It doesn't appear to be specific to the blob data protection package, as the issue occurs when using Redis as well.

@JoshLove-msft
Copy link
Member

This comment seems to indicate that the issue can occur when using the same blob for data protection across apps. @peterbomers are you using this same blob for multiple apps?

@peterbomers
Copy link
Author

peterbomers commented Oct 18, 2023 •
edited
Loading

This comment seems to indicate that the issue can occur when using the same blob for data protection across apps. @peterbomers are you using this same blob for multiple apps?

No, its one app. Its deployed to Azure and uses a service plan with multiple instances.

@JoshLove-msft
Copy link
Member

JoshLove-msft commented Oct 18, 2023 •
edited
Loading

I see - but the fact that this issue is occurring with multiple data protection storage repositories points to this being an issue in the core DataProtection APIs as opposed to the blobs extension. Can you add your details to the linked issue - dotnet/aspnetcore#33071?

@peterbomers
Copy link
Author

peterbomers commented Oct 19, 2023 •
edited
Loading

I see - but the fact that this issue is occurring with multiple data protection storage repositories points to this being an issue in the core DataProtection APIs as opposed to the blobs extension. Can you add your details to the linked issue - dotnet/aspnetcore#33071?

I found the code path causing this issue.

When the AzureBlobXmlRepository receives 404 from the blob-client it resets the local cached data. Apparently these 404 codes are also reported when there are network issues. (see docs)

Once this happens the default key is no longer available for the KeyRingProvider.
It will jump to line 110.
The now value used for activationDate is initialized with DateTime.UtcNow before the creationDate in the XmlKeyManager is initialized with DateTimeOffset.UtcNow

@JoshLove-msft
Copy link
Member

Great find! If I'm understanding correctly the issue stems from the ASP .NET data protection code base. Would you mind adding your findings to the linked issue?

@JoshLove-msft JoshLove-msft added the needs-author-feedback Workflow: More information is needed from author to address the issue. label Oct 19, 2023
@github-actions github-actions bot removed the needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team label Oct 19, 2023
@github-actions
Copy link

Hi @peterbomers. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

@peterbomers
Copy link
Author

Great find! If I'm understanding correctly the issue stems from the ASP .NET data protection code base. Would you mind adding your findings to the linked issue?

dotnet/aspnetcore#33071 (comment)

@github-actions github-actions bot added needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team and removed needs-author-feedback Workflow: More information is needed from author to address the issue. labels Oct 19, 2023
@JoshLove-msft
Copy link
Member

Given that the root issue is being tracked in the ASP .NET Core repo, I will go ahead and close out this issue. @peterbomers, feel free to reopen if there is anything specific to the extension that you believe should be addressed.

@github-actions github-actions bot locked and limited conversation to collaborators Jan 21, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@JoshLove-msft JoshLove-msft

Labels
Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. Extensions ASP.NET Core extensions needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team question The issue doesn't require a change to the product in order to be resolved. Most issues start as that
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jsquire @peterbomers @JoshLove-msft