PersistKeysToAzureBlobStorage: 'Unable to retrieve the decryption key.'

[peterbomers]

Azure.Extensions.AspNetCore.DataProtection.Blobs 1.2.1

We're using the ASP.NET Core Data Protection feature in a production environment with multiple instances and deployment slots.
Last week, after a deployment and a key rotation (default every 90 days) the Data Protection feature was no longer able to decrypt data from our database. It kept throwing CryptographicException: 'Unable to retrieve the decryption key.' and we lost fital information.

Our code setup looks like this:

services.AddDataProtection()
        .PersistKeysToAzureBlobStorage(options.BlobStorage, "data-protection", "keys.xml")
        .ProtectKeysWithCertificate(keyVault.GetCertificate("data-protection"));

During deployment (application startup) we saw ClientOtherError in the blob storage montoring.
The problem started after this deployment and keys file on the blob storage now contains weird entries.
Especially the creation, activation and expiration dates are strange. Below the last 3 entries. (total file size = 67kb)

<?xml version="1.0" encoding="utf-8"?>
<repository>
    <key id="d7720961-removed" version="1">
        <creationDate>2022-07-26T15:58:04.9292928Z</creationDate>
        <activationDate>2022-07-28T15:34:11.4371143Z</activationDate>
        <expirationDate>2022-10-24T15:58:04.907673Z</expirationDate>
        <descriptor deserializerType="Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60">
            <descriptor>
                <encryption algorithm="AES_256_CBC" />
                <validation algorithm="HMACSHA256" />
                <encryptedSecret decryptorType="Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60"
                    xmlns="http://schemas.asp.net/2015/03/dataProtection">
                    <EncryptedData Type="http://www.w3.org/2001/04/xmlenc#Element"
                        xmlns="http://www.w3.org/2001/04/xmlenc#">
                        <EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#aes256-cbc" />
                        <KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#">
                            <EncryptedKey xmlns="http://www.w3.org/2001/04/xmlenc#">
                                <EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#rsa-1_5" />
                                <KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#">
                                    <X509Data>
                                        <X509Certificate>removed</X509Certificate>
                                    </X509Data>
                                </KeyInfo>
                                <CipherData>
                                    <CipherValue>removed</CipherValue>
                                </CipherData>
                            </EncryptedKey>
                        </KeyInfo>
                        <CipherData>
                            <CipherValue>removed</CipherValue>
                        </CipherData>
                    </EncryptedData>
                </encryptedSecret>
            </descriptor>
        </descriptor>
    </key>
    <key id="a04a75a1-removed" version="1">
        <creationDate>2022-10-23T03:32:59.4371134Z</creationDate>
        <activationDate>2022-10-24T15:58:04.907673Z</activationDate>
        <expirationDate>2023-01-21T03:32:59.3917697Z</expirationDate>
        <descriptor deserializerType="Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60">
            <descriptor>
                <encryption algorithm="AES_256_CBC" />
                <validation algorithm="HMACSHA256" />
                <encryptedSecret decryptorType="Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60"
                    xmlns="http://schemas.asp.net/2015/03/dataProtection">
                    <EncryptedData Type="http://www.w3.org/2001/04/xmlenc#Element"
                        xmlns="http://www.w3.org/2001/04/xmlenc#">
                        <EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#aes256-cbc" />
                        <KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#">
                            <EncryptedKey xmlns="http://www.w3.org/2001/04/xmlenc#">
                                <EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#rsa-1_5" />
                                <KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#">
                                    <X509Data>
                                        <X509Certificate>removed</X509Certificate>
                                    </X509Data>
                                </KeyInfo>
                                <CipherData>
                                    <CipherValue>removed</CipherValue>
                                </CipherData>
                            </EncryptedKey>
                        </KeyInfo>
                        <CipherData>
                            <CipherValue>removed</CipherValue>
                        </CipherData>
                    </EncryptedData>
                </encryptedSecret>
            </descriptor>
        </descriptor>
    </key>
    <key id="2de0efd7-removed" version="1">
        <creationDate>2022-11-17T09:35:54.6857467Z</creationDate>
        <activationDate>2022-11-17T09:35:54.3656624Z</activationDate>
        <expirationDate>2023-02-15T09:35:54.3656624Z</expirationDate>
        <descriptor deserializerType="Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60">
            <descriptor>
                <encryption algorithm="AES_256_CBC" />
                <validation algorithm="HMACSHA256" />
                <encryptedSecret decryptorType="Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60"
                    xmlns="http://schemas.asp.net/2015/03/dataProtection">
                    <EncryptedData Type="http://www.w3.org/2001/04/xmlenc#Element"
                        xmlns="http://www.w3.org/2001/04/xmlenc#">
                        <EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#aes256-cbc" />
                        <KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#">
                            <EncryptedKey xmlns="http://www.w3.org/2001/04/xmlenc#">
                                <EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#rsa-1_5" />
                                <KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#">
                                    <X509Data>
                                        <X509Certificate>removed</X509Certificate>
                                    </X509Data>
                                </KeyInfo>
                                <CipherData>
                                    <CipherValue>removed</CipherValue>
                                </CipherData>
                            </EncryptedKey>
                        </KeyInfo>
                        <CipherData>
                            <CipherValue>removed</CipherValue>
                        </CipherData>
                    </EncryptedData>
                </encryptedSecret>
            </descriptor>
        </descriptor>
    </key>
</repository>

Our demand on the Data Protection feature is high and I suspect that the AzureBlobXmlRepository can't handle rotation very well in an environment with many concurrent requests.

[azure-sdk]

Label prediction was below confidence level 0.6 for Model:ServiceLabels: 'Storage:0.50965524,Service Bus:0.027342962,Search:0.025486905'

[jsquire]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[JoshLove-msft]

Hi @peterbomers, apologies for the delay. Do you happen to have a stack trace demonstrating the exception?

[github-actions]

Hi @peterbomers. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[github-actions]

Hi @peterbomers, we're sending this friendly reminder because we haven't heard back from you in 7 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!