DataProtection - Wrong activation time

[georgelivingston]

Describe the bug

Activation time is ahead of the creation time and causing (Cookies was not authenticated. Failure message: Unprotect ticket failed)

To Reproduce

Multiple service is deployed with the below code to protect the cookies. Everything works as expected. From yesterday, suddenly facing this error (Cookies was not authenticated. Failure message: Unprotect ticket failed) in one service, were the other services works as expected. On further analysis, found activate date is ahead of the creation date in the blob stored XML.

var dataProtectionBuilder = services.AddDataProtection()
                .PersistKeysToAzureBlobStorage(environmentVariable.AzureBlobStorage, "secret", "secretkey.xml")
                .SetApplicationName("cookiename");
dataProtectionBuilder?.ProtectKeysWithCertificate(x509ClientCertificate);

Below is the active key in the blob stored xml file.

<creationDate>2021-05-26T09:35:20.9324376Z</creationDate>
<activationDate>2021-05-26T09:35:20.078957Z</activationDate
<expirationDate>2021-08-24T09:35:20.078957Z</expirationDate>

Exceptions (if any)

Microsoft.AspNetCore.Authentication.Cookies.CookieAuthenticationHandler[7]
Cookies was not authenticated. Failure message: Unprotect ticket failed

Further technical details

• ASP.NET Core version - 3.1
• Microsoft.AspNetCore.DataProtection, Version=3.1.0.0

[blowdart]

That's very odd. To clarify the other services sharing the keyring work just fine, with the same cookies?

[georgelivingston]

That's very odd. To clarify the other services sharing the keyring work just fine, with the same cookies?

Yes, the 3 services are sharing the same keyring. It works for 2 services, and not for only one service, with the same cookies.

[blowdart]

If it's working for two apps, then the date doesn't have any effect. Do you have any way to repo this for us?

[ghost]

Thanks for contacting us.

We're moving this issue to the Next sprint planning milestone for future evaluation / consideration. We would like to keep this around to collect more feedback, which can help us with prioritizing this work. We will re-evaluate this issue, during our next planning meeting(s).
If we later determine, that the issue has no community involvement, or it's very rare and low-impact issue, we will close it - so that the team can focus on more important and high impact issues.
To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[georgelivingston]

If it's working for two apps, then the date doesn't have any effect. Do you have any way to repo this for us?

Sorry, I cannot be able to share the repo. But I have prepared the ConfigureServices details from startup, and data protection blob stored key.xml for your analysis. On further analysis, we found that, in one of our service has also faced this error for a second and it got recovered automatically. The other services didn't recover. After we manual refreshed the pod, it works as expected, and it occurred in our production, and we are down for more than an hour. We scared about this bug, that it may occur anytime again. We need your valuable suggestion or workaround to resolve this.

Additional note: Also, I have noted one more thing, the default rotation time is 90 days, but it is rotated irrespective of that, please refer the blob key.xml for more details.

public void ConfigureServices(IServiceCollection services)
 {

     services.AddAzureClients(builder =>
     {
         builder.AddBlobServiceClient(environmentVariable.AzureBlobStorage);
     });

     var keyVaultCache = new KeyVaultProxy();
     var clientSecretCredential = new ClientSecretCredential(environmentVariable.AzureADVaultTenantId, environmentVariable.AzureADVaultClientId, environmentVariable.AzureADVaultClientSecret);
     var cyptoOption = new CryptographyClientOptions();
     cyptoOption.AddPolicy(keyVaultCache, HttpPipelinePosition.PerCall);

     IKeyEncryptionKeyResolver keyEncryptionKeyResolver = new KeyResolver(clientSecretCredential, cyptoOption);
     services.AddSingleton(keyEncryptionKeyResolver);

     KeyClientOptions keyClientOptions = new KeyClientOptions();
     keyClientOptions.AddPolicy(keyVaultCache, HttpPipelinePosition.PerCall);

     SecretClientOptions secretClientOptions = new SecretClientOptions();
     secretClientOptions.AddPolicy(keyVaultCache, HttpPipelinePosition.PerCall);

     var secretClient = new SecretClient(new Uri(environmentVariable.VaultUri), clientSecretCredential, secretClientOptions);
     var keyClient = new KeyClient(new Uri(environmentVariable.VaultUri), clientSecretCredential, keyClientOptions);
     services.AddSingleton(secretClient);
     services.AddSingleton(keyClient);
     var dataProtectionBuilder = services.AddDataProtection()
         .PersistKeysToAzureBlobStorage(environmentVariable.AzureBlobStorage, "secret", "secretkey.xml")
         .SetApplicationName("cookiename");

     if (!webHostEnvironment.IsDevelopment())
     {
         KeyVaultSecret certificate = secretClient.GetSecret(environmentVariable.DataProtectionKeyVaultSecret);
         var x509ClientCertificate = new X509Certificate2(Convert.FromBase64String(certificate.Value));
         dataProtectionBuilder?.ProtectKeysWithCertificate(x509ClientCertificate);
         dataProtectionBuilder.DisableAutomaticKeyGeneration();
     }


     services.Configure<CookiePolicyOptions>(options =>
     {
         options.CheckConsentNeeded = context => false;
         options.MinimumSameSitePolicy = SameSiteMode.None;
     });

     services.AddAuthentication(options =>
     {
         options.DefaultAuthenticateScheme = CookieAuthenticationDefaults.AuthenticationScheme;
         options.DefaultSignInScheme = CookieAuthenticationDefaults.AuthenticationScheme;
         options.DefaultChallengeScheme = OpenIdConnectDefaults.AuthenticationScheme;
     })
     .AddCookie()
     .AddOpenIdConnect(OpenIdConnectDefaults.AuthenticationScheme, options =>
     {
         var handler = new JwtSecurityTokenHandler();
         handler.InboundClaimTypeMap.Clear();
         options.SecurityTokenValidator = handler;
         options.RequireHttpsMetadata = true;
         options.Authority = environmentVariable.LoginAuthority;
         options.SignInScheme = "Cookies";
         options.GetClaimsFromUserInfoEndpoint = true;

         // Configure the Auth0 Client ID and Client Secret.
         options.ClientId = environmentVariable.LoginClientId;
         options.ClientSecret = environmentVariable.LoginClientSecret;
         options.CallbackPath = new PathString("/signin-oidc");

         // options.Scope.Clear();
         options.Scope.Add("openid");
         options.Scope.Add("profile");
         options.ClaimActions.MapUniqueJsonKey("id", "id");
         options.ClaimActions.MapUniqueJsonKey("EmailId", "EmailId");
         options.ResponseType = OpenIdConnectResponseType.Code;
         options.UsePkce = true;
         options.ResponseMode = "query";

         options.Events.OnRemoteFailure = context =>
         {
             if (context.Failure.Message.Contains("Correlation failed", StringComparison.InvariantCultureIgnoreCase))
             {
                 context.Response.Redirect("/");
             }
             else
             {
                 context.Response.Redirect("/Account/Error");
             }

             context.HandleResponse();
             return Task.CompletedTask;
         };

         options.Events.OnRedirectToIdentityProvider = context =>
         {
             if (context.Request.Headers.ContainsKey("X-Requested-With"))
             {
                 context.Response.StatusCode = StatusCodes.Status401Unauthorized;
                 context.Response.WriteAsync(JsonConvert.SerializeObject(new { loginLink = environmentVariable.RedirectHomePageUrl, isPageRedirection = true }));
                 context.HandleResponse();
             }

             return Task.CompletedTask;
         };
     });
	
     services.AddJwtAuthentication(AppAuthenticationSchemes.IdJwtScheme);

     services.AddAuthentication(CertificateAuthenticationDefaults.AuthenticationScheme)
         .AddCertificate(options =>
         {
             options.AllowedCertificateTypes = CertificateTypes.All;
         });

}

Key.XML

<?xml version="1.0" encoding="utf-8"?><repository><key id="36c1e414-5b3c-438a-8b07-73ec5a528486" version="1"><creationDate>2020-04-23T12:31:14.818651Z</creationDate><activationDate>2020-04-23T12:31:13.543231Z</activationDate><expirationDate>2020-07-22T12:31:13.543231Z</expirationDate><descriptor deserializerType="Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=3.0.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60"><descriptor><encryption algorithm="AES_256_CBC" /><validation algorithm="HMACSHA256" /><masterKey p5:requiresEncryption="true" xmlns:p5="http://schemas.asp.net/2015/03/dataProtection"><!-- Warning: the key below is in an unencrypted form. --><value>***Redacted***</value></masterKey></descriptor></descriptor></key><key id="099f105a-e590-4e66-90d7-0883e8e2db72" version="1"><creationDate>2020-07-21T08:05:10.9149068Z</creationDate><activationDate>2020-07-22T12:31:13.543231Z</activationDate><expirationDate>2020-10-19T08:05:10.704182Z</expirationDate><descriptor deserializerType="Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=3.1.6.0, Culture=neutral, PublicKeyToken=adb9793829ddae60"><descriptor><encryption algorithm="AES_256_CBC" /><validation algorithm="HMACSHA256" /><encryptedSecret decryptorType="Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Version=3.1.6.0, Culture=neutral, PublicKeyToken=adb9793829ddae60" xmlns="http://schemas.asp.net/2015/03/dataProtection"><EncryptedData Type="http://www.w3.org/2001/04/xmlenc#Element" xmlns="http://www.w3.org/2001/04/xmlenc#"><EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#aes256-cbc" /><KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#"><EncryptedKey xmlns="http://www.w3.org/2001/04/xmlenc#"><EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#rsa-1_5" /><KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#"><X509Data><X509Certificate>***Redacted***</X509Certificate></X509Data></KeyInfo><CipherData><CipherValue>***Redacted***</CipherValue></CipherData></EncryptedKey></KeyInfo><CipherData><CipherValue>***Redacted***</CipherValue></CipherData></EncryptedData></encryptedSecret></descriptor></descriptor></key><key id="78f05c65-38d9-4b9e-bf26-7a4577a0f5f1" version="1"><creationDate>2020-09-03T09:05:48.0849065Z</creationDate><activationDate>2020-09-03T09:05:45.1695927Z</activationDate><expirationDate>2020-12-02T09:05:45.1695927Z</expirationDate><descriptor deserializerType="Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=3.1.4.0, Culture=neutral, PublicKeyToken=adb9793829ddae60"><descriptor><encryption algorithm="AES_256_CBC" /><validation algorithm="HMACSHA256" /><masterKey p5:requiresEncryption="true" xmlns:p5="http://schemas.asp.net/2015/03/dataProtection"><!-- Warning: the key below is in an unencrypted form. --><value>***Redacted***</value></masterKey></descriptor></descriptor></key><key id="2a89281f-9fd0-453c-ac37-c9aced5a0759" version="1"><creationDate>2020-11-30T10:07:35.2706401Z</creationDate><activationDate>2020-12-02T09:05:45.1695927Z</activationDate><expirationDate>2021-02-28T10:07:34.8158656Z</expirationDate><descriptor deserializerType="Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=3.1.9.0, Culture=neutral, PublicKeyToken=adb9793829ddae60"><descriptor><encryption algorithm="AES_256_CBC" /><validation algorithm="HMACSHA256" /><encryptedSecret decryptorType="Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Version=3.1.9.0, Culture=neutral, PublicKeyToken=adb9793829ddae60" xmlns="http://schemas.asp.net/2015/03/dataProtection"><EncryptedData Type="http://www.w3.org/2001/04/xmlenc#Element" xmlns="http://www.w3.org/2001/04/xmlenc#"><EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#aes256-cbc" /><KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#"><EncryptedKey xmlns="http://www.w3.org/2001/04/xmlenc#"><EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#rsa-1_5" /><KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#"><X509Data><X509Certificate>***Redacted***</CipherValue></CipherData></EncryptedData></encryptedSecret></descriptor></descriptor></key><key id="e000fae6-6a3b-46ae-b476-a95f30a522be" version="1"><creationDate>2021-02-15T06:18:04.1338347Z</creationDate><activationDate>2021-02-15T06:18:02.2434726Z</activationDate><expirationDate>2021-05-16T06:18:02.2434726Z</expirationDate><descriptor deserializerType="Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=3.1.12.0, Culture=neutral, PublicKeyToken=adb9793829ddae60"><descriptor><encryption algorithm="AES_256_CBC" /><validation algorithm="HMACSHA256" /><masterKey p5:requiresEncryption="true" xmlns:p5="http://schemas.asp.net/2015/03/dataProtection"><!-- Warning: the key below is in an unencrypted form. --><value>***Redacted***</value></masterKey></descriptor></descriptor></key><key id="cf7e1a32-3a8a-4f34-8eca-367e1aa06f79" version="1"><creationDate>2021-05-14T14:07:38.0258627Z</creationDate><activationDate>2021-05-16T06:18:02.2434726Z</activationDate><expirationDate>2021-08-12T14:07:37.7895481Z</expirationDate><descriptor deserializerType="Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=3.1.14.0, Culture=neutral, PublicKeyToken=adb9793829ddae60"><descriptor><encryption algorithm="AES_256_CBC" /><validation algorithm="HMACSHA256" /><encryptedSecret decryptorType="Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Version=3.1.14.0, Culture=neutral, PublicKeyToken=adb9793829ddae60" xmlns="http://schemas.asp.net/2015/03/dataProtection"><EncryptedData Type="http://www.w3.org/2001/04/xmlenc#Element" xmlns="http://www.w3.org/2001/04/xmlenc#"><EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#aes256-cbc" /><KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#"><EncryptedKey xmlns="http://www.w3.org/2001/04/xmlenc#"><EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#rsa-1_5" /><KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#"><X509Data><X509Certificate>***Redacted***</X509Certificate></X509Data></KeyInfo><CipherData><CipherValue>***Redacted***</CipherValue></CipherData></EncryptedKey></KeyInfo><CipherData><CipherValue>***Redacted***</CipherValue></CipherData></EncryptedData></encryptedSecret></descriptor></descriptor></key><key id="8b561d98-e699-4a50-8ab9-c4a70d13061a" version="1"><creationDate>2021-05-26T09:35:20.9324376Z</creationDate><activationDate>2021-05-26T09:35:20.078957Z</activationDate><expirationDate>2021-08-24T09:35:20.078957Z</expirationDate><descriptor deserializerType="Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=3.1.15.0, Culture=neutral, PublicKeyToken=adb9793829ddae60"><descriptor><encryption algorithm="AES_256_CBC" /><validation algorithm="HMACSHA256" /><masterKey p5:requiresEncryption="true" xmlns:p5="http://schemas.asp.net/2015/03/dataProtection"><!-- Warning: the key below is in an unencrypted form. --><value>***Redacted***</value></masterKey></descriptor></descriptor></key></repository>

[georgelivingston]

Due to no propagation time between the created and activation time, this issue occurs. I will explain the same with the below illustration.

The new key is generated from POD 1. The User A logs in and visits the first pod, and he claims the new protected cookie. Again, the same user visits using POD 2 (Old key), now the request failed with this “Cookies was not authenticated. Failure message: Unprotect ticket failed” and redirects to login.

User A again logs in and redirected to POD 1, it got successful, now again the consecutive request sends to POD 2. Again, it redirects to login. It became deadlock, POD 2 become useless, and the user facing repeated login.

As explained in the documentation it should have a provided an enough propagation time to get sync'd with the new key for all POD’s.

https://docs.microsoft.com/en-us/aspnet/core/security/data-protection/implementation/key-management?view=aspnetcore-5.0

[ajbeaven]

Thanks for reporting this one @georgelivingston.

I believe I am experiencing the same problem on my .NET5 applications. I have two apps storing the Data Protection keys in Azure Blob storage where the activation date of the key is >1 second after the creation date. One app is responsible for creating a cookie, which is then sent to the second app to authenticate the user there. This issue presents as the second app failing to authenticate.

Similarly the keys seem to be rotated extremely frequently (not every 90 days). This would happen at least once every 24-48 hours, after which time the second app would start to fail with an authentication error.

Both apps use the same code in Startup.cs:

var dataProtection = services.AddDataProtection()
	.SetApplicationName(siteSettings.SiteName)
	.PersistKeysToAzureBlobStorage(
		configuration.GetConnectionString("StorageConnectionString"),
		dataProtectionSettings.KeyStoreContainerName,
		dataProtectionSettings.KeyStoreBlobName)
	.ProtectKeysWithAzureKeyVault(
		new Uri(dataProtectionSettings.KeyVaultIdentifier),
		new DefaultAzureCredential());

Unlike @georgelivingston, I'm protecting the keys via Azure Key Vault. MS support seemed fairly confident it wasn't related to the Azure Key Vault side of things. That would seem to be supported by the fact that we both have the same problem despite using different protection mechanisms.

MS support is reportedly looking in to this for me now. Fingers crossed they identify the issue and fix shortly as this is causing real problems on my production application.

In the meantime I have set up a auto-heal rule to restart the second app when it sees a few 401 errors in quick succession. Restarting the app appears to resolve the error, presumably because it retrieves the new key at startup.

[ajbeaven]

The support team was able to identify the issue which was that multiple apps were using the same storage location to persist the Data Protection keys. Both of my apps were storing these keys in the same Azure Storage blob.

I figured this to be OK due to my reading of the docs relating to the SetApplicationName() method. This states that using different application names across apps will isolate those apps from one another. Unfortunately this isolation does not extend to the actual physical data protection keys rather, just the payload stored in them. Pointing to the same location with multiple apps using different application names will causes these issues with the activation date and subsequent problems with cross-app data protection.

@georgelivingston I'm unsure if this is the same setup you've got, but I'd recommend checking for this same thing. Using a different location to store the data protection keys immediately resolved all my issues.

[adityamandaleeka]

This is hard to investigate without a repro. If someone can provide one, we'd be happy to investigate.

[nolanbradshaw]

Any information on how to stop this issue from occurring currently? Running a .NET 6 app in production that has run into this issue.

Creation date: 2022-02-16T00:19:05.0768936Z
Activation date: 2022-02-16T00:19:04.5984669Z

Not much info to share on consistently being able to reproduce the issue. We are using .NET 6, Kubernetes, and Redis (Elasticache) to persist the keys. We do set the application name to use the same keys across all instances

Our dev environment has seen this behavior as well (with MANY keys):
Creation date: 2022-10-23T13:44:42.9233831Z
Activation date: 2022-10-23T13:44:42.8359936Z

program.cs configuration for data protection:

var redisConnection = ConnectionMultiplexer.Connect(redisConnectionStr);

builder.Services.AddDataProtection()
    .SetApplicationName("{app_name}")
    .PersistKeysToStackExchangeRedis(redisConnection, "DataProtection-Keys");

[ghost]

Thanks for contacting us.
We're moving this issue to the .NET 8 Planning milestone for future evaluation / consideration. Because it's not immediately obvious that this is a bug in our framework, we would like to keep this around to collect more feedback, which can later help us determine the impact of it. We will re-evaluate this issue, during our next planning meeting(s).
If we later determine, that the issue has no community involvement, or it's very rare and low-impact issue, we will close it - so that the team can focus on more important and high impact issues.
To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[krispenner]

I am also having a similar issue. I'm using .NET 6 with blob storage for persistence and key vault for wrapping the key. I have over 100 instances of my app running in Azure App Service Linux Docker containers.

I am finding that a new key is randomly being added to the blob XML file way before the current key expires. This new key has an activation date of two minutes before the creation date. The previous default key is not expired or revoked. This causes all my other instances to not work for up to 24 hours until their key ring cache expires.

There is this separate issue that mentioned when key vault fails to be accessed during app start-up a new key is just created instead. I'm trying to add more logging to capture this failure as app start-up logs in a Linux docker container hosted in App Service is not super easy to get logs for.

Is there a setting to always reload the keyring if there is a keyring cache miss? I see there is a two minute window during start-up, can I extend this to be indefinite?

aspnetcore/src/DataProtection/DataProtection/src/KeyManagement/KeyRingProvider.cs

Line 59 in c37ee91

internal bool InAutoRefreshWindow() => DateTime.UtcNow < AutoRefreshWindowEnd;

My sample keyring XML:

    <key id="ddcdd943-2a4a-41fa-814d-cefec5644516" version="1">
        <creationDate>2022-12-20T20:09:42.6476227Z</creationDate>
        <activationDate>2022-12-22T20:07:31.3953347Z</activationDate>
        <expirationDate>2023-01-19T20:09:41.8118313Z</expirationDate>
        <descriptor deserializerType="Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60">
            <descriptor>
                <encryption algorithm="AES_256_CBC"/>
                <validation algorithm="HMACSHA256"/>
                <encryptedSecret decryptorType="Azure.Extensions.AspNetCore.DataProtection.Keys.AzureKeyVaultXmlDecryptor, Azure.Extensions.AspNetCore.DataProtection.Keys, Version=1.1.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8" xmlns="http://schemas.asp.net/2015/03/dataProtection">
                    <encryptedKey xmlns="">
                        <!-- This key is encrypted with Azure Key Vault. -->
                        <kid>...</kid>
                        <key>...</key>
                        <iv>...</iv>
                        <value>...</value>
                    </encryptedKey>
                </encryptedSecret>
            </descriptor>
        </descriptor>
    </key>
    <key id="759405fc-01a6-492f-a06b-67ced9f05e6e" version="1">
        <creationDate>2023-01-05T17:59:15.444065Z</creationDate>
        <activationDate>2023-01-05T17:57:45.8103086Z</activationDate>
        <expirationDate>2023-02-04T17:57:45.8103086Z</expirationDate>
        <descriptor deserializerType="Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60">
            <descriptor>
                <encryption algorithm="AES_256_CBC"/>
                <validation algorithm="HMACSHA256"/>
                <encryptedSecret decryptorType="Azure.Extensions.AspNetCore.DataProtection.Keys.AzureKeyVaultXmlDecryptor, Azure.Extensions.AspNetCore.DataProtection.Keys, Version=1.1.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8" xmlns="http://schemas.asp.net/2015/03/dataProtection">
                    <encryptedKey xmlns="">
                        <!-- This key is encrypted with Azure Key Vault. -->
                        <kid>...</kid>
                        <key>...</key>
                        <iv>...</iv>
                        <value>...</value>
                    </encryptedKey>
                </encryptedSecret>
            </descriptor>
        </descriptor>
    </key>
    <key id="1e930dbd-b293-4001-981d-233315ae3826" version="1">
        <creationDate>2023-01-23T21:48:10.1974563Z</creationDate>
        <activationDate>2023-01-23T21:46:31.3660167Z</activationDate>
        <expirationDate>2023-02-22T21:46:31.3660167Z</expirationDate>
        <descriptor deserializerType="Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60">
            <descriptor>
                <encryption algorithm="AES_256_CBC"/>
                <validation algorithm="HMACSHA256"/>
                <encryptedSecret decryptorType="Azure.Extensions.AspNetCore.DataProtection.Keys.AzureKeyVaultXmlDecryptor, Azure.Extensions.AspNetCore.DataProtection.Keys, Version=1.1.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8" xmlns="http://schemas.asp.net/2015/03/dataProtection">
                    <encryptedKey xmlns="">
                        <!-- This key is encrypted with Azure Key Vault. -->
                        <kid>...</kid>
                        <key>...</key>
                        <iv>...</iv>
                        <value>...</value>
                    </encryptedKey>
                </encryptedSecret>
            </descriptor>
        </descriptor>
    </key>

[peterbomers]

This issue also occurs when the AzureBlobXmlRepository suffers from network issues.

I've found the code path causing it.
When there is no the default key available (in our case due to network issues with the blob storage) the KeyRingProvider jumps to line 110.

The now value used for activationDate is initialized with DateTime.UtcNow before the creationDate in the XmlKeyManager is initialized with DateTimeOffset.UtcNow

[lukasz-zoglowek]

We have also experienced similar case in our production system. From the telemetry we can conclude that:

1. A node downloaded keys.xml successful from the blob storage
2. A node failed to decrypt the keys because connection to KeyVault failed with number of retries
3. The logic managed to create a new key at the point when the connection to the KeyVault was successful

Would that be possible to add a an option to tell what should happen in that case? Issuing a new key with immediate activation is breaking all nodes in the system.

[jdlegan]

We have experienced this as well, three seperate times, three seperate environements, all related to transient connectivity issues in East US. Statistically the fact that these specific key operations have failed more than once when we run several million transactions through key vault monthly is staggering. This is causing us to reconsider how we are persisting and protecting keys that, by design, are meant to be self managed.

[ghost]

To learn more about what this message means, what to expect next, and how this issue will be handled you can read our Triage Process document.
We're moving this issue to the .NET 9 Planning milestone for future evaluation / consideration. Because it's not immediately obvious what is causing this behavior, we would like to keep this around to collect more feedback, which can later help us determine how to handle this. We will re-evaluate this issue, during our next planning meeting(s).
If we later determine, that the issue has no community involvement, or it's very rare and low-impact issue, we will close it - so that the team can focus on more important and high impact work.

[JarrodJ83]

Does anyone have a good mitigation strategy they are using here? Also what has been the best resolution for this when it occurs? We just had it happen in a lower environment and redeploying the application seems to have fixed the issue but it's strange because there is still a key in the ring with activation code prior to creation date which, if I'm following correctly, is what causes the issue to begin with.

[amcasey]

@JarrodJ83 Sorry for the late reply - this wasn't the main tracking issue for Data Protection races. There are multiple races in this area, especially prior to 9.0, and most effort to date has been focused on making these exceptions rarer (vs eliminating them).

An approach we've been exploring for addressing the root cause has been to have a separate component that is responsible for all key generation and disabling key generation in app instances. If there's only one writer, there's no race. Unfortunately, the approach isn't mature enough for us to have a doc or a sample at this point, but let me know if you have questions.

[amcasey]

I'm going to close this issue. Please post additional feedback in #36157 so we can track it centrally and don't drop any.