Fix CreateInNotExistAsync bug#3398
Merged
Aaronontheweb merged 1 commit intoakkadotnet:devfrom Feb 25, 2026
Merged
Conversation
This was referenced Mar 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3397
Problem
AzureApiImpl.ContainerClient()calledCreateIfNotExistsAsync()with no exception handling. The Azure SDK has known bugs where this method still throwsRequestFailedException(409)forContainerAlreadyExists. The unhandled exception propagates through the lease actor to the Split Brain Resolver, which reverses its decision toReverseDownIndirectlyConnectedand downs the entire cluster.Propagation path:
ContainerClient()throws 409 →LeaseResourceExists()catch block misidentifies it as a blob-level error → re-thrown asLeaseException→ReadOrCreateLeaseResource()(no catch) →LeaseActorPipeTo →Status.Failure→ SBR receives failure →ReverseDownIndirectlyConnected→ all nodes downed.Fix
Replaced
CreateIfNotExistsAsync()withCreateAsync()+ explicit 409 handling inContainerClient():ContainerAlreadyExists(benign) — silently handled, set_initialized = trueContainerInitializationExceptionNew
ContainerInitializationException— an internal exception type that distinguishes container-level errors from blob-levelRequestFailedException. This prevents the existingcatch (RequestFailedException)blocks (designed for blob operations) from mishandling container creation errors.Catch blocks added to
LeaseResourceExists(),CreateLeaseResource(), andGetLeaseResource()— each catchesContainerInitializationExceptionand returns its natural "try again" value (false,null,null), which feeds back into the existing retry loop inReadOrCreateLeaseResource().Files Changed
src/coordination/azure/Akka.Coordination.Azure/Internal/AzureApiImpl.csContainerInitializationException; rewriteContainerClient(); add catch blocks to caller methodssrc/coordination/azure/Akka.Coordination.Azure.Tests/AzureApiSpec.csTest Plan
ShouldHandleContainerAlreadyExists— reproduces the exact SBRlease-majorityfails to acquire lease due to HTTP 409ContainerAlreadyExists, causingReverseDownIndirectlyConnectedto down the entire cluster #3397 scenario: secondAzureApiImplinstance hits pre-existing container, verifies it succeeds instead of throwingMultipleInstancesShouldAcquireDifferentLeases— simulates multi-node cluster where two instances race to create the same container