aws-s3-deployment: custom `logGroup` of `BucketDeployment` is prevented from deletion

[nikmilson]

Describe the bug

When a custom logGroup is passed to BucketDeployment, the log group is not reliably deleted with the Cloudformation Stack, despite having "delete" deletion policy and "DELETE_COMPLETE" status in Cloudformation resources list.

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Library Version

No response

Expected Behavior

A custom logGroup with "delete" deletion policy passed to BucketDeployment is deleted, when corresponding Cloudformation Stack is deleted.

Current Behavior

To me it seems like the log group was actually recreated a couple of minutes after deletion (see the screenshot). Probably it happens because the BucketDeletion Lambda was executed after the Log Group was deleted. The Lambda was not triggered by us.

Reproduction Steps

1. Create the stack described below (requires also an "../assets" folder with files to upload)
2. Delete the stack
3. Eventually the log group is not deleted

We create two BucketDeployments, which upload files to the same bucket. One of them contains Cloudfront invalidation path.

Code snippet:

import * as path from 'path';
import { Construct } from 'constructs';
import * as cloudfront from 'aws-cdk-lib/aws-cloudfront';
import * as bucket from 'aws-cdk-lib/aws-s3';
import * as s3deploy from 'aws-cdk-lib/aws-s3-deployment';
import * as cdk from 'aws-cdk-lib';
import * as cloudfrontOrigins from 'aws-cdk-lib/aws-cloudfront-origins';
import * as logs from 'aws-cdk-lib/aws-logs';

export class ApplicationStack extends cdk.Stack {
  public constructor(scope?: Construct, id?: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const assetsBucket = new bucket.Bucket(this, 'assets-bucket', {
      publicReadAccess: false,
      bucketName: 'assets',
      autoDeleteObjects: true,
      removalPolicy: cdk.RemovalPolicy.DESTROY,
      encryption: bucket.BucketEncryption.S3_MANAGED,
      enforceSSL: true
    });

    const bucketOrigin = cloudfrontOrigins.S3BucketOrigin.withOriginAccessControl(assetsBucket);
    const cloudfrontDistribution = new cloudfront.Distribution(this, 'cloudfront-distribution', {
      priceClass: cloudfront.PriceClass.PRICE_CLASS_100,
      minimumProtocolVersion: cloudfront.SecurityPolicyProtocol.TLS_V1_2_2021,
      defaultBehavior: {
        origin: bucketOrigin,
        viewerProtocolPolicy: cloudfront.ViewerProtocolPolicy.REDIRECT_TO_HTTPS,
        allowedMethods: cloudfront.AllowedMethods.ALLOW_GET_HEAD,
        originRequestPolicy: cloudfront.OriginRequestPolicy.ALL_VIEWER_EXCEPT_HOST_HEADER
      }
    });

    const logGroup = new logs.LogGroup(this, 'upload-assets-log-group', {
      logGroupName: `/aws/lambda/upload-assets-log-group`,
      removalPolicy: cdk.RemovalPolicy.DESTROY,
      retention: logs.RetentionDays.FIVE_DAYS
    });

    new s3deploy.BucketDeployment(this, 'upload-assets-svg', {
      sources: [s3deploy.Source.asset(path.resolve(__dirname, '..', 'assets'))],
      include: ['*.svg'],
      destinationBucket: assetsBucket,
      cacheControl: [
        s3deploy.CacheControl.maxAge(cdk.Duration.hours(1)),
        s3deploy.CacheControl.staleWhileRevalidate(cdk.Duration.minutes(10)),
        s3deploy.CacheControl.staleIfError(cdk.Duration.days(1))
      ],
      memoryLimit: 1024,
      prune: false,
      logGroup
    });

    new s3deploy.BucketDeployment(this, 'upload-assets', {
      sources: [s3deploy.Source.asset(path.resolve(__dirname, '..', 'assets'))],
      exclude: ['*.svg'],
      destinationBucket: assetsBucket,
      distribution: cloudfrontDistribution,
      distributionPaths: ['/*'], // invalidate the whole cache on redeployment
      cacheControl: [
        s3deploy.CacheControl.maxAge(cdk.Duration.days(365)),
        s3deploy.CacheControl.staleWhileRevalidate(cdk.Duration.minutes(10)),
        s3deploy.CacheControl.staleIfError(cdk.Duration.days(1))
      ],
      memoryLimit: 1024,
      prune: true,
      logGroup
    });
  }
}

Possible Solution

We solved it by adding the Log Group to the dependencies of BucketDeployment's Lambda. I'd suggest to do it inside BucketDeployment in the CDK itself, when the logGroup property is passed.

import * as cdk from 'aws-cdk-lib';
import * as s3deploy from 'aws-cdk-lib/aws-s3-deployment';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import { IDependable } from 'constructs';

export class CustomBucketDeployment extends s3deploy.BucketDeployment {
  constructor(scope: cdk.Stack, id: string, options: s3deploy.BucketDeploymentProps) {
    super(scope, id, options);

    if (options.logGroup) {
      // Ensure that the log group won't be deleted before the custom resource lambda.
      // Otherwise the lambda can be called during the stack deletion after the log group is deleted,
      // which can lead to log group recreation even though it'll be marked as deleted in the stack.
      this.addCustomResourceHandlerLambdaDependency(options.logGroup);
    }
  }

  private addCustomResourceHandlerLambdaDependency(...deps: IDependable[]) {
    const customResourceHandler = this.node.tryFindChild('CustomResourceHandler');

    if (
      customResourceHandler &&
      'lambdaFunction' in customResourceHandler &&
      customResourceHandler.lambdaFunction instanceof lambda.Function
    ) {
      customResourceHandler.lambdaFunction.node.addDependency(...deps);
    }
  }
}

Additional Information/Context

No response

AWS CDK Library version (aws-cdk-lib)

2.211.0

AWS CDK CLI version

2.1025.0

Node.js Version

20.19.2

OS

Ubuntu 24.04.3 LTS

Language

TypeScript

Language Version

No response

Other information

No response

[pahud]

Hi @nikmilson,

Thank you for reporting this issue and providing such a detailed analysis with a working solution! This is indeed a valid bug related to CloudFormation resource deletion ordering.

Root Cause
The issue occurs because there's no explicit dependency between the custom log group and the BucketDeployment Lambda function. During stack deletion, CloudFormation may delete the log group before the Lambda function completes execution, causing AWS Lambda to automatically recreate the log group when it tries to write logs.

Your Solution
Your proposed fix is exactly right - we should add an explicit dependency when a custom logGroup is provided. This would ensure the Lambda function completes before the log group is deleted.

Implementation
The fix should be added to the BucketDeployment constructor around line 400:

if (props.logGroup) {
 handler.node.addDependency(props.logGroup);
}

Workaround
Until this is fixed, users can implement your custom wrapper class or manually add the dependency:

bucketDeployment.node.findChild('CustomResourceHandler').node.addDependency(logGroup);

Would you be interested in contributing a pull request for this fix? The change should be straightforward and would help other users facing the same issue.

Thanks again for the excellent bug report and solution!

[cnaples79]

I submitted a PR to fix this.