Skip to content

NLBs not deleted when the corresponding k8s services are deleted #4773

@PomeloWang

Description

@PomeloWang

I'm experiencing the same issue: resources in the EKS cluster have been deleted, but the NLB on AWS has not been deleted.

Description

NLB resources are orphaned in AWS when a Service with LoadBalancer is deleted during a Controller restart/upgrade. The Service is fully deleted from the cluster but the corresponding NLB remains in AWS with the cluster ownership tag.

Symptoms

  • ✅ NLB exists in AWS with correct elbv2.k8s.aws/cluster tag
  • ✅ NLB has elbv2.k8s.aws/resources/id service.k8s.aws/stack tag pointing to deleted Service
  • ❌ Service no longer exists in the cluster
  • ❌ No errors in Controller logs
  • ❌ Controller doesn't attempt to clean up the NLB

Expected Behavior

One of:

  1. Controller should be idempotent: If NLB is already deleted, treat it as success and remove finalizer
  2. Add GC mechanism: Controller should periodically scan AWS for orphaned NLBs with cluster tags and clean them up
  3. Handle missing NLB gracefully: When finalizer removal needed but NLB already gone, allow cleanup to continue

Controller Version

v2.6.1

Kubernetes Version

v1.32.12-eks

Originally posted by @PomeloWang in #3841

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions