I'm experiencing the same issue: resources in the EKS cluster have been deleted, but the NLB on AWS has not been deleted.
Description
NLB resources are orphaned in AWS when a Service with LoadBalancer is deleted during a Controller restart/upgrade. The Service is fully deleted from the cluster but the corresponding NLB remains in AWS with the cluster ownership tag.
Symptoms
- ✅ NLB exists in AWS with correct
elbv2.k8s.aws/cluster tag
- ✅ NLB has
elbv2.k8s.aws/resources/id service.k8s.aws/stack tag pointing to deleted Service
- ❌ Service no longer exists in the cluster
- ❌ No errors in Controller logs
- ❌ Controller doesn't attempt to clean up the NLB
Expected Behavior
One of:
- Controller should be idempotent: If NLB is already deleted, treat it as success and remove finalizer
- Add GC mechanism: Controller should periodically scan AWS for orphaned NLBs with cluster tags and clean them up
- Handle missing NLB gracefully: When finalizer removal needed but NLB already gone, allow cleanup to continue
Controller Version
v2.6.1
Kubernetes Version
v1.32.12-eks
Originally posted by @PomeloWang in #3841
Originally posted by @PomeloWang in #3841