second destroy (after successful first) returns 'The "count" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created.'

[jaffel-lc]

Terraform Version

Terraform v1.3.3                                                                                                                                                                                           on linux_amd64                                                                                                                                                                                             + provider registry.terraform.io/hashicorp/aws v4.34.0

Terraform Configuration Files

variable "external_acm" {
  type        = bool
  default     = false
}

variable "lb_certificate_arn" {
  type        = string
  default     = ""
  description = "If running HTTPS then a valid certificate arn must be provided."
}

variable "route53_zone_id" {
  type        = string
  default     = null
  description = "Route53 Zone ID for the domain served by this load balancer"
}

resource "aws_acm_certificate_validation" "certval" {
  count                   = (var.external_acm || var.lb_certificate_arn != "" || var.route53_zone_id == null) ? 0 : 1
  certificate_arn         = aws_acm_certificate.cert[0].arn
  validation_record_fqdns = [aws_route53_record.cert_validation[0].fqdn]

  lifecycle {
    create_before_destroy = true
  }
}

Debug Output

https://gist.github.com/jaffel-lc/78be590f8fddae0426adbffba844f374

Expected Behavior

second destroy should complete without error (just like the first) and without destroying anything.

Actual Behavior

Do you really want to destroy all resources? 
Terraform will destroy all your managed infrastructure, as shown above. 
There is no undo. Only 'yes' will be accepted to confirm. 
Enter a value: yes
╷                                                                                                                                                                                                          │ Error: Invalid count argument
│
on ../certificate.tf line 30, in resource "aws_acm_certificate_validation" "certval":
│   30:   count                   = (var.external_acm || var.lb_certificate_arn != "" || var.route53_zone_id == null) ? 0 : 1 
│
│ The "count" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how 
any instances will be created. To work around this, use the -target argument to
│ first apply only the resources that the count depends on.                                                                                                                                                ╵                                                                                                                                  

Steps to Reproduce

1. terraform init
2. terraform apply
3. terraform destroy
4. terraform destroy

Additional Context

No response

References

No response

[jaffel-lc]

While this appears to be a harmless error, the erroneously failing tf destroy command's exit status is 1 which makes our build/tests pipeline report failure form its last - global cleanup - step, despite having successfully destroyed all of the resources in the previous step.

[apparentlymart]

Hi @jaffel-lc! Thanks for reporting this.

I think what's going on here is that Terraform creates a normal plan as a precursor to creating a destroy plan because a normal plan refreshes the previous run state and can therefore detect if something has already been destroyed which therefore doesn't need to be "re-destroyed". However, a normal plan also needs to expand all resource blocks that have repetition arguments, and so it can run into this problem if the repetition depends on something that hasn't been created yet, in this case because you've literally just destroyed it.

If I'm right about the cause then the good news is that we changed the approach to that in #32051 for an unrelated reason, and so as of the next release Terraform will internally use a refresh-only plan for that initial refreshing step. Terraform didn't behave this way before just because the destroy feature has been around longer than the possibility of refresh-only plans, and so it was implemented in terms of the primitives that were available at the time.

That change was backported into the v1.3 branch and so will be included in the forthcoming v1.3.4 release. Once that's out (which should be in the next week or so), could you give that a try and see if the problem still occurs? Thanks!

[jaffel-lc]

That is great news.
I'll test again once I notice 1.3.4

[jaffel-lc]

Tested.

Now I am getting several "invalid index" errors.

Here is one example:

│ Error: Invalid index                                                                                                                                                                 
│                                                                                                                                                                                       
│   on .terraform/modules/t3_task1/main.tf line 5, in locals:
│    5:   log_group_name = var.log_group_name != "" ? var.log_group_name : aws_cloudwatch_log_group.task-log-group[0].name
│      ├────────────────
│      │ aws_cloudwatch_log_group.task-log-group is empty tuple
│
│ The given key does not identify an element in this collection value: the collection has no elements.                 

[updated error formatting for readability]

[apparentlymart]

Thanks @jaffel-lc!

It seems that Terraform is noticing that there are now no instances of aws_cloudwatch_log_group.task-log-group and so is rejecting this access of element zero as invalid.

I would agree that this doesn't seem right, but I'm also not really sure what Terraform ought to do instead here. It is true that there is not an index zero in the state, but there is still presumably an index zero declared in the configuration. It's weird to evaluate something in the configuration against the current state rather than the desired state, but in this context Terraform isn't actually building a desired state and so it can't refer to that.

With that said then: it does seem like there's an opportunity to improve this case, but it's not clear to me exactly what change is valid to make here while allowing the refresh phase prior to destroy still work. We'll need to think more about that before deciding how to proceed here.

In the meantime I think unfortunately the most viable strategy with today's Terraform is to somehow avoid running terraform destroy a second time. One possible answer to that would be to run terraform show -json and count how many resource instances are listed in the resulting JSON representation of the state; if you find none then you know that everything has already been destroyed and can skip running terraform destroy again.

[jaffel-lc]

With that said then: it does seem like there's an opportunity to improve this case, but it's not clear to me exactly what
change is valid to make here while allowing the refresh phase prior to destroy still work. We'll need to think more about
that before deciding how to proceed here.

One thought is that an error caused by outputs could have a different exit value than one cause by e.g. syntax error, and then our automations could choose to treat that specific exit code as not-an-error.

Another is that TF could not bother to lookup an output value if its originating resource is about to be destroyed, or does not exist, since the output will be cleared.

We have a cleanup task that runs a second destroy as a workaround failures we have encountered when ASGs, ECS services, and/or ecsclusters do not complete their destroys before terraform times out waiting for them to complete. The second destroy usually clears the reseources, or manages to resync the state file.

[jaffel-lc]

But it occurs to me now that I could wrap the resource[0].name reference in a try() call.

[jbardin]

I'm going to take a look into this because it's very similar to some related destroy time errors. The problem generally arises while attempting to refresh the instances, which during destroy is primarily used to ensure providers have the most recent values for their configuration, and remove any instances which may have already been deleted. In the meantime, a better workaround maybe to use -refresh=false to skip this process, which is probably not very useful to begin with if the destroy operations are run back-to-back.

[AmyJeanes]

This has also hit us too on 1.3.4, downgrading to 1.2.9 appears to work for now, fingers crossed for a proper fix soon! 😄

Interestingly I found that downgrading to 1.3.2 initially appeared to fix the issue for one case I was seeing but then it failed in other cases whereas 1.2.9 seems to work properly.

[sbocinec]

Now I am getting several "invalid index" errors.
The given key does not identify an element in this collection value: the collection has no elements.

Spent half of my day today investigating our integration test suite failing with the following error after upgrading from 1.2.9 to 1.3.4:

The given key does not identify an element in this collection value: the collection has no elements.
Error: Invalid index
  on main.tf line 49, in locals:
  49:     ? module.network[0].vpc_id
    ├────────────────
    │ module.network is empty tuple
The given key does not identify an element in this collection value: the collection has no elements.

trying to hunt down the issue and now I see the culprit is indeed in the destroy fixes in the 1.3.x causing this nasty bugs.. I hope this is going to be fixed soon 🤞