Terraform changes a lot of resources when removing an element from the middle of a list

[joshuaspence]

We have a lot of AWS Route53 zones which are setup in exactly the same way. As such, we are using count and a list variable to manage these. The code basically looks like this:

variable "zone_names" {
  type    = "list"
  default = []
}

resource "aws_route53_zone" "ctld" {
  name  = "${var.zone_names[count.index]}"
  count = "${length(var.zone_names)}"
}

resource "aws_route53_record" "www" {
  zone_id = "${var.zone_names[count.index]}"
  name    = "www"
  type    = "A"

  alias {
    # ...
  }

  count = "${length(var.zone_names)}"
}

However, the problem with this is that Terraform references resources using a numeric identifier. As such, if we remove an element from the middle of the list then Terraform will want to recreate all resources with a larger numeric index. This can be minimally reproduce with the following code snippet:

variable "names" {
  type    = "list"
  default = ["foo", "bar", "baz"]
}

resource "null_resource" "test" {
  triggers {
    name = "${var.names[count.index]}"
  }

  count = "${length(var.names)}"
}

Run terraform apply and then remove "bar" from var.names. The subsequent plan is as follows:

-/+ null_resource.test.1
    triggers.%:    "1" => "1"
    triggers.name: "bar" => "baz" (forces new resource)

- null_resource.test.2

I thought that one possible way to fix this would be if Terraform could index a list of resources by its identifier rather than its sequence in a list.

[apparentlymart]

Hi @joshuaspence!

This is an issue we've had on our radar for a while now and have some ideas to deal with it, which will hopefully be included as part of a set of configuration language improvements coming in the future. The most promising idea so far is to add a new for_each argument alongside count which can use a stable identifier for the resource identifiers, similar to what you suggested.

Thanks for reporting it!

[ewbankkit]

@apparentlymart Until said functionality is available, any thoughts around adding a CLI command to compact a list in the state file (e.g. terraform state compact)?
You could then:

• remove a list element in the code - e.g. at index 1
• remove the associated resource from state - terraform state rm aws_route53_zone.ctld[1]
• compact the counted resource in state - terraform state compact aws_route53_zone.ctld

At this point terraform plan should show no changes.

[apparentlymart]

That's an interesting idea, @ewbankkit!

My first reaction to it is being a little nervous about it, since it's a single command that could create a lot of disruption in a state that would be hard to recover from if it's a mistake.

However, perhaps as one of a suite of operations it would be okay, so that mistakes would be no more costly than accidentally running terraform state rm on the wrong resource...

• As you proposed, an operation to remove an instance from a counted list of instances
• Another operation to "insert" an instance at an index, which would really just be renumbering all of the indices >= to the specified one to make a space for a new instance to exist in the next plan.
• An operation to swap two instances as a single operation, so to help with recovery if instances end up in the wrong order.

This way if I make a mistake with the "remove one instance" operation I can use the "insert" operation to renumber everything back again, and then hopefully terraform import the thing I removed back into its previous place.

I expect no matter how we were to present it these commands would be a bit arcane and edge-casey, but this would still be superior to having users manually edit the state file to renumber things. I'm a little unsure as to what to name these three operations so that they are self-explaining and make sense together as a set. If you have any more thoughts here, maybe let's discuss this in a separate issue.

[giuliano108]

The way we worked around this is by scripting a .tfstate transform. It moves to the end of a list the resources we want to remove...

[goffinf]

We also use count to create a lot of instances of various resource types (aws_ecr_repository is one example). We provide a list of repo names and use count to iterate. But, as mentioned here if you either add a new entry to the list in any position other than the end, or, if you delete an entry from anywhere other than the last, Terraform we destroy and recreate every resource from that point.

Doesn't this sound like a case for passing a map into a resource so that resource creation / deletion is based on the explicit key rather than rely on the ordinal position in a list ?

I suppose if we could interpolate the resource name itself that might do it, but I'm not sure how practical that is ?

Anyway, whatever the underlaying implementation, this problem is really hurting us.

We currently use Terraform 0.9.11. Is a solution likely in the near term ?

Can you suggest any other work-arounds other than manipulating the data backend state file (not something I'm especially attracted towards) ?

Kind Regards

Fraser.

[coingraham]

I wrote a utility for editing count in the state file, but would like to see a fix as well.

[pschuegr]

@apparentlymart Do you have any idea of what the timeframe might be for a fix to this issue? Thanks!

[Farhie]

I am running into this issue with generation of ECR repos. I'd be interested to know where this is on your radar @apparentlymart? I assume the current workaround is to utilise a map as per @goffinf's suggestion?