on-destroy provisioners not being executed

[IOAyman]

Terraform Version

v0.9.2

Affected Resource(s)

• digitalocean_droplet
• provisioners (on-destroy)

Terraform Configuration Files

...
resource "digitalocean_droplet" "kr_manager" {
  name     = "${var.do_name}"
  image    = "${var.do_image}"
  region   = "${var.do_region}"
  size     = "${var.do_size}"
  ssh_keys = [XXX]

  provisioner "local-exec" {
    command = "echo ${digitalocean_droplet.kr_manager.ipv4_address} >> hosts"
  }
  provisioner "remote-exec" {
    inline = ["dnf install -y python python-dnf"]
    connection {
      type        = "ssh"
      user        = "${var.ssh_user}"
      private_key = "${file(var.ssh_key)}"
      timeout     = "1m"
    }
  }
  provisioner "local-exec" {
    command = "ansible-playbook ${var.play}"
  }
  provisioner "local-exec" {
    command = "docker-machine create --driver generic --generic-ip-address ${digitalocean_droplet.kr_manager.ipv4_address} --generic-ssh-key ${var.ssh_key} ${var.do_name}"
  }
  provisioner "local-exec" {
    when    = "destroy"
    command = "rm hosts"
  }
  provisioner "local-exec" {
    when    = "destroy"
    command = "docker-machine rm -f ${var.do_name}"
  }
}
...

Debug Output

https://gist.github.com/IOAyman/3e86d9c06d03640786184c1429376328

Expected Behavior

It should have run the on-destroy provisioners

Actual Behavior

It did not run the on-destroy provisioners

Steps to Reproduce

1. terraform apply -var-file=infrasecrets.tfvars
2. terraform destroy -var-file=infrasecrets.tfvars

References

Are there any other GitHub issues (open or closed) or Pull Requests that should be linked here? For example:

• GH-13395

[apparentlymart]

Hi @IOAyman! Thanks for opening this.

This is indeed related to both #13097 and #13395, and seems to be another example of the same general problem. However, each of these would be solved in a different part of the codebase so I'm going to leave all three open with these connectors between them though ultimately we will probably attempt to fix them all with one PR in the end.

[Gary-Armstrong]

I'm having this issue in a local-exec provisioner when I am tainting aws_instance resources, but I can't tell from this issue text if I'm hitting this particular bug. Are the same mechanism used to -/+ after taint as when a explicit terraform destroy is used?

[apparentlymart]

@Gary-Armstrong I'm almost certain that you've found another variant of the same root cause there. Thanks for mentioning it!

Having slept on it a bit I've changed my mind and am going to fold all of these issues into this one as an umbrella issue for addressing the various places that destroy provisioners aren't working yet, since I strongly suspect that the fix will be a holistic one that covers all of these cases together.

Note for me or someone else who picks this up in future: the root challenge here is that the destroy provisioners currently live only in config, but yet most of our destroy use-cases don't have access to config for one reason or another. I think the holistic fix here is to change how destroy provisioners work so that during apply we include the fully-resolved destroy provisioner configs as part of the instance state, and then when we destroy those instances we use the stashed config in the state to destroy them, thus avoiding the direct dependency on config during destroy. This can then address all of the variants of the problem we've seen so far, and probably ones we've not yet seen:

• Resource has been removed from config altogether ("orphaned", in core terminology)
• Instance has been deposed as part of the create_before_destroy lifecycle
• Instance has been tainted (this one is tricky, since we don't know if the create provisioner ran in this case; we may need to punt on making this work since we don't know if a particular destroy provisioner depends on the outcome of a particular create provisioner)

[Gary-Armstrong]

More info: I eliminated some instances by setting count = 0 and the local-exec ran properly.

[in4mer]

I'd like to add some color here:

when = "destroy" provisioners are also not being run for aws_instances that are marked as tainted, which are then destroyed and re-created. This is on v0.9.3.

I see that this was #3 in the above comment, sorry for adding more traffic. A word on the mind of the maintainer, here, though.. I consider it my responsibility to design idempotency around the create-destroy provisioners. I expect TF to be a trigger, based on when specific events start, not at some point during their execution. I can design my provisioners to error when I want them to, and ignore errors that I don't consider constructive; thereby avoiding the whole discussion of "when should we run these?"

Maybe the destroyers should have an option or two allowing us to customize when/where they fire? Food for thought.

My current destruction provisioner (This might be helpful to see my position):

  provisioner "local-exec" {
    when    = "destroy"
    command = "knife node delete ${self.tags.Name} -y; knife client delete ${self.tags.Name} -y; true"
   /* The reason we're ignoring the return values is either it works, or it
      wasnt there to begin with. Or you don't have permissions, but it's going
      to wind up desynchronized anyway. */
  }

An alternative would be a way to gate the overall instance termination on the relative success or failure of other designated dependent destructors prior to outright instance destruction. I wonder if that can be accomplished by putting the local-exec destructor before the Chef provisioner, but I haven't tested to see if that would work or not. Then I could save the desync by designing my destructor to stop terraform destroy in a better way.

[leftathome]

In the case of a tainted resource, tracking the individual run results and configurations of provisioners against the resource as part of resource state should help Terraform in deciding whether a provisioner needs to run its destroy action on the resource, with additional guidance from the resource's configuration.

For provisioners that kick off config management tools, a successful run usually indicates there's something there that needs to be torn down at destroy time. There will probably be a common set of actions that each CM tool uses for decommissioning, as in @in4mer's example where API calls need to get made to remove the instance from a Chef server.

(I actually found this thread because I thought that's how Terraform's Chef provisioner already worked!)

Remote or local exec are more open-ended, so they might need to have destroy-stage actions explicitly defined in the resource config, defaulting to noop.

[pierrebeaucamp]

Edit: Sorry, it appears to run correctly now. Please ignore this whole comment

I have the same issue with a null_resource and a local-exec provisioner in it. Creation works fine, but terraform destroy completely skips over the when = "destroy" command.