Persistent state and automatic reimport for resources that aren't fully deleted on `destroy`

[geekofalltrades]

Terraform Version

Terraform v1.5.0-beta1
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v4.67.0

Use Cases

Some resources, like aws_secretsmanager_secret or aws_kms_key, go into a pending deletion state outside of Terraform when destroyed, rather than being immediately deleted.

If a configuration containing a resource like this is intended to be applied and destroyed regularly -- for example, a dev or CI environment -- then it is currently already painful to handle. Because the resource can't be fully deleted by design, it must be manually undeleted and reimported into Terraform before the configuration can be applied again. The new import blocks coming in Terraform 1.5.0 seemed like they had the potential to improve this: if those resources could be enhanced to undelete themselves when imported, then the import block would get rid of any manual steps needed in between applies.

resource "aws_secretsmanager_secret" "test" {
  name = "test"
}

import {
  # Doesn't actually need to be a full ARN to be imported successfully, in spite of documentation.
  # This resource type gets some random characters added to the end of its ARN by AWS.
  id = "arn:aws:secretsmanager:us-west-2:xxx:secret:test"
  to = aws_secretsmanager_secret.test
}

But now the pain point comes on the first apply. In a configuration like this, the import block would be intended to be permanent. But this means that on the first apply, it needs to be commented out, otherwise it fails with Error: Cannot import non-existent remote object.

Attempted Solutions

The way we currently manage resources of this type is with a wrapper script that undeletes and reimports the resource outside of Terraform before running apply.

Proposal

A new concept of "persistence" is added to resources. A persistent resource continues to have its ID stored in the state after it is destroyed. Persistent bool is added as a new attribute to schema.Resource to effect this change. When a provider adds support for a resource that can't be immediately deleted by Terraform, they should set this to true on that resource.

Every time state is refreshed, if the resource has been previously destroyed, the resource is read via the stored ID, and is only removed from the state when it is found to have been fully deleted outside of Terraform.

When a destroyed persistent resource is applied, Terraform automatically proposes an import of the previously deleted resource, rather than creation of a new resource. It would be up to providers to ensure that import of the resource includes undeleting/restoring it.

terraform state list for a resource like this that has been destroyed might show:

aws_secretsmanager_secret.test [destroyed]

The resource could be terraform state rm'd to make Terraform fully forget about it and attempt to create a new resource instead of reimporting the previous one. A new lifecycle option, persistence, of type bool, can be used on persistent resources to optionally disable the behavior:

lifecycle {
  persistence = false
}

Trying to use this attribute on a resource that is not persistent would generate an error.

Open Questions

What should terraform taint do with a persistent resource? taint is intended to destroy and recreate, but this is likely to cause a conflict with the resource that was just destroyed, but isn't actually gone. Maybe it should be an error.

What happens if the configuration for a persistent resource is removed from the configuration? It should likely be retained in the state as if it were destroyed, since it still has the same property of not actually having been deleted. A user might be switching between git branches that do and don't have the resource.

References

• [Enhancement]: Support import of resources pending deletion terraform-provider-aws#31468
• Import by declarative code over imperative command #26364

[apparentlymart]

Hi @geekofalltrades! Thanks for sharing this use-case.

This is an interesting situation and technically an example of providers not implementing the providers protocol correctly: a provider should not report that deletion succeeded if the object hasn't actually been deleted at least enough that another object could be created to replace it.

Before we consider complicated new lifecycle behavior like this I think it would help to consider what the options are within the bounds of Terraform's current provider protocol.

The most direct design -- but also perhaps annoying -- would be for the provider to return an error whenever it's asked to plan deleting one of these objects that can't actually be deleted, and thus require the object to be "forgotten" with terraform state rm instead. This seems to solve half of the problem you described, but of course it doesn't deal with the "recreation" step you mentioned.

We've already been considering a new provider protocol capability that could be a different take on this problem with similar characteristics to what you described, which I'll describe here for comparison's sake:

The new idea is to allow providers to respond to PlanResourceChange with an extra annotation that says either that the provider is going to treat a create request as an "adoption" request (automatically take control of an existing object) or it needs to treat a destroy request as a "forget" request (remove the object from the state without destroying it in the remote system).

As you can see, "adopting" is like an automatic import and "forgetting" is like an automatic state rm, both controlled by the provider rather than the user so the provider can more accurately describe situations like taking control over a singleton object that always exists and thus cannot truly be created, or ceding control of such as object after the resource block has been removed.

This seems almost equivalent to what you proposed except that the verbs here are slightly different: instead of "adopt" we might say "reactivate", and instead of "forget" we might say "deactivate". We would of course need to consider how and whether to keep track of "deactivated" objects in the state as you noted... something living on in the state after it's been removed from configuration is unprecedented in Terraform so far, so we would need to consider carefully how to model that so it's clear that the object is still being tracked in some sense by Terraform even though there's no record of it in the most recently-applied configuration.

Another variation would be to require some kind of marker to remain in the configuration to commemorate the deactivated object, which could then be replaced by a normal resource block later if the object needs to be reactivated. This would be similar to my first idea above of returning an error when something can't be destroyed, except that the resolution of the error in this case would be to add the configuration block to mark the object as deactivated, rather than running terraform state rm.

If we did make this idea of "deactivating" explicit in the configuration then it could also work in situations where both deactivating and destroying are possible, such as suspending an EC2 instance instead of destroying it, although that is already technically possible via a resource-level argument today so perhaps this suggests that "deactivating" could be an idea fully implemented inside a provider without Terraform Core being aware of it... from Terraform Core's perspective a deactivated object is still an object that needs to be tracked, so in principle it could just be a normal resource block with a provider-specific setting to say that it should be made inactive.

I think there's a bunch of details to think about here but the overall use-case makes sense to me, so we'll use this issue to track it and gauge interest. If other folks find this issue and have similar use-cases we should consider, please upvote the original issue comment above with 👍 and leave a comment describing the resource type you are using and how it might benefit from explicit modelling of the object being inactive.

[geekofalltrades]

Adoption and forget decisions by the provider sound perfect, and a lot less complicated than my idea. I continued pulling the thread on it after updating this issue last night and was coming up with all sorts of weird edge cases, like how do you handle changes that force recreation of a persistent resource? Do you continue to store a list of all previous IDs of the resource? What if the user changes it back to the previous state? How does the Terraform core decide when the new state is equivalent to the previous state to propose a reimport?

What if the user tries to import a persistent resource to a different address after it has been destroyed?

All questions that the provider seems better equipped to answer.

[mikebranstein]

@apparentlymart I've been waiting for a feature like this so that I don't have to develop and integrate lengthy state removal scripts into our pipeline just to "forget" a resource without deleting it.

We've already been considering a new provider protocol capability that could be a different take on this problem with similar characteristics to what you described, which I'll describe here for comparison's sake:

The new idea is to allow providers to respond to PlanResourceChange with an extra annotation that says either that the provider is going to treat a create request as an "adoption" request (automatically take control of an existing object) or it needs to treat a destroy request as a "forget" request (remove the object from the state without destroying it in the remote system).

[AtelierSnek]

This would also be handy for cases in which a resource shouldn't be deleted in the normal sense (take a versioned artifact that's already in-use, for example), and what we want to indicate is "this is .e.g. deprecated, don't use it", and then "create" a new resource that we're now tracking with terraform. The old resource can be reactivated (to use @apparentlymart's terms) if it's the one we want to use again for whatever reason, but can be deactivated if it's not something terraform should be working on, but should still be notated as extant.

This would make certain actions taken on e.g. artifactory or maven or even a terraform network mirror much easier to manage and lifecycle. Currently, a lot of this can be handled transparently to terraform core (We currently use an optional, computed attribute, along with mangling of requests to ensure that Terraform doesn't clear its state completely even when it asks us to 'Delete' the resource), but it is both difficult to maintain and really frustrating to use/develop.

EDIT: On the flip side, consider the statement "Ideologically, Terraform should not manage such resources". I don't agree with this at face value, because 'Delete' in the context of a provider or use-case is not necessarily the same as "you can forget everything you knew about it".