Incremental backend persistence during Terraform operations

[alexng-canuck]

Current Terraform Version

terraform v0.11.7

Use-cases

In the course of using remote backends (such as HTTP), Terraform may crash or otherwise be forced to exit in the middle of an "apply" or "destroy" operation.

Currently, backend Puts are generally invoked only once at the end of such operations. This can lead to dangling of any resources that were created before an unexpected Terraform exit occurred.

Is it possible to have incremental backend Put (or persistence) of remote state to guard against such cases?

The timing of the Put calls might be analogous to the WriteState calls that occur when using a local backend that writes to a file.

Attempted Solutions

[apparentlymart]

Hi @alexng-canuck! Thanks for sharing this idea.

We attempted this a while back and had to revert it due to some synchronization bugs that were present at the time. The state-handling changes in v0.12 should've addressed the concerns we ran into there, so we are likely to try this again in a future release.

The behavior we were trying for before -- which I think is still what we want -- was to have a periodic interval during the apply phase where Terraform will take a snapshot of the state, check whether it's changed since the last snapshot, and persist it to the backend if so.

Although I think this behavior ought to be enabled by default since it's the safest behavior, I expect we'll need to provide mechanisms to disable it or customize the save frequency for those who are concerned about rate-limit or cost impacts of those partial writes. Once we get to implementing this again we'll need to see what's the best tradeoff for this to avoid adding lots of new complexity to the user experience.

We are planning to have general backend architecture improvements as a theme for a future major release, since we've collected quite a few of them and doing them all together allows us to amortize testing overhead, reduce context-switching, etc. I've labelled this one so we'll find it when planning that later release.

[brikis98]

I wanted to add a BIG +1 for this request. Having Terraform completely fail to save state—because of a connectivity issue, or the CI server it's running on crashing, or a Terraform bug that causes a panic, or simply hitting CTRL+C by accident—especially when deploying something new or a large batch of changes, is very painful. There's no easy way to recover from it, other than manually figuring out what had gotten deployed, and running terraform import over and over, possibly hundreds of times.

An incremental save mechanism would be invaluable here. In fact, I'd argue that each time the internal state representation changes, it should be saved, or at most once every few seconds. This would reduce the "blast radius" of Terraform failing to save state to a tiny number of resources, making it easier to recover. It would also remove the pain of having an apply running that you know will fail (e.g., due to some bug or misconfig), but having to wait for some timeout to be hit (which is sometimes an hour or more), because if you hit CTRL+C, you'll lose state.

If you're using a backend with locking, I would hope that saving the state file incrementally shouldn't cause any concurrency issues as long as you have the exclusive lock (and if you don't have locking on your backend, then errors when running things concurrently are to be expected), but I haven't dug into the implementation details, so perhaps there is more complexity there. At any rate, I'd list this functionality as one of my top 3 most desired Terraform improvements. Thanks for listening :)

[afallucc]

I know this one has been sitting a bit, was curious if any traction on the subject has occurred?

[kawsark]

@apparentlymart could you provide an overview (or link to documentation) for terraform 0.12 state handling changes? If a busy build or TFE server crashes, this could lead to "lost" state for any in-progress Runs.

The state-handling changes in v0.12 should've addressed the concerns we ran into there, so we are likely to try this again in a future release.

[stephanpelikan]

I've also run into it several times: If you use modules then a lot of tiny things are created. If one part within the module fails or another part of my scripts or even another module a lot of things has to be cleaned up before doing the next attempt. If this fails again because the fix was wrong or for another reason then it starts all over again. Importing the state of those successfully created fragments is also not really an option because its pretty cumbersome.

Are there any news regarding this topic / feature request?

[bondsb]

This tends to happen occasionally in our CI/CD pipelines. Sometimes it's due to a timeout in GitLab.

When it happens, I need to delete a few dozen resources in order to get the pipeline running again. Fixing this so that only the one long-running resource is in conflict would be a major time saver.

[andreamaruccia]

Is this planned in any release?

[youssef-chtourou]

Is this planned in any release?

[VincentGaoHJ]

Any update? For now, do we really have to do the ugly workarounds such as:

Manually find everything that got deployed, delete it, and re-run terraform apply?

[ynyway]

Bumping this thread for added traction and eyes. Is anyone working on fixing this issue from 3 years ago that still persists today? @apparentlymart