When Storing the Terraform State, commit updates more often ( configurable? )

[TechnicalMercenary]

Current Terraform Version

Terraform v0.12.21
+ provider.aws v2.51.0
+ provider.http v1.1.1
+ provider.local v1.4.0
+ provider.null v2.1.2
+ provider.template v2.1.2

Use-cases

Occasionally in our environment, our network connectivity fails. This causes terraform to fail as well, and prevent TF from updating the remote state in S3.

EDIT: For this case, we are executing a terraform project inside of a Docker container, either as a docker run or a k8s pod. When terraform is running and we lose the network connection to the container's interactive shell, the container is unceremoniously and immediately destroyed. Because of docker container being terminated, TF doesn't know it's being killed as well and doesn't have an opportunity to write any state.

When we restart Terraform, we receive errors like "Resource <> already exists". If terraform had been saving the remote state more often then TF shouldn't be 'confused' and will know not to re-create and already existing resource that it had created on the previous run.

If there has been any, more current, state stored in the file location where terraform is executing, that's lost as well, because the docker container ( where this all runs ) is also lost when we lose a network connection.

Attempted Solutions

I've tried monitoring the folder where TF is running to see if I can capture intermediate state files, additionally I've scanned the docs to see if there is any information or configuration on how state is written remotely.

Proposal

I would be interested in knowing, if after any "Creation complete" or "Destruction complete" message an updated state file could be pushed. I understand that any multi threading might make this challenging etc.

[jbardin]

Hi @TechnicalMercenary,

Even if state were updated more frequently, because providers (and the APIs with which they interact) are not transactional, there's no way to ensure all resources could be recorded in state at any moment, and there is always a period of time that complete failure would result in loss of information. We understand reducing this window has benefits, but there are many complications to doing so across all state storage backends.

While improving how state is handled in general is a long term goal, you should not lose state due to a backend connection failure. In the rare event that writing the remote state fails, an errored.tfstate should be recorded locally for manual recovery. If there is a situation where writing to s3 fails yet we don't record the state somewhere, that should filed as a bug.

[AeroNotix]

@TechnicalMercenary you could also execute your terraform changes in fewer steps such that you are not waiting on a large amount of resources to be created. It could help if your network is poor and you're only creating a handful of resources.

[TechnicalMercenary]

@jbardin : I'm not expecting perfection just 'more often'. In this situation, there is no errored.tfstate available.

@AeroNotix : I would prefer not to refactor 10,000 lines of Terraform, but it is certainly an option for some of the more critical errors.

thank you

[jbardin]

@TechnicalMercenary,

If you have any trace logs from that happening, it would be exceptionally useful. We have not seen a case where the state isn't written somewhere at all.

[TechnicalMercenary]

Sorry, See my edit above in the ticket details. I thought I had described the use case

[speller]

We're using GitLab runners autoscaled on AWS and state storage on S3. Very rare, but it still may happen that the runner gets killed by an underlying instance failure or job timeout. In this case, the state is completely lost. ANY (literally, ANY) intermediate state stored anywhere would be super helpful. We now have more than 100 resources managed by Terraform. If it fails, we have to clean up everything manually.

If you'll create the error.tfstate file locally it could be pulled by CI as an artifact and it would be possible to use it. But now there's no way and I think it's not a good situation.

[bondsb]

There is a previously existing request for a similar use case: #20718

[speller]

Today I faced this issue again. One of the provisioners hangs in the middle of the apply process. And viola, seeing a timeout and the lost state. When this annoying issue of such a great tool will be solved? Just add an option to save state after every resource application. Nothing difficult. But will save a lot of time and nerves for users.

[speller]

@jbardin

Even if state were updated more frequently, because providers (and the APIs with which they interact) are not transactional, there's no way to ensure all resources could be recorded in state at any moment

Just allow this feature and let providers implement it on their own. As soon as a provider implements it - it will be available for users of that specific provider. The most popular ones will implement it quickly.

and there is always a period of time that complete failure would result in loss of information.

First, cover the most common cases. Even if a couple of resources will be lost it's 100% better than the completely lost state. This issue is the biggest pain point in my TF experience.

We understand reducing this window has benefits

Fine, go ahead!

but there are many complications to doing so across all state storage backends.

As I said above, no need to implement it in every provider. Only in most popular first.

While improving how state is handled in general is a long term goal, you should not lose state due to a backend connection failure.

There can be plenty of possibilities to break the deployment process and lose the state. Don't assume should not, just allow users to be backed up.

In the rare event that writing the remote state fails, an errored.tfstate should be recorded locally for manual recovery. If there is a situation where writing to s3 fails yet we don't record the state somewhere, that should filed as a bug.

Should we file an additional issue for remote backends like S3?

[rpadovani]

I confirm what others have already reported: for reasons outside Terraform control, there could be a huge failure and no Terraform state could be written, nor errored.tfstate retrieved.

In my case, today, I was deploying a new project with more than 200 resources and modules, and the K8s node on which the CI/CD pod was running failed spectacularly, and now I have an account with ~90 resources/modules that I have to clean up. They are well-tagged, and it is not so much of a problem, but being able to restart from an intermediate point would be easier, 'cause some resources take a long time to be provisioned / unprovisioned.

[slalomark]

Echoing the sentiment here. Running Terraform through CI, if a 'job' is cancelled for whatever reason, the next run through usually means Terraform will fail to create due to duplicate resource names and whatnot since the original cancellation prevented state from being updated. The manual cleanup this forces really makes using TF a risk in CI. This is a pretty serious issue that I'm surprised to see not addressed after so many years... I saw an open issue about this dated in 2017!

[speller]

Any plans to work on this? Just faced this issue again during a heavy migration executed in a provisioning script. It was creating a new infra and timed-out in a CI/CD worker. Thus having a lot of resources out of control, no state was saved at all. This is really weird...