Add support for force pushing with the remote backend

[leetrout]

This PR adds support for the -force flag for state push with the remote backend. Per the documentation around manual state push / pull https://www.terraform.io/docs/backends/state.html#manual-state-pull-push both differing serials and lineage protections can be bypassed with the -force flag.

• The first change you'll notice is the addition of a forcePush property on the remoteClient struct. This allows the client to know if it is operating in the context of a force push.

• The second change is an optional interface ClientForcePusher for clients (like the remoteClient) that would need to know they are operating in the context of a force push. There was a force flag added to the Go TFE client that needs to be specified at persistence time.

• The last change is tracking the serial and lineage in addition to the state. As noted above, lineage and serial can be overwritten with -force so we must look at them to determine if we can exit early from writing state.

To prevent changing every method signature of PersistState to accept force bool I added the optional interface. That provides a hook to flag the Client as operating in a force push context. Changing the method signature would be more explicit at the cost of not being used anywhere else currently or the optional interface pattern could be applied to the state itself so it could be upgraded to support PersistState(force bool) only when needed. Let me know if either of these seem preferable instead!

Background

The Go TFE client received support for a Force attribute in March 2019 hashicorp/go-tfe#63 however that was not brought over into Terraform via the remote backend. It is possible it worked until changes were made to prevent sending snapshots unless the resources had changed (otherwise the function exits early) 85eda8a

Existing remote state behaviors

I’ve compiled examples of the behavior of two backends, Postgres and S3 and how they currently work with and without the force flag. Lineages and serials are both flagged when trying to push but are correctly overridden when using the force flag as evidence by the diff when pulling after the force push.

https://gist.github.com/leetrout/51e605000fa1f59d1f6902023d685c8c

Manual testing

Configure Terraform Cloud

1. Setup an organization
2. Setup a workspace

Add Terraform config

terraform {
  backend "remote" {
    hostname = "<Terraform Cloud/Enterprise>"
    organization = "terraform-testing"

    workspaces {
      name = "testing"
    }
  }
}

resource "random_integer" "rando" {
  min     = 1
  max     = 50000
}

Create resource

1. init
2. apply

Terraform will perform the following actions:

  # random_integer.rando will be created
  + resource "random_integer" "rando" {
      + id     = (known after apply)
      + max    = 50000
      + min    = 1
      + result = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions in workspace "testing"?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

random_integer.rando: Creating...
random_integer.rando: Creation complete after 0s [id=23721]

Mutate lineage

In this case, change all 1's to 2's in the lineage:

terraform state pull > remote.tfstate
jq ".lineage=\"$(jq -r '.lineage' remote.tfstate | sed 's/1/2/g')\"" remote.tfstate > edited.tfstate

Confirm what changed with diff remote.tfstate edited.tfstate

5c5
<   "lineage": "dd577646-0d5c-d9ba-93b2-d2e04881ebb7",
---
>   "lineage": "dd577646-0d5c-d9ba-93b2-d2e04882ebb7",
30d29
<

Pushing

Try pushing (without -force) to ensure it doesn't work.

terraform state push edited.tfstate

Failed to write state: cannot import state with lineage "dd577646-0d5c-d9ba-93b2-d2e04882ebb7" over unrelated state with lineage "dd577646-0d5c-d9ba-93b2-d2e04881ebb7"

Now add the force flag:

terraform state push -force edited.tfstate

There should be no output but exit status of 0 and you should see the new state in TFC.

[codecov]

Codecov Report

Merging #24696 into master will increase coverage by 0.03%.
The diff coverage is 88.23%.

Impacted Files	Coverage Δ
state/remote/remote.go	0.00% <ø> (ø)
backend/remote/backend_state.go	57.31% <33.33%> (-0.92%)	⬇️
state/remote/state.go	67.30% <100.00%> (+19.43%)	⬆️
dag/walk.go	92.54% <0.00%> (-0.63%)	⬇️
terraform/evaluate.go	53.15% <0.00%> (+0.45%)	⬆️

[pselle]

Some questions. As well, could you look into adding tests for this change? Both for the enhanced remote backend, and some of the changes you've made (that I think are great) that ensure a more granular checking of state when deciding whether/not to push the state.

[pselle]

I'm a bit confused why the additional interface is needed, since there already is this bool related to force. Wouldn't a general else be what is needed?

[leetrout]

Open to all suggestions on how to make the comment in L68-L71 make more sense 🙏

The CLI snags the flag and drops it in this function where the s.Client is just the Client interface so L72 checks to see if the optional ClientForcePusher is implemented.

Would you rather change it to inline the check and remove the prior check and descriptive isForcePusher for the if-ok pattern?

else if c, ok := s.Client.(ClientForcePusher); ok {
	c.EnableForcePush()
}

[pselle]

I appreciate the addition of these more granular checks -- we'll probably want to note this in the changelog as part of the changes included in this changeset (more specific checks for whether state needs to be updated)

[leetrout]

👍 will do

[pselle]

This looks great! One comment about a comment (that "Implements.." pattern we talked about, but this looks really good.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.