State snapshots have inconsistent ordering for resources differing only in module address

[a13xb]

When I do terraform refresh, the state file is rewritten each time, randomly reordering all resources, even with -parallelism=1, creating large diffs between the state file and its backup.

Terraform Version

Terraform v0.12.0
+ provider.aws v1.60.0-dev20190216h00-dev

Expected Behavior

Resource order changes arbitrarily.

Actual Behavior

Resources are sorted, so the order is preserved.

Steps to Reproduce

Take any semi-complicated configuration (I have multiple resources in multiple AWS regions) and run terraform refresh.

[a13xb]

I do not see this anymore with v0.12.3 with AWS provider v2.13.0.

[a13xb]

I still see this with v2.13.0, but with Datadog provider v2.0.2. Could be provider-specific?

[sbrandtb]

@a13xb Happens to us as well. It hinders us from using our workflow where we commit the state file to a repository and only commit the relevant changes.

• Terraform 0.12.6
• AWS provider 2.22.0
• Archive provider 1.2.2
• Null provider 2.1.2
• Template provider 2.1.2

[sbrandtb]

@a13xb Could you please reopen this?

[kamilbednarz]

Also experiencing this with:

$ terraform -v
Terraform v0.12.10
+ provider.aws v2.32.0
+ provider.random v2.1.2

[apparentlymart]

Hi all,

I spent a little time trying to reproduce this today. At first I was unable to see it, but I've now figured out what's required: you need to have multiple modules in the configuration that contain resources with the same local address.

For example, I made a test configuration that contains both null_resource.foo and module.child.null_resource.foo. The normalization logic is sorting by the mode, type, and name, but is not considering the module address. Therefore the ordering of those two resources in the state snapshot is undefined and changes between snapshots.

In order to observe this reliably it's also necessary to have something in the configuration that changes each time you run terraform refresh, or else Terraform will see that the state has not changed in any meaningful way and thus skip creating a new state snapshot. I achieved that by putting a fetch from random.org in my configuration as a data resource. The total configuration to reproduce, then, is:

resource "null_resource" "foo" {}

data "http" "a" {
  url = "https://www.random.org/integers/?num=2&min=1&max=6&col=1&base=10&format=plain&rnd=new"
}

module "child" {
  source = "./child"
}

...and then ./child contains just another resource "null_resource" "foo" {} block.

I think the problem is in sortResourcesV4: its Less implementation ought to have an additional initial case for case sr[i].Module != sr[j].Module to sort by the module address before sorting by the module-local identifiers.

Here's a useful jq command to summarize what's in the state. The result is a little messy (lots of null and not standard address syntax) but it's easier to read than the JSON directly:

$ jq -r '.resources[] | . as $r | .instances[] | "\($r.module).\($r.type).\($r.name) \(.index_key)"' < terraform.tfstate
null.http.a null
null.null_resource.foo null
module.child.null_resource.foo null