Terraform Show fails if plan has no changes

[dnascimento]

Terraform Version

Terraform v1.0.1 (not in v1.0.0)

Crash Output

2021-06-26T14:40:49.232+1000 [TRACE] provider.stdio: waiting for stdio data
2021-06-26T14:40:49.303+1000 [TRACE] GRPCProvider: Close
2021-06-26T14:40:49.304+1000 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-06-26T14:40:49.306+1000 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/aws/3.47.0/darwin_amd64/terraform-provider-aws_v3.47.0_x5 pid=20348
2021-06-26T14:40:49.306+1000 [DEBUG] provider: plugin exited
2021-06-26T14:40:49.306+1000 [TRACE] terraform.NewContext: complete
2021-06-26T14:40:49.307+1000 [TRACE] backend/local: finished building terraform.Context

panic: value is marked, so must be unmarked first

goroutine 1 [running]:
github.com/zclconf/go-cty/cty.Value.assertUnmarked(...)
	github.com/zclconf/go-cty@v1.8.4/cty/marks.go:141
github.com/zclconf/go-cty/cty.Value.LengthInt(0x38c2e30, 0xc0005e4000, 0x3159ee0, 0xc0004587e0, 0x38c2e01)
	github.com/zclconf/go-cty@v1.8.4/cty/value_ops.go:1050 +0x4f
github.com/zclconf/go-cty/cty.transform(0xc0005e4960, 0x1, 0x1, 0x38c2e30, 0xc0005e4000, 0x3159ee0, 0xc0004587e0, 0x388c048, 0xc000458b10, 0xc0004587e0, ...)
	github.com/zclconf/go-cty@v1.8.4/cty/walk.go:144 +0x2e8
github.com/zclconf/go-cty/cty.transform(0x0, 0x0, 0x0, 0x38c2df8, 0xc0005e4240, 0x3065f60, 0xc001a9d380, 0x388c048, 0xc000458b10, 0xc0005e48b0, ...)
	github.com/zclconf/go-cty@v1.8.4/cty/walk.go:209 +0x10ed
github.com/zclconf/go-cty/cty.TransformWithTransformer(...)
	github.com/zclconf/go-cty@v1.8.4/cty/walk.go:125
github.com/zclconf/go-cty/cty.Value.MarkWithPaths(0x38c2df8, 0xc0005e4240, 0x3065f60, 0xc001a9d380, 0xc000fc4660, 0x1, 0x1, 0x38221c0, 0x1, 0xc001929640, ...)
	github.com/zclconf/go-cty@v1.8.4/cty/marks.go:236 +0xc5
github.com/hashicorp/terraform/internal/command/jsonstate.marshalResources(0xc001b0d4d0, 0x0, 0x0, 0x0, 0xc00063efd0, 0x0, 0xc00108f410, 0x13dc26c, 0xc00108f4a0, 0x34366ba)
	github.com/hashicorp/terraform/internal/command/jsonstate/state.go:333 +0xe07
github.com/hashicorp/terraform/internal/command/jsonstate.marshalRootModule(0xc000eec0a0, 0xc00063efd0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	github.com/hashicorp/terraform/internal/command/jsonstate/state.go:200 +0x12d
github.com/hashicorp/terraform/internal/command/jsonstate.(*state).marshalStateValues(0xc001a9c1e0, 0xc000eec0a0, 0xc00063efd0, 0x0, 0x0)
	github.com/hashicorp/terraform/internal/command/jsonstate/state.go:166 +0x14a
github.com/hashicorp/terraform/internal/command/jsonstate.Marshal(0xc001b0d3e0, 0xc00063efd0, 0x0, 0x0, 0x0, 0x0, 0x2da4e55)
	github.com/hashicorp/terraform/internal/command/jsonstate/state.go:146 +0xf8
github.com/hashicorp/terraform/internal/command/jsonplan.Marshal(0xc0005ca000, 0xc000dd3a40, 0xc001b0d3e0, 0xc00063efd0, 0x0, 0x0, 0x0, 0x0, 0x0)
	github.com/hashicorp/terraform/internal/command/jsonplan/plan.go:152 +0x633
github.com/hashicorp/terraform/internal/command.(*ShowCommand).Run(0xc00048b6c0, 0xc00004e0e0, 0x1, 0x2, 0xc00058e590)
	github.com/hashicorp/terraform/internal/command/show.go:152 +0x966
github.com/mitchellh/cli.(*CLI).Run(0xc000837400, 0xc00049c060, 0x3435241, 0x4)
	github.com/mitchellh/cli@v1.1.2/cli.go:262 +0x41a
main.wrappedMain(0x0)
	github.com/hashicorp/terraform/main.go:384 +0x1112
main.realMain(0x0)
	github.com/hashicorp/terraform/main.go:115 +0x554
main.main()
	github.com/hashicorp/terraform/main.go:58 +0x25

Steps to Reproduce

terraform plan -out=plan.out
terraform show -json plan.out

Regression introduced by v1.0.1

[DotNetSparky]

I can confirm I get the same error even if the plan has changes (same situation: trying to generate a JSON version of a saved plan file).

I can do a non-JSON show (e.g. "terraform show plan.out") but the crash occurs when including the '-json' option.

[m-torin]

This error likes to show up with Google Cloud Run, where constant small meta tweaks by GCP means the state will have something to change every run. I'm also noticing it with GCP domain mapping and waiting for SSL certificates.

[blakemorgan]

I can confirm the same error using the AWS provider.

Debug output:

2021-06-28T08:04:52.742-0600 [DEBUG] [aws-sdk-go] {"Item":{"Digest":{"S":"*****************"},"LockID":{"S":"terraform-state-storage-********/app/dev/app.tfstate-md5"}}}
2021-06-28T08:04:52.774-0600 [DEBUG] backend/local: skipping refresh of managed resources
2021-06-28T08:04:52.877-0600 [DEBUG] created provider logger: level=debug
2021-06-28T08:04:52.877-0600 [INFO]  provider: configuring client automatic mTLS
2021-06-28T08:04:52.905-0600 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/hashicorp/aws/3.47.0/darwin_amd64/terraform-provider-aws_v3.47.0_x5 args=[.terraform/providers/registry.terraform.io/hashicorp/aws/3.47.0/darwin_amd64/terraform-provider-aws_v3.47.0_x5]
2021-06-28T08:04:52.915-0600 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/hashicorp/aws/3.47.0/darwin_amd64/terraform-provider-aws_v3.47.0_x5 pid=12169
2021-06-28T08:04:52.916-0600 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/hashicorp/aws/3.47.0/darwin_amd64/terraform-provider-aws_v3.47.0_x5
2021-06-28T08:04:52.987-0600 [INFO]  provider.terraform-provider-aws_v3.47.0_x5: configuring server automatic mTLS: timestamp=2021-06-28T08:04:52.986-0600
2021-06-28T08:04:53.027-0600 [DEBUG] provider.terraform-provider-aws_v3.47.0_x5: plugin address: address=/var/folders/h0/k8l81wx974j8bc_hvb41mp140000gp/T/plugin722281452 network=unix timestamp=2021-06-28T08:04:53.027-0600
2021-06-28T08:04:53.027-0600 [DEBUG] provider: using plugin: version=5
2021-06-28T08:04:53.139-0600 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-06-28T08:04:53.142-0600 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/aws/3.47.0/darwin_amd64/terraform-provider-aws_v3.47.0_x5 pid=12169
2021-06-28T08:04:53.142-0600 [DEBUG] provider: plugin exited
2021-06-28T08:04:53.142-0600 [DEBUG] created provider logger: level=debug
2021-06-28T08:04:53.142-0600 [INFO]  provider: configuring client automatic mTLS
2021-06-28T08:04:53.175-0600 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/hashicorp/local/2.1.0/darwin_amd64/terraform-provider-local_v2.1.0_x5 args=[.terraform/providers/registry.terraform.io/hashicorp/local/2.1.0/darwin_amd64/terraform-provider-local_v2.1.0_x5]
2021-06-28T08:04:53.178-0600 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/hashicorp/local/2.1.0/darwin_amd64/terraform-provider-local_v2.1.0_x5 pid=12170
2021-06-28T08:04:53.178-0600 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/hashicorp/local/2.1.0/darwin_amd64/terraform-provider-local_v2.1.0_x5
2021-06-28T08:04:53.197-0600 [INFO]  provider.terraform-provider-local_v2.1.0_x5: configuring server automatic mTLS: timestamp=2021-06-28T08:04:53.196-0600
2021-06-28T08:04:53.230-0600 [DEBUG] provider.terraform-provider-local_v2.1.0_x5: plugin address: address=/var/folders/h0/k8l81wx974j8bc_hvb41mp140000gp/T/plugin912420609 network=unix timestamp=2021-06-28T08:04:53.230-0600
2021-06-28T08:04:53.230-0600 [DEBUG] provider: using plugin: version=5
2021-06-28T08:04:53.268-0600 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-06-28T08:04:53.269-0600 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/local/2.1.0/darwin_amd64/terraform-provider-local_v2.1.0_x5 pid=12170
2021-06-28T08:04:53.269-0600 [DEBUG] provider: plugin exited
panic: value is marked, so must be unmarked first

goroutine 1 [running]:
github.com/zclconf/go-cty/cty.Value.assertUnmarked(...)
        /Users/distiller/go/pkg/mod/github.com/zclconf/go-cty@v1.8.4/cty/marks.go:141
github.com/zclconf/go-cty/cty.Value.LengthInt(0x38c25b0, 0xc0011c3fe0, 0x31596e0, 0xc001c5dd40, 0x38c2501)
        /Users/distiller/go/pkg/mod/github.com/zclconf/go-cty@v1.8.4/cty/value_ops.go:1050 +0x4f
github.com/zclconf/go-cty/cty.transform(0xc00161f840, 0x3, 0x4, 0x38c25b0, 0xc0011c3fe0, 0x31596e0, 0xc001c5dd40, 0x388b7c8, 0xc001cbaba0, 0xc001c5dd40, ...)
        /Users/distiller/go/pkg/mod/github.com/zclconf/go-cty@v1.8.4/cty/walk.go:144 +0x2e8
github.com/zclconf/go-cty/cty.transform(0xc0024e2a80, 0x2, 0x2, 0x38c2578, 0xc000924090, 0x3065760, 0xc0005ca210, 0x388b7c8, 0xc001cbaba0, 0xc0009240c0, ...)
        /Users/distiller/go/pkg/mod/github.com/zclconf/go-cty@v1.8.4/cty/walk.go:209 +0x10ed
github.com/zclconf/go-cty/cty.transform(0xc000925610, 0x1, 0x1, 0x38c2508, 0xc0009240c0, 0x2f46680, 0xc001c5de30, 0x388b7c8, 0xc001cbaba0, 0xc001c5de30, ...)
        /Users/distiller/go/pkg/mod/github.com/zclconf/go-cty@v1.8.4/cty/walk.go:156 +0x57f
github.com/zclconf/go-cty/cty.transform(0x0, 0x0, 0x0, 0x38c2578, 0xc000924100, 0x3065760, 0xc0005ca2d0, 0x388b7c8, 0xc001cbaba0, 0xc000925060, ...)
        /Users/distiller/go/pkg/mod/github.com/zclconf/go-cty@v1.8.4/cty/walk.go:209 +0x10ed
github.com/zclconf/go-cty/cty.TransformWithTransformer(...)
        /Users/distiller/go/pkg/mod/github.com/zclconf/go-cty@v1.8.4/cty/walk.go:125
github.com/zclconf/go-cty/cty.Value.MarkWithPaths(0x38c2578, 0xc000924100, 0x3065760, 0xc0005ca2d0, 0xc0015c7300, 0x3, 0x4, 0x3821940, 0x1, 0xc002399bb0, ...)
        /Users/distiller/go/pkg/mod/github.com/zclconf/go-cty@v1.8.4/cty/marks.go:236 +0xc5
github.com/hashicorp/terraform/internal/command/jsonstate.marshalResources(0xc00149d1a0, 0xc00187c040, 0x1, 0x1, 0xc001603a90, 0x100f39b, 0xc0014611a0, 0x18, 0x18, 0x3365ba0)
        /Users/distiller/project/project/internal/command/jsonstate/state.go:333 +0xe07
github.com/hashicorp/terraform/internal/command/jsonstate.marshalModules(0xc000790780, 0xc001603a90, 0xc0014611a0, 0x1, 0x1, 0xc000b1b1a8, 0x0, 0x1, 0x0, 0x0, ...)
        /Users/distiller/project/project/internal/command/jsonstate/state.go:254 +0x538
github.com/hashicorp/terraform/internal/command/jsonstate.marshalRootModule(0xc000790780, 0xc001603a90, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /Users/distiller/project/project/internal/command/jsonstate/state.go:234 +0x8c9
github.com/hashicorp/terraform/internal/command/jsonstate.(*state).marshalStateValues(0xc00147b4d0, 0xc000790780, 0xc001603a90, 0x0, 0x0)
        /Users/distiller/project/project/internal/command/jsonstate/state.go:166 +0x14a
github.com/hashicorp/terraform/internal/command/jsonstate.Marshal(0xc00149c1e0, 0xc001603a90, 0x0, 0x0, 0x0, 0x0, 0x2da4655)
        /Users/distiller/project/project/internal/command/jsonstate/state.go:146 +0xf8
github.com/hashicorp/terraform/internal/command/jsonplan.Marshal(0xc000813790, 0xc001ad3d60, 0xc00149c1e0, 0xc001603a90, 0x0, 0x0, 0x0, 0x0, 0x0)
        /Users/distiller/project/project/internal/command/jsonplan/plan.go:152 +0x633
github.com/hashicorp/terraform/internal/command.(*ShowCommand).Run(0xc000227340, 0xc00004e0b0, 0x1, 0x2, 0xc0008aeb60)
        /Users/distiller/project/project/internal/command/show.go:152 +0x966
github.com/mitchellh/cli.(*CLI).Run(0xc0008ea140, 0xc0009814a0, 0x3434a41, 0x4)
        /Users/distiller/go/pkg/mod/github.com/mitchellh/cli@v1.1.2/cli.go:262 +0x41a
main.wrappedMain(0x0)
        /Users/distiller/project/project/main.go:384 +0x1112
main.realMain(0x0)
        /Users/distiller/project/project/main.go:115 +0x554
main.main()
        /Users/distiller/project/project/main.go:58 +0x25

[davidalger]

Pretty sure this has to do with changes introduced in #28889. We updated a test branch to run Terraform (v1.0.1) plans on all of our pipelines and only a handful of our stacks crash with this new panic: value is marked, so must be unmarked first error but each one which does crash has the exact same trace as the OP. Our CI plan jobs show the JSON plan to print a summarized list of changed resources, so this bug completely fails the normal CI operations on affected pipelines. In every case though, showing the plan normally, i.e. terraform show plan.cache works as expected but terraform show -json plan.cache panics.

Notably this also doesn't appear to be specific to using a cached plan. The same environments for which JSON output of a cached plan fails will also fail when running terraform show -json leading me to believe some edge case in the state is triggering the bug.

[mildwonkey]

Thanks for reporting this, and sorry for the bad behavior!

It was easy enough to find where these panics occur, but I'm having trouble writing a test that can reproduce the issue. @blakemorgan or @dnascimento , can either of you share your configuration, or (better yet) provide a simplified reproduction case? It would help me to have an example I can run locally to confirm what's happening.

---

For a bit of an explanation:

The panicking code is the last line, here:
https://github.com/hashicorp/terraform/blob/main/internal/command/jsonstate/state.go#L326-L334

The call to ResourceInstanceObjectSrc.Decode (at the top of the snippet above) would have already re-applied any marks, so it seems like a redundant call. However writing a test with marked attribute values isn't enough to recreate the panic; I could be miswriting the test, or the problem might require some other element to reproduce.

[mildwonkey]

I've opened a PR that removes the redundancy from the jsonstate resource marshaling - the top of the stack trace - but I wasn't actually able to coerce the panic in a test. So while removing the redundancy is a good idea regardless, I'd still like to know what actual confluence of events causes the panic so I can extend the test cases and confirm that this fixes all of the issue, and not just a symptom.

If anyone experiencing this issue is interested and willing, I would greatly appreciate it if you could build terraform from my branch (in #29049) and see if it fixes the panic in terraform show -json planfile. Thanks!

[apparentlymart]

I've not dug in deep here so I'm not sure if this is actually relevant but I see that there's a call to cty.TransformWithTransformer in these stack traces and wanted to note that for an unrelated reason last week I pushed support for marks in the walk/transform functions which I think would make these situations not panic anymore.

I've not yet included it in a release of my cty library but if we can find some way to test with that cty commit in place and see that it helped then I'd be happy to expedite a cty release to help get this panic fixed.