NodeApplyableProvider Does Not Call ConfigureProvider for Destroy Plan With Deposed Instances

[ewbankkit]

SDK version

% go list -m github.com/hashicorp/terraform-plugin-sdk/...
github.com/hashicorp/terraform-plugin-sdk/v2 v2.8.0 => github.com/gdavison/terraform-plugin-sdk/v2 v2.7.1-0.20210913224932-c7c2dbd9e010

We are running on a forked version of v2.7.1 but have seen this issue on prior unforked versions.

Configuration and steps to repro: hashicorp/terraform-provider-aws#21181.

Terraform Configuration Files

provider "aws" {}

resource "aws_api_gateway_rest_api" "this" {
  name = "Test API"
  body = "invalid OpenAPI spec"
}

resource "aws_api_gateway_deployment" "default" {
  rest_api_id = aws_api_gateway_rest_api.this.id

  triggers = {
    redeployment = sha1("invalid OpenAPI spec")
  }

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_api_gateway_stage" "default" {
  deployment_id = aws_api_gateway_deployment.default.id
  rest_api_id   = aws_api_gateway_rest_api.this.id
  stage_name    = "default"
}

Steps to Reproduce

% terraform apply
...
Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_api_gateway_rest_api.this: Creating...
╷
│ Error: error creating API Gateway specification: BadRequestException: Invalid OpenAPI input.
│ 
│   with aws_api_gateway_rest_api.this,
│   on main.tf line 3, in resource "aws_api_gateway_rest_api" "this":
│    3: resource "aws_api_gateway_rest_api" "this" {
│ 
╵
% terraform apply
...
Plan: 3 to add, 0 to change, 1 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_api_gateway_rest_api.this: Creating...
╷
│ Error: error creating API Gateway specification: BadRequestException: Invalid OpenAPI input.
│ 
│   with aws_api_gateway_rest_api.this,
│   on main.tf line 3, in resource "aws_api_gateway_rest_api" "this":
│    3: resource "aws_api_gateway_rest_api" "this" {
│ 
╵
% terraform destroy
...
aws_api_gateway_rest_api.this: Refreshing state... [id=z9k6kjpm2e]
aws_api_gateway_rest_api.this (deposed object 11ed7aa0): Refreshing state... [id=kttrnchlwa]
aws_api_gateway_rest_api.this (deposed object 11ed7aa0): Refreshing state... [id=kttrnchlwa]
╷
│ Error: Plugin did not respond
│ 
│   with aws_api_gateway_rest_api.this,
│   on main.tf line 3, in resource "aws_api_gateway_rest_api" "this":
│    3: resource "aws_api_gateway_rest_api" "this" {
│ 
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ReadResource call. The plugin logs may contain more details.
╵

Stack trace from the terraform-provider-aws plugin:

panic: interface conversion: interface {} is nil, not *aws.AWSClient

goroutine 69 [running]:
github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsApiGatewayRestApiRead(0xc000776a80, 0x0, 0x0, 0xc976ad0, 0xc000680000)
	/Users/ewbankkit/altsrc/github.com/terraform-providers/terraform-provider-aws/aws/resource_aws_api_gateway_rest_api.go:328 +0x1aca
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0xc000894c40, 0x8eb88c8, 0xc00075e5c0, 0xc000776a80, 0x0, 0x0, 0x0, 0x0, 0x0)
	/Users/ewbankkit/go/pkg/mod/github.com/gdavison/terraform-plugin-sdk/v2@v2.7.1-0.20210913224932-c7c2dbd9e010/helper/schema/resource.go:335 +0x1ee
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).RefreshWithoutUpgrade(0xc000894c40, 0x8eb88c8, 0xc00075e5c0, 0xc00077a540, 0x0, 0x0, 0xc00077e400, 0x0, 0x0, 0x0)
	/Users/ewbankkit/go/pkg/mod/github.com/gdavison/terraform-plugin-sdk/v2@v2.7.1-0.20210913224932-c7c2dbd9e010/helper/schema/resource.go:624 +0x1cb
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadResource(0xc000112828, 0x8eb88c8, 0xc00075e5c0, 0xc00075e600, 0xc00075e5c0, 0x100d165, 0x7d913a0)
	/Users/ewbankkit/go/pkg/mod/github.com/gdavison/terraform-plugin-sdk/v2@v2.7.1-0.20210913224932-c7c2dbd9e010/helper/schema/grpc_provider.go:575 +0x43b
github.com/hashicorp/terraform-plugin-go/tfprotov5/server.(*server).ReadResource(0xc000753120, 0x8eb8970, 0xc00075e5c0, 0xc00076a540, 0xc000753120, 0xc001e60150, 0xc000092ba0)
	/Users/ewbankkit/go/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.3.0/tfprotov5/server/server.go:298 +0x105
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadResource_Handler(0x80885e0, 0xc000753120, 0x8eb8970, 0xc001e60150, 0xc00076a4e0, 0x0, 0x8eb8970, 0xc001e60150, 0xc001e17c00, 0x1e9)
	/Users/ewbankkit/go/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.3.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:344 +0x214
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0004b16c0, 0x8edc8b8, 0xc0022ba600, 0xc00076c200, 0xc001301260, 0xc935370, 0x0, 0x0, 0x0)
	/Users/ewbankkit/go/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1194 +0x52b
google.golang.org/grpc.(*Server).handleStream(0xc0004b16c0, 0x8edc8b8, 0xc0022ba600, 0xc00076c200, 0x0)
	/Users/ewbankkit/go/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1517 +0xd0c
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc0014aecd0, 0xc0004b16c0, 0x8edc8b8, 0xc0022ba600, 0xc00076c200)
	/Users/ewbankkit/go/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:859 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/Users/ewbankkit/go/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:857 +0x1fd

Error: The terraform-provider-aws plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

References

Relates:

• Invalid OpenAPI spec in aws_api_gateway_rest_api leads to plugin crash during destroy state refresh terraform-provider-aws#21181
• plug in crash when doing a terraform apply on EKS. terraform-provider-aws#21140
• Plugin crash on the refreshing state stage terraform-provider-aws#21132
• Error: The terraform-provider-aws_v3.60.0_x5 plugin crashed! Jenkins job failed. terraform-provider-aws#21079
• terraform destroy gives "Error: The terraform-provider-aws_v3.59.0_x5 plugin crashed!" terraform-provider-aws#21019
• "terraform-provider-aws_v3.57.0_x5 plugin crashed" when calling terraform destroy (EKS cluster) terraform-provider-aws#20941
• Terraform Destroy crashed  terraform-provider-aws#20878
• Plugin crashed terraform-provider-aws#20461
• terraform-provider-aws_v3.52.0_x5_plugin_crashed terraform-provider-aws#20438
• Terraform AWS plugin crashes when destroying items terraform-provider-aws#20131
• terraform-provider-aws_v3.47.0_x5 plugin - interface {} is nil, not *aws.AWSClient terraform-provider-aws#20015
• error running terraform 1.0 destroy on local windows env terraform-provider-aws#19807
• aws provider v3.45 crashes during deleting resources terraform-provider-aws#19788
• Crash while destroying state terraform-provider-aws#19740
• Error: Plugin did not respond terraform-provider-aws#19601

[bflad]

Hi @ewbankkit 👋 Thank you so much for submitting this.

This appears to be errant behavior within Terraform CLI for walking a destroy plan. When performing a terraform plan -destroy or terraform destroy there are two graphs that are first: plan and destroy plan. The plan graph was correctly calling the ConfigureProvider RPC before ReadResource, however the destroy plan is not. This appears to not be intended in the case of deposed nodes, which have special considerations in the graph.

I'm going to transfer this issue over to the Terraform CLI repository as there might be a relatively trivial code fix in the graph logic for NodeApplyableProvider (at least for the short term to prevent crashes).

General reproduction steps for Terraform CLI maintainers:

• Create resource that gets saved to state, but errors (so its now tainted)
• Repeat (so now we have a tainted instance and the deposed instance)
• Destroy

Example state prior to destroy:

{
  "version": 4,
  "terraform_version": "1.0.8",
  "serial": 3,
  "lineage": "3273cdc5-8c20-62b8-12f1-a59ea44e7575",
  "outputs": {},
  "resources": [
    {
      "mode": "managed",
      "type": "example_thing",
      "name": "this",
      "provider": "provider[\"registry.terraform.io/hashicorp/example\"]",
      "instances": [
        {
          "status": "tainted",
          "schema_version": 0,
          "attributes": {
            "id": "91m255pfj0"
          },
          "sensitive_attributes": [],
          "private": "bnVsbA==",
          "create_before_destroy": true
        },
        {
          "status": "tainted",
          "deposed": "31a2bd47",
          "schema_version": 0,
          "attributes": {
            "id": "x32esw9495"
          },
          "sensitive_attributes": [],
          "private": "bnVsbA==",
          "create_before_destroy": true
        }
      ]
    }
}

[bflad]

Just for some additional context for folks who may see this issue.

It was unintentionally introduced as part of #28634, which was released with Terraform CLI v0.15.4. As NodePlanDeposedResourceInstanceObject now includes provider side calls to refresh the deposed instance state and these nodes are included during destroy plans, the additional handling for calling ConfigureProvider is necessary before those are executed, otherwise (for most providers) the Terraform Plugin SDK provider instance will not be ready before resource operations.

A proposed quick fix is adding walkPlanDestroy to the NodeApplyableProvider Execute() method to ensure when that node is executed, that ConfigureProvider is called.

---

Using the -refresh=false flag (not typically recommended) may workaround the issue, however there can be other problems when attempting this approach. For example, provider resources that do not appropriately handle missing remote resource errors during deletion if they have already been deleted. In other cases, Terraform CLI may return missing deposed state errors during the destroy which can remove those resources from the Terraform state, while the remote resources still exist.

[Toilal]

Using the -refresh=false flag (not typically recommended) may workaround the issue, however there can be other problems when attempting this approach

This did the job for me, after reading almost all issues related to this crash :)

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.