terraform-provider-aws_v3.47.0_x5 plugin - interface {} is nil, not *aws.AWSClient

[Erucolindo96]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v1.0.1
on linux_amd64
+ provider registry.terraform.io/hashicorp/archive v2.2.0
+ provider registry.terraform.io/hashicorp/aws v3.47.0
+ provider registry.terraform.io/hashicorp/null v3.1.0

OS: Ubuntu 18.04, but on Windows there are same effect

Affected Resource(s)

Terraform Configuration Files

• Propably does not depends on config. When comment some failed code, the other is affected. I can upload code if nessecery.
• Terraform state shared betweent devs in S3 bucket, with DynamoDB locking table :
  terraform { backend "s3" { bucket = "bucket_name" key = "local/terraform.tfstate" region = "eu-central-1" dynamodb_table = "table_name" } }

Debug Output

Panic Output

Output at terraform destroy (only plan were executed, even no asking for confirmation to delete resources, crash before them)

│ Error: Request cancelled
│ 
│   with module.api_users.module.api_users_method_services_ls_fp_csrs_get.aws_api_gateway_integration_response.api_method_response_integration[3],
│   on modules/api_lambda_method/main.tf line 33, in resource "aws_api_gateway_integration_response" "api_method_response_integration":
│   33: resource "aws_api_gateway_integration_response" "api_method_response_integration" {
│ 
│ The plugin.(*GRPCProvider).UpgradeResourceState request was cancelled.
╵
╷
│ Error: Request cancelled
│ 
│   with module.api_users.module.api_users_method_services_ls_fp_csrs_info_get.aws_api_gateway_integration_response.api_method_response_integration[6],
│   on modules/api_lambda_method/main.tf line 33, in resource "aws_api_gateway_integration_response" "api_method_response_integration":
│   33: resource "aws_api_gateway_integration_response" "api_method_response_integration" {
│ 
│ The plugin.(*GRPCProvider).UpgradeResourceState request was cancelled.
╵
╷
│ Error: Request cancelled
│ 
│   with module.api_users.module.api_users_method_account_billing_snapshots.aws_api_gateway_integration_response.api_method_response_integration[5],
│   on modules/api_lambda_method/main.tf line 33, in resource "aws_api_gateway_integration_response" "api_method_response_integration":
│   33: resource "aws_api_gateway_integration_response" "api_method_response_integration" {
│ 
│ The plugin.(*GRPCProvider).UpgradeResourceState request was cancelled.
╵
╷
│ Error: Request cancelled
│ 
│   with module.api_users.module.api_users_method_services_ls_fp_install_post.aws_api_gateway_integration_response.api_method_response_integration[5],
│   on modules/api_lambda_method/main.tf line 33, in resource "aws_api_gateway_integration_response" "api_method_response_integration":
│   33: resource "aws_api_gateway_integration_response" "api_method_response_integration" {
│ 
│ The plugin.(*GRPCProvider).UpgradeResourceState request was cancelled.
╵
╷
│ Error: Plugin did not respond
│ 
│   with module.billing_snapshots.module.create_drm_billing_snapshots_lambda.aws_lambda_function.lambda,
│   on modules/lambda/main.tf line 5, in resource "aws_lambda_function" "lambda":
│    5: resource "aws_lambda_function" "lambda" {
│ 
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).UpgradeResourceState call. The plugin logs may contain more details.
╵
╷
│ Error: Plugin did not respond
│ 
│   with module.api_fairplay.module.api_fairplay_settings_cors.aws_api_gateway_method_response.cors_method_response,
│   on .terraform/modules/api_fairplay.api_fairplay_settings_cors/main.tf line 21, in resource "aws_api_gateway_method_response" "cors_method_response":
│   21: resource "aws_api_gateway_method_response" "cors_method_response" {
│ 
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).UpgradeResourceState call. The plugin logs may contain more details.
╵

Stack trace from the terraform-provider-aws_v3.47.0_x5 plugin:

panic: interface conversion: interface {} is nil, not *aws.AWSClient

goroutine 208 [running]:
github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsCognitoUserPoolRead(0xc001f54880, 0x0, 0x0, 0xb4d97e0, 0xc000100800)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-aws/aws/resource_aws_cognito_user_pool.go:804 +0x29ea
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0xc0008ff180, 0x7cdb5c8, 0xc000a7c2c0, 0xc001f54880, 0x0, 0x0, 0x0, 0x0, 0x0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.6.1/helper/schema/resource.go:335 +0x1ee
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).RefreshWithoutUpgrade(0xc0008ff180, 0x7cdb5c8, 0xc000a7c2c0, 0xc000e34310, 0x0, 0x0, 0xc00000f1b8, 0x0, 0x0, 0x0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.6.1/helper/schema/resource.go:624 +0x1cb
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadResource(0xc00011c618, 0x7cdb5c8, 0xc000a7c2c0, 0xc000a7c300, 0xc000a7c2c0, 0x40b965, 0x6c900a0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.6.1/helper/schema/grpc_provider.go:575 +0x43b
github.com/hashicorp/terraform-plugin-go/tfprotov5/server.(*server).ReadResource(0xc0020f8420, 0x7cdb670, 0xc000a7c2c0, 0xc001b48540, 0xc0020f8420, 0xc00122b3e0, 0xc0026a0ba0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.3.0/tfprotov5/server/server.go:298 +0x105
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadResource_Handler(0x6f637a0, 0xc0020f8420, 0x7cdb670, 0xc00122b3e0, 0xc001b484e0, 0x0, 0x7cdb670, 0xc00122b3e0, 0xc0026be000, 0x7c3)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.3.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:344 +0x214
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000239880, 0x7cfd138, 0xc000f88f00, 0xc000414a00, 0xc000eb98c0, 0xb498b30, 0x0, 0x0, 0x0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1194 +0x52b
google.golang.org/grpc.(*Server).handleStream(0xc000239880, 0x7cfd138, 0xc000f88f00, 0xc000414a00, 0x0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1517 +0xd0c
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc0014c69b0, 0xc000239880, 0x7cfd138, 0xc000f88f00, 0xc000414a00)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:859 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:857 +0x1fd

Error: The terraform-provider-aws_v3.47.0_x5 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

Expected Behavior

Display confirmation to delete resource's, and delete them after that

Actual Behavior

Crash before asking for confirmation to delete resource's

Steps to Reproduce

1. Before - some resource's were created.
2. Code were changed without destroy'ing resource's
3. terraform init
4. terraform apply - some resource were no created, because of changing code without previous deletion
5. terraform destroy

After that i can replace it every time when i do:

1. terraform init -upgrade
2. terraform destroy

Important Factoids

This issue not depends on TF version - tested with TF 1.0.0, 1.0.1 and 0.15.5 - same effect.

References

Something similar were reported in previous time:
#8681
#7772
https://discuss.hashicorp.com/t/terraform-aws-provider-panic-plugin-did-not-respond/23396

[ewbankkit]

Suspiciously similar to

• error running terraform 1.0 destroy on local windows env #19807
• Crash while destroying state #19740
• Error: Plugin did not respond #19601
• aws provider v3.45 crashes during deleting resources #19788

[Erucolindo96]

Temporary solution:

1. comment code which creates every resource in AWS (leave some code doesn't created resource's - to be sure that plan and apply have work)
2. Terraform init
3. Terraform plan
4. Terraform apply

It should destroy most or every of your resource. And restore TF state which does not trigger error.
After than terraform destroy/apply with all resource's should be fine.

[Erucolindo96]

Suspiciously similar to

* [error running terraform 1.0 destroy on local windows env #19807](https://github.com/hashicorp/terraform-provider-aws/issues/19807)

* [Crash while destroying state #19740](https://github.com/hashicorp/terraform-provider-aws/issues/19740)

* [Error: Plugin did not respond #19601](https://github.com/hashicorp/terraform-provider-aws/issues/19601)

* [aws provider v3.45 crashes during deleting resources #19788](https://github.com/hashicorp/terraform-provider-aws/issues/19788)

@ewbankkit Is there some plans to fix this bug?

[gwvandesteeg]

Still happening on 3.48.0

Stack trace from the terraform-provider-aws_v3.48.0_x5 plugin:

panic: interface conversion: interface {} is nil, not *aws.AWSClient

goroutine 48 [running]:
github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsIamInstanceProfileRead(0xc00207c580, 0x0, 0x0, 0xb572d60, 0xc000100c00)
    /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-aws/aws/resource_aws_iam_instance_profile.go:210 +0xcaa
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0xc00089dce0, 0x7d367a8, 0xc0026e8080, 0xc00207c580, 0x0, 0x0, 0x0, 0x0, 0x0)
    /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.7.0/helper/schema/resource.go:335 +0x1ee
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).RefreshWithoutUpgrade(0xc00089dce0, 0x7d367a8, 0xc0026e8080, 0xc0017b0150, 0x0, 0x0, 0xc0003ea270, 0x0, 0x0, 0x0)
    /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.7.0/helper/schema/resource.go:624 +0x1cb
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadResource(0xc000132678, 0x7d367a8, 0xc0026e8080, 0xc0026e80c0, 0xc0026e8080, 0x40b965, 0x6cdee60)
    /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.7.0/helper/schema/grpc_provider.go:575 +0x43b
github.com/hashicorp/terraform-plugin-go/tfprotov5/server.(*server).ReadResource(0xc001ecd7a0, 0x7d36850, 0xc0026e8080, 0xc002062060, 0xc001ecd7a0, 0xc0027bf410, 0xc002078ba0)
    /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.3.0/tfprotov5/server/server.go:298 +0x105
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadResource_Handler(0x6fb6880, 0xc001ecd7a0, 0x7d36850, 0xc0027bf410, 0xc002062000, 0x0, 0x7d36850, 0xc0027bf410, 0xc001fb2000, 0x36e)
    /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.3.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:344 +0x214
google.golang.org/grpc.(*Server).processUnaryRPC(0xc00023d6c0, 0x7d58318, 0xc0006e6600, 0xc0018d6600, 0xc0006d8ae0, 0xb531b30, 0x0, 0x0, 0x0)
    /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1194 +0x52b
google.golang.org/grpc.(*Server).handleStream(0xc00023d6c0, 0x7d58318, 0xc0006e6600, 0xc0018d6600, 0x0)
    /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1517 +0xd0c
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc00077e9d0, 0xc00023d6c0, 0x7d58318, 0xc0006e6600, 0xc0018d6600)
    /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:859 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
    /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:857 +0x1fd

Error: The terraform-provider-aws_v3.48.0_x5 plugin crashed!

The above was when attempting to do a destroy of all the resources, prior to this there was the following

 Error: Error updating Auto Scaling Group: ValidationError: You must use a valid fully-formed launch template. Value (arn:aws:iam::123456789012:instance-profile/dev-node-instances-2021070503153847200000000d) for parameter iamInstanceProfile.arn is invalid. Invalid IAM Instance Profile ARN