Error: validating provider credentials: retrieving caller identity from STS

[leonardoauribe]

Terraform Version

Terraform V1.6.3
on Linux amd64

Terraform Configuration Files

terraform {
  required_version = "~>1.6.0"
  backend "s3" {
    bucket         = "sfdc-site-tfstate-<redacted>"
    key            = "device-site"
    region         = "us-gov-west-1"
    dynamodb_table = "sfdc-site-tfstate-<redacted>"
    encrypt        = "true"
    kms_key_id     = "arn:aws-us-gov:kms:us-gov-west-1:01234567890:key/abcdefghi"
  }
  required_providers {
    fortios = {
      source  = "fortinetdev/fortios"
      version = "~>1.18.0"
    }
  }
}

provider "fortios" {
  hostname     = "device1-site.net"
  token        = var.device_token
  insecure     = "false"
  cabundlefile = "<redacted>.crt"
}

provider "fortios" {
  hostname     = "device2-site.net"
  alias        = "secondary"
  token        = var.device_token
  insecure     = "false"
  cabundlefile = "<redacted>.crt"
}

Debug Output

https://gist.github.com/leonardoauribe/e493440ee36679d9fef981b4f70f0bd1

Expected Behavior

While running our previous version, v1.5.2, we were able to initialize terraform and run plans/applies with no issues.

Actual Behavior

After upgrading to v1.6.3 we are unable to complete the initialization process. We are receiving error stating TF has exhausted all attempts to validate credentials to STS.

Steps to Reproduce

1. terraform init

Additional Context

We are running terraform in an isolated environment. We are using a proxy to handle/filter web calls and leveraging locally stored plugins/providers using the .terraformrc cli config file (contents below).

environment variable

HTTPS_PROXY=export HTTPS_PROXY=proxy1-site.net:8080

.terraformrc

provider installation {
filesystem mirror {
path = "/opt/terraform-<redacted>/"
include = ["registry.terraform.io/hashicorp/nxos", "registry.terraform.io/hashicorp/dns", "registry.terraform.io/hashicorp/aws", "registry.terraform.io/fortinetdev/fortios", "registry.terraform.io/hashicorp/dns", "registry. terraform.io/<redacted>/gus"]

AWS CLI and Curls to STS tests are successful, providing the expected output. We also see the tests reflected in our proxy logs. When running TF, we aren't getting hits on the proxy for the traffic pointing to TF being the problem. It appears as if the traffic never leaves the host.

AWS CLI

aws sts get-caller-identity
"Account" : “<redacted>”
"UserId": "<redacted>: user1"
"Arn" :
"arn:aws-us-gov:sts::<redacted>:assumed-role/<redacted>/user1"

Curl

curl -v -× proxy1-site.net:8080 https://sts.us-gov-west-1.amazonaws.com
About to connect () to proxy proxy1-site.net port 8080 (#0)
Trying 10.1.1.1..
Connected to proxy1-site.net (10.1.1.1) port 8080 (#0)
Establish HTTP proxy tunnel to sts.us-gov-west-1. amazonaws. com: 443
> CONNECT sts. us -gov-west -1. amazonaws.com: 443 HTTP/1.1
> Host: sts.us -gov -west - 1.amazonaws.com: 443
> User- Agent: curl/7.29.0
> Proxy -Connection: Keep-Alive
< HTTP/1.1 200 Connection established
Proxy replied OK to CONNECT request
Initializing NSS with certpath: sql: /etc/pki/nssdb
CAfile: <redacted>.crt
CApath: none
SSL connection using TLS ECDHE RSA WITH AES 128 GCM SHA256
Server certificate:
subject: CN=sts.us - gov -west -1. amazonaws.com
start date: Mar 08 00:00:00 2023 GMT
expire date: Mar 07 23:59:59 2024 GMT
common name: sts.us -gov-west - 1. amazonaws.com
issuer: CN=Amazon RSA 2048 M01, 0=Amazon, C=US
> GET / HTTP/1. 1
User- Agent: curl/7.29.0
> Host: sts.us-gov-west-1.amazonaws.com
> Accept: */*
< HTTP/1.1 302 Found
< x-amzn-RequestId: cdd256af - 5ee0 - 4004-b959 -6a110cdbda2b
Location: https://aws.amazon.com/iam
< Content - Lenath: 0
< Date: Wed, 01 Nov 2023 23:10:40 GMT
Connection #0 to host proxy1-site.net left intact

References

No response

[apparentlymart]

Hi @leonardoauribe! Thanks for reporting this.

Terraform does not configure any providers during the "init" command and so I think it's the "s3" backend returning this error. Terraform v1.6 included updating that backend to use the latest AWS SDK and so it's plausible for that to have caused a regression like what you've described.

I'm going to label this so that the AWS provider team can find it, since the AWS provider team also maintains the S3 backend in this codebase.

Thanks again!

[leonardoauribe]

Thanks @apparentlymart!

[gdavison]

Thanks for reporting this, @leonardoauribe. This will be fixed by #34243

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.