[Bug]: Failed to save state .... https response error StatusCode: 400

[oligirling]

Current Behavior

When we run terraform apply the state in SOS can be read but when it tries to save the state in SOS it fails.

This is only happening when upgrading terraform >1.6. Works fine on terraform 1.5.7. This could be an issue with terraform but since the issue is happening with the state and the http code is 400, thought i would ask here.

Copy of our provider.tf:

terraform {
  required_providers {
    exoscale = {
      source  = "exoscale/exoscale"
      version = "~> 0.50"
    }
  }
}

provider "exoscale" {
  key    = var.exoscale_vstest_key
  secret = var.exoscale_vstest_secret
}

terraform {
  backend "s3" {
    bucket                      = "bucketname"
    key                         = "terraform.tfstate"
    region                      = "de-muc-1"
    endpoint                    = "https://sos-de-muc-1.exo.io"
    profile                     = "profilename"
    shared_credentials_file     = "terraform.tfstate-credentials.txt"
    skip_region_validation      = true
    skip_credentials_validation = true
    skip_requesting_account_id  = true
    skip_metadata_api_check     = true
    force_path_style            = true
  }
}

Expected Behavior

We run terraform apply and the state is read and written to SOS

Steps To Reproduce

No response

Provider Version

0.53.1

Terraform Version

1.6.3

Relevant log output

exoscale_compute_instance.instancename[3]: Still creating... [30s elapsed]
exoscale_compute_instance.instancename[3]: Creation complete after 34s [id=xxxxxxxxx]
╷
│ Error: Failed to save state
│ 
│ Error saving state: failed to upload state: operation error S3: PutObject, exceeded maximum number of attempts, 5, https response error StatusCode: 400, RequestID:
│ 088a3b3c-ca72-4b64-9796-49049d108f96, HostID: 088a3b3c-ca72-4b64-9796-49049d108f96, api error RequestTimeout: No data received during the last 30 secs.
╵
╷
│ Error: Failed to persist state to backend
│ 
│ The error shown above has prevented Terraform from writing the updated state to the configured backend. To allow for recovery, the state has been written to the file
│ "errored.tfstate" in the current working directory.
│ 
│ Running "terraform apply" again at this point will create a forked state, making it harder to recover.
│ 
│ To retry writing this state, use the following command:
│     terraform state push errored.tfstate

[PhilippeChepy]

Hello @oligirling,

We are aware of this unfortunate situation.
The root cause of this issue is in the core Terraform CLI that now uses a more recent version of the AWS SDK to write its state in an S3-compatible backend.

We don't have yet a long-term solution for that but we still have to investigate further this issue.

In the meantime, we recommend you to stick with terraform < 1.6. You can add this definition to enforce that:

terraform {
  # Terraform
  required_version = "< 1.6"
}

We will update this thread as soon as we have an update.

[PhilippeChepy]

Hello @oligirling,

I just tried to use SOS as a state backend with Terraform 1.6.3. It works with this configuration:

terraform {
  required_providers {
    exoscale = {
      source  = "exoscale/exoscale"
    }
  }

  backend "s3" {
    bucket   = "test-tf-backend-bug"
    key      = "test.tfstate"
    region   = "<your-zone>"
    
    endpoints = {
      s3 = "https://sos-<your-zone>.exo.io"
    }
    
    access_key = "<your-api-key>"
    secret_key = "<your-api-secret>"

    # Disable AWS-specific features
    skip_credentials_validation = true
    skip_region_validation      = true
    skip_requesting_account_id = true
    skip_s3_checksum = true
  }
}

It seems that the AWS SDK used by Terraform now requires an "x-amz-sdk-checksum-algorithm" to be present. Most s3-compatible providers don't provide this header. That's why Hashicorp deployed a workaround in Terraform 1.6.3: you can set the skip_s3_checksum to true.

I'm following up on this issue with our SOS team to see if we need to implement this header or not.

[PhilippeChepy]

We plan to implement SOS missing parts, but we don't have an ETA yet.
In the meantime, we updated our documentation to reflect the updated configuration requirements (See #318).