0.12.0 terraform init failed: Error refreshing state: Invalid state file format: The state file field "hash" has invalid value number 10669755453527594976

[minamijoyo]

I have a v0.12 upgrade issue related to migrating old tfstate file format.

Terraform Version

$ terraform --version
Terraform v0.12.0
+ provider.aws v2.11.0
+ provider.template v2.1.2

Terraform Configuration Files

terraform {
  backend "s3" {
    region = "ap-northeast-1"
    bucket = "[REDACTED]"
    key    = "services/yadockeri/development/terraform.tfstate"
  }
}

Debug Output

I omitted aws-sdk-go requests and responses for security reason.

$ TF_LOG=TRACE terraform init
2019/05/28 15:38:41 [INFO] Terraform version: 0.12.0
2019/05/28 15:38:41 [INFO] Go runtime version: go1.12.4
2019/05/28 15:38:41 [INFO] CLI args: []string{"/usr/local/Cellar/tfenv/0.6.0/versions/0.12.0/terraform", "init"}
2019/05/28 15:38:41 [DEBUG] Attempting to open CLI config file: /Users/masayuki-morita/.terraformrc
2019/05/28 15:38:41 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2019/05/28 15:38:41 [INFO] CLI command args: []string{"init"}
2019/05/28 15:38:41 [TRACE] ModuleInstaller: installing child modules for . into .terraform/modules
Initializing modules...
2019/05/28 15:38:41 [DEBUG] Module installer: begin yadockeri_development_db
2019/05/28 15:38:41 [TRACE] ModuleInstaller: Module installer: yadockeri_development_db <nil> already installed in ../../../modules/rds_resources
2019/05/28 15:38:41 [DEBUG] Module installer: begin yadockeri_development_swarm
2019/05/28 15:38:41 [TRACE] ModuleInstaller: Module installer: yadockeri_development_swarm <nil> already installed in ../../../modules/swarm
2019/05/28 15:38:41 [TRACE] modsdir: writing modules manifest to .terraform/modules/modules.json

Initializing the backend...
2019/05/28 15:38:41 [TRACE] Meta.Backend: built configuration for "s3" backend with hash value 413735613
2019/05/28 15:38:41 [TRACE] Preserving existing state lineage "97959b01-d89b-1ffa-adf9-e34df9c3e2cd"
2019/05/28 15:38:41 [TRACE] Preserving existing state lineage "97959b01-d89b-1ffa-adf9-e34df9c3e2cd"
2019/05/28 15:38:41 [TRACE] Meta.Backend: working directory was previously initialized for "s3" backend
2019/05/28 15:38:41 [TRACE] backendConfigNeedsMigration: configuration values have changed, so migration is required
2019/05/28 15:38:41 [TRACE] Meta.Backend: backend configuration has changed (from type "s3" to type "s3")
2019/05/28 15:38:41 [WARN] backend config has changed since last init
Backend configuration changed!

Terraform has detected that the configuration specified for the backend
has changed. Terraform will now check for existing state in the backends.


2019/05/28 15:38:41 [INFO] Setting AWS metadata API timeout to 100ms
2019/05/28 15:38:42 [INFO] Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id
2019/05/28 15:38:42 [INFO] AWS Auth provider used: "EnvProvider"
2019/05/28 15:38:42 [DEBUG] Trying to get account information via sts:GetCallerIdentity
2019/05/28 15:38:42 [DEBUG] [aws-sdk-go] DEBUG: Request sts/GetCallerIdentity Details:
[REDACTED]
2019/05/28 15:38:42 [DEBUG] [aws-sdk-go] DEBUG: Response sts/GetCallerIdentity Details:
[REDACTED]
2019/05/28 15:38:42 [DEBUG] [aws-sdk-go] <GetCallerIdentityResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
[REDACTED]
2019/05/28 15:38:42 [INFO] Setting AWS metadata API timeout to 100ms
2019/05/28 15:38:43 [INFO] Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id
2019/05/28 15:38:43 [INFO] AWS Auth provider used: "EnvProvider"
2019/05/28 15:38:43 [DEBUG] Trying to get account information via sts:GetCallerIdentity
2019/05/28 15:38:43 [DEBUG] [aws-sdk-go] DEBUG: Request sts/GetCallerIdentity Details:
[REDACTED]
2019/05/28 15:38:44 [DEBUG] [aws-sdk-go] DEBUG: Response sts/GetCallerIdentity Details:
[REDACTED]
2019/05/28 15:38:44 [DEBUG] [aws-sdk-go] <GetCallerIdentityResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
[REDACTED]
2019/05/28 15:38:44 [TRACE] backendMigrateState: need to migrate from "s3" to "s3" backend config
2019/05/28 15:38:44 [DEBUG] [aws-sdk-go] DEBUG: Request s3/ListObjects Details:
[REDACTED]
2019/05/28 15:38:44 [DEBUG] [aws-sdk-go] DEBUG: Response s3/ListObjects Details:
[REDACTED]
2019/05/28 15:38:44 [DEBUG] [aws-sdk-go] <?xml version="1.0" encoding="UTF-8"?>
[REDACTED]
2019/05/28 15:38:44 [DEBUG] [aws-sdk-go] DEBUG: Request s3/ListObjects Details:
[REDACTED]
2019/05/28 15:38:44 [DEBUG] [aws-sdk-go] DEBUG: Response s3/ListObjects Details:
[REDACTED]
2019/05/28 15:38:44 [DEBUG] [aws-sdk-go] <?xml version="1.0" encoding="UTF-8"?>
[REDACTED]
2019/05/28 15:38:44 [TRACE] backendMigrateState: migrating "default" workspace to "default" workspace
2019/05/28 15:38:44 [DEBUG] [aws-sdk-go] DEBUG: Request s3/ListObjects Details:
[REDACTED]
2019/05/28 15:38:44 [DEBUG] [aws-sdk-go] DEBUG: Response s3/ListObjects Details:
[REDACTED]
2019/05/28 15:38:44 [DEBUG] [aws-sdk-go] <?xml version="1.0" encoding="UTF-8"?>
[REDACTED]
2019/05/28 15:38:44 [DEBUG] [aws-sdk-go] DEBUG: Request s3/GetObject Details:
[REDACTED]
2019/05/28 15:38:44 [DEBUG] [aws-sdk-go] DEBUG: Response s3/GetObject Details:
[REDACTED]
2019/05/28 15:38:44 [DEBUG] [aws-sdk-go]

Error: Error loading state:
    Invalid state file format: The state file field "hash" has invalid value number 10669755453527594976

Terraform failed to load the default state from the "s3" backend.
State migration cannot occur unless the state can be loaded. Backend
modification and state migration has been aborted. The state in both the
source and the destination remain unmodified. Please resolve the
above error and try again.

Crash Output

No crash

Expected Behavior

terraform init should be succeeded.

Actual Behavior

Failed to initialize with the following error message:

Error: Error loading state:
    Invalid state file format: The state file field "hash" has invalid value number 10669755453527594976

Steps to Reproduce

1. terraform init with v0.11.14
2. terraform 0.12checklist with v0.11.14 => no warning
3. terraform init with v0.12.0

I'm not sure exactly how to reproduce this.
In my paticular environment, errors always occur. But I cannot create a minimum reproduction case.

Additional Context

My tfstate file in the S3 bucket is as follows:

$ terraform state pull | head -20
{
    "version": 3,
    "terraform_version": "0.11.14",
    "serial": 130,
    "lineage": "13c44505-731a-456d-8f94-932507ef55c9",
    "backend": {
        "type": "s3",
        "config": {
            "bucket": "[REDACTED]",
            "key": "services/yadockeri/development/terraform.tfstate",
            "region": "ap-northeast-1"
        },
        "hash": 10669755453527594976
    },
    "modules": [
        {
            "path": [
                "root"
            ],
            "outputs": {},

10669755453527594976 is greater than int64 max (9223372036854775807)
So, this issue is maybe related to: #19688

I tried to create a minimum reproduction case, but I found that initializing a new tfstate with v0.11.14 no longer stores the backend field.
This environment was initialized with too old terraform(maybe v0.6 or 0.7), and upgraded to 0.8.8 => 0.9.10 => 0.10.7 => 0.11.10 => 0.11.14 => 0.12.0.
So, I tried to reproduced its upgrade sequence in other new environment, it has backend field but doesn't contain hash field in it. I don't know why.

If I omitted backend field and overwrite the tfstate file in S3 bucket, then terraform init is succeeded, but now terraform plan failed with the following error message:

2019/05/28 14:51:56 [TRACE] <root>: eval: *terraform.EvalWriteState
2019/05/28 14:51:56 [TRACE] EvalWriteState: writing current state object for aws_route53_record.y_dev_crowdworks_jp_local_ns
2019/05/28 14:51:56 [TRACE] [walkRefresh] Exiting eval tree: aws_route53_record.y_dev_crowdworks_jp_local_ns
2019/05/28 14:51:56 [TRACE] vertex "aws_route53_record.y_dev_crowdworks_jp_local_ns": visit complete
2019/05/28 14:51:56 [TRACE] vertex "aws_route53_record.y_dev_crowdworks_jp_local_ns": dynamic subgraph completed successfully
2019/05/28 14:51:56 [TRACE] vertex "aws_route53_record.y_dev_crowdworks_jp_local_ns": visit complete
2019/05/28 14:51:56 [TRACE] dag/walk: upstream of "provider.aws (close)" errored, so skipping
2019/05/28 14:51:56 [TRACE] dag/walk: upstream of "root" errored, so skipping

Error: Invalid state file format

The state file field "hash" has invalid value number 15487971632617800249

2019-05-28T14:51:56.775+0900 [DEBUG] plugin: plugin process exited: path=/Users/masayuki-morita/src/github.com/crowdworksjp/terraform-aws/services/yadockeri/development/.terraform/plugins/darwin_amd64/terraform-provider-template_v2.1.2_x4 pid=78070
2019-05-28T14:51:56.775+0900 [DEBUG] plugin: plugin exited
2019-05-28T14:51:56.777+0900 [DEBUG] plugin: plugin process exited: path=/Users/masayuki-morita/src/github.com/crowdworksjp/terraform-aws/services/yadockeri/development/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.11.0_x4 pid=78069
2019-05-28T14:51:56.777+0900 [DEBUG] plugin: plugin exited

How to avoid this error and upgrade to v0.12 ?

References

#19688

[minamijoyo]

If I omitted backend field and overwrite the tfstate file in S3 bucket, then terraform init is succeeded, but now terraform plan failed with the following error message:

I wondered why the error hash value on the plan (15487971632617800249) was different from the first one (10669755453527594976), and after further investigation, I found that this was another tfstate file referenced by terraform_remote_state.
After removing the backend attribute from this tfstate file, the plan now succeeded.

I submitted a Pull Request #21484 to fix this issue.
If you have the same issue, removing the backend attribute from your tfstate file manually is a workaround.

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.