Panic: value for "variable" was requested before it was provided

[sjparkinson]

Terraform Version

Terraform v1.8.2
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v5.47.0
+ provider registry.terraform.io/hashicorp/random v3.6.1

Terraform Configuration Files

tests/s3.tftest.hcl:

variables {
  image_tag = "latest"
}

mock_provider "aws" {}

override_data {
  target = data.aws_secretsmanager_secret_version.production
  values = {
    secret_string = <<-EOT
    {
      "SPLUNK_TOKEN": "12345678-1234-1234-1234-123456789012",
      "UUID_1": "12345678-1234-1234-1234-123456789012",
      "UUID_2": "12345678-1234-1234-1234-123456789012"
    }
    EOT
  }
}

override_data {
  target = data.aws_iam_policy_document.s3_bucket_policy
  values = {
    json = "{}"
  }
}

override_resource {
  target = aws_s3_bucket.this
  values = {
    id  = "otel-graphite-collector"
    arn = "arn:aws:s3:::otel-graphite-collector"
  }
}

override_resource {
  target = aws_s3_bucket_policy.s3_bucket_policy
  values = {
    id = "otel-graphite-collector"
  }
}

# Override the module to avoid creating resources.
override_module {
  target  = module.production
  outputs = {}
}

run "aws_s3_object_uuids_formats_correctly" {
  command = plan

  assert {
    condition     = aws_s3_object.uuids.content == "UUID_1=12345678-1234-1234-1234-123456789012\nUUID_2=12345678-1234-1234-1234-123456789012"
    error_message = "UUIDs must be formatted into a valid .env file"
  }
}

s3.tf:

data "aws_secretsmanager_secret" "production" {
  name = "..."
}

data "aws_secretsmanager_secret_version" "production" {
  secret_id = data.aws_secretsmanager_secret.production.id
}

resource "aws_s3_bucket" "this" {
  bucket = "otel-graphite-collector"

  tags = {
    environment = "p"
  }
}

data "aws_iam_policy_document" "s3_bucket_policy" {
  statement {
    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::11111111111:role/..."]
    }

    actions = ["s3:GetObject"]

    resources = ["${aws_s3_bucket.this.arn}/*"]
  }

  statement {
    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::11111111111:role/..."]
    }

    actions = ["s3:*"]

    resources = [
      aws_s3_bucket.this.arn,
      "${aws_s3_bucket.this.arn}/*",
    ]
  }
}

# !!!
# Changes to this resource must be bootstrapped locally using the FT-Engineer role.
# !!!
resource "aws_s3_bucket_policy" "s3_bucket_policy" {
  bucket = aws_s3_bucket.this.id
  policy = data.aws_iam_policy_document.s3_bucket_policy.json
}

# Save the Graphite UUIDs from Secrets Manager into an .env file in S3. This is referenced by
# the ECS task definition as an `environmentFiles`.
resource "aws_s3_object" "uuids" {
  bucket = aws_s3_bucket.this.bucket
  key    = "resources/ecs_task_definition/uuids.env"

  # Build an .env file from the JSON object in Secrets Manager, containing only the UUID_ prefixed environment variables.
  content = join("\n", [
    for key, uuid in jsondecode(data.aws_secretsmanager_secret_version.production.secret_string) : "${key}=${uuid}"
    if startswith(key, "UUID_")
  ])

  content_type       = "text/plain"
  checksum_algorithm = "SHA256"

  tags = {
    environment = "p"
  }
}

Debug Output

https://gist.githubusercontent.com/sjparkinson/19680d8d703d8b81ca5e959cf5b95586/raw/45a13c77ee709e62cf293f674e808e448824c593/terraform-trace-output.txt

Expected Behavior

tests/s3.tftest.hcl... in progress
  run "aws_s3_object_uuids_formats_correctly"... pass
tests/s3.tftest.hcl... tearing down
tests/s3.tftest.hcl... pass

Success! 1 passed, 0 failed.

Actual Behavior

tests/s3.tftest.hcl... in progress

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Terraform crashed! This is always indicative of a bug within Terraform.
Please report the crash with Terraform[1] so that we can fix this.

When reporting bugs, please include your terraform version, the stack trace
shown below, and any additional information which may help replicate the issue.

[1]: https://github.com/hashicorp/terraform/issues

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

panic: value for module.production.var.tags was requested before it was provided
goroutine 503 [running]:
runtime/debug.Stack()
        runtime/debug/stack.go:24 +0x5e
github.com/hashicorp/terraform/internal/logging.PanicHandler()
        github.com/hashicorp/terraform/internal/logging/panic.go:84 +0x18b
panic({0x2e0b4c0?, 0xc00515b670?})
        runtime/panic.go:770 +0x132
github.com/hashicorp/terraform/internal/namedvals.(*values[...]).GetExactResult(0x3da61a0, {{0xc0029fa240, 0x1, 0x1}, {{}, {0xc0003d57c8, 0x4}}})
        github.com/hashicorp/terraform/internal/namedvals/values.go:88 +0x219
github.com/hashicorp/terraform/internal/namedvals.(*State).GetInputVariableValue(0x2f4bc40?, {{0xc0029fa240, 0x1, 0x1}, {{}, {0xc0003d57c8, 0x4}}})
        github.com/hashicorp/terraform/internal/namedvals/state.go:47 +0xe8
github.com/hashicorp/terraform/internal/terraform.(*evaluationStateData).GetInputVariable(0xc0036aacf0, {{}, {0xc0003d57c8?, 0xf153e7?}}, {{0xc00037de30, 0x2c}, {0x16, 0x12, 0x1b5}, {0x16, ...}})
        github.com/hashicorp/terraform/internal/terraform/evaluate.go:286 +0x246
github.com/hashicorp/terraform/internal/lang.(*Scope).evalContext(0xc0036aad80, {0xc00515b5d0, 0x2, 0x2}, {0x0, 0x0})
        github.com/hashicorp/terraform/internal/lang/eval.go:385 +0x1c72
github.com/hashicorp/terraform/internal/lang.(*Scope).EvalContext(...)
        github.com/hashicorp/terraform/internal/lang/eval.go:244
github.com/hashicorp/terraform/internal/lang.(*Scope).EvalBlock(0xc0036aad80, {0x3d863a0, 0xc003ca54f0}, 0xc001162c00)
        github.com/hashicorp/terraform/internal/lang/eval.go:57 +0xc5
github.com/hashicorp/terraform/internal/terraform.(*BuiltinEvalContext).EvaluateBlock(0xc002208b40?, {0x3d86630, 0xc000b21c30}, 0xc001162c00, {0x0?, 0x0?}, {{{{0x0, 0x0}}, {0x0, 0x0}}, ...})
        github.com/hashicorp/terraform/internal/terraform/eval_context_builtin.go:315 +0x145
github.com/hashicorp/terraform/internal/terraform.(*NodeApplyableProvider).ConfigureProvider(0xc001dba5f0, {0x3da90c8, 0xc002208b40}, {0x3d9add0, 0xc0027a8a80}, 0x0)
        github.com/hashicorp/terraform/internal/terraform/node_provider.go:115 +0x316
github.com/hashicorp/terraform/internal/terraform.(*NodeApplyableProvider).Execute(0xc001dba5f0, {0x3da90c8, 0xc002208b40}, 0x2)
        github.com/hashicorp/terraform/internal/terraform/node_provider.go:46 +0x5fb
github.com/hashicorp/terraform/internal/terraform.(*ContextGraphWalker).Execute(0xc001dc6dc0, {0x3da90c8, 0xc002208b40}, {0x7fb47b8d0068, 0xc001dba5f0})
        github.com/hashicorp/terraform/internal/terraform/graph_walk_context.go:153 +0xbb
github.com/hashicorp/terraform/internal/terraform.(*Graph).walk.func1({0x33015c0, 0xc001dba5f0})
        github.com/hashicorp/terraform/internal/terraform/graph.go:118 +0x795
github.com/hashicorp/terraform/internal/dag.(*Walker).walkVertex(0xc000b71020, {0x33015c0, 0xc001dba5f0}, 0xc0030cb400)
        github.com/hashicorp/terraform/internal/dag/walk.go:384 +0x2d7
created by github.com/hashicorp/terraform/internal/dag.(*Walker).Update in goroutine 324
        github.com/hashicorp/terraform/internal/dag/walk.go:307 +0xff3

Exited with code exit status 11

Steps to Reproduce

1. terraform init
2. terraform test

Additional Context

This looks to have been introduced in 5.47.0, no issues with 5.46.0.

Actually, seems more like an issue in the latest version of Terraform rather than the AWS provider, as we're seeing terraform test fail on other branches still using the older version of the provider too.

References

No response

[sjparkinson]

This looks to have been introduced in https://github.com/hashicorp/terraform/releases/tag/v1.8.2. Pinning to v1.8.1 works.

At a guess in #35030.

[sjparkinson]

At a guess this might be related to our use of the aws_default_tags data source, which we make use of in both a module we source, and also in the root module we deploy. In both we also specify default_tags in the provider block. I can see this getting a bit too interconnected?

In main.tf:

data "aws_default_tags" "this" {}

module "production" {
  source = "./modules/example"

  tags = merge(data.aws_default_tags.this.tags, {
    environment = "prod"
  })
}

provider "aws" {
  region = "eu-west-1"

  default_tags {
    tags = {
      system = "example"
    }
  }
}

And in the module:

data "aws_default_tags" "this" {}

variable "tags" {
  type        = map(string)
  description = "This variable specifies the AWS tags to apply to all resources."
  nullable    = true
}

provider "aws" {
  region = "eu-west-1"

  default_tags {
    tags = merge(var.tags, {
      "environment:id" = "example-${var.environment}"
    })
  }
}

[jbardin]

Hi @sjparkinson,

Thanks for filing the issue. Yes, the linked PR looks to have been the root cause of the change.

The bug is being triggered by the provider configuration existing in the module. Providers defined within a module are supported in limited cases for compatibility with early versions of Terraform, but it is not a recommended pattern to use. Optimally the provider configuration should be defined in the root module and passed into the child module which requires it.

I think we can avoid the panic, but there may still be other compatibility issues with a legacy module structure like this and the new testing framework.

Thanks!

[sjparkinson]

Providers defined within a module are supported in limited cases for compatibility with early versions of Terraform, but it is not a recommended pattern to use. Optimally the provider configuration should be defined in the root module and passed into the child module which requires it.

That makes sense, what we are doing does feel a bit off so great to have this pointer!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.