Since 1.4 output is flagged as sensitive

[bianchi2]

Terraform Version

terraform version
Terraform v1.4.2
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v4.45.0
+ provider registry.terraform.io/hashicorp/cloudinit v2.2.0
+ provider registry.terraform.io/hashicorp/helm v2.7.1
+ provider registry.terraform.io/hashicorp/kubernetes v2.16.1
+ provider registry.terraform.io/hashicorp/random v3.4.3
+ provider registry.terraform.io/hashicorp/time v0.9.1
+ provider registry.terraform.io/hashicorp/tls v4.0.4

Terraform Configuration Files

Previously, the following output in the root module wasn't marked as sensitive:

output "cluster_details" {
  description = "EKS cluster information"

  value = {
    cluster_name     = local.cluster_name
    cluster_asg_name = module.base-infrastructure.eks.cluster_asg_name
  }
}

locals {
  cluster_name = format("test-%s-cluster", var.environment_name)
}

Debug Output

Error: Output refers to sensitive values
  on outputs.tf line 11:
  11: output "cluster_details" {
To reduce the risk of accidentally exporting sensitive data that was intended
to be only internal, Terraform requires that any root module output
containing sensitive data be explicitly marked as sensitive, to confirm your
intent.
If you do intend to export this data, annotate the output value as sensitive
by adding the following argument:
    sensitive = true

Expected Behavior

No error should be presented as there's nothing sensitive, just cluster and ASG name.

Actual Behavior

Terraform apply fails with an error.

Steps to Reproduce

Just do terraform apply to a project with eks module and the above output.

Additional Context

No response

References

No response

[sushant-kapoor17]

Hello,
There are a couple of things, which can be looked at:

1. Are you using terraform cloud by any chance ? There have been issues seen in the past similar to this , which occurred due to some misconfiguration under "Remote state sharing" settings.
2. Could there be a possibility, that module.base-infrastructure.eks contains a sensitive value ? and , while fetching cluster_asg_name it is complaining ?
3. It is weird that it is only happening since 1.4 though ? Have you tried updating aws and kubernetes providers version and see if the problem still persists ?

To confirm point 2, can you please try replacing the output as following and run terraform apply again ?

output "cluster_details" {
  description = "EKS cluster information"

  value = {
    cluster_name     = local.cluster_name
    cluster_asg_name = nonsensitive(module.base-infrastructure.eks.cluster_asg_name)
  }
}

Additionally, it will be really great if you can please run apply using this command , and post the log here.
TF_LOG=DEBUG terraform apply

Thanks

[freakinhippie]

Terraform Version:

> terraform version
Terraform v1.4.2
on linux_amd64
+ provider registry.terraform.io/hashicorp/local v2.4.0

I've also been battling this issue since terraform 1.4.0 was released. There is an inconsistency with how terraform is handling sensitive input values in a root module vs. in a child module. I've created a repo that demonstrates the issue as concisely as possible, though my real use case is far more intricate.

This terraform stack creates a mock AWS EC2 instance, using entirely local values. The important point is that the module has an input variable that is marked sensitive = true. In the example below, the root module passes a key_name (sensitive = false) and a bootstrap_token (sensitive = true) into the module call. The module uses the token to construct an user_data object. The module then returns the objects that were created as outputs.

# clone the demo repo
git clone https://github.com/freakinhippie/terraform-bug-reports.git
# move into the repo and checkout the appropriate revision
cd terraform-bug-reports/
git checkout terraform-32880
# move into the stack directory to illustrate the error
cd stacks/error_with_module/
# run terraform
terraform init
terraform apply

The root module:

# cat ./stacks/error_with_module/*.tf

module "host" {
  source = "../../modules/mock_ec2_instance"

  instances       = var.instances
  bootstrap_token = var.bootstrap_token
}

locals {
  unique_keys = { for k in toset(module.host.instances[*].key_name) : k => true }
}

resource "local_sensitive_file" "keys" {
  for_each = local.unique_keys

  filename        = "${path.module}/${each.key}"
  content         = each.value
  file_permission = "0600"
}

variable "bootstrap_token" {
  type      = string
  sensitive = true
  default   = ""
}

variable "instances" {
  description = "List of instance meta-data objects"
  type = list(object({
    name     = string
    key_name = string
  }))
}

terraform {
  required_version = ">= 1.4"
}

The child module:

# cat ./modules/mock_ec2_instance/*.tf

# mock ec2 instance to demonstrate issue
locals {
  instances = [
    for i, v in var.instances : merge(
      v,
      {
        user_data = "# ${var.bootstrap_token}"
      }
    )
  ]
}

output "instances" {
  value = local.instances
}

variable "bootstrap_token" {
  type      = string
  sensitive = true
  default   = ""
}

variable "instances" {
  description = "List of instance meta-data objects"
  type = list(object({
    name     = string
    key_name = string
  }))
}

terraform {
  required_version = ">= 1.4"
}

Attempting to apply this configuration with terraform 1.4.0 or greater results in the following error:

> terraform apply -auto-approve

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # local_sensitive_file.keys["my-ec2-key"] will be created
  + resource "local_sensitive_file" "keys" {
      + content              = (sensitive value)
      + content_base64sha256 = (known after apply)
      + content_base64sha512 = (known after apply)
      + content_md5          = (known after apply)
      + content_sha1         = (known after apply)
      + content_sha256       = (known after apply)
      + content_sha512       = (known after apply)
      + directory_permission = "0700"
      + file_permission      = "0600"
      + filename             = "./my-ec2-key"
      + id                   = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.
╷
│ Error: Invalid for_each argument
│ 
│   on main.tf line 13, in resource "local_sensitive_file" "keys":
│   13:   for_each = local.unique_keys
│     ├────────────────
│     │ local.unique_keys has a sensitive value
│ 
│ Sensitive values, or values derived from sensitive values, cannot be used as for_each arguments. If used, the sensitive value could be exposed as a resource instance key.

The problem arises from the bootstrap_token input variable within the ec2_instance module. It is marked as sensitive = true and therefore the module.host.instances[*].user_data is presumably also marked as sensitive. The sensitive flag is passed back in the output where it ultimately results in the error shown above.

In my scenario I have no need to use the sensitive user_data value in a for_each statement. I do need the key_name value though.

In an attempt to mitigate the issue, I tried changing the unique_keys local value to use the nonsensitive() function like so (as well as every other permutation of nonsensitive() I could think of):

locals {
  unique_keys = { for k in toset(nonsensitive(module.host.instances[*].key_name)) : k => true }
}

But that results in a new error:

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: Invalid function argument
│ 
│   on main.tf line 77, in locals:
│   77:   unique_keys = { for k in toset(nonsensitive(module.host.instances[*].key_name)) : k => true }
│     ├────────────────
│     │ while calling nonsensitive(value)
│     │ module.host.instances is tuple with 1 element
│ 
│ Invalid value for "value" parameter: the given value is not sensitive, so
│ this call is redundant.
╵

Incidentally, the original error which said "If used, the sensitive value could be exposed as a resource instance key.", that ship had already sailed because the planning output includes:

# SNIP
Terraform will perform the following actions:

  # local_sensitive_file.keys["my-ec2-key"] will be created
  + resource "local_sensitive_file" "keys" {
      + content              = (sensitive value)
      + content_base64sha256 = (known after apply)
      + content_base64sha512 = (known after apply)
      + content_md5          = (known after apply)
      + content_sha1         = (known after apply)
      + content_sha256       = (known after apply)
      + content_sha512       = (known after apply)
      + directory_permission = "0700"
      + file_permission      = "0600"
      + filename             = "./my-ec2-key"
      + id                   = (known after apply)
    } 
# SNIP

In trying to simplify the use case enough to submit a meaningful bug report I found that if I move all of the logic from the ec2_instance child module into the root module, including the sensitive input variable bootstrap_token, I do not encounter the problem.

# move into the stack directory to illustrate the error
cd stacks/no_error_in_root/
# run terraform
terraform init
terraform apply

# cat ./stacks/no_error_in_root/*.tf

locals {
  instances = [
    for i, v in var.instances : merge(
      v,
      {
        user_data = "# ${var.bootstrap_token}"
      }
    )
  ]
  unique_keys = { for k in toset(local.instances[*].key_name) : k => true }
}

resource "local_sensitive_file" "keys" {
  for_each = local.unique_keys

  filename        = "${path.module}/${each.key}"
  content         = each.value
  file_permission = "0600"
}

variable "bootstrap_token" {
  type      = string
  sensitive = true
  default   = ""
}

variable "instances" {
  description = "List of instance meta-data objects"
  type = list(object({
    name     = string
    key_name = string
  }))
}

terraform {
  required_version = ">= 1.4"
}

I cannot say if this is a bug or not, but the inconsistent operation when operating at the root module level vs using a child module suggest a bug.

Any suggestions for a work-around, other than moving module logic into the root module would be very much appreciated!

Thank you!

[bianchi2]

@sushant-kapoor17 yes, nonsensitive helped. I started seeing this behavior recently with terraform version upgrade. All modules versions haven't been changed. Just wanted to figure out why the same module was non sensitive pre 1.4 and it is sensitive now.

Interesting... when running terraform apply for the second time I get:

Error: Invalid function argument
  on outputs.tf line 16, in output "eks":
  16:     cluster_asg_name = nonsensitive(module.base-infrastructure.eks.cluster_asg_name)
    ├────────────────
    │ while calling nonsensitive(value)
    │ module.base-infrastructure.eks.cluster_asg_name is "eks-appNode-m5_2xlarge"
Invalid value for "value" parameter: the given value is not sensitive, so
this call is redundant.

Which makes absolute sense. However, without nonsensitive (on the first apply) it errors out complaining about the value being sensitive.

[sushant-kapoor17]

Hello @freakinhippie,

Thank you for explaining the issue in detail and providing a sample repo, that usually helps alot !!
I can understand where you are coming from about this issue of sensitive variables not working unless substituted back into root modules.
It seems to me that, when Terraform uses functions to create values out of sensitive object, it tends to mark the created objects as sensitive as well and it has been there by design.

One interesting thing that I found was another Github issue, which is basically the same problem we are trying to solve including the problem mentioned by the original poster @bianchi2.

It looks like this is a problem , which is being mentioned quite frequently , but I could not find much in terms of drafting of some kind of a feature request for this issue.The Github issue discusses more workarounds to deal with these kind of situations.Hopefully,in future, there will be a feature request or an explanation further , if it remains as-is by design.

Anyhow, I spent some time looking at your repo and have been able to create workarounds for you ,which will unblock you for now.There should be a PR available in your repo now.
Note,neither of these workarounds require you to move all the logic from child module to root module , so happy days 😃

I have used the following strategies:

1. Use count instead of for-each evaluation on transformed non-sensitive values. However, to achieve this, I had to tweak your transformed object a little to make it look more realistic.

unique_keys = [for k in toset(module.host.instances[*].key_name) : { "key_name" : k, "content" : "true" }]

 resource "local_sensitive_file" "keys" {
  count = length(local.unique_keys)

  filename        = "${path.module}/${lookup(local.unique_keys[count.index], "key_name")}"
  content         = lookup(local.unique_keys[count.index], "content")
  file_permission = "0600"
}

2. Convert the non sensitive value to a string first in json and then again convert it back.

locals {
  unique_spot_keys = { for k in nonsensitive(jsondecode(jsonencode(module.host.spot_instances[*]))) : k.key_name => true }
}
resource "local_sensitive_file" "spot_keys" {
  for_each = local.unique_spot_keys

  filename        = "${path.module}/${each.key}"
  content         = each.value
  file_permission = "0600"
}

With both of these strategies, I can run terraform apply successfully with no errors.

I am hoping the strategies I have provided are helpful for you. If you like the solutions, you can merge the PR into your repo and use them.

@bianchi2, I am sorry the nonsensitive() function didn't work out for you. It seems like the way nonsensitive() works is a bit complicated for now. I would request you to try the no.2 approach(mentioned above) for the field module.base-infrastructure.eks.cluster_asg_name. I think converting it into string and back again might do the trick.

Hope this helps.
Thanks