0.14.2 panic with nested optional - module_variable_optional_attrs experiment

[bsch150]

While trying out the new module_variable_optional_attrs experiment, I ran into this issue with nested optional attributes. The exact same thing works if I explicitly set the optional_attr to null, and remove the experiment.

Terraform Version

0.14.2

Terraform Configuration Files

Please see the minimal github repo here

There are two examples in the repo. working-without-optional and broken-with-optional. They are identical except the broken example attempts to remove the explicit optional_attr = null using the optional type.

Debug Output

See the debug output in the repo here

Crash Output

See the crash.log in the repo here

Expected Behavior

The two examples should behave exactly the same way. The optional_attr should be implicitly set to null in the broken-with-optional example, and pass tf plan.

Actual Behavior

Terraform panics and crashes.

Steps to Reproduce

git clone https://github.com/bsch150/terraform-14.2-optional-issue.git
cd broken-with-optional
terraform init
terraform plan

[snowsky]

Does 'optional' need to be used in 'object' type?

variable "in" { type = object({ required = string optional = optional(string) }) }

[bsch150]

Does 'optional' need to be used in 'object' type?

variable "in" { type = object({ required = string optional = optional(string) }) }

@snowsky I'm not sure I understand your question. Are you asking whether the optional is needed there? Or do you mean to point out some way I'm misusing the experiment?

[mildwonkey]

Thank you for providing the reproduction repo @bsch150 , and I'm sorry about the panic! I've confirmed that the issue still exists in the 0.14 release branch and our master branch.

---

The crash output gives us a good hint about the problem: convert (in the upstream go-cty library) is getting inconsistent types; in one the list_with_optional is cty.ObjectWithOptionalAttrs and in the other it's just a cty.Object. This is enough information for us to dig into where the problem is occurring. I can't guarantee that we won't find that there's a configuration error in here somewhere, but it definitely should not panic.

Here's the main panic (with some editing and extra newlines, for easier reading):

inconsistent list element types 
(cty.Object(map[string]cty.Type{"nullable_object":cty.Object(map[string]cty.Type{"list_with_optional":cty.List(cty.ObjectWithOptionalAttrs(map[string]cty.Type{"attr":cty.String, "optional_attr":cty.String}, []string{"", "optional_attr"}))})}) 
 cty.Object(map[string]cty.Type{"nullable_object":cty.Object(map[string]cty.Type{"list_with_optional":cty.List(cty.Object(map[string]cty.Type{"attr":cty.String, "optional_attr":cty.String}))})}))

---

And finally to answer @snowsky 's question, any attribute inside an object can be optional, it does not need to be the entire object.

[preventhar]

I also faced a similar issue today. But, the variable structure I used is a little different. Just wanted to share that.

The variable structure in the bug's description is using optional in a nested object. Like this,

variable "list_of_objects_var" {
  type = list(object({
    nullable_object = object({
      list_with_optional = list(object({
        attr          = string
        optional_attr = optional(string)
      }))
    })
  }))
}

The one I used was a nested optional object. Like this,

variable "test_optional" {
  type = list(object({
    level_one_req_attr = string
    level_one_optional_attr = optional(object({
      level_two_req_attr = string
      level_two_optional_attr = optional(string)
    }))
  }))
}

And the input to the above variable is like this,

test_optional = [
    {
      level_one_req_attr = "sample1"
    },
    {
      level_one_req_attr = "sample2"
      level_one_optional_attr = {
        level_two_req_attr = "sample2"
      }
    },
    {
      level_one_req_attr = "sample3"
      level_one_optional_attr = {
        level_two_req_attr = "sample3"
        level_two_optional_attr = "sample3"
      }
    }
  ]

And the main panic is,

inconsistent list element types
(cty.Object(map[string]cty.Type{"level_one_optional_attr":cty.ObjectWithOptionalAttrs(map[string]cty.Type{"level_two_optional_attr":cty.String, "level_two_req_attr":cty.String}, []string{"", "level_two_optional_attr"}), "level_one_req_attr":cty.String}) then
cty.Object(map[string]cty.Type{"level_one_optional_attr":cty.Object(map[string]cty.Type{"level_two_optional_attr":cty.String, "level_two_req_attr":cty.String}), "level_one_req_attr":cty.String}))

The reason for inconsistent list element types is exactly the same as @mildwonkey described above.

[bsch150]

Thanks for the responses all - especially @mildwonkey!

I just made an interesting discovery around this issue - if you flip the order of the input array, plan passes.

in my broken-with-optional folder, my inputs to the module are:

module "test" {
  source = "./module"
  list_of_objects_var = [
    {
      nullable_object = null
    },
    {
      nullable_object = {
        list_with_optional = [
          {
            attr = "line"
          }
        ]
      }
    }
  ]
}

Now, if I switch the two elements of the list:

module "test" {
  source = "./module"
  list_of_objects_var = [
    {
      nullable_object = {
        list_with_optional = [
          {
            attr = "line"
          }
        ]
      }
    },
    {
      nullable_object = null
    }
  ]
}

With this change I can run plan successfully. However, this is not a solution for my use case as the order does matter for me.

Having never looked into your source code I'll avoid making any guesses why that would be - hopefully this is helpful.

[Matthewsre]

@mildwonkey, this issue has come up a few times for a microservices module I have been working on and is a big key to unlock some of the features we have been holding back on. Is there any way for me to view how items like this are prioritized?

You can see how we are leveraging optional in our module and have been unable to expand:

Module:
https://registry.terraform.io/modules/Matthewsre/microservices/azurerm/latest

Direct link to variable we would like to expand:
https://github.com/Matthewsre/terraform-azurerm-microservices/blob/master/variables.tf#L88