Default (optional) values in nested null object cause it to be non-null

[slwbuild]

Terraform Version

Terraform v1.3.4

Terraform Configuration Files

module "defaulting_module" {
    source = "./modules/default-module"
    request = {}
}

output "output" {
    value = module.defaulting_module
}

variable "request" {
    type = object({
        thing = optional(object({
            flag = optional(bool, false)
        }))
    })
}

output "request" {
    value = var.request
}

Debug Output

Changes to Outputs:
  + output = {
      + request = {
          + thing = {
              + flag = false
            }
        }
    }

Expected Behavior

thing should be null

Actual Behavior

thing is non-null and contains defaulted values

Steps to Reproduce

terraform plan

Additional Context

Output in TF 1.3.3 matched expected results. Changed in 1.3.4

References

No response

[GRBurst]

I came across a similar issue and are currently trying to figure out what I should expect. From the docs:

An optional attribute with a non-null default value is guaranteed to never have the value null within the receiving module. Terraform will substitute the default value both when a caller omits the attribute altogether and when a caller explicitly sets it to null, thereby avoiding the need for additional checks to handle a possible null value.

Terraform applies object attribute defaults top-down in nested variable types. This means that Terraform applies the default value you specify in the optional modifier first and then later applies any nested default values to that attribute.

Especially the second paragraph seems like it initializes and applies the nested default values to the parent object, hence creating an object?
So this might be the intended use, but it is definitely not something I would expect or design.

[slwbuild]

It isn't what i expected either and the behavior completely changed between 1.3.3 and 1.3.4 so i assumed it was not intentional

[apparentlymart]

Thanks for reporting this, @slwbuild.

Indeed, I think this is a regression that resulted from an attempt to address a nearby bug related to the population of default values in nested objects.

Based on my understanding of the intended design here, I would expect that leaving thing unassigned would cause it to be null, because you didn't specify an empty object as its default value.

I would expect the output you shared if you had assigned it like this:

  request = {
    thing = {}
  }

...because once thing is actually an object the optional attribute rules declared inside it should be effective.

I would also expect the output you shared if the type declaration included a non-null default value for thing, like this:

variable "request" {
    type = object({
        thing = optional(object({
            flag = optional(bool, false)
        }), {})
    })
}

In the above case, the default value for thing must be converted to the declared object type and so the default value is effectively { flag = false } after that conversion.

Without either of these things being true, I would agree with you that the final value of this variable ought to have been:

{
  thing = null /* of object({ flag = bool }) */
}

[apparentlymart]

Here are the three cases I described above represented as code:

variable "thing_default_null_unset" {
  type = object({
    thing = optional(object({
      flag = optional(bool, false)
    }))
  })
  default = {}
}

variable "thing_default_null_but_set" {
  type = object({
    thing = optional(object({
      flag = optional(bool, false)
    }))
  })
  default = {
    thing = {}
  }
}

variable "thing_default_obj" {
  type = object({
    thing = optional(object({
      flag = optional(bool, false)
    }), {})
  })
  default = {}
}

output "thing_default_null_unset" {
  value = var.thing_default_null_unset

  precondition {
    condition     = var.thing_default_null_unset.thing == null
    error_message = "thing should be null."
  }
}

output "thing_default_null_but_set" {
  value = var.thing_default_null_but_set

  precondition {
    condition     = !var.thing_default_null_but_set.thing.flag
    error_message = "thing.flag should be false."
  }
}

output "thing_default_obj" {
  value = var.thing_default_obj

  precondition {
    condition     = !var.thing_default_obj.thing.flag
    error_message = "thing.flag should be false."
  }
}

When I run this the first output value's precondition fails:

╷
│ Error: Module output value precondition failed
│ 
│   on optional-object-null.tf line 34, in output "thing_default_null_unset":
│   34:     condition     = var.thing_default_null_unset.thing == null
│     ├────────────────
│     │ var.thing_default_null_unset.thing is object with 1 attribute "flag"
│ 
│ thing should be null.
╵

I believe all three of these should pass, per my understanding of the design intent of this feature.

[stevehipwell]

We're seeing a similar regression from v1.3.3 to v1.3.4 where we have optional nested attributes with defaults.

[liamcervante]

I think the root cause of this is the same as #32157.

In the other case the problem is being exposed as a crash because the created objects are trying to be pushed into a collection which then fails as they have different types. Here it is just creating the bad objects from the nulls.