Terraform 0.14 crashes, Terraform 1.0 reports inconsistent data type only when array is in conditional statement

[AndreiPotemkin]

I think there might be an issue with the way Terraform handles list elements when in conditional statement compared to unconditional logic.
Actual problem has been encountered with Azure network security rule definitions but simplified code reproduces the problem.

Consider 2 local variables:

  a = {
    val = 1, arr = null
  }
  b = {
    val = null, arr = [1]
  }

if these 2 values are placed in an array unconditionally

c = [local.a, local.b]

it works fine but if same array is built conditionally

c = local.v ? [local.a, local.b] : []

it crashes Terraform 0.14 and in later versions reports error
│ │ local.a is object with 2 attributes
│ │ local.b is object with 2 attributes

To reproduce the issue attempt to execute 'terraform plan' on following:

locals {
  v = true
  a = {
    val = 1, arr = null
  }
  b = {
    val = null, arr = [1]
  }
  c = concat(
    [local.a,local.b],
    local.v ? [local.a,local.b]: []
  )
}

When v = false plan succeeds, when v = true plan fails

Possible solution to change definition of a to

  a = {
    val = 1, arr = []
  }

would pass Terraform check but will fail on apply as Azure network security rule only supports one of the 2 values to be provided, for example source_port_range and source_port_ranges (https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/network_security_rule#source_port_range)

[laszlocsomor]

I believe this is the same bug I just saw with Terraform 1.1.7 on Linux x86-64.

Actual encounter was with jsonencode( { widgets = [ ... ] } ) with a computed list of widgets; simplified repro is:

locals {
  value = concat(
    [{ x = 0 }],
    var.some_input_var == null ? [] : [
      {
        key1 = [
          [
            { x = 1 }
          ]
        ]
      },
      {
        key2 = ["y"]
      },
  ])
}

This gives:

│ The true and false result expressions must have consistent types. The given expressions are tuple and tuple, respectively.

Replacing the true branch with null gives a different error, I guess originating from concat:

│ Invalid value for "seqs" parameter: argument must not be null.

Removing the conditional and keeping either branch makes the problem vanish.

[apparentlymart]

Hi @AndreiPotemkin,

Aside from this error message being confusing, this does seem like correct behavior to me. Both the true and false arms of a conditional expression must have the same type so that Terraform can determine the result type even if the condition isn't known yet, and a tuple with two elements is a different type than a tuple with no elements.

One possible answer is to convert them both to be lists, because (unlike for tuple types) the numby of elements is part of the value of a list, not part of the type: a list of two strings is the same type as a list of zero strings.

That would not work for @laszlocsomor's example though, because in that case there really is no single type that both the true and false expressions could be converted to: the false arm itself can't be a list because the two elements of the tuple are of different types. That problem will require a different solution that doesn't use a conditional expression, though I'm not sure what to suggest since the example seems to be contrived rather than a real configuration.

[apparentlymart]

(regarding the confusing error message: Terraform really wants to say that the two tuple types have different numbers/types of elements, but it's only reporting the top level type "tuple" and not describing the elements inside.)

[AndreiPotemkin]

@apparentlymart, unless I misunderstand your response I would disagree with your point about how handling of arms of conditional expression plays into this problem.
As I mentioned above if arr attribute is defined as a list in both a and b definitions then Terraform does not have any problem evaluating conditional expression

locals {
  v = true
  a = {
    val = 1, arr = []
  }
  b = {
    val = null, arr = [1]
  }
  c = concat(
    [local.a,local.b],
    local.v ? [local.a,local.b]: []
  )
}

However if attribute arr is defined as a list in one variable but as null in the other one it causes the problem only if list [local.a, local.b] is constructed in one of the arms of conditional statement.

Maybe you can suggest an alternative solution to the actual problem I am trying to solve which can be described as follows:
I am building a list of network rules to be created where each rule is an object with multiple attributes and I would like to be able to conditionally include certain rules in the list hence my attempt to use conditional

list = concat(
   [unconditional rules],
   condition : [rules for condition] : []
)

Problem arises from scenario where some rules may require a single port to be supplied and others require a list of ports meaning that some objects being added into a list have ports attribute set to a list and other have same attribute set to null which my simplified code simulates.

[laszlocsomor]

Both the true and false arms of a conditional expression must have the same type so that Terraform can determine the result type even if the condition isn't known yet, and a tuple with two elements is a different type than a tuple with no elements.

Thanks for the explanation @apparentlymart. This makes sense.

I'm not sure what to suggest since the example seems to be contrived rather than a real configuration.

That's OK, thanks anyway! It was an actual example -- a JSON string passed to aws_cloudwatch_dashboard.dashboard_body's widgets element. To work around the behavior, I ended up separately jsonencode()'ing both arms of the conditional.

[apparentlymart]

Thanks for clarifying your concern, @laszlocsomor! The underlying concern here was obscured a bit by you only sharing a part of the error message, making me think you were reporting that the error message was incorrect rather than that the behavior was incorrect.

However, I can see one specific misbehavior here which I think is the root cause of the problem. Trying some expressions in terraform console:

> type(tolist([1, 2]))
list(number)
> type(tolist([1, null]))
list(number)
> type(tostring(null))
dynamic

The first two of these are correct:

• Converting tuple([number, number]) to a list produces a list(number) because all of the tuple elements have the same type.
• Converting tuple([number, any]) to a list produces a list(number) because any can convert to number. (Note that the inherent type of null is "any" because it doesn't specify any type information itself; Terraform infers the type of a null value only by its context.)
• However, converting any to a string as we're doing in tostring(null) should also produce type string, for the same reason as the previous conversion worked, and yet tostring is actually returning "dynamic", which is just another way of saying any when we're talking about the type of a value rather than a type constraint. The information added by the tostring call here is therefore lost.

I can see the likely cause of this unexpected result in the generic definition that we share across all of these to... functions:

terraform/internal/lang/funcs/conversion.go

Lines 24 to 36 in 3a72639

	{
	Name: "v",
	// We use DynamicPseudoType rather than wantTy here so that
	// all values will pass through the function API verbatim and
	// we can handle the conversion logic within the Type and
	// Impl functions. This allows us to customize the error
	// messages to be more appropriate for an explicit type
	// conversion, whereas the cty function system produces
	// messages aimed at _implicit_ type conversions.
	Type: cty.DynamicPseudoType,
	AllowNull: true,
	AllowMarked: true,
	},

The above states that the argument can be of any type (cty.DynamicPseudoType is the internal representation of any), it's okay for it to be null, and that the function is able to handle the function being "marked" (which is how Terraform tracks a value being "sensitive"). However, a crucial omission here is that this definition doesn't include AllowDynamicType: true, which tells the language runtime that this function isn't capable of handling a non-precisely-typed value itself and so the language runtime should just immediately return "dynamic" whenever such an argument is presented.

I think that just adding AllowDynamicType: true here would be sufficient to fix this inconsistency, because the function itself is already just delegating to the generic type conversion functionality and that already knows how to convert an untyped null into a typed null.

Doing the above would not make the original configuration as presented immediately work, because the given values are ambiguous enough about the intended result type that the automatic type inference algorithm prefers to give up and return an error rather than make an non-confident guess what the author intended.

However, I believe fixing the above so that tostring(null) returns a string-typed null would allow this to work with the addition of some explicit type conversions to guide the language runtime toward converting the values toward a single result type:

locals {
  v = true
  a = {
    val = 1, arr = tolist(null)
  }
  b = {
    val = tonumber(null), arr = tolist([1])
  }
  c = concat(
    [local.a,local.b],
    local.v ? [local.a,local.b] : []
  )
}

The tonumber(null) and tolist(null) are the most relevant to what I've described above -- that will make the val attribute have type number in both objects and the arr attribute in both cases be a list, which still doesn't exactly match but I expect would be close enough that the automatic type inference would be satisfied that it understands what you intend and thus be able to convert both [local.a, local.b] and [] to list(object({ val = number, arr = list(number) })) so that the conditional expression would be valid and produce that type as its result.

[AndreiPotemkin]

@apparentlymart, I tried few more scenarios in Terraform console and it does appear [] construct is handled differently in conditional statement.

> [{a:1, b:[]},{a:2, b:null}]
[
  {
    "a" = 1
    "b" = []
  },
  {
    "a" = 2
    "b" = null
  },
]

works however

> true ? [{a:1, b:[]},{a:2, b:null}] :[]
╷
│ Error: Inconsistent conditional result types
│ 
│   on <console-input> line 1:
│   (source code not available)
│ 
│ The true result value has the wrong type: element types must all match for conversion to list.
╵

fails

Further analysis shows this interesting behavior

> [{a:1, b:[]},{a:2, b:[1]}]
[
  {
    "a" = 1
    "b" = []
  },
  {
    "a" = 2
    "b" = [
      1,
    ]
  },
]

evaluates b in both elements to [] however when used in the conditional the same construct evaluates to tolist conversion

> true ? [{a:1, b:[]},{a:2, b:[1]}] :[]
tolist([
  {
    "a" = 1
    "b" = tolist([])
  },
  {
    "a" = 2
    "b" = tolist([
      1,
    ])
  },
])

And finally when b array in both objects has same number of elements it does not produce tolist conversion

> true ? [{a:1, b:[3]},{a:2, b:[1]}] :[]
tolist([
  {
    "a" = 1
    "b" = [
      3,
    ]
  },
  {
    "a" = 2
    "b" = [
      1,
    ]
  },
])

I sure hope PR #30879 properly handles presented failing case