Unknown values when decoding empty but yaml documents

[lexton]

Terraform Version

Terraform v1.1.9
on darwin_arm64

Terraform code

output "test" {
    value = {
        empty = try(yamldecode(file("./empty.yml")), {})
        started = yamldecode(file("./started.yml"))
        val = yamldecode(file("./val.yml"))
    }
}

Debug Output

$ terraform plan

  + test = {
      + empty   = {}
      + started = (known after apply)
      + val     = {
          + a = "b"
          + c = 1
        }
    }

$ terraform apply -auto-approve 

Changes to Outputs:
  + test = {
      + empty   = {}
      + started = (known after apply)
      + val     = {
          + a = "b"
          + c = 1
        }
    }

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

test = {
  "empty" = {}
  "started" = null
  "val" = {
    "a" = "b"
    "c" = 1
  }
}

$ terraform apply -auto-approve 

Changes to Outputs:
  ~ test = {
      ~ started = null -> (known after apply)
        # (2 unchanged elements hidden)
    }

You can apply this plan to save these new output values to the Terraform state, without changing any real infrastructure.

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

test = {
  "empty" = {}
  "started" = null
  "val" = {
    "a" = "b"
    "c" = 1
  }
}

Expected Behavior

started should have value null at plan time.

I'm not sure that this behavior is defined in the docs - but its important because the apply promise of 'unknown' is a contagious property, and will force values to be reloaded and re-evaluated each terraform apply.

Actual Behavior

started has an unknown value at plan.

Steps to Reproduce

#!/bin/bash   
touch empty.yml

cat << EOF > started.yml
---

EOF

cat << EOF > val.yml
a: b
c: 1

EOF

cat << EOF > main.tf
output "test" {
    value = {
        empty = try(yamldecode(file("./empty.yml")), {})
        started = yamldecode(file("./started.yml"))
        val = yamldecode(file("./val.yml"))
    }
}
EOF
terraform init
terraform plan

Additional Context

no additional context

References

Unrelated but indicative reading: https://log.martinatkins.me/2021/06/14/terraform-plan-unknown-values/

[alisdair]

Thanks for the report! I can confirm this behaviour using the reproduction case given. The issue only seems to occur with YAML files which consistent entirely of the document separator (---). I'm not sure why this is the case, but it seems like a bug in the yamldecode function to me.

[alisdair]

A little more digging shows that the yamldecode implementation decodes this essentially empty document as a null value of unknown type, which results in Terraform deferring its evaluation until apply time. I think to be consistent, we ought to decode truly empty documents and those which contain only a document separator similarly, although I'm not sure that our backwards-compatibility promises will permit making that change.

[apparentlymart]

For what it's worth, replacing something unknown with something known is typically not a breaking change as long as the change of value upholds the same rules Terraform would normally check between the plan phase and the apply phase.

For a null value with an incomplete type constraint it should typically be okay to replace it with a null value of an exact type as long as the exact type conforms to the constraint. Any type conforms to an entirely unknown type, so that should be okay here as far as I can reason.

[apparentlymart]

Looking at the implementation, it seems like the main logic error is in assuming that an unknown return type means to return DynamicVal (unknown value of unknown type), when as you say in this case it seems like the type function is intending it to mean a null value of an unknown type, which would be okay in principle.

I think there is some question here about what an entirely empty YAML document is meant to represent. I guess this current logic (aside from erroneously handling the null as an unknown value) is trying to treat a totally empty document the same as a YAML null (~). Perhaps the YAML spec had something to say about whether treating it as null is correct or if we should be doing something more magical like inferring an empty object. 🤔

[apparentlymart]

The specification seems to support the intended behavior of returning a null value of an unknown type here, although it's split into two parts:

• The section about explicitly-delimited documents says that a consequence of explicit delimiters is that it becomes possible to encode a completely empty document, and links to the "empty nodes" section to define.
• The empty content example has some surrounding text that says that an empty node is "typically" treated as null, and so it seems like the actual behavior is undefined but we have no context that would imply doing anything else, and so I guess it's good for us to do what is "typical".

I think then the upstream fix here might be to just remove that special case about the DynamicPseudoType from the value function, or to make it explicitly return a null value of that type in that situation instead.

If course this is an upstream dependency rather than part of Terraform, so it will need to be addressed upstream and then upgraded here for Terraform to get the new behavior.