usage of distinct hurts performance

[freedge]

Terraform Version

Terraform v1.5.0-dev
on linux_amd64

Terraform Configuration Files

https://github.com/freedge/tftest

we use a large expression containing the distinct keyword

locals {
  address_map = {
      for address in distinct(flatten([for endpoint in var.content: endpoint])): address.name => address
  }
}

resource "null_resource" "null_objects" {
  lifecycle {
    create_before_destroy = true
  }
  for_each    = local.address_map
  triggers = {
        name        = each.key,
  }
}

Debug Output

...
2023-03-29T06:23:54.760Z [TRACE] vertex "module.addresses.local.address_map (expand)": starting visit (*terraform.nodeExpandLocal)
2023-03-29T06:23:54.760Z [TRACE] vertex "module.addresses.local.address_map (expand)": expanding dynamic subgraph
2023-03-29T06:23:54.760Z [TRACE] Expanding local: adding module.addresses.local.address_map as *terraform.NodeLocal
2023-03-29T06:23:54.760Z [TRACE] vertex "module.addresses.local.address_map (expand)": entering dynamic subgraph
2023-03-29T06:23:54.760Z [TRACE] vertex "module.addresses.local.address_map": starting visit (*terraform.NodeLocal)
2023-03-29T06:23:59.003Z [TRACE] dag/walk: vertex "provider["registry.terraform.io/hashicorp/null"] (close)" is waiting for "module.addresses.null_resource.null_objects (expand)"
2023-03-29T06:23:59.003Z [TRACE] dag/walk: vertex "root" is waiting for "provider["registry.terraform.io/hashicorp/null"] (close)"
2023-03-29T06:23:59.003Z [TRACE] dag/walk: vertex "module.addresses (close)" is waiting for "module.addresses.null_resource.null_objects (expand)"
2023-03-29T06:23:59.041Z [TRACE] dag/walk: vertex "module.addresses.null_resource.null_objects (expand)" is waiting for "module.addresses.local.address_map (expand)"
2023-03-29T06:23:59.760Z [TRACE] dag/walk: vertex "root" is waiting for "module.addresses.local.address_map"
...

Expected Behavior

terraform plan, apply, show, should be reasonably slow

Actual Behavior

terraform takes CPU during several minutes, which is unreasonably slow

this is aggravated by the usage of the distinct function. If we remove it it's more reasonable:

• x: the amount of elements in the content input array
• y: the time in second to complete the terraform plan

Steps to Reproduce

git clone https://github.com/freedge/tftest.git
terraform init
terraform plan

initially reproduced on v1.4.2 and on current main branch.

Additional Context

this was observed initially while using the https://github.com/PaloAltoNetworks/terraform-provider-panos provider.
We are considering running a preprocessing script to clean up our input content, so that terraform eat pre-mashed data that are unique already, but it would be so great to have improvements in terraform itself.

References

• Backport of core: Reduce string allocations for addrs Equal into v0.15 #28000

[jbardin]

Hi @freedge,

Thanks for filing the issue! It does look like that distinct function in the upstream library has a quadratic complexity, so large numbers of inputs will impact runtime significantly. We can look into improving that package.

Thanks!

[freedge]

awesome!
these many objects are way above best practices from https://cloud.google.com/docs/terraform/best-practices-for-terraform#minimize-resources

I was also looking into possible ways to use bulk methods (like https://github.com/PaloAltoNetworks/terraform-provider-panos/blob/b4ad451eb47b2c7e463d94b13fd7e2a5a62db158/docs/resources/address_objects.md, with a dynamic object with 8k elements) and hit other performance issues. I don't know if a separate case would be helpful (or a ticket on the provider side) nevertheless the pprof looked like this:

do you know if terraform is tested using that many resources? Big cheers!

[jbardin]

Large single objects should generally be ok, but Terraform is optimized for resolving dependencies and ordering of operations, so large amounts of raw data are not always handled optimally. For example, evaluating large numbers of resource instances can run into #26355, for which there is no good workaround right now with the given language design.

There is no set limit for how many resources is too many, and we can make improvements as we run into limitations, but users generally make "takes too long to plan" the self-imposed limit. If you do have specific cases that are of concern, please file new issues. This one will be closed if changes are made upstream to the distinct function.

Thanks!