'plan -out ...' fails with a brand new state file when dealing with some 'data' resources

[usrme]

Terraform Version

Terraform v1.4.1
on linux_amd64
+ provider registry.terraform.io/carlpett/sops v0.7.2
+ provider registry.terraform.io/gitlabhq/gitlab v15.9.0
+ provider registry.terraform.io/hashicorp/azuread v2.20.0
+ provider registry.terraform.io/hashicorp/azurerm v3.47.0
+ provider registry.terraform.io/hashicorp/google v4.9.0
+ provider registry.terraform.io/hashicorp/null v3.1.1
+ provider registry.terraform.io/hashicorp/random v3.1.3

Terraform Configuration Files

# module.key_vaults.data.sops_file.secrets
data "sops_file" "secrets" {
  source_file = "secrets.enc.yaml"
}

# module.storage_accounts.data.azurerm_resources.files_storage_account
data "azurerm_resources" "files_storage_account" {
  resource_group_name = var.resource_group.name
  type                = "Microsoft.Storage/storageAccounts"

  required_tags = {
    type = "general-templates"
  }
}

Debug Output

Expected Behavior

terraform plan -out ... should have finished as it would normally in any other case.

Actual Behavior

$ terraform plan -var "admin_user_ssh_key_path=~/.ssh/id_rsa_.pub" -out plan.tfstate
data.azuread_group.admins: Reading...
data.azuread_group.admins: Read complete after 1s [id=...]
data.azurerm_subscription.primary: Reading...
data.azurerm_subscription.primary: Read complete after 0s [id=...]
╷
│ Error: Failed to write plan file
│ 
│ The plan file could not be written: failed to write state snapshot: 2 problems:
│ 
│ - Failed to serialize resource instance in state: Instance module.key_vaults.data.sops_file.secrets has status ObjectPlanned, which cannot be saved in state.
│ - Failed to serialize resource instance in state: Instance module.storage_accounts.data.azurerm_resources.files_storage_account has status ObjectPlanned, which cannot be saved in
│ state..

Steps to Reproduce

Additional Context

This is something that started happening after upgrading to 1.4.0 from 1.3.9 and persists even in 1.4.1. Curiously, the same configuration passes terraform plan -out ... without issues if I'm working on an existing state, but if I am creating a new state file by way of terraform workspace new <name> and run terraform plan -out ..., then it crashes.

I am also using the following data resources, but those do not seem to be affected:

data "azurerm_subscription" "primary" {
}

data "azuread_group" "admins" {
  display_name = "Admins"
}

I'm using GCS as the back-end for my state files.

References

No response

[alisdair]

Hi @usrme, thanks for reporting this.

I'm having difficulty reproducing this issue. Since your report only includes a snippet, I've been trying this analogous configuration:

resource "null_resource" "null" {
}

# Reads at plan time
data "null_data_source" "one" {
  inputs = {
    id = "foo"
  }
}

# Deferred read
data "null_data_source" "two" {
  inputs = {
    id = null_resource.null.id
  }
}

This plans (and saves the plan with -out) using Terraform 1.4.0 and 1.4.1.

Are you able to share the full configuration, or a cut-down version of it which demonstrates the same bug?

[usrme]

@alisdair, thanks for having a look! The configuration that is failing since 1.4.0 is quite complex and in its entirety would expose things I don't want being exposed, and I'm not as versed with Terraform as I'd like to be able to provide a minimally reproducible example that cuts through all the depends_on and other bits, but I'll try to get something for you within 24 hours.

I'm considering doing a git bisect procedure with all of the commits from 1.3.9 to 1.4.0 to see where the potential bug was introduced as then I wouldn't have to faff about with the intricacies of the configuration.

Here's what I am able to share though:

main.tf:

module "storage_accounts" {
  source         = "./modules/storage-accounts"
  resource_group = module.resource_groups.mgmt_rg
  depends_on = [
    module.resource_groups
  ]
}

module "key_vaults" {
  source         = "./modules/key-vaults"
  tenant_id      = data.azurerm_subscription.primary.tenant_id
  admin_group_id = data.azuread_group.admins.object_id
  resource_groups = {
    mgmt = module.resource_groups.mgmt_rg
  }
  mgmt_secrets = [
    module.storage_accounts.files_storage_account.secrets
  ]
  depends_on = [
    module.service_principals,
    module.user_assigned_identities
  ]
}

module.storage_accounts.main.tf:

resource "random_string" "files_storage_account_suffix" {
  length  = 12
  special = false
  upper   = false
}

resource "azurerm_storage_account" "files" {
  name                     = local.files_storage_account
  resource_group_name      = var.resource_group.name
  location                 = var.resource_group.location
  account_kind             = "StorageV2"
  account_tier             = "Standard"
  account_replication_type = "LRS"
  access_tier              = "Hot"

  tags = {
    type = "general-templates"
  }

  lifecycle {
    ignore_changes = [
      tags,
    ]
  }
}

resource "azurerm_storage_container" "general_templates" {
  name                  = "general-templates"
  storage_account_name  = azurerm_storage_account.files.name
  container_access_type = "private"
}

module.storage_accounts.outputs.tf:

output "files_storage_account" {
  description = "'files' storage account object."
  value = {
    files = azurerm_storage_account.files
    secrets = {
      "${var.storage_account_prefix}-${terraform.workspace}-files-name" = azurerm_storage_account.files.name
      "${var.storage_account_prefix}-${terraform.workspace}-files-key"  = azurerm_storage_account.files.primary_access_key
    }
  }
}

module.storage_accounts.variables.tf:

variable "resource_group" {
  default = null
}

variable "location" {
  default = null
}

variable "storage_account_prefix" {
  type    = string
  default = "sa"
}

# Use 'azurerm_resources' instead of 'azurerm_storage_account'
# when exact name can't be given ahead of time due to the fact
# that all storage account names must be globally unique
data "azurerm_resources" "files_storage_account" {
  resource_group_name = var.resource_group.name
  type                = "Microsoft.Storage/storageAccounts"

  required_tags = {
    type = "general-templates"
  }
}

locals {
  fetched_files_storage_account_name = try(data.azurerm_resources.files_storage_account.resources.*.name[0], "")
  files_storage_account              = local.fetched_files_storage_account_name != "" ? local.fetched_files_storage_account_name : "files${resource.random_string.files_storage_account_suffix.result}"
}

module.key_vaults.main.tf:

terraform {
  required_providers {
    sops = {
      source  = "carlpett/sops"
      version = "~> 0.7"
    }
  }
}

resource "azurerm_key_vault" "kv" {
  for_each            = toset(var.kv_names)
  name                = "${var.kv_prefix}-global-${terraform.workspace}-${each.value}"
  location            = var.resource_groups[each.value].location
  resource_group_name = var.resource_groups[each.value].name
  tenant_id           = var.tenant_id
  sku_name            = "standard"

  lifecycle {
    ignore_changes = [
      tags,
    ]
  }
}

data "sops_file" "secrets" {
  source_file = "secrets.enc.yaml"
}

resource "azurerm_key_vault_secret" "lab_secret" {
  for_each     = nonsensitive(data.sops_file.secrets.data)
  name         = each.key
  value        = each.value
  key_vault_id = azurerm_key_vault.kv["lab"].id

  depends_on = [
    azurerm_key_vault_access_policy.admins["lab"]
  ]
}

resource "azurerm_key_vault_secret" "mgmt_secret" {
  for_each = { for pair in local.mgmt_secret_pairs : "${pair.secret_name}" => pair.secret_value }
  name     = each.key
  # Accommodate values that are maps and contain key "value" that holds
  # actual value, plus additional meta-data like an expiration date
  value = try(lookup(each.value, "value", each.value), each.value)
  # When no "expiration" key exists for "value", then it is safe to assign
  # a "null" value and then no expiration date is set
  expiration_date = try(each.value.expiration, null)
  key_vault_id    = azurerm_key_vault.kv["mgmt"].id

  depends_on = [
    azurerm_key_vault_access_policy.admins["mgmt"]
  ]
}

module.key_vaults.variables.tf:

variable "kv_prefix" {
  type    = string
  default = "kv"
}

variable "tenant_id" {
  type    = string
  default = null
}

variable "admin_group_id" {
  type    = string
  default = null
}

variable "mgmt_secrets" {
  type    = any
  default = []
}

variable "resource_groups" {
  type    = map(any)
  default = null
}

variable "kv_names" {
  description = "List of Key Vaults to create"
  type        = list(string)
  default     = ["mgmt", "lab"]
}

locals {
  mgmt_secret_pairs = flatten([
    for secret_group in var.mgmt_secrets : [for k, v in secret_group : { "secret_name" = k, "secret_value" = v }]
  ])
}

I realize this is too much, but hopefully it is something.

[jbardin]

Thanks @usrme, I have a handle on where this is going wrong. I think there's an error in your plan which we aren't getting to see, and that error path is skipping some of the state cleanup which normally happens. Once we fixup state handling, hopefully the remaining error is something easy to diagnose.

[usrme]

@jbardin, that's great to hear! I'm in no hurry to get this resolved as 1.3.9 is fine for me, but are you able to perhaps share an ETA for the fix? Also, should I still try and get that minimally viable reproduction or not?

[jbardin]

@usrme, don't worry about the reproduction, I'll have to come up with something more specific for a test anyway. If the problem I found is the only part of this issue we need to worry about, it should be fixed in an upcoming patch release.