State migration issue when upgrading Helm provider from v2.x to v3.x — old resources not read correctly #1722
Description
State migration issue when upgrading Helm provider from v2.x to v3.x — old resources not read correctly
Hello, I have a Terraform state migration issue for helm_release resources when upgrading the hsshicorp/helm provider from 2.x to 3.x.
Affected versions
I have tried several versions for terraform as well as the helm provider
Terraform v1.12.2 (I also tried v1.12.0, 1.13.0, 1.14.0)
provider "helm" {
version = "3.1.0" (I also tried 3.0.0-pre2, 3.0.0, 3.0.1, 3.0.2)
}
previous provider version: 2.17.0
helm version: version.BuildInfo{Version:"v3.19.0", GitCommit:"3d8990f0836691f0229297773f3524598f46bda6", GitTreeState:"clean", GoVersion:"go1.25.1"}
OS: macOS 15.6.1
Affected resources
This is happening for all "helm_release" resources that already are in the terraform state file.
The issue
After the version upgrade and the adjustments based on the changes to the resources the existing state objects cant be read by the new provider shich results in terraform wanting to create a resource that already exists.
During the plan/apply there are the following Warn Logs:
╷
│ Warning: Failed to decode resource from state
│
│ Error decoding "helm_release.regsync_secret" from prior state: missing expected {
╵
There is a workaround that seems to work gfine, which is to remove the affected resources from the tfstate and import them afterwards but I feel this is something that should not be necessary.
My configuration
I only post one example here but there are several other (much more lenghty) resources that are also affected.
For the 2.17.0 version of the hashicorp/helm provider I had this setup:
terraform {
required_version = ">= 1.13.0, < 1.14.0"
required_providers {
helm = {
source = "hashicorp/helm"
version = "~> 2.17.0"
}
kubernetes = {
source = "hashicorp/kubernetes"
version = "~> 2.38.0"
}
}
}
provider "kubernetes" {
host = module.aks.host
cluster_ca_certificate = base64decode(module.aks.cluster_ca_certificate)
exec {
api_version = "client.authentication.k8s.io/v1beta1"
command = "kubelogin"
args = ["get-token", "--login", "azurecli", "--server-id", module.conventions.aks_add_server_application_id]
}
}
provider "helm" {
kubernetes {
host = module.aks.host
cluster_ca_certificate = base64decode(module.aks.cluster_ca_certificate)
exec {
api_version = "client.authentication.k8s.io/v1beta1"
command = "kubelogin"
args = ["get-token", "--login", "azurecli", "--server-id", module.conventions.aks_add_server_application_id]
}
}
resource "helm_release" "regsync_secret" {
name = local.application_name_regsync
namespace = kubernetes_namespace.tools.id
chart = "../../../kubernetes/charts/aks-secret-provider-class-2.0.0"
set {
name = "vaultName"
value = module.key_vault.name
}
set {
name = "tenantId"
value = module.conventions.azure_tenant_id
}
set {
name = "clientId"
value = module.workload_identity_regsync.client_id
}
set_list {
name = "secrets"
value = [
data.azurerm_key_vault_secret.gebit_quay_username.name,
data.azurerm_key_vault_secret.gebit_quay_password.name
]
}
force_update = true
}After the changes for the 3.x version:
terraform {
required_version = ">= 1.13.0, < 1.14.0"
required_providers {
helm = {
source = "hashicorp/helm"
version = "~> 3.1.0"
}
kubernetes = {
source = "hashicorp/kubernetes"
version = "~> 2.38.0"
}
}
}
provider "kubernetes" {
host = module.aks.host
cluster_ca_certificate = base64decode(module.aks.cluster_ca_certificate)
exec {
api_version = "client.authentication.k8s.io/v1beta1"
command = "kubelogin"
args = ["get-token", "--login", "azurecli", "--server-id", module.conventions.aks_add_server_application_id]
}
}
provider "helm" {
registries = [{
url = "oci://${module.conventions.acr_name_shared}.azurecr.io/helm"
username = "00000000-0000-0000-0000-000000000000" # Dummy-User for ACR OAuth2
password = local.acr_token
}]
}
resource "helm_release" "regsync_secret" {
name = local.application_name_regsync
namespace = kubernetes_namespace.tools.id
chart = "../../../kubernetes/charts/aks-secret-provider-class-2.0.0"
set = [
{
name = "vaultName"
value = module.key_vault.name
},
{
name = "tenantId"
value = module.conventions.azure_tenant_id
},
{
name = "clientId"
value = module.workload_identity_regsync.client_id
}
]
set_list = [
{
name = "secrets"
value = [
data.azurerm_key_vault_secret.gebit_quay_username.name,
data.azurerm_key_vault_secret.gebit_quay_password.name
]
}
]
force_update = true
}
Expected Behaviour
I would ewxpect that a major upgrade of the hashicorp/helm provider (from 2.x to 3.x in this case) can handle tfstate objects from the previous version.
Actual Behaviour
A terraform plan outut after version upgrade and code adjustments results in something like this (target is in effect since the whole plan for this state is super long):
helm_release.regsync_secret: Refreshing state... [id=regsync]
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following
symbols:
+ create
Terraform will perform the following actions:
# helm_release.regsync_secret will be created
+ resource "helm_release" "regsync_secret" {
+ atomic = false
+ chart = "../../../kubernetes/charts/aks-secret-provider-class-2.0.0"
+ cleanup_on_fail = false
+ create_namespace = false
+ dependency_update = false
+ disable_crd_hooks = false
+ disable_openapi_validation = false
+ disable_webhooks = false
+ force_update = true
+ id = (known after apply)
+ lint = false
+ max_history = 0
+ metadata = (known after apply)
+ name = "regsync"
+ namespace = "tools"
+ pass_credentials = false
+ recreate_pods = false
+ render_subchart_notes = true
+ replace = false
+ reset_values = false
+ reuse_values = false
+ set = [
+ {
+ name = "vaultName"
+ value = "kv-main-shared-gwc-01"
# (1 unchanged attribute hidden)
},
+ {
+ name = "tenantId"
+ value = "857a7b86-2d66-46f2-92e1-25be0c27e398"
# (1 unchanged attribute hidden)
},
+ {
+ name = "clientId"
+ value = "c723985c-e1cf-4cca-9382-d2953ad579dc"
# (1 unchanged attribute hidden)
},
]
+ set_list = [
+ {
+ name = "secrets"
+ value = [
+ "regsync-gebit-quay-username",
+ "regsync-gebit-quay-password",
]
},
]
+ set_wo = (write-only attribute)
+ skip_crds = false
+ status = "deployed"
+ timeout = 300
+ verify = false
+ version = "2.0.0"
+ wait = true
+ wait_for_jobs = false
}
Plan: 1 to add, 0 to change, 0 to destroy.
╷
│ Warning: Resource targeting is in effect
│
│ You are creating a plan with the -target option, which means that the result of this plan may not represent all of the changes
│ requested by the current configuration.
│
│ The -target option is not for routine use, and is provided only for exceptional situations such as recovering from errors or
│ mistakes, or when Terraform specifically suggests to use it as part of an error message.
╵
╷
│ Warning: Failed to decode resource from state
│
│ Error decoding "helm_release.internal_nginx_ingress_controller" from prior state: missing expected {
╵
╷
│ Warning: Failed to decode resource from state
│
│ Error decoding "helm_release.cert_manager_cluster_issuer_prod_public" from prior state: missing expected {
╵
╷
│ Warning: Failed to decode resource from state
│
│ Error decoding "helm_release.kuberpult_environment_update_secret_provider_class" from prior state: missing expected {
╵
╷
│ Warning: Failed to decode resource from state
│
│ Error decoding "helm_release.nginx_ingress_controller" from prior state: missing expected {
╵
╷
│ Warning: Failed to decode resource from state
│
│ Error decoding "helm_release.private_external_dns" from prior state: missing expected {
╵
╷
│ Warning: Failed to decode resource from state
│
│ Error decoding "helm_release.cert_manager_cluster_issuer_stage_priv" from prior state: missing expected {
╵
╷
│ Warning: Failed to decode resource from state
│
│ Error decoding "helm_release.public_external_dns" from prior state: missing expected {
╵
╷
│ Warning: Failed to decode resource from state
│
│ Error decoding "helm_release.datadog_agent" from prior state: missing expected {
╵
╷
│ Warning: Failed to decode resource from state
│
│ Error decoding "helm_release.kuberpult" from prior state: missing expected {
╵
╷
│ Warning: Failed to decode resource from state
│
│ Error decoding "helm_release.cert_manager" from prior state: missing expected {
╵
╷
│ Warning: Failed to decode resource from state
│
│ Error decoding "helm_release.kuberpult_test_environment_update_secret_provider_class" from prior state: missing expected {
╵
╷
│ Warning: Failed to decode resource from state
│
│ Error decoding "helm_release.cert_manager_cluster_issuer_stage_public" from prior state: missing expected {
╵
╷
│ Warning: Failed to decode resource from state
│
│ Error decoding "helm_release.argo_cd" from prior state: missing expected {
╵
╷
│ Warning: Failed to decode resource from state
│
│ Error decoding "helm_release.kuberpult_test" from prior state: missing expected {
╵
╷
│ Warning: Failed to decode resource from state
│
│ Error decoding "helm_release.cert_manager_cluster_issuer_prod_priv" from prior state: missing expected {
╵
╷
│ Warning: UpgradeState Triggered
│
│ with helm_release.regsync_secret,
│ on k8s-regsync.tf line 134, in resource "helm_release" "regsync_secret":
│ 134: resource "helm_release" "regsync_secret" {
│
│ Successfully migrated state from SDKv2 to Plugin Framework
╵
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run
"terraform apply" now.
Releasing state lock. This may take a few moments...
Note that only with the --target option the last warning reads different. It suggests that the migration from SDKv2 to Plugin Framework worked for the targeted resource but for none of the other helm_release resource, but based on the planned action - a new resource to be cerated - this seems not to work and the old state object was not recognized correctly.
When looking into the tfstate file I can see that the current schema_version is 1:
{
"mode": "managed",
"type": "helm_release",
"name": "regsync_secret",
"provider": "provider[\"registry.terraform.io/hashicorp/helm\"]",
"instances": [
{
"schema_version": 1,
"attributes": {
"atomic": false,
"chart": "../../../kubernetes/charts/aks-secret-provider-class-2.0.0",
"cleanup_on_fail": false,
"create_namespace": false,
"dependency_update": false,
"description": null,
"devel": null,
"disable_crd_hooks": false,
"disable_openapi_validation": false,
"disable_webhooks": false,
"force_update": true,
"id": "regsync",
"keyring": null,
"lint": false,
"manifest": null,
"max_history": 0,
"metadata": [
{
"app_version": "1.0.0",
"chart": "aks-secret-provider-class",
"first_deployed": 1721382452,
"last_deployed": 1724683527,
"name": "regsync",
"namespace": "tools",
"notes": "",
"revision": 5,
"values": "<my-values :)>",
"version": "2.0.0"
}
],
"name": "regsync",
"namespace": "tools",
"pass_credentials": false,
"postrender": [],
"recreate_pods": false,
"render_subchart_notes": true,
"replace": false,
"repository": null,
"repository_ca_file": null,
"repository_cert_file": null,
"repository_key_file": null,
"repository_password": null,
"repository_username": null,
"reset_values": false,
"reuse_values": false,
"set": [
{
"name": "clientId",
"type": "",
"value": "clientId"
},
{
"name": "tenantId",
"type": "",
"value": "tenantId"
},
{
"name": "vaultName",
"type": "",
"value": "vaultName"
}
],
"set_list": [
{
"name": "secrets",
"value": [
"regsync-gebit-quay-username",
"regsync-gebit-quay-password"
]
}
],
"set_sensitive": [],
"skip_crds": false,
"status": "deployed",
"timeout": 300,
"upgrade_install": null,
"values": [],
"verify": false,
"version": "2.0.0",
"wait": true,
"wait_for_jobs": false
},
"sensitive_attributes": [
[
{
"type": "get_attr",
"value": "repository_password"
}
]
],
"identity_schema_version": 0,
"private": "private",
"dependencies": [
"azurerm_public_ip.aks_outbound",
"azurerm_resource_group.main",
"data.azurerm_key_vault_secret.gebit_quay_password",
"data.azurerm_key_vault_secret.gebit_quay_username",
"data.external.acr_token",
"kubernetes_namespace.tools",
"module.aks.azurerm_kubernetes_cluster.this",
"module.aks.azurerm_role_assignment.uai_network_contributor",
"module.aks.azurerm_role_assignment.uai_private_dns_zone_contributor",
"module.aks.azurerm_user_assigned_identity.this",
"module.aks.data.azurerm_virtual_network.this",
"module.aks_private_dns_zone.azurerm_private_dns_zone.this",
"module.aks_private_dns_zone.azurerm_private_dns_zone_virtual_network_link.vnet",
"module.key_vault.azurerm_key_vault.this",
"module.vnet.azurerm_subnet.this",
"module.vnet.azurerm_virtual_network.this",
"module.vnet.data.azurerm_virtual_network.this",
"module.workload_identity_regsync.azurerm_user_assigned_identity.this"
]
}
]
}
References
I found several online resources that address the migration to the 3.x mayor version.
There is the official migration guide in which the migration from SDKv2 to Framework plugin is also mentioned:
https://github.com/hashicorp/terraform-provider-helm/blob/v3.0.0-pre1/docs/guides/v3-upgrade-guide.md
This closed github issue seems to address the same problem:
#1574
But it was closed after a found workaround that involves manually adjusting the tfstate file
This existing issue seems slightly different to me:
#1720
This issue is similar also but here the schema_version was 0:
#1698
None of thema are still open or specifically matching this case so I decieded to create a new issue. I hope you find all the relevant information here - if not I am happy to provide more.