gracefull upgrades through terraform

[camaeel]

Currently there is no option to trigger talos upgrade through terraform provider.
If I aplly new kubernetes version through talos resources all nodes get patched simultaneously.
It would be great to have resources to upgrade both talos itself and kubernetes in a gracefull way one by one node (like talosctl upgrade and taloscyl upgrade-k8s does).

Another option could be, to have builtin upgrade controller in talos controllers.

[M4t7e]

IMHO, this is by far one of the most important functionalities missing in the Talos Terraform provider. To manage a Talos K8s cluster using only Terraform, we need a declarative and idempotent way to specify a target version. The lifecycle of a cluster should be considered more thoroughly. The Talos and Kubernetes versions also have be compatible with the additional components that may be needed, such as external CCM, CNI, CSI, etc. Talos introduces imperative change management here, which is undesirable in a Terraform environment. Therefore, I don't see any feasible way to conduct proper lifecycle management with this module.

[hegerdes]

As a workaround you can specify the parallelism Terrafrom Docs arg and setting it to 1 default is 10. So terraform will only do one resource at the time. When you pair this with the terraform time resource you can archive wait till the node is done.

Not pretty but bast we can do right now besides from gracefully replacing nodes

[perfectra1n]

Yeah, without a way to run talosctl upgrade when changing Talos versions or extensions via this provider is rough - it makes all the work done on this provider for naught. It is possible to first create the cluster using this provider, but being unable to upgrade the cluster through this provider is a huge bummer.

[nwmcsween]

This could be done with different "vm instance/server/etc" blocks, the initial block would create and destroy on any update and the second block would depend on data http to connect before creating/destroying. This is only for upgrading Talos and not k8s.

[divStar]

I actually am interested in this, too. I suppose one could use local-exec, but it becomes very ugly very quickly.
There is also some idea I saw in https://blog.stonegarden.dev/articles/2024/08/talos-proxmox-tofu/, but to be perfectly honest I don't understand how it works.

And since I am on my homelab server with somewhat-limited resources, I wouldn't mind some downtime while upgrading, but I'd need it to e.g. destroy the current nodes and create new ones, mount the disk device blocks and boot everything - just like it currently does, but with VMs with newer versions of Talos.

Isn't it enough to just destroy and re-create the nodes with newer versions? Or should we prefer to "upgrade" running nodes?

[amaol-vestas]

We need this feature in talos terraform provider, or the alternative is just a nasty bad code practice and very unstable kubernetes and talos upgrades, please @smira take this need into account, thanks

[ionfury]

I have the same issue. I implemented a nasty workaround with local-exec and a script.

[Auroch]

I'm also very interested in this function. If you're working on it, think about the --preserve option in talosctl so that you can choose not to delete EPHEMERAL (for longhorn users, for example).

[soostdijck]

This would be a good feature to have, but as a cynical person, I doubt it will happen any time soon. Why would it, Sidero would much rather have you use Omni instead 💰

[ktoulliou]

We need this feature in talos terraform provider, or the alternative is just a nasty bad code practice and very unstable kubernetes and talos upgrades, please @smira take this need into account, thanks

I agree, we need this feature or at least an official upgrade procedure for Terraform workflows...

For bare-metal setups with multiple disks, we cannot simply destroy and recreate resources, we need a robust upgrade process for several reasons:

• Continuity of service for workloads
• Consistency with Rook-Ceph
• Patching various random bugs during node-by-node upgrades (because no one follows the same procedure)

Currently, I have tested the following approaches:

• local-exec with bash scripts (very poor solution, hard to maintain Terraform state)
• parallelism=1 (not feasible with Terragrunt)
• Ansible provider (not tested yet, seems too heavy and also hard to maintain Terraform state like local-exec)

The main problem is the Terraform for_each loop, which cannot be executed sequentially. The only "eventual" workaround is to use count.index + depends_on. However, the depends_on argument cannot contain a conditional expression... like this:

depends_on = count.index == 0 ? [] : [null_resource.upgrade_talos[count.index - 1]]

Does anyone have alternatives? Is this planned for a future provider update? Thanks