Current Terraform Version
/terraform -v
Terraform v0.11.10
Use-cases
We have a wide deployment of Consul in our organization. We also use Terraform to manage various resources in each of datacenters.
Using Consul remote state backend sounds like a natural solution but even with compression enabled we quickly ran into Consul's key size limit. It would've been great if the backend was able to somehow split the state in chunks to manage states of arbitrary size.
Attempted Solutions
Tried enabling compression. Help for a bit but then state grew and doesn't help anymore.
Switched to etcdv3 but would prefer to use Consul so that we do not need to run etcd.
Proposal
I believe there many ways to do this and I am not sure I can come up with best proposals.
From the top of my head, I can think of pre-processing the state by compressing and/or Base64 encoding it into a string blob that can be split over multiple keys that would be combined together.
This would require using Consul transaction API and it means that we can use up to 64 keys but that should provide significant improvement over current limitation.
Current Terraform Version
Use-cases
We have a wide deployment of Consul in our organization. We also use Terraform to manage various resources in each of datacenters.
Using Consul remote state backend sounds like a natural solution but even with compression enabled we quickly ran into Consul's key size limit. It would've been great if the backend was able to somehow split the state in chunks to manage states of arbitrary size.
Attempted Solutions
Tried enabling compression. Help for a bit but then state grew and doesn't help anymore.
Switched to
etcdv3but would prefer to use Consul so that we do not need to run etcd.Proposal
I believe there many ways to do this and I am not sure I can come up with best proposals.
From the top of my head, I can think of pre-processing the state by compressing and/or Base64 encoding it into a string blob that can be split over multiple keys that would be combined together.
This would require using Consul transaction API and it means that we can use up to 64 keys but that should provide significant improvement over current limitation.