Lock Files, Plugin Cache and using several architectures fails `terraform init`

[LDVSOFT]

So, since I am using several stacks of Terraform configuration quite regularly and their plugins in use are mostly the same, I didn't want providers to be downloaded for each one, so I enabled plugin cache directory (see below). Also, at some point in time I started using lock files instead of relying on using version = "= x.y" to always reuse the same version.

Since we use Terraform on several machines, and some developers are using different architectures (I'm using Linux Amd64, my colleague is using Darwin Amd64, more architectures might come with M1), I expect lock files to support being used on several architectures. Documentation states that even though h1s only for current architectures are populated, since zhs are always present they are enough to check either plugin for a new architecture already matches known hashes, so that Terraform can append new h1 hash on a first download. Sounds great, actually (or one can assume we can remember h1s directly, but it's not my expertise).

The only problem is that, somehow, Terraform always does a little different thing for me:

1. When I run terraform init that resolved to a plugin version that is not in the cache, Terraform actually does it's job: it downloads the plugin, populates the lock file with all zhs and a single h1. If I will use this lock file on a different machine everything works as expected.
2. When I run terraform init for the same plugin version in another directory, it is accessed from the cache, and only h1 hash is written into the lock file.
3. When my colleague will run terraform init on the second stack with my lock file (if I commit it into VCS), they will get an error.

As I have found out, instead of trying to use different hardware I can emulate the behaviour by just running terraform providers lock -platform=darwin_amd64, where platform can be anything but not the current one.

Terraform Version

Terraform v1.0.11
on linux_amd64
+ provider registry.terraform.io/hashicorp/null v3.1.0

Also I've seen this on a lot of versions, since, I think, around 0.15.

Terraform Configuration Files

There is nothing relevant beyond version requirements:

terraform {
  required_version = ">= 1.0.4"
  required_providers {
    null = {
      source  = "hashicorp/null"
      version = "~> 3.1"
    }
  }
}

Relevant ~/.terraformrc:

plugin_cache_dir = "$HOME/.cache/terraform/plugin-cache"

Debug Output

This Gist includes all logs

Expected Behavior

terraform init should populate lock file well enough for future use or switch to redownloading plugin in case lock file doesn't provide enough information.

Actual Behavior

Terraform disrupts developer experience by forcing to do terraform providers lock -platform=… with every platform in use so that terraform init wouldn't fail in the future.

Steps to Reproduce

Here I assume that I'm running on linux_amd64, and I'm using darwin_amd64 and darwin_arm64 as different achitectures. However, I experienced this bug with other combinations, I don't think exact platforms are relevant here.

1. Create ~/.terraformrc file with line above, create given directory / check that it's empty.
2. Create an empty directory a with versions.tf file as above.
3. terraform init in it. Should work like a charm.
4. Open .terraform.lock.hcl to see that there are a lot of hash lines.
5. terraform providers lock -platform=darwin_amd64. Works, lock file is updated with a new h1 hash.
6. Create an empty direcory b with the same versions.tf file.
7. terraform init in it. Should work.
8. However, terraform.lock.hcl only contains a single h1 hash.
9. terraform providers lock -platform=darwin_arm64. Fails:

   ╷
   │ Error: Could not retrieve providers for locking
   │
   │ Terraform failed to fetch the requested providers for darwin_arm64 in order
   │ to calculate their checksums: some providers could not be installed:
   │ - registry.terraform.io/hashicorp/null: the current package for
   │ registry.terraform.io/hashicorp/null 3.1.0 doesn't match any of the
   │ checksums previously recorded in the dependency lock file.
   ╵
   

Additional Context

I am sorry, this can be already discussed in another Cache+Lock-related issue, but I haven't seen exactly my case being described. I am opening this one with the exact reproducible case.

References

For a lot of time I was following #28041 for this problem, but it seems to me it has lost the direction I am interested in.

[apparentlymart]

Hi @LDVSOFT! Thanks for raising this.

Unfortunately what you've described here is the current expected behavior of Terraform: if you activate any configuration that prevents Terraform from accessing the origin registry then it won't have access to the official signed checksums from the registry and thus it must conservatively only record what it can determine from the information already available locally.

In your case, you've enabled the plugin cache and therefore effectively disabled Terraform accessing the registry to reinstall the same version of a previously-seen provider, which causes this problem.

terraform providers lock is the current way to get a comprehensive set of checksums in any situation where you can't or don't want to use the default provider installation behaviors, which would normally be able to populate the lock file manually. I think #27388 is covering that concern.

Towards the end of your comment here you also mentioned terraform providers lock returning an error, which does seem different than what was discussed before. Based on your reproduction steps I think that happened because in your step 5 you only included darwin_amd64, and so you generated a lock file that only supports that platform. To get a successful result from that series of steps you would need to include all of the different platforms your team uses in your initial terraform providers lock call, which would then generate a sufficient lock file for everyone on your team.

[LDVSOFT]

Thank you, @apparentlymart, for explaining the current situation for me.

I find it a bit misleading that the cache for plugins prevents accessing the origin registry, it's strange. A lot of other tools use dependency caching by default and also support version or/and integrity locking (Maven, Gradle, Pip, NPM, Yarn, Cargo come to my mind), but when those miss out on information they don't hesitate accessing the origin. Maybe just store validated hashes (zhs for now) in the cache too? In my eyes, the cache is helping with storage space & network traffic and is not a source of trust for any user beyond local machine, so it will fit there.

I don't know how exactly Terraform operates it's cache from a security standpoint (especially since the whole process of unzipping the distribution), so I don't insist on anything.

At least the docs can be improved with a notice about the behaviour: as opposed to provider_installation that obviously intervents into where plugins are installed from. Users may end up with very non-trivial problems from lock files and will blame those, not the cache.

[Phylu]

I ran in the very same issue during a different workflow in a similar situation:

• We have different repositories which all use the same provider version. To prevent re-downloading it for every repository, I am using the plugin_cache_dir.
• We have 4 different architectures (darwin_amd64, darwin_arm64, linux_adm64 & linux_arm64), for local developers and our pipeline servers.

On a version update

• we update the versions.tf file
• Run terraform init --upgrade
• Run providers lock -platform=darwin_amd64 -platform=darwin_arm64 -platform=linux_arm64 -platform=linux_amd64.

Then providers lock command fails with Terraform failed to fetch the requested providers for darwin_arm64 in order to calculate their checksums.

I agree with @LDVSOFT here. It would be great to have the possibility to use the providers lock command in addition with the plugin cache directory.

[apparentlymart]

Based on the discussion here so far, it seems like there may be a more tactical fix that we could employ to smooth out this terraform providers lock workflow in the meantime, until the registry can be updated to support the new-style hashing scheme:

Based on my testing, it seems like terraform providers lock can only add new checksums to an existing lock entry if there's already a saved checksum matching each of the downloaded packages. That means it's able to "upgrade" an existing entry from the zh: scheme to the h1: scheme, but it fails with the error discussed above if there is no existing zh: checksum to correlate with.

That rule makes sense for the "trust on first use" (TOFU) paradigm of terraform init, but it's not so obviously a correct behavior for terraform providers lock whose explicit purpose is to add new checksums to the lock file. Instead, it seems to me like terraform providers lock should do as it's told and just add the requested package checksums, regardless of what was already in the file for that provider before.

A detail here is to think through what extra output, if any, terraform providers lock ought to produce in that case. In the case where there was no existing entry for a particular provider this command will happily record an entirely new set of checksums without any special notice, and so arguably it could do the same for newly-discovered checksums of an existing provider, but since this behavior would differ from the TOFU behavior of terraform init it might be better to be a bit more "chatty" in that situation and announce what happened:

hashicorp/null v3.1.0 for darwin_amd64 checksum not previously known; now recorded in the lock file

For consistency, it may be best for the command to be a little more chatty in general by saying something about all of the packages it considers:

hashicorp/null v3.1.0 for linux_amd64 checksum already tracked in the lock file
hashicorp/aws v2.0.0 checksums recorded for all requested platforms

From a practical standpoint I do expect that most folks will not carefully inspect the output here, but in some environments there is a policy to be extra careful with adding new checksums to the lock file and so by prompting about it we at least give the opportunity to notice that something unusual happened. While it would be possible to see some information by looking directly at the diff for .terraform.lock.hcl, that just records an undifferentiated set of checksums and so can't explain where each of the new checksums came from.

---

I have a slightly shorter contrived reproduction case here which I'm sharing just in the hope that it's useful to others who might work on this issue. This is not a realistic user pattern, but allows quickly getting into the same situation this issue is about without a bunch of fiddly system configuration.

First, create a Terraform configuration which refers to the null provider. For example:

provider "null" {
}

In the same directory, create a .terraform.lock.hcl file with the following content:

# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.

provider "registry.terraform.io/hashicorp/null" {
  version = "3.1.0"
  hashes = [
    "h1:invalid",
  ]
}

The h1:invalid here is standing in for the hash of the package for whatever platform the user initially ran terraform init on, which therefore doesn't cover any packages for other platforms.

In that directory, run terraform providers lock with any explicit platform selections you like:

$ terraform providers lock -platform=linux_amd64
- Fetching hashicorp/null 3.1.0 for linux_amd64...
╷
│ Error: Could not retrieve providers for locking
│ 
│ Terraform failed to fetch the requested providers for linux_amd64
│ in order to calculate their checksums: some providers could not be
│ installed:
│ - registry.terraform.io/hashicorp/null: the current package for
│ registry.terraform.io/hashicorp/null 3.1.0 doesn't match any of the
│ checksums previously recorded in the dependency lock file.
╵

If we were to implement the TOFU exception I described above, I'd expect terraform providers lock to instead just add the new checksums to the existing block in the lock file, preserving the existing ones:

--- .terraform.lock.hcl
+++ .terraform.lock.hcl
@@ -5,5 +5,6 @@
   version = "3.1.0"
   hashes = [
     "h1:invalid",
+    "h1:vpC6bgUQoJ0znqIKVFevOdq+YQw42bRq0u+H3nto8nA=",
   ]
 }

Of course, exactly what output the command would produce in the terminal in this case remains to be decided. I trust that if someone picks up this issue to work on they will try some different permutations and see what level of "chattiness" feels most appropriate and most consistent, and record that rationale as a comment in the implementation so that we can refer to it again in future.

[jeff-french]

FWIW the way I encountered this issue was by creating a brand new Terraform configuration on my Mac darwin_amd64 that uses Terraform Cloud with remote runs (v1.1.0 using the new cloud provider stanza). Running terraform init generated the lockfile with darwin_amd64 h1 hashes. Then running terraform plan or committing the code and attempting a plan from the Terraform Cloud UI I would get these same error messages as the Terraform Cloud runners use the linux_amd64 arch. My first workaround was to simply delete the lockfile. After finding this issue, I was able to generate a completely new lockfile using terraform providers lock -platform=darwin_amd64 -platform=darwin_arm64 -platform=linux_amd64 -platform=linux_arm64 which worked with Terraform Cloud.

When this issue gets worked on it might be a good idea to automatically include the linux_amd64 hashes if the configuration is using Terraform Cloud.

[danielcompton]

The docs for Dependency Lock File say:

If you install a provider from an origin registry which provides checksums that are signed with a cryptographic signature, Terraform will treat all of the signed checksums as valid as long as one checksum matches. The lock file will therefore include checksums for both the package you installed for your current platform and any other packages that might be available for other platforms.

I might be misunderstanding the docs, but it seems like this is only partly happening? All zh checksums are downloaded, but it seems like h1 checksums are only downloaded when you run terraform providers lock specifying other platforms? The more nuanced reading makes sense after reading https://www.terraform.io/docs/language/dependency-lock.html#new-provider-package-checksums, but it wasn't what I'd expected.

Would it be possible for all the h1 checksums to also be downloaded when installing a package from the registry?

[apparentlymart]

Hi @danielcompton,

The current provider registry protocol only returns checksums compatible with the zh: scheme, and so until there's a new version of the registry protocol which can return h1: checksums (which is a change outside the direct scope of Terraform CLI) there is no way to calculate those except to download the packages and calculate them client-side, which is what terraform providers lock does.

The automatic "upgrade" from an existing zh: to its corresponding h1: as described in that documentation section is the compromise to allow this to work with the current registry protocol in the common case where folks are installing directly from provider registries. Things get more complex when the original installation is from somewhere other than an origin registry, because in that case Terraform doesn't even have access to the zh: checksums (they come from the origin registry, signed by the author), and so it can only record h1: checksums for exactly the packages it has access to.

[bflad]

I ran into an almost identical situation as @jeff-french above in #29958 (comment), running darwin_amd64 locally while attempting to use Terraform Cloud which is based on linux_amd64. I'm using a shared provider cache because I'm constantly working with a lot of various Terraform configurations and really do not want to re-download providers constantly.

$ cat .terraformrc
plugin_cache_dir = "/Users/bflad/.terraform.d/plugin-cache"

$ tree /Users/bflad/.terraform.d/plugin-cache/registry.terraform.io/hashicorp/random/3.1.0/
/Users/bflad/.terraform.d/plugin-cache/registry.terraform.io/hashicorp/random/3.1.0/
└── darwin_amd64
    └── terraform-provider-random_v3.1.0_x5

1 directory, 1 file

$ terraform init

Initializing Terraform Cloud...

Initializing provider plugins...
- Finding hashicorp/random versions matching "3.1.0"...
- Using hashicorp/random v3.1.0 from the shared cache directory

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform Cloud has been successfully initialized!

You may now begin working with Terraform Cloud. Try running "terraform plan" to
see any changes that are required for your infrastructure.

If you ever set or change modules or Terraform Settings, run "terraform init"
again to reinitialize your working directory.

$ cat .terraform.lock.hcl 
# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.

provider "registry.terraform.io/hashicorp/random" {
  version     = "3.1.0"
  constraints = "3.1.0"
  hashes = [
    "h1:rKYu5ZUbXwrLG1w81k7H3nce/Ys6yAxXhWcbtk36HjY=",
  ]
}

$ terraform apply
Running apply in Terraform Cloud. Output will stream here. Pressing Ctrl-C
will cancel the remote apply if it's still pending. If the apply started it
will stop streaming the logs, but will not stop the apply running remotely.

Preparing the remote apply...

To view this run in a browser, visit:
https://app.terraform.io/app/bflad-hashicorp/onedotonedemo-moved/runs/run-xGmjEfin46ajHgQr

Waiting for the plan to start...

Terraform v1.1.0
on linux_amd64
Configuring remote state backend...
Initializing Terraform configuration...

Setup failed: Failed terraform init (exit 1): <nil>

Output:

Initializing Terraform Cloud...

Initializing provider plugins...
- Reusing previous version of hashicorp/random from the dependency lock file
- Installing hashicorp/random v3.1.0...
╷
│ Error: Failed to install provider
│ 
│ Error while installing hashicorp/random v3.1.0: the current package for
│ registry.terraform.io/hashicorp/random 3.1.0 doesn't match any of the
│ checksums previously recorded in the dependency lock file
╵

The terraform init command didn't provide any warning that I would not be able to use Terraform Cloud immediately. When receiving the terraform apply error, there was not an indication where this lock file was located (only the local init message mentions .terraform.lock.hcl) or to use the terraform providers lock command to potentially fix the issue. There was also no specific recommendation how to fix this for the platform it was running on (Terraform Cloud's linux_amd64).

Maybe the checksum mismatch error could include some additional diagnostic information, such as:

Error while installing hashicorp/random v3.1.0: the current package for
registry.terraform.io/hashicorp/random 3.1.0 doesn't match any of the
checksums previously recorded in the dependency lock file (.terraform.lock.hcl)

Use the `terraform providers lock` command to include all necessary
platforms where Terraform will be executed in the dependency lock file.
Ensure the current platform, linux_amd64, is included.

Deleting .terraform.lock.hcl and running terraform providers lock -platform=darwin_amd64 -platform=linux_amd64 fixed my situation, given the current behaviors.