"terraform providers lock" does not lock providers required by tests

[zanecodes]

Terraform Version

Terraform v1.13.3
on darwin_arm64

Terraform Configuration Files

versions.tf

terraform {
  required_providers {
    dns = {
      source  = "hashicorp/dns"
      version = "~> 3.4"
    }
  }
}

tests/versions.tf

terraform {
  required_providers {
    vault = {
      source  = "hashicorp/vault"
      version = "~> 5.3"
    }
  }
}

tests/test.tftest.hcl

run "test" {
  module {
    source = "./tests"
  }
}

Debug Output

N/A

Expected Behavior

Checksums for both hashicorp/dns (required by the root Terraform module) and hashicorp/vault (required by the tests Terraform module, which itself is required by tests/test.tftest.hcl) should be recorded in .terraform.lock.hcl, so that terraform init -lockfile=readonly can install the locked provider versions required by both terraform plan and terraform test commands

Actual Behavior

Only the checksum for hashicorp/dns is recorded in .terraform.lock.hcl, causing terraform init -lockfile=readonly to fail when it tries to add hashicorp/vault to .terraform.lock.hcl, since terraform init respects the providers required by modules in tests/test.tftest.hcl, but terraform providers lock does not

Steps to Reproduce

1. terraform providers lock
2. terraform init -lockfile=readonly

Additional Context

No response

References

No response

Generative AI / LLM assisted development?

No response

[crw]

Thanks for this report!

[zanecodes]

This may be out of scope for this specific issue, but the UX around provider locking is a bit confusing in general. My desired workflow would be something like:

• Run terraform init locally after setting up a new project, to generate a .terraform.lock.hcl
• Run terraform init -lockfile=readonly in CI/CD pipelines, which should use the exact same provider versions that I used in local testing, and the resulting .terraform directory should be cacheable across CI/CD steps within a pipeline

In practice, if I run terraform init locally, only the hashes for my local platform are added to .terraform.lock.hcl. I develop on darwin_arm64, and our CI/CD runs on linux_amd64, so when the CI/CD pipeline runs terraform init -lockfile=readonly, it installs the provider versions recorded in .terraform.lock.hcl, but also tries to add the hashes for the linux_amd64 packages, which emits a warning due to -lockfile=readonly. Terraform then re-installs the providers in every subsequent CI/CD step, since the hashes for linux_amd64 are not present in .terraform.lock.hcl, defeating our caching.

My vague understanding (which could very well be wrong) is that, due to how the Terraform registry works and how providers are packaged, Terraform has to download a provider in order to compute its hash, which is why it re-installs the providers in every CI/CD step, since Terraform won't use the provider it finds in .terraform if there are no matching hashes in the lock file, even if the version number matches, and it has to re-download the provider version in the lock file so it can hash it. Is my use case what the TF_PLUGIN_CACHE_MAY_BREAK_DEPENDENCY_LOCK_FILE setting is intended to facilitate? Or should I be setting TF_PLUGIN_CACHE_DIR rather than relying on the default .terraform directory (is the global plugin cache treated differently than a local .terraform directory?)

I tried to solve this by running terraform providers lock -platform=linux_amd64 -platform=darwin_arm64 and committing the resulting lock file, but due to this issue, the providers required by terraform test are not included in the lock file, so terraform init -lockfile=readonly fails in CI/CD, since it tries to add the providers needed by terraform test to the lock file. As a user, it's not clear to me why terraform init only adds the hashes for the current platform to the lock file, rather than all of the hashes for every supported platform, which would eliminate the need for the terraform providers lock command and solve my issue without any special handling to support CI/CD pipelines.

[liamcervante]

Hi @zanecodes, thanks for this! It definitely is an oversight that terraform providers lock isn't including providers needed by test when terraform init does include them by default. We'll look into fixing that.

A longer explanation for the rest of your questions can be found here. The short explanation is that Terraform includes all the zh hashes for every platform in the lock file, but can only compute the more complete h1 hashes locally. So, by default Terraform doesn't download all the different builds of the providers to compute the h1 hashes but only the ones needed by the local system and those specifically requested by providers lock. Despite that it does still include all the zh hashes, and still verifies them when a the lock file is introduced to a new system.

Hopefully the warning you're seeing is this one:

╷
│ Warning: Provider lock file not updated
│ 
│ Changes to the provider selections were detected, but not saved in the .terraform.lock.hcl file. To record these selections, run "terraform init" without the "-lockfile=readonly" flag.
╵

What this warning is likely saying is that a new h1 hash has been computed for the provider downloaded on the linux system, and this would normally be added to the lockfile but can't be because of the readonly command. Given the full context, I think you can safely ignore this warning in your CI/CD pipeline - the zh hashes will still have been checked and validated so Terraform is sure the downloaded provider is the correct one.

But, that doesn't change the fact we should support the test providers in the lock command which would also allow you to make the warning go away.

[apparentlymart]

The bigger problem with that -lockfile=readonly situation happens as soon as you run some other command (e.g. terraform test) after terraform init -lockfile=readonly succeeds.

The installation will succeed with that warning because the installer can use the zh: checksums to verify the previously-unseen package. But every other command relies on h1: checksums to verify the extracted packages in .terraform/providers, so those commands can't work unless there's a h1: package for the current platform, and -lockfile=readonly prevents terraform init from adding the extra h1: checksum that those later commands would rely on.

So -lockfile=readonly is only useful today if you've used terraform providers lock to make sure you have h1: checksums for at least the platform where you'll be using -lockfile=readonly. And because of this bug, even that won't work when there's a testing-only dependency.

Removing -lockfile=readonly from the preparation to run the tests is the only way to make this work today, I think. If this bug were fixed then the terraform providers lock workaround would work, at least.