`init` fails for development overrides

[tombuildsstuff]

Terraform Version

$ terraform version
Terraform v0.14.4

Description

With the following ~/.terraformrc:

provider_installation {
   dev_overrides {
     "hashicorp/azurerm" = "/Users/tharvey/code/bin"
   }

   # For all other providers, install them directly from their origin provider
   # registries as normal. If you omit this, Terraform will _only_ use
   # the dev_overrides block, and so no other providers will be available.
   direct {}
 }

And a new directory containing the following main.tf:

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
    }
  }
}

provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "test" {
  name = "tom-dev"
  location = "west europe"
}

Then terraform init should pick up the local override, however it instead pulls this from the registry:

$ ls -la
total 8
drwxr-xr-x  3 tharvey  staff   96 11 Jan 09:30 .
drwxr-xr-x  8 tharvey  staff  256 11 Jan 09:17 ..
-rw-r--r--  1 tharvey  staff  229 11 Jan 09:30 main.tf

$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/azurerm...
- Installing hashicorp/azurerm v2.42.0...
- Installed hashicorp/azurerm v2.42.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.


Warning: Provider development overrides are in effect

The following provider development overrides are set in the CLI configuration:
 - hashicorp/azurerm in /Users/tharvey/code/bin

The behavior may therefore not match any released version of the provider and
applying changes may cause the state to become incompatible with published
releases.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary

However this actually installs the latest version from the registry, currently v2.42.0:

$ tree -la
.
├── .terraform
│   └── providers
│       └── registry.terraform.io
│           └── hashicorp
│               └── azurerm
│                   └── 2.42.0
│                       └── darwin_amd64
│                           └── terraform-provider-azurerm_v2.42.0_x5
├── .terraform.lock.hcl
└── main.tf

7 directories, 3 files

$ cat .terraform.lock.hcl
# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.

provider "registry.terraform.io/hashicorp/azurerm" {
  version = "2.42.0"
  hashes = [
    "h1:oxyi21Gz35Cup/JPqPaGtxunuqMuJHHYMBzBEieMhbw=",
    "zh:5b6d6b9d53efb09d1bff83fff019107ffaf2a79a68c9f59b9541098503c9d7a8",
    "zh:87886113155f25a9875e5b0812568953bf6a523b2c604ae0860054b464d1e2fe",
    "zh:9022504ad36b9ec3132cf8a74ae150583a3798f4aea39efd323db52fe180cdbb",
    "zh:98b06bababc7dfc888e8723f563f3da8bc87ee9e0e2a6fe681a157b54e1eeb01",
    "zh:b58b6e959ce7f5d37545e9abe2375adab474845e37d6b192dd9f94ca213bc461",
    "zh:c36fd2e03bf83b6e36a59712d1e57234194455ff761ea01e62af959d08fb51b4",
    "zh:c3ac50af9682c9719f3953ade2bd545cdc02e3d23741b805dc4b891916df0d73",
    "zh:c84e0c84ae17a8d2322a6a4d3a9e48077247683591b113cf1dd14cef7e64e1d8",
    "zh:f328085cb6f6ec5b6b38476b219f9caca8c02c86b0013e5377cd100a8e9169e5",
    "zh:f33cd91ad1a173ded18fe2ccf5648d4f40bcfab893c1b5b80df7a3123f7b8297",
  ]
}

Removing the .terraform and .terraform.lock.hcl files, it's possible to run terraform plan|apply|destroy just fine, using the local/development provider however:

$ terraform plan

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
 + create

Terraform will perform the following actions:

 # azurerm_resource_group.test will be created
 + resource "azurerm_resource_group" "test" {
     + id       = (known after apply)
     + location = "westeurope"
     + name     = "tom-dev"
   }

Plan: 1 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

$ terraform apply

Warning: Provider development overrides are in effect

The following provider development overrides are set in the CLI configuration:
- hashicorp/azurerm in /Users/tharvey/code/bin

The behavior may therefore not match any released version of the provider and
applying changes may cause the state to become incompatible with published
releases.


An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
 + create

Terraform will perform the following actions:

 # azurerm_resource_group.test will be created
 + resource "azurerm_resource_group" "test" {
     + id       = (known after apply)
     + location = "westeurope"
     + name     = "tom-dev"
   }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
 Terraform will perform the actions described above.
 Only 'yes' will be accepted to approve.

 Enter a value: yes

azurerm_resource_group.test: Creating...
azurerm_resource_group.test: Creation complete after 2s [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/tom-dev]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

$ terraform destroy

Warning: Provider development overrides are in effect

The following provider development overrides are set in the CLI configuration:
 - hashicorp/azurerm in /Users/tharvey/code/bin

The behavior may therefore not match any released version of the provider and
applying changes may cause the state to become incompatible with published
releases.


An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # azurerm_resource_group.test will be destroyed
  - resource "azurerm_resource_group" "test" {
      - id       = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/tom-dev" -> null
      - location = "westeurope" -> null
      - name     = "tom-dev" -> null
    }

Plan: 0 to add, 0 to change, 1 to destroy.

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

azurerm_resource_group.test: Destroying... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/tom-dev]
azurerm_resource_group.test: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/tom-dev, 10s elapsed]
azurerm_resource_group.test: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/tom-dev, 20s elapsed]
azurerm_resource_group.test: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/tom-dev, 30s elapsed]
azurerm_resource_group.test: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/tom-dev, 40s elapsed]
azurerm_resource_group.test: Destruction complete after 47s

Destroy complete! Resources: 1 destroyed.

---

It's worth noting that if using a provider not available in the registry, such as an alias in this case to intentionally ignore it, terraform init fails instead. For example, updating the ~/.terraformrc to:

provider_installation {
   dev_overrides {
     "tombuildsstuff/azurerm" = "/Users/tharvey/code/bin"
   }

   # For all other providers, install them directly from their origin provider
   # registries as normal. If you omit this, Terraform will _only_ use
   # the dev_overrides block, and so no other providers will be available.
   direct {}
 }

and using the following main.tf:

terraform {
  required_providers {
    azurerm = {
      source  = "tombuildsstuff/azurerm"
    }
  }
}

provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "test" {
  name = "tom-dev"
  location = "west europe"
}

Returns a failure when using terraform init:

$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding latest version of tombuildsstuff/azurerm...

Warning: Provider development overrides are in effect

The following provider development overrides are set in the CLI configuration:
 - tombuildsstuff/azurerm in /Users/tharvey/code/bin

The behavior may therefore not match any released version of the provider and
applying changes may cause the state to become incompatible with published
releases.


Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider
tombuildsstuff/azurerm: provider registry registry.terraform.io does not have
a provider named registry.terraform.io/tombuildsstuff/azurerm

If you have just upgraded directly from Terraform v0.12 to Terraform v0.14
then please upgrade to Terraform v0.13 first and follow the upgrade guide for
that release, which might help you address this problem.

But again a terraform plan (etc) work fine:

$ terraform plan

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
 + create

Terraform will perform the following actions:

 # azurerm_resource_group.test will be created
 + resource "azurerm_resource_group" "test" {
     + id       = (known after apply)
     + location = "westeurope"
     + name     = "tom-dev"
   }

Plan: 1 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

As such it appears this failure is specific to init.

References

• in-house provider not working #27358
• "missing provider" when attempting to override default #27409 - although this appears to be the inverse

[apparentlymart]

As noted in the documentation for dev_overrides, these settings override how Terraform runs providers, and not how Terraform installs providers. A development override bypasses all of Terraform's usual machinery for checking whether a suitable version is installed, whether the checksums match, etc, but as a result a development override can't participate in the installation process: it doesn't provide any of the necessary metadata to satisfy version constraints, populate the lock file, etc.

However, the fact that the dev_overrides feature completely bypasses the usual metadata checks is also what avoids this behavior being a significant problem in most cases: you can skip running terraform init at all, and just skip straight to running terraform plan or terraform apply. In that case Terraform would normally complain that the local cache of the provider doesn't exist or doesn't match the recorded metadata, but the dev_override disables all of that and just instructs Terraform to run the specified binary unconditionally, in a very similar way to how it deals with the provider acceptance test framework forcing it to use an already-running copy of the provider plugin (which also doesn't rely on or affect the behavior of terraform init, for the same reasons).

This feature was just a bonus stretch goal at the end of another project, so I think it's unlikely we'll be able to spend much more time on it in the near future, but nonetheless I'm going to relabel this as an enhancement with the intent that it represent a more complete design for development overrides, without this confusing oddity. The assumption that we'd do such a thing at some future point is what prompted the inclusion of the final paragraph of the docs:

This development overrides mechanism is intended as a pragmatic way to enable smoother provider development. The details of how it behaves, how to configure it, and how it interacts with the dependency lock file may all evolve in future Terraform releases, including possible breaking changes. We therefore recommend using development overrides only temporarily during provider development work.

[bflad]

Other provider developers including myself have run into similarly confusing situations with the development overrides and terraform init. Would you accept adding a warning to the top of the output noting that the configuration is not compatible/necessary for the command?

[bflad]

Actually it seems like the logic is already present and working for me with Terraform 0.14.4:

$ terraform init

Initializing the backend...

Initializing provider plugins...
- Reusing previous version of hashicorp/random from the dependency lock file
- Installing hashicorp/random v3.0.1...
- Installed hashicorp/random v3.0.1 (signed by HashiCorp)


Warning: Provider development overrides are in effect

The following provider development overrides are set in the CLI configuration:
 - hashicorp/aws in /Users/bflad/go/bin

The behavior may therefore not match any released version of the provider and
applying changes may cause the state to become incompatible with published
releases.

Terraform has been successfully initialized!

Maybe moving that above everything, similar to #27507 can make the behavior more obvious. 👍