Improve user feedback about 'bad' values of `TF_CLI_CONFIG_FILE` environment variable

[SarahFrench]

Terraform Version

Terraform v1.3.5-dev
on darwin_arm64
+ provider registry.terraform.io/hashicorp/google v4.51.0

Terraform Configuration Files

Any config is relevant to this issue.

Here I've included the one I used to generate the debug output, but the config's contents are not specific to this issue.

More importantly, here's the CLI config file that I use in this issue:

provider_installation {

  dev_overrides {
    "hashicorp/google" = "/Users/sarahfrench/go/bin"
    "hashicorp/google-beta" = "/Users/sarahfrench/go/bin"
  }

  # For all other providers, install them directly from their origin provider
  # registries as normal. If you omit this, Terraform will _only_ use
  # the dev_overrides block, and so no other providers will be available.
  direct {}
}

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 4.0"
    }
  }
}

provider "google" {
  project = var.gcp_project_id
  region  = var.default_region
  zone    = var.default_zone
}

variable "gcp_project_id" {
  type = string
}
variable "default_region" {
  type = string
}
variable "default_zone" {
  type = string
}

resource "google_cloud_scheduler_job" "job" {
  name             = "tf-test-my-example-job"
  description      = "This is a test cloud scheduler job"
  schedule         = "*/8 * * * *"
  time_zone        = "America/New_York"
  attempt_deadline = "320s"
  retry_config {
    retry_count = 1
  }
  http_target {
    http_method = "POST"
    uri         = "https://example.com/ping"
    body        = base64encode("{\"foo\":\"bar\"}")
  }
}

Debug Output

In these examples I am trying to use a CLI configuration file to use dev_overrides, and I set the path to the file via TF_CLI_CONFIG_FILE.

--

Here's what happens when I run terraform plan and set TF_CLI_CONFIG_FILE to:

1. an absolute path to a file that exists
   • https://gist.github.com/SarahFrench/ec5c2960df30f7b7c98a8ee99a686b39
2. an absolute path to a file that DOESN'T exist
   • https://gist.github.com/SarahFrench/ef49697f904f68ad466d766d52f60550
3. a path to a file that exists, which starts with ~/
   • https://gist.github.com/SarahFrench/7e1fda557400cca155169ff169685ed8

In each of the 3 examples I've truncated the logs so they finish at this log : [INFO] backend/local: starting Plan operation. All logs related to the CLI config are still there.

Expected Behavior

I would expect Terraform to provide feedback to the user if it cannot use the path supplied via the TF_CLI_CONFIG_FILE environment variable - either due to it pointing at a non-existent file or unable to use for other reasons (e.g. starting with ~/).

For the 3 scenarios my debug outputs above show, I would expect:

1. Already shows expected behaviour - a warning about dev_overrides is shown at the top of the terminal output. This confirms that TF_CLI_CONFIG_FILE is doing what I expect.
2. I would expect a similar warning to show, except it tells me that a file cannot be found at the path supplied and that TF_CLI_CONFIG_FILE is not impacting the CLI configuration.
3. Ideally would be able to use file paths starting with ~/ and behave like (1). Alternatively, it could behave like my expectations for (2) by printing a warning.

Actual Behavior

1. A warning about dev_overrides is shown at the top of the terminal output. This confirms that TF_CLI_CONFIG_FILE is doing what I expect.
2. There is no warning about TF_CLI_CONFIG_FILE not contributing to CLI config, and the CLI proceeds without being configured
3. There is no warning about TF_CLI_CONFIG_FILE not contributing to CLI config, and the CLI proceeds without being configured

Steps to Reproduce

1. Create a small TF configuration using a provider (e.g. google)
2. Create a CLI configuration file that configures a developer override for that provider
3. Run terraform init, without TF_CLI_CONFIG_FILE set, to create lock file etc
4. Run TF_CLI_CONFIG_FILE=<path> terraform plan where the path value is an absolute path to the file made in step 2. You will see a warning about developer overrides in effect, and then the plan is displayed.
5. Run TF_CLI_CONFIG_FILE=<path> terraform plan where the path value is an absolute path to a file that doesn't exist. You will see no warnings about developer overrides, the command proceed with a provider downloaded from the Registry, and then the plan is displayed.
6. Run TF_CLI_CONFIG_FILE=<path> terraform plan where the path value is a path to the file made in step 2 that starts with "~/". You will see no warnings about developer overrides, the command proceed with a provider downloaded from the Registry, and then the plan is displayed.

Note: if (3) isn't done then steps (5) and (6) will instead have an inconsistent dependency lock file error.

Additional Context

No other ENVs are set that impact Terraform (e.g. TERRAFORM_CONFIG is not being used)

References

No response

[alisdair]

Thanks for reporting this! I can confirm the behaviour you're seeing here, and it is unnecessarily unfriendly.

I think with some restructuring of the CLI config loader we could reasonably emit a warning, along the lines you suggested. At present we don't support the CLI colourizer this early in the process, so it wouldn't look particularly pretty, but it'd be functional:

The code in question already supports returning diagnostics to the main program flow, so this would be a relatively small change to the internal/command/cliconfig package.

[SarahFrench]

Thanks for looking at this! Adding a warning like the one you posted would be a great help when onboarding newbies to provider development.

If I get time I'd be interested in having a go at a implementing the new warning(s), but equally happy for others to pick it up

[rkhaled0]

indeed, quite helpful to warn people if bad CLI config location.
I picked it up @SarahFrench.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.