PSS: Add alternative, experimental version of init command that downloads providers in two stages

[SarahFrench]

Description

This PR introduces an alternative version of the logic controlling events during an init command. Instead of downloading providers in a single step, the new logic downloads providers in two steps:

• Providers described in the configuration.
• Providers described in the state.

The dependency locks identified after each provider download action are tracked. The locks identified from the config are used as an input for downloading providers present in the state, and this prevents providers being re-downloaded or another version of a provider being downloaded. The locks identified from the config and the state are later combined and written to the dependency lock file.

This PR also changes how we control the parsing of the state_store block to gate this feature as experimental. The old approach (implemented here) is not compatible with multiple tests running in parallel, so this commit changes the way the feature is gated.

Details

The original version of the init logic (internal/command/init_run.go) roughly does this:

• Loads the root module's configuration
• Uses that config to initialise the backend
• Reads the state using that backend
• Loads the complete configuration (incl. child modules)
• Makes a call to the getProviders method, with the full config and state data as inputs to that process.
  • This downloads providers and updates the dependency lock file.

In the new version (internal/command/init_run_experiment.go) there are these changes:

• Load the complete configuration
• Download providers described by the config using getProvidersFromConfig (step1)
• Initialise the backend (which may depend on a provider in the context of PSS)
• Read in the state
• Download any remaining needed providers using getProvidersFromState (step2)
• Update the dependency lock file using the combination of locks identified from step 1 and 2.

The original version of this logic is used by default. The new init logic is only used when Terraform has experiments enabled and either the -enable-pluggable-state-storage-experiment flag is passed to the init command or an environment variable called TF_ENABLE_PLUGGABLE_STATE_STORAGE has a non-empty value.

Target Release

N/A

Rollback Plan

• If a change needs to be reverted, we will roll out an update to the code within 7 days.

Changes to Security Controls

Are there any changes to security controls (access controls, encryption, logging) in this pull request? If so, explain.

CHANGELOG entry

• This change is user-facing and I added a changelog entry.
• This change is not user-facing.

[SarahFrench]

TestInit_stateStoreBlockIsExperimental is passing when run in isolation, but is failing when run in parallel to other tests. Looking into why that's the case...

Edit: it's due to the modification of the package level terraformBlockSchema variable. 🤦🏻 - this is fine for gating the feature for real users, but means test can interfere with each other.

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.