backport of commit 5c0310c

Author: dbanck

.go-version

@@ -1 +1 @@
-1.19.4
+1.19.6

CHANGELOG.md

@@ -1,14 +1,64 @@
-## 1.5.0 (Unreleased)
+## 1.4.0 (Unreleased)
+
+UPGRADE NOTES:
+
+- config: The `textencodebase64` function when called with encoding "GB18030" will now encode the euro symbol € as the two-byte sequence `0xA2,0xE3`, as required by the GB18030 standard, before applying base64 encoding.
+- config: The `textencodebase64` function when called with encoding "GBK" or "CP936" will now encode the euro symbol € as the single byte `0x80` before applying base64 encoding. This matches the behavior of the Windows API when encoding to this Windows-specific character encoding.
+- `terraform init`: When interpreting the hostname portion of a provider source address or the address of a module in a module registry, Terraform will now use _non-transitional_ IDNA2008 mapping rules instead of the transitional mapping rules previously used.
+
+    This matches a change to [the WHATWG URL spec's rules for interpreting non-ASCII domain names](https://url.spec.whatwg.org/#concept-domain-to-ascii) which is being gradually adopted by web browsers. Terraform aims to follow the interpretation of hostnames used by web browsers for consistency. For some hostnames containing non-ASCII characters this may cause Terraform to now request a different "punycode" hostname when resolving.
+- The Terraform plan renderer has been completely rewritten to aid with future Terraform Cloud integration. Users should not see any material change in the plan output between 1.3 and 1.4. Users are encouraged to file reports if they notice material differences, or encounter any bugs or panics during their normal execution of Terraform.
+
+    The diff computation and the rendering are now split into separate packages, while previously the rendering was handled as the diff was computed. Going forward, making small changes to the format of the plan will be easier and introducing new types of renderer will be simplified.
+
+BUG FIXES:
+
+* The module installer will now record in its manifest a correct module source URL after normalization when the URL given as input contains both a query string portion and a subdirectory portion. Terraform itself doesn't currently make use of this information and so this is just a cosmetic fix to make the recorded metadata more correct. ([#31636](https://github.com/hashicorp/terraform/issues/31636))
+* config: The `yamldecode` function now correctly handles entirely-nil YAML documents. Previously it would incorrectly return an unknown value instead of a null value. It will now return a null value as documented. ([#32151](https://github.com/hashicorp/terraform/issues/32151))
+* Ensure correct ordering between data sources and the deletion of managed resource dependencies. ([#32209](https://github.com/hashicorp/terraform/issues/32209))
+* Fix Terraform creating objects that should not exist in variables that specify default attributes in optional objects. ([#32178](https://github.com/hashicorp/terraform/issues/32178))
+* Fix several Terraform crashes that are caused by HCL creating objects that should not exist in variables that specify default attributes in optional objects within collections. ([#32178](https://github.com/hashicorp/terraform/issues/32178))
+* Fix inconsistent behaviour in empty vs null collections. ([#32178](https://github.com/hashicorp/terraform/issues/32178))
+* `terraform workspace` now returns a non-zero exit when given an invalid argument ([#31318](https://github.com/hashicorp/terraform/issues/31318))
+* Terraform would always plan changes when using a nested set attribute ([#32536](https://github.com/hashicorp/terraform/issues/32536))
+* Terraform can now better detect when complex optional+computed object attributes are removed from configuration ([#32551](https://github.com/hashicorp/terraform/issues/32551))
+* A new methodology for planning set elements can now better detect optional+computed changes within sets ([#32563](https://github.com/hashicorp/terraform/issues/32563))
+* Fix state locking and releasing messages when in `-json` mode, messages will now be written in JSON format ([#32451](https://github.com/hashicorp/terraform/issues/32451))
+
 
 ENHANCEMENTS:
-* Terraform CLI's local operations mode will now attempt to persist state snapshots to the state storage backend periodically during the apply step, thereby reducing the window for lost data if the Terraform process is aborted unexpectedly. [GH-32680]
-* If Terraform CLI recieves SIGINT (or its equivalent on non-Unix platforms) during the apply step then it will immediately try to persist the latest state snapshot to the state storage backend, with the assumption that a graceful shutdown request often typically followed by a hard abort some time later if the graceful shutdown doesn't complete fast enough. [GH-32680]
+
+* `terraform plan` can now store a plan file even when encountering errors, which can later be inspected to help identify the source of the failures ([#32395](https://github.com/hashicorp/terraform/issues/32395))
+* `terraform_data` is a new builtin managed resource type, which can replace the use of `null_resource`, and can store data of any type ([#31757](https://github.com/hashicorp/terraform/issues/31757))
+* `terraform init` will now ignore entries in the optional global provider cache directory unless they match a checksum already tracked in the current configuration's dependency lock file. This therefore avoids the long-standing problem that when installing a new provider for the first time from the cache we can't determine the full set of checksums to include in the lock file. Once the lock file has been updated to include a checksum covering the item in the global cache, Terraform will then use the cache entry for subsequent installation of the same provider package. There is an interim CLI configuration opt-out for those who rely on the previous incorrect behavior. ([#32129](https://github.com/hashicorp/terraform/issues/32129))
+* Interactive input for sensitive variables is now masked in the UI ([#29520](https://github.com/hashicorp/terraform/issues/29520))
+* A new `-or-create` flag was added to `terraform workspace select`, to aid in creating workspaces in automated situations ([#31633](https://github.com/hashicorp/terraform/issues/31633))
+* A new command was added for exporting Terraform function signatures in machine-readable format: `terraform metadata functions -json` ([#32487](https://github.com/hashicorp/terraform/issues/32487))
+* The "Failed to install provider" error message now includes the reason a provider could not be installed. ([#31898](https://github.com/hashicorp/terraform/issues/31898))
+* backend/gcs: Add `kms_encryption_key` argument, to allow encryption of state files using Cloud KMS keys. ([#24967](https://github.com/hashicorp/terraform/issues/24967))
+* backend/gcs: Add `storage_custom_endpoint` argument, to allow communication with the backend via a Private Service Connect endpoint. ([#28856](https://github.com/hashicorp/terraform/issues/28856))
+* backend/gcs: Update documentation for usage of `gcs` with `terraform_remote_state` ([#32065](https://github.com/hashicorp/terraform/issues/32065))
+* backend/gcs: Update storage package to v1.28.0 ([#29656](https://github.com/hashicorp/terraform/issues/29656))
+* When removing a workspace from the `cloud` backend `terraform workspace delete` will use Terraform Cloud's [Safe Delete](https://developer.hashicorp.com/terraform/cloud-docs/api-docs/workspaces#safe-delete-a-workspace) API if the `-force` flag is not provided. ([#31949](https://github.com/hashicorp/terraform/pull/31949))
+* backend/oss: More robustly handle endpoint retrieval error ([#32295](https://github.com/hashicorp/terraform/issues/32295))
+* local-exec provisioner: Added `quiet` argument. If `quiet` is set to `true`, Terraform will not print the entire command to stdout during plan. ([#32116](https://github.com/hashicorp/terraform/issues/32116))
+* backend/http: Add support for mTLS authentication. ([#31699](https://github.com/hashicorp/terraform/issues/31699))
+* cloud: Add support for using the [generic hostname](https://developer.hashicorp.com/terraform/cloud-docs/registry/using#generic-hostname-terraform-enterprise) localterraform.com in module and provider sources as a substitute for the currently configured cloud backend hostname. This enhancement was also applied to the remote backend.
+* `terraform show` will now print an explanation when called on a Terraform workspace with empty state detailing why no resources are shown. ([#32629](https://github.com/hashicorp/terraform/issues/32629))
+* backend/gcs: Added support for `GOOGLE_BACKEND_IMPERSONATE_SERVICE_ACCOUNT` env var to allow impersonating a different service account when `GOOGLE_IMPERSONATE_SERVICE_ACCOUNT` is configured for the GCP provider. ([#32557](https://github.com/hashicorp/terraform/issues/32557))
+* backend/cos: Add support for the `assume_role` authentication method with the `tencentcloud` provider. This can be configured via the Terraform config or environment variables.
+* backend/cos: Add support for the `security_token` authentication method with the `tencentcloud` provider. This can be configured via the Terraform config or environment variables.
+
+EXPERIMENTS:
+
+* Since its introduction the `yamlencode` function's documentation carried a warning that it was experimental. This predated our more formalized idea of language experiments and so wasn't guarded by an explicit opt-in, but the intention was to allow for small adjustments to its behavior if we learned it was producing invalid YAML in some cases, due to the relative complexity of the YAML specification.
+
+    From Terraform v1.4 onwards, `yamlencode` is no longer documented as experimental and is now subject to the Terraform v1.x Compatibility Promises. There are no changes to its previous behavior in v1.3 and so no special action is required when upgrading.
 
 ## Previous Releases
 
 For information on prior major and minor releases, see their changelogs:
 
-* [v1.4](https://github.com/hashicorp/terraform/blob/v1.4/CHANGELOG.md)
 * [v1.3](https://github.com/hashicorp/terraform/blob/v1.3/CHANGELOG.md)
 * [v1.2](https://github.com/hashicorp/terraform/blob/v1.2/CHANGELOG.md)
 * [v1.1](https://github.com/hashicorp/terraform/blob/v1.1/CHANGELOG.md)

docs/resource-instance-change-lifecycle.md

@@ -33,11 +33,9 @@ The various object values used in different parts of this process are:
   which a provider may use as a starting point for its planning operation.
 
     The built-in logic primarily deals with the expected behavior for attributes
-    marked in the schema as "computed". If an attribute is only "computed",
-    Terraform expects the value to only be chosen by the provider and it will
-    preserve any Prior State. If an attribute is marked as "computed" and
-    "optional", this means that the user may either set it or may leave it
-    unset to allow the provider to choose a value.
+    marked in the schema as both "optional" _and_ "computed", which means that
+    the user may either set it or may leave it unset to allow the provider
+    to choose a value instead.
 
     Terraform Core therefore constructs the proposed new state by taking the
     attribute value from Configuration if it is non-null, and then using the

internal/backend/local/backend_apply.go

@@ -5,7 +5,6 @@ import (
 	"errors"
 	"fmt"
 	"log"
-	"time"
 
 	"github.com/hashicorp/terraform/internal/backend"
 	"github.com/hashicorp/terraform/internal/command/views"
@@ -75,10 +74,6 @@ func (b *Local) opApply(
 		op.ReportResult(runningOp, diags)
 		return
 	}
-	// stateHook uses schemas for when it periodically persists state to the
-	// persistent storage backend.
-	stateHook.Schemas = schemas
-	stateHook.PersistInterval = 20 * time.Second // arbitrary interval that's hopefully a sweet spot
 
 	var plan *plans.Plan
 	// If we weren't given a plan, then we refresh/plan

internal/backend/local/hook_state.go

@@ -1,9 +1,7 @@
 package local
 
 import (
-	"log"
 	"sync"
-	"time"
 
 	"github.com/hashicorp/terraform/internal/states"
 	"github.com/hashicorp/terraform/internal/states/statemgr"
@@ -17,21 +15,6 @@ type StateHook struct {
 	sync.Mutex
 
 	StateMgr statemgr.Writer
-
-	// If PersistInterval is nonzero then for any new state update after
-	// the duration has elapsed we'll try to persist a state snapshot
-	// to the persistent backend too.
-	// That's only possible if field Schemas is valid, because the
-	// StateMgr.PersistState function for some backends needs schemas.
-	PersistInterval time.Duration
-
-	// Schemas are the schemas to use when persisting state due to
-	// PersistInterval. This is ignored if PersistInterval is zero,
-	// and PersistInterval is ignored if this is nil.
-	Schemas *terraform.Schemas
-
-	lastPersist  time.Time
-	forcePersist bool
 }
 
 var _ terraform.Hook = (*StateHook)(nil)
@@ -40,56 +23,11 @@ func (h *StateHook) PostStateUpdate(new *states.State) (terraform.HookAction, er
 	h.Lock()
 	defer h.Unlock()
 
-	if h.lastPersist.IsZero() {
-		// The first PostStateUpdate starts the clock for intermediate
-		// calls to PersistState.
-		h.lastPersist = time.Now()
-	}
-
 	if h.StateMgr != nil {
 		if err := h.StateMgr.WriteState(new); err != nil {
 			return terraform.HookActionHalt, err
 		}
-		if mgrPersist, ok := h.StateMgr.(statemgr.Persister); ok && h.PersistInterval != 0 && h.Schemas != nil {
-			if h.forcePersist || time.Since(h.lastPersist) >= h.PersistInterval {
-				err := mgrPersist.PersistState(h.Schemas)
-				if err != nil {
-					return terraform.HookActionHalt, err
-				}
-				h.lastPersist = time.Now()
-			}
-		}
 	}
 
 	return terraform.HookActionContinue, nil
 }
-
-func (h *StateHook) Stopping() {
-	h.Lock()
-	defer h.Unlock()
-
-	// If Terraform has been asked to stop then that might mean that a hard
-	// kill signal will follow shortly in case Terraform doesn't stop
-	// quickly enough, and so we'll try to persist the latest state
-	// snapshot in the hope that it'll give the user less recovery work to
-	// do if they _do_ subsequently hard-kill Terraform during an apply.
-
-	if mgrPersist, ok := h.StateMgr.(statemgr.Persister); ok && h.Schemas != nil {
-		err := mgrPersist.PersistState(h.Schemas)
-		if err != nil {
-			// This hook can't affect Terraform Core's ongoing behavior,
-			// but it's a best effort thing anyway so we'll just emit a
-			// log to aid with debugging.
-			log.Printf("[ERROR] Failed to persist state after interruption: %s", err)
-		}
-
-		// While we're in the stopping phase we'll try to persist every
-		// new state update to maximize every opportunity we get to avoid
-		// losing track of objects that have been created or updated.
-		// Terraform Core won't start any new operations after it's been
-		// stopped, so at most we should see one more PostStateUpdate
-		// call per already-active request.
-		h.forcePersist = true
-	}
-
-}

internal/backend/local/hook_state_test.go

@@ -1,12 +1,8 @@
 package local
 
 import (
-	"fmt"
 	"testing"
-	"time"
 
-	"github.com/google/go-cmp/cmp"
-	"github.com/hashicorp/terraform/internal/states"
 	"github.com/hashicorp/terraform/internal/states/statemgr"
 	"github.com/hashicorp/terraform/internal/terraform"
 )
@@ -31,125 +27,3 @@ func TestStateHook(t *testing.T) {
 		t.Fatalf("bad state: %#v", is.State())
 	}
 }
-
-func TestStateHookStopping(t *testing.T) {
-	is := &testPersistentState{}
-	hook := &StateHook{
-		StateMgr:        is,
-		Schemas:         &terraform.Schemas{},
-		PersistInterval: 4 * time.Hour,
-		lastPersist:     time.Now(),
-	}
-
-	s := statemgr.TestFullInitialState()
-	action, err := hook.PostStateUpdate(s)
-	if err != nil {
-		t.Fatalf("unexpected error from PostStateUpdate: %s", err)
-	}
-	if got, want := action, terraform.HookActionContinue; got != want {
-		t.Fatalf("wrong hookaction %#v; want %#v", got, want)
-	}
-	if is.Written == nil || !is.Written.Equal(s) {
-		t.Fatalf("mismatching state written")
-	}
-	if is.Persisted != nil {
-		t.Fatalf("persisted too soon")
-	}
-
-	// We'll now force lastPersist to be long enough ago that persisting
-	// should be due on the next call.
-	hook.lastPersist = time.Now().Add(-5 * time.Hour)
-	hook.PostStateUpdate(s)
-	if is.Written == nil || !is.Written.Equal(s) {
-		t.Fatalf("mismatching state written")
-	}
-	if is.Persisted == nil || !is.Persisted.Equal(s) {
-		t.Fatalf("mismatching state persisted")
-	}
-	hook.PostStateUpdate(s)
-	if is.Written == nil || !is.Written.Equal(s) {
-		t.Fatalf("mismatching state written")
-	}
-	if is.Persisted == nil || !is.Persisted.Equal(s) {
-		t.Fatalf("mismatching state persisted")
-	}
-
-	gotLog := is.CallLog
-	wantLog := []string{
-		// Initial call before we reset lastPersist
-		"WriteState",
-
-		// Write and then persist after we reset lastPersist
-		"WriteState",
-		"PersistState",
-
-		// Final call when persisting wasn't due yet.
-		"WriteState",
-	}
-	if diff := cmp.Diff(wantLog, gotLog); diff != "" {
-		t.Fatalf("wrong call log so far\n%s", diff)
-	}
-
-	// We'll reset the log now before we try seeing what happens after
-	// we use "Stopped".
-	is.CallLog = is.CallLog[:0]
-	is.Persisted = nil
-
-	hook.Stopping()
-	if is.Persisted == nil || !is.Persisted.Equal(s) {
-		t.Fatalf("mismatching state persisted")
-	}
-
-	is.Persisted = nil
-	hook.PostStateUpdate(s)
-	if is.Persisted == nil || !is.Persisted.Equal(s) {
-		t.Fatalf("mismatching state persisted")
-	}
-	is.Persisted = nil
-	hook.PostStateUpdate(s)
-	if is.Persisted == nil || !is.Persisted.Equal(s) {
-		t.Fatalf("mismatching state persisted")
-	}
-
-	gotLog = is.CallLog
-	wantLog = []string{
-		// "Stopping" immediately persisted
-		"PersistState",
-
-		// PostStateUpdate then writes and persists on every call,
-		// on the assumption that we're now bailing out after
-		// being cancelled and trying to save as much state as we can.
-		"WriteState",
-		"PersistState",
-		"WriteState",
-		"PersistState",
-	}
-	if diff := cmp.Diff(wantLog, gotLog); diff != "" {
-		t.Fatalf("wrong call log once in stopping mode\n%s", diff)
-	}
-}
-
-type testPersistentState struct {
-	CallLog []string
-
-	Written   *states.State
-	Persisted *states.State
-}
-
-var _ statemgr.Writer = (*testPersistentState)(nil)
-var _ statemgr.Persister = (*testPersistentState)(nil)
-
-func (sm *testPersistentState) WriteState(state *states.State) error {
-	sm.CallLog = append(sm.CallLog, "WriteState")
-	sm.Written = state
-	return nil
-}
-
-func (sm *testPersistentState) PersistState(schemas *terraform.Schemas) error {
-	if schemas == nil {
-		return fmt.Errorf("no schemas")
-	}
-	sm.CallLog = append(sm.CallLog, "PersistState")
-	sm.Persisted = sm.Written
-	return nil
-}

internal/builtin/providers/terraform/provider.go

@@ -119,7 +119,11 @@ func (p *Provider) ApplyResourceChange(req providers.ApplyResourceChangeRequest)
 }
 
 // ImportResourceState requests that the given resource be imported.
-func (p *Provider) ImportResourceState(providers.ImportResourceStateRequest) providers.ImportResourceStateResponse {
+func (p *Provider) ImportResourceState(req providers.ImportResourceStateRequest) providers.ImportResourceStateResponse {
+	if req.TypeName == "terraform_data" {
+		return importDataStore(req)
+	}
+
 	panic("unimplemented - terraform_remote_state has no resources")
 }
 

internal/builtin/providers/terraform/resource_data.go

@@ -146,3 +146,24 @@ func applyDataStoreResourceChange(req providers.ApplyResourceChangeRequest) (res
 
 	return resp
 }
+
+// TODO: This isn't very useful even for examples, because terraform_data has
+// no way to refresh the full resource value from only the import ID. This
+// minimal implementation allows the import to succeed, and can be extended
+// once the configuration is available during import.
+func importDataStore(req providers.ImportResourceStateRequest) (resp providers.ImportResourceStateResponse) {
+	schema := dataStoreResourceSchema()
+	v := cty.ObjectVal(map[string]cty.Value{
+		"id": cty.StringVal(req.ID),
+	})
+	state, err := schema.Block.CoerceValue(v)
+	resp.Diagnostics = resp.Diagnostics.Append(err)
+
+	resp.ImportedResources = []providers.ImportedResource{
+		{
+			TypeName: req.TypeName,
+			State:    state,
+		},
+	}
+	return resp
+}

internal/terraform/context.go

@@ -193,12 +193,6 @@ func (c *Context) Stop() {
 		c.runContextCancel = nil
 	}
 
-	// Notify all of the hooks that we're stopping, in case they want to try
-	// to flush in-memory state to disk before a subsequent hard kill.
-	for _, hook := range c.hooks {
-		hook.Stopping()
-	}
-
 	// Grab the condition var before we exit
 	if cond := c.runCond; cond != nil {
 		log.Printf("[INFO] terraform: waiting for graceful stop to complete")