lang/funcs: Allow some more expression types in templatestring

The templatestring function has some special constraints on its first
argument that are included to add some intentional friction for those who
are new to Terraform, want to do some simple template rendering, but have
only found the templatestring function so far.

We know from previous experience with the hashicorp/template provider that
this sort of functionality tends to attract those who haven't yet learned
that the Terraform language has built-in support for string templates
(without calling any function), who would then get confused by the need
for an extra level of escaping to render a template string only indirectly
through this function.

However, this rule is not intended to be onerous and require writing the
rest of the containing module in an unnatural way to work around it, so
here we loosen the rule to allow some additional forms:
 - An index expression whose collection operand meets these rules.
 - A relative traversal whose source operand meets these rules.

In particular this makes it possible to write an expression like:
    data.example.example[each.key].result
...which is a relative traversal from an index from a scope traversal,
and is a very reasonable thing to write if you've retrieved multiple
templates using a data resource that uses for_each.

This also treats splat expressions in the same way as index expressions
at the static check stage, but that's only to allow us to reach the
dynamic type check that will ultimately report that a string is required,
because the result of a splat expression is a tuple. The type-related
error message is (subjectively) more helpful/relevant than the
syntax-related one for this case.

Finally, this includes some revisions to the documentation for this
function to correct some editing errors from the first pass and to slightly
loosen the language about what's allowed. It's still a little vague about
what exactly is allowed, but I'm doubtful that a precise definition in
terms of HCL's expression types would be very enlightening for a typical
reader anyway. We can tweak the specificity of the language here if we
start to see repeated questions about what is and is not valid.

Author: apparentlymart

internal/lang/funcs/string.go

@@ -213,8 +213,6 @@ func MakeTemplateStringFunc(funcsCb func() (funcs map[string]function.Function,
 			// the function documentation to learn about the other options that
 			// are probably more suitable for what they need.
 			switch expr := templateClosure.Expression.(type) {
-			case *hclsyntax.ScopeTraversalExpr:
-				// A standalone traversal is always acceptable.
 			case *hclsyntax.TemplateWrapExpr:
 				// This situation occurs when someone writes an interpolation-only
 				// expression as was required in Terraform v0.11 and earlier.
@@ -249,16 +247,17 @@ func MakeTemplateStringFunc(funcsCb func() (funcs map[string]function.Function,
 					)
 				}
 			default:
-				// Nothing else is allowed.
-				// Someone who really does want to construct a template dynamically
-				// can factor out that construction into a local value and refer
-				// to it in the templatestring call, but it's not really feasible
-				// to explain that clearly in a short error message so we'll deal
-				// with that option on the function's documentation page instead,
-				// where we can show a full example.
-				return cty.UnknownVal(retType), function.NewArgErrorf(
-					0, "invalid template expression: must be a direct reference to a single string from elsewhere, containing valid Terraform template syntax",
-				)
+				if !isValidTemplateStringExpr(expr) {
+					// Someone who really does want to construct a template dynamically
+					// can factor out that construction into a local value and refer
+					// to it in the templatestring call, but it's not really feasible
+					// to explain that clearly in a short error message so we'll deal
+					// with that option on the function's documentation page instead,
+					// where we can show a full example.
+					return cty.UnknownVal(retType), function.NewArgErrorf(
+						0, "invalid template expression: must be a direct reference to a single string from elsewhere, containing valid Terraform template syntax",
+					)
+				}
 			}
 
 			templateVal, diags := templateClosure.Value()
@@ -385,6 +384,55 @@ func makeRenderTemplateFunc(funcsCb func() (funcs map[string]function.Function,
 	}
 }
 
+func isValidTemplateStringExpr(expr hcl.Expression) bool {
+	// Our goal with this heuristic is to be as permissive as possible with
+	// allowing things that authors might try to use as references to a
+	// template string defined elsewhere, while rejecting complex expressions
+	// that seem like they might be trying to construct templates dynamically
+	// or might have resulted from a misunderstanding that "templatestring" is
+	// the only way to render a template, because someone hasn't learned
+	// about template expressions yet.
+	//
+	// This is here only to give better feedback to folks who seem to be using
+	// templatestring for something other than what it's intended for, and not
+	// to block dynamic template generation altogether. Authors who have a
+	// genuine need for dynamic template generation can always assert that to
+	// Terraform by factoring out their dynamic generation into a local value
+	// and referring to it; this rule is just a little speedbump to prompt
+	// the author to consider whether there's a better way to solve their
+	// problem, as opposed to just using the first solution they found.
+	switch expr := expr.(type) {
+	case *hclsyntax.ScopeTraversalExpr:
+		// A simple static reference from the current scope is always valid.
+		return true
+
+	case *hclsyntax.RelativeTraversalExpr:
+		// Relative traversals are allowed as long as they begin from
+		// something that would otherwise be allowed.
+		return isValidTemplateStringExpr(expr.Source)
+
+	case *hclsyntax.IndexExpr:
+		// Index expressions are allowed as long as the collection is
+		// also specified using an expression that conforms to these rules.
+		// The key operand is intentionally unconstrained because that
+		// is a rule for how to select an element, and so doesn't represent
+		// a source from which the template string is being retrieved.
+		return isValidTemplateStringExpr(expr.Collection)
+
+	case *hclsyntax.SplatExpr:
+		// Splat expressions would be weird to use because they'd typically
+		// return a tuple and that wouldn't be valid as a template string,
+		// but we allow it here (as long as the operand would otherwise have
+		// been allowed) because then we'll let the type mismatch error
+		// show through, and that's likely a more helpful error message.
+		return isValidTemplateStringExpr(expr.Source)
+
+	default:
+		// Nothing else is allowed.
+		return false
+	}
+}
+
 // Replace searches a given string for another given substring,
 // and replaces all occurences with a given replacement string.
 func Replace(str, substr, replace cty.Value) (cty.Value, error) {

internal/lang/funcs/string_test.go

@@ -335,6 +335,55 @@ func TestTemplateString(t *testing.T) {
 			cty.StringVal(`it's a value`),
 			``,
 		},
+		{
+			`data.whatever.whatever[each.key].result`,
+			map[string]cty.Value{
+				"data": cty.ObjectVal(map[string]cty.Value{
+					"whatever": cty.ObjectVal(map[string]cty.Value{
+						"whatever": cty.MapVal(map[string]cty.Value{
+							"foo": cty.ObjectVal(map[string]cty.Value{
+								"result": cty.StringVal("it's ${a}"),
+							}),
+						}),
+					}),
+				}),
+				"each": cty.ObjectVal(map[string]cty.Value{
+					"key": cty.StringVal("foo"),
+				}),
+			},
+			cty.ObjectVal(map[string]cty.Value{
+				"a": cty.StringVal("a value"),
+			}),
+			cty.StringVal(`it's a value`),
+			``,
+		},
+		{
+			`data.whatever.whatever[*].result`,
+			map[string]cty.Value{
+				"data": cty.ObjectVal(map[string]cty.Value{
+					"whatever": cty.ObjectVal(map[string]cty.Value{
+						"whatever": cty.TupleVal([]cty.Value{
+							cty.ObjectVal(map[string]cty.Value{
+								"result": cty.StringVal("it's ${a}"),
+							}),
+						}),
+					}),
+				}),
+				"each": cty.ObjectVal(map[string]cty.Value{
+					"key": cty.StringVal("foo"),
+				}),
+			},
+			cty.ObjectVal(map[string]cty.Value{
+				"a": cty.StringVal("a value"),
+			}),
+			cty.NilVal,
+			// We have an intentional hole in our heuristic for whether the
+			// first argument is a suitable expression which permits splat
+			// expressions just so that we can return the type mismatch error
+			// from the result not being a string, instead of the more general
+			// error about it not being a supported expression type.
+			`invalid template value: a string is required`,
+		},
 		{
 			`"can't write $${not_allowed}"`,
 			map[string]cty.Value{},

website/docs/language/functions/templatestring.mdx

@@ -8,13 +8,13 @@ description: |-
 
 -> **Note:** The `templatestring` function is intended for advanced use cases. Most use cases require only a [string template expression](/terraform/language/expressions/strings#string-templates). To render a template from a file, use the [`templatefile` function](/terraform/language/functions/templatefile).
 
-`templatestring` renders a template using a supplied set of template variables.
+`templatestring` renders a template given as a string value, using a supplied set of template variables.
 
 ```hcl
-templatefile(ref, vars)
+templatestring(ref, vars)
 ```
 
-The first parameter must be a simple reference to string value containing the template: for example, `data.aws_s3_object.example.body` or `local.inline_template`.
+The first parameter must be a direct reference to string value containing the template: for example, `data.aws_s3_object.example.body` or `local.inline_template`.
 
 It is **not** valid to supply the template expression directly as the first argument:
 
@@ -25,23 +25,18 @@ templatestring("Hello, $${name}", {
 })
 ```
 
-Instead of the above, you should instead use a string template expression:
+Instead of the above, you should use a normal string template expression:
 
 ```hcl
 "Hello, ${var.name}"
 ```
 
-The `templatestring` function is needed only when the template is available as a named object in the current module.
+The `templatestring` function is needed only when the template is available from a named object, such as a data resource, declared in the current module.
 
 The template syntax is the same as for
 [string templates](/terraform/language/expressions/strings#string-templates)
 in the main Terraform language, including interpolation sequences delimited with
-`${` ... `}`. 
-
-Strings in the Terraform language are sequences of Unicode characters, so
-this function will interpret the file contents as UTF-8 encoded text and
-return the resulting Unicode characters. If the template contains invalid UTF-8
-sequences then this function will produce an error.
+`${` ... `}`.
 
 ## Example
 
@@ -60,7 +55,17 @@ output "example" {
 }
 ```
 
-For more examples of how to use templates, please see the documentation for the [`templatefile`](/terraform/language/functions/templatefile#Examples) function.
+For more examples of how to use templates, refer to the documentation for [the `templatefile` function](/terraform/language/functions/templatefile#Examples).
+
+## Dynamic Template Construction
+
+This function is primarily intended for rendering templates fetched as a single string from remote locations, often using data resources.
+
+You can also use this as a way to construct a template dynamically and then render it, but we recommend treating that only as a last resort because the result tends to be hard to understand, hard to maintain, and fragile to unexpected input.
+
+The restrictions on what kind of syntax is allowed in the first argument are a guardrail to help avoid those new to Terraform thinking that this function is the primary way to render templates in Terraform, but you can bypass those restrictions if you wish by writing an expression that builds a template dynamically and then assigning that expression to a [local value](/terraform/language/values/locals). You can then use a reference to that local value as the first argument to `templatestring`.
+
+If you _do_ choose to construct templates from parts dynamically, be mindful that Terraform has built-in functions that can interact with the local filesystem, and so maliciously-crafted input might produce a template whose result includes data from arbitrary files on the system where Terraform is running.
 
 ## Related Functions