Skip to content

terraform test: include expected diagnostics in verbose mode #37362

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
liamcervante merged 1 commit into from
Jul 24, 2025
Merged

terraform test: include expected diagnostics in verbose mode #37362

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changes/v1.14/ENHANCEMENTS-20250723-122922.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
kind: ENHANCEMENTS
body: 'terraform test: expected diagnostics will be included in test output when running in verbose mode"'
time: 2025-07-23T12:29:22.244611+02:00
custom:
Issue: "37362"
31 changes: 31 additions & 0 deletions internal/command/test_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,37 @@ func TestTest_Runs(t *testing.T) {
expectedOut: []string{"1 passed, 0 failed."},
code: 0,
},
"expect_failures_outputs": {
expectedOut: []string{"1 passed, 0 failed."},
code: 0,
},
"expect_failures_checks_verbose": {
override: "expect_failures_checks",
args: []string{"-verbose"},
expectedOut: []string{"1 passed, 0 failed.", "Warning: Check block assertion failed"},
code: 0,
},
"expect_failures_inputs_verbose": {
override: "expect_failures_inputs",
args: []string{"-verbose"},
expectedOut: []string{"1 passed, 0 failed."},
expectedErr: []string{"Error: Invalid value for variable"},
code: 0,
},
"expect_failures_resources_verbose": {
override: "expect_failures_resources",
args: []string{"-verbose"},
expectedOut: []string{"1 passed, 0 failed."},
expectedErr: []string{"Error: Resource postcondition failed"},
code: 0,
},
"expect_failures_outputs_verbose": {
override: "expect_failures_outputs",
args: []string{"-verbose"},
expectedOut: []string{"1 passed, 0 failed."},
expectedErr: []string{"Error: Module output value precondition failed"},
code: 0,
},
"multiple_files": {
expectedOut: []string{"2 passed, 0 failed"},
code: 0,
Expand Down
14 changes: 11 additions & 3 deletions internal/moduletest/graph/apply.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ func (n *NodeTestRun) testApply(ctx *EvalContext, variables terraform.InputValue

// Remove expected diagnostics, and add diagnostics in case anything that should have failed didn't.
// We'll also update the run status based on the presence of errors or missing expected failures.
failOrErr := n.checkForMissingExpectedFailures(run, applyDiags)
failOrErr := n.checkForMissingExpectedFailures(ctx, run, applyDiags)
if failOrErr {
// Even though the apply operation failed, the graph may have done
// partial updates and the returned state should reflect this.
Expand Down Expand Up @@ -172,11 +172,19 @@ func (n *NodeTestRun) apply(tfCtx *terraform.Context, plan *plans.Plan, progress

// checkForMissingExpectedFailures checks for missing expected failures in the diagnostics.
// It updates the run status based on the presence of errors or missing expected failures.
func (n *NodeTestRun) checkForMissingExpectedFailures(run *moduletest.Run, diags tfdiags.Diagnostics) (failOrErr bool) {
func (n *NodeTestRun) checkForMissingExpectedFailures(ctx *EvalContext, run *moduletest.Run, diags tfdiags.Diagnostics) (failOrErr bool) {
// Retrieve and append diagnostics that are either unrelated to expected failures
// or report missing expected failures.
unexpectedDiags := run.ValidateExpectedFailures(diags)
run.Diagnostics = run.Diagnostics.Append(unexpectedDiags)

if ctx.Verbose() {
// in verbose mode, we still add all the original diagnostics for
// display even if they are expected.
run.Diagnostics = run.Diagnostics.Append(diags)
} else {
run.Diagnostics = run.Diagnostics.Append(unexpectedDiags)
}
Comment on lines +180 to +186
Copy link
Copy Markdown
Member

@dsa0x dsa0x Jul 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this not likely change the status of the run?

Copy link
Copy Markdown
Member

@dsa0x dsa0x Jul 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or are we saying that's okay in verbose mode?

Copy link
Copy Markdown
Contributor Author

@liamcervante liamcervante Jul 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it does anymore. I made a change recently where it is much more explicit about setting the status. So, I think from after this point it doesn't look at the set of run.Diagnostics as a whole to work out the test status anymore but sets it's explicitly based on checks made at each stage.

For example see here for apply and here for plan. In both cases, the status is computed based on just the diagnostics of the current step and then set separately to the big collection of diagnostics.

So, I think at this stage it is safe to add error diagnostics into the general diagnostics as long as we're not actually changing the result of the test when doing this.


for _, diag := range unexpectedDiags {
// // If any diagnostic indicates a missing expected failure, set the run status to fail.
if ok := moduletest.DiagnosticFromMissingExpectedFailure(diag); ok {
Expand Down
14 changes: 11 additions & 3 deletions internal/moduletest/graph/plan.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,11 +32,19 @@ func (n *NodeTestRun) testPlan(ctx *EvalContext, variables terraform.InputValues
tfCtx, _ := terraform.NewContext(n.opts.ContextOpts)

// execute the terraform plan operation
planScope, plan, planDiags := n.plan(ctx, tfCtx, setVariables, providers, mocks, waiter)
planScope, plan, originalDiags := n.plan(ctx, tfCtx, setVariables, providers, mocks, waiter)
// We exclude the diagnostics that are expected to fail from the plan
// diagnostics, and if an expected failure is not found, we add a new error diagnostic.
planDiags = run.ValidateExpectedFailures(planDiags)
run.Diagnostics = run.Diagnostics.Append(planDiags)
planDiags := run.ValidateExpectedFailures(originalDiags)

if ctx.Verbose() {
// in verbose mode, we still add all the original diagnostics for
// display.
run.Diagnostics = run.Diagnostics.Append(originalDiags)
} else {
run.Diagnostics = run.Diagnostics.Append(planDiags)
}

if planDiags.HasErrors() {
run.Status = moduletest.Error
return
Expand Down