File provisioning hangs with Terraform 12.* terraforming a Windows server with Powershell 5 installed

[ericblackburn]

Terraform Version

Terraform v0.12.3
+ provider.aws v2.17.0

Terraform Configuration Files

data "aws_ami" "ami" {
  most_recent = true
  owners      = ["self"]

  filter {
    name   = "name"
    values = ["TestPowershell5Ami*"]
  }
}
resource "aws_instance" "TestTerraform12" {
  ami                    = data.aws_ami.ami.id
  count                  = 1
  instance_type          = "t2.medium"
  availability_zone      = "us-east-1a"
  key_name               = "****"
  subnet_id              = "subnet-*********"
  vpc_security_group_ids = ["sg-*******"]
  root_block_device {
    volume_size = 80
  }
  provisioner "file" {
    source      = "TestFolder1"
    destination = "C:/Terraform/TestFolder1"
    connection {
      host     = coalesce(self.public_ip, self.private_ip)
      type     = "winrm"
      user     = "Test"
      password = "Password"
      timeout  = "15m"
      https    = true
      port     = "5986"
      insecure = true
    }
  }
  provisioner "file" {
    source      = "TestFolder2"
    destination = "C:/Terraform/TestFolder2"
    connection {
      host     = coalesce(self.public_ip, self.private_ip)
      type     = "winrm"
      user     = "Test"
      password = "Password"
      timeout  = "15m"
      https    = true
      port     = "5986"
      insecure = true
    }
  }
}

Debug Output

terraform.exe apply -auto-approve -no-color -target aws_instance.TestTerraform12
data.aws_ami.ami: Refreshing state...
aws_instance.TestTerraform12[0]: Creating...
aws_instance.TestTerraform12[0]: Still creating... [10s elapsed]
aws_instance.TestTerraform12[0]: Still creating... [20s elapsed]
aws_instance.TestTerraform12[0]: Still creating... [30s elapsed]
aws_instance.TestTerraform12[0]: Provisioning with 'file'...
aws_instance.TestTerraform12[0]: Still creating... [40s elapsed]
aws_instance.TestTerraform12[0]: Still creating... [50s elapsed]
aws_instance.TestTerraform12[0]: Still creating... [1m0s elapsed]
aws_instance.TestTerraform12[0]: Still creating... [1m10s elapsed]
aws_instance.TestTerraform12[0]: Still creating... [1m20s elapsed]
aws_instance.TestTerraform12[0]: Still creating... [1m30s elapsed]
...

aws_instance.TestTerraform12[0]: Still creating... [14m40s elapsed]
aws_instance.TestTerraform12[0]: Still creating... [14m50s elapsed]
aws_instance.TestTerraform12[0]: Still creating... [15m0s elapsed]
aws_instance.TestTerraform12[0]: Still creating... [15m10s elapsed]
aws_instance.TestTerraform12[0]: Still creating... [15m20s elapsed]
aws_instance.TestTerraform12[0]: Still creating... [15m30s elapsed]
aws_instance.TestTerraform12[0]: Still creating... [15m40s elapsed]
Interrupt received.
Please wait for Terraform to exit or data loss may occur.
Gracefully shutting down...
Stopping operation...
Two interrupts received. Exiting immediately. Note that data
loss may have occurred.

The interrupt received was me cancelling the cmd line process.

Crash Output

N/A

Expected Behavior

On a Windows Server with Powershell 5 installed, be able to use Terraform 12.* to complete file provisioning of TestFolder1, copying all content and subfolder content, and continue to the next file provisioning step.

This works in Terrafform 11.*
I am using terraform v0.11.8 successfully with provider.aws v1.35.0

Actual Behavior

Since upgrading to Terraform 12.*, the file provisioning of TestFolder1 copies a couple of the 1 KB files over, seems to copy over what can be done in 1 minute, and then stops copying over content. The Terraform console logging continues to report "Still creating..." without end and doesn't obey the timeout. The Terraform process never completes.

Steps to Reproduce

Setup Windows Server 2012 to have Powershell 5

1. Create a Window Server 2012 R2 64 bit instance and manually or via cli
   For AWS, I am using the source AMI filter

"source_ami_filter": {
			"filters": {
				"virtualization-type": "hvm",
				"name": "Windows_Server-2012-R2_RTM-English-64Bit-Base-*",
				"root-device-type": "ebs"
			},
			"owners": ["amazon"],
			"most_recent": true
		},

2. Install Powershell 5 directly via the MSU or through Chocolatey, both have the same effect.
   2.1 Chocolatey via, "choco install powershell"
   2.2 Manually by downloading Win8.1AndW2K12R2-KB3191564-x64.msu from https://www.microsoft.com/en-us/download/details.aspx?id=54616, Windows Management Framework 5.1 (KB3191564).
   2.2.1 Install via the GUI manually or run "Win8.1AndW2K12R2-KB3191564-x64.msu /quiet" in a cmd line Administrator elevated prompt.
3. Restart the system to complete the install
4. Save the image as TestPowershell5Ami

Another option is to leverage a Windows Server 2016 instance, which natively uses Powershell 5.1. For AWS, I used the "name": "Windows_Server-2016-English-Full-Containers-*" source_ami_filter.

Attempt to terraform

1. Setup a TestFolder1 folder with over 10 1kb files, something that would take over a minute to file provision on your network.
2. terraform init
3. terraform apply

Additional Context

Terraform 11.* has never been an issue. The Terraform file isn't new, outside of being upgraded to Terraform 12's syntax, using the automated upgrade cmd. Without Powershll 5 installed, I can terraform the Windows Server system, with Powershell 4.0 natively, just fine. After installing Powershell 5 or against a Windows Server 2016 instance with Powershell 5.1 natively installed, Terraform hangs and never completes or errors out.

The Windows system being terraformed has a Powershell execution policy set to: localmachine bypass.
"set-executionpolicy bypass -force"

The server I run Terraform from has Powershell 5.1 installed.

References

N/A

[acbreeze]

I have the exact same issue. I switched to 11.14 and now file provisioner step completes no problem.

[anmiles]

Have the same issue on 0.12.3.
It's being reproduced on the file provisioners when either instance or null_resource created.
For example, here is what I see when file provisioner working when aws_instance created. It waits 5 minutes until instance created and then starting uploading the files. Finally all the files are being uploaded (I've double-checked this) but provisioner will never stop. Although timeout is set to 15m.

aws_instance.default: Still creating... [5m0s elapsed]
2019-07-17T15:51:35.754+0300 [DEBUG] plugin.terraform.exe: file-provisioner (internal) 2019/07/17 15:51:35 [DEBUG] connecting to remote shell using WinRM
2019-07-17T15:51:37.325+0300 [DEBUG] plugin.terraform.exe: file-provisioner (internal) 2019/07/17 15:51:37 [DEBUG] Uploading dir 'scripts/remote/' to 'C:/scripts/'
aws_instance.default: Still creating... [5m10s elapsed]
aws_instance.default: Still creating... [5m20s elapsed]
aws_instance.default: Still creating... [5m30s elapsed]

(still waiting and waiting)

aws_instance.default: Still creating... [30m20s elapsed]
aws_instance.default: Still creating... [30m30s elapsed]

(etc)

[mc-jared]

I also have this problem with terraform 0.12.5 and aws 2.19
Including the timeout failing to be hit.
I let it run through a meeting and came back to over 90 minutes still waiting.

aws_instance.DIR[0]: Still creating... [1h30m20s elapsed]
aws_instance.DIR[2]: Still creating... [1h30m20s elapsed]
aws_instance.DIR[0]: Still creating... [1h30m30s elapsed]
aws_instance.DIR[2]: Still creating... [1h30m30s elapsed]
aws_instance.DIR[0]: Still creating... [1h30m40s elapsed]
aws_instance.DIR[2]: Still creating... [1h30m40s elapsed]
aws_instance.DIR[0]: Still creating... [1h30m50s elapsed]
aws_instance.DIR[2]: Still creating... [1h30m50s elapsed]
aws_instance.DIR[0]: Still creating... [1h31m0s elapsed]
aws_instance.DIR[2]: Still creating... [1h31m0s elapsed]
aws_instance.DIR[0]: Still creating... [1h31m10s elapsed]
(etc)

My behavior is that occasionally 1-2 of the connections will occur so long as you only have 1 resource defined in the .tf file. If you have more than 1 resource that have the file provisioner none of them will connect using winRM.

[zlilA]

Having the same issue, terraform 0.12 using the google provider.
Trying to create a file using the file provisioner and the contents is generated by templatefile.

[mauri]

Having the same issue, using aws provider

Terraform v0.12.7
+ provider.aws v2.17.0

Creating an aws_instance with

provisioner "file" {
    destination = "/root/foo"
    content     = <<-EOF
    NODE_ENV=production
    APPLICATION_URL=foo
EOF
}

hangs on Still creating... for 5 minutes and then fails with timeout. The instance is created successfully but the file is never created.

[kchoudhury]

Facing the same issue with windows 10 over winrm using vsphere provider and file provisioner , from the folder only some of the files get copied and the creation goes into endless loop .

If inside the folder i am copying i only have 1 file it always works as expected

[mildwonkey]

Hi folks! I'm sorry that y'all are experiencing this behavior.

Please do not post "+1" comments here, since it creates noise for others watching the issue and ultimately doesn't influence our prioritization because we can't actually report on these. Instead, react to the original issue comment with 👍, which we can and do report on during prioritization.

That being said, if you have an example of this issue that includes new information, please do continue to share!

[Bandyman]

I saw this issue as well when upgrading to terraform v12.3 as soon as it became viable for us to update but I assumed it was something wrong with our windows template or VSphere instance as changes had been done to these at the same time.

I have noticed this behaviour on WIndows 10 versions, 1511, 1803 and 1903.
This has been an issue for us since 0.12.3 after upgrading from 0.11.14, and still an issue when upgrading from 0.12.3 to 0.12.6.
Updated to 0.12.7 a couple of days ago and still have this issue.

During this version timeframe we have been using Vpshere provider 1.11.0 and 1.12.0.

The issue seems inconsitend however, Say I roll out with count=10, perhaps the first 3 make it ok and the rest just hang.

We install a bunch of things into the template but the only things we touch with powershell is PowerCLI and AD Module.

Hope this info helps.

edit
Forgot to mention, we see this with both provisioning a file and when trying to write text to a .txt file. for example:

  provisioner "file" {
    content     = var.local_admin_password
    destination = "C:\\scripts\\file_local.txt"
    connection {
      host     = self.default_ip_address
      type     = "winrm"
      user     = "Administrator"
      password = var.local_admin_password
    }
  }

  provisioner "file" {
    source      = "${var.project_dir}/scripts/"
    destination = "C:\\scripts"
    connection {
      host     = self.default_ip_address
      type     = "winrm"
      user     = "Administrator"
      password = var.local_admin_password
    }
  }

[0xVox]

This issue appears to occur with remote-exec's as well. I replaced all my file provisioners with a combination of local-exec and remote-exec that would transfer the files to my instance(s) via AWS S3 - now the unzip stage (Of very small files <1MB) will hang in this same way.

Experienced this on Windows Server 2019 DC Edition