terraform apply creates temporary files via provisioner "file" + connection "ssh"

[marsskop]

Terraform Version

Terraform v1.1.9
on linux_amd64
+ provider registry.terraform.io/hashicorp/null v3.2.0

Terraform Configuration Files

resource "null_resource" "spread_files" {
  for_each = local.nodes  # set of IPs
  triggers = {
    user        = "user"
    password    = local.ssh_password
    file_name   = "dummy.txt"
    dest_path   = "/home/user"
  }
  connection {
    type        = "ssh"
    user        = self.triggers.user
    host        = each.key
    password    = self.triggers.password
  }

  provisioner "file" {
    source      = "./${self.triggers.file_name}"
    destination = "${self.triggers.dest_path}/${self.triggers.file_name}"
  }
}

Debug Output

...
2022-11-05T07:49:25.370Z [DEBUG] Connection established. Handshaking for user user
2022-11-05T07:49:25.370Z [DEBUG] Connection established. Handshaking for user user
2022-11-05T07:49:25.371Z [DEBUG] Connection established. Handshaking for user user
2022-11-05T07:49:25.618Z [DEBUG] starting ssh KeepAlives
2022-11-05T07:49:25.618Z [DEBUG] starting ssh KeepAlives
2022-11-05T07:49:25.620Z [DEBUG] opening new ssh session
2022-11-05T07:49:25.620Z [DEBUG] opening new ssh session
2022-11-05T07:49:25.639Z [DEBUG] starting ssh KeepAlives
2022-11-05T07:49:25.642Z [DEBUG] opening new ssh session
2022-11-05T07:49:25.672Z [DEBUG] Starting remote scp process: 'scp' -vt /home/user
2022-11-05T07:49:25.697Z [DEBUG] Starting remote scp process: 'scp' -vt /home/user
2022-11-05T07:49:25.703Z [DEBUG] Started SCP session, beginning transfers...
2022-11-05T07:49:25.707Z [DEBUG] Copying input data into temporary file so we can read the length
2022-11-05T07:49:25.709Z [DEBUG] Beginning file upload...
2022-11-05T07:49:25.722Z [DEBUG] Starting remote scp process: 'scp' -vt /home/user
2022-11-05T07:49:25.740Z [DEBUG] Started SCP session, beginning transfers...
2022-11-05T07:49:25.744Z [DEBUG] Copying input data into temporary file so we can read the length
2022-11-05T07:49:25.745Z [DEBUG] SCP session complete, closing stdin pipe.
2022-11-05T07:49:25.745Z [DEBUG] Waiting for SSH session to complete.
2022-11-05T07:49:25.747Z [DEBUG] Beginning file upload...
2022-11-05T07:49:25.796Z [DEBUG] Started SCP session, beginning transfers...
2022-11-05T07:49:25.797Z [ERROR] scp stderr: "Sink: C0644 17 dummy.txt\n"
2022-11-05T07:49:25.797Z [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState to workingState for null_resource.spread_files["xxx"]
2022-11-05T07:49:25.797Z [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState: writing state object for null_resource.spread_files["xxx"]
null_resource.spread_files["xxx"]: Creation complete after 1s [id=8674665223082153551]
...

Expected Behavior

Provisioner "file" with connection "ssh" should copy files only to the destination.

Actual Behavior

Provisioner "file" with connection "ssh" creates temporary files in TMPDIR:

> fswatch -0 -x tmp/ | xargs -0 -n1 echo
...
/tmp/terraform_issues/tmp/terraform-upload1615626800 Created IsFile Updated Removed
/tmp/terraform_issues/tmp/terraform-upload439229309 Created IsFile Updated Removed
/tmp/terraform_issues/tmp/terraform-upload60184181 Created IsFile Updated Removed
...

Steps to Reproduce

1. TMPDIR=/tmp/terraform_issues/tmp terraform init
2. TMPDIR=/tmp/terraform_issues/tmp terraform apply

Additional Context

We run Terraform in a Docker image:

> cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.9 (Maipo)

We use provisioner "file" to distribute big files between machines, for example, a 10 Gb file between 10 machines simultaneously. This means that we (and our customers) should have 100+ Gb of memory in TMPDIR which is unviable. Some customers want to set custom TMPDIR with hard limitations on the size of the directory.
I believe the issue lies within this part of code: https://github.com/hashicorp/terraform/blob/v1.1.9/internal/communicator/ssh/communicator.go#L630-L664
The same part of code is in newer versions of Terraform too.

References

No response

[crw]

Thanks for this report!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.