SSH connections are kept open

[Michkov]

Terraform Version

Terraform v1.6.2
on linux_amd64
+ provider registry.terraform.io/hashicorp/null v3.1.1

Terraform Configuration Files

terraform {
  required_version = "~> 1.1"
  required_providers {
    null = {
      source  = "hashicorp/null"
      version = "~> 3.1.1"
    }
  }
}

variable "host" {
  type = string
  default = "139.144.128.15"
}
variable "user" {
  type = string
  default = "root"
}
variable "private_key" {
  type = string
  default = "/home/mkovarik/.ssh/mkovarik-dev"
}


resource "null_resource" "ssh_connection" {
  connection {
    type                = "ssh"
    host                = var.host
    agent               = "false"
    user                = var.user
    timeout             = "600s"
    private_key         = file(var.private_key)
  }
  provisioner "remote-exec" {
    inline = [
       "touch abc",
    ]
  }
}
resource "null_resource" "more_ssh_connections" {
  connection {
    type                = "ssh"
    host                =  var.host
    agent               = "false"
    user                =  var.user
    timeout             = "600s"
    private_key         = file(var.private_key)
  }
  provisioner "remote-exec" {
    inline = [
       "touch abc",
       "touch abc",
    ]
  }
  provisioner "remote-exec" {
    inline = [
       "touch abc",
    ]
  }
}
resource "null_resource" "local-sleep" {
  depends_on = [
     null_resource.ssh_connection,
     null_resource.more_ssh_connections
  ]
  provisioner "local-exec" {
       command = "sleep 10; netstat -natp | grep ${var.host}"
  }
}

Debug Output

https://gist.github.com/Michkov/302f06370581fa66f216ed80bac09097

Expected Behavior

Execution of netstat should not show ESTABLISHED connection to the host, SSH connection should be closed directly after execution.

Actual Behavior

SSH connections to the host are still ESTABLISHED even the commands were executed.

Steps to Reproduce

1. Set variables -> host, user and private_key
2. terraform init
3. terraform apply

Additional Context

Terraform keeps SSH connections opened. This is problematic when hosts are having connection limits set. Also when using Bastions there are too many unnecessary connections kept established.

References

No response

[apparentlymart]

This is giving me some very faint memories of some decisions made many years ago now as tradeoffs for folks who wanted to be able to, for various reasons, use a provisioner to tell a VM to reboot itself.

Originally the SSH communicator would sometimes consider that as an error because the reboot command would race with the SSH server, sometimes forcing the SSH server to shut down before the reboot command was able to report success.

I don't remember exactly what we did to try to accommodate that situation, but I wonder if whatever that solution was also led to the possibility of the SSH connection being possibly left open if the server doesn't actively close the connection itself.

Provisioners are a last resort and so we typically just try to maintain their current functionality as best we can. If fixing this would risk regressing whatever we made work earlier then we should probably tread carefully. But to answer that we'll first need to track down the details of what I'm faintly remembering to see if it's related to this problem after all.

[Michkov]

Seems that change happened between v0.14 (connection closed) and v0.15 (connection opened).

[jbardin]

This is a small bug which was exacerbated when converting provisioners from being plugins, while still retaining the same plugin API internally. The context used to close the connection was actually tied to the plugin lifetime not just the remote-exec execution, but the plugin and the connection had roughly the same lifetime in simple cases before so it was never noticed. It should just be a matter of updating the connection context.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.