git clean deletes module folders when removing .terraform directory

[pmcevoy]

Terraform Version

Terraform v0.11.10

Also note that this is Windows 10, with git version 2.19.1.windows.1

Expected Behavior

I have authored some modules in a local git repo. Modules are in one path, and terraform files in another path. When I git clean the repo, I expect only .terraform folder to be removed

Actual Behavior

Module files are deleted from the filesystem when the .terraform dir is removed

Steps to Reproduce

From a clean git repo containing terraform modules in one path, and terraform files in another such as:

├───Environment
│   └───DingDev
│       ├───Core
│       ├───Dev
│       └───Networks
└───Modules
    ├───Stripe
    ├───Vpc
    └───Vpn

1 terraform init in (say) the Dev folder which contains .tf files importing modules from the Modules folder

 [master ≡]> terraform init
Initializing modules...
- module.pink
  Getting source "../../../Modules/Stripe"

2 git clean -fxd to remove the .terraform directory:

[master ≡]> git clean -fxd
Removing .terraform/

3. Discover that all the files in ../../../Modules/Stripe have been removed:

 [master ≡ +0 ~0 -2 !]> git status
On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        deleted:    ../../../Modules/Stripe/inputs.tf
        deleted:    ../../../Modules/Stripe/main.tf

no changes added to commit (use "git add" and/or "git commit -a")

Additional Context

This is my .gitignore:

terraform.tfvars
terraform.tfstate
*.backup
.terraform
*.swp

[apparentlymart]

Hi @pmcevoy! I'm sorry things didn't work out as expected here.

As far as I can tell, this is a git question rather than a Terraform question. Terraform is not involved in the execution of git clean here. Did I misunderstand?

Please note that we use GitHub issues for tracking bugs and enhancements rather than for questions. While we may be able to help with certain simple problems here it's generally better to use one of the community forums where there are far more people ready to help, whereas the GitHub issues here are generally monitored only by our few core maintainers.

[pmcevoy]

That's an interesting response... Let me try to convince you that this is not just a git clean issue. After investigating a little more about the structure of the .terraform directory I can see that it contains a hardlink/junction point that links to the Module folder.

C:\Source\Infrastructure\Src\Environment\DingDev\Dev>dir /al /s
 Volume in drive C is Windows
 Volume Serial Number is 3036-115F

 Directory of C:\Source\Infrastructure\Src\Environment\DingDev\Dev\.terraform\modules

12/13/2018  11:23 AM    <JUNCTION>     9f8cd12450a82da6f3038a999a83c745 [C:\Source\Infrastructure\Src\Modules\Stripe]
               0 File(s)              0 bytes

     Total Files Listed:
               0 File(s)              0 bytes
               1 Dir(s)  342,818,054,144 bytes free

I presume this is so that the modules can be edited in place and not need to be explicitly updated each time they are changed

However, this then means that a recursive delete of the .terraform directory will traverse the link and delete the referenced files. Here is a PowerShell that will delete files in the .terraform directory from the leaf inwards (files with the longest path length are deleted first)

Get-ChildItem .terraform -Recurse | Sort-Object @{e={$_.FullName.Length}} -Descending | Remove-Item -Force

After executing this, the source files in the original Modules folder are deleted in the same way that they get deleted using git clean.

You may say that no-one deletes folders that way, but I would say that the usage of a link in the .terraform folder is surprising/unexpected and potentially dangerous with other tools - not just git clean

[apparentlymart]

Thanks for that additional context, @pmcevoy.

Indeed, Terraform currently creates a Directory Junction to connect the entry in .terraform/modules to a local directory when running on Windows. This appears to be an attempt to mimic the behavior of creating a symlink when on a Unix system. When running on a Unix system, Git recognizes when it encounters a symlink during clean and does not recurse into it.

I found some interesting discussion over in git-for-windows/git#437 where someone makes a convincing argument that a junction point is more comparable to a Unix mount point than to a symlink, and I can see that perspective. The behavior of Terraform here (or rather, of the underlying go-getter library that Terraform uses for this purpose) is therefore not ideal. Ideally it would instead create an NTFS symlink reparse point, which is a more direct analog of a Unix symlink (though git may not handle that consistently either).

It seems that the discussion didn't yield an actual change yet and Git for Windows is still treating a directory junction just as a directory. As long as that remains true, it's not safe to use directory junctions under a git work tree.

The good news here is that this will become a moot point in the forthcoming v0.12.0 release because we've introduced a new module installer that no longer uses go-getter to deal with local directories, instead just referring directly to the local module directory without creating any links at all. This means those directories will no longer be created when installing modules in v0.12.0, and any pre-existing directory junctions there will be entirely ignored by Terraform after upgrading and can be deleted using a directory-junction-aware delete command.

Now that I understand the details, I agree that the go-getter behavior here is flawed and it should instead be either creating a real NTFS symlink or just copying the directory. Other go-getter callers may benefit from such a fix. The only other caller at HashiCorp is Nomad, which has a different set of use-cases that don't involve writing into Git work trees, but I know there is a small number of third-party codebases that import it, so I'm going to file an upstream go-getter issue to record this problem for posterity.

[apparentlymart]

I have created the upstream go-getter issue, which GitHub has linked above ☝️ .

The change for v0.12.0 I mentioned is already in the master branch ready to be included in the release, so there's no further development work required in Terraform for this. The latest alpha builds should reflect this new behavior if you'd like to verify, though of course I'd suggest not using those prerelease builds in any "real" workspaces since there are various known and unknown bugs in other subsystems.

Since a fix for this is already in, I'm going to close this out. Thanks for pointing out this unexpected interaction, though!

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.