Unlink of file '.git/...' failed

[thany]

• I was not able to find an open or closed issue matching what I'm seeing

Setup

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

$ git --version --build-options

git version 2.32.0.windows.2
cpu: x86_64
built from commit: 3d45ac813c4adf97fe3733c1f763ab6617d5add5
sizeof-long: 4
sizeof-size_t: 8
shell-path: /bin/sh
feature: fsmonitor--daemon

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

$ cmd.exe /c ver

Microsoft Windows [Version 10.0.19043.1110]

That's the 64 bits version, obviously.

• What options did you set as part of the installation? Or did you choose the
  defaults?

# One of the following:
> type "C:\Program Files\Git\etc\install-options.txt"
> type "C:\Program Files (x86)\Git\etc\install-options.txt"
> type "%USERPROFILE%\AppData\Local\Programs\Git\etc\install-options.txt"
$ cat /etc/install-options.txt

Editor Option: VisualStudioCode
Custom Editor Path:
Default Branch Option: master
Path Option: Cmd
SSH Option: OpenSSH
Tortoise Option: false
CURL Option: OpenSSL
CRLF Option: CRLFCommitAsIs
Bash Terminal Option: ConHost
Git Pull Behavior Option: Rebase
Use Credential Manager: Core
Performance Tweaks FSCache: Enabled
Enable Symlinks: Disabled
Enable Pseudo Console Support: Disabled
Enable FSMonitor: Enabled

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

I also use GitKraken, and this issues happens a lot while that's running, but also while it isn't (and even hasn't been sine the latest reboot). I also have VScode open all the time, which has a built-in Git client, but I'm not using it.

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

CMD

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

git pull --rebase

But really many other commands trigger an automatic git gc, as you (authors) probably know ;)

• What did you expect to occur after running these commands?

When git gc triggers automatically, don't complain when files can't be unlinked. As a user, I shouldn't have to care about that. Either git should ignore such problem (if it's harmless), or recover from it transparently.

• What actually happened instead?

(after doing a successful command)

Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
Enumerating objects: 297431, done.
Counting objects: 100% (297431/297431), done.
Delta compression using up to 12 threads
Compressing objects: 100% (74044/74044), done.
Writing objects: 100% (297431/297431), done.
Total 297431 (delta 204855), reused 296282 (delta 203853), pack-reused 0
Unlink of file '.git/objects/pack/pack-2ebaef8c3398262c246a752a06git branch24602c42c80332.pack' failed. Should I try again? (y/n)

This "unlink" question repeats itself seemingly endlessly whether I answer yes or no. It's also impossible to know what the "right" answer is. Should I try again - I don't know, should you? Either way, answering yes just repeats the question for the same file. Answering no advances it to the next file, but god knows how many there are.

• If the problem was occurring with a specific repository, can you provide the
  URL to that repository to help us with testing?

It's a general problem. Happens with multiple repo's, but most notably with very large ones. I wonder if this can happen with the repo for Git itself...

However, it does only happen after doing "some work". I'm not sure exactly when git gc triggers automatically, but it does. And I'm not sure when it decides unlinking a file cannot be done and bothers the user with it, but it will happen "at some point".

[dscho]

Could you try to figure out what process is holding an open handle to that .pack file, e.g. using Sysinternals' HANDLE tool?

[thany]

This might take a while. I'd have to wait until the problem reproduces itself first 😀

In the mean time, is it really a problem if Git would simply ignore being unable to delete in-use .pack files? Does Git really need those files gone?

[dscho]

In the mean time, is it really a problem if Git would simply ignore being unable to delete in-use .pack files? Does Git really need those files gone?

That is an interesting idea.

Obviously, disk space would be wasted more easily this way, and if there was a consistent bug in the future that always keeps a handle to the pack(s) open while trying to clean up no-longer-needed packs, we would easily miss that bug, and disk space usage would skyrocket with every gc.

But as a band-aid, guarded by an opt-in config setting, this could be very useful.

Are you up to implementing it? I would try to find some time to provide you with enough pointers to get started.

[thany]

I was able te reproduce the issue while GitKraken was running. I suspect if GK isn't running, it might be VScode's internal Git client causing this problem in a similar way. I've send some stern feedback GK's way, since I have proof that that's what is keeping those files locked. I haven't been able to prove the same thing for VScode, so I shouldn't be sending feedback their way just yet.

Be that as it may, I'm not actually sure where the fix should be. I'm pretty sure software like that doesn't invent totally custom Git clients, but instead are using ready-to-use packages (or even the commandline) under the bonnet. I do think it's somewhat common to use the CLI and a GUI side-by-side.

As for implementing that option: I'm sorry, but I'm not that kind of programmer. I don't think I'd be able to do a very good job. I can help testing though.

[mikekaganski]

The process holding the files is ... git itself - namely, the other instance of git.

The process tree (as seen in Process Explorer) is:

git.exe		53 556 K	43 744 K	17608	Git for Windows	The Git Development Community	"C:\Program Files\Git\mingw64\bin\git.exe" pull -r
    git.exe		35 964 K	36 600 K	23236	Git for Windows	The Git Development Community	git fetch --update-head-ok
        git.exe		2 756 K	6 996 K	20140	Git for Windows	The Git Development Community	git maintenance run --auto --no-quiet
            git.exe		2 956 K	8 192 K	14024	Git for Windows	The Git Development Community	git gc --auto --no-quiet
                git.exe		2 748 K	7 048 K	28192	Git for Windows	The Git Development Community	git repack -d -l -A --unpack-unreachable=2.weeks.ago

The topmost one holds all those handles to the files failing to unlink; the bottommost one tries to unlink them.

[dscho]

I guess it is that read_cache_unmerged() call that causes Git to open those packs, and the repack later in the call chain needs them not to be opened. The solution probably looks somewhat like this:

diff --git a/builtin/pull.c b/builtin/pull.c
index 3e13f810843..5d51c84d305 100644
--- a/builtin/pull.c
+++ b/builtin/pull.c
@@ -998,6 +998,7 @@ int cmd_pull(int argc, const char **argv, const char *prefix)
 			oidclr(&rebase_fork_point);
 	}
 
+	close_object_store(the_repository->objects);
 	if (run_fetch(repo, refspecs))
 		return 1;
 

@mikekaganski do you think you could try that?

[mikekaganski]

@mikekaganski do you think you could try that?

Yes, I can try to build, but I need help to get an exact command/sequence of actions that would trigger the GC during the pull command.

[dscho]

@mikekaganski do you think you could try that?

Yes, I can try to build, but I need help to get an exact command/sequence of actions that would trigger the GC during the pull command.

I opened a PR, and then used the PR branch to kick off a build the Git artifacts (installer and portable Git, for x86_64): https://github.com/git-for-windows/git/actions/runs/1209162450. Once that run succeeds (knock on wood), it will have the installer and the portable Git as a build artifact for you to test.

If you want to try other things, just fetch the PR branch, make your own modifications, push it to your fork on GitHub, then kick off your own git-artifacts build (see https://github.blog/changelog/2020-07-06-github-actions-manual-triggers-with-workflow_dispatch/ for details how to trigger GitHub workflows manually).

[mikekaganski]

@mikekaganski do you think you could try that?

Yes, I can try to build, but I need help to get an exact command/sequence of actions that would trigger the GC during the pull command.

I opened a PR, and then used the PR branch to kick off a build the Git artifacts (installer and portable Git, for x86_64): https://github.com/git-for-windows/git/actions/runs/1209162450. Once that run succeeds (knock on wood), it will have the installer and the portable Git as a build artifact for you to test.

Thank you!
I only wanted to know if there is something simple that I can make in my local git repo, so that the next time I run git pull there, I know that GC will start, so that it's possible to see immediately if things change or not (GC succeeds with patch, show errors without, repeatable). Sorry, I'm not an experienced git user :)

[dscho]

@mikekaganski hmm. I am not quite sure. You could try to create tons of loose objects, but that might not trigger any repack. Or you could potentially lower the threshold for an auto-gc (see here for details).