libwinpthread seems to be causing issues when cloning repositories in 2.33.0.2

[cwgreene]

• I was not able to find an open or closed issue matching what I'm seeing

For an large internal repository, git clone fails with the following:

Cloning into 'localcheckout'...
remote: Enumerating objects: 159058, done.
remote: Counting objects: 100% (159058/159058), done.
remote: Compressing objects: 100% (31922/31922), done.
fetch-pack: unexpected disconnect while reading sideband packet/s
fatal: early EOF
fatal: fetch-pack: invalid index-pack output

Copying the libwinpthread-1.dll file from the 2.33.0 release onto the 2.33.2 release prevents this from happening.

How can I find out which version of libwinpthread was used in each version? The dll itself does not seem to provide any version information.

sha256sums:

libwinpthread 2.33.0   287106dea25588079daeb3c038181a01c7008eb342f904d0a5071e7d66a987dd
libpwinthread 2.33.0.2 d1fae9e4b08e8d106f037a6225c885abf013f8723cf0d54a69e2dc56e507dbd6

Setup

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?
  64 bit

$ git --version --build-options

git version 2.33.0.windows.2
cpu: x86_64
built from commit: 8735530946cced809cc6cc4c2ca3b078cdb3dfc8
sizeof-long: 4
sizeof-size_t: 8
shell-path: /bin/sh
feature: fsmonitor--daemon

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

$ cmd.exe /c ver

Microsoft Windows [Version 10.0.19042.1237]

• What options did you set as part of the installation? Or did you choose the
  defaults?

# One of the following:
> type "C:\Program Files\Git\etc\install-options.txt"
> type "C:\Program Files (x86)\Git\etc\install-options.txt"
> type "%USERPROFILE%\AppData\Local\Programs\Git\etc\install-options.txt"
$ cat /etc/install-options.txt

Editor Option: VIM
Custom Editor Path:
Default Branch Option:
Path Option: Cmd
SSH Option: OpenSSH
Tortoise Option: false
CURL Option: OpenSSL
CRLF Option: CRLFAlways
Bash Terminal Option: MinTTY
Git Pull Behavior Option: Merge
Use Credential Manager: Core
Performance Tweaks FSCache: Enabled
Enable Symlinks: Disabled
Enable Pseudo Console Support: Disabled
Enable FSMonitor: Disabled

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

** insert your response here **

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

Powershell

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

git clone git@internal.git.server:/path/to/repo

git clone git@internal.git.server:/path/to/repo

• What did you expect to occur after running these commands?

Cloned repository

• What actually happened instead?

Cloning into 'local_checkout'...
remote: Enumerating objects: 159058, done.
remote: Counting objects: 100% (159058/159058), done.
remote: Compressing objects: 100% (31922/31922), done.
fetch-pack: unexpected disconnect while reading sideband packet/s
fatal: early EOF
fatal: fetch-pack: invalid index-pack

• If the problem was occurring with a specific repository, can you provide the
  URL to that repository to help us with testing?

Internal, cannot link I'm afraid.

[rimrul]

How can I find out which version of libwinpthread was used in each version?

We keep a list of package versions for each release

[cwgreene]

How can I find out which version of libwinpthread was used in each version?

We keep a list of package versions for each release

Awesome! Thank you! I'll dig into that. :)

[dscho]

How can I find out which version of libwinpthread was used in each version?

We keep a list of package versions for each release

Awesome! Thank you! I'll dig into that. :)

@cwgreene any news?

[cwgreene]

@dscho no unfortunately. I have honestly been having trouble setting up the environment to build the dlls. I think I somehow managed to build the linux target, but not the windows one (as no dlls were created after running make). Attempts to build using the windows sdk install environment ran into issues (missing a header). Will be resuming attacking this, my pardon for the delays on more information. If I run into the header issue again today I'll update this comment with the name of the header so that others can find the issue if I manage to find a workaround for it :)

[dscho]

FWIW the Pacman package containing this library is mingw-w64-x86_64-winpthreads-git. To build it,

1. install Git for Windows' SDK,
2. sdk cd mingw-w64-winpthreads-git,
3. sdk build
4. install it into the SDK via pacman -U mingw-w64-x86_64-winpthreads-git*.pkg.tar.xz

[dscho]

I see that libwinpthreads was updated recently: git-for-windows/git-sdk-64@25ace34. @cwgreene could you please test the latest snapshot?

[dscho]

I'll just close this until we have further information.

[pe224]

FWIW I ran into this as well.
For me, already 2.33.0.rc0.windows.1 is showing this error, also the most recent release 2.33.1.windows.1

This is effectively breaking git clone for one of our repos (it is not even a particularly large one, total size ~14.4MB).
The only workaround I found is to downgrade to 2.32.0.windows.2.
I would vote to keep the issue open for awareness.

Edit: I tried the latest snapshot 2.33.1.windows.1.13.g3f28c5f141.20211015131715, issue still persists.

[dscho]

I would vote to keep the issue open for awareness.

I don't keep issues open just for the fun of it. If there is no progress, there is no progress, and if that is the case, there is no use keeping the issue open.

The only way to convince me to reopen is to make progress on the issue.

For me, already 2.33.0.rc0.windows.1 is showing this error, also the most recent release 2.33.1.windows.1

You could, of course, bisect even further between those versions in the snapshots. The easiest way would be to use the Portable Git editions, because they can be unpacked side-by-side.

Once you identified the earliest version with the breakage, you can compare their respective /etc/package-versions.txt to verify that your hunch is correct, and it is indeed the libwinpthread difference.

Once the component is identified that causes the reported change of behavior, you can copy over files from the working to the non-working Portable Git, until you identify the culprit (see e.g. https://github.com/git-for-windows/git-sdk-64/blob/main/var/lib/pacman/local/mingw-w64-i686-winpthreads-git-9.0.0.6316.acdc7adc9-1/files for a list of files included in a particular component).

Again, once I see progress on this ticket, I have no problem reopening the ticket.

[pe224]

Fair enough. It was not my intention to step on anybody's toes here.

Bisecting the snapshots, I found that 1abb447 is the first commit that breaks while bdcbb00 is still ok.
I used the 64bit portable MinGit version as this is what I normally use.

The diff of the two /etc/package-versions.txt is

-mingw-w64-x86_64-git 2.32.0.windows.2.13.gbdcbb00aca.20210802230436-1
+mingw-w64-x86_64-git 2.33.0.rc0.windows.1.1.g1abb4477f4.20210803184744-1

-mingw-w64-x86_64-libwinpthread-git 9.0.0.6246.ae63cde27-1
+mingw-w64-x86_64-libwinpthread-git 9.0.0.6270.0e0eed623-1

It does not seem to be connected to the libwinpthread as copying over libwinpthread-1.dll (the only relevant file in the package) from the working commit did not fix the issue on the broken one.

It thus seems to be related to the git package itself, as replacing /mingw64/bin/git.exe from the working commit fixed it in the broken one.

Edit:
it only breaks when cloning via SSH. Same repository as HTTPS works fine.
I am using the Windows-native OpenSSH client via git config core.sshCommand C:/Windows/System32/OpenSSH/ssh.exe
Unfortunately, same as for the issue creator, the only repo with which I can reproduce this is internal and thus non-shareable.