cache trying to use git for windows' tar.exe on self-hosted runner, failing to find correct gzip

[jeremyd2019]

On a self-hosted Windows runner, I just noticed that caches are failing now with errors like:

"C:\Program Files\Git\usr\bin\tar.exe" --posix -cf cache.tgz --exclude cache.tgz -P -C C:/runner/_work/msys2-autobuild/msys2-autobuild --files-from manifest.txt --force-local -z
/bin/sh: line 1: gzip: command not found
/usr/bin/tar: cache.tgz: Cannot write: Broken pipe
/usr/bin/tar: Child returned status 127
/usr/bin/tar: Error is not recoverable: exiting now
Warning: Failed to save: "C:\Program failed with error: The process 'C:\Program Files\Git\usr\bin\tar.exe' failed with exit code 2

Caches used to work fine, using the version of tar.exe in system32. My understanding is that the programs in git for windows usr\bin are supposed to be regarded as internal to Git for Windows and should not be relied upon by external things like this. @dscho?

Current theory is that Git for Windows' tar is looking on the path for gzip, either not finding it in my case, or in other cases finding one incompatible with that tar (due to being based on a different Cygwin).

[jeremyd2019]

Maybe it is failing to find gzip because git for windows' usr\bin is not on my path? But it really shouldn't be on the path.

[me-and]

I just came to report what is, I suspect, the same bug. Do you have Cygwin installed on these runners?

The problem I'm seeing is that Git for Windows is based on MSYS2, which is downstream from Cygwin, and in particular is based on Cygwin DLLs that are a few years old. If you have a more recent regular Cygwin installation on the same system, and it's in the PATH, the caching code will still run the GfW tar, but that will try to link against the Cygwin DLL from your Cygwin installation, and promptly fall over with the error you're seeing.

Assuming it is the same problem, I've just created a simple testcase using GitHub hosted runners, at https://github.com/me-and/repro-cygwin-cache-woe.

[jeremyd2019]

I don't have cygwin on the path, but I could see where that could cause problems too.

What's worse, we just discovered this over at MSYS2:

toolkit/packages/cache/src/internal/tar.ts

Line 17 in 6c1f9ea

exportVariable('MSYS', 'winsymlinks:nativestrict')

This exports the variable for all subsequent steps too, switching the default behavior of any MSYS2 used later in the workflow.

So there are at least two pretty major issues with this change:

1. tar is looking on the path for the compression program, which may not be on the path, or may be an incompatible variant (such as one from a Cygwin or different MSYS2 install).
2. the code here is setting (and/or potentially overriding an existing setting) an environment variable globally, affecting all future actions unexpectedly (based on whether something used a cache prior to those steps)

[lazka]

What's worse, we just discovered this over at MSYS2:

toolkit/packages/cache/src/internal/tar.ts

Line 17 in 6c1f9ea

exportVariable('MSYS', 'winsymlinks:nativestrict')

@Phantsure it would be nice if an alternative to this could be found. There is no good reason to set global env vars and change the default behavior for all MSYS2 users.

[lazka]

I've created a separate issue for the env var issue, to not sidetrack this issue: #1312

[me-and]

The problem I'm seeing is that Git for Windows is based on MSYS2, which is downstream from Cygwin, and in particular is based on Cygwin DLLs that are a few years old.

No, MSYS2 is based on the latest Cygwin.

"A few years" was an overstatement (I think I got confused by the the calendar turning over, and overcompensated), but I believe Git for Windows is, as of v2.38.0.windows.1, based on Cygwin 3.3.6, while the latest upstream Cygwin release is v3.4.3.

Downgrading to use Cygwin v3.3.6 gets the tar call working, so I'm pretty sure that the Cygwin DLL compatibility is the problem (although I don't think just switching library versions is the correct solution).

[dscho]

Git for Windows is based on MSYS2,

correct

which is downstream from Cygwin,

correct

and in particular is based on Cygwin DLLs that are a few years old.

Incorrect. Your information is more than 7 years old; Git for Windows v1.x was based on MSys which indeed was forked off of an ancient Cygwin version. But MSYS2 frequently updates to the latest Cygwin; Git for Windows lags behind a little to take off the edge of some of the more fragile developments, at the moment Git for Windows uses a derivative of Cygwin v3.3.6 (which was released on September 6th 2022, hardly "several years old").

the caching code will still run the GfW tar, but that will try to link against the Cygwin DLL from your Cygwin installation

That, too, is incorrect. Git for Windows' tar.exe links to msys-2.0.dll and will never "magically" link to cygwin1.dll. In other words, it will never pick up the DLL from the Cygwin installation.

So how about the MSYS2 installation that's in C:\msys64 on hosted runners? Git for Windows' tar.exe won't use C:\msys64\usr\bin\msys-2.0.dll either because there is a msys-2.0.dll right next to tar.exe, in C:\Program Files\Git\usr\bin, and that will be used.

My understanding is that the programs in git for windows usr\bin are supposed to be regarded as internal to Git for Windows and should not be relied upon by external things like this. @dscho?

I tried to make the case and I even almost got a change accepted to prevent the internal tools of Git for Windows to be picked up. However, too many users already relied on them tools to be in the PATH and we could not merge that change. I've made my peace with it.