Look into MSys2 #25

dscho · 2015-02-13T09:41:05Z

I am looking into using MSys2 for future Git for Windows development, for a couple of reasons:

MSys2 packages are much more up-to-date than MSys ones
MSys2 offers a nice, stable package management system, pacman (ported from Arch Linux)
MSys2 already ships with mintty, a powerful alternative to the cmd.exe command-line window
MSys2 has 64-bit support
MSys2 allows us to (re-)build Perl with Subversion bindings easily (at least that is the impression I got)

This ticket mainly serves as a notepad listing quirks and issues I find out while working on getting Git for Windows to build with MSys2.

It is quite easy to get started with MSys2:

download and install MSys2 as mentioned on https://msys2.github.io/
run the MINGW shell (not the MSys2 shell) from the Start Menu
run pacman -Sy git mingw-w64-x86_64-gcc to install Git and GCC (-S means "synchronize", i.e. install, and -y does not mean "yes" but "refresh the package list before installing the newest versions of the specified packages")
clone https://github.com/git-for-windows/MINGW-packages, say, into /usr/src/
in MINGW-packages/mingw-w64-git/, run makepkg-mingw -s (the -s says: install all dependencies, including build time dependencies, via pacman)
Play with the unpacked sources (they are in the src/build-x86_64/ or the src/build-i686/ subdirectory)

So it looks that it is pretty easy to get started, as well as to rebuild the MSys2 installer (which we might want to do in order to continue offering a net installer to contributors that avoids requiring manual steps).

Having said that, there are a few bumps on the road to heave: The following issues need to be solved before we can start bundling up even so much as a portable application:

tab completion in the MSys2 shell seems to be flakey (it sometimes does not find anything, and pressing the Tab key repeatedly will not do anything in that case, but often it helps to delete the last character, retype it, and then press the Tab key again). This did not happen with the MinGW shell, funnily enough, so it might not be a huge problem for us; However, it could indicate a more problematic, deeper bug
MSys2's openssh refuses to heed the HOME variable and always looks into /home/.ssh/
MinGW Git built with makepkg-mingw hardcodes the absolute exec path (e.g. C:/msys64/mingw64/git-core)
it seems that git ls-remote origin, run from a freshly-built and installed MinGW Git fails to be able to output anything, and git ls-remote origin | cat (a trick learned from working with old MSys'/MinGW's quirks) only says: output is not a tty (the exit code is 127, suggesting that some executable was not found, but it is very difficult to say which one because not even debug print statements to stderr are shown; It seems that in case of a crash or of a die(), stderr is not flushed)
we #define some S_* flags in mingw.h that mingw-w64's headers already define. This results in tons of compile warnings that might hide valid problems
MSys2's build recipe (called PKGBUILD; very similar to msysGit's home-brewn release.sh scripts) lists a dozen patches; We need to determine whether to reject them or whether to integrate them in our Git fork proper)
MSys2 builds Git with ./configure while our strategy has always been to enhance the configuration in config.mak.uname; We need to decide whether we want to follow the ./configure strategy or rather continue with our tried and tested approach
the mingw-w64-x86_64-gdb package is a 64-bit executable, but handles only 32-bit executables, it seems. This will make debugging substantially harder
we will also need to add support for the -f flag to MSys2's kill command (Note: MSys2's kill already knows the -f flag, but there is a Bash builtin that takes precedence when calling kill without an explicit path, and that kill does not know the -f flag)

I will keep updating this report as I find out more.

The text was updated successfully, but these errors were encountered:

dscho · 2015-02-13T10:54:36Z

Update: to rebuild MinGW curl, I had to import Daniel Sternberg's (i.e. the main cURL developer's) key into gpg, not into pacman via pacman-key.

In other words, I had to call gpg --keyserver hkp://pgp.mit.edu --recv-keys 78E11C6B279D5C91 before running makepkg-mingw -s in MINGW-packages' mingw-w64-curl/ directory. The symptom that this is needed:

==> Verifying source file signatures with gpg...
    curl-7.40.0.tar.bz2 ... FAILED (unknown public key 78E11C6B279D5C91)
==> ERROR: One or more PGP signatures could not be verified!

dscho · 2015-02-13T12:40:22Z

Another update, this time really dramatic:

git merge origin/msys2
merge: origin\msys2 - not something we can merge

In other words, MSys2 has an even worse path translation than MSys... 😦

dscho · 2015-02-13T14:59:21Z

Good news: the funny issue with git ls-remote goes away when recompiling myself. It seems that there was some problem with curl, maybe even caused by a version skew.

dscho · 2015-02-13T15:45:10Z

Sadly, there are also bad news: t0001 already fails with a Segmentation fault (!!!) when running git aliasedinit... It keeps being a challenging project!

oysteinkrog · 2015-02-14T14:12:49Z

I just want to let you know that I appreciate your work:)
I use MSYS2 myself and I agree that it is much better than old msys, the main problem for me is that the cygwin-based git is so slow.
If you get mingw-git working it would be awesome.
I tried to look at it myself but it is really very hard, especially if one is not all that experienced with native code and buildsystems.

dscho · 2015-02-15T09:58:49Z

merge: origin\msys2 - not something we can merge

This issue was probably caused by mingw_startup() not being called in the Git built from MSys2's MinGW Git build script. I will not investigate the issue more because it cannot be reproduced with the Git I built from scratch.

Given this insight, I decided that it would be better to stick with our config.mak.uname-based approach to configure the build (as opposed to the ./configure-based method proposed by MSys2's mingw-w64-git).

Sidenote: I do plan to offer a PR to MSys2's MINGW-packages repository once I completed my work successfully.

The hard crash I saw in t0001 (i.e. when running git aliasedinit) was caused by unsetenv()'s not knowing about our UTF-8 environment: when looking up an environment variable, our binary search would try to compare a string against NULL. I added a commit that errors out explicitly when NO_UNSETENV is defined and the mingw_startup() code is active at the same time.

dscho · 2015-02-15T17:05:16Z

Another set of updates: the two most crucial problems I addressed today are:

The TZ environment variable is stripped out by the MSys2 runtime when executing non-MSys2 programs, causing regressions in the test suite. While the rationale is sound (Win32 API treats TZ differently than POSIX), making the information unavailable is not a proper solution. My proposed fix renames the environment variable to MSYS2_TZ before passing it on to non-MSys2 programs (including Git).
The MSys2 POSIX <-> Windows conversion (p2w) is a complete rewrite of MSys1' p2w code, unfortunately re-introducing some bugs such as converting ~/file to ~C:\msys64\file. My current version of a fix most likely will evolve, but for the moment, we simply skip the conversion when a tilde, a quote or wildcards were found before the first slash or white space character.

The current version of my fixes can be seen here: https://github.com/dscho/Cygwin/branches

dscho · 2015-02-16T11:37:05Z

Update: as of last night, the following tests still fail: t0060, t1504, t1506, t1508, t1513, t3300, t3301, t3400, t3404, t3600, t3707, t3902, t4016, t4056, t4135, t4208, t4211, t5000, t5500, t5516, t5570, t5601, t5602, t5801, t6006, t6132, t7201, t7400, t7800, t9001, t9010, t9350, t9700, t9903 and t9300 waits forever.

While this is an impressive list of failures, the number of root causes is likely much smaller. I expect most of the issues to go away with a more careful p2w handling.

dscho · 2015-02-18T15:27:47Z

Progress: after skipping filenames-with-tabs tests explicitly (MSys2 allows you to create such files even if the Win32 API cannot handle them), and after working quite a bit on the posix-to-windows path mangling as well as the environment mangling of MSys2, the list of failing tests shrunk to t3703, t4056, t5000, t5500, t5516, t5570, t5601, t5602, t5801, t7400, t7800, t9001, t9010, t9350, t9700 and t9300 still has five test cases that wait for input on stdin by mistake.

The current patches can be seen here: https://github.com/dscho/Cygwin/compare/git-for-windows and dscho/git@git-for-windows...msys2 (to be cleaned up)

dscho · 2015-02-20T16:35:20Z

A ton of work later, t7800 and 9700 are the only tests that expose problems.

hvoigt · 2015-02-22T08:47:39Z

Wow, @dscho I just have to say: Your work is awesome!

dscho · 2015-02-22T13:34:16Z

t7800 down, t9700 to go.

dscho · 2015-02-22T16:11:13Z

Done. While at it, I also documented packaging a bit more, as well as started describing how to run and debug the regression tests.

dscho · 2015-02-22T20:01:25Z

Okay, it is done. I merged the fixes. Note: this is still only Git 2.1.2. I already performed a merging rebase (see the tentative-2.3.0 branch in my repository), but there are test failures. Those will have to wait for tomorrow, but this ticket can be closed already because I am really confident now that we need to switch to MSys2.

The virtual file system code incorrectly treated symlinks as directories instead of regular files. This meant symlinks were not included even if they are listed in the list of files returned by the core.virtualFilesystem hook proc. Fixes git-for-windows#25 Signed-off-by: Ben Peart <[email protected]>

It was recently reported that concurrent reads and writes may cause the reftable backend to segfault. The root cause of this is that we do not properly keep track of reftable readers across reloads. Suppose that you have a reftable iterator and then decide to reload the stack while iterating through the iterator. When the stack has been rewritten since we have created the iterator, then we would end up discarding a subset of readers that may still be in use by the iterator. The consequence is that we now try to reference deallocated memory, which of course segfaults. One way to trigger this is in t5616, where some background maintenance jobs have been leaking from one test into another. This leads to stack traces like the following one: + git -c protocol.version=0 -C pc1 fetch --filter=blob:limit=29999 --refetch origin AddressSanitizer:DEADLYSIGNAL ================================================================= ==657994==ERROR: AddressSanitizer: SEGV on unknown address 0x7fa0f0ec6089 (pc 0x55f23e52ddf9 bp 0x7ffe7bfa1700 sp 0x7ffe7bfa1700 T0) ==657994==The signal is caused by a READ memory access. #0 0x55f23e52ddf9 in get_var_int reftable/record.c:29 #1 0x55f23e53295e in reftable_decode_keylen reftable/record.c:170 #2 0x55f23e532cc0 in reftable_decode_key reftable/record.c:194 #3 0x55f23e54e72e in block_iter_next reftable/block.c:398 #4 0x55f23e5573dc in table_iter_next_in_block reftable/reader.c:240 #5 0x55f23e5573dc in table_iter_next reftable/reader.c:355 #6 0x55f23e5573dc in table_iter_next reftable/reader.c:339 #7 0x55f23e551283 in merged_iter_advance_subiter reftable/merged.c:69 #8 0x55f23e55169e in merged_iter_next_entry reftable/merged.c:123 #9 0x55f23e55169e in merged_iter_next_void reftable/merged.c:172 #10 0x55f23e537625 in reftable_iterator_next_ref reftable/generic.c:175 #11 0x55f23e2cf9c6 in reftable_ref_iterator_advance refs/reftable-backend.c:464 #12 0x55f23e2d996e in ref_iterator_advance refs/iterator.c:13 #13 0x55f23e2d996e in do_for_each_ref_iterator refs/iterator.c:452 #14 0x55f23dca6767 in get_ref_map builtin/fetch.c:623 #15 0x55f23dca6767 in do_fetch builtin/fetch.c:1659 #16 0x55f23dca6767 in fetch_one builtin/fetch.c:2133 #17 0x55f23dca6767 in cmd_fetch builtin/fetch.c:2432 #18 0x55f23dba7764 in run_builtin git.c:484 #19 0x55f23dba7764 in handle_builtin git.c:741 #20 0x55f23dbab61e in run_argv git.c:805 #21 0x55f23dbab61e in cmd_main git.c:1000 #22 0x55f23dba4781 in main common-main.c:64 #23 0x7fa0f063fc89 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 #24 0x7fa0f063fd44 in __libc_start_main_impl ../csu/libc-start.c:360 #25 0x55f23dba6ad0 in _start (git+0xadfad0) (BuildId: 803b2b7f59beb03d7849fb8294a8e2145dd4aa27) While it is somewhat awkward that the maintenance processes survive tests in the first place, it is totally expected that reftables should work alright with concurrent writers. Seemingly they don't. The only underlying resource that we need to care about in this context is the reftable reader, which is responsible for reading a single table from disk. These readers get discarded immediately (unless reused) when calling `reftable_stack_reload()`, which is wrong. We can only close them once we know that there are no iterators using them anymore. Prepare for a fix by converting the reftable readers to be refcounted. Reported-by: Jeff King <[email protected]> Signed-off-by: Patrick Steinhardt <[email protected]> Signed-off-by: Junio C Hamano <[email protected]>

dscho added the sdk label Feb 13, 2015

dscho self-assigned this Feb 13, 2015

dscho closed this as completed Feb 22, 2015

tkelman mentioned this issue Mar 22, 2015

bundle Julia with a better terminal on Windows? JuliaLang/julia#7267

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Look into MSys2 #25

Look into MSys2 #25

dscho commented Feb 13, 2015

dscho commented Feb 13, 2015

dscho commented Feb 13, 2015

dscho commented Feb 13, 2015

dscho commented Feb 13, 2015

oysteinkrog commented Feb 14, 2015

dscho commented Feb 15, 2015

dscho commented Feb 15, 2015

dscho commented Feb 16, 2015

dscho commented Feb 18, 2015

dscho commented Feb 20, 2015

hvoigt commented Feb 22, 2015

dscho commented Feb 22, 2015

dscho commented Feb 22, 2015

dscho commented Feb 22, 2015

Look into MSys2 #25

Look into MSys2 #25

Comments

dscho commented Feb 13, 2015

dscho commented Feb 13, 2015

dscho commented Feb 13, 2015

dscho commented Feb 13, 2015

dscho commented Feb 13, 2015

oysteinkrog commented Feb 14, 2015

dscho commented Feb 15, 2015

dscho commented Feb 15, 2015

dscho commented Feb 16, 2015

dscho commented Feb 18, 2015

dscho commented Feb 20, 2015

hvoigt commented Feb 22, 2015

dscho commented Feb 22, 2015

dscho commented Feb 22, 2015

dscho commented Feb 22, 2015