Crashing during sparse-checkout operations

[aaronberger-msft]

• I was not able to find an open
  or closed issue matching
  what I'm seeing, including in the git-for-windows/git tracker.

Setup

• Which version of microsoft/git are you using? Is it 32-bit or 64-bit?

2.41.0.vfs.0.5 64 Bit

Are you using Scalar or VFS for Git?

Scalar

If VFS for Git, then what version?

$ gvfs version

** insert your machine's response here **

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

$ cmd.exe /c ver
This is manifesting on a few different machines but they should all be close to this version.
Microsoft Windows [Version 10.0.23541.1000]

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

No

Details

4 users reporting crashes while running sparse checkout operations. I assume more user's have also encountered this but not with enough frequency to merit making noise about it.

The users mention that the crashes have started within the last month or so. They seem to be mostly random and don’t immediately repro if trying to re-run the same command that originally failed.

Is there further information around the scenarios or other info I/users can provide that is helpful to look into this?

One user provided a dump collected at time of crash, I've provided some details from it here.
Exception Info:

CONTEXT:  (.ecxr)
rax=00000000c0d2c425 rbx=0000036c600b82a8 rcx=0000000000000000
rdx=0000036c61fe0420 rsi=00000018ad7fe140 rdi=0000036c601aa5f0
rip=0000000000000000 rsp=00000018ad7fe0c8 rbp=0000036c7e6a65b4
r8=00000018ad7fe140  r9=0000036c7e6a65b4 r10=0000000000000012
r11=8101010101010100 r12=00000018ad7fe430 r13=0000036c7e6a65b4
r14=000000000000000c r15=0000036c601aa5f0
iopl=0         nv up ei pl zr na po nc
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00010244
00000000`00000000 ??              ???
Resetting default scope

EXCEPTION_RECORD:  (.exr -1)
ExceptionAddress: 0000000000000000
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 0000000000000008
   Parameter[1]: 0000000000000000
Attempt to execute non-executable address 0000000000000000

PROCESS_NAME:  git.exe

EXECUTE_ADDRESS: 0

FAILED_INSTRUCTION_ADDRESS: 
+0
ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.

EXCEPTION_CODE_STR:  c0000005

EXCEPTION_PARAMETER1:  0000000000000008

EXCEPTION_PARAMETER2:  0000000000000000

Stack from main thread:

git.exe!hashmap_get() Line 100
              at hashmap.c(100)
git.exe!hash_dir_entry() Line 45
              at name-hash.c(45)
git.exe!hash_index_entry() Line 92
              at name-hash.c(92)
git.exe!add_path_to_index() Line 162
              at strbuf.h(162)
git.exe!read_tree_at() Line 42
              at tree.c(42)
git.exe!expand_index() Line 397
              at sparse-index.c(397)
git.exe!index_name_stage_pos() Line 549
              at read-cache.c(549)
git.exe!index_name_pos() Line 589
              at read-cache.c(589)
git.exe!add_patterns() Line 927
              at dir.c(927)
git.exe!prep_exclude() Line 1753
              at dir.c(1753)
git.exe!last_matching_pattern() Line 1781
              at dir.c(1781)
git.exe!is_excluded() Line 1809
              at dir.c(1809)
git.exe!treat_path() Line 2422
              at dir.c(2422)
git.exe!read_directory_recursive() Line 2671
              at dir.c(2671)
git.exe!read_directory() Line 3104
              at dir.c(3104)
git.exe!fill_directory() Line 243
              at dir.c(243)
git.exe!update_working_directory() Line 189
              at builtin\sparse-checkout.c(189)
git.exe!write_patterns_and_update() Line 345
              at builtin\sparse-checkout.c(345)
git.exe!modify_pattern_list() Line 695
              at builtin\sparse-checkout.c(695)
git.exe!sparse_checkout_set() Line 854
              at builtin\sparse-checkout.c(854)
git.exe!cmd_sparse_checkout() Line 1056
              at builtin\sparse-checkout.c(1056)
git.exe!handle_builtin() Line 540
              at git.c(540)
git.exe!cmd_main() Line 803
              at git.c(803)
git.exe!main() Line 84
              at common-main.c(84)
git.exe!wmain() Line 4058
              at compat\mingw.c(4058)
git.exe!__tmainCRTStartup() Line 268
              at C:\M\B\src\mingw-w64\mingw-w64-crt\crt\crtexe.c(268)
git.exe!mainCRTStartup() Line 190
              at C:\M\B\src\mingw-w64\mingw-w64-crt\crt\crtexe.c(190)
kernel32.dll!BaseThreadInitThunk(unsigned long RunProcessInit, long(*)(void *) StartAddress, void * Argument) Line 75
              at clientcore\base\win32\client\thread.c(75)
ntdll.dll!RtlUserThreadStart(long(*)(void *) StartAddress, void * Argument) Line 1166
              at minkernel\ntdll\rtlstrt.c(1166)

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

CMD

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

git sparse-checkout  set
(have also seen crashes from sparse-checkout reapply)

• What did you expect to occur after running these commands?

No crashes

• What actually happened instead?

Experienced a crash, but not consistently

• If the problem was occurring with a specific repository, can you specify
  the repository?

  • Public repo: insert URL here
  • Windows monorepo
  • Office monorepo
  • Other Microsoft-internal repo: insert name here
  • Other internal repo.

[vdye]

Hello, thanks for the report! I have a couple of follow-up questions to hopefully narrow down the root cause:

• Has this issue only been occurring in v2.41.0.vfs.0.5? If so, which version did the affected users have installed previously?
• The call stack indicates that the error happened in git sparse-checkout set, but later the "CMD" section indicates that it happens in git sparse-checkout apply. Is this crash happening in multiple subcommands of sparse-checkout, or is it only one of them (with one being a typo)?

As a final note: if other users encounter this issue, any additional exception/call stack info they could provide would be extremely helpful to our investigation of this bug.

[aaronberger-msft]

Thanks for the prompt response!

• Has this issue only been occurring in v2.41.0.vfs.0.5? If so, which version did the affected users have installed previously?

I only know of it happening with v2.41.0.vfs.0.5, the first report of it was on 2023-08-30 and user specified they were on the latest version at that point. The callstack and exception info I provided is from a different user that explicitly called out 2.41.0.vfs.0.5.

• The call stack indicates that the error happened in git sparse-checkout set, but later the "CMD" section indicates that it happens in git sparse-checkout apply. Is this crash happening in multiple subcommands of sparse-checkout, or is it only one of them (with one being a typo)?

Sorry, had one report during re-apply, but this stack was during set, I have edited the original text to reflect this

As a final note: if other users encounter this issue, any additional exception/call stack info they could provide would be extremely helpful to our investigation of this bug.

I will point users at this issue to share further info.

[tiagomacarios]

@vdye What would be the best option for us to move forward with this? Could you perhaps leave Teams open for a couple of days so we can engage you when we have a "live" repro?

We are seeing multiple people hitting this. The only artificial repro we have is something like:

git sparse-checkout add folderA
mkdir folderA/subfolderB
git sparse-checkout set folderC

It looks like the empty folder causes the crash.

[vdye]

We are seeing multiple people hitting this. The only artificial repro we have is something like:

git sparse-checkout add folderA
mkdir folderA/subfolderB
git sparse-checkout set folderC

It looks like the empty folder causes the crash.

Thanks, this artificial repro is extremely helpful. With a working test case, we should be able to make better progress on finding a root cause. We should be able to give you an update sometime in the next couple days!

@vdye What would be the best option for us to move forward with this? Could you perhaps leave Teams open for a couple of days so we can engage you when we have a "live" repro?

If for some reason the artificial repro isn't enough to track down the bug (although I strongly suspect it will), I or someone else on my team will follow up here to work out what to do next.

CC: @jeffhostetler

[vdye]

We were able to figure out the root cause and merged a fix in #607. I'll publish a new release with the fix shortly. Thanks for your patience!