Extension process using too much memory in the filename cache

[bobbrow]

@sean-mcmanus Yes, it's happens during code write, I get (cpptools:.... )"Connection to server got closed. Server will not be restarted", but I still get this message. And in task explorer I see how "Microsoft.VSCode.CPP.IntelliSense.Msvc.exe" allocate memory and then it up to 4G is closed.
Reproducing steps is not clear. But rollback to 0.26.1 helps until restart the VCCode (I can mistake, need more investigation).
When I got clear steps, I will inform you. I'll try to get the "Microsoft.VSCode.CPP.IntelliSense.Msvc.exe" dump during crush.

My configuration is still the same:
{
"compileCommands": ".../compile_commands.json",
"compilerPath": ".../14.16.27023/bin/Hostx64/x64/cl.exe",
"intelliSenseMode": "msvc-x64",
"cppStandard": "c++17",
"cStandard": "c11",
"defines": [.....],
"includePath": [....].
"browse": {
"limitSymbolsToIncludedHeaders": true
}
}

Originally posted by @Gargony in #3270 (comment)

[bobbrow]

@Gargony replied:

@sean-mcmanus may be this help

WER/CrashAPI:2655: ERROR Invalid args, too  big block
WER/CrashAPI:2655: ERROR Invalid args, too  big block
WER/CrashAPI:2655: ERROR Invalid args, too  big block
WER/CrashAPI:2655: ERROR Invalid args, too  big block
(1460.88ec): C++ EH exception - code e06d7363 (first chance)
WER/CrashAPI:2655: ERROR Invalid args, too  big block
WER/CrashAPI:2655: ERROR Invalid args, too  big block
WER/CrashAPI:2655: ERROR Invalid args, too  big block
WER/CrashAPI:2655: ERROR Invalid args, too  big block
WER/CrashAPI:2655: ERROR Invalid args, too  big block
WER/CrashAPI:2655: ERROR Invalid args, too  big block
WER/CrashAPI:2655: ERROR Invalid args, too  big block
WER/CrashAPI:2655: ERROR Invalid args, too  big block
WER/CrashAPI:2655: ERROR Invalid args, too  big block
WER/CrashAPI:2655: ERROR Invalid args, too  big block
WER/CrashAPI:2655: ERROR Invalid args, too  big block
WER/CrashAPI:2655: ERROR Invalid args, too  big block
WER/CrashAPI:2655: ERROR Invalid args, too  big block
WER/CrashAPI:2655: ERROR Invalid args, too  big block
WER/CrashAPI:2655: ERROR Invalid args, too  big block
WER/CrashAPI:2655: ERROR Invalid args, too  big block
WER/CrashAPI:2655: ERROR Invalid args, too  big block
WER/CrashAPI:2655: ERROR Invalid args, too  big block
WER/CrashAPI:2655: ERROR Invalid args, too  big block
(1460.7e10): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=062cc044 ebx=0340fb88 ecx=0340fb08 edx=00000000 esi=7fffffff edi=0340fae0
eip=0046d46d esp=0340f90c ebp=0340f978 iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010246
Microsoft_VSCode_CPP_IntelliSense_Msvc!edge::scoped_query_manager::check_and_compile+0x31f:
0046d46d 0fb64008        movzx   eax,byte ptr [eax+8]       ds:002b:062cc04c=??

[bobbrow]

@sean-mcmanus replied:

@Gargony We fixed a high hitting crash with 0.26.0.1 involving "check_and_compile"...if you have 0.26.1 or later, then you could be hitting a less commonly hitting crash with that method. We might need an isolated/sample repro to fix that.

Is the memory usage related to the crash or are they separate issues? What causes the memory to get to 4 GB? Just opening a file? Or does it accumulate as the cursor is moved around the file? Or only after edits?

[Gargony]

@bobbrow I got crash with memory up to 4 GB on Microsoft.VSCode.CPP.Extension.exe (0.26.2.2)
After that the Microsoft.VSCode.CPP.Extension.exe restarted and all repeat again.

Microsoft_VSCode_CPP_Extension!__scrt_throw_std_bad_alloc+0x1f
0x382f000
Microsoft_VSCode_CPP_Extension!std::_Allocate_manually_vector_aligned<std::_Default_allocate_traits>+0x17
Microsoft_VSCode_CPP_Extension!std::_Allocate<8,std::_Default_allocate_traits,0>+0x13
Microsoft_VSCode_CPP_Extension!std::allocator<std::_List_unchecked_const_iterator<std::_List_val<std::_List_simple_types<filename_entry *> >,std::_Iterator_base0> >::allocate+0x1c
Microsoft_VSCode_CPP_Extension!std::vector<std::_List_unchecked_const_iterator<std::_List_val<std::_List_simple_types<filename_entry *> >,std::_Iterator_base0>,std::allocator<std::_List_unchecked_const_iterator<std::_List_val<std::_List_simple_types<filename_entry *> >,std::_Iterator_base0> > >::_Reallocate_exactly+0x19
Microsoft_VSCode_CPP_Extension!std::_Hash<std::_Uset_traits<filename_entry *,std::_Uhash_compare<filename_entry *,std::hash<filename_entry *>,std::equal_to<filename_entry *> >,std::allocator<filename_entry *>,0> >::_Init+0x1d
Microsoft_VSCode_CPP_Extension!std::_Hash<std::_Uset_traits<filename_entry *,std::_Uhash_compare<filename_entry *,std::hash<filename_entry *>,std::equal_to<filename_entry *> >,std::allocator<filename_entry *>,0> >::_Check_size+0x39
Microsoft_VSCode_CPP_Extension!std::_Hash<std::_Uset_traits<filename_entry *,std::_Uhash_compare<filename_entry *,std::hash<filename_entry *>,std::equal_to<filename_entry *> >,std::allocator<filename_entry *>,0> >::_Insert<filename_entry *,std::_Not_a_node_tag>+0xa7
Microsoft_VSCode_CPP_Extension!browse_engine::populate_filename_cache+0x7a8
Microsoft_VSCode_CPP_Extension!browse_engine::query_translation_unit_source+0x141
Microsoft_VSCode_CPP_Extension!create_sync_work+0x13f
Microsoft_VSCode_CPP_Extension!create_async_work+0x9d
Microsoft_VSCode_CPP_Extension!<lambda_22a3a4f55eafaf5e1ed922229c820ce6>::operator()+0xf9
Microsoft_VSCode_CPP_Extension!std::_Func_impl_no_alloc<<lambda_66f0a2a1d5b8c10b4ef2a449fd6e2e9e>,void>::_Do_call+0xf
Microsoft_VSCode_CPP_Extension!msvc::thread_pool::do_work+0x149
Microsoft_VSCode_CPP_Extension!std::_LaunchPad<std::unique_ptr<std::tuple<<lambda_a3c66ca3d4c2719be92bdb8d8b1f4c0e> >,std::default_delete<std::tuple<<lambda_a3c66ca3d4c2719be92bdb8d8b1f4c0e> > > > >::_Go+0x26
Microsoft_VSCode_CPP_Extension!std::_Pad::_Call_func+0xa
Microsoft_VSCode_CPP_Extension!thread_start<unsigned int (__stdcall*)(void *)>+0x58
KERNEL32!BaseThreadInitThunk+0x24
ntdll!__RtlUserThreadStart+0x2f
ntdll!_RtlUserThreadStart+0x1b

[Colengms]

Hi @Gargony . Do you have a huge number of files in your project? It looks like you're running out of memory (on a 32-bit system?) while building a table of header/source relationships from data in the tag parser database.

[Gargony]

Hi @Colengms . I'm not sure about border huge and less files ))) We have a mobile application about ~1K+ cpp files. boost, stl, templates like this https://github.com/DevUtilsNet/linqcpp/blob/master/include/linqcpp/linqcpp.h and etc. Nothing special, usual project ))) Please tell me what information I can provide to you? I can provide compile_commands.json (3Mb+), build.ninja (4Mb+) or somphing like this.
"ipch" folder (3Gb+)
FYI: command "C/C++: Reset IntelliSense Database" helps in short terms

[sean-mcmanus]

Does the bug re-appear after doing the Reset IntelliSense Database, i.e. does it seem like the database was in a bad state due to a previously fixed issue?

[Gargony]

@sean-mcmanus Yes, the bug re-appear after I doing the Reset IntelliSense Database.
Ok, I'll remove folder "ipch" and let you know it fixed or not.

[Gargony]

@sean-mcmanus after clear "ipch" the process Microsoft.VSCode.CPP.Extension.exe works and after about ~3h is began crush ("ipch" ~830Mb)

[Colengms]

Hi @Gargony .

The specific allocation that failed is an association between a header and a file that includes it. This information is used to select an appropriate source file when creating an IntelliSense Translation Unit, as a header file is opened. It's possible that a very large number of #include statements (not necessarily files) could lead to excessive memory usage. Running out of memory here is surprising. It sounds like we may need to rethink how this is implemented.

Do you consistently see the same stack (within populate_filename_cache) ? If you see any other stacks associated with the issue, they may provide some clues.

In the meanwhile, I will see what we can do to reduce the overhead involved in includer/includee associations.

[Gargony]

Hi @Colengms the stack is the same
please look at dump, maybe it help.
Microsoft.VSCode.CPP.Extension.exe.dmp.zip