Use clang4 by default, add setting to opt into clang5 until clang 5 works on Chrome #93

jacobdufault · 2017-11-27T16:41:29Z

No description provided.

topisani · 2017-11-27T16:42:31Z

any idea why it doesnt work?

jacobdufault · 2017-11-27T16:43:15Z

Nope, completion works, but indexing fails for some reason. I'll have to do some debugging. I'm guessing there is a fatal error somewhere in clang.

topisani · 2017-11-27T16:43:53Z

and chrome compiles with clang 5?

jacobdufault · 2017-11-27T16:44:32Z

Yea, Chrome compiles with near-head clang.

elpixo · 2017-11-28T07:04:36Z

I've had some crashes during indexing with clang5 too (non-chrome code base) - everything rock solid on clang4. If you want some stack traces, let me know and I can add them here.

jacobdufault · 2017-11-28T09:22:37Z

Yea, that'd be great

elpixo · 2017-11-29T13:36:22Z

Not sure how useful this is;

(gdb) bt                                                                                                                                                                                                                         
#0  x86_64_fallback_frame_state (context=0x7f70f7ffc400, context=0x7f70f7ffc400, fs=0x7f70f7ffc4f0) at ./md-unwind-support.h:58                                                                                                  
#1  uw_frame_state_for (context=context@entry=0x7f70f7ffc400, fs=fs@entry=0x7f70f7ffc4f0) at /var/tmp/portage/sys-devel/gcc-5.4.0-r3/work/gcc-5.4.0/libgcc/unwind-dw2.c:1249                                                     
#2  0x0000003805410368 in _Unwind_Backtrace (trace=0x3803907e10 <backtrace_helper>, trace_argument=0x7f70f7ffc6b0) at /var/tmp/portage/sys-devel/gcc-5.4.0-r3/work/gcc-5.4.0/libgcc/unwind.inc:290                               
#3  0x0000003803907fe2 in __GI___backtrace (array=<optimized out>, size=<optimized out>) at ../sysdeps/x86_64/backtrace.c:110                                                                                                    
#4  0x000000000053c6cd in loguru::stacktrace_as_stdstring[abi:cxx11](int) (skip=4) at ../third_party/loguru/loguru.hpp:2185                                                                                                      
#5  0x000000000053ccdb in loguru::stacktrace (skip=<optimized out>) at ../third_party/loguru/loguru.hpp:2245                                                                                                                     
#6  loguru::log_message (stack_trace_skip=<optimized out>, message=..., with_indentation=false, abort_if_fatal=false) at ../third_party/loguru/loguru.hpp:2295                                                                   
#7  0x000000000053ef0d in loguru::signal_handler (signal_number=11) at ../third_party/loguru/loguru.hpp:2842                                                                                                                     
#8  <signal handler called>                                                                                                                                                                                                      
#9  x86_64_fallback_frame_state (context=0x7f70f7ffd7c0, context=0x7f70f7ffd7c0, fs=0x7f70f7ffd8b0) at ./md-unwind-support.h:58                                                                                                  
#10 uw_frame_state_for (context=context@entry=0x7f70f7ffd7c0, fs=fs@entry=0x7f70f7ffd8b0) at /var/tmp/portage/sys-devel/gcc-5.4.0-r3/work/gcc-5.4.0/libgcc/unwind-dw2.c:1249                                                     
#11 0x0000003805410368 in _Unwind_Backtrace (trace=0x3803907e10 <backtrace_helper>, trace_argument=0x7f70f7ffda70) at /var/tmp/portage/sys-devel/gcc-5.4.0-r3/work/gcc-5.4.0/libgcc/unwind.inc:290                               
#12 0x0000003803907fe2 in __GI___backtrace (array=<optimized out>, size=<optimized out>) at ../sysdeps/x86_64/backtrace.c:110                                                                                                    
#13 0x00007f7100d7d7ba in llvm::sys::PrintStackTrace(llvm::raw_ostream&) () from /usr/lib/llvm/5/lib64/../lib64/libLLVMSupport.so.5                                                                                              
#14 0x00007f7100d7b5ce in llvm::sys::RunSignalHandlers() () from /usr/lib/llvm/5/lib64/../lib64/libLLVMSupport.so.5                                                                                                                                                                    
#15 0x00007f7100d7b85c in SignalHandler(int) () from /usr/lib/llvm/5/lib64/../lib64/libLLVMSupport.so.5                                                                                                                                                                                
#16 <signal handler called>                                                                                                                                                                                                                                                            
#17 0x0000000000000171 in ?? ()                                                                                                                                                                                                                                                        
#18 0x00007f71023fc18c in clang::ASTUnit::~ASTUnit() () from /usr/lib/llvm/5/lib64/../lib64/libclangFrontend.so.5                                                                                                                                                                      
#19 0x00007f71023fc4b2 in llvm::CrashRecoveryContextDeleteCleanup<clang::ASTUnit>::recoverResources() () from /usr/lib/llvm/5/lib64/../lib64/libclangFrontend.so.5                                                                                                                     
#20 0x00007f7100cf1e89 in llvm::CrashRecoveryContext::~CrashRecoveryContext() () from /usr/lib/llvm/5/lib64/../lib64/libLLVMSupport.so.5                                                                                                                                               
#21 0x00007f7102d4f7a0 in clang_parseTranslationUnit2FullArgv () from /usr/lib/llvm/5/lib64/libclang.so.5                                                                                                                                                                              
#22 0x0000000000418562 in ClangTranslationUnit::Create (index=<optimized out>, filepath=..., arguments=..., unsaved_files=..., flags=<optimized out>) at ../src/clang_translation_unit.cc:29                                                                                           
#23 0x00000000004118b3 in (anonymous namespace)::TryEnsureDocumentParsed (manager=<optimized out>, session=<error reading variable: Cannot access memory at address 0x66>, tu=0x7f70f7ffee68, index=0x4b98d88) at ../src/clang_complete.cc:290                                         
#24 0x0000000000412f6d in (anonymous namespace)::CompletionParseMain (completion_manager=<optimized out>) at ../src/clang_complete.cc:339                                                                                                                                              
#25 ClangCompleteManager::ClangCompleteManager(Config*, Project*, WorkingFiles*, std::function<void (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, NonElidedVector<lsDiagnostic>)>, std::function<void (ClangTranslationUnit*, std::vector<CXUnsavedFile, std::allocator<CXUnsavedFile> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&)>)::$_1::operator()() const (this=<optimized out>) at ../src/clang_complete.cc:549                                                                                 
#26 std::_Bind_simple<ClangCompleteManager::ClangCompleteManager(Config*, Project*, WorkingFiles*, std::function<void (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, NonElidedVector<lsDiagnostic>)>, std::function<void (ClangTranslationUnit*, std::vector<CXUnsavedFile, std::allocator<CXUnsavedFile> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&)>)::$_1 ()>::_M_invoke<>(std::_Index_tuple<>) (this=<optimized out>) at /usr/lib/gcc/x86_64-pc-linux-gnu/5.4.0/include/g++-v5/functional:1530    
#27 std::_Bind_simple<ClangCompleteManager::ClangCompleteManager(Config*, Project*, WorkingFiles*, std::function<void (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, NonElidedVector<lsDiagnostic>)>, std::function<void (ClangTranslationUnit*, std::vector<CXUnsavedFile, std::allocator<CXUnsavedFile> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&)>)::$_1 ()>::operator()() (this=<optimized out>) at /usr/lib/gcc/x86_64-pc-linux-gnu/5.4.0/include/g++-v5/functional:1520                        
#28 std::thread::_Impl<std::_Bind_simple<ClangCompleteManager::ClangCompleteManager(Config*, Project*, WorkingFiles*, std::function<void (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, NonElidedVector<lsDiagnostic>)>, std::function<void (ClangTranslationUnit*, std::vector<CXUnsavedFile, std::allocator<CXUnsavedFile> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&)>)::$_1 ()> >::_M_run() (this=<optimized out>) at /usr/lib/gcc/x86_64-pc-linux-gnu/5.4.0/include/g++-v5/thread:115                                               
#29 0x00007f7102c03022 in std::execute_native_thread_routine (__p=0x610ea0) at /var/tmp/portage/sys-devel/gcc-5.4.0-r3/work/gcc-5.4.0/libstdc++-v3/src/c++11/thread.cc:84                                                                                                                                                                                       
#30 0x0000003804007637 in start_thread (arg=0x7f70f7fff700) at pthread_create.c:456                                                                                                                                                                                                                                                                             
#31 0x00000038038f8bbf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

This is from the core file dumped by ./app

I looked in the Chrome console and got a
[Extension Host] An error occurred while running command togglehs.toggleHS DISPOSED: TextEditor(vs.editor.ICodeEditor:1,$model17) has been disposed followed by a bunch of Language client is not ready yet

I'm using my system clang 5.0.0 RELEASE on Gentoo linux.

If there's some setting I need to set to get more useful information, please let me know & I'll keep trying.

So far today with Clang 5 - I've had about 15 of these where with Clang 4 it's never yet crashed.

jacobdufault · 2017-11-29T19:25:22Z

Ooh, based on that stack trace it looks like this is caused by the same issue as #43 (but that stack is originating from ClangCompleteManager, and iirc the crashes are from the indexer thread)

MaskRay · 2017-11-29T20:11:09Z

This may expose a different race condition in libclang when cquery calls clang_parseTranslationUnit2FullArgv.

When clang_parseTranslationUnit2FullArgv is called, libclang.so will create a new thread.

In @elpixo 's comment we can only see the libclang thread, not the cquery "indexer" thread spawning the libclang thread. New thread inherits pthread_setname_np(3) of the old thread, and they should have the same name (say "indexer7"). Can you confirm that the parent thread was calling clang_parseTranslationUnit2FullArgv?

If the problem is easy to reproduce, you may gdb -p after starting the cquery build/app process. Then you can do more interactive operations when the thread dies (likely std::terminate())

jacobdufault · 2017-11-30T04:29:05Z

I played around with serializing clang_parseTranslationUnit2FullArgv and it does not seem to fix the issue - there seem to be fewer crashes the the index is still empty. See 473b827 and e73f749.

MaskRay · 2017-11-30T04:42:09Z

Do you mean in 473b827 , some indexes (.json files I suppose) may be empty? But there is no deadlock in RegisterTarget and lookupTarget?

+  {
 -      clang_defaultReparseOptions(tu->cx_tu));		 +    std::lock_guard<std::mutex> lock(g_reparse_translation_unit_mutex);
 +    error_code = clang_reparseTranslationUnit(
 +        tu->cx_tu, (unsigned)unsaved.size(), unsaved.data(),
 +        clang_defaultReparseOptions(tu->cx_tu));
 +  }

jacobdufault · 2017-11-30T05:02:20Z

Yep - but I don't recall seeing deadlocks either way. I'm planning on testing clang --emit-ast via command line tonight or tomorrow.

jacobdufault · 2017-12-01T05:58:41Z

I don't think we will ever switch to clang 5 by default, the indexing API seems to be buggy. clang trunk works though, so when clang 6 becomes stable we can switch to that.

Got this backtrace when indexing this file.

Thread 58 "indexer29" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fcad57fa700 (LWP 46669)]
0x00007fcbbd69f160 in clang::Decl::castFromDeclContext(clang::DeclContext const*) ()
   from /usr/local/google/home/jdufault/cquery-wip/build/clang+llvm-5.0.0-x86_64-linux-gnu-debian8/lib/libclang.so.5
(gdb) bt
#0  0x00007fcbbd69f160 in clang::Decl::castFromDeclContext(clang::DeclContext const*) ()
   from /usr/local/google/home/jdufault/cquery-wip/build/clang+llvm-5.0.0-x86_64-linux-gnu-debian8/lib/libclang.so.5
#1  0x00007fcbbd62629a in clang::cxindex::CXIndexDataConsumer::handleReference(clang::NamedDecl const*, clang::SourceLocation, CXCursor, clang::NamedDecl const*, clang::DeclContext const*, clang::Expr const*, CXIdxEntityRefKind) ()
   from /usr/local/google/home/jdufault/cquery-wip/build/clang+llvm-5.0.0-x86_64-linux-gnu-debian8/lib/libclang.so.5
#2  0x00007fcbbd625ae1 in clang::cxindex::CXIndexDataConsumer::handleDeclOccurence(clang::Decl const*, unsigned int, llvm::ArrayRef<clang::index::SymbolRelation>, clang::FileID, unsigned int, clang::index::IndexDataConsumer::ASTNodeInfo) ()
   from /usr/local/google/home/jdufault/cquery-wip/build/clang+llvm-5.0.0-x86_64-linux-gnu-debian8/lib/libclang.so.5
#3  0x00007fcbbd904e65 in clang::index::IndexingContext::handleDeclOccurrence(clang::Decl const*, clang::SourceLocation, bool, clang::Decl const*, unsigned int, llvm::ArrayRef<clang::index::SymbolRelation>, clang::Expr const*, clang::Decl const*, clang::DeclContext const*
) () from /usr/local/google/home/jdufault/cquery-wip/build/clang+llvm-5.0.0-x86_64-linux-gnu-debian8/lib/libclang.so.5
#4  0x00007fcbbd905186 in clang::index::IndexingContext::handleReference(clang::NamedDecl const*, clang::SourceLocation, clang::NamedDecl const*, clang::DeclContext const*, unsigned int, llvm::ArrayRef<clang::index::SymbolRelation>, clang::Expr const*, clang::Decl const*)
 () from /usr/local/google/home/jdufault/cquery-wip/build/clang+llvm-5.0.0-x86_64-linux-gnu-debian8/lib/libclang.so.5
#5  0x00007fcbbd903d60 in (anonymous namespace)::IndexingDeclVisitor::handleTemplateArgumentLoc(clang::TemplateArgumentLoc const&, clang::NamedDecl const*, clang::DeclContext const*) ()
   from /usr/local/google/home/jdufault/cquery-wip/build/clang+llvm-5.0.0-x86_64-linux-gnu-debian8/lib/libclang.so.5
#6  0x00007fcbbd903c79 in (anonymous namespace)::IndexingDeclVisitor::VisitTemplateDecl(clang::TemplateDecl const*) ()
   from /usr/local/google/home/jdufault/cquery-wip/build/clang+llvm-5.0.0-x86_64-linux-gnu-debian8/lib/libclang.so.5
#7  0x00007fcbbd901959 in clang::index::IndexingContext::indexDecl(clang::Decl const*) ()
   from /usr/local/google/home/jdufault/cquery-wip/build/clang+llvm-5.0.0-x86_64-linux-gnu-debian8/lib/libclang.so.5
#8  0x00007fcbbd901b89 in clang::declvisitor::Base<clang::declvisitor::make_const_ptr, (anonymous namespace)::IndexingDeclVisitor, bool>::Visit () from /usr/local/google/home/jdufault/cquery-wip/build/clang+llvm-5.0.0-x86_64-linux-gnu-debian8/lib/libclang.so.5
#9  0x00007fcbbd901959 in clang::index::IndexingContext::indexDecl(clang::Decl const*) ()
   from /usr/local/google/home/jdufault/cquery-wip/build/clang+llvm-5.0.0-x86_64-linux-gnu-debian8/lib/libclang.so.5
#10 0x00007fcbbd901b89 in clang::declvisitor::Base<clang::declvisitor::make_const_ptr, (anonymous namespace)::IndexingDeclVisitor, bool>::Visit () from /usr/local/google/home/jdufault/cquery-wip/build/clang+llvm-5.0.0-x86_64-linux-gnu-debian8/lib/libclang.so.5
#11 0x00007fcbbd901959 in clang::index::IndexingContext::indexDecl(clang::Decl const*) ()
   from /usr/local/google/home/jdufault/cquery-wip/build/clang+llvm-5.0.0-x86_64-linux-gnu-debian8/lib/libclang.so.5
#12 0x00007fcbbd85722a in clang::ASTUnit::visitLocalTopLevelDecls(void*, bool (*)(void*, clang::Decl const*)) ()
   from /usr/local/google/home/jdufault/cquery-wip/build/clang+llvm-5.0.0-x86_64-linux-gnu-debian8/lib/libclang.so.5
#13 0x00007fcbbd8fc89a in clang::index::indexASTUnit(clang::ASTUnit&, std::shared_ptr<clang::index::IndexDataConsumer>, clang::index::IndexingOptions) () from /usr/local/google/home/jdufault/cquery-wip/build/clang+llvm-5.0.0-x86_64-linux-gnu-debian8/lib/libclang.so.5
---Type <return> to continue, or q <return> to quit---q

MaskRay · 2017-12-08T02:23:23Z

Reverse engineering is fun.

tools/clang/tools/libclang/CXIndexDataConsumer.cpp:935

bool CXIndexDataConsumer::handleReference(const NamedDecl *D, SourceLocation Loc,
                                      CXCursor Cursor,
                                      const NamedDecl *Parent,
                                      const DeclContext *DC,
                                      const Expr *E,
                                      CXIdxEntityRefKind Kind) {

935:  getContainerInfo(DC, Container);

void CXIndexDataConsumer::getContainerInfo(const DeclContext *DC,
                                       ContainerInfo &ContInfo) {
  ContInfo.cursor = getCursor(cast<Decl>(DC));
  ContInfo.DC = DC;
  ContInfo.IndexCtx = this;
}

`cast<Decl>(DC)` calls `Decl::castFromDeclContext`

tools/clang/lib/AST/DeclBase.cpp:832

Decl *Decl::castFromDeclContext (const DeclContext *D) {
  Decl::Kind DK = D->getDeclKind();
  switch(DK) {

while here D is a null pointer.

llvm 5.0.1-rc3 has been tagged and the final release will be next week.

MaskRay · 2017-12-08T02:29:30Z

It would be helpful if you could a compile an llvm with debug info

Here is the command I use to build llvm.

# https://llvm.org/docs/GettingStarted.html#git-mirror
% git clone https://git.llvm.org/git/llvm.git
% cd llvm/tools
% git clone https://git.llvm.org/git/clang.git

# cd back to llvm/

% mkdir build; cd build
% cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -G Ninja -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_C_COMPILER=clang -DCMAKE_ASM_COMPILER=clang -DCMAKE_EXE_LINKER_FLAGS=-fuse-ld=gold -DLLVM_USE_SPLIT_DWARF=On -DBUILD_SHARED_LIBS=ON -DLLVM_OPTIMIZED_TABLEGEN=On -DLLVM_ENABLE_ASSERTIONS=On -DLLVM_ENABLE_RTTI=On ..
% ninja

# lib/ is what we want

MaskRay · 2017-12-18T07:01:49Z

--bundled-clang=5.0.0 has been supported but it is not recommended (see wiki for details)

jacobdufault mentioned this issue Nov 27, 2017

Upgrade bundled llvm+clang to 5.0.0 #88

Merged

MaskRay mentioned this issue Nov 27, 2017

waf configure --bundled-clang=5.0.0 to use prebuilt clang+llvm 5.0.0,… #95

Merged

MaskRay closed this as completed Dec 18, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Use clang4 by default, add setting to opt into clang5 until clang 5 works on Chrome #93

Use clang4 by default, add setting to opt into clang5 until clang 5 works on Chrome #93

jacobdufault commented Nov 27, 2017

topisani commented Nov 27, 2017

Uh oh!

jacobdufault commented Nov 27, 2017

Uh oh!

topisani commented Nov 27, 2017

Uh oh!

jacobdufault commented Nov 27, 2017

Uh oh!

elpixo commented Nov 28, 2017

Uh oh!

jacobdufault commented Nov 28, 2017

Uh oh!

elpixo commented Nov 29, 2017 •

edited

Loading

Uh oh!

jacobdufault commented Nov 29, 2017 •

edited

Loading

Uh oh!

MaskRay commented Nov 29, 2017 •

edited

Loading

Uh oh!

jacobdufault commented Nov 30, 2017

Uh oh!

MaskRay commented Nov 30, 2017

Uh oh!

jacobdufault commented Nov 30, 2017 •

edited

Loading

Uh oh!

jacobdufault commented Dec 1, 2017 •

edited

Loading

Uh oh!

MaskRay commented Dec 8, 2017 •

edited

Loading

Uh oh!

MaskRay commented Dec 8, 2017 •

edited

Loading

Uh oh!

MaskRay commented Dec 18, 2017

Uh oh!

Use clang4 by default, add setting to opt into clang5 until clang 5 works on Chrome #93

Use clang4 by default, add setting to opt into clang5 until clang 5 works on Chrome #93

Comments

jacobdufault commented Nov 27, 2017

topisani commented Nov 27, 2017

Uh oh!

jacobdufault commented Nov 27, 2017

Uh oh!

topisani commented Nov 27, 2017

Uh oh!

jacobdufault commented Nov 27, 2017

Uh oh!

elpixo commented Nov 28, 2017

Uh oh!

jacobdufault commented Nov 28, 2017

Uh oh!

elpixo commented Nov 29, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jacobdufault commented Nov 29, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MaskRay commented Nov 29, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jacobdufault commented Nov 30, 2017

Uh oh!

MaskRay commented Nov 30, 2017

Uh oh!

jacobdufault commented Nov 30, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jacobdufault commented Dec 1, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MaskRay commented Dec 8, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

cast<Decl>(DC) calls Decl::castFromDeclContext

Uh oh!

MaskRay commented Dec 8, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MaskRay commented Dec 18, 2017

Uh oh!

elpixo commented Nov 29, 2017 •

edited

Loading

jacobdufault commented Nov 29, 2017 •

edited

Loading

MaskRay commented Nov 29, 2017 •

edited

Loading

jacobdufault commented Nov 30, 2017 •

edited

Loading

jacobdufault commented Dec 1, 2017 •

edited

Loading

MaskRay commented Dec 8, 2017 •

edited

Loading

`cast<Decl>(DC)` calls `Decl::castFromDeclContext`

MaskRay commented Dec 8, 2017 •

edited

Loading