Jim sync #1

jimwwalker · 2014-07-24T16:03:43Z

Some changes to sync-with-meta.

Moved logging to stderr and we're now much quieter if nothing is happening.

Fixed up usage to include source/dest IPs and list of keys

Tool to synchronize a document *and* it's metadata between two clusters. Example usage: sync-with-meta.py --verbose 192.168.10.1 192.168.10.2 key_to_sync key_to_sync2 Add --force to overwrite existing documents on the destination. Change-Id: I61dd0e825baf029373f64af3993973926e8a95a7

Change-Id: Ic7ac39164f7d2d57476a1a459cc1ab9f58ca1a9f

Change-Id: I69e1b91dbd0f022336c03ab67f032e4b5648540a

Requires new option --allow-source-changes Change-Id: Icad7802f0da05120fbf3dc9052c4bcfc1b507520

Change-Id: I66c7153077a7c0e3b3eabe3082ba30ce923cfff7

Change-Id: Iefe365ef453760b04e06b894f25001fcd3fe8217

Change-Id: Ic239cb3a7be486bf56707b7f0ed2b5be27cec8c3

Change-Id: I269fd2a21c04b58d58427b34a59a6c73b51b8c56

Change-Id: Ibac54d0dfcdac37e1c98d901abae5cc1ed6db078

Change-Id: I078aade4d556a5d541b9393bf2e8436f57f409dc

Change-Id: I001cd5d434b4ba564269d6c68e3d7630375ca5d3

Change-Id: I1a303d296913012e2a6887fd4d070b8976fb5d2c

Change-Id: I9e5f3d2dad67b2bdeeb507e365f355b05e4816f1

jimwwalker · 2014-07-24T16:14:44Z

wrong repo :) - closed

As identified by ThreadSanitizer: WARNING: ThreadSanitizer: data race (pid=17259) Write of size 8 at 0x7fec86e44de0 by main thread (mutexes: write M1161): #0 gmtime ??:0 (libtsan.so.0+0x000000025135) #1 EventuallyPersistentEngine::doEngineStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:3369 (ep.so+0x00000010f4be) #2 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:4339 (ep.so+0x000000113c35) #3 EvpGetStats /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:217 (ep.so+0x000000102b14) #4 mock_get_stats /repos/couchbase/server/source/memcached/programs/engine_testapp/engine_testapp.c:195 (exe+0x0000000026de) #5 get_int_stat(engine_interface*, engine_interface_v1*, char const*, char const*) /repos/couchbase/server/source/ep-engine/tests/ep_test_apis.cc:799 (ep_testsuite.so+0x0000000832d8) #6 wait_for_stat_change(engine_interface*, engine_interface_v1*, char const*, int, char const*) /repos/couchbase/server/source/ep-engine/tests/ep_test_apis.cc:846 (ep_testsuite.so+0x0000000838d6) #7 test_setup /repos/couchbase/server/source/ep-engine/tests/ep_testsuite.cc:178 (ep_testsuite.so+0x00000001ee69) #8 execute_test /repos/couchbase/server/source/memcached/programs/engine_testapp/engine_testapp.c:1050 (exe+0x000000005970) #9 main /repos/couchbase/server/source/memcached/programs/engine_testapp/engine_testapp.c:1313 (exe+0x000000006606) Previous write of size 8 at 0x7fec86e44de0 by thread T7: #0 gmtime ??:0 (libtsan.so.0+0x000000025135) #1 EventuallyPersistentEngine::doEngineStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:3369 (ep.so+0x00000010f4be) #2 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:4339 (ep.so+0x000000113c35) #3 EventuallyPersistentStore::snapshotStats() /repos/couchbase/server/source/ep-engine/src/ep.cc:1465 (ep.so+0x0000000e150e) #4 StatSnap::run() /repos/couchbase/server/source/ep-engine/src/tasks.cc:79 (ep.so+0x000000174db2) #5 ExecutorThread::run() /repos/couchbase/server/source/ep-engine/src/executorthread.cc:110 (ep.so+0x00000014a0e7) #6 launch_executor_thread /repos/couchbase/server/source/ep-engine/src/executorthread.cc:34 (ep.so+0x000000149930) #7 platform_thread_wrap /repos/couchbase/server/source/platform/src/cb_pthreads.c:19 (libplatform.so.0.1.0+0x000000002d8b) #8 __tsan_write_range ??:0 (libtsan.so.0+0x00000001b1c9) Switch to gmtime_r which is thread-safe. Change-Id: Id0773df65f4fc569c0a173b6185b1ef8bd91862d Reviewed-on: http://review.couchbase.org/43312 Reviewed-by: Chiyoung Seo <[email protected]> Tested-by: Chiyoung Seo <[email protected]>

Fix race as identified by ThreadSanitizer: WARNING: ThreadSanitizer: data race (pid=4243) Read of size 8 at 0x7d04000fde60 by main thread (mutexes: write M1367): #0 std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> >::operator++() /usr/include/c++/4.8/bits/stl_list.h:235 (ep.so+0x00000013a129) #1 std::iterator_traits<std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> > >::difference_type std::__distance<std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> > >(std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> >, std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> >, std::input_iterator_tag) /usr/include/c++/4.8/bits/stl_iterator_base_funcs.h:82 (ep.so+0x000000138b67) #2 std::iterator_traits<std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> > >::difference_type std::distance<std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> > >(std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> >, std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> >) /usr/include/c++/4.8/bits/stl_iterator_base_funcs.h:118 (ep.so+0x000000136b63) #3 std::list<SingleThreadedRCPtr<GlobalTask>, std::allocator<SingleThreadedRCPtr<GlobalTask> > >::size() const /usr/include/c++/4.8/bits/stl_list.h:874 (ep.so+0x000000135538) #4 ExecutorPool::doTaskQStat(EventuallyPersistentEngine*, void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /repos/couchbase/server/source/ep-engine/src/executorpool.cc:654 (ep.so+0x0000001331b6) #5 EventuallyPersistentEngine::doWorkloadStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:4198 (ep.so+0x000000112ed3) #6 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:4454 (ep.so+0x000000114cb4) #7 EvpGetStats /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:217 (ep.so+0x000000102b14) #8 mock_get_stats /repos/couchbase/server/source/memcached/programs/engine_testapp/engine_testapp.c:195 (exe+0x0000000026de) #9 test_workload_stats /repos/couchbase/server/source/ep-engine/tests/ep_testsuite.cc:7094 (ep_testsuite.so+0x00000004e931) #10 execute_test /repos/couchbase/server/source/memcached/programs/engine_testapp/engine_testapp.c:1055 (exe+0x0000000059fc) #11 main /repos/couchbase/server/source/memcached/programs/engine_testapp/engine_testapp.c:1313 (exe+0x000000006606) Previous write of size 8 at 0x7d04000fde60 by thread T5 (mutexes: write M45): #0 RCValue::_rc_decref() const /repos/couchbase/server/source/ep-engine/src/atomic.h:293 (ep.so+0x000000096797) #1 __gnu_cxx::new_allocator<std::_List_node<SingleThreadedRCPtr<GlobalTask> > >::allocate(unsigned long, void const*) /usr/include/c++/4.8/ext/new_allocator.h:104 (ep.so+0x00000017b42b) #2 std::_List_base<SingleThreadedRCPtr<GlobalTask>, std::allocator<SingleThreadedRCPtr<GlobalTask> > >::_M_get_node() /usr/include/c++/4.8/bits/stl_list.h:334 (ep.so+0x00000017b0dc) #3 std::_List_node<SingleThreadedRCPtr<GlobalTask> >* std::list<SingleThreadedRCPtr<GlobalTask>, std::allocator<SingleThreadedRCPtr<GlobalTask> > >::_M_create_node<SingleThreadedRCPtr<GlobalTask> const&>(SingleThreadedRCPtr<GlobalTask> const&) /usr/include/c++/4.8/bits/stl_list.h:502 (ep.so+0x00000017a25a) #4 void std::list<SingleThreadedRCPtr<GlobalTask>, std::allocator<SingleThreadedRCPtr<GlobalTask> > >::_M_insert<SingleThreadedRCPtr<GlobalTask> const&>(std::_List_iterator<SingleThreadedRCPtr<GlobalTask> >, SingleThreadedRCPtr<GlobalTask> const&) /usr/include/c++/4.8/bits/stl_list.h:1561 (ep.so+0x000000178e7b) #5 std::list<SingleThreadedRCPtr<GlobalTask>, std::allocator<SingleThreadedRCPtr<GlobalTask> > >::push_back(SingleThreadedRCPtr<GlobalTask> const&) /usr/include/c++/4.8/bits/stl_list.h:1016 (ep.so+0x00000017817b) #6 TaskQueue::_fetchNextTask(ExecutorThread&, bool) /repos/couchbase/server/source/ep-engine/src/taskqueue.cc:126 (ep.so+0x000000176932) #7 TaskQueue::fetchNextTask(ExecutorThread&, bool) /repos/couchbase/server/source/ep-engine/src/taskqueue.cc:142 (ep.so+0x000000176acf) #8 ExecutorPool::_nextTask(ExecutorThread&, unsigned char) /repos/couchbase/server/source/ep-engine/src/executorpool.cc:214 (ep.so+0x000000130707) #9 ExecutorPool::nextTask(ExecutorThread&, unsigned char) /repos/couchbase/server/source/ep-engine/src/executorpool.cc:229 (ep.so+0x0000001307a5) #10 ExecutorThread::run() /repos/couchbase/server/source/ep-engine/src/executorthread.cc:78 (ep.so+0x000000149da0) #11 launch_executor_thread /repos/couchbase/server/source/ep-engine/src/executorthread.cc:34 (ep.so+0x00000014990a) #12 platform_thread_wrap /repos/couchbase/server/source/platform/src/cb_pthreads.c:19 (libplatform.so.0.1.0+0x000000002d8b) #13 __tsan_write_range ??:0 (libtsan.so.0+0x00000001b1c9) Fix by adding ew helper methods which will return the size of the various queues (while holding the TaskQueue's mutex). Change-Id: If5e8d357e45803d78c4ba6ed1475e6e1a90e1c89 Reviewed-on: http://review.couchbase.org/43313 Reviewed-by: Chiyoung Seo <[email protected]> Tested-by: Chiyoung Seo <[email protected]>

As identified by ThreadSantizer: WARNING: ThreadSanitizer: data race (pid=4009) Write of size 8 at 0x7d440000c5b0 by thread T6 (mutexes: write M55): #0 KVStatsCallback::callback(KVStatsCtx&) /repos/couchbase/server/source/ep-engine/src/ep.cc:933 (ep.so+0x0000000f0a22) #1 CouchKVStore::commit2couchstore(Callback<KVStatsCtx>*, unsigned long, unsigned long) /repos/couchbase/server/source/ep-engine/src/couch-kvstore/couch-kvstore.cc:1697 (ep.so+0x0000001aa8c6) #2 CouchKVStore::commit(Callback<KVStatsCtx>*, unsigned long, unsigned long) /repos/couchbase/server/source/ep-engine/src/couch-kvstore/couch-kvstore.cc:1040 (ep.so+0x0000001a6483) #3 EventuallyPersistentStore::flushVBucket(unsigned short) /repos/couchbase/server/source/ep-engine/src/ep.cc:2909 (ep.so+0x0000000e780b) #4 Flusher::flushVB() /repos/couchbase/server/source/ep-engine/src/flusher.cc:283 (ep.so+0x00000013f363) #5 Flusher::step(GlobalTask*) /repos/couchbase/server/source/ep-engine/src/flusher.cc:174 (ep.so+0x00000013e9c8) #6 FlusherTask::run() /repos/couchbase/server/source/ep-engine/src/tasks.cc:44 (ep.so+0x000000174a85) #7 ExecutorThread::run() /repos/couchbase/server/source/ep-engine/src/executorthread.cc:110 (ep.so+0x00000014a0c1) #8 launch_executor_thread /repos/couchbase/server/source/ep-engine/src/executorthread.cc:34 (ep.so+0x00000014990a) #9 platform_thread_wrap /repos/couchbase/server/source/platform/src/cb_pthreads.c:19 (libplatform.so.0.1.0+0x000000002d8b) #10 __tsan_write_range ??:0 (libtsan.so.0+0x00000001b1c9) Previous read of size 8 at 0x7d440000c5b0 by main thread (mutexes: write M193510842443017784): #0 VBucketCountVisitor::visitBucket(RCPtr<VBucket>&) /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:2889 (ep.so+0x00000010c631) #1 VBucketCountAggregator::visitBucket(RCPtr<VBucket>&) /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:2926 (ep.so+0x000000121392) #2 EventuallyPersistentStore::visit(VBucketVisitor&) /repos/couchbase/server/source/ep-engine/src/ep.cc:3278 (ep.so+0x0000000e99d9) #3 EventuallyPersistentEngine::doEngineStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:2955 (ep.so+0x00000010cb46) #4 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:4344 (ep.so+0x000000113c0f) #5 EvpGetStats /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:217 (ep.so+0x000000102b14) #6 mock_get_stats /repos/couchbase/server/source/memcached/programs/engine_testapp/engine_testapp.c:195 (exe+0x0000000026de) #7 get_int_stat(engine_interface*, engine_interface_v1*, char const*, char const*) /repos/couchbase/server/source/ep-engine/tests/ep_test_apis.cc:799 (ep_testsuite.so+0x0000000832d8) #8 wait_for_flusher_to_settle(engine_interface*, engine_interface_v1*) /repos/couchbase/server/source/ep-engine/tests/ep_test_apis.cc:900 (ep_testsuite.so+0x000000083cd4) #9 wait_for_persisted_value(engine_interface*, engine_interface_v1*, char const*, char const*, unsigned short) /repos/couchbase/server/source/ep-engine/tests/ep_test_apis.cc:917 (ep_testsuite.so+0x000000083de5) #10 test_io_stats /repos/couchbase/server/source/ep-engine/tests/ep_testsuite.cc:6279 (ep_testsuite.so+0x0000000468b3) #11 execute_test /repos/couchbase/server/source/memcached/programs/engine_testapp/engine_testapp.c:1055 (exe+0x0000000059fc) #12 main /repos/couchbase/server/source/memcached/programs/engine_testapp/engine_testapp.c:1313 (exe+0x000000006606) Change to AtomicValues to fix this. Change-Id: Ie7f9a403f809e751ff1802cceb5d2a77a483a586 Reviewed-on: http://review.couchbase.org/43314 Reviewed-by: Chiyoung Seo <[email protected]> Tested-by: Chiyoung Seo <[email protected]>

Fix race identified by ThreadSanitizer: WARNING: ThreadSanitizer: data race (pid=10664) Read of size 4 at 0x7d4400009888 by main thread (mutexes: write M18321): #0 Flusher::stateName() const /home/vagrant/couchbase-server/ep-engine/src/flusher.cc:119 (ep.so+0x0000001fc22e) #1 EventuallyPersistentEngine::doEngineStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/vagrant/couchbase-server/ep-engine/src/ep_engine.cc:3025 (ep.so+0x000000180649) #2 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/vagrant/couchbase-server/ep-engine/src/ep_engine.cc:4374 (ep.so+0x00000018d960) #3 EvpGetStats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/vagrant/couchbase-server/ep-engine/src/ep_engine.cc:214 (ep.so+0x000000174062) #4 mock_get_stats /home/vagrant/couchbase-server/memcached/programs/engine_testapp/engine_testapp.c:196 (exe+0x0000000a741d) #5 get_int_stat(engine_interface*, engine_interface_v1*, char const*, char const*) /home/vagrant/couchbase-server/ep-engine/tests/ep_test_apis.cc:799 (ep_testsuite.so+0x0000000dae46) #6 verify_curr_items(engine_interface*, engine_interface_v1*, int, char const*) /home/vagrant/couchbase-server/ep-engine/tests/ep_test_apis.cc:834 (ep_testsuite.so+0x0000000e1d31) #7 test_dcp_producer_stream_req_disk(engine_interface*, engine_interface_v1*) /home/vagrant/couchbase-server/ep-engine/tests/ep_testsuite.cc:3589 (ep_testsuite.so+0x000000094c9a) #8 execute_test /home/vagrant/couchbase-server/memcached/programs/engine_testapp/engine_testapp.c:1055 (exe+0x0000000a3e83) #9 main /home/vagrant/couchbase-server/memcached/programs/engine_testapp/engine_testapp.c:1313 (exe+0x0000000a1d84) Previous write of size 4 at 0x7d4400009888 by thread T5: #0 Flusher::transition_state(flusher_state) /home/vagrant/couchbase-server/ep-engine/src/flusher.cc:114 (ep.so+0x0000001fb7df) #1 Flusher::step(GlobalTask*) /home/vagrant/couchbase-server/ep-engine/src/flusher.cc:167 (ep.so+0x0000001fc9d1) #2 FlusherTask::run() /home/vagrant/couchbase-server/ep-engine/src/tasks.cc:44 (ep.so+0x00000027870e) #3 ExecutorThread::run() /home/vagrant/couchbase-server/ep-engine/src/executorthread.cc:110 (ep.so+0x0000002160ff) #4 launch_executor_thread(void*) /home/vagrant/couchbase-server/ep-engine/src/executorthread.cc:34 (ep.so+0x00000021560a) #5 platform_thread_wrap /home/vagrant/couchbase-server/platform/src/cb_pthreads.c:19 (libplatform.so.0.1.0+0x0000000033f4) Change-Id: Iaeb60efdc1032de7ba344e04cce454cc1d876d40 Reviewed-on: http://review.couchbase.org/43315 Reviewed-by: Chiyoung Seo <[email protected]> Tested-by: Chiyoung Seo <[email protected]>

Race identified by ThreadSanitizer: Read of size 1 at 0x7d480000b458 by main thread (mutexes: write M18384, write M898, write M18351): #0 ActiveStream::addStats(void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/vagrant/couchbase-server/ep-engine/src/dcp-stream.cc:386 (ep.so+0x0000000ff1a9) #1 DcpProducer::addStats(void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/vagrant/couchbase-server/ep-engine/src/dcp-producer.cc:502 (ep.so+0x0000000efcd3) #2 ConnStatBuilder::operator()(SingleThreadedRCPtr<ConnHandler>&) /home/vagrant/couchbase-server/ep-engine/src/ep_engine.cc:3725 (ep.so+0x0000001ba97d) #3 ConnStatBuilder std::for_each<std::_List_iterator<SingleThreadedRCPtr<ConnHandler> >, ConnStatBuilder>(std::_List_iterator<SingleThreadedRCPtr<ConnHandler> >, std::_List_iterator<SingleThreadedRCPtr<ConnHandler> >, ConnStatBuilder) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/bits/stl_algo.h:4417 (ep.so+0x0000001ba5b5) #4 void ConnMap::each_UNLOCKED<ConnStatBuilder>(ConnStatBuilder) /home/vagrant/couchbase-server/ep-engine/src/connmap.h:148 (ep.so+0x0000001ba519) #5 void ConnMap::each<ConnStatBuilder>(ConnStatBuilder) /home/vagrant/couchbase-server/ep-engine/src/connmap.h:140 (ep.so+0x0000001b2c4e) #6 EventuallyPersistentEngine::doDcpStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/vagrant/couchbase-server/ep-engine/src/ep_engine.cc:3978 (ep.so+0x00000018928e) #7 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/vagrant/couchbase-server/ep-engine/src/ep_engine.cc:4384 (ep.so+0x00000018dc73) #8 EvpGetStats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/vagrant/couchbase-server/ep-engine/src/ep_engine.cc:214 (ep.so+0x000000174062) #9 mock_get_stats /home/vagrant/couchbase-server/memcached/programs/engine_testapp/engine_testapp.c:196 (exe+0x0000000a741d) #10 get_int_stat(engine_interface*, engine_interface_v1*, char const*, char const*) /home/vagrant/couchbase-server/ep-engine/tests/ep_test_apis.cc:799 (ep_testsuite.so+0x0000000dae46) #11 dcp_stream(engine_interface*, engine_interface_v1*, char const*, void const*, unsigned short, unsigned int, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, int, int, int, int, int, bool) /home/vagrant/couchbase-server/ep-engine/tests/ep_testsuite.cc:3365 (ep_testsuite.so+0x0000000afc65) #12 test_dcp_producer_stream_req_disk(engine_interface*, engine_interface_v1*) /home/vagrant/couchbase-server/ep-engine/tests/ep_testsuite.cc:3596 (ep_testsuite.so+0x000000094ed0) #13 execute_test /home/vagrant/couchbase-server/memcached/programs/engine_testapp/engine_testapp.c:1055 (exe+0x0000000a3e83) #14 main /home/vagrant/couchbase-server/memcached/programs/engine_testapp/engine_testapp.c:1313 (exe+0x0000000a1d84) Previous write of size 1 at 0x7d480000b458 by thread T10 (mutexes: write M18363, write M18357): #0 ActiveStream::markDiskSnapshot(unsigned long, unsigned long) /home/vagrant/couchbase-server/ep-engine/src/dcp-stream.cc:202 (ep.so+0x0000000fd0c7) #1 DCPBackfill::create() /home/vagrant/couchbase-server/ep-engine/src/dcp-backfill.cc:138 (ep.so+0x0000000d057c) #2 DCPBackfill::run() /home/vagrant/couchbase-server/ep-engine/src/dcp-backfill.cc:95 (ep.so+0x0000000cfe3b) #3 BackfillManager::backfill() /home/vagrant/couchbase-server/ep-engine/src/dcp-backfill-manager.cc:171 (ep.so+0x0000000c778d) #4 BackfillManagerTask::run() /home/vagrant/couchbase-server/ep-engine/src/dcp-backfill-manager.cc:50 (ep.so+0x0000000c745a) #5 ExecutorThread::run() /home/vagrant/couchbase-server/ep-engine/src/executorthread.cc:110 (ep.so+0x00000021646f) #6 launch_executor_thread(void*) /home/vagrant/couchbase-server/ep-engine/src/executorthread.cc:34 (ep.so+0x00000021597a) #7 platform_thread_wrap /home/vagrant/couchbase-server/platform/src/cb_pthreads.c:19 (libplatform.so.0.1.0+0x0000000033f4) Change-Id: I02cdf26c697f559a0afde91b377f1462c482349c Reviewed-on: http://review.couchbase.org/43316 Reviewed-by: Chiyoung Seo <[email protected]> Tested-by: Chiyoung Seo <[email protected]>

Race identified by ThreadSanitizer: Read of size 8 at 0x7d480000b518 by main thread (mutexes: write M18375, write M898, write M18336): #0 void STATWRITER_NAMESPACE::add_casted_stat<unsigned long>(char const*, unsigned long const&, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/vagrant/couchbase-server/ep-engine/src/statwriter.h:45 (ep.so+0x0000000aacf9) #1 ActiveStream::addStats(void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/vagrant/couchbase-server/ep-engine/src/dcp-stream.cc:378 (ep.so+0x0000000fee1e) #2 DcpProducer::addStats(void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/vagrant/couchbase-server/ep-engine/src/dcp-producer.cc:502 (ep.so+0x0000000efcd3) #3 ConnStatBuilder::operator()(SingleThreadedRCPtr<ConnHandler>&) /home/vagrant/couchbase-server/ep-engine/src/ep_engine.cc:3725 (ep.so+0x0000001ba92d) #4 ConnStatBuilder std::for_each<std::_List_iterator<SingleThreadedRCPtr<ConnHandler> >, ConnStatBuilder>(std::_List_iterator<SingleThreadedRCPtr<ConnHandler> >, std::_List_iterator<SingleThreadedRCPtr<ConnHandler> >, ConnStatBuilder) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/bits/stl_algo.h:4417 (ep.so+0x0000001ba565) #5 void ConnMap::each_UNLOCKED<ConnStatBuilder>(ConnStatBuilder) /home/vagrant/couchbase-server/ep-engine/src/connmap.h:148 (ep.so+0x0000001ba4c9) #6 void ConnMap::each<ConnStatBuilder>(ConnStatBuilder) /home/vagrant/couchbase-server/ep-engine/src/connmap.h:140 (ep.so+0x0000001b2bfe) #7 EventuallyPersistentEngine::doDcpStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/vagrant/couchbase-server/ep-engine/src/ep_engine.cc:3978 (ep.so+0x00000018923e) #8 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/vagrant/couchbase-server/ep-engine/src/ep_engine.cc:4384 (ep.so+0x00000018dc23) #9 EvpGetStats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/vagrant/couchbase-server/ep-engine/src/ep_engine.cc:214 (ep.so+0x000000174012) #10 mock_get_stats /home/vagrant/couchbase-server/memcached/programs/engine_testapp/engine_testapp.c:196 (exe+0x0000000a741d) #11 get_ull_stat(engine_interface*, engine_interface_v1*, char const*, char const*) /home/vagrant/couchbase-server/ep-engine/tests/ep_test_apis.cc:817 (ep_testsuite.so+0x0000000e1aa6) #12 dcp_stream(engine_interface*, engine_interface_v1*, char const*, void const*, unsigned short, unsigned int, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, int, int, int, int, int, bool) /home/vagrant/couchbase-server/ep-engine/tests/ep_testsuite.cc:3370 (ep_testsuite.so+0x0000000afd38) #13 test_dcp_producer_stream_req_disk(engine_interface*, engine_interface_v1*) /home/vagrant/couchbase-server/ep-engine/tests/ep_testsuite.cc:3596 (ep_testsuite.so+0x000000094ed0) #14 execute_test /home/vagrant/couchbase-server/memcached/programs/engine_testapp/engine_testapp.c:1055 (exe+0x0000000a3e83) #15 main /home/vagrant/couchbase-server/memcached/programs/engine_testapp/engine_testapp.c:1313 (exe+0x0000000a1d84) Previous write of size 8 at 0x7d480000b518 by thread T10 (mutexes: write M276545986342477096, write M18342): #0 ActiveStream::backfillReceived(Item*, backfill_source_t) /home/vagrant/couchbase-server/ep-engine/src/dcp-stream.cc:229 (ep.so+0x0000000fd6ab) #1 CacheCallback::callback(CacheLookup&) /home/vagrant/couchbase-server/ep-engine/src/dcp-backfill.cc:58 (ep.so+0x0000000cf5cc) #2 CouchKVStore::recordDbDump(_db*, _docinfo*, void*) /home/vagrant/couchbase-server/ep-engine/src/couch-kvstore/couch-kvstore.cc:1577 (ep.so+0x0000002f3fe7) #3 recordDbDumpC(_db*, _docinfo*, void*) /home/vagrant/couchbase-server/ep-engine/src/couch-kvstore/couch-kvstore.cc:74 (ep.so+0x0000002f2b70) #4 lookup_callback(couchfile_lookup_request*, _sized_buf const*, _sized_buf const*) /home/vagrant/couchbase-server/couchstore/src/couch_db.cc:729 (libcouchstore.so+0x000000019e46) #5 btree_lookup_inner(couchfile_lookup_request*, unsigned long, int, int) /home/vagrant/couchbase-server/couchstore/src/btree_read.cc:101 (libcouchstore.so+0x0000000132d7) #6 btree_lookup_inner(couchfile_lookup_request*, unsigned long, int, int) /home/vagrant/couchbase-server/couchstore/src/btree_read.cc:71 (libcouchstore.so+0x000000012bd3) #7 btree_lookup /home/vagrant/couchbase-server/couchstore/src/btree_read.cc:133 (libcouchstore.so+0x00000001236b) #8 couchstore_changes_since /home/vagrant/couchbase-server/couchstore/src/couch_db.cc:774 (libcouchstore.so+0x0000000196ca) #9 CouchKVStore::scan(ScanContext*) /home/vagrant/couchbase-server/ep-engine/src/couch-kvstore/couch-kvstore.cc:1227 (ep.so+0x0000002f2868) #10 DCPBackfill::scan() /home/vagrant/couchbase-server/ep-engine/src/dcp-backfill.cc:150 (ep.so+0x0000000d08ea) #11 DCPBackfill::run() /home/vagrant/couchbase-server/ep-engine/src/dcp-backfill.cc:97 (ep.so+0x0000000cfeb3) #12 BackfillManager::backfill() /home/vagrant/couchbase-server/ep-engine/src/dcp-backfill-manager.cc:171 (ep.so+0x0000000c778d) #13 BackfillManagerTask::run() /home/vagrant/couchbase-server/ep-engine/src/dcp-backfill-manager.cc:50 (ep.so+0x0000000c745a) #14 ExecutorThread::run() /home/vagrant/couchbase-server/ep-engine/src/executorthread.cc:110 (ep.so+0x00000021641f) #15 launch_executor_thread(void*) /home/vagrant/couchbase-server/ep-engine/src/executorthread.cc:34 (ep.so+0x00000021592a) #16 platform_thread_wrap /home/vagrant/couchbase-server/platform/src/cb_pthreads.c:19 (libplatform.so.0.1.0+0x0000000033f4) Change-Id: I330979be66ea5113f28846f664af70559379f1ae Reviewed-on: http://review.couchbase.org/43317 Reviewed-by: Chiyoung Seo <[email protected]> Tested-by: Chiyoung Seo <[email protected]>

Race identified by ThreadSanitizer: Read of size 8 at 0x7d08000c1430 by thread T10: #0 ConnNotifier::notifyConnections() /home/vagrant/couchbase-server/ep-engine/src/connmap.cc:127 (ep.so+0x0000002537e8) #1 ConnNotifierCallback::run() /home/vagrant/couchbase-server/ep-engine/src/connmap.cc:80 (ep.so+0x000000277e8b) #2 ExecutorThread::run() /home/vagrant/couchbase-server/ep-engine/src/executorthread.cc:110 (ep.so+0x0000002163af) #3 launch_executor_thread(void*) /home/vagrant/couchbase-server/ep-engine/src/executorthread.cc:34 (ep.so+0x0000002158ba) #4 platform_thread_wrap /home/vagrant/couchbase-server/platform/src/cb_pthreads.c:19 (libplatform.so.0.1.0+0x0000000033f4) Previous write of size 8 at 0x7d08000c1430 by main thread: #0 ConnNotifier::start() /home/vagrant/couchbase-server/ep-engine/src/connmap.cc:99 (ep.so+0x00000025336f) #1 ConnMap::initialize(conn_notifier_type) /home/vagrant/couchbase-server/ep-engine/src/connmap.cc:194 (ep.so+0x0000002544a6) #2 EventuallyPersistentEngine::initialize(char const*) /home/vagrant/couchbase-server/ep-engine/src/ep_engine.cc:2060 (ep.so+0x000000179b4a) #3 EvpInitialize(engine_interface*, char const*) /home/vagrant/couchbase-server/ep-engine/src/ep_engine.cc:135 (ep.so+0x000000173675) #4 init_engine /home/vagrant/couchbase-server/memcached/utilities/engine_loader.c:116 (libmcd_util.so.1.0.0+0x000000003fac) #5 start_your_engines /home/vagrant/couchbase-server/memcached/programs/engine_testapp/engine_testapp.c:913 (exe+0x0000000a2fb5) #6 execute_test /home/vagrant/couchbase-server/memcached/programs/engine_testapp/engine_testapp.c:1048 (exe+0x0000000a3d29) #7 main /home/vagrant/couchbase-server/memcached/programs/engine_testapp/engine_testapp.c:1313 (exe+0x0000000a1d84) Change-Id: I16cfdff1ea363bbb07a62a92f09f829483276b3d Reviewed-on: http://review.couchbase.org/43318 Reviewed-by: Chiyoung Seo <[email protected]> Tested-by: Chiyoung Seo <[email protected]>

Race identified by ThreadSanitizer: Read of size 4 at 0x7d480000b554 by main thread (mutexes: write M18379, write M899, write M18337): #0 void STATWRITER_NAMESPACE::add_casted_stat<unsigned int>(char const*, unsigned int const&, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/vagrant/couchbase-server/ep-engine/src/statwriter.h:45 (ep.so+0x000000107639) #1 ActiveStream::addStats(void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/vagrant/couchbase-server/ep-engine/src/dcp-stream.cc:388 (ep.so+0x0000000ff272) #2 DcpProducer::addStats(void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/vagrant/couchbase-server/ep-engine/src/dcp-producer.cc:502 (ep.so+0x0000000efcd3) #3 ConnStatBuilder::operator()(SingleThreadedRCPtr<ConnHandler>&) /home/vagrant/couchbase-server/ep-engine/src/ep_engine.cc:3725 (ep.so+0x0000001ba8bd) #4 ConnStatBuilder std::for_each<std::_List_iterator<SingleThreadedRCPtr<ConnHandler> >, ConnStatBuilder>(std::_List_iterator<SingleThreadedRCPtr<ConnHandler> >, std::_List_iterator<SingleThreadedRCPtr<ConnHandler> >, ConnStatBuilder) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/bits/stl_algo.h:4417 (ep.so+0x0000001ba4f5) #5 void ConnMap::each_UNLOCKED<ConnStatBuilder>(ConnStatBuilder) /home/vagrant/couchbase-server/ep-engine/src/connmap.h:148 (ep.so+0x0000001ba459) #6 void ConnMap::each<ConnStatBuilder>(ConnStatBuilder) /home/vagrant/couchbase-server/ep-engine/src/connmap.h:140 (ep.so+0x0000001b2b8e) #7 EventuallyPersistentEngine::doDcpStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/vagrant/couchbase-server/ep-engine/src/ep_engine.cc:3978 (ep.so+0x0000001891ce) #8 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/vagrant/couchbase-server/ep-engine/src/ep_engine.cc:4384 (ep.so+0x00000018dbb3) #9 EvpGetStats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/vagrant/couchbase-server/ep-engine/src/ep_engine.cc:214 (ep.so+0x000000173fa2) #10 mock_get_stats /home/vagrant/couchbase-server/memcached/programs/engine_testapp/engine_testapp.c:196 (exe+0x0000000a741d) #11 get_ull_stat(engine_interface*, engine_interface_v1*, char const*, char const*) /home/vagrant/couchbase-server/ep-engine/tests/ep_test_apis.cc:817 (ep_testsuite.so+0x0000000e1aa6) #12 dcp_stream(engine_interface*, engine_interface_v1*, char const*, void const*, unsigned short, unsigned int, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, int, int, int, int, int, bool) /home/vagrant/couchbase-server/ep-engine/tests/ep_testsuite.cc:3370 (ep_testsuite.so+0x0000000afd38) #13 test_dcp_producer_stream_req_disk(engine_interface*, engine_interface_v1*) /home/vagrant/couchbase-server/ep-engine/tests/ep_testsuite.cc:3596 (ep_testsuite.so+0x000000094ed0) #14 execute_test /home/vagrant/couchbase-server/memcached/programs/engine_testapp/engine_testapp.c:1055 (exe+0x0000000a3e83) #15 main /home/vagrant/couchbase-server/memcached/programs/engine_testapp/engine_testapp.c:1313 (exe+0x0000000a1d84) Previous write of size 4 at 0x7d480000b554 by thread T10 (mutexes: write M277108936295898408, write M18343): #0 ActiveStream::backfillReceived(Item*, backfill_source_t) /home/vagrant/couchbase-server/ep-engine/src/dcp-stream.cc:216 (ep.so+0x0000000fd3d3) #1 CacheCallback::callback(CacheLookup&) /home/vagrant/couchbase-server/ep-engine/src/dcp-backfill.cc:58 (ep.so+0x0000000cf5cc) #2 CouchKVStore::recordDbDump(_db*, _docinfo*, void*) /home/vagrant/couchbase-server/ep-engine/src/couch-kvstore/couch-kvstore.cc:1577 (ep.so+0x0000002f3f77) #3 recordDbDumpC(_db*, _docinfo*, void*) /home/vagrant/couchbase-server/ep-engine/src/couch-kvstore/couch-kvstore.cc:74 (ep.so+0x0000002f2b00) #4 lookup_callback(couchfile_lookup_request*, _sized_buf const*, _sized_buf const*) /home/vagrant/couchbase-server/couchstore/src/couch_db.cc:729 (libcouchstore.so+0x000000019e46) #5 btree_lookup_inner(couchfile_lookup_request*, unsigned long, int, int) /home/vagrant/couchbase-server/couchstore/src/btree_read.cc:101 (libcouchstore.so+0x0000000132d7) #6 btree_lookup_inner(couchfile_lookup_request*, unsigned long, int, int) /home/vagrant/couchbase-server/couchstore/src/btree_read.cc:71 (libcouchstore.so+0x000000012bd3) #7 btree_lookup /home/vagrant/couchbase-server/couchstore/src/btree_read.cc:133 (libcouchstore.so+0x00000001236b) #8 couchstore_changes_since /home/vagrant/couchbase-server/couchstore/src/couch_db.cc:774 (libcouchstore.so+0x0000000196ca) #9 CouchKVStore::scan(ScanContext*) /home/vagrant/couchbase-server/ep-engine/src/couch-kvstore/couch-kvstore.cc:1227 (ep.so+0x0000002f27f8) #10 DCPBackfill::scan() /home/vagrant/couchbase-server/ep-engine/src/dcp-backfill.cc:150 (ep.so+0x0000000d08ea) #11 DCPBackfill::run() /home/vagrant/couchbase-server/ep-engine/src/dcp-backfill.cc:97 (ep.so+0x0000000cfeb3) #12 BackfillManager::backfill() /home/vagrant/couchbase-server/ep-engine/src/dcp-backfill-manager.cc:171 (ep.so+0x0000000c778d) #13 BackfillManagerTask::run() /home/vagrant/couchbase-server/ep-engine/src/dcp-backfill-manager.cc:50 (ep.so+0x0000000c745a) #14 ExecutorThread::run() /home/vagrant/couchbase-server/ep-engine/src/executorthread.cc:110 (ep.so+0x0000002163af) #15 launch_executor_thread(void*) /home/vagrant/couchbase-server/ep-engine/src/executorthread.cc:34 (ep.so+0x0000002158ba) #16 platform_thread_wrap /home/vagrant/couchbase-server/platform/src/cb_pthreads.c:19 (libplatform.so.0.1.0+0x0000000033f4) Change-Id: Ia80928489df9b57b64b463c67fafee5ba15d2aac Reviewed-on: http://review.couchbase.org/43319 Reviewed-by: Chiyoung Seo <[email protected]> Tested-by: Chiyoung Seo <[email protected]>

Identified by ThreadSanitizer running ep_perfsuite.so: WARNING: ThreadSanitizer: data race (pid=51118) Atomic write of size 1 at 0x7d4400009d58 by main thread (mutexes: write M5599): #0 __tsan_atomic8_compare_exchange_val <null>:0 (engine_testapp+0x000000093f50) #1 CouchbaseAtomic<bool>::compare_exchange_strong(bool&, bool, memory_order) /root/couchbase-3.0/ep-engine/src/atomic.h:92 (ep.so+0x00000005575d) #2 Flusher::notifyFlushEvent() /root/couchbase-3.0/ep-engine/src/flusher.h:88 (ep.so+0x0000000c6fec) #3 EventuallyPersistentStore::queueDirty(RCPtr<VBucket>&, StoredValue*, LockHolder*, bool, bool, bool) /root/couchbase-3.0/ep-engine/src/ep.cc:2826 (ep.so+0x00000009c796) #4 EventuallyPersistentStore::add(Item const&, void const*) /root/couchbase-3.0/ep-engine/src/ep.cc:728 (ep.so+0x00000009f673) #5 EventuallyPersistentEngine::store(void const*, void*, unsigned long*, ENGINE_STORE_OPERATION, unsigned short) /root/couchbase-3.0/ep-engine/src/ep_engine.cc:2135 (ep.so+0x000000100980) #6 EvpStore(engine_interface*, void const*, void*, unsigned long*, ENGINE_STORE_OPERATION, unsigned short) /root/couchbase-3.0/ep-engine/src/ep_engine.cc:229 (ep.so+0x0000000fa21d) #7 mock_store /root/couchbase-3.0/memcached/programs/engine_testapp/engine_testapp.c:227 (engine_testapp+0x0000000ade2e) #8 storeCasVb11(engine_interface*, engine_interface_v1*, void const*, ENGINE_STORE_OPERATION, char const*, char const*, unsigned long, unsigned int, void**, unsigned long, unsigned short, unsigned int, unsigned char) /root/couchbase-3.0/ep-engine/tests/ep_test_apis.cc:654 (ep_perfsuite.so+0x000000018ec3) #9 perf_latency(engine_interface*, engine_interface_v1*, char const*) /root/couchbase-3.0/ep-engine/tests/ep_perfsuite.cc:104 (ep_perfsuite.so+0x0000000097e2) #10 perf_latency_baseline(engine_interface*, engine_interface_v1*) /root/couchbase-3.0/ep-engine/tests/ep_perfsuite.cc:169 (ep_perfsuite.so+0x0000000090b7) #11 execute_test /root/couchbase-3.0/memcached/programs/engine_testapp/engine_testapp.c:1042 (engine_testapp+0x0000000ab933) #12 main /root/couchbase-3.0/memcached/programs/engine_testapp/engine_testapp.c:1296 (engine_testapp+0x0000000a9a19) Previous read of size 1 at 0x7d4400009d58 by thread T20: #0 CouchbaseAtomic<bool>::load(memory_order) const /root/couchbase-3.0/ep-engine/src/atomic.h:79 (ep.so+0x00000005288c) #1 Flusher::canSnooze() /root/couchbase-3.0/ep-engine/src/flusher.h:104 (ep.so+0x00000018e92b) #2 Flusher::computeMinSleepTime() /root/couchbase-3.0/ep-engine/src/flusher.cc:239 (ep.so+0x00000018dc87) #3 Flusher::step(GlobalTask*) /root/couchbase-3.0/ep-engine/src/flusher.cc:187 (ep.so+0x00000018cb35) Change-Id: Ie32ca8fa4e662e1244362cbdb0cb2573f80665e2 Reviewed-on: http://review.couchbase.org/51968 Tested-by: buildbot <[email protected]> Reviewed-by: Jim Walker <[email protected]> Reviewed-by: Chiyoung Seo <[email protected]>

As identified by ThreadSantizer: WARNING: ThreadSanitizer: data race (pid=59833) Write of size 8 at 0x7d240000d3f0 by thread T8: #0 CouchbaseAtomic<unsigned long>::store(unsigned long const&, memory_order) /root/couchbase-3.0/ep-engine/src/atomic.h:84 (ep.so+0x0000000414ef) #1 CouchbaseAtomic<unsigned long>::operator=(unsigned long const&) /root/couchbase-3.0/ep-engine/src/atomic.h:105 (ep.so+0x0000000401f5) #2 Warmup::scheduleEstimateDatabaseItemCount() /root/couchbase-3.0/ep-engine/src/warmup.cc:500 (ep.so+0x000000277991) #3 Warmup::step() /root/couchbase-3.0/ep-engine/src/warmup.cc:816 (ep.so+0x000000275124) #4 Warmup::transition(int, bool) /root/couchbase-3.0/ep-engine/src/warmup.cc:853 (ep.so+0x0000002754ff) #5 Warmup::createVBuckets(unsigned short) /root/couchbase-3.0/ep-engine/src/warmup.cc:491 (ep.so+0x00000027785f) #6 WarmupCreateVBuckets::run() /root/couchbase-3.0/ep-engine/src/warmup.h:234 (ep.so+0x00000028cde9) #7 ExecutorThread::run() /root/couchbase-3.0/ep-engine/src/executorthread.cc:110 (ep.so+0x0000001a2581) #8 launch_executor_thread(void*) /root/couchbase-3.0/ep-engine/src/executorthread.cc:34 (ep.so+0x0000001a1a5a) #9 platform_thread_wrap /root/couchbase-3.0/platform/src/cb_pthreads.c:19 (libplatform.so.0.1.0+0x000000002f14) Previous atomic write of size 8 at 0x7d240000d3f0 by main thread (mutexes: write M670470284968504712): #0 __tsan_atomic64_fetch_add <null>:0 (engine_testapp+0x00000008cb48) #1 CouchbaseAtomic<unsigned long>::load(memory_order) const /root/couchbase-3.0/ep-engine/src/atomic.h:77 (ep.so+0x0000000446b4) #2 CouchbaseAtomic<unsigned long>::operator unsigned long() const /root/couchbase-3.0/ep-engine/src/atomic.h:101 (ep.so+0x000000044575) #3 Warmup::addStats(void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) const /root/couchbase-3.0/ep-engine/src/warmup.cc:901 (ep.so+0x00000027d4ea) #4 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /root/couchbase-3.0/ep-engine/src/ep_engine.cc:4405 (ep.so+0x0000001155a9) #5 EvpGetStats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /root/couchbase-3.0/ep-engine/src/ep_engine.cc:214 (ep.so+0x0000000fa102) #6 mock_get_stats /root/couchbase-3.0/memcached/programs/engine_testapp/engine_testapp.c:194 (engine_testapp+0x0000000aeecd) #7 wait_for_warmup_complete(engine_interface*, engine_interface_v1*) /root/couchbase-3.0/ep-engine/tests/ep_test_apis.cc:864 (ep_perfsuite.so+0x00000001b1cb) #8 test_setup(engine_interface*, engine_interface_v1*) /root/couchbase-3.0/ep-engine/tests/ep_testsuite_common.cc:128 (ep_perfsuite.so+0x00000000dc03) #9 execute_test /root/couchbase-3.0/memcached/programs/engine_testapp/engine_testapp.c:1037 (engine_testapp+0x0000000ab85a) #10 main /root/couchbase-3.0/memcached/programs/engine_testapp/engine_testapp.c:1296 (engine_testapp+0x0000000a9a19) Change-Id: I260942712ca471c0d2e0fa3ebc4793d694f9b237 Reviewed-on: http://review.couchbase.org/51973 Tested-by: buildbot <[email protected]> Reviewed-by: Jim Walker <[email protected]> Reviewed-by: Chiyoung Seo <[email protected]>

WARNING: ThreadSanitizer: data race (pid=20056) Write of size 8 at 0x7d600000f038 by main thread (mutexes: write M1412): #0 ConnHandler::setCookie(void const*) /home/abhinav/couchbase/ep-engine/src/tapconnection.h:344 (ep.so+0x000000042367) #1 EventuallyPersistentEngine::createTapQueue(void const*, std::string&, unsigned int, void const*, unsigned long) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:2655 (ep.so+0x0000000b86da) #2 EvpGetTapIterator(engine_interface*, void const*, void const*, unsigned long, unsigned int, void const*, unsigned long) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:1462 (ep.so+0x0000000b46a3) #3 mock_get_tap_iterator(engine_interface*, void const*, void const*, unsigned long, unsigned int, void const*, unsigned long) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:467 (engine_testapp+0x0000000bae3e) #4 test_tap_ack_stream(engine_interface*, engine_interface_v1*) /home/abhinav/couchbase/ep-engine/tests/ep_testsuite.cc:7341 (ep_testsuite.so+0x000000050416) #5 execute_test(test, char const*, char const*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1090 (engine_testapp+0x0000000b946c) #6 __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287 (libc.so.6+0x000000021ec4) Previous read of size 8 at 0x7d600000f038 by thread T9 (mutexes: write M1411): #0 ConnHandler::getCookie() const /home/abhinav/couchbase/ep-engine/src/tapconnection.h:340 (ep.so+0x00000004067c) #1 bool TapConnMap::performOp<Item*>(std::string const&, TapOperation<Item*>&, Item*) /home/abhinav/couchbase/ep-engine/src/connmap.h:389 (ep.so+0x00000001fa08) #2 ItemResidentCallback::callback(CacheLookup&) /home/abhinav/couchbase/ep-engine/src/backfill.cc:63 (ep.so+0x00000001d9ca) #3 CouchKVStore::recordDbDump(_db*, _docinfo*, void*) /home/abhinav/couchbase/ep-engine/src/couch-kvstore/couch-kvstore.cc:1654 (ep.so+0x000000180ca0) #4 recordDbDumpC(_db*, _docinfo*, void*) /home/abhinav/couchbase/ep-engine/src/couch-kvstore/couch-kvstore.cc:66 (ep.so+0x00000017fe95) #5 lookup_callback(couchfile_lookup_request*, _sized_buf const*, _sized_buf const*) /home/abhinav/couchbase/couchstore/src/couch_db.cc:767 (libcouchstore.so+0x00000000d7e5) #6 btree_lookup_inner(couchfile_lookup_request*, unsigned long, int, int) /home/abhinav/couchbase/couchstore/src/btree_read.cc:99 (libcouchstore.so+0x00000000b5a2) #7 btree_lookup /home/abhinav/couchbase/couchstore/src/btree_read.cc:131 (libcouchstore.so+0x00000000affc) #8 couchstore_changes_since /home/abhinav/couchbase/couchstore/src/couch_db.cc:812 (libcouchstore.so+0x00000000d5f1) #9 CouchKVStore::scan(ScanContext*) /home/abhinav/couchbase/ep-engine/src/couch-kvstore/couch-kvstore.cc:1264 (ep.so+0x00000017f94e) #10 BackfillDiskLoad::run() /home/abhinav/couchbase/ep-engine/src/backfill.cc:131 (ep.so+0x00000001e449) #11 ExecutorThread::run() /home/abhinav/couchbase/ep-engine/src/executorthread.cc:112 (ep.so+0x0000000f8956) #12 launch_executor_thread(void*) /home/abhinav/couchbase/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f84f5) #13 platform_thread_wrap /home/abhinav/couchbase/platform/src/cb_pthreads.c:23 (libplatform.so.0.1.0+0x000000003d31) Change-Id: I8a668f17013c95abc9786d853ed2c6462cae5320 Reviewed-on: http://review.couchbase.org/62978 Well-Formed: buildbot <[email protected]> Reviewed-by: Will Gardner <[email protected]> Tested-by: buildbot <[email protected]>

As reported by ThreadSanitizer (see below), there is a dirty read on {{buffer.items}} & {{buffer.bytes}} during stat writing, due to PassiveStream::addStats not acquiring the {{bufMutex}} lock before reading. This appears benign as the stat is only user sent to users, not used by ns_server etc for any calculation. WARNING: ThreadSanitizer: data race (pid=28418) Read of size 8 at 0x7d5000018128 by main thread (mutexes: write M23810, write M969): #0 void STATWRITER_NAMESPACE::add_casted_stat<unsigned long>(char const*, unsigned long const&, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) ep-engine/src/statwriter.h:47 (ep.so+0x0000000afe89) #1 PassiveStream::addStats(void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) ep-engine/src/dcp-stream.cc:1512 (ep.so+0x0000002a04ba) #2 DcpConsumer::addStats(void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) ep-engine/src/dcp-consumer.cc:555 (ep.so+0x00000026e867) Previous write of size 8 at 0x7d5000018128 by thread T18 (mutexes: write M23762): #0 PassiveStream::processBufferedMessages(unsigned int&) ep-engine/src/dcp-stream.cc:1311 (ep.so+0x00000029e196) #1 DcpConsumer::processBufferedItems() ep-engine/src/dcp-consumer.cc:599 (ep.so+0x0000002647e9) #2 Processer::run() ep-engine/src/dcp-consumer.cc:48 (ep.so+0x0000002643ef) Change-Id: I443e85d59ffda3827b670e747794b3fcb69c4f7c Reviewed-on: http://review.couchbase.org/63026 Well-Formed: buildbot <[email protected]> Reviewed-by: Will Gardner <[email protected]> Tested-by: buildbot <[email protected]>

WARNING: ThreadSanitizer: data race (pid=139161) Read of size 4 at 0x7d480000b150 by thread T31 (mutexes: write M51120): #0 DCPBackfill::scan() ep-engine/src/dcp/stream.h:126 (ep.so+0x000000053391) #1 DCPBackfill::run() ep-engine/src/dcp/backfill.cc:118 (ep.so+0x000000052737) #2 BackfillManager::backfill() ep-engine/src/dcp/backfill-manager.cc:240 (ep.so+0x00000004cf65) #3 BackfillManagerTask::run() ep-engine/src/dcp/backfill-manager.cc:43 (ep.so+0x00000004cb8f) #4 ExecutorThread::run() ep-engine/src/executorthread.cc:115 (ep.so+0x0000000eb94d) #5 launch_executor_thread(void*) ep-engine/src/executorthread.cc:33 (ep.so+0x0000000eb515) #6 platform_thread_wrap(void*) platform/src/cb_pthreads.cc:53 (libplatform.so.0.1.0+0x0000000048ab) Previous write of size 4 at 0x7d480000b150 by main thread (mutexes: write M1241, write M32448, write M51071, write M51087): #0 ActiveStream::transitionState(stream_state_t) ep-engine/src/dcp/stream.cc:829 (ep.so+0x00000006accb) #1 ActiveStream::endStream(end_stream_status_t) ep-engine/src/dcp/stream.cc:688 (ep.so+0x00000006a8c2) #2 ActiveStream::setDead(end_stream_status_t) ep-engine/src/dcp/stream.cc:654 (ep.so+0x00000006f27b) #3 DcpProducer::setDisconnect(bool) ep-engine/src/dcp/producer.cc:835 (ep.so+0x000000065605) #4 DcpConnMap::disconnect_UNLOCKED(void const*) ep-engine/src/connmap.cc:1116 (ep.so+0x000000045d6c) #5 DcpConnMap::disconnect(void const*) ep-engine/src/connmap.cc:1109 (ep.so+0x000000045c8b) #6 EventuallyPersistentEngine::handleDisconnect(void const*) ep-engine/src/ep_engine.cc:6265 (ep.so+0x0000000ca38a) #7 EvpHandleDisconnect(void const*, ENGINE_EVENT_TYPE, void const*, void const*) ep-engine/src/ep_engine.cc:1802 (ep.so+0x0000000af976) #8 destroy_mock_cookie memcached/programs/engine_testapp/mock_server.cc:325 (engine_testapp+0x0000004f4082) #9 dcp_stream_req(engine_interface*, engine_interface_v1*, unsigned int, unsigned short, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, ENGINE_ERROR_CODE) ep-engine/tests/ep_testsuite.cc:4331 (ep_testsuite.so+0x000000090b06) #10 test_failover_log_dcp(engine_interface*, engine_interface_v1*) ep-engine/tests/ep_testsuite.cc:14127 (ep_testsuite.so+0x00000007ce7a) #11 execute_test(test, char const*, char const*) memcached/programs/engine_testapp/engine_testapp.cc:1091 (engine_testapp+0x0000004cb315) #12 __libc_start_main /build/buildd/eglibc-2.15/csu/libc-start.c:226 (libc.so.6+0x00000002176c) Change-Id: Icfc82fa877999d128184c9cac8f8c0e1cafc4e67 Reviewed-on: http://review.couchbase.org/63027 Well-Formed: buildbot <[email protected]> Reviewed-by: Will Gardner <[email protected]> Tested-by: buildbot <[email protected]>

WARNING: ThreadSanitizer: data race (pid=236996) Write of size 8 at 0x7d380000dbd8 by thread T5: #0 ExecutorThread::run() ep-engine/src/executorthread.cc:105 (ep.so+0x0000000ee0cc) #1 launch_executor_thread(void*) ep-engine/src/executorthread.cc:33 (ep.so+0x0000000edd75) #2 platform_thread_wrap(void*) platform/src/cb_pthreads.cc:54 (libplatform.so.0.1.0+0x000000004e7b) Previous read of size 8 at 0x7d380000dbd8 by main thread (mutexes: write M266835151185571736): #0 addWorkerStats(char const*, ExecutorThread*, void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) ep-engine/src/executorthread.h:108 (ep.so+0x0000000ea8dc) #1 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) ep-engine/src/ep_engine.cc:4363 (ep.so+0x0000000c05dd) #2 EvpGetStats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) ep-engine/src/ep_engine.cc:219 (ep.so+0x0000000af40e) #3 mock_get_stats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) memcached/programs/engine_testapp/engine_testapp.cc:240 (engine_testapp+0x0000004cc71d) #4 test_worker_stats(engine_interface*, engine_interface_v1*) ep-engine/tests/ep_testsuite.cc:9464 (ep_testsuite.so+0x00000003c038) #5 execute_test(test, char const*, char const*) memcached/programs/engine_testapp/engine_testapp.cc:1091 (engine_testapp+0x0000004cb315) #6 __libc_start_main /build/buildd/eglibc-2.15/csu/libc-start.c:226 (libc.so.6+0x00000002176c) Change-Id: If01aba3cf6b3591f19c5bb62119e40998f12c8ff Reviewed-on: http://review.couchbase.org/63028 Well-Formed: buildbot <[email protected]> Reviewed-by: Jim Walker <[email protected]> Reviewed-by: Will Gardner <[email protected]> Tested-by: buildbot <[email protected]>

This one looks benign - we only perform the dirty read when calculating the %s:waketime stat which is not used by anyone apart from end-users. WARNING: ThreadSanitizer: data race (pid=41666) Read of size 8 at 0x7d4400008370 by main thread (mutexes: write M21616): #0 ExecutorThread::getWaketime() ep-engine/src/executorthread.h:120 (ep.so+0x0000001cac0e) #1 addWorkerStats(char const*, ExecutorThread*, void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) ep-engine/src/executorpool.cc:692 (ep.so+0x0000001b6f06) #2 ExecutorPool::doWorkerStat(EventuallyPersistentEngine*, void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) ep-engine/src/executorpool.cc:706 (ep.so+0x0000001b6734) #3 EventuallyPersistentEngine::doDispatcherStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) ep-engine/src/ep_engine.cc:4139 (ep.so+0x00000015223e) #4 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) ep-engine/src/ep_engine.cc:4375 (ep.so+0x000000155336) Previous write of size 8 at 0x7d4400008370 by thread T6 (mutexes: write M14985): #0 TaskQueue::_fetchNextTask(ExecutorThread&, bool) ep-engine/src/taskqueue.cc:125 (ep.so+0x00000025b3fd) #1 TaskQueue::fetchNextTask(ExecutorThread&, bool) ep-engine/src/taskqueue.cc:161 (ep.so+0x00000025bfcf) #2 ExecutorPool::_nextTask(ExecutorThread&, unsigned char) ep-engine/src/executorpool.cc:217 (ep.so+0x0000001afc67) #3 ExecutorPool::nextTask(ExecutorThread&, unsigned char) ep-engine/src/executorpool.cc:232 (ep.so+0x0000001afe3f) #4 ExecutorThread::run() ep-engine/src/executorthread.cc:81 (ep.so+0x0000001e85c9) Change-Id: I34b12681dd9dfc87c889f301692ca714f04d2a82 Reviewed-on: http://review.couchbase.org/63029 Well-Formed: buildbot <[email protected]> Reviewed-by: Jim Walker <[email protected]> Reviewed-by: Will Gardner <[email protected]> Tested-by: buildbot <[email protected]>

As identified by ThreadSanitizer: WARNING: ThreadSanitizer: data race (pid=17259) Write of size 8 at 0x7fec86e44de0 by main thread (mutexes: write M1161): #0 gmtime ??:0 (libtsan.so.0+0x000000025135) #1 EventuallyPersistentEngine::doEngineStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:3369 (ep.so+0x00000010f4be) #2 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:4339 (ep.so+0x000000113c35) #3 EvpGetStats /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:217 (ep.so+0x000000102b14) #4 mock_get_stats /repos/couchbase/server/source/memcached/programs/engine_testapp/engine_testapp.c:195 (exe+0x0000000026de) #5 get_int_stat(engine_interface*, engine_interface_v1*, char const*, char const*) /repos/couchbase/server/source/ep-engine/tests/ep_test_apis.cc:799 (ep_testsuite.so+0x0000000832d8) #6 wait_for_stat_change(engine_interface*, engine_interface_v1*, char const*, int, char const*) /repos/couchbase/server/source/ep-engine/tests/ep_test_apis.cc:846 (ep_testsuite.so+0x0000000838d6) #7 test_setup /repos/couchbase/server/source/ep-engine/tests/ep_testsuite.cc:178 (ep_testsuite.so+0x00000001ee69) #8 execute_test /repos/couchbase/server/source/memcached/programs/engine_testapp/engine_testapp.c:1050 (exe+0x000000005970) #9 main /repos/couchbase/server/source/memcached/programs/engine_testapp/engine_testapp.c:1313 (exe+0x000000006606) Previous write of size 8 at 0x7fec86e44de0 by thread T7: #0 gmtime ??:0 (libtsan.so.0+0x000000025135) #1 EventuallyPersistentEngine::doEngineStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:3369 (ep.so+0x00000010f4be) #2 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:4339 (ep.so+0x000000113c35) #3 EventuallyPersistentStore::snapshotStats() /repos/couchbase/server/source/ep-engine/src/ep.cc:1465 (ep.so+0x0000000e150e) #4 StatSnap::run() /repos/couchbase/server/source/ep-engine/src/tasks.cc:79 (ep.so+0x000000174db2) #5 ExecutorThread::run() /repos/couchbase/server/source/ep-engine/src/executorthread.cc:110 (ep.so+0x00000014a0e7) #6 launch_executor_thread /repos/couchbase/server/source/ep-engine/src/executorthread.cc:34 (ep.so+0x000000149930) #7 platform_thread_wrap /repos/couchbase/server/source/platform/src/cb_pthreads.c:19 (libplatform.so.0.1.0+0x000000002d8b) #8 __tsan_write_range ??:0 (libtsan.so.0+0x00000001b1c9) Switch to gmtime_r which is thread-safe. Change-Id: Id0773df65f4fc569c0a173b6185b1ef8bd91862d Reviewed-on: http://review.couchbase.org/63080 Well-Formed: buildbot <[email protected]> Reviewed-by: Jim Walker <[email protected]> Reviewed-by: Will Gardner <[email protected]> Tested-by: buildbot <[email protected]>

As reported by ThreadSanitizer. CouchKVStore maintains a map of vBucketID to counter - dbFileRevMap. This is read by some of the stats functions (e.g. getNumPersistedDeletes) without a lock and hence there is a potential race. Solve this by changing the type of these counters to RelaxedAtomic<>. WARNING: ThreadSanitizer: data race (pid=10155) Read of size 8 at 0x7d9000008000 by main thread (mutexes: write M21730): #0 CouchKVStore::getNumPersistedDeletes(unsigned short) ep-engine/src/couch-kvstore/couch-kvstore.cc:2095 (ep.so+0x000000326779) #1 EventuallyPersistentEngine::doDcpVbTakeoverStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), std::string&, unsigned short) ep-engine/src/ep_engine.cc:5312 (ep.so+0x000000155ca5) #2 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) ep-engine/src/ep_engine.cc:4462 (ep.so+0x000000154622) Previous write of size 8 at 0x7d9000008000 by thread T8 (mutexes: write M15079): #0 CouchKVStore::updateDbFileMap(unsigned short, unsigned long) ep-engine/src/couch-kvstore/couch-kvstore.cc:1306 (ep.so+0x000000311d3c) #1 CouchKVStore::openDB(unsigned short, unsigned long, _db**, unsigned long, unsigned long*) ep-engine/src/couch-kvstore/couch-kvstore.cc:1336 (ep.so+0x00000030f7ae) #2 CouchKVStore::setVBucketState(unsigned short, vbucket_state&, Callback<KVStatsCtx>*) ep-engine/src/couch-kvstore/couch-kvstore.cc:981 (ep.so+0x00000031a557) #3 CouchKVStore::snapshotVBucket(unsigned short, vbucket_state&, Callback<KVStatsCtx>*) ep-engine/src/couch-kvstore/couch-kvstore.cc:891 (ep.so+0x00000031a11c) #4 EventuallyPersistentStore::snapshotVBuckets(Priority const&, unsigned short) ep-engine/src/ep.cc:949 (ep.so+0x0000000dce69) Change-Id: I83db17ffce0d0a49cfe80f23a34e5dac25ede719 Reviewed-on: http://review.couchbase.org/63081 Well-Formed: buildbot <[email protected]> Reviewed-by: Jim Walker <[email protected]> Reviewed-by: Will Gardner <[email protected]> Tested-by: buildbot <[email protected]>

std::unordered_map::size() is not thread-safe unless other threads are read-only. WARNING: ThreadSanitizer: data race (pid=131734) Write of size 8 at 0x7d580002fc50 by thread T19 (mutexes: read M15526, write M20152): #0 std::_Hashtable<unsigned short, std::pair<unsigned short const, SingleThreadedRCPtr<Stream> >, std::allocator<std::pair<unsigned short const, SingleThreadedRCPtr<Stream> > >, std::__detail::_Select1st, std::equal_to<unsigned short>, std::hash<unsigned short>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_insert_unique_node(unsigned long, unsigned long, std::__detail::_Hash_node<std::pair<unsigned short const, SingleThreadedRCPtr<Stream> >, false>*) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/hashtable.h:1625 (ep.so+0x000000069fa7) #1 AtomicUnorderedMap<unsigned short, Stream, SingleThreadedRCPtr<Stream>, std::hash<unsigned short>, std::equal_to<unsigned short>, std::allocator<std::pair<unsigned short const, SingleThreadedRCPtr<Stream> > > >::insert(std::pair<unsigned short const, SingleThreadedRCPtr<Stream> > const&) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/hashtable.h:1724 (ep.so+0x0000000613e9) #2 EvpDcpStreamReq(engine_interface*, void const*, unsigned int, unsigned int, unsigned short, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long*, ENGINE_ERROR_CODE (*)(vbucket_failover_t*, unsigned long, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep_engine.cc:1512 (ep.so+0x0000000b3737) ... Previous read of size 8 at 0x7d580002fc50 by main thread (mutexes: write M14188, write M20259, write M14598): #0 DcpProducer::addStats(void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/hashtable.h:500 (ep.so+0x0000000640f4) #1 ConnStatBuilder::operator()(SingleThreadedRCPtr<ConnHandler>&) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep_engine.cc:3859 (ep.so+0x0000000e02c1) #2 EventuallyPersistentEngine::doDcpStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep_engine.cc:4116 (ep.so+0x0000000bf61a) Change-Id: I139ca87ee1e4a1615f84ffc5318f577eee162106 Reviewed-on: http://review.couchbase.org/63363 Tested-by: buildbot <[email protected]> Reviewed-by: Chiyoung Seo <[email protected]>

…t lock + [Not a backport, this code was altered/removed in master] + This scenario should not occur in real operation. + Most DCP unit tests would however flag this as an issue because of how we do things in the tests --> [setting vbucket's state to replica at the very beginning (by the main thread)]. + Suppressing this lock inversion, by moving the function call to update the vbucket's snapshot range to outside the state lock context in setState(), as it isn't necessary to acquire the state lock to update the snapshot range. WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=39750) Cycle in lock order graph: M43306 (0x7d640000fcf0) => M43309 (0x7d640000fe18) => M43306 Mutex M43309 acquired here while holding mutex M43306 in main thread: #0 pthread_mutex_lock <null> (engine_testapp+0x000000474420) #1 cb_mutex_enter /home/daver/repos/couchbase/server/platform/src/cb_pthreads.c:85 (libplatform.so.0.1.0+0x0000000034a0) #2 Mutex::acquire() /home/daver/repos/couchbase/server/ep-engine/src/mutex.cc:31 (ep.so+0x0000001c611e) #3 LockHolder::lock() /home/daver/repos/couchbase/server/ep-engine/src/locks.h:71 (ep.so+0x00000006a4e3) #4 LockHolder /home/daver/repos/couchbase/server/ep-engine/src/locks.h:48 (ep.so+0x00000006a172) #5 VBucket::setCurrentSnapshot(unsigned long, unsigned long) /home/daver/repos/couchbase/server/ep-engine/src/vbucket.h:217 (ep.so+0x0000000e5ee5) #6 VBucket::setState(vbucket_state_t, server_handle_v1_t*) /home/daver/repos/couchbase/server/ep-engine/src/vbucket.cc:196 (ep.so+0x0000002932e9) #7 EventuallyPersistentStore::setVBucketState(unsigned short, vbucket_state_t, bool, bool) /home/daver/repos/couchbase/server/ep-engine/src/ep.cc:1060 (ep.so+0x0000000c0b61) #8 EventuallyPersistentEngine::setVBucketState(unsigned short, vbucket_state_t, bool) /home/daver/repos/couchbase/server/ep-engine/src/ep_engine.h:628 (ep.so+0x000000188a12) #9 setVBucket(EventuallyPersistentEngine*, void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) /home/daver/repos/couchbase/server/ep-engine/src/ep_engine.cc:824 (ep.so+0x00000014aaaa) #10 processUnknownCommand(EventuallyPersistentEngine*, void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) /home/daver/repos/couchbase/server/ep-engine/src/ep_engine.cc:1118 (ep.so+0x000000147707) #11 EvpUnknownCommand(engine_interface*, void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) /home/daver/repos/couchbase/server/ep-engine/src/ep_engine.cc:1312 (ep.so+0x00000011a055) #12 mock_unknown_command /home/daver/repos/couchbase/server/memcached/programs/engine_testapp/engine_testapp.c:335 (engine_testapp+0x0000004be97a) #13 set_vbucket_state(engine_interface*, engine_interface_v1*, unsigned short, vbucket_state_t) /home/daver/repos/couchbase/server/ep-engine/tests/ep_test_apis.cc:484 (ep_testsuite.so+0x0000000e1562) #14 test_dcp_replica_stream_backfill(engine_interface*, engine_interface_v1*) /home/daver/repos/couchbase/server/ep-engine/tests/ep_testsuite.cc:5278 (ep_testsuite.so+0x00000008c84a) #15 execute_test /home/daver/repos/couchbase/server/memcached/programs/engine_testapp/engine_testapp.c:1042 (engine_testapp+0x0000004ba8ff) #16 main /home/daver/repos/couchbase/server/memcached/programs/engine_testapp/engine_testapp.c:1296 (engine_testapp+0x0000004b8861) Mutex M43306 acquired here while holding mutex M43309 in thread T17: #0 pthread_rwlock_rdlock <null> (engine_testapp+0x000000457ca0) #1 cb_rw_reader_enter /home/daver/repos/couchbase/server/platform/src/cb_pthreads.c:264 (libplatform.so.0.1.0+0x0000000042e0) #2 RWLock::readerLock() /home/daver/repos/couchbase/server/ep-engine/src/rwlock.h:38 (ep.so+0x000000115cf0) #3 ReaderLockHolder /home/daver/repos/couchbase/server/ep-engine/src/locks.h:167 (ep.so+0x0000000dbbe7) #4 EventuallyPersistentStore::addTAPBackfillItem(Item const&, unsigned char, bool) /home/daver/repos/couchbase/server/ep-engine/src/ep.cc:851 (ep.so+0x0000000be35d) #5 PassiveStream::commitMutation(MutationResponse*, bool) /home/daver/repos/couchbase/server/ep-engine/src/dcp-stream.cc:1370 (ep.so+0x00000027e8cc) #6 PassiveStream::processMutation(MutationResponse*) /home/daver/repos/couchbase/server/ep-engine/src/dcp-stream.cc:1346 (ep.so+0x00000027d680) #7 PassiveStream::processBufferedMessages(unsigned int&) /home/daver/repos/couchbase/server/ep-engine/src/dcp-stream.cc:1286 (ep.so+0x00000027cfbc) #8 DcpConsumer::processBufferedItems() /home/daver/repos/couchbase/server/ep-engine/src/dcp-consumer.cc:599 (ep.so+0x0000002454cc) #9 Processer::run() /home/daver/repos/couchbase/server/ep-engine/src/dcp-consumer.cc:48 (ep.so+0x0000002450ef) #10 ExecutorThread::run() /home/daver/repos/couchbase/server/ep-engine/src/executorthread.cc:109 (ep.so+0x0000001c76d9) #11 launch_executor_thread(void*) /home/daver/repos/couchbase/server/ep-engine/src/executorthread.cc:34 (ep.so+0x0000001c6caa) #12 platform_thread_wrap /home/daver/repos/couchbase/server/platform/src/cb_pthreads.c:19 (libplatform.so.0.1.0+0x00000000325c) Change-Id: I2f3cf88e6aebbf6078f533fb1ed87bd9fe618616 Reviewed-on: http://review.couchbase.org/63319 Well-Formed: buildbot <[email protected]> Reviewed-by: Dave Rigby <[email protected]> Tested-by: buildbot <[email protected]> Reviewed-by: Will Gardner <[email protected]>

…t lock + [Not a backport, this code was altered/removed in master] + Address this lock inversion by moving reading the vbucket snapshot range to outside the vbucket's state lock context. WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=245522) Cycle in lock order graph: M21518 (0x7d640003e220) => M21515 (0x7d640003e0f0) => M21518 Mutex M21515 acquired here while holding mutex M21518 in thread T17: #0 pthread_rwlock_rdlock <null> (engine_testapp+0x000000462260) #1 cb_rw_reader_enter <null> (libplatform.so.0.1.0+0x000000004800) #2 RWLock::readerLock() ep-engine/src/rwlock.h:38 (ep.so+0x000000132360) #3 ReaderLockHolder::ReaderLockHolder(RWLock&) ep-engine/src/locks.h:167 (ep.so+0x0000000f8087) #4 EventuallyPersistentStore::addTAPBackfillItem(Item const&, unsigned char, bool) ep-engine/src/ep.cc:851 (ep.so+0x0000000d9ba7) #5 PassiveStream::commitMutation(MutationResponse*, bool) ep-engine/src/dcp-stream.cc:1370 (ep.so+0x00000029dd8c) #6 PassiveStream::processMutation(MutationResponse*) ep-engine/src/dcp-stream.cc:1346 (ep.so+0x00000029cbd0) #7 PassiveStream::processBufferedMessages(unsigned int&) ep-engine/src/dcp-stream.cc:1286 (ep.so+0x00000029c522) #8 DccpConsumer::processBufferedItems() ep-engine/src/dcp-consumer.cc:599 (ep.so+0x000000262e04) #9 Processer::run() ep-engine/src/dcp-consumer.cc:48 (ep.so+0x0000002629ff) #10 ExecutorThread::run() ep-engine/src/executorthread.cc:109 (ep.so+0x0000001e38f1) #11 launch_executor_thread(void*) ep-engine/src/executorthread.cc:34 (ep.so+0x0000001e2f1a) #12 platform_thread_wrap platform/src/cb_pthreads.c (libplatform.so.0.1.0+0x00000000377c) Mutex M21518 acquired here while holding mutex M21515 in main thread: #0 pthread_mutex_lock <null> (engine_testapp+0x00000047e9e0) #1 cb_mutex_enter <null> (libplatform.so.0.1.0+0x0000000039c0) #2 Mutex::acquire() ep-engine/src/mutex.cc:31 (ep.so+0x0000001e241e) #3 LockHolder::lock() ep-engine/src/locks.h:71 (ep.so+0x000000080bc3) #4 LockHolder::LockHolder(Mutex&, bool) ep-engine/src/locks.h:48 (ep.so+0x000000080832) #5 VBucket::getCurrentSnapshot(unsigned long&, unsigned long&) ep-engine/src/vbucket.h:233 (ep.so+0x0000000fae05) #6 EventuallyPersistentEngine::doSeqnoStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), char const*, int) ep-engine/src/ep_engine.cc:4255 (ep.so+0x00000014f202) #7 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) ep-engine/src/ep_engine.cc:4372 (ep.so+0x000000150bcb) #8 EvpGetStats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) ep-engine/src/ep_engine.cc:214 (ep.so+0x000000136a72) #9 mock_get_stats memcached/programs/engine_testapp/engine_testapp.c (engine_testapp+0x0000004c8403) #10 get_int_stat(engine_interface*, engine_interface_v1*, char const*, char const*) ep-engine/tests/ep_test_apis.cc:771 (ep_testsuite.so+0x0000000e21ea) #11 wait_for_stat_to_be(engine_interface*, engine_interface_v1*, char const*, int, char const*) ep-engine/tests/ep_test_apis.cc:860 (ep_testsuite.so+0x0000000e8d2b) #12 test_dcp_replica_stream_backfill(engine_interface*, engine_interface_v1*) ep-engine/tests/ep_testsuite.cc:5306 (ep_testsuite.so+0x00000008e601) #13 execute_test memcached/programs/engine_testapp/engine_testapp.c (engine_testapp+0x0000004c4e9f) #14 main crtstuff.c (engine_testapp+0x0000004c2e01) Change-Id: Ia4dd34ab152d1cc1d1658ebe957da7c3b8d32c06 Reviewed-on: http://review.couchbase.org/63366 Well-Formed: buildbot <[email protected]> Tested-by: buildbot <[email protected]> Reviewed-by: Will Gardner <[email protected]>

A mistake in 495e00a means that no variable is created for the ReaderLockHolder, the compiler either optimises away the lock constructor/destructor or the lock scope is wrong. Either way we need to create a variable. Includes some lock ordering changes as per ThreadSantitiser warnings. (Reviewed-on: http://review.couchbase.org/56978) This will address the following lock inversion: 11:56:19 WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=51509) 11:56:19 Cycle in lock order graph: M21441 (0x7d780000f450) => M21477 (0x7d640005edf0) => M21441 11:56:19 11:56:19 Mutex M21477 acquired here while holding mutex M21441 in main thread: 11:56:19 #0 pthread_rwlock_rdlock <null> (engine_testapp+0x000000462260) 11:56:19 #1 cb_rw_reader_enter <null> (libplatform.so.0.1.0+0x000000004800) 11:56:19 #2 RWLock::readerLock() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/rwlock.h:38 (ep.so+0x000000132360) 11:56:19 #3 ReaderLockHolder::ReaderLockHolder(RWLock&) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/locks.h:167 (ep.so+0x0000000f8087) 11:56:19 #4 EventuallyPersistentStore::getAndUpdateTtl(std::string const&, unsigned short, void const*, long) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/ep.cc:1970 (ep.so+0x0000000e6b45) 11:56:19 #5 EventuallyPersistentEngine::touch(void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/ep_engine.cc:4619 (ep.so+0x000000155fe8) 11:56:19 #6 processUnknownCommand(EventuallyPersistentEngine*, void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/ep_engine.cc:1126 (ep.so+0x000000163a9b) 11:56:19 #7 EvpUnknownCommand(engine_interface*, void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/ep_engine.cc:1312 (ep.so+0x000000137365) 11:56:19 #8 mock_unknown_command /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/memcached/programs/engine_testapp/engine_testapp.c (engine_testapp+0x0000004c8f1a) 11:56:19 #9 gat(engine_interface*, engine_interface_v1*, char const*, unsigned short, unsigned int, bool) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/tests/ep_test_apis.cc:348 (ep_testsuite.so+0x0000000e2d6b) 11:56:19 #10 test_expired_item_with_item_eviction(engine_interface*, engine_interface_v1*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/tests/ep_testsuite.cc:11401 (ep_testsuite.so+0x0000000acbd4) 11:56:19 #11 execute_test /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/memcached/programs/engine_testapp/engine_testapp.c (engine_testapp+0x0000004c4e9f) 11:56:19 #12 main crtstuff.c (engine_testapp+0x0000004c2e01) 11:56:19 11:56:19 Mutex M21441 acquired here while holding mutex M21477 in thread T8: 11:56:19 #0 pthread_mutex_lock <null> (engine_testapp+0x00000047e9e0) 11:56:19 #1 cb_mutex_enter <null> (libplatform.so.0.1.0+0x0000000039c0) 11:56:19 #2 Mutex::acquire() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/mutex.cc:31 (ep.so+0x0000001e241e) 11:56:19 #3 LockHolder::lock() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/locks.h:71 (ep.so+0x000000080bc3) 11:56:19 #4 LockHolder::LockHolder(Mutex&, bool) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/locks.h:48 (ep.so+0x000000080832) 11:56:19 #5 HashTable::getLockedBucket(int, int*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/stored-value.h:1266 (ep.so+0x00000008280a) 11:56:19 #6 HashTable::getLockedBucket(std::string const&, int*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/stored-value.h:1295 (ep.so+0x00000007c61b) 11:56:19 #7 EventuallyPersistentStore::deleteExpiredItem(unsigned short, std::string&, long, unsigned long) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/ep.cc:481 (ep.so+0x0000000d4d80) 11:56:19 #8 ExpiredItemsCallback::callback(compaction_ctx&) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/ep.cc:1258 (ep.so+0x000000123ecb) 11:56:19 #9 CouchKVStore::compactVBucket(unsigned short, compaction_ctx*, Callback<compaction_ctx>&, Callback<KVStatsCtx>&) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/couch-kvstore/couch-kvstore.cc:862 (ep.so+0x0000003159f7) 11:56:19 #10 EventuallyPersistentStore::compactVBucket(unsigned short, compaction_ctx*, void const*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/ep.cc:1326 (ep.so+0x0000000df2ec) 11:56:19 #11 CompactVBucketTask::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/tasks.cc:76 (ep.so+0x000000251ed1) 11:56:19 #12 ExecutorThread::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/executorthread.cc:109 (ep.so+0x0000001e38f1) 11:56:19 #13 launch_executor_thread(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/executorthread.cc:34 (ep.so+0x0000001e2f1a) 11:56:19 #14 platform_thread_wrap /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/platform/src/cb_pthreads.c (libplatform.so.0.1.0+0x00000000377c) Change-Id: I5d5ca33fdd3c17df2be9d2b2d6acc8c254f1cb2d Reviewed-on: http://review.couchbase.org/63418 Well-Formed: buildbot <[email protected]> Tested-by: buildbot <[email protected]> Reviewed-by: Will Gardner <[email protected]> Reviewed-by: Chiyoung Seo <[email protected]>

[Not a backport, this code was altered/removed in master] 11:56:19 Read of size 8 at 0x7d6400050df8 by main thread (mutexes: write M45364, write M44294): 11:56:19 #0 std::_Hashtable<std::string, std::pair<std::string const, std::list<VBucketBGFetchItem*, std::allocator<VBucketBGFetchItem*> > >, std::allocator<std::pair<std::string const, std::list<VBucketBGFetchItem*, std::allocator<VBucketBGFetchItem*> > > >, std::__detail::_Select1st, std::equal_to<std::string>, std::hash<std::string>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::size() const /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/hashtable.h:500 (ep.so+0x00000008d98e) 11:56:19 #1 std::unordered_map<std::string, std::list<VBucketBGFetchItem*, std::allocator<VBucketBGFetchItem*> >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::list<VBucketBGFetchItem*, std::allocator<VBucketBGFetchItem*> > > > >::size() const /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/unordered_map.h:264 (ep.so+0x000000085be0) 11:56:19 #2 VBucket::numPendingBGFetchItems() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/vbucket.h:333 (ep.so+0x000000101789) 11:56:19 #3 EventuallyPersistentStore::bgFetch(std::string const&, unsigned short, unsigned long, void const*, bool) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/ep.cc:1651 (ep.so+0x0000000d71f7) 11:56:19 #4 EventuallyPersistentStore::getInternal(std::string const&, unsigned short, void const*, bool, bool, vbucket_state_t, bool) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/ep.cc:1708 (ep.so+0x0000000e3e21) 11:56:19 #5 EventuallyPersistentStore::get(std::string const&, unsigned short, void const*, bool, bool, bool) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/ep.h:242 (ep.so+0x00000019eee9) 11:56:19 #6 EventuallyPersistentEngine::get(void const*, void**, void const*, int, unsigned short, bool) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/ep_engine.h:259 (ep.so+0x00000016fe75) 11:56:19 #7 EvpGet(engine_interface*, void const*, void**, void const*, int, unsigned short) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/ep_engine.cc:202 (ep.so+0x000000136911) 11:56:19 #8 mock_get /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/memcached/programs/engine_testapp/engine_testapp.c (engine_testapp+0x0000004c6e84) 11:56:19 #9 get_item_info(engine_interface*, engine_interface_v1*, item_info*, char const*, unsigned short) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/tests/ep_test_apis.cc:356 (ep_testsuite.so+0x0000000e2f2d) 11:56:19 #10 check_key_value(engine_interface*, engine_interface_v1*, char const*, char const*, unsigned long, unsigned short) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/tests/ep_testsuite.cc:155 (ep_testsuite.so+0x0000000b2590) 11:56:19 #11 test_duplicate_items_disk(engine_interface*, engine_interface_v1*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/tests/ep_testsuite.cc:7839 (ep_testsuite.so+0x00000005a4b6) 11:56:19 #12 execute_test /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/memcached/programs/engine_testapp/engine_testapp.c (engine_testapp+0x0000004c4e9f) 11:56:19 #13 main crtstuff.c (engine_testapp+0x0000004c2e01) 11:56:19 11:56:19 Previous write of size 8 at 0x7d6400050df8 by thread T1 (mutexes: write M44318): 11:56:19 #0 std::_Hashtable<std::string, std::pair<std::string const, std::list<VBucketBGFetchItem*, std::allocator<VBucketBGFetchItem*> > >, std::allocator<std::pair<std::string const, std::list<VBucketBGFetchItem*, std::allocator<VBucketBGFetchItem*> > > >, std::__detail::_Select1st, std::equal_to<std::string>, std::hash<std::string>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::clear() /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/hashtable.h:1943 (ep.so+0x000000087ba8) 11:56:19 #1 std::unordered_map<std::string, std::list<VBucketBGFetchItem*, std::allocator<VBucketBGFetchItem*> >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::list<VBucketBGFetchItem*, std::allocator<VBucketBGFetchItem*> > > > >::clear() /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/unordered_map.h:528 (ep.so+0x000000086f40) 11:56:19 #2 VBucket::getBGFetchItems(std::unordered_map<std::string, std::list<VBucketBGFetchItem*, std::allocator<VBucketBGFetchItem*> >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::list<VBucketBGFetchItem*, std::allocator<VBucketBGFetchItem*> > > > >&) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/vbucket.cc:294 (ep.so+0x0000002b4026) 11:56:19 #3 BgFetcher::run(GlobalTask*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/bgfetcher.cc:155 (ep.so+0x00000008449f) 11:56:19 #4 BgFetcherTask::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/tasks.cc:89 (ep.so+0x00000025205e) 11:56:19 #5 ExecutorThread::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/executorthread.cc:109 (ep.so+0x0000001e38f1) 11:56:19 #6 launch_executor_thread(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/executorthread.cc:34 (ep.so+0x0000001e2f1a) 11:56:19 #7 platform_thread_wrap /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/platform/src/cb_pthreads.c (libplatform.so.0.1.0+0x00000000377c) Change-Id: I66e3c2af1f58448a68fbfedf3dfa030a657ed9a7 Reviewed-on: http://review.couchbase.org/63369 Well-Formed: buildbot <[email protected]> Tested-by: buildbot <[email protected]> Reviewed-by: Will Gardner <[email protected]> Reviewed-by: Chiyoung Seo <[email protected]>

WARNING: ThreadSanitizer: data race (pid=14344) Read of size 8 at 0x7d780000fa58 by thread T6: #0 void STATWRITER_NAMESPACE::add_casted_stat<long>(char const*, long const&, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/abhinav/couchbase/ep-engine/src/statwriter.h:45 (ep.so+0x000000037ff5) #1 EventuallyPersistentEngine::doEngineStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:3557 (ep.so+0x0000000be990) #2 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:4554 (ep.so+0x0000000c5c8c) #3 EventuallyPersistentStore::snapshotStats() /home/abhinav/couchbase/ep-engine/src/ep.cc:1671 (ep.so+0x00000008f1fe) #4 StatSnap::run() /home/abhinav/couchbase/ep-engine/src/tasks.cc:97 (ep.so+0x00000013cea6) #5 ExecutorThread::run() /home/abhinav/couchbase/ep-engine/src/executorthread.cc:112 (ep.so+0x0000000f94e3) #6 launch_executor_thread(void*) /home/abhinav/couchbase/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f9065) #7 platform_thread_wrap /home/abhinav/couchbase/platform/src/cb_pthreads.c:23 (libplatform.so.0.1.0+0x000000003d31) Previous write of size 8 at 0x7d780000fa58 by main thread: #0 EventuallyPersistentEngine::initialize(char const*) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:2167 (ep.so+0x0000000b728a) #1 EvpInitialize(engine_interface*, char const*) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:133 (ep.so+0x0000000b4aa8) #2 init_engine_instance /home/abhinav/couchbase/memcached/utilities/engine_loader.c:157 (libmcd_util.so.1.0.0+0x0000000058bb) #3 create_bucket(bool, char const*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:980 (engine_testapp+0x0000000b9e12) #4 execute_test(test, char const*, char const*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1083 (engine_testapp+0x0000000b93db) #5 __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287 (libc.so.6+0x000000021ec4) (Reviewed-on: http://review.couchbase.org/55776) Change-Id: Ibec6c267f9138aab626359c703fc067f91e1ee43 Reviewed-on: http://review.couchbase.org/63419 Well-Formed: buildbot <[email protected]> Reviewed-by: Will Gardner <[email protected]> Reviewed-by: Chiyoung Seo <[email protected]> Tested-by: abhinav dangeti <[email protected]>

…t lock + [Not a backport, this code was altered/removed in master] + Address this lock inversion by moving the code that reads the vbucket snapshot range to outside the vbucket's state lock context. 15:30:43 WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=235352) 15:30:43 Cycle in lock order graph: M21536 (0x7d640002f720) => M21533 (0x7d640002f5f0) => M21536 15:30:43 15:30:43 Mutex M21533 acquired here while holding mutex M21536 in thread T17: 15:30:43 #0 pthread_rwlock_rdlock <null> (engine_testapp+0x000000462260) 15:30:43 #1 cb_rw_reader_enter <null> (libplatform.so.0.1.0+0x000000004800) 15:30:43 #2 RWLock::readerLock() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/rwlock.h:38 (ep.so+0x0000001327e0) 15:30:43 #3 ReaderLockHolder::ReaderLockHolder(RWLock&) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/locks.h:167 (ep.so+0x0000000f84c7) 15:30:43 #4 EventuallyPersistentStore::addTAPBackfillItem(Item const&, unsigned char, bool) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/ep.cc:851 (ep.so+0x0000000d9c67) 15:30:43 #5 PassiveStream::commitMutation(MutationResponse*, bool) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/dcp-stream.cc:1370 (ep.so+0x00000029e25c) 15:30:43 #6 PassiveStream::processMutation(MutationResponse*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/dcp-stream.cc:1346 (ep.so+0x00000029d0a0) 15:30:43 #7 PassiveStream::processBufferedMessages(unsigned int&) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/dcp-stream.cc:1286 (ep.so+0x00000029c9f2) 15:30:43 #8 DcpConsumer::processBufferedItems() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/dcp-consumer.cc:599 (ep.so+0x0000002632d4) 15:30:43 #9 Processer::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/dcp-consumer.cc:48 (ep.so+0x000000262ecf) 15:30:43 #10 ExecutorThread::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/executorthread.cc:109 (ep.so+0x0000001e3dc1) 15:30:43 #11 launch_executor_thread(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/executorthread.cc:34 (ep.so+0x0000001e33ea) 15:30:43 #12 platform_thread_wrap /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/platform/src/cb_pthreads.c (libplatform.so.0.1.0+0x00000000377c) 15:30:43 15:30:43 Mutex M21536 acquired here while holding mutex M21533 in main thread: 15:30:43 #0 pthread_mutex_lock <null> (engine_testapp+0x00000047e9e0) 15:30:43 #1 cb_mutex_enter <null> (libplatform.so.0.1.0+0x0000000039c0) 15:30:43 #2 Mutex::acquire() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/mutex.cc:31 (ep.so+0x0000001e28ee) 15:30:43 #3 LockHolder::lock() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/locks.h:71 (ep.so+0x000000080bc3) 15:30:43 #4 LockHolder::LockHolder(Mutex&, bool) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/locks.h:48 (ep.so+0x000000080832) 15:30:43 #5 VBucket::getCurrentSnapshot(unsigned long&, unsigned long&) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/vbucket.h:233 (ep.so+0x0000000fb245) 15:30:43 #6 ActiveStream::ActiveStream(EventuallyPersistentEngine*, SingleThreadedRCPtr<DcpProducer>, std::string const&, unsigned int, unsigned int, unsigned short, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/dcp-stream.cc:293 (ep.so+0x000000291276) 15:30:43 #7 DcpProducer::streamRequest(unsigned int, unsigned int, unsigned short, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long*, ENGINE_ERROR_CODE (*)(vbucket_failover_t*, unsigned long, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/dcp-producer.cc:259 (ep.so+0x00000027b9a3) 15:30:43 #8 EvpDcpStreamReq(engine_interface*, void const*, unsigned int, unsigned int, unsigned short, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long*, ENGINE_ERROR_CODE (*)(vbucket_failover_t*, unsigned long, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/ep_engine.cc:1471 (ep.so+0x0000001395e3) 15:30:43 #9 mock_dcp_stream_req /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/memcached/programs/engine_testapp/engine_testapp.c (engine_testapp+0x0000004caf81) 15:30:43 #10 dcp_stream(engine_interface*, engine_interface_v1*, char const*, void const*, unsigned short, unsigned int, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, int, int, int, int, int, bool, bool, unsigned long, bool) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/tests/ep_testsuite.cc:3427 (ep_testsuite.so+0x0000000b357e) 15:30:43 #11 test_dcp_replica_stream_backfill(engine_interface*, engine_interface_v1*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/tests/ep_testsuite.cc:5311 (ep_testsuite.so+0x00000008e78a) 15:30:43 #12 execute_test /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/memcached/programs/engine_testapp/engine_testapp.c (engine_testapp+0x0000004c4e9f) 15:30:43 #13 main crtstuff.c (engine_testapp+0x0000004c2e01) Change-Id: Idc09ce9af98669f74f28d1fd4b1cc15f7d8b1152 Reviewed-on: http://review.couchbase.org/63379 Well-Formed: buildbot <[email protected]> Tested-by: buildbot <[email protected]> Reviewed-by: Will Gardner <[email protected]>

WARNING: ThreadSanitizer: data race (pid=3212) Write of size 8 at 0x7d5000016908 by thread T5 (mutexes: write M26478): #0 PassiveStream::reconnectStream(RCPtr<VBucket>&, unsigned int, unsigned long) /home/abhinav/couchbase/ep-engine/src/dcp/stream.cc:1097 (ep.so+0x000000076c0f) #1 DcpConsumer::doRollback(unsigned int, unsigned short, unsigned long) /home/abhinav/couchbase/ep-engine/src/dcp/consumer.cc:676 (ep.so+0x00000005db67) #2 RollbackTask::run() /home/abhinav/couchbase/ep-engine/src/dcp/consumer.cc:574 (ep.so+0x00000005d9d4) #3 ExecutorThread::run() /home/abhinav/couchbase/ep-engine/src/executorthread.cc:112 (ep.so+0x0000000f8916) #4 launch_executor_thread(void*) /home/abhinav/couchbase/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f84b5) #5 platform_thread_wrap /home/abhinav/couchbase/platform/src/cb_pthreads.c:23 (libplatform.so.0.1.0+0x000000003d31) Previous read of size 8 at 0x7d5000016908 by main thread (mutexes: write M1367): #0 PassiveStream::setDead_UNLOCKED(end_stream_status_t, LockHolder*) /home/abhinav/couchbase/ep-engine/src/dcp/stream.cc:1046 (ep.so+0x0000000759ca) #1 PassiveStream::setDead(end_stream_status_t) /home/abhinav/couchbase/ep-engine/src/dcp/stream.cc:1056 (ep.so+0x0000000766d7) #2 DcpConsumer::closeAllStreams() /home/abhinav/couchbase/ep-engine/src/dcp/consumer.cc:860 (ep.so+0x00000005a006) #3 DcpConnMap::disconnect_UNLOCKED(void const*) /home/abhinav/couchbase/ep-engine/src/connmap.cc:1137 (ep.so+0x000000049972) #4 DcpConnMap::disconnect(void const*) /home/abhinav/couchbase/ep-engine/src/connmap.cc:1111 (ep.so+0x00000004969b) #5 EventuallyPersistentEngine::handleDisconnect(void const*) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:6224 (ep.so+0x0000000d3bea) #6 EvpHandleDisconnect(void const*, ENGINE_EVENT_TYPE, void const*, void const*) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:1783 (ep.so+0x0000000b7046) #7 mock_perform_callbacks /home/abhinav/couchbase/memcached/programs/engine_testapp/mock_server.c:296 (engine_testapp+0x0000000bd420) #8 test_rollback_to_zero(engine_interface*, engine_interface_v1*) /home/abhinav/couchbase/ep-engine/tests/ep_testsuite.cc:5434 (ep_testsuite.so+0x00000007f45f) #9 execute_test(test, char const*, char const*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1090 (engine_testapp+0x0000000b946c) #10 __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287 (libc.so.6+0x000000021ec4) (Reviewed-on: http://review.couchbase.org/55785) Change-Id: I287bd95f8b03cb207419d0a0e57ca71be6058b19 Reviewed-on: http://review.couchbase.org/63446 Well-Formed: buildbot <[email protected]> Reviewed-by: Dave Rigby <[email protected]> Tested-by: buildbot <[email protected]> Reviewed-by: Chiyoung Seo <[email protected]>

…stats request WARNING: ThreadSanitizer: data race (pid=82258) Read of size 1 at 0x7d4c0000a208 by thread T11 (mutexes: write M2483, write M19044): #0 DcpProducer::addStats(void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/dcp/producer.cc:601 (ep.so+0x000000063e2d) #1 ConnStatBuilder::operator()(SingleThreadedRCPtr<ConnHandler>&) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep_engine.cc:3903 (ep.so+0x0000000d6931) #2 EventuallyPersistentEngine::doDcpStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep_engine.cc:4160 (ep.so+0x0000000b904a) #3 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep_engine.cc:4580 (ep.so+0x0000000bcba4) #4 EventuallyPersistentStore::snapshotStats() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep.cc:1700 (ep.so+0x000000088386) #5 StatSnap::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/tasks.cc:98 (ep.so+0x00000012ba26) #6 ExecutorThread::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/executorthread.cc:115 (ep.so+0x0000000eaeed) #7 launch_executor_thread(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000eaab5) #8 platform_thread_wrap(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/platform/src/cb_pthreads.cc:53 (libplatform.so.0.1.0+0x0000000048bb) Previous write of size 1 at 0x7d4c0000a208 by main thread: #0 DcpProducer::handleResponse(protocol_binary_response_header*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/dcp/producer.cc:547 (ep.so+0x000000063231) #1 EvpDcpResponseHandler(engine_interface*, void const*, protocol_binary_response_header*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep_engine.cc:1765 (ep.so+0x0000000ae08b) #2 mock_dcp_response_handler(engine_interface*, void const*, protocol_binary_response_header*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/memcached/programs/engine_testapp/engine_testapp.cc:796 (engine_testapp+0x0000004c68e5) #3 sendDcpAck(engine_interface*, engine_interface_v1*, void const*, protocol_binary_command, protocol_binary_response_status, unsigned int) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/tests/ep_test_apis.cc:983 (ep_testsuite.so+0x0000000a6a22) #4 test_dcp_noop(engine_interface*, engine_interface_v1*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/tests/ep_testsuite.cc:3975 (ep_testsuite.so+0x000000068deb) #5 execute_test(test, char const*, char const*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/memcached/programs/engine_testapp/engine_testapp.cc:1090 (engine_testapp+0x0000004c4192) #6 __libc_start_main /build/buildd/eglibc-2.15/csu/libc-start.c:226 (libc.so.6+0x00000002176c) (Reviewed-on: http://review.couchbase.org/56306) Change-Id: Ice7236da5cc885d9e7612894ba3d37e357e13b4a Reviewed-on: http://review.couchbase.org/63440 Well-Formed: buildbot <[email protected]> Reviewed-by: Dave Rigby <[email protected]> Tested-by: buildbot <[email protected]> Reviewed-by: Chiyoung Seo <[email protected]>

As identified by threadSanitizer (see below). Note that the read of the disk highseqno is essentially pointless (it's only used in debug logs), so just remove that to avoid the race. WARNING: ThreadSanitizer: data race (pid=94198) Write of size 8 at 0x7d1000017fd8 by thread T14 (mutexes: write M15079): #0 CouchKVStore::saveDocs(unsigned short, unsigned long, _doc**, _docinfo**, unsigned long, KVStatsCtx&, unsigned long, unsigned long) ep-engine/src/couch-kvstore/couch-kvstore.cc:1804 (ep.so+0x000000325c3a) #1 CouchKVStore::commit2couchstore(Callback<KVStatsCtx>*, unsigned long, unsigned long) ep-engine/src/couch-kvstore/couch-kvstore.cc:1669 (ep.so+0x00000031f94f) #2 CouchKVStore::commit(Callback<KVStatsCtx>*, unsigned long, unsigned long) ep-engine/src/couch-kvstore/couch-kvstore.cc:1080 (ep.so+0x00000031f247) #3 EventuallyPersistentStore::flushVBucket(unsigned short) ep-engine/src/ep.cc:2790 (ep.so+0x0000000edbce) #4 Flusher::flushVB() ep-engine/src/flusher.cc:281 (ep.so+0x0000001ced95) #5 Flusher::step(GlobalTask*) ep-engine/src/flusher.cc:173 (ep.so+0x0000001cdce0) #6 FlusherTask::run() ep-engine/src/tasks.cc:45 (ep.so+0x000000251ffe) ... Previous read of size 8 at 0x7d1000017fd8 by thread T15: #0 CouchKVStore::getLastPersistedSeqno(unsigned short) ep-engine/src/couch-kvstore/couch-kvstore.cc:1091 (ep.so+0x00000031fe25) #1 DCPBackfill::run() ep-engine/src/dcp-stream.cc:149 (ep.so+0x00000028e590) Location is heap block of size 64 at 0x7d1000017fc0 allocated by thread T8: #0 operator new(unsigned long) <null> (engine_testapp+0x00000045c6ed) #1 CouchKVStore::snapshotVBucket(unsigned short, vbucket_state&, Callback<KVStatsCtx>*) ep-engine/src/couch-kvstore/couch-kvstore.cc:888 (ep.so+0x00000031b7f4) #2 EventuallyPersistentStore::snapshotVBuckets(Priority const&, unsigned short) ep-engine/src/ep.cc:970 (ep.so+0x0000000da799) #3 VBSnapshotTask::run() ep-engine/src/tasks.cc:49 (ep.so+0x0000002520be) Change-Id: Iae368af8c2f0eceb67ce50d5e9358af56a75b6b2 Reviewed-on: http://review.couchbase.org/64211 Well-Formed: buildbot <[email protected]> Tested-by: buildbot <[email protected]> Reviewed-by: Will Gardner <[email protected]>

ItemPager::phase is accessed by multiple threads (when different vbuckets are being processed by different executors), so it needs to be atomic. As reported by ThreadSanitizer: WARNING: ThreadSanitizer: data race (pid=180366) Read of size 4 at 0x7d1c0000d650 by thread T18 (mutexes: write M18437): #0 PagingVisitor::visit(StoredValue*) ep-engine/src/item_pager.cc:86 (ep.so+0x000000107734) #1 HashTable::visit(HashTableVisitor&) ep-engine/src/stored-value.cc:475 (ep.so+0x00000011d1b8) #2 VBCBAdaptor::run() ep-engine/src/ep.cc:3929 (ep.so+0x0000000a3c2f) #3 ExecutorThread::run() ep-engine/src/executorthread.cc:115 (ep.so+0x0000000fb086) Previous write of size 4 at 0x7d1c0000d650 by thread T17: #0 PagingVisitor::complete() ep-engine/src/item_pager.cc:179 (ep.so+0x000000108004) #1 VBCBAdaptor::run() ep-engine/src/ep.cc:3937 (ep.so+0x0000000a3d7e) #2 ExecutorThread::run() ep-engine/src/executorthread.cc:115 (ep.so+0x0000000fb086) Change-Id: I9652a9f7f36f18967b6ec5903db53f3c7fb917fe Reviewed-on: http://review.couchbase.org/64161 Tested-by: buildbot <[email protected]> Well-Formed: buildbot <[email protected]> Reviewed-by: Will Gardner <[email protected]>

Fix the data race http://cv.jenkins.couchbase.com/job/ep-engine-threadsanitizer-3.0.x/266/consoleFull WARNING: ThreadSanitizer: data race (pid=109115) Write of size 8 at 0x7d480000b088 by thread T16: #0 ActiveStream::processItems(std::deque<SingleThreadedRCPtr<Item>, std::allocator<SingleThreadedRCPtr<Item> > >&) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/dcp-stream.cc:760 (ep.so+0x000000297e65) #1 ActiveStream::nextCheckpointItemTask() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/dcp-stream.cc:724 (ep.so+0x0000002976ab) #2 ActiveStreamCheckpointProcessorTask::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/dcp-stream.cc:679 (ep.so+0x0000002973ad) #3 ExecutorThread::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/executorthread.cc:109 (ep.so+0x0000001e3fe1) #4 launch_executor_thread(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/executorthread.cc:34 (ep.so+0x0000001e360a) #5 platform_thread_wrap /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/platform/src/cb_pthreads.c (libplatform.so.0.1.0+0x00000000377c) Previous read of size 8 at 0x7d480000b088 by main thread: [failed to restore the stack] SUMMARY: ThreadSanitizer: data race /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/dcp-stream.cc:760 ActiveStream::processItems(std::deque<SingleThreadedRCPtr<Item>, std::allocator<SingleThreadedRCPtr<Item> > >&) Change-Id: I7fa5dd9110342ca836b6b0b0f203dd8b063cf20d Reviewed-on: http://review.couchbase.org/64862 Well-Formed: buildbot <[email protected]> Tested-by: buildbot <[email protected]> Reviewed-by: Dave Rigby <[email protected]>

…ker threads Fix the thread sanitizer warning http://cv.jenkins.couchbase.com/job/ep-engine-threadsanitizer-3.0.x/258/console WARNING: ThreadSanitizer: data race (pid=102290) Read of size 4 at 0x7d580000f71c by thread T14 (mutexes: write M969): #0 void ConnHandler::addStat<unsigned int>(char const*, unsigned int const&, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) const /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/tapconnection.h:294 (ep.so+0x00000020e9f3) #1 DcpProducer::addStats(void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/dcp-producer.cc:557 (ep.so+0x00000027fbe5) #2 ConnStatBuilder::operator()(SingleThreadedRCPtr<ConnHandler>&) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/ep_engine.cc:3701 (ep.so+0x000000183c44) #3 ConnStatBuilder std::for_each<std::_List_iterator<SingleThreadedRCPtr<ConnHandler> >, ConnStatBuilder>(std::_List_iterator<SingleThreadedRCPtr<ConnHandler> >, std::_List_iterator<SingleThreadedRCPtr<ConnHandler> >, ConnStatBuilder) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/stl_algo.h:3755 (ep.so+0x0000001838a5) #4 void ConnMap::each_UNLOCKED<ConnStatBuilder>(ConnStatBuilder) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/connmap.h:148 (ep.so+0x000000183808) #5 void ConnMap::each<ConnStatBuilder>(ConnStatBuilder) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/connmap.h:140 (ep.so+0x00000017732e) #6 EventuallyPersistentEngine::doDcpStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/ep_engine.cc:3954 (ep.so+0x00000014c5ed) Previous write of size 4 at 0x7d580000f71c by main thread: #0 DcpProducer::maybeSendNoop(dcp_message_producers*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/dcp-producer.cc:740 (ep.so+0x00000027d8ce) #1 DcpProducer::step(dcp_message_producers*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/dcp-producer.cc:323 (ep.so+0x00000027c920) #2 EvpDcpStep(engine_interface*, void const*, dcp_message_producers*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-3.0.x/ep-engine/src/ep_engine.cc:1404 (ep.so+0x000000138baa) Change-Id: I2a2b0b0f01b10ecb31701bfc2330881bbafc6b74 Reviewed-on: http://review.couchbase.org/64959 Well-Formed: buildbot <[email protected]> Tested-by: buildbot <[email protected]> Reviewed-by: Dave Rigby <[email protected]>

…sLock As identified by ThreadSanitizer (see below). The issue is that the DcpConsumer::bufMutex and connsLock mutexes are acquired in different orders: A) As part of processing incoming DCP messages (PassiveStream::processBufferedMessages) we acquire (1) bufMutex, then (2) acquire connsLock as part of DcpConnMap::vbucketStateChanged. B) When disconnecting a connection from the DcpConnMap, we acquire (1) connsLock to locate the connection to be closed, and then (2) acquire bufMutex as part of PassiveStream::setDead. Address this by changing (B) - update the data structures maintaining the map of cookie -> connection (`all` and `map_`), *release connsLock* and then call PassiveStream::setDead. Finally we re-acquire connsLock to add the (now closed) stream to the deadConnections list. WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=393) Cycle in lock order graph: M22701 (0x7d5000018140) => M969 (0x7d840001cc50) => M22701 Mutex M969 acquired here while holding mutex M22701 in thread T10: #0 pthread_mutex_lock <null> (engine_testapp+0x00000047e980) #1 cb_mutex_enter <null> (libplatform.so.0.1.0+0x000000003960) #2 Mutex::acquire() ep-engine/src/mutex.cc:31 (ep.so+0x0000001e60ce) #3 LockHolder::lock() ep-engine/src/locks.h:71 (ep.so+0x000000080b33) #4 LockHolder::LockHolder(Mutex&, bool) ep-engine/src/locks.h:48 (ep.so+0x0000000807a2) #5 DcpConnMap::vbucketStateChanged(unsigned short, vbucket_state_t) ep-engine/src/connmap.cc:1044 (ep.so+0x00000023b3ba) #6 EventuallyPersistentStore::setVBucketState(unsigned short, vbucket_state_t, bool, bool) ep-engine/src/ep.cc:1057 (ep.so+0x0000000dc240) #7 PassiveStream::processSetVBucketState(SetVBucketState*) ep-engine/src/dcp-stream.cc:1483 (ep.so+0x0000002a3125) #8 PassiveStream::processBufferedMessages(unsigned int&) ep-engine/src/dcp-stream.cc:1313 (ep.so+0x0000002a0b58) #9 DcpConsumer::processBufferedItems() ep-engine/src/dcp-consumer.cc:608 (ep.so+0x0000002650c4) #10 Processer::run() ep-engine/src/dcp-consumer.cc:48 (ep.so+0x000000264cbf) #11 ExecutorThread::run() ep-engine/src/executorthread.cc:109 (ep.so+0x0000001e75a1) #12 launch_executor_thread(void*) ep-engine/src/executorthread.cc:34 (ep.so+0x0000001e6bca) #13 platform_thread_wrap platform/src/cb_pthreads.c (libplatform.so.0.1.0+0x00000000371c) Mutex M22701 previously acquired by the same thread here: #0 pthread_mutex_lock <null> (engine_testapp+0x00000047e980) #1 cb_mutex_enter <null> (libplatform.so.0.1.0+0x000000003960) #2 Mutex::acquire() ep-engine/src/mutex.cc:31 (ep.so+0x0000001e60ce) #3 LockHolder::lock() ep-engine/src/locks.h:71 (ep.so+0x000000080b33) #4 LockHolder::LockHolder(Mutex&, bool) ep-engine/src/locks.h:48 (ep.so+0x0000000807a2) #5 PassiveStream::processBufferedMessages(unsigned int&) ep-engine/src/dcp-stream.cc:1286 (ep.so+0x0000002a085d) #6 DcpConsumer::processBufferedItems() ep-engine/src/dcp-consumer.cc:608 (ep.so+0x0000002650c4) #7 Processer::run() ep-engine/src/dcp-consumer.cc:48 (ep.so+0x000000264cbf) #8 ExecutorThread::run() ep-engine/src/executorthread.cc:109 (ep.so+0x0000001e75a1) #9 launch_executor_thread(void*) ep-engine/src/executorthread.cc:34 (ep.so+0x0000001e6bca) #10 platform_thread_wrap platform/src/cb_pthreads.c (libplatform.so.0.1.0+0x00000000371c) Mutex M22701 acquired here while holding mutex M969 in main thread: #0 pthread_mutex_lock <null> (engine_testapp+0x00000047e980) #1 cb_mutex_enter <null> (libplatform.so.0.1.0+0x000000003960) #2 Mutex::acquire() ep-engine/src/mutex.cc:31 (ep.so+0x0000001e60ce) #3 LockHolder::lock() ep-engine/src/locks.h:71 (ep.so+0x000000080b33) #4 LockHolder::LockHolder(Mutex&, bool) ep-engine/src/locks.h:48 (ep.so+0x0000000807a2) #5 PassiveStream::clearBuffer() ep-engine/src/dcp-stream.cc:1573 (ep.so+0x00000029e6a1) #6 PassiveStream::setDead(end_stream_status_t) ep-engine/src/dcp-stream.cc:1171 (ep.so+0x00000029dffc) #7 DcpConsumer::closeAllStreams() ep-engine/src/dcp-consumer.cc:722 (ep.so+0x00000026fb66) #8 DcpConnMap::disconnect_UNLOCKED(void const*) ep-engine/src/connmap.cc:1096 (ep.so+0x00000023bbe3) #9 DcpConnMap::disconnect(void const*) ep-engine/src/connmap.cc:1069 (ep.so+0x00000023b664) #10 EventuallyPersistentEngine::handleDisconnect(void const*) ep-engine/src/ep_engine.cc:5711 (ep.so+0x0000001617ab) #11 EvpHandleDisconnect(void const*, ENGINE_EVENT_TYPE, void const*, void const*) ep-engine/src/ep_engine.cc:1702 (ep.so+0x00000013d415) #12 mock_perform_callbacks memcached/programs/engine_testapp/mock_server.c (engine_testapp+0x0000004d04db) #13 disconnect_mock_connection <null> (engine_testapp+0x0000004d10c6) #14 destroy_mock_cookie <null> (engine_testapp+0x0000004d0fa7) #15 test_mb19982(engine_interface*, engine_interface_v1*) ep-engine/tests/ep_testsuite.cc:12020 (ep_testsuite.so+0x0000000b1d7d) #16 execute_test memcached/programs/engine_testapp/engine_testapp.c (engine_testapp+0x0000004c50bf) #17 main crtstuff.c (engine_testapp+0x0000004c2ea8) Mutex M969 previously acquired by the same thread here: #0 pthread_mutex_lock <null> (engine_testapp+0x00000047e980) #1 cb_mutex_enter <null> (libplatform.so.0.1.0+0x000000003960) #2 Mutex::acquire() ep-engine/src/mutex.cc:31 (ep.so+0x0000001e60ce) #3 LockHolder::lock() ep-engine/src/locks.h:71 (ep.so+0x000000080b33) #4 LockHolder::LockHolder(Mutex&, bool) ep-engine/src/locks.h:48 (ep.so+0x0000000807a2) #5 DcpConnMap::disconnect(void const*) ep-engine/src/connmap.cc:1068 (ep.so+0x00000023b64e) #6 EventuallyPersistentEngine::handleDisconnect(void const*) ep-engine/src/ep_engine.cc:5711 (ep.so+0x0000001617ab) #7 EvpHandleDisconnect(void const*, ENGINE_EVENT_TYPE, void const*, void const*) ep-engine/src/ep_engine.cc:1702 (ep.so+0x00000013d415) #8 mock_perform_callbacks memcached/programs/engine_testapp/mock_server.c (engine_testapp+0x0000004d04db) #9 disconnect_mock_connection <null> (engine_testapp+0x0000004d10c6) #10 destroy_mock_cookie <null> (engine_testapp+0x0000004d0fa7) #11 test_mb19982(engine_interface*, engine_interface_v1*) ep-engine/tests/ep_testsuite.cc:12020 (ep_testsuite.so+0x0000000b1d7d) #12 execute_test memcached/programs/engine_testapp/engine_testapp.c (engine_testapp+0x0000004c50bf) #13 main crtstuff.c (engine_testapp+0x0000004c2ea8) SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) ??:0 __interceptor_pthread_mutex_lock Change-Id: Ia441d5f5898516e3526a610426fa81f5df0e35e6 Reviewed-on: http://review.couchbase.org/65297 Well-Formed: buildbot <[email protected]> Tested-by: buildbot <[email protected]> Reviewed-by: Jim Walker <[email protected]>

Regression test for MB-20054 - the following abort is encountered when a DCPBackfill task is still running when a bucket is deleted: Assertion failed: (engine), function verifyEngine, file ep-engine/src/objectregistry.cc, line 58. This issue occurs because the DCPBackfill object (and associated objects ActiveStream and importantly ActiveStreams' readyQ of Items) is not deleted earlier in the shutdown sequence (via EvpDestroy), as we use ref-counted pointers for it and there is a still an outstanding reference by the AuxIO Thread which is running the task. Hence the DCPBackfill object is only deleted when we finally unregister the deleted bucket from the shared ExecutorPool - see the following backtrace: #1 0x00007f513b75a085 in abort () from /lib64/libc.so.6 #2 0x00007f51337034e2 in ObjectRegistry::onDeleteItem (pItem=<value optimized out>) at ep-engine/src/objectregistry.cc:157 #3 0x00007f5133652094 in ~Item (this=<value optimized out>) at ep-engine/src/item.h:352 #4 SingleThreadedRCPtr<Item>::~SingleThreadedRCPtr (this=<value optimized out>) at ep-engine/src/atomic.h:430 #5 0x00007f51336c7f47 in ~MutationResponse (this=0x3cd87880) at ep-engine/src/dcp-response.h:275 #6 MutationResponse::~MutationResponse (this=0x3cd87880) at ep-engine/src/dcp-response.h:275 #7 0x00007f51336d86aa in clear_UNLOCKED (this=0x7a3f5fa0) at ep-engine/src/dcp-stream.cc:201 #8 ~ActiveStream (this=0x7a3f5fa0) at ep-engine/src/dcp-stream.h:178 #9 ActiveStream::~ActiveStream (this=0x7a3f5fa0) at ep-engine/src/dcp-stream.h:179 #10 0x00007f51336cc808 in RCPtr<Stream>::~RCPtr (this=0xb1823780) at ep-engine/src/atomic.h:348 #11 0x00007f51336d77c7 in ~DCPBackfill (this=0xb1823740) at ep-engine/src/dcp-stream.cc:114 #12 DCPBackfill::~DCPBackfill (this=0xb1823740) at ep-engine/src/dcp-stream.cc:114 #13 0x00007f513368d95f in ~SingleThreadedRCPtr (this=0x5b55a20, e=0x59c4000, taskType=NO_TASK_TYPE) at ep-engine/src/atomic.h:430 #14 ExecutorPool::_stopTaskGroup (this=0x5b55a20, e=0x59c4000, taskType=NO_TASK_TYPE) at ep-engine/src/executorpool.cc:532 #15 0x00007f513368dad3 in ExecutorPool::_unregisterBucket (this=0x5b55a20, engine=0x59c4000) at ep-engine/src/executorpool.cc:551 #16 0x00007f513368e143 in ExecutorPool::unregisterBucket (this=0x5b55a20, engine=0x59c4000) at ep-engine/src/executorpool.cc:602 #17 0x00007f5133655f82 in EventuallyPersistentStore::~EventuallyPersistentStore (this=0x59e6000) at ep-engine/src/ep.cc:365 #18 0x00007f5133672a25 in EventuallyPersistentEngine::~EventuallyPersistentEngine (this=0x59c4000) at ep-engine/src/ep_engine.cc:5791 #19 0x00007f5133672c95 in EvpDestroy (handle=0x59c4000, force=<value optimized out>) at ep-engine/src/ep_engine.cc:143 To actually reproduce the issue is somewhat involved - we need to orchestrate the world such that we delete the engine while a DCPBackfill task is still running. We spin up a separate thread which will run the DCPBackfill task concurrently with destroy - specifically DCPBackfill must start running (and add items to the readyQ) before destroy(), it must then continue running (stop after) _stopTaskGroup is invoked. To achieve this we use a couple of condition variables to synchronise between the two threads - the timeline needs to look like: auxIO thread: [------- DCPBackfill ----------] main thread: [--destroy()--] [ExecutorPool::_stopTaskGroup] --------------------------------------------------------> time Change-Id: Ic64c419cb8e4e0af2378efba9711b121aacee15b Reviewed-on: http://review.couchbase.org/65520 Well-Formed: buildbot <[email protected]> Tested-by: buildbot <[email protected]> Reviewed-by: Jim Walker <[email protected]>

================== WARNING: ThreadSanitizer: data race (pid=49270) Read of size 8 at 0x7d900001c000 by thread T4: #0 vbucket_state** std::__uninitialized_copy<false>::__uninit_copy<__gnu_cxx::__normal_iterator<vbucket_state* const*, std::vector<vbucket_state*, std::allocator<vbucket_state*> > >, vbucket_state**>(__gnu_cxx::__normal_iterator<vbucket_state* const*, std::vector<vbucket_state*, std::allocator<vbucket_state*> > >, __gnu_cxx::__normal_iterator<vbucket_state* const*, std::vector<vbucket_state*, std::allocator<vbucket_state*> > >, vbucket_state**) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/stl_construct.h:75 (ep-engine_ep_unit_tests+0x0000006e3980) #1 Warmup::populateShardVbStates() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/warmup.cc:1030 (ep-engine_ep_unit_tests+0x00000067999a) #2 Warmup::initialize() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/warmup.cc:459 (ep-engine_ep_unit_tests+0x0000006797ec) Previous write of size 8 at 0x7d900001c000 by thread T12 (mutexes: write M119516, write M119521): #0 KVStore::updateCachedVBState(unsigned short, vbucket_state const&) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/kvstore.cc:122 (ep-engine_ep_unit_tests+0x0000006ade1c) #1 ForestKVStore::snapshotVBucket(unsigned short, vbucket_state&, VBStatePersist) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/forest-kvstore/forest-kvstore.cc:1413 (ep-engine_ep_unit_tests+0x0000006e4249) #2 EventuallyPersistentStore::persistVBState(unsigned short) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep.cc:1355 (ep-engine_ep_unit_tests+0x0000005b0fb8) #3 VBStatePersistTask::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/tasks.cc:55 (ep-engine_ep_unit_tests+0x000000660a55) #4 ExecutorThread::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/executorthread.cc:113 (ep-engine_ep_unit_tests+0x000000616ef1) #5 launch_executor_thread(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/executorthread.cc:31 (ep-engine_ep_unit_tests+0x000000616aa5) #6 platform_thread_wrap(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/platform/src/cb_pthreads.cc:54 (libplatform.so.0.1.0+0x00000000586b) Change-Id: I4e807ac9a23baf9e4d352f13ef65f3f5a345cd60 Reviewed-on: http://review.couchbase.org/66050 Tested-by: buildbot <[email protected]> Reviewed-by: Dave Rigby <[email protected]>

As identified by TSan. Seen whilst testing sherlock->watson merge, analysed the code and it seems this is a latent issue and hard to re-produce. The issue is that when the executor thread does a reset on the current task, the VBCBAdapator is the last one holding the ref-counted vbucket, so destruction occurs and ~VBucket calls the destructor of the checkpoint manager, which is the reverse locks ordering of a previous code path. Number of threads in play here, but main ones of interest: WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=170834) Cycle in lock order graph: M_checkpoint (0x7d640002e9a8) => M_exepool (0x7d4c00008288) => M_taskqueue (0x7d4400008e80) => M_exethread (0x7d380000df60) => M_checkpoint Mutex M_exepool acquired here while holding mutex M_checkpoint in thread T35: #0 pthread_mutex_lock <null> (engine_testapp+0x000000486760) #1 std::mutex::lock() /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/x86_64-linux-gnu/c++/4.9/bits/gthr-default.h:748 (ep.so+0x0000000f47b0) #2 ExecutorPool::wake(unsigned long) ep-engine/src/executorpool.cc:355 (ep.so+0x0000000f48f1) #3 Flusher::wake() ep-engine/src/flusher.cc:155 (ep.so+0x000000101ee6) #4 NotifyFlusherCB::callback(unsigned short&) ep-engine/src/flusher.h:88 (ep.so+0x00000010d194) #5 Checkpoint::queueDirty(SingleThreadedRCPtr<Item> const&, CheckpointManager*) ep-engine/src/checkpoint.h:675 (ep.so+0x0000000271b0) #6 CheckpointManager::closeOpenCheckpoint_UNLOCKED() ep-engine/src/checkpoint.cc:454 (ep.so+0x000000028dcb) #7 CheckpointManager::addNewCheckpoint_UNLOCKED(unsigned long, unsigned long, unsigned long) ep-engine/src/checkpoint.cc:371 (ep.so+0x00000002881f) #8 CheckpointManager::checkOpenCheckpoint_UNLOCKED(bool, bool) ep-engine/src/checkpoint.cc:361 (ep.so+0x00000002bd71) #9 CheckpointVisitor::visitBucket(RCPtr<VBucket>&) ep-engine/src/checkpoint_remover.cc:43 (ep.so+0x00000003c3bd) #10 VBCBAdaptor::run() ep-engine/src/ep.cc:3924 (ep.so+0x0000000a6174) #11 ExecutorThread::run() ep-engine/src/executorthread.cc:115 (ep.so+0x0000000fe1b6) #12 launch_executor_thread(void*) ep-engine/src/executorthread.cc:33 (ep.so+0x0000000fdd15) #13 platform_thread_wrap(void*) platform/src/cb_pthreads.cc:54 (libplatform.so.0.1.0+0x0000000057fb) ... Mutex M_checkpoint acquired here while holding mutex M_exethread in thread T36: #0 pthread_mutex_lock <null> (engine_testapp+0x000000486760) #1 CheckpointManager::~CheckpointManager() /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/x86_64-linux-gnu/c++/4.9/bits/gthr-default.h:748 (ep.so+0x000000027fdd) #2 VBucket::~VBucket() ep-engine/src/vbucket.cc:152 (ep.so+0x00000014a018) #3 PagingVisitor::~PagingVisitor() ep-engine/src/atomic.h:190 (ep.so+0x00000010a5e6) #4 PagingVisitor::~PagingVisitor() ep-engine/src/item_pager.cc:43 (ep.so+0x00000010a645) #5 std::_Sp_counted_ptr<PagingVisitor*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/shared_ptr_base.h:373 (ep.so+0x00000010a2b0) #6 VBCBAdaptor::~VBCBAdaptor() /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/shared_ptr_base.h:149 (ep.so+0x0000000aea7e) #7 ExecutorThread::run() ep-engine/src/atomic.h:325 (ep.so+0x0000000fdee4) #8 launch_executor_thread(void*) ep-engine/src/executorthread.cc:33 (ep.so+0x0000000fdd15) #9 platform_thread_wrap(void*) platform/src/cb_pthreads.cc:54 (libplatform.so.0.1.0+0x0000000057fb) Change-Id: I0a966b3d112963243e17647184123fd8b3200656 Reviewed-on: http://review.couchbase.org/66357 Well-Formed: buildbot <[email protected]> Tested-by: buildbot <[email protected]> Reviewed-by: Will Gardner <[email protected]> Reviewed-by: Jim Walker <[email protected]>

Address race condition identified by ThreadSanitizer on BloomFilter::keyCounter. WARNING: ThreadSanitizer: data race (pid=341) Write of size 8 at 0x7d140000df80 by thread T11 (mutexes: write M20582, write M20817, write M19369): #0 BloomFilter::addKey() ep-engine/src/bloomfilter.cc:119 (ep.so+0x00000002453d) #1 VBucket::addToFilter() ep-engine/src/vbucket.cc:482 (ep.so+0x000000149704) #2 PersistenceCallback::callback() ep-engine/src/ep.cc:3211 (ep.so+0x0000000b27c1) #3 non-virtual thunk to PersistenceCallback::callback(int&) ep-engine/src/ep.cc:3167 (ep.so+0x0000000b36c2) #4 CouchKVStore::commitCallback() ep-engine/src/couch-kvstore/couch-kvstore.cc:1898 (ep.so+0x00000017aa51) #5 _ZN12CouchKVStore17commit2couchstoreEP8CallbackIJ10KVStatsCtxEE ep-engine/src/couch-kvstore/couch-kvstore.cc:1712 (ep.so+0x00000017208b) #6 _ZN12CouchKVStore6commitEP8CallbackIJ10KVStatsCtxEE ep-engine/src/couch-kvstore/couch-kvstore.cc:954 (ep.so+0x000000171c92) #7 EventuallyPersistentStore::commit(unsigned short) ep-engine/src/ep.cc:3451 (ep.so+0x0000000a29e6) #8 EventuallyPersistentStore::flushVBucket(unsigned short) ep-engine/src/ep.cc:3399 (ep.so+0x0000000a1935) #9 Flusher::flushVB() ep-engine/src/flusher.cc:293 (ep.so+0x0000001038ae) #10 Flusher::step(GlobalTask*) ep-engine/src/flusher.cc:183 (ep.so+0x000000101f7e) #11 FlusherTask::run() ep-engine/src/tasks.cc:138 (ep.so+0x00000013ac42) #12 ExecutorThread::run() ep-engine/src/executorthread.cc:115 (ep.so+0x0000000fd53f) #13 launch_executor_thread(void*) ep-engine/src/executorthread.cc:33 (ep.so+0x0000000fd095) #14 platform_thread_wrap(void*) platform/src/cb_pthreads.cc:55 (libplatform_so.so.0.1.0+0x000000005deb) Previous read of size 8 at 0x7d140000df80 by main thread (mutexes: write M15538, write M1828880534441356728): #0 BloomFilter::getNumOfKeysInFilter() ep-engine/src/bloomfilter.cc:142 (ep.so+0x0000000246a0) #1 VBucket::addStats() ep-engine/src/vbucket.cc:597 (ep.so+0x00000014a527) #2 EventuallyPersistentEngine::doVBucketStats() ep-engine/src/ep_engine.cc:3837 (ep.so+0x0000000c69f6) #3 EventuallyPersistentEngine::doVBucketStats() ep-engine/src/ep_engine.cc:3861 (ep.so+0x0000000c664a) #4 EventuallyPersistentEngine::getStats() ep-engine/src/ep_engine.cc:4774 (ep.so+0x0000000cbbb3) #5 EvpGetStats() ep-engine/src/ep_engine.cc:231 (ep.so+0x0000000baafe) #6 mock_get_stats() memcached/programs/engine_testapp/engine_testapp.cc:215 (engine_testapp+0x0000004ce40d) #7 std::string get_stat<std::string>() ep-engine/tests/ep_test_apis.cc:1140 (ep_testsuite_xdcr.so+0x000000020c34) #8 unsigned long get_stat<unsigned long>() ep-engine/tests/ep_test_apis.cc:1179 (ep_testsuite_xdcr.so+0x00000002089b) #9 get_ull_stat() ep-engine/tests/ep_test_apis.cc:1174 (ep_testsuite_xdcr.so+0x00000001e73e) #10 test_del_with_meta_and_check_drift_stats() ep-engine/tests/ep_testsuite_xdcr.cc:1711 (ep_testsuite_xdcr.so+0x0000000143f1) #11 execute_test() memcached/programs/engine_testapp/engine_testapp.cc:1118 (engine_testapp+0x0000004cc997) #12 __libc_start_main /build/buildd/eglibc-2.15/csu/libc-start.c:226 (libc.so.6+0x00000002176c) Change-Id: I4e9a589e9286e72646df515db891c57b143d1fdd Reviewed-on: http://review.couchbase.org/68803 Tested-by: buildbot <[email protected]> Reviewed-by: Daniel Owen <[email protected]>

When developing additional stats test for ep-engine_perfsuite, the following data race was uncovered. The patch ensures that accesses to vals are protected with a mutex. Also get_stat and get_histo_stat can only be called one at a time as they use the three global variables (requested_stat_name, actual_stat_value and histogram_stat_int_value). Therefore a secondary mutex is required to enforce this. It has been confirmed that with this patch the ThreadSanitizer issue no longer occurs. 30: WARNING: ThreadSanitizer: data race (pid=42065) 30: Read of size 8 at 0x7d1800060a38 by thread T18 (mutexes: write M1704): 30: #0 std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_S_right(std::_Rb_tree_node_base*) <null> (engine_testapp+0x00000049c503) 30: #1 std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_M_lower_bound(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*, std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (engine_testapp+0x0000004a30db) 30: #2 std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::lower_bound(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (engine_testapp+0x00000049c5f6) 30: #3 std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::lower_bound(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (engine_testapp+0x000000499047) 30: #4 std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::operator[](std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (engine_testapp+0x00000048e3ba) 30: #5 add_stats /home/owend/master/ep-engine/tests/ep_test_apis.cc:236 (ep_perfsuite.so+0x00000008f5b9) 30: #6 STATWRITER_NAMESPACE::add_casted_stat(char const*, char const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/owend/master/ep-engine/src/statwriter.h:39 (ep.so+0x0000000e5673) 30: #7 StatCheckpointVisitor::addCheckpointStat(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), EventuallyPersistentStore*, RCPtr<VBucket>&) <null> (ep.so+0x0000001c120c) 30: #8 StatCheckpointVisitor::visitBucket(RCPtr<VBucket>&) <null> (ep.so+0x0000001c10e5) 30: #9 EventuallyPersistentStore::visit(VBucketVisitor&) /home/owend/master/ep-engine/src/ep.cc:3796 (ep.so+0x000000174559) 30: #10 StatCheckpointTask::run() <null> (ep.so+0x0000001c155b) 30: #11 ExecutorThread::run() /home/owend/master/ep-engine/src/executorthread.cc:113 (ep.so+0x0000001ea96d) 30: #12 launch_executor_thread /home/owend/master/ep-engine/src/executorthread.cc:31 (ep.so+0x0000001ea0d0) 30: #13 CouchbaseThread::run() /home/owend/master/platform/src/cb_pthreads.cc:58 (libplatform_so.so.0.1.0+0x00000000c9c7) 30: #14 platform_thread_wrap /home/owend/master/platform/src/cb_pthreads.cc:71 (libplatform_so.so.0.1.0+0x00000000affe) 30: #15 <null> <null> (libtsan.so.0+0x0000000230d9) 30: 30: Previous write of size 8 at 0x7d1800060a38 by main thread (mutexes: write M1575): 30: #0 operator new(unsigned long) <null> (libtsan.so.0+0x000000025a33) 30: #1 __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::allocate(unsigned long, void const*) <null> (engine_testapp+0x0000004a6ca7) 30: #2 std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >::allocate(std::allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, unsigned long) <null> (engine_testapp+0x0000004a5168) 30: #3 std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_M_get_node() <null> (engine_testapp+0x0000004a316f) 30: #4 std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >* std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_M_create_node<std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>, std::tuple<> >(std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&, std::tuple<>&&) <null> (engine_testapp+0x00000049c6b4) 30: #5 std::_Rb_tree_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_M_emplace_hint_unique<std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>, std::tuple<> >(std::_Rb_tree_const_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&, std::tuple<>&&) <null> (engine_testapp+0x00000049928e) 30: #6 std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::operator[](std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (engine_testapp+0x00000048e488) 30: #7 add_stats /home/owend/master/ep-engine/tests/ep_test_apis.cc:236 (ep_perfsuite.so+0x00000008f5b9) 30: #8 STATWRITER_NAMESPACE::add_casted_stat(char const*, char const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/owend/master/ep-engine/src/statwriter.h:39 (ep.so+0x0000000e5673) 30: #9 void STATWRITER_NAMESPACE::add_casted_stat<unsigned long>(char const*, unsigned long const&, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/owend/master/ep-engine/src/statwriter.h:48 (ep.so+0x0000000e96ae) 30: #10 EventuallyPersistentEngine::addSeqnoVbStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), RCPtr<VBucket> const&) /home/owend/master/ep-engine/src/ep_engine.cc:4445 (ep.so+0x0000001ad56d) 30: #11 EventuallyPersistentEngine::doSeqnoStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), char const*, int) /home/owend/master/ep-engine/src/ep_engine.cc:4490 (ep.so+0x0000001ad95f) 30: #12 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/owend/master/ep-engine/src/ep_engine.cc:4595 (ep.so+0x0000001ae4de) 30: #13 EvpGetStats /home/owend/master/ep-engine/src/ep_engine.cc:232 (ep.so+0x00000019851e) 30: #14 ENGINE_ERROR_CODE std::_Bind<ENGINE_ERROR_CODE (*(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)))(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*))>::__call<ENGINE_ERROR_CODE, , 0ul, 1ul, 2ul, 3ul, 4ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul, 2ul, 3ul, 4ul>) <null> (engine_testapp+0x00000049f211) 30: #15 ENGINE_ERROR_CODE std::_Bind<ENGINE_ERROR_CODE (*(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)))(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*))>::operator()<, ENGINE_ERROR_CODE>() <null> (engine_testapp+0x00000049aa02) 30: #16 std::_Function_handler<ENGINE_ERROR_CODE (), std::_Bind<ENGINE_ERROR_CODE (*(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)))(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*))> >::_M_invoke(std::_Any_data const&) <null> (engine_testapp+0x00000049145b) 30: #17 std::function<ENGINE_ERROR_CODE ()>::operator()() const <null> (engine_testapp+0x000000484e28) 30: #18 call_engine_and_handle_EWOULDBLOCK /home/owend/master/memcached/programs/engine_testapp/engine_testapp.cc:127 (engine_testapp+0x00000047cf00) 30: #19 mock_get_stats /home/owend/master/memcached/programs/engine_testapp/engine_testapp.cc:220 (engine_testapp+0x00000047d896) 30: #20 perf_stat_latency_core /home/owend/master/ep-engine/tests/ep_perfsuite.cc:1238 (ep_perfsuite.so+0x000000063d3d) 30: #21 perf_stat_latency /home/owend/master/ep-engine/tests/ep_perfsuite.cc:1306 (ep_perfsuite.so+0x00000006427a) 30: #22 perf_slow_stat_latency_100vb /home/owend/master/ep-engine/tests/ep_perfsuite.cc:1365 (ep_perfsuite.so+0x00000006498c) 30: #23 execute_test /home/owend/master/memcached/programs/engine_testapp/engine_testapp.cc:962 (engine_testapp+0x000000481776) 30: #24 main /home/owend/master/memcached/programs/engine_testapp/engine_testapp.cc:1359 (engine_testapp+0x000000482ab1) Change-Id: I7bdc847c0913244409fa044e312d53b484dc2dab Reviewed-on: http://review.couchbase.org/68813 Reviewed-by: Dave Rigby <[email protected]> Tested-by: buildbot <[email protected]>

…kfills The existing code was safe, however ThreadSanitizer doesn't know about our own Spinlocks. Given this shouldn't be a hot path, switch to normal std:mutex. ThreadSanitizer report: WARNING: ThreadSanitizer: data race (pid=23569) Read of size 2 at 0x7d840000eca2 by thread T8 (mutexes: write M294, read M27095, write M66205, write M102676, write M95235): #0 DcpConnMap::canAddBackfillToActiveQ() ep-engine/src/connmap.cc:1308 (ep.so+0x000000045ac5) #1 BackfillManager::schedule() ep-engine/src/dcp/backfill-manager.cc:142 (ep.so+0x00000005b0eb) #2 DcpProducer::scheduleBackfillManager() ep-engine/src/dcp/producer.cc:702 (ep.so+0x000000078fe3) #3 ActiveStream::scheduleBackfill_UNLOCKED(bool) ep-engine/src/dcp/stream.cc:1016 (ep.so+0x00000008f280) #4 ActiveStream::transitionState(stream_state_t) ep-engine/src/dcp/stream.cc:1145 (ep.so+0x000000090589) #5 ActiveStream::setActive() ep-engine/src/dcp/stream.h:204 (ep.so+0x00000009958e) #6 DcpProducer::streamRequest() ep-engine/src/dcp/producer.cc:327 (ep.so+0x00000007f85d) #7 EvpDcpStreamReq ep-engine/src/ep_engine.cc:1573 (ep.so+0x0000000cea78) #8 dcp_stream_req_executor memcached/daemon/mcbp_executors.cc:2272 (memcached+0x00000045925c) #9 process_bin_packet memcached/daemon/mcbp_executors.cc:4650 (memcached+0x00000046481d) #10 mcbp_complete_nread(McbpConnection*) memcached/daemon/mcbp_executors.cc:4759 (memcached+0x00000046481d) #11 conn_nread(McbpConnection*) memcached/daemon/statemachine_mcbp.cc:314 (memcached+0x000000472678) #12 McbpStateMachine::execute(McbpConnection&) memcached/daemon/statemachine_mcbp.h:43 (memcached+0x000000447054) #13 McbpConnection::runStateMachinery() memcached/daemon/connection_mcbp.cc:1003 (memcached+0x000000447054) #14 McbpConnection::runEventLoop(short) memcached/daemon/connection_mcbp.cc:1274 (memcached+0x0000004470dd) #15 run_event_loop memcached/daemon/connections.cc:147 (memcached+0x00000044b9e9) #16 event_handler(int, short, void*) memcached/daemon/memcached.cc:851 (memcached+0x00000041466c) #17 event_persist_closure src/libevent/event.c:1319 (libevent_core-2.0.so.5+0x00000000b6b7) #18 event_process_active_single_queue src/libevent/event.c:1363 (libevent_core-2.0.so.5+0x00000000b6b7) #19 event_process_active src/libevent/event.c:1438 (libevent_core-2.0.so.5+0x00000000b6b7) #20 event_base_loop src/libevent/event.c:1639 (libevent_core-2.0.so.5+0x00000000b6b7) #21 CouchbaseThread::run() platform/src/cb_pthreads.cc:54 (libplatform.so.0.1.0+0x0000000057a5) #22 platform_thread_wrap platform/src/cb_pthreads.cc:66 (libplatform.so.0.1.0+0x0000000057a5) Previous write of size 2 at 0x7d840000eca2 by thread T55: #0 DcpConnMap::decrNumActiveSnoozingBackfills() ep-engine/src/connmap.cc:1319 (ep.so+0x000000045b7b) #1 BackfillManager::backfill() ep-engine/src/dcp/backfill-manager.cc:273 (ep.so+0x00000005a783) #2 BackfillManagerTask::run() ep-engine/src/dcp/backfill-manager.cc:62 (ep.so+0x00000005ac1c) #3 ExecutorThread::run() ep-engine/src/executorthread.cc:115 (ep.so+0x000000108d96) #4 launch_executor_thread ep-engine/src/executorthread.cc:33 (ep.so+0x000000109675) #5 CouchbaseThread::run() platform/src/cb_pthreads.cc:54 (libplatform.so.0.1.0+0x0000000057a5) #6 platform_thread_wrap platform/src/cb_pthreads.cc:66 (libplatform.so.0.1.0+0x0000000057a5) Change-Id: I88df57b268c1e615b7c5d2b7caf5f038a53692ba Reviewed-on: http://review.couchbase.org/69235 Reviewed-by: Trond Norbye <[email protected]> Tested-by: buildbot <[email protected]>

As identified by ThreadSantizer: WARNING: ThreadSanitizer: data race (pid=23560) Write of size 8 at 0x7ff76a2e7e50 by thread T52 (mutexes: write M64345, write M85935): #0 VBucket::adjustCheckpointFlushTimeout(unsigned long) ep-engine/src/vbucket.cc:393 (ep.so+0x00000016c806) #1 VBucket::notifyOnPersistence(EventuallyPersistentEngine&, unsigned long, bool) ep-engine/src/vbucket.cc:332 (ep.so+0x00000016c806) #2 EventuallyPersistentStore::flushVBucket(unsigned short) ep-engine/src/ep.cc:3475 (ep.so+0x0000000be736) #3 Flusher::flushVB() ep-engine/src/flusher.cc:284 (ep.so+0x0000001100d1) #4 Flusher::step(GlobalTask*) ep-engine/src/flusher.cc:183 (ep.so+0x0000001110b7) #5 FlusherTask::run() ep-engine/src/tasks.cc:138 (ep.so+0x000000156642) #6 ExecutorThread::run() ep-engine/src/executorthread.cc:115 (ep.so+0x000000108d96) #7 launch_executor_thread ep-engine/src/executorthread.cc:33 (ep.so+0x000000109675) #8 CouchbaseThread::run() platform/src/cb_pthreads.cc:54 (libplatform.so.0.1.0+0x0000000057a5) #9 platform_thread_wrap platform/src/cb_pthreads.cc:66 (libplatform.so.0.1.0+0x0000000057a5) Previous read of size 8 at 0x7ff76a2e7e50 by thread T9 (mutexes: write M295): #0 VBucket::getCheckpointFlushTimeout() ep-engine/src/vbucket.cc:406 (ep.so+0x0000001683e6) #1 EventuallyPersistentEngine::doEngineStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) ep-engine/src/ep_engine.cc:3628 (ep.so+0x0000000e9678) #2 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) ep-engine/src/ep_engine.cc:4680 (ep.so+0x0000000ea930) #3 EvpGetStats ep-engine/src/ep_engine.cc:235 (ep.so+0x0000000eb9fe) #4 stat_executor memcached/daemon/mcbp_executors.cc:3367 (memcached+0x00000046c94b) #5 process_bin_packet memcached/daemon/mcbp_executors.cc:4650 (memcached+0x00000046481d) #6 mcbp_complete_nread(McbpConnection*) memcached/daemon/mcbp_executors.cc:4759 (memcached+0x00000046481d) #7 conn_nread(McbpConnection*) memcached/daemon/statemachine_mcbp.cc:314 (memcached+0x000000472678) #8 McbpStateMachine::execute(McbpConnection&) memcached/daemon/statemachine_mcbp.h:43 (memcached+0x000000447054) #9 McbpConnection::runStateMachinery() memcached/daemon/connection_mcbp.cc:1003 (memcached+0x000000447054) #10 McbpConnection::runEventLoop(short) memcached/daemon/connection_mcbp.cc:1274 (memcached+0x0000004470dd) #11 run_event_loop memcached/daemon/connections.cc:147 (memcached+0x00000044b9e9) #12 event_handler(int, short, void*) memcached/daemon/memcached.cc:851 (memcached+0x00000041466c) #13 event_persist_closure /home/couchbase/serverjenkins/workspace/cbdeps-platform-build/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1319 (libevent_core-2.0.so.5+0x00000000b6b7) #14 event_process_active_single_queue /home/couchbase/serverjenkins/workspace/cbdeps-platform-build/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1363 (libevent_core-2.0.so.5+0x00000000b6b7) #15 event_process_active /home/couchbase/serverjenkins/workspace/cbdeps-platform-build/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1438 (libevent_core-2.0.so.5+0x00000000b6b7) #16 event_base_loop /home/couchbase/serverjenkins/workspace/cbdeps-platform-build/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1639 (libevent_core-2.0.so.5+0x00000000b6b7) #17 CouchbaseThread::run() platform/src/cb_pthreads.cc:54 (libplatform.so.0.1.0+0x0000000057a5) #18 platform_thread_wrap platform/src/cb_pthreads.cc:66 (libplatform.so.0.1.0+0x0000000057a5) Location is global 'VBucket::chkFlushTimeout' of size 8 at 0x7ff76a2e7e50 (ep.so+0x000000431e50) Change-Id: I2d6f928b8a5552cf08a91c6134ad5134810966c2 Reviewed-on: http://review.couchbase.org/69234 Reviewed-by: Trond Norbye <[email protected]> Tested-by: buildbot <[email protected]>

tMutex must be acquired before accessing the threadQ data structure. 07:01:46 WARNING: ThreadSanitizer: data race (pid=32512) 07:01:46 Write of size 8 at 0x7d1c00016eb8 by thread T9 (mutexes: write M27337): 07:01:46 #0 operator delete(void*) <null> (engine_testapp+0x000000464cbb) 07:01:46 #1 <null> <null> (libstdc++.so.6+0x0000000c1ac7) 07:01:46 #2 ExecutorThread::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/executorthread.cc:127 (ep.so+0x0000000f7ff5) 07:01:46 #3 launch_executor_thread(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/executorthread.cc:32 (ep.so+0x0000000f78a5) 07:01:46 #4 CouchbaseThread::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/platform/src/cb_pthreads.cc:58 (libplatform_so.so.0.1.0+0x000000007b91) 07:01:46 #5 platform_thread_wrap(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/platform/src/cb_pthreads.cc:71 (libplatform_so.so.0.1.0+0x000000005e92) 07:01:46 07:01:46 Previous read of size 8 at 0x7d1c00016eb8 by main thread (mutexes: write M20977, write M26658): 07:01:46 #0 strlen <null> (engine_testapp+0x0000004652af) 07:01:46 #1 showJobLog(char const*, char const*, std::vector<TaskLogEntry, std::allocator<TaskLogEntry> > const&, void const*, void (char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/statwriter.h:39 (ep.so+0x0000000f346b) 07:01:46 #2 ExecutorPool::doWorkerStat(EventuallyPersistentEngine*, void const*, void (char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/executorpool.cc:801 (ep.so+0x0000000f2ce2) 07:01:46 #3 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep_engine.cc:4383 (ep.so+0x0000000c30bf) Change-Id: Iae7aa0fe9da15805671762516221f25da69fe5d8 Reviewed-on: http://review.couchbase.org/70262 Reviewed-by: Dave Rigby <[email protected]> Tested-by: buildbot <[email protected]>

HashTable will (must) be rather cleared in the destructor of VBucket object. Otherwise, race condition flagged by thread/address sanitizers when StoredValue is stored in more than one data structure. (like in both HashTable and Linked list in http://review.couchbase.org/#/c/74440) 16:55:17 ==27000==ERROR: AddressSanitizer: heap-use-after-free on address 0x60700001be60 at pc 0x2b5e33efd0ae bp 0x7ffd62e2d270 sp 0x7ffd62e2d268 16:55:17 READ of size 8 at 0x60700001be60 thread T0 16:55:17 #0 0x2b5e33efd0ad in boost::intrusive::list_node_traits<void*>::get_next(boost::intrusive::list_node<void*>* const&) /home/couchbase/jenkins/workspace/ep-engine-addresssanitizer-master/build/tlm/deps/boost.exploded/include/boost/intrusive/detail/list_node.hpp:63:13 16:55:17 #1 0x2b5e33efd0ad in boost::intrusive::list_iterator<boost::intrusive::mhtraits<StoredValue, boost::intrusive::list_member_hook<void, void, void>, &StoredValue::hook_>, true>::operator++() /home/couchbase/jenkins/workspace/ep-engine-addresssanitizer-master/build/tlm/deps/boost.exploded/include/boost/intrusive/detail/list_iterator.hpp:81 16:55:17 #2 0x2b5e33efd0ad in void boost::intrusive::list_impl<boost::intrusive::mhtraits<StoredValue, boost::intrusive::list_member_hook<void, void, void>, &StoredValue::hook_>, unsigned long, true, void>::clear_and_dispose<boost::intrusive::detail::null_disposer>(boost::intrusive::detail::null_disposer) /home/couchbase/jenkins/workspace/ep-engine-addresssanitizer-master/build/tlm/deps/boost.exploded/include/boost/intrusive/list.hpp:744 16:55:17 #3 0x2b5e33efd0ad in boost::intrusive::list_impl<boost::intrusive::mhtraits<StoredValue, boost::intrusive::list_member_hook<void, void, void>, &StoredValue::hook_>, unsigned long, true, void>::clear() /home/couchbase/jenkins/workspace/ep-engine-addresssanitizer-master/build/tlm/deps/boost.exploded/include/boost/intrusive/list.hpp:719 16:55:17 #4 0x2b5e33efd0ad in BasicLinkedList::~BasicLinkedList() /home/couchbase/jenkins/workspace/ep-engine-addresssanitizer-master/ep-engine/src/linkedlist.cc:30 16:55:17 #5 0x2b5e33efd0ad in BasicLinkedList::~BasicLinkedList() /home/couchbase/jenkins/workspace/ep-engine-addresssanitizer-master/ep-engine/src/linkedlist.cc:25 16:55:17 #6 0x2b5e33e5ece3 in std::default_delete<SequenceList>::operator()(SequenceList*) const /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/unique_ptr.h:76:2 16:55:17 #7 0x2b5e33e5ece3 in std::unique_ptr<SequenceList, std::default_delete<SequenceList> >::~unique_ptr() /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/unique_ptr.h:236 16:55:17 #8 0x2b5e33e5ece3 in EphemeralVBucket::~EphemeralVBucket() /home/couchbase/jenkins/workspace/ep-engine-addresssanitizer-master/ep-engine/src/ephemeral_vb.h:24 16:55:17 #9 0x2b5e33e5ece3 in EphemeralVBucket::~EphemeralVBucket() /home/couchbase/jenkins/workspace/ep-engine-addresssanitizer-master/ep-engine/src/ephemeral_vb.h:24 16:55:17 #10 0x2b5e33eaee1c in RCPtr<VBucket>::~RCPtr() /home/couchbase/jenkins/workspace/ep-engine-addresssanitizer-master/ep-engine/src/atomic.h:158:13 16:55:17 #11 0x2b5e33eaee1c in KVBucket::deleteVBucket(unsigned short, void const*) /home/couchbase/jenkins/workspace/ep-engine-addresssanitizer-master/ep-engine/src/kv_bucket.cc:902 16:55:17 #12 0x2b5e33e2cce5 in EventuallyPersistentEngine::deleteVBucket(unsigned short, void const*) /home/couchbase/jenkins/workspace/ep-engine-addresssanitizer-master/ep-engine/src/ep_engine.h:532:16 Change-Id: Ib46448cd396396dff183775a27c17c9f27d6a7e0 16:55:17 0x60700001be60 is located 0 bytes inside of 78-byte region [0x60700001be60,0x60700001beae) 16:55:17 freed by thread T12 (mc:nonIO_2) here: 16:55:17 #0 0x4ee992 in operator delete(void*) (/home/couchbase/jenkins/workspace/ep-engine-addresssanitizer-master/build/memcached/engine_testapp+0x4ee992) 16:55:17 #1 0x2b5e33e89a45 in StoredValue::operator delete(void*) /home/couchbase/jenkins/workspace/ep-engine-addresssanitizer-master/ep-engine/src/stored-value.h:41:9 16:55:17 #2 0x2b5e33e89a45 in HashTable::clear(bool) /home/couchbase/jenkins/workspace/ep-engine-addresssanitizer-master/ep-engine/src/hash_table.cc:98 16:55:17 #3 0x2b5e33f8aa49 in VBucketMemoryDeletionTask::run() /home/couchbase/jenkins/workspace/ep-engine-addresssanitizer-master/ep-engine/src/vbucketmemorydeletiontask.cc:45:5 Reviewed-on: http://review.couchbase.org/74500 Tested-by: Build Bot <[email protected]> Reviewed-by: Dave Rigby <[email protected]>

daverigby and others added 14 commits June 16, 2014 11:54

sync-with-meta: support different src/dest buckets

f4a8a31

Change-Id: Ic7ac39164f7d2d57476a1a459cc1ab9f58ca1a9f

sync-with-meta: Add check for revIDs being greater on dest.

60ae1e6

Change-Id: I69e1b91dbd0f022336c03ab67f032e4b5648540a

sync-with-meta: Allow updating of source revID if too low

e121049

Requires new option --allow-source-changes Change-Id: Icad7802f0da05120fbf3dc9052c4bcfc1b507520

sync-with-meta: Delete on dest if deleted on src

f6237f4

Change-Id: I66c7153077a7c0e3b3eabe3082ba30ce923cfff7

sync-with-meta: Simplify printing of docs

7c61e62

Change-Id: Iefe365ef453760b04e06b894f25001fcd3fe8217

sync-with-meta: Rename --force to --overwrite

45552f8

Change-Id: Ic239cb3a7be486bf56707b7f0ed2b5be27cec8c3

sync-with-meta: Handle document deletion. cleanup / refactor

34b8e7e

Change-Id: I269fd2a21c04b58d58427b34a59a6c73b51b8c56

sync-with-meta: Handle document not existing on dest or src

6147836

Change-Id: Ibac54d0dfcdac37e1c98d901abae5cc1ed6db078

sync-with-meta: Handle document not existing on dest or src - take2

80128b1

Change-Id: I078aade4d556a5d541b9393bf2e8436f57f409dc

compare-meta.py

13713b1

Change-Id: I001cd5d434b4ba564269d6c68e3d7630375ca5d3

compare-meta.py: Add brief mode

acb4065

Change-Id: I1a303d296913012e2a6887fd4d070b8976fb5d2c

compare-meta.py: Flush brief mode output

9c2aa6c

Change-Id: I9e5f3d2dad67b2bdeeb507e365f355b05e4816f1

Tidy up help and use stderr

5c84a10

jimwwalker closed this Jul 24, 2014

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Jim sync #1

Jim sync #1

Uh oh!

jimwwalker commented Jul 24, 2014

Uh oh!

jimwwalker commented Jul 24, 2014

Uh oh!

Uh oh!

Jim sync #1

Jim sync #1

Uh oh!

Conversation

jimwwalker commented Jul 24, 2014

Uh oh!

jimwwalker commented Jul 24, 2014

Uh oh!

Uh oh!