ElasticSearch plugin startup check is incomplete

[abitmore]

Bug Description

The checkES() function returns true even if there is an error in the response:

bitshares-core/libraries/utilities/elasticsearch.cpp

Lines 39 to 49 in c06d09c

	bool checkES(ES& es)
	{
	graphene::utilities::CurlRequest curl_request;
	curl_request.handler = es.curl;
	curl_request.url = es.elasticsearch_url + "_nodes";
	curl_request.auth = es.auth;
	curl_request.type = "GET";
	if(doCurl(curl_request).empty())
	return false;
	return true;

bitshares-core/libraries/plugins/elasticsearch/elasticsearch_plugin.cpp

Lines 535 to 543 in c06d09c

	void elasticsearch_plugin::plugin_startup()
	{
	graphene::utilities::ES es;
	es.curl = my->curl;
	es.elasticsearch_url = my->_elasticsearch_node_url;
	es.auth = my->_elasticsearch_basic_auth;
	if(!graphene::utilities::checkES(es))
	FC_THROW_EXCEPTION(fc::exception, "ES database is not up in url ${url}", ("url", my->_elasticsearch_node_url));

It is inappropriate to use the _nodes query to check, because the user may not have specific permissions.

Actually the check should be moved to plugin_initialize(), because plugin_startup() is called after database::open() which could do a partial replay which would already need to insert data into ES. Update: this has been addressed via 679e645 in #1997.

Note: the same check is also used in elasticsearch_plugin::get_account_history(...):

bitshares-core/libraries/plugins/elasticsearch/elasticsearch_plugin.cpp

Lines 620 to 623 in 1d1fb99

	vector<operation_history_object> result;
	if(!graphene::utilities::checkES(es))
	return result;

By the way, the same check in es_objects has a line es.auth = my->_es_objects_index_prefix; which is obviously wrong. This has been fixed via 8cc9e4c in #2570.

bitshares-core/libraries/plugins/es_objects/es_objects.cpp

Lines 403 to 414 in 6b61772

	void es_objects_plugin::plugin_startup()
	{
	graphene::utilities::ES es;
	es.curl = my->curl;
	es.elasticsearch_url = my->_es_objects_elasticsearch_url;
	es.auth = my->_es_objects_auth;
	es.auth = my->_es_objects_index_prefix;
	if(!graphene::utilities::checkES(es))
	FC_THROW_EXCEPTION(fc::exception, "ES database is not up in url ${url}", ("url", my->_es_objects_elasticsearch_url));
	ilog("elasticsearch OBJECTS: plugin_startup() begin");
	}

Impacts
Describe which portion(s) of BitShares Core may be impacted by this bug. Please tick at least one box.

• API (the application programming interface)
• Build (the build process or something prior to compiled code)
• CLI (the command line wallet)
• Deployment (the deployment process after building such as Docker, Travis, etc.)
• DEX (the Decentralized EXchange, market engine, etc.)
• P2P (the peer-to-peer network for transaction/block propagation)
• Performance (system or user efficiency, etc.)
• Protocol (the blockchain logic, consensus, validation, etc.)
• Security (the security of system or user data, etc.)
• UX (the User Experience)
• Other (please add below)

CORE TEAM TASK LIST

• Evaluate / Prioritize Bug Report
• Refine User Stories / Requirements
• Define Test Cases
• Design / Develop Solution
• Perform QA/Testing
• Update Documentation