File-based settings health indicator

[prdoyle]

We have been trying to alert on file-based settings failures by inferring badness from logs. We've made progress there, but ultimately we're having trouble with the alert recovery conditions.

This PR adds a file-based settings Health Indicator, and we can alert directly on that instead of the logs.

[prdoyle]

Ok I've taken a different approach.

• All failures are now YELLOW
• The details include a field called most_recent_failure which is the Exception.toString of the exception
• Our alerting can differentiate based on most_recent_failure

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[dakrone]

LGTM, I left one comment. I wanted to do some manual testing, but it's not blocking.

[dakrone]

In the failure case you call .failureOccurred prior to the completion invocation, but here the order is switched. Should the .successOccurred() call be before the completion.onResponse(null)?

[prdoyle]

I didn't want to report a success if onResponse throws. Conversely, I did want to report a failure even if onFailure throws.

[rjernst]

What happens for nodes that don't ever read file settings? They are only used by ECK/serverless, and even then only the master node actually reads them. So on other nodes, the indicator would always be in a yellow state?

[prdoyle]

@rjernst that should be covered by the NO_CHANGES_SYMPTOM case. At least, that's the intent.

[prdoyle]

I ran it locally, and the curl -u elastic-admin:elastic-password http://localhost:9200/_health_report response includes this:

"file_settings" : {
   "status" : "green",
   "symptom" : "No file-based setting changes have occurred"
},