Allow exclusion of health indicators from status rollup

[fitzoh]

We've been having intermittent issues with spring-cloud-config-server. When our config server is down, the config server health check returns a status of DOWN, which in turn marks the entire application as DOWN, even though the application is still functional.

While we could just disable the health check, we would like to include it in the full health endpoint response to make monitoring easier.

It would be nice if individual health checks could be configured as optional so that they are excluded from the overall status rollup, but included in the full response.

Alternatives

I asked about this on the spring boot gitter channel, and @snicoll recommended using custom statuses.

I looked into this a bit, and there appear to be a couple issues with this approach:

• Does not work with built in health indicators. If you want to use a custom status, you would need to rewrite/copy paste the indicators included in spring boot.
• Potentially difficult to be expressive. For example, let's say that I wanted to mark the application as DEGRADED when elasticsearch was unavailable. The custom health indicator would be unintuitive, as it would have to return an application level status (DEGRADED) when the component it's monitoring is DOWN.

[philwebb]

I understand the desire here, but I'm worried that optional health indicators might make the system generally harder to understand.

For example, the status page might show that an indicator is down, but still return an overall health of OK. The HealthAggregator would also need to change to either never receive optional indicators, or to somehow know they are optional and take the appropriate action.

As of Spring Boot 2.0 we've actually split the /health endpoint into two. We now have /status (for a non details view) and /health (for a full details view). This might actually help with your use-case as you could monitor on /status and use /health for the details (just ignoring any returned status code).

[philwebb]

I've flagged this for team attention to see if anyone else has an opinion.

[wilkinsona]

#3441 is somewhat similar to this.

[wilkinsona]

I am inclined to agree with @philwebb on this one. I think this could be implemented using a custom HealthAggregator. Assuming that's correct, I don't think we should add anything more in this area.

[snicoll]

Agreed. @fitzoh please give the custom HealthAggregator a try and we can resume the discussion if necessary.