Allow certain indices be excluded from disk watermark write block

[Leaf-Lin]

Today when disk usage reaches the low watermark on the node, indices will be marked as read-only block from writing.

This is reasonably except that for certain system indices like .kibana, the size of the indices are usually very small, yet blocking on these system indices may have much bigger impact, for example causing upgrade failure.

Is would be nice to be able to set certain indices always exempt from blocking.

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[jrodewig]

If we proceed, we'll want to update the related troubleshooting docs here:
https://www.elastic.co/guide/en/elasticsearch/reference/master/fix-common-cluster-issues.html#_error_disk_usage_exceeded_flood_stage_watermark_index_has_read_only_allow_delete_block

[rdnm]

@Leaf-Lin is this something that you have seen occurring regularly?

[jaymode]

@rdnm another instance of this was seen on Cloud with an internal cluster. Specifically this prevented an upgrade plan from going through and took some time to get everything sorted again, so hopefully we can get the team to discuss this again in the context of system indices. I'll ping you separately with additional details.

[williamrandolph]

We discussed this issue in the Core/Infra team sync today. Team members had ideas for approaches other than a simple exemption from the watermark. For example, we could reserve a certain amount of additional "overflow" space for crucial indices. It's hard to know what approach to choose without more details about which indices are most important and which tasks can cause upgrade failures or plan failures for Cloud.

Also, is this something that we are sure would be applied only to system indices, or are there non-system-indices that could cause such failures?

[jaymode]

It's hard to know what approach to choose without more details about which indices are most important and which tasks can cause upgrade failures or plan failures for Cloud.

That's valid and a lot of things can cause upgrade/plan failures in Cloud; many of which would be unrelated to this issue. One of the tricky things with upgrades is cloud doesn't know before hand what will happen. Essentially what we're asking for is the ability to upgrade the stack even when we're over the low disk watermark. In my opinion, Cloud is a consumer of the stack and ideally the stack would be able to do some of the following in the future:

1. Indicate that an upgrade cannot be completed due to the amount of disk space or another reason before we attempt to upgrade versions of any component
2. Enable upgrades to complete even if we are past the low watermark and/or pass the watermark as part of the upgrade. In this case Kibana upgrades require enough space to reindex
3. Gracefully fallback (maybe just even validation upfront) if upgrades cannot be completed due to lack of space or other resource

Obviously the ask here requires collaboration with more than just the Elasticsearch team; any component that has a system index that needs to be written to during an upgrade would need to be part of the discussion and design for future work.

Also, is this something that we are sure would be applied only to system indices, or are there non-system-indices that could cause such failures?

This is a question that the stack teams would need to answer IMO. When we do a version upgrade for a stack component, does the stack component require writes made to non-system indices to succeed?

I'll note, this is not just a Cloud problem; this affects on premise users as well. It is a bad experience to find out in the middle of a stack upgrade that the upgrade is going to fail and then have to dig to find out why. System indices didn't have this aspect in its scope originally but I feel like this fits well with the project and could enable us to deliver a better experience in Cloud and on premise.

[colings86]

The original issue description here mentions the low disk watermark but its the flood stage watermark that adds the write block? So I think we are talking about cases where either before or during the upgrade the flood stage is reached?

I am going to assume the above is correct for now but please correct me if it's wrong.

Flood stage is the last line of defence before an out of memory. If we get to flood stage we are already in a very bad situation and we need to halt everything in order to preserve the little amount of disk space we have left so the cluster can still function, albeit limping, while the situation is resolved. No index, system or otherwise, should be considered special in this case. Just because an index is a system index doesn't mean its small in size (as mentioned above), but we also can't assume that a system index that is small today will remain small. The best approach to getting to flood stage is to stop everything IMO.

We do however need to have a solution to the problem of upgrades failing and leaving the cluster in a bad state because the flood stage has been reached either before the upgrade started or during the upgrade. I think the approaches here should be as @jaymode described:

1. Elasticsearch should be able to indicate (maybe through the health API) whether it's safe for an upgrade to go ahead. Part of this indication should take into account disk space
2. Elasticsearch should leave the cluster in a good state even if upgrades can't be completed (this could be by having a rollback)

For 1 above, I don't think the heuristic needs to be particularly complex. I think it would be fine for us to start with a rule that doesn't allow upgrades if the high disk watermark has been reached on a node. Since upgrades are not essential operations in Cloud[1] I think its fine to block an upgrade because the high watermark has been reached and require the user to fix that before upgrading.

[1] By essential operation I mean an operation that must make progress and succeed no matter what state the cluster is in. An example would be vacating an allocator

[Leaf-Lin]

I am happy with the approaches above.

Just came across #71486 having a similar concern.