zone_failures_availability and zone_failures_data suddenly going to zero and never recovering

[czephyr]

Hello,

I have noticed this pattern on our metrics where the metric fdb_cluster_fault_tolerance_zone_failures_data and the one for zone availability sometimes plummet to zero and then keep being reported as 0 until the exporter is restarted.

Do you happen to have witnessed such behavior in your exporter deployments?

[pboros]

I am familiar with this category of issues, usually this happens if a metric appears, but instead of it having a value change it just doesn't disappear. We had this issues with the roles of the processes and we fixed it by re-initializing tally before generating the stats. This was long ago, I don't expect that the exporter would have issues like this.

I just checked out data as well, here is an example when it bounces right back:

Next time this happens, can you also curl the exporter to see what is the raw data? Happy to help debugging this.