You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Compactor can fail with "block with not healthy index found ... series have an average of 1.000 out-of-order chunks: 0.000 of these are exact duplicates (in terms of data and time range)" message #3569
Compactor can fail to compact block with message like this:
msg="failed to compact user blocks" err="compaction: group 0@8712473450002685162: block with not healthy index found /data/compact/0@8712473450002685162/01EJEXEW6XQ37G17Q4JH9M2KF1; Compaction level 1; Labels: map[__org_id__:...]: 1/457844 series have an average of 1.000 out-of-order chunks: 0.000 of these are exact duplicates (in terms of data and time range)"
When this happens, compaction for given user will not continue, because compactor will retry to compact this block over and over, failing each time.
Upon further investigation, this is a 2h block produced by ingester. It's not clear why out-of-order chunks would be written. This is bug likely in Prometheus TSDB code.