Ingesters do not wait long enough for transfer

[bboreham]

We used to tell ours to wait for two minutes, but that was torn up by #1185

Today I see this:

level=info ts=2019-03-28T16:41:45.673567131Z caller=lifecycler.go:456 msg="changing ingester state from" old_state=ACTIVE new_state=LEAVING
level=error ts=2019-03-28T16:41:45.688616369Z caller=transfer.go:199 msg="transfer failed" err="cannot find ingester to transfer chunks to: no pending ingesters"
level=error ts=2019-03-28T16:41:45.695897001Z caller=transfer.go:199 msg="transfer failed" err="cannot find ingester to transfer chunks to: no pending ingesters"
level=error ts=2019-03-28T16:41:45.746735562Z caller=transfer.go:199 msg="transfer failed" err="cannot find ingester to transfer chunks to: no pending ingesters"
level=error ts=2019-03-28T16:41:45.807703134Z caller=transfer.go:199 msg="transfer failed" err="cannot find ingester to transfer chunks to: no pending ingesters"
level=error ts=2019-03-28T16:41:46.334636223Z caller=transfer.go:199 msg="transfer failed" err="cannot find ingester to transfer chunks to: no pending ingesters"
level=error ts=2019-03-28T16:41:46.46819466Z caller=transfer.go:199 msg="transfer failed" err="cannot find ingester to transfer chunks to: no pending ingesters"
level=error ts=2019-03-28T16:41:47.212946695Z caller=transfer.go:199 msg="transfer failed" err="cannot find ingester to transfer chunks to: no pending ingesters"
level=error ts=2019-03-28T16:41:47.370324186Z caller=transfer.go:199 msg="transfer failed" err="cannot find ingester to transfer chunks to: no pending ingesters"
level=error ts=2019-03-28T16:41:48.672456187Z caller=transfer.go:199 msg="transfer failed" err="cannot find ingester to transfer chunks to: no pending ingesters"
level=error ts=2019-03-28T16:41:51.35543268Z caller=transfer.go:199 msg="transfer failed" err="cannot find ingester to transfer chunks to: no pending ingesters"
level=error ts=2019-03-28T16:41:51.355469649Z caller=lifecycler.go:466 msg="Failed to transfer chunks to another ingester" err="terminated after 10 retries"

so it isn't even waiting the 100ms minimum I expect from cursory examination of the code.

Possibly related: #792

[goelankitt]

I have faced this same issue a couple of times. What if we make the retry backoff duration configurable?

[pracucci]

@bboreham I think this issue (min backoff period not honored) has been fixed by #1599.

[bboreham]

@pracucci I agree "it isn't even waiting the 100ms minimum I expect" should be fixed, but @ankit1ank reported seeing similar symptoms, presumably with #1599 included.

I see you've commented on #1844 - maybe the fix for this issue (#1307) is to raise the default -ingester.max-transfer-retries ?

[pracucci]

Yes, I think it's valuable to allow to customize the min/max backoff period (PR #1844), but I was just wondering if it doesn't really wait long enough, given we can already set the max number of transfers and we can estimate how long will retry within a [x, 2x] time range (which I do understand it pretty large, but still).

[bboreham]

Looking at a recent log the first 10 attempts took 25 seconds, so that's probably OK as a default.