Failed to transfer chunks to another ingester: rpc error: code = Unavailable desc = there is no connection available

[tomwilkie]

level=info ts=2019-01-11T20:17:39.047328696Z caller=transfer.go:195 msg="sending chunks" to_ingester=10.52.9.93:9095
level=error ts=2019-01-11T20:17:39.047697551Z caller=lifecycler.go:450 msg="Failed to transfer chunks to another ingester" err="rpc error: code = Unavailable desc = there is no connection available"

[tomwilkie]

Something to do with #1160 or #1138?

[tomwilkie]

Yeah looks like it:

WARNING: 2019/01/11 20:59:57 grpc: addrConn.createTransport failed to connect to {[2607:f8b0:4001:c0f::5f]:443 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp [2607:f8b0:4001:c0f::5f]:443: connect: network is unreachable". Reconnecting...
WARNING: 2019/01/11 21:00:06 grpc: addrConn.createTransport failed to connect to {[2607:f8b0:4001:c15::5f]:443 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp [2607:f8b0:4001:c15::5f]:443: connect: network is unreachable". Reconnecting...
level=info ts=2019-01-11T21:00:11.69841949Z caller=transfer.go:195 msg="sending chunks" to_ingester=10.52.9.110:9095
INFO: 2019/01/11 21:00:11 parsed scheme: ""
INFO: 2019/01/11 21:00:11 scheme "" not registered, fallback to default scheme
INFO: 2019/01/11 21:00:11 ccResolverWrapper: sending new addresses to cc: [{10.52.9.110:9095 0  <nil>}]
INFO: 2019/01/11 21:00:11 balancerWrapper: got update addr from Notify: [{10.52.9.110:9095 <nil>}]
level=error ts=2019-01-11T21:00:11.699025003Z caller=lifecycler.go:450 msg="Failed to transfer chunks to another ingester" err="rpc error: code = Unavailable desc = there is no connection available"
WARNING: 2019/01/11 21:00:11 grpc: the naming watcher stops working due to watcher has been closed.

[tomwilkie]

Actually I believe those ipv6 addr are for bigtable, so might be a read herring.