Document the ingester hand-over process #1560
Conversation
Signed-off-by: Bryan Boreham <[email protected]>
| short time, if nothing happens, it adds itself to the ring and goes | ||
| into state ACTIVE. | ||
|
|
||
| A running ingester is notified to shut down by Unix signal |
There was a problem hiding this comment.
This paragraph should mention the -ingester.claim-on-rollout flag.
docs/ingester-handover.md
Outdated
| `ACTIVE`, taking over ownership of the leaver's | ||
| [ring tokens](architecture.md#hashing). | ||
|
|
||
| If a leaving ingester does not find a pending ingester, it will flush |
There was a problem hiding this comment.
-ingester.search-pending-for I believe?
There was a problem hiding this comment.
Ah no, its ingester.max-transfer-retries - it will try, backoff for 5 secs (max).
There was a problem hiding this comment.
Yeah if you win the lottery. See #1307
docs/ingester-handover.md
Outdated
|
|
||
| On start-up, an ingester first goes into state `PENDING`. After a | ||
| short time, if nothing happens, it adds itself to the ring and goes | ||
| into state ACTIVE. |
There was a problem hiding this comment.
Tuneable with -ingester.join-after
There was a problem hiding this comment.
If you add that property can you tell under which circumstances you may want to tune it?
Why wouldn't I like to have a new ingester to immediately join the ring (i. e. setting -ingester.join-after to 0). Would that introduce the risk the ingester is considered as additional ingester (rather than the new ingester which is supposed to receive the data from the terminating ingester?) or why does that property even exist?
I am missing some more context to understand what is happening there.
There was a problem hiding this comment.
The leaving ingester probes periodically for a joining one - need to allow time for this to happen. I guess it's tuneable because we don't trust our initial guesses.
Personally I don't want to litter the main narrative with all the tuning options - maybe add a pointer to their description in arguments.md ?
There was a problem hiding this comment.
I think linking to arguments.md makes a lot of sense for the detailed parameters.
|
LGTM with a few nits. |
|
I added the two args. |
Signed-off-by: Bryan Boreham <[email protected]>
| ingester goes into state `JOINING` and the leaver transfers all its | ||
| in-memory data over to the joiner. On successful transfer the leaver | ||
| removes itself from the ring and exits and the joiner changes to | ||
| `ACTIVE`, taking over ownership of the leaver's |
There was a problem hiding this comment.
Should we also mention about unhealthy ingester state?
Suggested at #1553 (comment)