Smarter and more Leadership election schemes #149
Description
Discussion: #112 (comment)
Currently Leadership* is very trivial, a version of the "Ranking" election, we "rank" members by sorting them by their address, requiring they are .joining or .up and pick the "first". This is simple, and works well enough to pick anyone to start moving members joining->up.
It does mean though that each partition gets its own leader in case of partitions. this is on purpose (!), since it allows us to add members in case of a partition quickly -- this works well if a cluster never needs consensus of other peers but instead operates either almost statelessly or is eventually consistent to begin with, and we'd rather add more nodes serving "old" data than not add any nodes (as .up).
This is but one side of the tradeoff, we could envision a scheme where adding members is a big deal and may have strong impact on consistency. This matters if we had any "singletons" in the cluster and we must ensure we don't run two of them, so we don't want to move nodes to .up until we know for certain the old nodes have been removed for example, or similar.
This is a meta ticket to collect specific use cases we have, and which strategies would serve it well.