In today’s entry I would like to introduce quorum and epsilon concepts. Both are related with cluster health, needed in case of potential communications and connectivity issues.
What is Quorum?
When a cluster is in quorum, it means that majority of nodes are healthy and can communicate with each other. In other words, quorum is a precondition for a functioning cluster. When quorum is lost, the cluster does not have the ability to accomplish normal operations. To better understand what Quorum is, imagine 6-node cluster. Only one collection of nodes can have a quorum, to keep the data integrity in the cluster. To better understand last sentence, imagine we are talking about 6-nodes cluster. One HA pair (two nodes) loses connection with other 4 nodes. Both “set of nodes” are online, for the time being if such incident, only collection of nodes that have quorum can operate as a cluster. This way, once the connectivity issue is solved, the remaining nodes can reconcile the data into a single data view.
In the cluster there is always a master node (also called RDB Master). This node is responsible for synchronizing information across the cluster. Master node is chosen by a voting protocol, each node in the cluster take part in election of a master node. In case of unlikely failure of a master node, a new master is elected by the remaining nodes.
When cluster is in a quorum?
Cluster is in a quorum when a simple majority of nodes are operational. The situation becomes more complicated, when, during the connectivity issues, a even number of nodes are split. For example 4-nodes cluster (2 HA pairs), are split evenly. Here comes very handy the concept of epsilon.
What is Epsilon?
Epsilon is assigned to a single node in the cluster. It gives an extra fractional voting weight, which is a deciding factor when the connectivity between two equal portions of a large cluster fails. The group of nodes containing epsilon maintains quorum. It is needed to avoid “Split Brain” situation, when both halves of the cluster could end up having two separate read/write copies of replicated databases. Epsilon is automatically assigned to the first node when the cluster is created. In case of a node failure, the epsilon is automatically reassigned to a healthy node.
How to verify which host have Epsilon and how to reassign it?
Epsilon can be also manually reassigned. To verify which host have the epsilon assigned:
cluster01::> set -privilege advanced
Warning: These advanced commands are potentially dangerous; use them only when directed to do so by NetApp personnel.
Do you want to continue? {y|n}: y
cluster01::*> cluster show -epsilon *
Node Health Eligibility Epsilon
-------------------- ------- ------------ ------------
cluster01_13 true true true
cluster01_14 true true false
cluster01_15 true true false
cluster01_16 true true false
cluster01_17 true true false
cluster01_18 true true false
6 entries were displayed.
cluster01::*>
To re-assign a epsilon to another node in the cluster, I’ll make sure to be in the advanced privilege level (set advanced), and use cluster modify command, first to remove epsilon from a current node, and than assign epsilon to another node. As on below example:
cluster01::*> cluster modify -node cluster01_13 -epsilon false cluster01::*> cluster modify -node cluster01_16 -epsilon true cluster01::*> cluster show -epsilon * Node Health Eligibility Epsilon -------------------- ------- ------------ ------------ cluster01_13 true true false cluster01_14 true true false cluster01_15 true true false cluster01_16 true true true cluster01_17 true true false cluster01_18 true true false 6 entries were displayed. cluster01::*> set admin cluster01::>
Two node cluster
In a two-node cluster, neither node holds epsilon, both nodes are equally important. If one node goes down, the other node must be able to become the RDB Master. In such configuration, instead of holding epsilon, both nodes are continuously polled to ensure that both are operational. In other word, each node will monitor the other node using disk connectivity status (since in two-node, HA pair, cluster both nodes have connectivity to same disks). In case of a failure, remaining node is taking over the operations, and becoming a RDB Master node.