NetApp cDOT – quorum and epsilon

In today’s entry I would like to introduce quorum and epsilon concepts. Both are related with cluster health, needed in case of potential communications and connectivity issues.

What is Quorum?

When a cluster is in quorum, it means that majority of nodes are healthy and can communicate with each other. In other words, quorum is a precondition for a functioning cluster. When quorum is lost, the cluster does not have the ability to accomplish normal operations. To better understand what Quorum is, imagine 6-node cluster. Only one collection of nodes can have a quorum, to keep the data integrity in the cluster. To better understand last sentence, imagine we are talking about 6-nodes cluster. One HA pair (two nodes) loses connection with other 4 nodes. Both “set of nodes” are online, for the time being if such incident, only collection of nodes that have quorum can operate as a cluster. This way, once the connectivity issue is solved, the remaining nodes can reconcile the data into a single data view.

In the cluster there is always a master node (also called RDB Master). This node is responsible for synchronizing information across the cluster. Master node is chosen by a voting protocol, each node in the cluster take part in election of a master node. In case of unlikely failure of a master node, a new master is elected by the remaining nodes.

When cluster is in a quorum?

Cluster is in a quorum when a simple majority of nodes are operational. The situation becomes more complicated, when, during the connectivity issues, a even number of nodes are split. For example 4-nodes cluster (2 HA pairs), are split evenly. Here comes very handy the concept of epsilon.

What is Epsilon?

Epsilon is assigned to a single node in the cluster. It gives an extra fractional voting weight, which is a deciding factor when the connectivity between two equal portions of a large cluster fails. The group of nodes containing epsilon maintains quorum. It is needed to avoid “Split Brain” situation, when both halves of the cluster could end up having two separate read/write copies of replicated databases. Epsilon is automatically assigned to the first node when the cluster is created. In case of a node failure, the epsilon is automatically reassigned to a healthy node.

How to verify which host have Epsilon and how to reassign it?

Epsilon can be also manually reassigned. To verify which host have the epsilon assigned:

cluster01::> set -privilege advanced

Warning: These advanced commands are potentially dangerous; use them only when directed to do so by NetApp personnel.
Do you want to continue? {y|n}: y

cluster01::*> cluster show -epsilon *
Node                 Health  Eligibility   Epsilon
-------------------- ------- ------------  ------------
cluster01_13          true    true          true
cluster01_14          true    true          false
cluster01_15          true    true          false
cluster01_16          true    true          false
cluster01_17          true    true          false
cluster01_18          true    true          false
6 entries were displayed.

cluster01::*>

To re-assign a epsilon to another node in the cluster, I’ll make sure to be in the advanced privilege level (set advanced), and use cluster modify command, first to remove epsilon from a current node, and than assign epsilon to another node. As on below example:

cluster01::*> cluster modify -node cluster01_13 -epsilon false

cluster01::*> cluster modify -node cluster01_16 -epsilon true

cluster01::*> cluster show -epsilon *
Node                 Health  Eligibility   Epsilon
-------------------- ------- ------------  ------------
cluster01_13          true    true          false
cluster01_14          true    true          false
cluster01_15          true    true          false
cluster01_16          true    true          true
cluster01_17          true    true          false
cluster01_18          true    true          false
6 entries were displayed.

cluster01::*> set admin

cluster01::> 

Two node cluster

In a two-node cluster, neither node holds epsilon, both nodes are equally important. If one node goes down, the other node must be able to become the RDB Master. In such configuration, instead of holding epsilon, both nodes are continuously polled to ensure that both are operational. In other word, each node will monitor the other node using disk connectivity status (since in two-node, HA pair, cluster both nodes have connectivity to same disks). In case of a failure, remaining node is taking over the operations, and becoming a RDB Master node.

Leave a Reply

Your email address will not be published. Required fields are marked *