Since the arbiter brick does not store file data, its disk usage will be considerably less than the other bricks of the replica. The sizing of the brick will depend on how many files you plan to store in the volume. A good estimate will be 4kb times the number of files in the replica.

When a file is in split-brain, there is an inconsistency in either data or metadata (permissions, uid/gid, extended attributes etc.)of the file amongst the bricks of a replica and we do not have enough information to authoritatively pick a copy as being pristine and heal to the bad copies, despite all bricks being up and online. For directories, there is also an entry-split brain where a file inside it has different gfids/ file-type (say one is a file and another is a directory of the same name) across the bricks of a replica.

This document describes how to resolve files that are in split-brain using gluster cli or the mount point. Almost always, split-brains occur due to network disconnects (where a client temporarily loses connection to the bricks) and very rarely due to the gluster brick processes going down or returning an error.

This document provides a detailed description of this feature. The volume options for server-quorum are:

The cluster.server-quorum-ratio is a percentage figure and is cluster wide- i.e. you cannot have a different ratio for different volumes in the same trusted pool.

For a two-node trusted storage pool it is important to set this value to be greater than 50% so that two nodes separated from each other do not both believe they have quorum simultaneously. For a 2 node plain replica volume, this would mean both nodes need to be up and running. So there is no notion of HA/failover.

There are users who create a replica 2 volume from 2 nodes and peer-probe a `dummy' node without bricks and enable server quorum with a ratio of 51%. This does not prevent files from getting into split-brain. For example, if B1 and B2 are the bricks/nodes of the replica and B3 is the dummy node, we can still end up in split-brain like so:

In the author’s opinion, server quorum is useful if you want to avoid split-brains to the volume(s) configuration across the nodes and not in the I/O path. Unlike in client-quorum where the volume becomes read-only when quorum is lost, loss of server-quorum in a particular node makes glusterd kill the brick processes on that node (for the participating volumes) making even reads impossible.

Client-quorum is a feature implemented in AFR to prevent split-brains in the I/O path for replicate/distributed-replicate volumes. By default, if the client-quorum is not met for a particular replica subvol, it becomes read-only. The other subvols (in a dist-rep volume) will still have R/W access.

The following volume set options are used to configure it: >Option: cluster.quorum-type Default Value: none Value Description: none|auto|fixed If set to fixed'', this option allows writes to a file only if the number of active bricks in that replica set (to which the file belongs) is greater than or equal to the count specified in the `quorum-count' option. If set to auto'', this option allows writes to the file only if number of bricks that are up >= ceil (of the total number of bricks that constitute that replica/2). If the number of replicas is even, then there is a further check: If the number of up bricks is exactly equal to n/2, then the first brick must be one of the bricks that is up. If it is more than n/2 then it is not necessary that the first brick is one of the up bricks.

From the above descriptions, it is clear that client-quorum cannot really be applied to a replica 2 volume:(without costing HA). If the quorum-type is set to auto, then by the description given earlier, the first brick must always be up, irrespective of the status of the second brick. IOW, if only the second brick is up, the subvol becomes EROFS, i.e. no HA. If quorum-type is set to fixed, the the quorum-count has to be two to prevent split-brains. (otherwise a write can succeed in brick1, another in brick2 ⇒split-brain). So for all practical purposes, if you want high availabilty in a replica 2 volume, it is recommended not to enable client-quorum.

In a replica 3 volume, client-quorum is enabled by default and set to `auto'. This means 2 bricks need to be up for the writes to succeed. Here is how this configuration prevents files from ending up in split-brain:

Say B1, B2 and B3 are the bricks: 1. B3 is down, quorum is met, write happens on the file on B1 and B2 2. B3 comes up, B2 is down, quorum is again met, a write happens on B1 and B3. 3. B2 comes up, B1 goes down, quorum is met. Now when a write is issued, AFR sees that B2 and B3’s pending xattrs blame each other and therefore the write is not allowed and is failed with an EIO.

There is a corner case even with replica 3 volumes where the file can end up in a split-brain. AFR usually takes range locks for the \{offset, length} of the write. If 3 writes happen on the same file at non-overlapping \{offset, length} and each write fails on (only) one different brick, then we have AFR xattrs of the file blaming each other.