跳转至

High availability cluster with GlusterFS

Prerequisites

  • Proficiency with a command line editor (using vi in this example)
  • A heavy comfort level with issuing commands from the command-line, viewing logs, and other general systems administrator duties
  • All commands are run as the root user or sudo

Introduction

GlusterFS is a distributed file system.

It allows storing large amounts of data distributed across clusters of servers with very high availability.

It is composed of a server part to be installed on all the nodes of the server clusters.

Clients can access the data via the glusterfs client or the mount command.

GlusterFS can operate in two modes:

  • replicated mode: each node of the cluster has all the data.
  • distributed mode: no data redundancy. If a storage fails, the data on the failed node is lost.

Both modes can be used together to provide a replicated and distributed file system if you have the correct number of servers.

Data is stored inside bricks.

A Brick is the basic unit of storage in GlusterFS, represented by an export directory on a server in the trusted storage pool.

Test platform

Our fictitious platform is comprises two servers and a client, all Rocky Linux servers.

  • First node: node1.cluster.local - 192.168.1.10
  • Second node: node2.cluster.local - 192.168.1.11
  • Client1: client1.clients.local - 192.168.1.12

Note

Make sure you have the necessary bandwidth between the servers of the cluster.

Each server in the cluster has a second disk for data storage.

Preparation of the disks

We will create a new LVM logical volume that will be mounted on /data/glusterfs/vol0 on both of the cluster's servers:

$ sudo pvcreate /dev/sdb
$ sudo vgcreate vg_data /dev/sdb
$ sudo lvcreate -l 100%FREE -n lv_data vg_data
$ sudo mkfs.xfs /dev/vg_data/lv_data
$ sudo mkdir -p /data/glusterfs/volume1

Note

If LVM is not available on your servers, just install it with the following command:

$ sudo dnf install lvm2

We can now add that logical volume to the /etc/fstab file:

/dev/mapper/vg_data-lv_data /data/glusterfs/volume1        xfs     defaults        1 2

And mount it:

$ sudo mount -a

As the data is stored in a sub-volume called brick, we can create a directory in this new data space dedicated to it:

$ sudo mkdir /data/glusterfs/volume1/brick0

Installation

At the time of writing this documentation, the original CentOS Storage SIG repository is no longer available and the RockyLinux repository is not yet available.

However, we will use (for the time being) the archived version.

First of all, it is necessary to add the dedicated repository to gluster (in version 9) on both servers:

sudo dnf install centos-release-gluster9

Note

Later, when it is ready on the Rocky side, we can change the name of this package.

As the repo list and url is not available anymore, let's change the content of the /etc/yum.repos.d/CentOS-Gluster-9.repo:

[centos-gluster9]
name=CentOS-$releasever - Gluster 9
#mirrorlist=http://mirrorlist.centos.org?arch=$basearch&release=$releasever&repo=storage-gluster-9
baseurl=https://dl.rockylinux.org/vault/centos/8.5.2111/storage/x86_64/gluster-9/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Storage

We are now ready to install the glusterfs server:

$ sudo dnf install glusterfs glusterfs-libs glusterfs-server

Firewall rules

A few rules are necessary for the service to work:

$ sudo firewall-cmd --zone=public --add-service=glusterfs --permanent
$ sudo firewall-cmd --reload

or:

$ sudo firewall-cmd --zone=public --add-port=24007-24008/tcp --permanent
$ sudo firewall-cmd --zone=public --add-port=49152/tcp --permanent
$ sudo firewall-cmd --reload

Name resolution

You can let DNS handle the name resolution of the servers in your cluster, or you can choose to relieve the servers of this task by inserting records for each of them in your /etc/hosts files. This will also keep things running even during a DNS failure.

192.168.10.10 node1.cluster.local
192.168.10.11 node2.cluster.local

Starting the service

Without further delay, let's start the service:

$ sudo systemctl enable glusterfsd.service glusterd.service
$ sudo systemctl start glusterfsd.service glusterd.service

We are ready to join the two nodes in the same pool.

This command is to be performed only once on a single node (here on node1):

sudo gluster peer probe node2.cluster.local
peer probe: success

Verify:

node1 $ sudo gluster peer status
Number of Peers: 1

Hostname: node2.cluster.local
Uuid: c4ff108d-0682-43b2-bc0c-311a0417fae2
State: Peer in Cluster (Connected)
Other names:
192.168.10.11
node2 $ sudo gluster peer status
Number of Peers: 1

Hostname: node1.cluster.local
Uuid: 6375e3c2-4f25-42de-bbb6-ab6a859bf55f
State: Peer in Cluster (Connected)
Other names:
192.168.10.10

We can now create a volume with 2 replicas:

$ sudo gluster volume create volume1 replica 2 node1.cluster.local:/data/glusterfs/volume1/brick0/ node2.cluster.local:/data/glusterfs/volume1/brick0/
Replica 2 volumes are prone to split-brain. Use Arbiter or Replica 3 to avoid this. See: https://docs.gluster.org/en/latest/Administrator-Guide/Split-brain-and-ways-to-deal-with-it/.
Do you still want to continue?
 (y/n) y
volume create: volume1: success: please start the volume to access data

Note

As the return command says, a 2-node cluster is not the best idea in the world against split brain. But this will suffice for our test platform.

We can now start the volume to access data:

$ sudo gluster volume start volume1

volume start: volume1: success

Check the volume state:

$ sudo gluster volume status
Status of volume: volume1
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick node1.cluster.local:/data/glusterfs/v
olume1/brick0                               49152     0          Y       1210
Brick node2.cluster.local:/data/glusterfs/v
olume1/brick0                               49152     0          Y       1135
Self-heal Daemon on localhost               N/A       N/A        Y       1227
Self-heal Daemon on node2.cluster.local     N/A       N/A        Y       1152

Task Status of Volume volume1
------------------------------------------------------------------------------
There are no active volume tasks
$ sudo gluster volume info

Volume Name: volume1
Type: Replicate
Volume ID: f51ca783-e815-4474-b256-3444af2c40c4
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: node1.cluster.local:/data/glusterfs/volume1/brick0
Brick2: node2.cluster.local:/data/glusterfs/volume1/brick0
Options Reconfigured:
cluster.granular-entry-heal: on
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off

The status must be "Started".

We can already restrict access on the volume a little bit:

$ sudo gluster volume set volume1 auth.allow 192.168.10.*

It is as simple as that.

Clients access

There are several ways to access our data from a client.

The preferred method:

$ sudo dnf install glusterfs-client
$ sudo mkdir /data
$ sudo mount.glusterfs node1.cluster.local:/volume1 /data

There are no additional repositories to configure. The client is already present in the base repos.

Create a file and check that it is present on all the nodes of the cluster:

On client:

sudo touch /data/test

On both servers:

$ ll /data/glusterfs/volume1/brick0/
total 0
-rw-r--r--. 2 root root 0 Feb  3 19:21 test

Sounds good! But what happens if node 1 fails? It is the one that was specified when mounting the remote access.

Let's stop node one:

$ sudo shutdown -h now

Check status on node2:

$ sudo gluster peer status
Number of Peers: 1

Hostname: node1.cluster.local
Uuid: 6375e3c2-4f25-42de-bbb6-ab6a859bf55f
State: Peer in Cluster (Disconnected)
Other names:
192.168.10.10
[antoine@node2 ~]$ sudo gluster volume status
Status of volume: volume1
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick node2.cluster.local:/data/glusterfs/v
olume1/brick0                               49152     0          Y       1135
Self-heal Daemon on localhost               N/A       N/A        Y       1152

Task Status of Volume volume1
------------------------------------------------------------------------------
There are no active volume tasks

The node1 is away.

And on client:

$ ll /data/test
-rw-r--r--. 1 root root 0 Feb  4 16:41 /data/test

File is already there.

Upon connection, the glusterfs client receives a list of nodes it can address, which explains the transparent switchover we just witnessed.

Conclusions

While there are no current repositories, using the archived repositories that CentOS had for GlusterFS will still work. As outlined, GlusterFS is pretty easy to install and maintain. Using the command line tools is a pretty straight forward process. GlusterFS will help create and maintain high-availability clusters for data storage and redundancy. You can find more information on GlusterFS and tool usage from the official documentation pages.

Author: Antoine Le Morvan

Contributors: Steven Spencer, Ganna Zhyrnova