Galera cluster, be it MariaDB Cluster, Percona XtraDB Cluster or Galera Cluster from Codership uses quorum calculations to make sure the data consistency is safe in case of a split-brain and that the majority of the nodes will continue to operate while nodes that are in the minority will stop executing changes. We have discussed the quorum calculation in detail in one of our blog posts. Let’s say we want to build a very small Galera Cluster, the minimal version. We don’t want to spend resources unless it’s necessary. How can it be done? In the earlier blog we have discussed that two nodes cannot form a cluster that will be able to survive failure of one of them. If surviving failure of one node is a requirement, three nodes are required in the cluster. Can we still do something about it? All the steps are executed on Centos7, for other distributions you may have to introduce some changes.
Galera Arbitrator
Enters Galera Arbitrator, or garbd for short. It will provide us with a solution that will allow us to minimize the operational expenses while still providing us with a high availability that we need.
What garbd does is it joins the cluster as a node but it does not contain any data on its own. It takes part of the communication between nodes, it takes part in the quorum calculation, relaying the traffic if needed and so on but when it comes to the data, all communication related to the data will be discarded. What does it change? Quite a lot. As we know, Galera, PXC or MariaDB Cluster is as fast as its slowest node – every node will have to process at least data modifications. Even if you won’t send the application traffic to it, it will still receive updates from other nodes and writes will have to happen quickly. When we discard all of that traffic, you will see that there’s no need for tens of gigabytes of RAM, no need for 16 CPU cores to handle the queries nor fast SSD to ensure data is quickly persisted on disk. What we need is just a small VM that’s connected to the rest of the cluster, a decent Internet connection and we are good to go. Such VM doesn’t have to run on VM with specs of a production-grade database node, thus reducing the cost of the whole setup.
Deploying two node MariaDB Cluster
In this blog we will go with MariaDB Cluster but the same process will work with any other flavour of Galera.
This is the topology that we want to deploy – two Galera nodes and one garbd. Such a setup will allow us to handle failure of one node and it’s pretty much the same as if using two nodes only. There is no scale-out (if one node goes down, you end up with one node only so every node has to be able to handle the traffic on its own) but there is high availability.
Installing MariaDB
When it comes to deployment, we start with two MariaDB nodes. It can be done in numerous ways and it really doesn’t matter how you do it. We used documentation from MariaDB website.
First, we download the script intended to set up the repositories.
[root@localhost ~]# wget https://downloads.mariadb.com/MariaDB/mariadb_repo_setup --2021-12-23 12:04:20-- https://downloads.mariadb.com/MariaDB/mariadb_repo_setup Resolving downloads.mariadb.com (downloads.mariadb.com)... 104.18.135.24, 104.17.191.14, 2606:4700::6811:bf0e, ... Connecting to downloads.mariadb.com (downloads.mariadb.com)|104.18.135.24|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 26626 (26K) [application/octet-stream] Saving to: ‘mariadb_repo_setup’ 100%[============================================================================================================================================================================================================================================>] 26,626 --.-K/s in 0.001s 2021-12-23 12:04:20 (28.4 MB/s) - ‘mariadb_repo_setup’ saved [26626/26626]
Once it’s done, we can make it executable and execute it to set up repositories for latest MariaDB version:
[root@localhost ~]# chmod +x mariadb_repo_setup [root@localhost ~]# sudo ./mariadb_repo_setup \ > --mariadb-server-version="mariadb-10.6" [info] Checking for script prerequisites. [info] Repository file successfully written to /etc/yum.repos.d/mariadb.repo [info] Adding trusted package signing keys... /etc/pki/rpm-gpg ~ ~ [info] Successfully added trusted package signing keys [info] Cleaning package cache... Loaded plugins: fastestmirror Cleaning repos: base extras mariadb-main mariadb-maxscale mariadb-tools updates Cleaning up list of fastest mirrors
When the repositories are set up, the remaining part is to install the software:
[root@localhost ~]# yum install MariaDB-server MariaDB-backup . . . Verifying : libpmem-1.5.1-2.1.el7.x86_64 19/20 Verifying : 1:mariadb-libs-5.5.68-1.el7.x86_64 20/20 Installed: MariaDB-backup.x86_64 0:10.6.5-1.el7.centos MariaDB-compat.x86_64 0:10.6.5-1.el7.centos MariaDB-server.x86_64 0:10.6.5-1.el7.centos Dependency Installed: MariaDB-client.x86_64 0:10.6.5-1.el7.centos MariaDB-common.x86_64 0:10.6.5-1.el7.centos boost-program-options.x86_64 0:1.53.0-28.el7 galera-4.x86_64 0:26.4.9-1.el7.centos libaio.x86_64 0:0.3.109-13.el7 libpmem.x86_64 0:1.5.1-2.1.el7 lsof.x86_64 0:4.87-6.el7 pcre2.x86_64 0:10.23-2.el7 perl-Compress-Raw-Bzip2.x86_64 0:2.061-3.el7 perl-Compress-Raw-Zlib.x86_64 1:2.061-4.el7 perl-DBI.x86_64 0:1.627-4.el7 perl-Data-Dumper.x86_64 0:2.145-3.el7 perl-IO-Compress.noarch 0:2.061-2.el7 perl-Net-Daemon.noarch 0:0.48-5.el7 perl-PlRPC.noarch 0:0.2020-14.el7 socat.x86_64 0:1.7.3.2-2.el7 Replaced: mariadb-libs.x86_64 1:5.5.68-1.el7 Complete!
That’s it, MariaDB has been installed. Now, we can proceed to the next step.
Configuring nodes of MariaDB Cluster
We should continue now with configuring the MariaDB nodes to use Galera replication. This is the /etc/my.cnf.d/server.cnf that we will use. Of course, IP addresses will differ between nodes, we need to keep it in mind.
# # These groups are read by MariaDB server. # Use it for options that only the server (but not clients) should see # # See the examples of server my.cnf files in /usr/share/mysql/ # # this is read by the standalone daemon and embedded servers [server] # this is only for the mysqld standalone daemon [mysqld] # # * Galera-related settings # [galera] # Mandatory settings wsrep_on=ON wsrep_provider=/usr/lib64/galera-4/libgalera_smm.so wsrep_cluster_address=gcomm://192.168.10.161,192.168.10.162 binlog_format=row default_storage_engine=InnoDB innodb_autoinc_lock_mode=2 # # Allow server to accept connections on all interfaces. # bind-address=0.0.0.0 # # Optional setting #wsrep_slave_threads=1 #innodb_flush_log_at_trx_commit=0 wsrep_node_address=192.168.10.162 wsrep_node_incoming_address=192.168.10.162 # this is only for embedded server [embedded] # This group is only read by MariaDB servers, not by MySQL. # If you use the same .cnf file for MySQL and MariaDB, # you can put MariaDB-only options here [mariadb] # This group is only read by MariaDB-10.6 servers. # If you use the same .cnf file for MariaDB of different versions, # use this group for options that older servers don't understand [mariadb-10.6]
Once the configuration file is ready, we will bootstrap the cluster on one node and, once that node is up, start the MariaDB service on the second node:
[root@node1 ~]# galera_new_cluster
[root@node2 ~]# systemctl start mariadb
If everything will go as planned, we should see in MariaDB logs that two Galera nodes have been discovered:
Dec 23 12:33:39 localhost mariadbd: 2021-12-23 12:33:39 0 [Note] WSREP: Quorum results: Dec 23 12:33:39 localhost mariadbd: version = 6, Dec 23 12:33:39 localhost mariadbd: component = PRIMARY, Dec 23 12:33:39 localhost mariadbd: conf_id = 3, Dec 23 12:33:39 localhost mariadbd: members = 1/2 (joined/total), Dec 23 12:33:39 localhost mariadbd: act_id = 8, Dec 23 12:33:39 localhost mariadbd: last_appl. = 5, Dec 23 12:33:39 localhost mariadbd: protocols = 2/10/4 (gcs/repl/appl), Dec 23 12:33:39 localhost mariadbd: vote policy= 0,
We should also see that the state transfer has been completed and both nodes (0.0 and 1.0) are synced with the cluster:
Dec 23 12:33:39 localhost mariadbd: 2021-12-23 12:33:39 0 [Note] WSREP: 0.0 (localhost.localdomain): State transfer to 1.0 (localhost.localdomain) complete. Dec 23 12:33:39 localhost mariadbd: 2021-12-23 12:33:39 0 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 9) Dec 23 12:33:39 localhost mariadbd: 2021-12-23 12:33:39 0 [Note] WSREP: Donor monitor thread ended with total time 0 sec Dec 23 12:33:39 localhost mariadbd: 2021-12-23 12:33:39 0 [Note] WSREP: Member 0.0 (localhost.localdomain) synced with group. Dec 23 12:33:39 localhost mariadbd: 2021-12-23 12:33:39 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 9) Dec 23 12:33:39 localhost mariadbd: 2021-12-23 12:33:39 1 [Note] WSREP: Server localhost.localdomain synced with group Dec 23 12:33:39 localhost mariadbd: 2021-12-23 12:33:39 1 [Note] WSREP: Server status change joined -> synced Dec 23 12:33:39 localhost mariadbd: 2021-12-23 12:33:39 1 [Note] WSREP: Synchronized with group, ready for connections Dec 23 12:33:39 localhost mariadbd: 2021-12-23 12:33:39 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. Dec 23 12:33:41 localhost mariadbd: 2021-12-23 12:33:41 0 [Note] WSREP: async IST sender served Dec 23 12:33:41 localhost mariadbd: 2021-12-23 12:33:41 0 [Note] WSREP: 1.0 (localhost.localdomain): State transfer from 0.0 (localhost.localdomain) complete. Dec 23 12:33:41 localhost mariadbd: 2021-12-23 12:33:41 0 [Note] WSREP: Member 1.0 (localhost.localdomain) synced with group. Dec 23 12:33:41 localhost mariadbd: 2021-12-23 12:33:41 0 [Note] WSREP: (68e07207-8163, 'tcp://0.0.0.0:4567') turning message relay requesting off
At this moment we have two nodes working in a MariaDB Cluster. No scale out and no high availability is provided by such a setup. We can now proceed to setting up Galera Arbitrator.
Deploying Galera Arbitrator for MariaDB Cluster
Once the cluster is up and running, we should proceed with deploying the Galera Arbitrator. The deployment process is easy. Garbd is provided by galera-4 package on Centos7
[root@localhost ~]# yum install galera-4
This is it, Galera Arbitrator has been installed. Now, we must configure it. For our setup we have to provide a list of Galera nodes and we also have to pass the contents of wsrep_cluster_name:
MariaDB [(none)]> show global variables like 'wsrep_cluster_name'; +--------------------+------------------+ | Variable_name | Value | +--------------------+------------------+ | wsrep_cluster_name | my_wsrep_cluster | +--------------------+------------------+ 1 row in set (0.000 sec)
For Centos7 the configuration file is located in /etc/sysconfig/garb:
[root@localhost ~]# cat /etc/sysconfig/garb # Copyright (C) 2012 Codership Oy # This config file is to be sourced by garb service script. # A comma-separated list of node addresses (address[:port]) in the cluster GALERA_NODES="192.168.10.161:4567, 192.168.10.162:4567" # Galera cluster name, should be the same as on the rest of the nodes. GALERA_GROUP="my_wsrep_cluster" # Optional Galera internal options string (e.g. SSL settings) # see http://galeracluster.com/documentation-webpages/galeraparameters.html # GALERA_OPTIONS="" # Log file for garbd. Optional, by default logs to syslog # LOG_FILE=""
Additional configuration options are required if you happen to use SSL for intra-cluster communication. As you may expect, if the Galera exchanges messages over SSL, garbd will have to have SSL configured to be able to understand what the communication is about. In our case there’s no need as SSL is not in use. Please keep in mind that, for production systems, you most likely would want to have SSL encryption. Once the configuration file is ready, we can start the garbd and watch what is happening:
Dec 27 14:43:51 localhost garb-systemd: 2021-12-27 14:43:51.281 INFO: Quorum results: Dec 27 14:43:51 localhost garb-systemd: version = 6, Dec 27 14:43:51 localhost garb-systemd: component = PRIMARY, Dec 27 14:43:51 localhost garb-systemd: conf_id = 4, Dec 27 14:43:51 localhost garb-systemd: members = 2/3 (joined/total), Dec 27 14:43:51 localhost garb-systemd: act_id = 9, Dec 27 14:43:51 localhost garb-systemd: last_appl. = 5, Dec 27 14:43:51 localhost garb-systemd: protocols = 2/10/4 (gcs/repl/appl), Dec 27 14:43:51 localhost garb-systemd: vote policy= 0, Dec 27 14:43:51 localhost garb-systemd: group UUID = 0299a397-63ea-11ec-8609-b75e95d2f49a Dec 27 14:43:51 localhost garb-systemd: 2021-12-27 14:43:51.282 INFO: Flow-control interval: [9999999, 9999999] Dec 27 14:43:51 localhost garb-systemd: 2021-12-27 14:43:51.282 INFO: Shifting OPEN -> PRIMARY (TO: 10) Dec 27 14:43:51 localhost garb-systemd: 2021-12-27 14:43:51.282 INFO: Sending state transfer request: 'trivial', size: 7 Dec 27 14:43:51 localhost garb-systemd: 2021-12-27 14:43:51.286 INFO: Member 0.0 (garb) requested state transfer from '*any*'. Selected 1.0 (localhost.localdomain)(SYNCED) as donor. Dec 27 14:43:51 localhost garb-systemd: 2021-12-27 14:43:51.286 INFO: Shifting PRIMARY -> JOINER (TO: 10) Dec 27 14:43:51 localhost garb-systemd: 2021-12-27 14:43:51.293 INFO: 0.0 (garb): State transfer from 1.0 (localhost.localdomain) complete. Dec 27 14:43:51 localhost garb-systemd: 2021-12-27 14:43:51.293 INFO: Shifting JOINER -> JOINED (TO: 10) Dec 27 14:43:51 localhost garb-systemd: 2021-12-27 14:43:51.299 INFO: Member 0.0 (garb) synced with group. Dec 27 14:43:51 localhost garb-systemd: 2021-12-27 14:43:51.299 INFO: Shifting JOINED -> SYNCED (TO: 10) Dec 27 14:43:51 localhost garb-systemd: 2021-12-27 14:43:51.305 INFO: 1.0 (localhost.localdomain): State transfer to 0.0 (garb) complete. Dec 27 14:43:51 localhost garb-systemd: 2021-12-27 14:43:51.307 INFO: Member 1.0 (localhost.localdomain) synced with group. Dec 27 14:43:53 localhost garb-systemd: 2021-12-27 14:43:53.777 INFO: (67afd3af-afea, 'tcp://0.0.0.0:4567') turning message relay requesting off
As you can see, all looks good. Third node has been discovered and it is now synced with the group. The remaining, optional step would be to add Galera Arbitrator’s hostname or IP address to wsrep_cluster_address:
MariaDB [(none)]> SHOW GLOBAL VARIABLES LIKE 'wsrep_cluster_address'; +-----------------------+---------------------------------------+ | Variable_name | Value | +-----------------------+---------------------------------------+ | wsrep_cluster_address | gcomm://192.168.10.161,192.168.10.162 | +-----------------------+---------------------------------------+ 1 row in set (0.001 sec)
[root@localhost ~]# grep wsrep_cluster_address /etc/my.cnf.d/server.cnf wsrep_cluster_address=gcomm://192.168.10.161,192.168.10.162
Currently only two Galera nodes are stored in this variable. It is not required as Arbitrator won’t be able to run IST or SST but in case of transient network issues it may act as a relay and improve overall durability.
This is pretty much it – we are now operating the cheapest Galera Cluster that provides a degree of high availability. What is very important to keep in mind is that there is no scale-out here. We mentioned it earlier but it is paramount to remember. With two nodes each node has to be able to handle the traffic alone, in case the other “database” node will fail.
