A Galera cluster that became unstable when it was stretched to a second data centre

A company ran its main database on a Percona XtraDB Cluster (Galera) in one data centre. When two more nodes were added in a second data centre, the cluster started to misbehave: nodes dropped out and rejoined, and at least once the whole cluster stopped accepting writes. The servers were barely loaded and their configuration was mostly sound. The error logs from both sites showed the cause — a link between the data centres that kept dropping for a few seconds at a time, and Galera timeouts tuned for a local network.

Times in the incident timelines below are relative to the start of each incident.

The starting point

Load was not the problem. The node in the main data centre used about 60% of one CPU core out of 12; the node in the second data centre served no application traffic at all and only applied replicated writes. In steady state Galera itself was healthy: replication queues were empty, flow control was almost never triggered and certification conflicts were rare.

• ~500 UPDATE/s against ~100 SELECT/s: a write-heavy workload.
• ~1,000 pages/s written to tablespaces, and as many redo-log writes.
• Almost no flow-control pauses and an empty replication queue in steady state.
• 3 s of silence before Galera reported a peer connection as timed out — and it happened again and again.

The second data centre had been configured with its own Galera segment (gmcast.segment), which is the right choice: it keeps replication traffic over the WAN to a minimum. So the topology was sensible. The question was what happened during the incidents.

Finding 1: the logs show a flapping link, not a failing server

The audit followed two representative incidents through the error logs of both sides and lined the timestamps up.

Incident A: the second site is evicted and resynchronises

This is Galera behaving as designed: the majority stays up, the minority steps aside and rejoins incrementally. It is still a disruption for anything using the second site, and it was happening repeatedly.

Incident B: the whole cluster stops

The second incident started the same way. Over about 35 seconds the connections from the main site to both remote nodes dropped and came back several times. Then, within one second, every node — including the five in the main data centre — went to non-Primary. Losing two nodes out of seven should never cost the majority its quorum. For about 16 seconds the cluster accepted no writes anywhere, and it only re-formed when all seven nodes could see each other again (full re-merge of primary: 7 of 7).

The report explained the difference. When nodes disappear one after another, Galera can exclude them in turn and carry on with the rest. When, from the cluster’s point of view, connectivity is lost everywhere at the same moment, it cannot tell which part should continue. It then waits for an operator to bootstrap one side, or for every node to come back so it can merge. Why the majority lost its quorum in this case was not fully explained by the logs, and the report said so rather than guessing; a software bug in that release was one possibility.

Finding 2: Galera’s defaults assume a local network

Both incidents had the same signature: links between the data centres going from working to silent and back within a few seconds. Galera’s default failure detector is tuned for a LAN, where three seconds of silence really does mean a node is gone. Over an unstable WAN those defaults turn every short network hiccup into a membership change.

The best fix is a stable link. Where that is not possible, the timeouts can be made tolerant of short outages:

wsrep_provider_options="evs.suspect_timeout=PT15S;evs.inactive_timeout=PT30S;evs.inactive_check_period=PT3S"

The trade-off is stated in the report: when a node really does fail, the cluster now takes longer to remove it — up to 15 seconds instead of 5 — and keeps waiting for it in the meantime, so commits stall for longer during a genuine failure.

A second, optional step tunes Galera for a high-latency link by letting more packets be in flight and sending larger ones: evs.send_window=1024, evs.user_send_window=512 and gcs.max_packet_size=2097152. The latency between the sites was not known, so the report recommended testing these and noted that the timeouts alone might be enough.

Finding 3: the nodes were not identical

A Galera cluster is only as fast as its slowest node, and troubleshooting assumes every node is configured the same. The two nodes reviewed were not:

The GCache size matters directly here. It decides how long a disconnected node can be away and still catch up with a quick incremental transfer instead of a full copy of the data. The report included a query to measure how many minutes of writes the GCache actually holds, and recommended the same size on every node.

Finding 4: configuration and housekeeping

Much was right and was left alone: O_DIRECT, file-per-table, the query cache off, per-session buffers near their defaults, a large table cache, xtrabackup-v2 for state transfers, and innodb_flush_log_at_trx_commit=2, a reasonable compromise on a Galera cluster where the other nodes hold any committed data.

• Background I/O. innodb_io_capacity was still at the default 200 on a write-heavy cluster, and the checkpoint age was 450–520 MB of a 2 GB redo log. Recommended: 2,000, with a maximum of 4,000.
• Memory. The 32 GB buffer pool was full while plenty of memory went to the OS cache. Recommended: grow it a few gigabytes at a time while watching memory, more carefully on the smaller node.
• Swappiness at 60; recommended 1.
• Version. 5.6.46 was behind its release line and affected by published denial-of-service CVEs. Recommended: the latest 5.6 release on all nodes, then a plan for 8.0.
• Accounts. Two accounts without a password and four using the old password-hash format.
• Disk. Data volumes were 73–76% full: not urgent, but worth tracking.

Finding 5: monitoring could not answer the question

Monitoring in place covered only the hosts. Nothing recorded Galera’s state over time — cluster size, flow control, queue lengths, state transfers — so the incidents could only be reconstructed from error logs after the fact, and there would be no baseline to tell whether the WAN tuning helped. The report recommended database- and Galera-level monitoring, such as Percona Monitoring and Management, before making the changes. For the timeouts the result would be visible as fewer evictions; for the rest of the tuning, only a before-and-after comparison can show the effect.

The recommendations, in order

1. Raise evs.suspect_timeout to 15 s, evs.inactive_timeout to 30 s and evs.inactive_check_period to 3 s.
2. Raise InnoDB I/O capacity to 2,000 (maximum 4,000).
3. Add database- and Galera-level monitoring.
4. Grow the buffer pool gradually while watching memory.
5. Set swappiness to 1.
6. Upgrade to the latest 5.6 release on every node.
7. Make node configuration identical, starting with the GCache size.
8. Fix or remove the accounts without a password or with old-format hashes.
9. Test WAN tuning: send windows and maximum packet size.
10. Plan the upgrade to version 8.0.

What this case shows

Stretching a Galera cluster across data centres changes what “a failure” looks like. The servers were idle and healthy; the instability came from a network that went silent for a few seconds at a time, and a failure detector set for a LAN that treated each of those moments as a lost node. Lining up the logs from both sides was enough to show it, and the fix started with three timeouts — with an honest note that longer timeouts also mean slower reactions to real failures.