MySQL backup lab, part 7 – point-in-time recovery to a timestamp and a GTID

This is part 7 of nine. The whole series builds one MySQL backup verification lab and explains every playbook, template and script in it.

---

Part 6 ended on the limitation that makes this part necessary: a full backup can only take you back to the instant it was taken. Recover a dropped table from this morning’s backup and you also discard everything written since.

Point-in-time recovery closes that gap. Restore the full backup, then replay the binary log on top of it and stop at a chosen moment — one second before the DELETE, or exactly at a named transaction boundary. You lose the mistake and keep everything else.

That requires the binary logs written since the backup to exist somewhere durable. That is what the binlog server is for, and it is the piece that takes the most explaining.

Three scenarios: recovery to a timestamp (t09, 355 s), recovery to a GTID (t10, 336 s), and an impossible target that must fail cleanly (t11, 24 s).

The binlog server

spec:
  backup:
    pitr:
      enabled: true
      binlogServer:
        size: 1
        image: percona/percona-server-mysql-operator:1.2.0-binlog-server-0.4.1
        serverId: 1001
        checkpointInterval: 30s
        checkpointSize: 16M
        logLevel: info
        storage:
          s3:
            bucket: mysql-lab-binlogs
            credentialsSecret: minio-backup-credentials
            endpointUrl: http://minio.mysql.svc.cluster.local:9000
            region: us-east-1
            prefix: lab

This deploys one pod, lab-binlog-server-0, which does something conceptually simple and operationally clever: it registers with the primary as if it were an ordinary replica, receives the binary log stream, and writes it to object storage. It applies nothing. It has no data directory to speak of. It is a tap on the replication stream whose output is S3 objects.

That design has a direct consequence for configuration:

serverId: 1001 must be unique across every server in the replication topology. MySQL identifies replicas by server ID, and two connections claiming the same ID cause the source to disconnect one of them. This lab keeps a simple register: the cluster’s binlog server is 1001, and each restore-time binlog server gets its own — 1011 for t09, 1012 for t10, 1013 for t11. Part 8’s verification cluster is a real MySQL server and gets its ID from the operator, which is separately checked to differ from the live one.

The image tag is not the operator’s tag. 1.2.0-binlog-server-0.4.1 is a distinct artefact; the plain 1.2.0 tag is the operator. Using the operator image here produces a pod that starts and does not stream.

Two settings that define your data loss

• checkpointInterval: 30s — how often a partial binary log is flushed to object storage.
• checkpointSize: 16M — how much accumulates before a flush happens regardless of the interval.

Anything written after the last checkpoint and before a total loss of the primary is gone. That — not the backup schedule — is this lab’s real RPO. A daily full backup with a 30-second binlog checkpoint has a worst-case data loss of about 30 seconds, not 24 hours. Conversely, a five-minute checkpoint interval quietly makes your RPO five minutes no matter how often you take full backups.

The trade is the usual one: more frequent checkpoints mean more, smaller objects and more requests.

The recovery window has two ends

A recovery window begins at the oldest full backup you still have and ends at the newest binlog checkpoint. Binary logs alone recover nothing — there is no base to replay them onto.

This connects back to part 5 in a way that is easy to miss: retention that prunes backups also shortens your recovery window from the left. A keep: 2 policy with hourly backups gives you a two-hour window, no matter how much binlog archive you have kept.

Why a separate bucket

mysql-lab-backups   ← XtraBackup output
mysql-lab-binlogs   ← the binary log archive

They could share a bucket under different prefixes. Keeping them apart makes “are the binlogs actually being archived?” a one-command question:

mysql-lab-binlogs/lab/binlog.000001        128MiB
mysql-lab-binlogs/lab/binlog.000001.json      212B
mysql-lab-binlogs/lab/binlog.index             48B
mysql-lab-binlogs/lab/metadata.json            27B

Part 4’s t04 turns that question into an assertion — 20,000 rows of random data produced 20,808,764 bytes of archive growth on the validation run — so PITR never has to assume its inputs exist.

Recovery histories: the trap that is not in the documentation

This is the most important idea in this part, and it is not obvious.

Restoring a backup creates a new history. After part 6’s t08 rolled the cluster back to t01-full, the transactions that had happened between the backup and the restore no longer exist in the database — but they are still sitting in the binlog archive, because the archive is append-only and knows nothing about restores.

Now take a new backup and try a PITR. The replay reads the archive and finds, in among the transactions you want, the transactions the restore threw away. It applies them.

This is not theoretical. The lab’s second shakeout run recorded it happening: the PITR markers were correct, and TABLES_AFTER_RESTORE=9 — because an old DROP TABLE from an abandoned history had been replayed back on top of the new one.

The fix is to give each recovery history its own archive prefix, allocated before the base backup that it will be replayed onto:

- name: Allocate an archive prefix for this recovery timeline
  ansible.builtin.set_fact:
    recovery_binlog_prefix: "{{ binlog_prefix }}/{{ scenario_id }}-{{ lookup('pipe', 'date -u +%Y%m%dT%H%M%S') }}"

- name: Remember the collector pod before changing its configuration
  kubernetes.core.k8s_info:
    kind: Pod
    name: "{{ mysql_cluster_name }}-binlog-server-0"
  register: timeline_old_pod

- name: Switch the binlog collector to the new archive
  kubernetes.core.k8s:
    state: patched
    api_version: ps.percona.com/v1
    kind: PerconaServerMySQL
    name: "{{ mysql_cluster_name }}"
    definition:
      spec:
        backup:
          pitr:
            binlogServer:
              storage:
                s3:
                  prefix: "{{ recovery_binlog_prefix }}"

ansible/roles/test_common/tasks/pitr_timeline.yml

The prefix carries the scenario and a UTC timestamp — lab/t09-20260912T102428 on the validation run — so it is unique and self-describing. Old archives are kept rather than deleted; they are evidence, and deleting them would destroy the ability to investigate a bad recovery.

Waiting for a configuration change to actually take effect

Patching the CR is the easy part. Knowing the running process picked it up is where this task earns its length.

- name: Wait for the updated collector process and StatefulSet revision
  ansible.builtin.shell: |
    set -euo pipefail
    kubectl … get sts {{ mysql_cluster_name }}-binlog-server -o json | python3 -c '
    import json,sys
    sts=json.load(sys.stdin)
    status=sts.get("status",{})
    assert status.get("observedGeneration",0) >= sts["metadata"]["generation"]
    assert status.get("currentRevision") == status.get("updateRevision")
    assert status.get("readyReplicas",0) == 1
    '
    kubectl … get pod {{ mysql_cluster_name }}-binlog-server-0 -o json | python3 -c '
    import json,sys
    pod=json.load(sys.stdin)
    assert pod["metadata"]["uid"] != "{{ timeline_old_pod.resources[0].metadata.uid }}"
    assert any(c["type"]=="Ready" and c["status"]=="True" for c in pod["status"].get("conditions",[]))
    '
  until: timeline_rollout.rc == 0
  retries: 60
  delay: 5

Four separate conditions, each ruling out a specific way of being wrong:

The UID comparison is the one that matters most and is the easiest to leave out. A pod name is stable across a StatefulSet restart — lab-binlog-server-0 before and after — so “the pod is Ready” can be true of the pod you were trying to replace. The UID is regenerated; comparing it is how you know this is a different process.

And then, because even a new pod could conceivably start from cached configuration, it reads the config out of the running container:

- name: Wait for the binlog pod to use the new archive
  ansible.builtin.shell: |
    kubectl … exec {{ mysql_cluster_name }}-binlog-server-0 -c binlog-server -- \
      cat /etc/binlog_server/config/config.json | python3 -c '
    import json,sys
    config=json.load(sys.stdin)
    assert "{{ recovery_binlog_prefix }}" in json.dumps(config), "archive prefix not yet updated"
    '

Kubernetes said the rollout finished. This asks the process what it thinks its own configuration is. For a change whose failure mode is “silently archive to the wrong place and produce a subtly wrong recovery three minutes later”, that is worth five extra lines.

t09 — Point-in-time recovery to a timestamp

Proves: recovery can land between two known writes.

The scenario builds a timeline it can check:

q "CREATE TABLE IF NOT EXISTS sbtest.pitr_markers
     (id INT AUTO_INCREMENT PRIMARY KEY, label VARCHAR(32), at DATETIME(6))"
q "DELETE FROM sbtest.pitr_markers"
q "INSERT INTO sbtest.pitr_markers (label, at) VALUES ('good-{{ t_start }}', UTC_TIMESTAMP(6))"
sleep 20
# No Z suffix and no offset. search_by_timestamp in the binlog server
# rejects 2026-09-12T09:00:00Z as an invalid timestamp format, even
# though the upstream sample CR shows exactly that. Times are UTC.
echo "PITR_TARGET=$(q "SELECT DATE_FORMAT(UTC_TIMESTAMP(), '%Y-%m-%dT%H:%i:%s')")"
sleep 20
q "INSERT INTO sbtest.pitr_markers (label, at) VALUES ('bad-{{ t_start }}', UTC_TIMESTAMP(6))"

ansible/roles/t09_pitr_timestamp/tasks/main.yml

good → wait 20 s → capture the target → wait 20 s → bad. A successful recovery to that target keeps good and discards bad.

The marker labels include {{ t_start }}, the scenario’s start epoch. Markers from a previous attempt cannot be mistaken for this one’s — and since a PITR replay can resurrect rows from an earlier history, that is not paranoia. A shipped offline test (test_pitr_markers_identify_this_attempt_and_are_written_after_the_base) keeps it that way.

The 20-second gaps exist because the target has one-second resolution. Writing good, capturing a timestamp and writing bad within the same second would make the test meaningless.

The date format that costs an afternoon

date: "2026-09-12T10:25:07"

YYYY-MM-DDTHH:MM:SS, UTC, no timezone suffix. Not RFC 3339. And Percona’s own sample restore.yaml shows date: "2024-11-18T11:10:48Z" — with the Z, which this build rejects:

reconcile pitr config: search binlogs: exec binlog_server search_by_timestamp:
{"version":1,"status":"error","message":"Invalid timestamp format"}

Probing binlog_server search_by_timestamp directly inside the binlog server pod settled it. Of 2026-09-12T09:00:00Z, 2026-09-12 09:00:00, a Unix epoch, 2026-09-12T09:00:00+00:00 and 2026-09-12T09:00:00, only the last is accepted.

Which is why the target is produced by the database, not by the laptop:

SELECT DATE_FORMAT(UTC_TIMESTAMP(), '%Y-%m-%dT%H:%i:%s')

UTC_TIMESTAMP() rather than NOW() removes the server’s time_zone from the equation. DATE_FORMAT produces exactly the accepted layout. And taking it from the server removes the laptop’s clock and timezone from the equation entirely — a real risk when the laptop is on CEST and the container is on UTC, and the resulting one-hour error lands the recovery at a plausible-looking but completely wrong moment.

Waiting for the archive

- name: Let the binlog server checkpoint both markers to MinIO
  ansible.builtin.pause:
    seconds: "{{ pitr_settle_seconds }}"    # 120

The markers are in MySQL’s binary log. PITR replays from object storage. Between those two facts is checkpointInterval: 30s, and 120 seconds is four intervals of margin — which is most of why t09 takes 355 seconds.

This is a lab making a real property visible: you cannot recover to a moment that has not been archived yet. The most recent seconds of your database are always, briefly, unrecoverable.

The restore object

spec:
  clusterName: lab
  backupName: t09-base
  pitr:
    type: date
    date: "{{ t09_target }}"
    backupSource:
      binlogServer:
        image: "{{ binlog_server_image }}"
        # A literal int: the CRD types serverId as integer, and a
        # templated "{{ }}" reaches the API server as a string.
        # Must not collide with the cluster binlog server (1001).
        serverId: 1011
        storage:
          s3:
            bucket: "{{ binlog_bucket }}"
            credentialsSecret: "{{ minio_credentials_secret }}"
            endpointUrl: "http://minio.{{ mysql_namespace }}.svc.cluster.local:9000"
            region: us-east-1
            prefix: "{{ recovery_binlog_prefix }}"

serverId: 1011 is a hard-coded literal, and the comment explains why. The CRD types it as an integer; Ansible renders "{{ binlog_server_id }}" as a string, and the API server rejects a string where an integer is required. A literal is the unglamorous, correct fix, and writing down why it is not a variable is what stops someone from helpfully “cleaning it up” later.

The prefix is recovery_binlog_prefix, the fresh archive allocated at the top of the scenario — not binlog_prefix. This is the whole recovery-history mechanism arriving at its point of use.

Measured:

BASE_BACKUP=t09-base
BASE_DESTINATION=s3://mysql-lab-backups/lab/lab-2026-09-12-10:24:35-full
GOOD_MARKER_AT=2026-09-12 10:24:47.265418
PITR_TARGET=2026-09-12T10:25:07
BAD_MARKER_AT=2026-09-12 10:25:27.838550
MARKERS_BEFORE_RESTORE=good-1789208667,bad-1789208667
PITR_TYPE=date
RESTORE_STATE=Succeeded
GOOD_AFTER_RESTORE=1
BAD_AFTER_RESTORE=0
TABLES_AFTER_RESTORE=10
HEALTH=SYSBENCH_TABLES=10 | PRIMARY=lab-mysql-0 | REPLICA_OK=1 | BINLOG_SERVER=lab-binlog-server-0=Running

The marker at 10:24:47 survived. The marker at 10:25:27 did not. The target was 10:25:07. The recovery landed inside a forty-second window between two known writes, and all ten tables came back — that last assertion being the one the shakeout failure taught the suite to make.

t10 — Point-in-time recovery to a GTID

Proves: recovery can be aimed at an exact transaction boundary rather than a clock reading.

A timestamp is approximate by nature. If you know which transaction did the damage, a GTID target is exact — no risk of landing a fraction of a second on the wrong side.

q "INSERT INTO sbtest.gtid_markers (label) VALUES ('keep-me-{{ t_start }}')"
echo "ROWS_BEFORE_DAMAGE=$(q "SELECT COUNT(*) FROM sbtest.sbtest9")"
echo "GTID_TARGET=$(q "SELECT GTID_SUBTRACT(@@GLOBAL.gtid_executed, '{{ t10_inherited.stdout | trim }}')")"
sleep 20
q "DELETE FROM sbtest.sbtest9"                       # the damage: no WHERE clause
q "INSERT INTO sbtest.gtid_markers (label) VALUES ('after-damage-{{ t_start }}')"

ansible/roles/t10_pitr_gtid/tasks/main.yml

GTID_SUBTRACT, and why the target is a difference

The naive target would be @@GLOBAL.gtid_executed at the moment before the damage. That set includes every transaction the server has ever executed, including everything already inside the base backup and everything inherited from earlier restores — five UUIDs’ worth on this lab by the time t10 runs.

Handing that to the replay asks it to cover transactions that are not in the fresh archive, because they happened before the archive existed. The operator says so:

The specified GTID set cannot be covered

So the scenario captures @@GLOBAL.gtid_executed before taking its base backup — the inherited set — and computes the difference:

SELECT GTID_SUBTRACT(@@GLOBAL.gtid_executed, '<inherited>')

That yields only the transactions committed since the base, which is exactly what the archive contains and exactly what needs replaying. The inherited history arrives with the base backup itself, via gtid_purged.

Validation: inherited was

5fccd34e-…:1-925,a84e643f-…:1-38,d97cef97-…:1-71,f25e09c8-…:1-214

and the target was

GTID_TARGET=a84e643f-ae94-11f1-8ddd-2ec4c43ab3c3:39-52

Fourteen transactions. Everything else came from the backup.

--raw, and a bug that reads like corruption

kubectl … exec … -- mysql -uroot -pmysql --raw -N -e "SELECT @@GLOBAL.gtid_executed" | tr -d '\r\n'

A multi-UUID GTID set contains newlines. The MySQL client’s default output mode escapes them as the two characters \ and n. You can see it in part 4’s t01 evidence:

GTID_AT_BACKUP=5fccd34e-…:1-925,\nf25e09c8-…:1-212

Harmless in an evidence line. Fatal when fed back into GTID_SUBTRACT or into a restore object, where it produces:

cannot parse GTID set

--raw disables the escaping; tr -d '\r\n' then collapses the real newlines into a single-line set. There is a shipped offline test — test_gtid_capture_handles_multiple_uuid_lines_without_escaped_newlines — because this failure looks like a corrupt GTID set rather than a client formatting default.

Measured:

ROWS_BEFORE_DAMAGE=200000
GTID_TARGET=a84e643f-ae94-11f1-8ddd-2ec4c43ab3c3:39-52
ROWS_AFTER_DAMAGE=0
PITR_TYPE=gtid
RESTORE_STATE=Succeeded
ROWS_AFTER_RESTORE=200000
KEEP_MARKER=1
DAMAGE_MARKER=0

A DELETE with no WHERE removed 200,000 rows. The recovery put all of them back, kept the marker written before the mistake, and discarded the one written after.

The final assertion compares against the captured value, not a constant:

- (t10_after.stdout_lines | select('match', '^ROWS_AFTER_RESTORE=') | first).split('=')[1] | int
  == (t10_markers.stdout_lines | select('match', '^ROWS_BEFORE_DAMAGE=') | first).split('=')[1] | int

t11 — An impossible recovery target fails cleanly

Proves: a request that cannot be satisfied is reported, not silently downgraded.

This is the scenario that exists because of a specific fear. If you ask for recovery to a point that predates every backup and binlog you have, the worst possible behaviour is a system that shrugs, restores the plain backup, and reports success. You would restore, see data, believe you had recovered to the requested point, and be wrong about the state of your database.

spec:
  clusterName: lab
  backupName: t09-base
  pitr:
    type: date
    date: "2001-01-01T00:00:00"
    backupSource:
      binlogServer:
        serverId: 1013
        storage:
          s3:
            prefix: "{{ t11_prefix.stdout | trim }}"

A valid format (part 7’s lesson applied), a target years before anything existed. The prefix is read from the live cluster:

kubectl -n mysql get ps lab -o jsonpath='{.spec.backup.pitr.binlogServer.storage.s3.prefix}'

— because by the time t11 runs, t09 and t10 have each rotated it.

Asserting the right failure

- name: Require a clean, explained failure
  ansible.builtin.assert:
    that:
      - (t11_rst.resources[0].status.state | default('') | lower) in ['failed', 'error']
      - t11_rst.resources[0].status.stateDescription is search('Timestamp is too old')
      - "'Invalid timestamp format' not in t11_rst.resources[0].status.stateDescription"

Three conditions, and the third is the clever one. Without it, t11 would pass if the timestamp were merely malformed — the scenario would be testing the date parser rather than the recovery-window check, and a regression that broke out-of-range detection would be invisible.

Measured:

RESTORE_STATE=Error
RESTORE_REASON=reconcile pitr config: search binlogs: search binlogs: exec binlog_server search_by_timestamp: stdout: {"version":1,"status":"error","message":"Timestamp is too old"}
 stderr: : command terminated with exit code 1
TABLES_AFTER_FAILED_RESTORE=10

The right error, for the right reason.

And the data must be untouched

- name: Snapshot table checksums before the invalid request
  ansible.builtin.shell: |
    for i in $(seq 1 {{ sysbench_tables }}); do
      kubectl -n mysql exec lab-mysql-0 -c mysql -- mysql -uroot -pmysql -N -e "CHECKSUM TABLE sbtest.sbtest$i"
    done
  register: t11_before_checksums
  failed_when: t11_before_checksums.rc != 0 or 'NULL' in t11_before_checksums.stdout

…and afterwards:

  register: t11_after_checksums
  failed_when: t11_after_checksums.rc != 0 or t11_after_checksums.stdout != t11_before_checksums.stdout

CHECKSUM TABLE on all ten tables, before and after, compared byte for byte. A table count would not catch a restore that ran and produced different data. A checksum does.

The 'NULL' in stdout guard on the before snapshot rejects a baseline taken against a missing table — CHECKSUM TABLE on a non-existent table returns NULL rather than erroring, and a NULL baseline compared to a NULL result would pass while proving nothing.

The failed restore object is then deleted so it cannot block later work, and the health gate runs again. The evidence line shows something worth noticing:

BINLOG_SERVER=lab-binlog-server-0=Running lab-binlog-server-r-t11-restore-0=Running

The restore-time binlog server the operator created for the rejected attempt was still running at that moment — which is exactly the sort of thing you only find out by recording the full health line instead of a boolean.

Doing it by hand

export KUBECONFIG="$PWD/.kube/config"

# 1. give this recovery its own archive, and wait for the collector to adopt it
PREFIX="lab/manual-$(date -u +%Y%m%dT%H%M%S)"
kubectl -n mysql patch ps lab --type merge \
  -p "{\"spec\":{\"backup\":{\"pitr\":{\"binlogServer\":{\"storage\":{\"s3\":{\"prefix\":\"$PREFIX\"}}}}}}}"
kubectl -n mysql rollout status sts/lab-binlog-server --timeout=300s
kubectl -n mysql exec lab-binlog-server-0 -c binlog-server -- \
  cat /etc/binlog_server/config/config.json | grep -o "$PREFIX"

# 2. base backup AFTER the archive switch
kubectl -n mysql apply -f - <<'YAML'
apiVersion: ps.percona.com/v1
kind: PerconaServerMySQLBackup
metadata: { name: manual-base, namespace: mysql }
spec: { clusterName: lab, storageName: minio, type: full }
YAML
kubectl -n mysql get ps-backup manual-base -w

# 3. build a timeline
kubectl -n mysql exec lab-mysql-0 -c mysql -- mysql -uroot -pmysql -e \
  "CREATE TABLE IF NOT EXISTS sbtest.pitr_markers (id INT AUTO_INCREMENT PRIMARY KEY, label VARCHAR(32), at DATETIME(6));
   INSERT INTO sbtest.pitr_markers (label, at) VALUES ('good', UTC_TIMESTAMP(6));"
sleep 20
TARGET=$(kubectl -n mysql exec lab-mysql-0 -c mysql -- mysql -uroot -pmysql -N -e \
  "SELECT DATE_FORMAT(UTC_TIMESTAMP(), '%Y-%m-%dT%H:%i:%s')" | tr -d '\r')
echo "target=$TARGET"          # no Z, no offset
sleep 20
kubectl -n mysql exec lab-mysql-0 -c mysql -- mysql -uroot -pmysql -e \
  "INSERT INTO sbtest.pitr_markers (label, at) VALUES ('bad', UTC_TIMESTAMP(6));"

# 4. let it checkpoint
sleep 120

# 5. recover
kubectl -n mysql apply -f - <<YAML
apiVersion: ps.percona.com/v1
kind: PerconaServerMySQLRestore
metadata: { name: manual-pitr, namespace: mysql }
spec:
  clusterName: lab
  backupName: manual-base
  pitr:
    type: date
    date: "$TARGET"
    backupSource:
      binlogServer:
        image: percona/percona-server-mysql-operator:1.2.0-binlog-server-0.4.1
        serverId: 1021
        storage:
          s3:
            bucket: mysql-lab-binlogs
            credentialsSecret: minio-backup-credentials
            endpointUrl: http://minio.mysql.svc.cluster.local:9000
            region: us-east-1
            prefix: $PREFIX
YAML

kubectl -n mysql get ps-restore manual-pitr -w
kubectl -n mysql exec lab-mysql-0 -c mysql -- mysql -uroot -pmysql -N -e \
  "SELECT label FROM sbtest.pitr_markers ORDER BY id"     # expect: good only

For the GTID variant, capture the inherited set with --raw before step 2, then after the damage compute GTID_SUBTRACT(@@GLOBAL.gtid_executed, '<inherited>') and use pitr: { type: gtid, gtid: "<difference>" }.

Or:

./scripts/lab.sh t09
./scripts/lab.sh t10
./scripts/lab.sh t11     # requires t09-base

What can go wrong here

• Invalid timestamp format. A Z, an offset, or a space instead of T. Take the value from DATE_FORMAT(UTC_TIMESTAMP(), '%Y-%m-%dT%H:%i:%s').
• Timestamp is too old. The target is outside the recovery window — before the base backup, or in an archive prefix that does not cover it.
• The specified GTID set cannot be covered. The target includes inherited transactions. Subtract the pre-backup set.
• cannot parse GTID set. Escaped newlines. Use mysql --raw.
• The recovery succeeds and the data is wrong. Very likely a mixed archive — a prefix that spans a previous in-place restore, replaying an abandoned history. Allocate a new prefix before each base backup.
• serverId collision. A restore-time binlog server sharing an ID with the running one, or with a MySQL instance. Keep a register.
• The replay finds nothing recent. You did not wait for a checkpoint. 30 seconds is the interval; wait longer than one.

Next

Parts 4 through 7 have now proved that backups are produced and that they restore — in place, after a mistake, and to a chosen moment. Every one of those proofs was made by restoring over the live cluster, which is both destructive and, for a verification routine, entirely impractical.

Part 8 is the one that settles recoverability without taking production down: restore the backup into a separate cluster, attach that cluster as a replica of the live one, let it catch up, and compare every row.