MySQL backup lab, part 4 – running backups: full, from the replica, incremental

This is part 4 of nine. The whole series builds one MySQL backup verification lab and explains every playbook, template and script in it.

---

A backup here is not a command. It is a Kubernetes object, and the operator reacts to it.

apiVersion: ps.percona.com/v1
kind: PerconaServerMySQLBackup
metadata:
  name: t01-full
  namespace: mysql
spec:
  clusterName: lab
  storageName: minio
  type: full

That is the entire request. What follows is what the operator does with it, and four scenarios — t01 to t04 — that check the result rather than the status.

All measured values in this post come from the ordered validation run recorded in docs/results/20260912T101400Z-validation-run.md.

What actually happens

The operator sees the object and creates a Kubernetes Job named xb-<backup>-<storage>-<hash> running percona/percona-xtrabackup:8.4.0-6.1. XtraBackup reads a cluster member’s data directory directly and streams the result to object storage with xbcloud. Nothing is staged on a local disk first, which matters on a laptop and matters much more on a real server where “make a local copy first” means provisioning a second copy of your database’s worth of disk.

This is a physical backup — InnoDB pages, not INSERT statements. Three consequences run through the rest of this series:

1. It is fast and cheap relative to a logical dump, because there is no SQL layer involved.
2. It restores as a data directory, so the restored server inherits the source’s mysql schema — its user accounts included. Part 8 is largely about the trouble that causes.
3. It restores gtid_purged along with the data, so the restored server knows exactly which transactions it already contains. That is what makes part 8’s replication test possible at all.

When the job finishes, the answer is in status:

state:       Succeeded
destination: s3://mysql-lab-backups/lab/lab-2026-09-12-10:13:48-full
completed:   2026-09-12T10:13:58Z
type:        full

Three fields on spec cover everything in this post:

The simple operator path: make backup

Before the scenarios, the lab ships a plain on-demand backup. It is the shortest complete example of the pattern every scenario uses.

- name: Record sysbench row counts before backup
  kubernetes.core.k8s_exec:
    pod: "{{ mysql_cluster_name }}-mysql-0"
    container: mysql
    command: >
      mysql -uroot -p{{ mysql_root_password }} -N -e "SELECT table_name, table_rows
      FROM information_schema.tables
      WHERE table_schema='{{ sysbench_database }}' AND table_name LIKE 'sbtest%'
      ORDER BY table_name"
  register: rows_before
  failed_when: false

ansible/roles/backup_cluster/tasks/main.yml

Record the state the backup is capturing, before capturing it. failed_when: false because this is context for a human, not a gate — a backup should not be blocked because an informational query hiccuped.

- name: Delete previous on-demand backup object if present
  kubernetes.core.k8s:
    state: absent
    api_version: ps.percona.com/v1
    kind: PerconaServerMySQLBackup
    name: lab-ondemand
    wait: true
    wait_timeout: 60

Backup objects are effectively immutable — re-applying lab-ondemand does not take a new backup, it does nothing. Deleting first makes make backup mean “take a backup now” rather than “take a backup the first time and be a no-op afterwards”. wait: true matters because deleting the object triggers the operator’s finalizer, which removes the backup’s data from object storage; racing that with a new backup of the same name is asking for trouble.

- name: Wait for backup to succeed
  kubernetes.core.k8s_info:
    api_version: ps.percona.com/v1
    kind: PerconaServerMySQLBackup
    name: lab-ondemand
  register: bkp
  until:
    - bkp.resources | length == 1
    - (bkp.resources[0].status.state | default('') | lower) in ['succeeded', 'ready', 'done']
      or (bkp.resources[0].status.state | default('') | lower) in ['failed', 'error']
  retries: 90
  delay: 10

- name: Require backup success
  ansible.builtin.assert:
    that:
      - (bkp.resources[0].status.state | default('') | lower) in ['succeeded', 'ready', 'done']
    fail_msg: "Backup did not succeed: {{ bkp.resources[0].status | default({}) }}"

The wait-for-terminal-then-assert split again, and the | lower with a list of accepted spellings is deliberate tolerance: the exact success word has varied across operator versions, and a suite that breaks on Ready versus Succeeded is testing the operator’s vocabulary rather than its behaviour.

t01 — On-demand full backup to MinIO

Proves: a backup object produces real bytes in object storage. Measured: 27 s.

The scenario records what it is about to capture:

kubectl -n mysql exec lab-mysql-0 -c mysql -- \
  mysql -uroot -pmysql -N -e \
  "SELECT CONCAT('TABLES=', COUNT(*)) FROM information_schema.tables
     WHERE table_schema='sbtest' AND table_name LIKE 'sbtest%';
   SELECT CONCAT('SIZE_MB=', IFNULL(ROUND(SUM(data_length+index_length)/1024/1024),0))
     FROM information_schema.tables WHERE table_schema='sbtest';
   SELECT CONCAT('GTID_AT_BACKUP=', @@GLOBAL.gtid_executed);"

The CONCAT('KEY=', …) idiom is the evidence convention from part 3, applied inside SQL so that the output lines are already in the right shape to be recorded.

GTID_AT_BACKUP is the interesting one. It is the exact set of transactions the source had executed when the backup was requested — the backup’s position on the replication timeline. It is what makes a statement like “this backup is an ancestor of the current state” checkable instead of rhetorical, and part 8 uses the same value in the form XtraBackup restores it: gtid_purged.

Then the request, the wait, and three assertions:

- name: Require the full backup to have succeeded
  ansible.builtin.assert:
    that:
      - (t01_bkp.resources[0].status.state | default('') | lower) in ['succeeded', 'ready', 'done']
      - t01_bkp.resources[0].status.destination | default('') | length > 0
      - backup_bucket in t01_bkp.resources[0].status.destination

Not just Succeeded — a destination, and one that points into the bucket we configured. A green state with an empty or unexpected destination would be a different kind of problem, and the assertion distinguishes them.

Then it looks at the storage itself:

- name: Measure the backup in MinIO
  ansible.builtin.include_role:
    name: test_common
    tasks_from: mc.yml
  vars:
    mc_args: "mc du local/{{ backup_bucket }}/{{ mysql_cluster_name }} 2>/dev/null | tail -1"

Recorded evidence:

TABLES=10
SIZE_MB=504
GTID_AT_BACKUP=5fccd34e-ae85-11f1-a9d5-0e01098e2e59:1-925,\nf25e09c8-ae91-11f1-b5a6-72d21a7478f2:1-212
BACKUP_STATE=Succeeded
BACKUP_TYPE=full
BACKUP_DESTINATION=s3://mysql-lab-backups/lab/lab-2026-09-12-10:13:48-full
BACKUP_COMPLETED=2026-09-12T10:13:58Z
MINIO_USAGE=10GiB	2353 objects	mysql-lab-backups/lab

Two things to read there. 10GiB / 2353 objects is the cumulative content of the cluster prefix at that moment — a suite’s worth of full, incremental and scheduled backups, not the size of this one. And the \n inside GTID_AT_BACKUP is an artefact: the MySQL client escaped the newline between two UUID entries. It is harmless in an evidence line and is not harmless when the value is fed back into MySQL — part 7 hits that exact problem and fixes it with --raw.

The mc helper, and a quoting problem worth showing

There is no S3 client on the laptop and no port-forward to MinIO. Every storage-side check runs mc in a throwaway pod inside the cluster:

- name: "mc: {{ mc_args }}"
  ansible.builtin.shell: |
    set -euo pipefail
    export KUBECONFIG="{{ kubeconfig_path }}"
    pod="mc-$RANDOM-$(date +%s)"
    trap 'kubectl -n {{ mysql_namespace }} delete pod "$pod" --ignore-not-found --wait=false >/dev/null 2>&1' EXIT
    kubectl -n {{ mysql_namespace }} run "$pod" --restart=Never --quiet \
      --image={{ minio_mc_image }} --command -- \
      sh -c "echo {{ mc_script | b64encode }} | base64 -d | sh" >/dev/null
    if ! kubectl -n {{ mysql_namespace }} wait --for=jsonpath='{.status.phase}'=Succeeded "pod/$pod" --timeout=120s >/dev/null; then
      kubectl -n {{ mysql_namespace }} logs "$pod"
      exit 1
    fi
    kubectl -n {{ mysql_namespace }} logs "$pod"
  vars:
    mc_script: |
      set -eu
      mc alias set local http://minio:9000 {{ minio_root_user }} {{ minio_root_password }} >/dev/null 2>&1
      {{ mc_args }}
  register: mc_out
  retries: 3
  delay: 5
  until: mc_out.rc == 0

ansible/roles/test_common/tasks/mc.yml

The base64 encoding is the point of the file. The script being run is composed of caller-supplied mc_args that routinely contain quotes, pipes and awk/sed fragments. Interpolating that into sh -c '…' inside a YAML string inside a kubectl run command line means four layers of quoting, and it breaks in ways that produce bizarre errors. Encoding the script to base64 and decoding it inside the container reduces the transport to a single opaque token that no layer can mangle. There is a shipped offline test for exactly this (test_mc_script_is_separated_from_kubectl_flags).

The trap … EXIT deletes the pod even when the task fails, so a broken run does not leave mc-* pods behind.

--wait=false on the delete so cleanup does not add latency to a task that already has its answer.

The pod name includes $RANDOM and a timestamp so concurrent or rapid successive calls cannot collide on a name.

retries: 3 absorbs the transient “pod was deleted before logs could be read” class of failure without hiding a genuine MinIO error, which fails all three times.

t02 — Backup taken from the replica, primary untouched

Proves: sourcePod moves XtraBackup’s read load and its brief locking off the pod serving writes. Measured: 14 s.

This is the scenario with the most direct production value in the whole produced-backups group. XtraBackup reads the entire data directory and takes a short lock near the end to capture a consistent position. On a busy primary that is real, measurable interference. Pointing it at a replica moves all of it.

First, find the replica — by asking, not by name:

for p in $(kubectl -n mysql get pods -l app.kubernetes.io/instance=lab,app.kubernetes.io/component=database \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | sort); do
  ro=$(kubectl -n mysql exec "$p" -c mysql -- mysql -uroot -pmysql -N -e "SELECT @@read_only" | tr -d '\r')
  if [ "$ro" = "1" ]; then echo "$p"; exit 0; fi
done
exit 1

Same discipline as part 3: @@read_only = 1 identifies the replica. Hard-coding lab-mysql-1 would be correct today and wrong after any failover.

spec:
  clusterName: lab
  storageName: minio
  type: full
  sourcePod: lab-mysql-1

While the backup runs, the scenario writes to the primary:

CREATE TABLE IF NOT EXISTS sbtest.t02_writes (id INT AUTO_INCREMENT PRIMARY KEY, at DATETIME);
INSERT INTO sbtest.t02_writes (at) VALUES (UTC_TIMESTAMP());
SELECT CONCAT('PRIMARY_WRITABLE=', IF(@@read_only=0,'yes','no'));
SELECT CONCAT('PRIMARY_WRITES=', COUNT(*)) FROM sbtest.t02_writes;

And then asserts four things:

- ...status.state in ['succeeded', 'ready', 'done']
- backup_bucket in status.destination
- (status.backupSource | default('')).split('.')[0] == (replica_pod.stdout | trim)
- (replica_pod.stdout | trim) != <the detected primary>
- "'PRIMARY_WRITABLE=yes' in primary_writes.stdout"

The third is the one that makes the scenario mean something. status.backupSource is the operator’s own report of which pod the backup was taken from; the assertion requires it to be the replica we selected. Without it, the test would pass even if sourcePod were silently ignored — a green backup would be taken from the primary and the scenario would call it success.

Measured:

SOURCE_POD=lab-mysql-1
ACTUAL_BACKUP_SOURCE=lab-mysql-1.lab-mysql.mysql
BACKUP_STATE=Succeeded
BACKUP_DESTINATION=s3://mysql-lab-backups/lab/lab-2026-09-12-10:14:15-full
BACKUP_COMPLETED=2026-09-12T10:14:24Z
PRIMARY_WRITABLE=yes
PRIMARY_WRITES=2

.split('.')[0] on lab-mysql-1.lab-mysql.mysql extracts the pod name from the FQDN form the operator reports.

What this does not prove, stated plainly because the catalogue states it: this is a sampled availability check. It shows the primary was writable and accepted a write while the replica-sourced backup was in flight. It does not measure primary latency during the backup, and it is not a claim about impact under load. Proving that would need a sustained workload and latency percentiles, which is a different lab.

PRIMARY_WRITES=2 rather than 1, incidentally, because t02_writes was created by an earlier attempt in the same lab lifetime and the insert appended to it. The evidence reports the table’s total, not this attempt’s insert.

t03 — Incremental backup on a full base

Proves: type: incremental with incrementalBaseBackupName captures only what changed. Measured: 19 s.

An incremental backup contains only the InnoDB pages whose log sequence number is newer than the base backup’s. The mechanism depends on a specific base:

spec:
  clusterName: lab
  storageName: minio
  type: incremental
  incrementalBaseBackupName: t01-full

So the scenario begins by requiring that base to exist and be good:

- name: Require the t01 full backup as the incremental base
  kubernetes.core.k8s_info:
    api_version: ps.percona.com/v1
    kind: PerconaServerMySQLBackup
    name: t01-full
  register: t03_base
  failed_when:
    - (t03_base.resources | default([]) | length) == 0
      or (t03_base.resources[0].status.state | default('') | lower) not in ['succeeded', 'ready', 'done']

This is the first explicit inter-scenario dependency, and it is checked rather than assumed. Running ./scripts/lab.sh t03 on a fresh lab fails immediately with a clear reason instead of producing an incremental against nothing.

Then it creates a measurable delta:

CREATE TABLE IF NOT EXISTS sbtest.t03_delta (id INT AUTO_INCREMENT PRIMARY KEY, pad CHAR(120));
INSERT INTO sbtest.t03_delta (pad)
  SELECT LEFT(MD5(RAND()), 120) FROM sbtest.sbtest1 LIMIT 50000;

SELECT … FROM sbtest1 LIMIT 50000 is a row generator — the source rows are irrelevant, the count is not. MD5(RAND()) makes the payload incompressible-ish and non-repeating, so the delta is real bytes rather than something a storage layer can collapse.

Sizing both backups is where the mc helper earns its keep:

- name: Derive the MinIO prefixes for both backups
  ansible.builtin.set_fact:
    t03_full_prefix: "{{ t03_base.resources[0].status.destination | regex_replace('^s3://' ~ backup_bucket ~ '/', '') }}"
    t03_incr_prefix: "{{ t03_bkp.resources[0].status.destination  | regex_replace('^s3://' ~ backup_bucket ~ '/', '') }}"

  vars:
    mc_args: "mc du --json local/{{ backup_bucket }}/{{ t03_full_prefix | trim }} | tail -1"

--json because parsing mc du’s human output (708MiB, 15MiB) means parsing units. The JSON has an exact size in bytes.

- name: Require the incremental to be smaller than the full
  ansible.builtin.assert:
    that:
      - (t03_incr_bytes | int) > 0
      - (t03_incr_bytes | int) < (t03_full_bytes | int)

Both bounds matter. Smaller-than-full alone would pass for a zero-byte backup, which is the exact failure an incremental scenario should catch.

Measured:

DELTA_ROWS=100000
BASE_BACKUP=t01-full
FULL_DESTINATION=s3://mysql-lab-backups/lab/lab-2026-09-12-10:13:48-full
INCR_DESTINATION=s3://mysql-lab-backups/lab/lab-2026-09-12-10:13:48-full.incr/lab-2026-09-12-10:14:30-incr
FULL_BYTES=708418853
INCR_BYTES=15510469
INCR_PCT_OF_FULL=2

2% of the full backup. That is the number the whole scenario exists to produce, and it is the argument for incrementals in one line.

Three details worth reading in that output:

• DELTA_ROWS=100000 when the insert was 50,000. The evidence reports COUNT(*) on t03_delta, and the table already held an earlier attempt’s rows in that lab lifetime. The assertion does not depend on the number, but it is worth knowing that this line is a table total, not an insert size.
• The incremental’s destination is nested under the full’s (…-full.incr/…-incr). The chain is expressed in the object path, which is how a restore can find its base.
• FULL_BYTES=708418853 — 708 MB — against a 504 MB dataset. No contradiction: SIZE_MB is data_length + index_length for the sbtest schema, while XtraBackup copies the whole data directory — every schema, InnoDB’s system tablespace, the redo log, the doublewrite buffer. A physical backup is bigger than the logical data it contains, and knowing that stops you from reading a perfectly healthy backup as suspiciously large.

What this does not prove: that an incremental restores. Creating one and measuring it is a different claim from replaying a chain onto a base. The catalogue says so explicitly, and it is not tested here. Do not tell yourself otherwise because the byte count looked right.

t04 — Backup and binlog objects present in MinIO

Proves: the objects are real files, and the binlog stream is live rather than merely configured. Measured: 40 s.

t01 asked MinIO how much space a prefix used. t04 goes further: it lists every object, requires the XtraBackup chunks to be non-empty, requires the checksum sidecar to exist, and then proves the binlog archive is actively growing by writing data and watching the bytes.

This one is a Python script rather than YAML, and the reason is honest: the logic is list-filter-assert-poll with retry, which is three lines of Python and an unreadable pile of Jinja filters.

- name: Verify backup chunks and prove binlog byte growth after controlled writes
  ansible.builtin.command:
    argv:
      - "{{ ansible_playbook_python }}"
      - "{{ role_path }}/files/probe.py"
  environment:
    NS: "{{ mysql_namespace }}"
    CLUSTER: "{{ mysql_cluster_name }}"
    KUBECONFIG: "{{ kubeconfig_path }}"
    MC_IMAGE: "{{ minio_mc_image }}"
    MC_USER: "{{ minio_root_user }}"
    MC_PASSWORD: "{{ minio_root_password }}"
    MYSQL_PASSWORD: "{{ mysql_root_password }}"
    DATABASE: "{{ sysbench_database }}"
    BINLOG_BUCKET: "{{ binlog_bucket }}"
    BINLOG_PREFIX: "{{ binlog_prefix }}"

Configuration arrives as environment variables, not command-line arguments or template substitution into the script. The script is a plain file — runnable by hand, testable offline, never rendered — and there is no path by which a lab value gets interpolated into Python source.

ansible_playbook_python is the interpreter Ansible itself is running under, which is the project virtualenv from part 1. The script gets the same Python the rest of the lab uses without hard-coding a path.

What the probe checks

backup = json.loads(kubectl('get', 'ps-backup', 't01-full', '-o', 'json'))
assert backup['spec']['clusterName'] == cluster
assert backup['status']['state'].lower() == 'succeeded'
destination = backup['status']['destination']
listing = objects(destination.removeprefix('s3://') + '/')
chunks = [x for x in listing if not x['key'].endswith('.md5')]
assert any('xtrabackup' in x['key'] for x in chunks), 'XtraBackup metadata chunks missing'
checksums = [x for x in listing if x['key'].endswith('.md5')]
# xbcloud may store its checksum sidecar next to the destination prefix.
if not checksums:
    checksums = [x for x in objects(destination.removeprefix('s3://') + '.md5') if x['key'].endswith('.md5')]
assert chunks and all(x['size'] > 0 for x in chunks), 'No nonzero backup chunks'
assert checksums and all(x['size'] > 0 for x in checksums), 'No nonzero MD5 objects'

ansible/roles/t04_storage_contents/files/probe.py

Requiring a key containing xtrabackup is requiring XtraBackup’s own metadata (xtrabackup_checkpoints, xtrabackup_info) to be present — those files carry the LSN and GTID position a restore needs. A pile of data chunks without them is not a restorable backup.

The .md5 fallback exists because xbcloud places the checksum sidecar in one of two places depending on version: inside the destination prefix, or beside it as <destination>.md5. Accepting both is tolerance for a real, observed variation — and the same sidecar shows up again in part 5, where it survives retention pruning and gets reported separately rather than failing the test.

The listing helper enforces something subtle:

item = json.loads(line)
if item.get('status') == 'error':
    raise RuntimeError('MinIO listing failed: ' + str(item))
if item.get('type') == 'file':
    result.append(item)

A failed listing must not look like an empty prefix. mc ls --json reports errors as JSON lines, and code that only collects type == 'file' would treat an authentication failure as “the prefix is empty” — which in part 5’s retention test would read as successful deletion. There is a shipped offline test named test_minio_error_cannot_be_mistaken_for_empty_prefix guarding precisely this.

Proving the binlog stream is alive

active = json.loads(kubectl('get', 'ps', cluster, '-o', 'json'))['spec']['backup']['pitr']['binlogServer']['storage']['s3']
prefix = active['bucket'] + '/' + active.get('prefix', '').strip('/') + '/'
before = sum(x['size'] for x in objects(prefix))
assert before > 0, 'Binlog prefix is empty'

The archive prefix is read from the live cluster spec, not from group_vars. That matters because part 7 changes this prefix at runtime; a probe reading the static default would look in a stale location and report an empty archive.

sql(primaries[0], f"CREATE TABLE IF NOT EXISTS `{database}`.t04_stream_probe "
                  f"(id BIGINT AUTO_INCREMENT PRIMARY KEY, pad VARBINARY(2048)); "
                  f"INSERT INTO `{database}`.t04_stream_probe(pad) "
                  f"SELECT RANDOM_BYTES(1024) FROM `{database}`.sbtest1 LIMIT 20000; "
                  f"FLUSH BINARY LOGS;")

• RANDOM_BYTES(1024) — 20,000 rows of a kilobyte of incompressible random data, about 20 MB, chosen to exceed the 16 MB checkpointSize from part 2 so a flush is guaranteed rather than hoped for.
• FLUSH BINARY LOGS rotates the binary log, which prompts the binlog server to finalise and upload rather than waiting on its interval.
• VARBINARY, not TEXT — no charset conversion between what MySQL stores and what lands in the binary log.

deadline = time.monotonic() + 300
while time.monotonic() < deadline:
    after = sum(x['size'] for x in objects(prefix))
    if after > before:
        evidence('BINLOG_BYTES_AFTER', after)
        evidence('BINLOG_BYTE_GROWTH', after - before)
        return
    time.sleep(10)
raise RuntimeError('Binlog bytes did not grow after controlled writes and rotation')

time.monotonic() rather than time.time() — a wall clock that steps (NTP, a laptop waking up) can shorten or extend the deadline. For a five-minute timeout around a storage operation, use the clock that only goes forward.

Also note the primary check just above it:

primaries = [p['metadata']['name'] for p in pods if sql(p['metadata']['name'], 'SELECT @@read_only') == '0']
assert len(primaries) == 1

Exactly one writable server. Not “at least one”. Two writable pods means a split brain, and writing test data into that state would corrupt the lab in a way later scenarios would report as mysterious replication errors.

Measured:

BACKUP_DESTINATION=s3://mysql-lab-backups/lab/lab-2026-09-12-10:13:48-full
BACKUP_CHUNKS=127
BACKUP_BYTES=708418853
MD5_OBJECTS=1
CONTROLLED_WRITE_ROWS=20000
BINLOG_BYTES_BEFORE=42303809
BINLOG_BYTES_AFTER=63112573
BINLOG_BYTE_GROWTH=20808764

127 chunks, 708 MB, and 20.8 MB of binary log archived in response to a 20 MB write. That last line is the one that matters for part 7: point-in-time recovery is only possible if the binlogs are actually reaching durable storage, and this is the scenario that proves they are rather than trusting that pitr.enabled: true did what it said.

Doing it by hand

export KUBECONFIG="$PWD/.kube/config"

# t01 — a full backup
kubectl -n mysql apply -f - <<'YAML'
apiVersion: ps.percona.com/v1
kind: PerconaServerMySQLBackup
metadata: { name: t01-full, namespace: mysql }
spec:
  clusterName: lab
  storageName: minio
  type: full
YAML

kubectl -n mysql get ps-backup t01-full -w
kubectl -n mysql get ps-backup t01-full -o jsonpath='{.status.state} {.status.destination}{"\n"}'

# watch the job the operator made for it
kubectl -n mysql get jobs | grep xb-t01-full
kubectl -n mysql logs job/$(kubectl -n mysql get jobs -o name | grep xb-t01-full | head -1 | cut -d/ -f2)

# t02 — from the replica
kubectl -n mysql exec lab-mysql-1 -c mysql -- mysql -uroot -pmysql -N -e "SELECT @@read_only"   # expect 1
kubectl -n mysql apply -f - <<'YAML'
apiVersion: ps.percona.com/v1
kind: PerconaServerMySQLBackup
metadata: { name: t02-replica, namespace: mysql }
spec:
  clusterName: lab
  storageName: minio
  type: full
  sourcePod: lab-mysql-1
YAML
kubectl -n mysql get ps-backup t02-replica -o jsonpath='{.status.backupSource}{"\n"}'

# t03 — an incremental on that base
kubectl -n mysql exec lab-mysql-0 -c mysql -- mysql -uroot -pmysql -e \
  "CREATE TABLE IF NOT EXISTS sbtest.t03_delta (id INT AUTO_INCREMENT PRIMARY KEY, pad CHAR(120));
   INSERT INTO sbtest.t03_delta (pad) SELECT LEFT(MD5(RAND()),120) FROM sbtest.sbtest1 LIMIT 50000;"
kubectl -n mysql apply -f - <<'YAML'
apiVersion: ps.percona.com/v1
kind: PerconaServerMySQLBackup
metadata: { name: t03-incr, namespace: mysql }
spec:
  clusterName: lab
  storageName: minio
  type: incremental
  incrementalBaseBackupName: t01-full
YAML

# t04 — look at the bytes
kubectl -n mysql run mc --rm -i --restart=Never \
  --image=quay.io/minio/mc:RELEASE.2024-11-21T17-21-54Z --command -- sh -c '
    mc alias set local http://minio:9000 minio minio12345 >/dev/null
    mc du local/mysql-lab-backups/lab
    mc ls --recursive local/mysql-lab-binlogs/lab/ | tail -5'

Or, with the harness:

./scripts/lab.sh t01
./scripts/lab.sh t02
./scripts/lab.sh t03
./scripts/lab.sh t04

What can go wrong here

• Re-applying a backup object does nothing. It is not an error and it is not a new backup. Delete and recreate, or use a new name.
• sourcePod names a pod that does not exist. The job is created and fails. Check status.stateDescription.
• The incremental’s base was deleted. Deleting a backup object removes its data from storage; an incremental whose base is gone is an unrestorable chain. Retention (part 5) can do this to you on a schedule.
• mc says the prefix is empty. Check the prefix in the cluster’s storage definition. s3://mysql-lab-backups/lab/… — the lab segment is the storage prefix, and forgetting it is the same mistake that breaks cross-cluster restores in part 8.
• Binlog bytes never grow. Either PITR is not enabled, the binlog server pod is not Running, or it cannot authenticate to MinIO. kubectl -n mysql logs lab-binlog-server-0 -c binlog-server says which.

Next

Backups happen when a human asks. Part 5 makes them happen when nobody is watching, and then proves the much less popular half of a backup policy: that old backups actually go away, data and all, when retention says they should.