MySQL backup lab, part 5 – scheduled backups and retention

This is part 5 of nine. The whole series builds one MySQL backup verification lab and explains every playbook, template and script in it.

---

Part 4’s backups all happened because somebody asked. Real backups happen at 03:15 while everyone is asleep, and the two properties that matter about them are:

1. they run without anyone doing anything, and
2. the old ones go away — object and data — so the bucket does not grow without limit.

The second is the one nobody tests. It is also the one that fails in the most expensive way, in either direction: retention that does not prune means an ever-growing storage bill; retention that prunes too eagerly means the backup you need is the one that was deleted this morning. Worse, it can silently break an incremental chain (part 4) by removing a base.

Scenarios t05 and t06 cover both. Measured durations from the ordered validation run: 207 s and 11 s.

How the operator schedules backups

A schedule lives on the cluster, not on a separate object:

spec:
  backup:
    schedule:
      - name: hourly-full
        schedule: "15 * * * *"
        keep: 3
        storageName: minio
        type: full

The operator runs its own cron loop and creates PerconaServerMySQLBackup objects when an entry is due. They are the same kind as the ones part 4 created by hand, which is a good design: everything that works for an on-demand backup — status, destination, restore — works identically for a scheduled one.

The difference is in the name and two labels:

name: cron-lab-minio-20260912101829-efvvj

percona.com/backup-type: cron
percona.com/backup-ancestor: f7ac8-suite-fast

Naming is worth spelling out because it causes confusion in kubectl get output. The backup object is cron-<cluster>-<storage>-<stamp>-<hash>. The xb-cron-* names you also see belong to the Jobs and Pods the operator created to execute it. They are different objects; grepping for the wrong one makes scheduled backups look like they vanished.

The backup-ancestor label is <hash>-<schedule name>. It ties every backup to the schedule entry that produced it, which is what makes keep: N scoped rather than global: retention counts backups per ancestor, so two schedules with different retention do not prune each other’s history.

t05 — Scheduled backup runs unattended

Proves: the operator’s cron produces a successful backup with nobody creating the object.

Waiting for 15 * * * * would make this scenario take up to an hour. So it temporarily patches the schedule to every three minutes — and the design problem that creates is the interesting part of the scenario.

Save first, and restore no matter what

- block:
    - name: Save the actual backup schedule before testing
      kubernetes.core.k8s_info:
        api_version: ps.percona.com/v1
        kind: PerconaServerMySQL
        name: "{{ mysql_cluster_name }}"
      register: t05_cluster_before

    - name: Preserve the prior backup schedule
      ansible.builtin.set_fact:
        t05_prior_schedule: "{{ t05_cluster_before.resources[0].spec.backup.schedule | default([]) }}"

    # … patch, watch, assert …

  always:
    - name: Restore the exact prior backup schedule
      kubernetes.core.k8s:
        state: patched
        api_version: ps.percona.com/v1
        kind: PerconaServerMySQL
        name: "{{ mysql_cluster_name }}"
        definition:
          spec:
            backup:
              schedule: "{{ t05_prior_schedule }}"
      when: t05_prior_schedule is defined

ansible/roles/t05_backup_scheduled/tasks/main.yml

Three decisions here, and they are the ones that separate a test you can run in a suite from one you can run once.

The prior schedule is read from the live cluster, not from group_vars. They should be the same. They are not necessarily the same — a previous scenario, an interrupted run, or a manual experiment may have left something else in place. Restoring what was actually there is the only correct behaviour.

The restore is in always:. If the patch succeeds and the watch then times out, the failure path still puts the schedule back. Without this, one failed scenario leaves a cluster taking a full backup every three minutes, forever, which quietly fills the 5Gi MinIO volume and makes every subsequent scenario fail for reasons that have nothing to do with what they test.

The when: guard covers the case where the scenario failed before the fact was even set, so cleanup does not itself fail and mask the original error.

There is a shipped offline test for this — test_schedule_scenarios_restore_the_actual_prior_schedule — because it is exactly the kind of property that survives review and then gets refactored away.

Knowing which backup is new

- name: Note the scheduled backups that already exist
  kubernetes.core.k8s_info:
    api_version: ps.percona.com/v1
    kind: PerconaServerMySQLBackup
    label_selectors:
      - percona.com/backup-type=cron
  register: t05_before

- name: Remember their names
  ansible.builtin.set_fact:
    t05_seen_before: "{{ t05_before.resources | default([]) | map(attribute='metadata.name') | list }}"

The lab’s default hourly-full schedule has probably already produced backups. Any of them could satisfy a naive “is there a successful cron backup?” check — and the scenario would pass having proven nothing about the current cluster. Recording the names first turns the question into “did a backup appear that did not exist when I started?”, which is the question actually worth asking.

The validation run recorded PRE_EXISTING_CRON_BACKUPS=3.

The watcher

- name: Patch the cluster to a fast backup schedule
  kubernetes.core.k8s:
    state: patched
    definition:
      spec:
        backup:
          schedule:
            - name: "{{ suite_schedule_name }}"      # suite-fast
              schedule: "{{ suite_fast_schedule }}"  # */3 * * * *
              keep: "{{ backup_keep }}"
              storageName: "{{ backup_storage_name }}"
              type: full

- name: Wait for a new successful backup of the exact cluster and schedule
  ansible.builtin.command:
    argv: ["{{ ansible_playbook_python }}", "{{ role_path }}/files/watch.py"]
  environment:
    KUBECONFIG: "{{ kubeconfig_path }}"
    NS: "{{ mysql_namespace }}"
    CLUSTER: "{{ mysql_cluster_name }}"
    SCHEDULE: "{{ suite_schedule_name }}"
    SEEN: "{{ t05_seen_before | to_json }}"

state: patched is a strategic merge patch, not an apply — it changes spec.backup.schedule and leaves the rest of a large CR untouched. This is also what makes jsonpatch a real dependency (part 1).

The eligibility test is the whole scenario in one function:

def eligible(item, cluster, schedule, seen):
    labels = item['metadata'].get('labels', {})
    return (item['metadata']['name'] not in seen
            and item.get('spec', {}).get('clusterName') == cluster
            and labels.get('percona.com/backup-type') == 'cron'
            and labels.get('percona.com/backup-ancestor', '').partition('-')[2] == schedule
            and item.get('status', {}).get('state', '').lower() == 'succeeded'
            and bool(item.get('status', {}).get('destination')))

ansible/roles/t05_backup_scheduled/files/watch.py

Five conditions, each removing a specific way to pass without proving anything:

.partition('-')[2] splits f7ac8-suite-fast after the first hyphen, keeping the schedule name — which is why a schedule name containing a hyphen still works.

The loop polls for 15 minutes, which is five chances at a three-minute schedule.

Measured:

SCHEDULE_PATCHED=*/3 * * * * name=suite-fast
PRE_EXISTING_CRON_BACKUPS=3
NEW_BACKUP=cron-lab-minio-20260912101829-efvvj
NEW_BACKUP_ANCESTOR=f7ac8-suite-fast
NEW_BACKUP_STATE=Succeeded
NEW_BACKUP_DESTINATION=s3://mysql-lab-backups/lab/lab-2026-09-12-10:18:29-full
NEW_BACKUP_COMPLETED=2026-09-12T10:18:40Z

207 seconds — the scenario is almost entirely waiting for a cron tick. It is also, by some distance, the least clever scenario in the suite, and that is correct: the thing being tested is that nothing clever is required.

t06 — Retention prunes scheduled backups down to keep

Proves: keep: N deletes the oldest backups of that schedule, and removes their data from object storage, not just the Kubernetes object.

That second clause is the entire point. A retention policy that deletes CRs and orphans their data in the bucket is worse than no retention: your object storage keeps growing and your kubectl get ps-backup output says everything is tidy.

schedule:
  - name: suite-fast
    schedule: "*/3 * * * *"
    keep: "{{ suite_fast_keep }}"      # 2
    storageName: minio
    type: full

Same save-and-restore-in-always structure as t05. Same watcher shape, doing considerably more work.

Observing rather than assuming

def matching(items, cluster, schedule):
    return {x['metadata']['name']: x for x in items
            if x.get('spec', {}).get('clusterName') == cluster
            and x['metadata'].get('labels', {}).get('percona.com/backup-type') == 'cron'
            and x['metadata'].get('labels', {}).get('percona.com/backup-ancestor', '').partition('-')[2] == schedule}


def succeeded(item):
    return item.get('status', {}).get('state', '').lower() == 'succeeded' and bool(item.get('status', {}).get('destination'))


def main():
    keep = int(os.environ['KEEP'])
    seen = {}
    deadline = time.monotonic() + 1500
    while time.monotonic() < deadline:
        current = matching(json.loads(kubectl('get', 'ps-backup', '-o', 'json'))['items'], …)
        for name, item in current.items():
            if succeeded(item):
                seen[name] = item['status']['destination']
        pruned = sorted(set(seen) - set(current))
        surviving = [name for name, item in current.items() if succeeded(item)]
        if len(seen) >= keep + 1 and len(surviving) == keep and pruned:
            …

ansible/roles/t06_backup_retention/files/watch.py

seen accumulates every successful backup this watcher has personally observed, keyed by name, with its destination. current is what exists right now. The difference is what was pruned while the watcher was looking.

The comment in the source states the property this buys:

Each removed CR was observed successfully completed, so failed jobs and transient name-list changes cannot masquerade as retention.

Without it, a scheduled backup that failed and was cleaned up would count as evidence of retention. The test would pass in a world where retention is broken.

The gate is three conditions together:

• len(seen) >= keep + 1 — at least one more successful backup existed than keep allows, so pruning had something to do;
• len(surviving) == keep — exactly keep remain, not “at most”;
• pruned is non-empty — something actually disappeared.

Then it checks the bucket

for name in pruned:
    path = seen[name].removeprefix('s3://').rstrip('/')
    parent, basename = path.rsplit('/', 1)
    listing = objects(parent + '/')
    remaining = [x for x in listing if x['key'].startswith(basename + '/')]
    if remaining:
        break
    absent.append(name)
    leftovers.extend(x['key'] for x in listing if x['key'] == basename + '.md5')
else:
    evidence('KEEP', keep)
    …

It lists the parent prefix and filters, rather than listing the deleted prefix directly. Listing a prefix that does not exist returns an empty result, which is indistinguishable from “it exists and is empty” — and, as part 4 noted, also indistinguishable from an error if you are careless. Listing the parent and looking for children is a positive check with an unambiguous answer.

The for … else is Python’s loop-else: the else block runs only if the loop completed without break. So the evidence is emitted only when every pruned backup’s data prefix was confirmed empty. One survivor breaks out and the watcher keeps polling.

The .md5 sidecar from part 4 reappears here — and it is reported, not asserted:

evidence('MD5_SIDECARS_REMAINING', json.dumps(leftovers))
evidence('RETENTION_SCOPE', 'successful backup CRs and data prefixes; sibling MD5 sidecars reported separately')

This is an observed behaviour of the operator in this build: the backup’s data is removed, and a small checksum object beside it is not. It is a handful of bytes, so it is not a storage problem; it is a fact about the system, so it is recorded rather than hidden. Documenting the limit of what a test proves is part of the test. An assertion that ignored the sidecar would be a quiet lie about completeness; one that failed on it would report a broken cluster.

Measured:

SCHEDULE=*/3 * * * *
KEEP=2
PRODUCED_COUNT=3
PRODUCED=cron-lab-minio-20260912090638-mu0gh,cron-lab-minio-20260912090938-mu0gh,cron-lab-minio-20260912101829-efvvj
SURVIVING_COUNT=2
SURVIVING=cron-lab-minio-20260912090938-mu0gh,cron-lab-minio-20260912101829-efvvj
PRUNED=cron-lab-minio-20260912090638-mu0gh
REMOVED_DATA_PREFIX=s3://mysql-lab-backups/lab/lab-2026-09-12-09:06:38-full/
MD5_SIDECARS_REMAINING=["lab-2026-09-12-09:06:38-full.md5"]
RETENTION_SCOPE=successful backup CRs and data prefixes; sibling MD5 sidecars reported separately

11 seconds, because two of the three observed backups already existed — one from 09:06, one from 09:09, plus the one t05 had just produced at 10:18. The scenario did not have to wait for three fresh cron ticks; it had history to prune immediately. That is why its duration is not comparable to t05’s, and why the catalogue notes it explicitly rather than letting a reader infer that retention is twenty times faster than scheduling.

What this pair does and does not establish

Established: the operator’s cron creates backups unattended; keep: N leaves exactly N successful backups of that schedule; pruning removes the backup object and its data prefix.

Not established:

• Retention across multiple schedules interacting.
• What happens when retention prunes the base of an incremental chain. The operator counts backups per ancestor; it does not, as far as this lab tested, refuse to delete a full backup that an incremental depends on. If you run incrementals, treat that as an open question in your own environment rather than an answered one.
• Anything about a restore. Everything in parts 4 and 5 concerns backups being produced. Whether they can be used starts in part 6.

Doing it by hand

export KUBECONFIG="$PWD/.kube/config"

# what is scheduled right now — save this
kubectl -n mysql get ps lab -o jsonpath='{.spec.backup.schedule}' | tee /tmp/prior-schedule.json; echo

# what scheduled backups already exist — note these names
kubectl -n mysql get ps-backup -l percona.com/backup-type=cron \
  -o custom-columns=NAME:.metadata.name,STATE:.status.state,ANCESTOR:.metadata.labels.percona\\.com/backup-ancestor

# speed the schedule up, keeping only two
kubectl -n mysql patch ps lab --type merge -p '{"spec":{"backup":{"schedule":[
  {"name":"suite-fast","schedule":"*/3 * * * *","keep":2,"storageName":"minio","type":"full"}]}}}'

# watch new objects appear, and old ones disappear
kubectl -n mysql get ps-backup -l percona.com/backup-type=cron -w

# confirm a pruned backup's data is really gone (use its recorded destination)
kubectl -n mysql run mc --rm -i --restart=Never \
  --image=quay.io/minio/mc:RELEASE.2024-11-21T17-21-54Z --command -- sh -c '
    mc alias set local http://minio:9000 minio minio12345 >/dev/null
    mc ls --recursive local/mysql-lab-backups/lab/ | grep "2026-09-12-09:06:38" || echo "PREFIX EMPTY"'

# PUT THE SCHEDULE BACK
kubectl -n mysql patch ps lab --type merge \
  -p "{\"spec\":{\"backup\":{\"schedule\":$(cat /tmp/prior-schedule.json)}}}"

That last step is not optional. A forgotten */3 * * * * will fill the MinIO volume overnight.

Or let the harness handle the save/restore for you:

./scripts/lab.sh t05
./scripts/lab.sh t06

What can go wrong here

• No new cron backup within the timeout. The patch did not apply (check kubectl -n mysql get ps lab -o jsonpath='{.spec.backup.schedule}'), or the backup is failing — look for xb-cron-* Jobs and read their logs.
• SURVIVING_COUNT is higher than keep. The ancestor label does not match the schedule name you patched, so the retention counter is scoped to a different history than the one you are watching.
• The data prefix is not empty after pruning. The object was removed and its data was not. That is the failure this scenario exists to catch — do not dismiss it as a listing artefact without reading the listing.
• Storage fills up. Almost always a fast schedule left in place by an interrupted run. Check the schedule before blaming the backups.

Next

Five parts in, the lab has produced backups on demand, from a replica, as incrementals, on a schedule, and pruned them. Not one of those proves a database can be recovered. Part 6 restores one over the running cluster and measures what it costs — including the part nobody enjoys, which is that the cluster is down while it happens.