MySQL backup lab, part 1 – the kind cluster underneath it

This is part 1 of nine. The whole series builds one MySQL backup verification lab and explains every playbook, template and script in it.

---

This series takes one lab apart line by line. The lab’s question is narrow and unfashionable: is this backup actually any good? Not “did the backup job go green”, which is a different and much weaker claim.

Answering it needs a database worth backing up, an object store to back it up into, a way to break things on purpose, and a way to record what happened that does not depend on anybody’s memory. That is what gets built over the next nine posts, and every file in it is explained — why it exists, what it produces, and what the alternative would have cost.

This first part is the substrate: a three-node Kubernetes cluster running as Docker containers on a laptop, and the Ansible that creates it.

Everything here lives in the lab directory 20260911-mysql-replication-backups/. Run commands from there.

What the finished thing looks like

Twelve scenarios then run against it. They are the subject of parts 4 through 8.

Why kind, and why three nodes

A single-node cluster would have been simpler. Three nodes were chosen because the operator’s own behaviour is node-aware: it schedules the source, the replica, HAProxy, Orchestrator and the binlog server as separate pods, and with one node every anti-affinity decision collapses into a no-op that hides scheduling mistakes until you try the same manifests somewhere real.

kind runs each Kubernetes node as a Docker container on the same host, so three nodes cost three containers, not three VMs. The trade is honest and worth stating up front: the nodes share a kernel, a page cache and one disk. Any timing number produced here is a laptop number. The lab is a correctness demonstration, not a benchmark, and nothing in the series quotes a duration as an RTO target.

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: {{ kind_cluster_name }}
nodes:
  - role: control-plane
    extraPortMappings:
      - containerPort: {{ kind_nodeport_mysql }}      # 30306
        hostPort: {{ kind_host_mysql_port }}          # 3307
        protocol: TCP
      - containerPort: {{ kind_nodeport_grafana }}    # 30300
        hostPort: {{ kind_host_grafana_port }}        # 3001
        protocol: TCP
      - containerPort: {{ kind_nodeport_prometheus }} # 30091
        hostPort: {{ kind_host_prometheus_port }}     # 9091
        protocol: TCP
  - role: worker
  - role: worker

ansible/roles/kind_cluster/templates/kind-config.yaml.j2

Why the port mappings are only on the control-plane

A Kubernetes NodePort service listens on every node, and kube-proxy forwards from whichever node received the packet to whichever node runs the pod. So one node’s ports being reachable from the host is enough to reach any pod in the cluster. Mapping the same three ports on all three containers would add two more chances for a port conflict and buy nothing.

The result is three fixed host endpoints:

Why those odd host ports

3307, 3001 and 9091 rather than 3306, 3000 and 9090. Each lab in this repo picks a non-overlapping host port range so two labs can run at once, and so a local MySQL or Grafana already on the standard port is not shadowed. The NodePort numbers on the cluster side (30306, 30300, 30091) encode the same intent.

Every one of these values is a variable, not a literal, so a port clash is a one-line fix in group_vars rather than an edit spread across templates:

kind_host_mysql_port: 3307
kind_host_grafana_port: 3001
kind_host_prometheus_port: 9091
kind_nodeport_mysql: 30306
kind_nodeport_grafana: 30300
kind_nodeport_prometheus: 30091

ansible/inventory/group_vars/all.yml

The driver: Ansible against localhost

There is exactly one host in the inventory, and it is this laptop:

all:
  hosts:
    localhost:
      ansible_connection: local
      ansible_python_interpreter: "{{ inventory_dir }}/../../.venv/bin/python"

ansible/inventory/localhost.yml

Two deliberate details.

ansible_connection: local — no SSH, no control master, no inventory of remote machines. Ansible here is not a configuration-management-over-the-network tool; it is a task runner with idempotence, retries and templating already solved. The actual remote system is the Kubernetes API, and the kubernetes.core modules talk to it directly.

The pinned interpreter is not a style preference, it is the single most common way this lab fails for someone else. The kubernetes.core modules import the kubernetes Python package in the interpreter Ansible is using. If that resolves to a system Python — Homebrew’s, or macOS’s — the import fails with a message about a missing library that is, in fact, installed, just not where the playbook is looking. Pinning the interpreter to the project virtualenv removes the ambiguity.

[defaults]
inventory = inventory/localhost.yml
roles_path = roles
collections_path = ./collections:~/.ansible/collections
timeout = 1200
host_key_checking = False
retry_files_enabled = False
stdout_callback = default
interpreter_python = auto_silent

ansible/ansible.cfg

collections_path puts ./collections first, so ansible-galaxy collection install -p ansible/collections produces a copy that belongs to this lab and cannot be changed by another project’s install. timeout = 1200 is generous because several tasks here legitimately wait many minutes — a restore of a 500 MB dataset on a laptop is not fast. retry_files_enabled = False keeps .retry droppings out of the tree.

Bootstrap: one virtualenv, pinned ranges

python3 -m venv .venv
.venv/bin/pip install -U pip
.venv/bin/pip install -r requirements.txt
.venv/bin/ansible-galaxy collection install -r ansible/requirements.yml -p ansible/collections

That is ./scripts/lab.sh bootstrap and make bootstrap, identically.

ansible-core>=2.17,<2.19
kubernetes>=31.0.0
PyYAML>=6.0
jsonpatch>=1.33
jinja2>=3.1

requirements.txt

ansible-core is range-pinned rather than floating. The suite leans heavily on templated when: and until: conditions, and a floating requirement means a routine pip install -U can change how those evaluate on a machine that ran the lab successfully last week. A bounded range makes “it worked yesterday” a checkable statement. jsonpatch is not decorative — kubernetes.core.k8s needs it for state: patched, which parts 5 and 7 use to change the backup schedule and the binlog archive prefix on a live cluster.

collections:
  - name: kubernetes.core
    version: ">=5.0.0"
  - name: community.general
    version: ">=10.0.0"

ansible/requirements.yml

community.general is here for exactly one module — homebrew — used to install kind, kubectl and helm on macOS.

Two entry points, one set of commands

The repo ships both a Makefile and scripts/lab.sh, with the same targets:

./scripts/lab.sh bootstrap     # venv + Ansible + collections
./scripts/lab.sh deploy        # kind + operator + MinIO + cluster + monitoring
./scripts/lab.sh load-data     # sysbench prepare
./scripts/lab.sh verify        # assert the cluster is in the expected state
./scripts/lab.sh test-backups  # the twelve scenarios, in order, then the report
./scripts/lab.sh teardown      # delete the kind cluster

make deploy, make test-backups and so on are equivalent. The duplication exists because make is not always present on a fresh macOS install without Xcode command-line tools, and asking a reader to install a build toolchain before they can create a database cluster is a bad first step.

The shell wrapper is deliberately thin — it resolves the repo root, picks the virtualenv’s ansible-playbook, and dispatches:

ROOT="$(cd "$(dirname "$0")/.." && pwd)"
cd "$ROOT"
VENV="${ROOT}/.venv"
ANSIBLE="${VENV}/bin/ansible-playbook"
...
  deploy)
    (cd ansible && "$ANSIBLE" playbooks/site.yml "$@")
    ;;

scripts/lab.sh

Two things it does that matter later. It forwards "$@", so ./scripts/lab.sh deploy -e mysql_size=3 -vv works without the wrapper knowing anything about Ansible flags. And for the scenario targets it supplies a run identifier by default:

  t0[1-9]|t1[0-2])
    playbook=$(basename "$(ls ansible/playbooks/tests/${cmd}-*.yml | head -1)")
    (cd ansible && "$ANSIBLE" "playbooks/tests/$playbook" -e run_id="${RUN_ID:-$(date -u +%Y%m%dT%H%M%SZ)}" "$@")
    ;;

Every scenario run is stamped with a run_id, and every result it records lands in a file named after it. Part 9 is about that machinery; the important thing here is that it is impossible to run a scenario without recording evidence.

The deploy playbook

- name: Deploy localhost MySQL replication lab
  hosts: localhost
  gather_facts: true
  vars:
    kubeconfig_path: "{{ (playbook_dir ~ '/../..') | realpath }}/.kube/config"
  environment:
    PATH: "/opt/homebrew/bin:/usr/local/bin:{{ lookup('env', 'PATH') }}"
    KUBECONFIG: "{{ (playbook_dir ~ '/../..') | realpath }}/.kube/config"
  roles:
    - prereqs
    - kind_cluster
    - mysql_operator
    - minio
    - mysql_cluster
    - monitoring

ansible/playbooks/site.yml

Six roles, in a strict order, and the order is a dependency chain:

1. prereqs — the tools exist and Docker is running.
2. kind_cluster — there is a Kubernetes API to talk to.
3. mysql_operator — the ps.percona.com CRDs exist, so a PerconaServerMySQL object is something the API server will accept.
4. minio — the backup target exists before a cluster is told to back up into it.
5. mysql_cluster — the database itself.
6. monitoring — Prometheus and Grafana last, because they observe everything above and nothing above depends on them.

The environment: block is worth a paragraph. It sets two variables for every task in the play:

• PATH is prefixed with Homebrew’s locations. Ansible’s local connection does not run a login shell, so a PATH that works in your terminal is not necessarily the PATH the playbook sees. Without this, kind and helm resolve on an Intel Mac and vanish on Apple silicon, or the reverse.
• KUBECONFIG points at a repo-local kubeconfig, not ~/.kube/config.

That second one is a deliberate safety property. This lab creates clusters, destroys clusters, and runs destructive scenarios that drop tables and roll databases back. It should not be able to do any of that to whatever cluster your personal kubeconfig happens to have as its current context. Writing the kubeconfig into the project directory — gitignored — means the blast radius is the lab:

export KUBECONFIG="$PWD/.kube/config"
kubectl get ps,pods -n mysql

Note that both vars: and environment: compute the same path independently. The variable is for Ansible modules that take a kubeconfig: parameter; the environment variable is for the raw kubectl invocations inside shell tasks. They must agree, so both are derived from playbook_dir rather than typed twice.

Role 1: prereqs — fail early, fail clearly

- name: Require Docker daemon
  ansible.builtin.command: docker info
  register: docker_info
  changed_when: false
  failed_when: docker_info.rc != 0

- name: Install Homebrew packages on macOS
  community.general.homebrew:
    name: [kind, kubectl, helm]
    state: present
  when: ansible_facts['os_family'] == 'Darwin'

- name: Check kind is on PATH
  ansible.builtin.command: kind version
  changed_when: false

ansible/roles/prereqs/tasks/main.yml (abridged — kubectl and helm get the same treatment)

docker info is the first task in the whole deploy for a reason. Without it, the failure arrives eight minutes later as a timeout inside kind create cluster, and the message is about a container runtime socket rather than “Docker Desktop is not running”. One second spent on a clear error beats eight minutes spent on a confusing one.

changed_when: false on the version checks is not cosmetic. Ansible treats a command as changed by default; marking read-only probes correctly means the final changed=N count is a real signal, and a re-run of deploy against an existing lab reports almost nothing changed — which is how you can tell idempotence is working.

The Homebrew install is gated on macOS. On Linux the tools are expected to be present already, and the three version checks that follow turn their absence into a named failure rather than a command not found inside a later role.

Role 2: kind_cluster — creating, and not resurrecting

The straightforward half:

- name: Ensure kubeconfig directory exists
  ansible.builtin.file:
    path: "{{ kubeconfig_path | dirname }}"
    state: directory
    mode: "0700"

- name: Ensure rendered manifest directories exist
  ansible.builtin.file:
    path: "{{ repo_root }}/k8s/{{ item }}"
    state: directory
    mode: "0755"
  loop: [mysql, monitoring, backup]

- name: Render kind cluster config
  ansible.builtin.template:
    src: kind-config.yaml.j2
    dest: "{{ repo_root }}/k8s/kind-config.yaml"
    mode: "0644"

- name: Check whether kind cluster exists
  ansible.builtin.command: kind get clusters
  register: kind_clusters
  changed_when: false

- name: Create kind cluster
  ansible.builtin.command: >-
    kind create cluster
    --name {{ kind_cluster_name }}
    --config {{ repo_root }}/k8s/kind-config.yaml
    --kubeconfig {{ kubeconfig_path }}
  when: kind_cluster_name not in kind_clusters.stdout_lines
  timeout: 600

0700 on the kubeconfig directory because it holds cluster admin credentials. kind get clusters plus a when: is the idempotence guard: kind create cluster is not idempotent and errors on an existing name, so the check is explicit rather than implied by a swallowed failure. timeout: 600 covers a first run that has to pull the node image.

Rendered manifests are an output, not an input

Every template in this lab renders to k8s/ before being applied:

k8s/kind-config.yaml
k8s/mysql/cluster.yaml
k8s/mysql/minio.yaml
k8s/mysql/secrets.yaml
k8s/mysql/nodeports.yaml
k8s/mysql/verify-cluster.yaml
k8s/monitoring/kube-prometheus-values.yaml
k8s/monitoring/servicemonitors.yaml
k8s/monitoring/grafana-dashboard.yaml
k8s/backup/ondemand.yaml
k8s/backup/restore.yaml

The alternative — building the manifest inline in the module call — works fine and is used in this lab for small, single-purpose objects. For anything with structure, rendering to a file first buys three things: you can cat exactly what was sent to the API server, you can kubectl apply -f it yourself without Ansible, and a review diff shows the rendered change rather than a change to a template you then have to evaluate in your head.

The docker update --restart=no block

This is the least obvious code in the role, and it exists because of a real failure:

- name: Inspect kind node restart policies
  ansible.builtin.command:
    argv: [docker, inspect, --format, "{{ '{{' }}.HostConfig.RestartPolicy.Name{{ '}}' }}", "{{ item }}"]
  loop: "{{ kind_node_ids.stdout_lines }}"
  register: kind_restart_inspect
  changed_when: false

# kind defaults to --restart=on-failure:1, which starts nodes again when
# Docker comes up after an unclean stop (exit 137). Keep them stopped until
# an explicit deploy or docker start.
- name: Disable Docker auto-start of kind node containers
  ansible.builtin.command:
    argv: [docker, update, --restart=no, "{{ item.item }}"]
  loop: "{{ kind_restart_inspect.results }}"
  when: item.stdout != "no"

kind creates its node containers with --restart=on-failure:1. When Docker Desktop is quit or the laptop sleeps badly, the containers exit non-zero (137, SIGKILL), and Docker helpfully restarts them the next time it starts — which means a three-node Kubernetes cluster and a MySQL cluster silently come back to life in the background hours later, eating RAM and, worse, resuming the binlog server’s stream into object storage. Setting --restart=no makes cluster lifetime explicit: it runs when you deploy it, and it is off otherwise.

The {{ '{{' }} … {{ '}}' }} escaping is Jinja getting out of the way of Go templates — docker inspect --format uses the same brace syntax Ansible does, so the braces have to be emitted as literals.

The when: item.stdout != "no" guard keeps a re-run from reporting three changed tasks every single time.

The complementary task starts anything that was left stopped:

- name: Start stopped kind node containers
  ansible.builtin.command:
    argv: "{{ ['docker', 'start'] + kind_stopped_node_ids.stdout_lines }}"
  when: kind_stopped_node_ids.stdout_lines | length > 0

Note the argv list built by concatenation rather than a shell string. No quoting, no word splitting, no surprises if a container ID ever contains something exotic.

Finally, the role does not declare victory until Kubernetes agrees:

- name: Wait for Kubernetes nodes to be Ready
  ansible.builtin.command: >-
    kubectl wait --for=condition=Ready nodes --all --timeout=180s
    --kubeconfig {{ kubeconfig_path }}
  changed_when: false

kind create cluster returning 0 means the containers started. kubectl wait means the kubelets registered and the CNI is up. The next role installs a Helm chart and would fail confusingly against a cluster that is not yet schedulable.

Teardown, and why it deletes the kubeconfig too

- name: Delete kind cluster
  ansible.builtin.command:
    cmd: "kind delete cluster --name {{ kind_cluster_name }}"
  register: kind_delete
  changed_when: kind_delete.rc == 0
  failed_when: false

- name: Remove project kubeconfig
  ansible.builtin.file:
    path: "{{ kubeconfig_path }}"
    state: absent

ansible/playbooks/teardown.yml

failed_when: false makes teardown safe to run against a lab that is already gone — which is the state you are usually in when you reach for it. Deleting the kubeconfig matters because a stale one is actively misleading: kubectl get pods against a deleted cluster produces a connection-refused error that reads like a network problem rather than “this cluster does not exist”.

Teardown deletes the kind cluster and everything inside it, including the MinIO PVC holding every backup. That is intentional. These labs exist to be rebuilt.

Doing it by hand

Without Ansible, the whole of this post is:

# tools
brew install kind kubectl helm
docker info    # must succeed

# cluster
cat > kind-config.yaml <<'YAML'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: mysql-lab
nodes:
  - role: control-plane
    extraPortMappings:
      - { containerPort: 30306, hostPort: 3307, protocol: TCP }
      - { containerPort: 30300, hostPort: 3001, protocol: TCP }
      - { containerPort: 30091, hostPort: 9091, protocol: TCP }
  - role: worker
  - role: worker
YAML

kind create cluster --name mysql-lab --config kind-config.yaml --kubeconfig "$PWD/.kube/config"
export KUBECONFIG="$PWD/.kube/config"
kubectl wait --for=condition=Ready nodes --all --timeout=180s

# stop Docker from resurrecting the nodes later
docker ps -aq --filter 'label=io.x-k8s.kind.cluster=mysql-lab' | xargs docker update --restart=no

kubectl get nodes

Expected:

NAME                      STATUS   ROLES           AGE   VERSION
mysql-lab-control-plane   Ready    control-plane   1m    v1.36.1
mysql-lab-worker          Ready    <none>          1m    v1.36.1
mysql-lab-worker2         Ready    <none>          1m    v1.36.1

(Server version v1.36.1 is what the recorded validation run captured in its environment snapshot; yours depends on the kind release you have.)

What can go wrong here

• Docker has too little memory. The lab wants about 8 GB and 4 CPUs. Below that, MySQL pods are OOM-killed during sysbench load and the failure surfaces as a replica that will not catch up — a symptom that sends you looking at replication when the problem is RAM.
• A host port is already bound. 3307, 3001, 9091. kind create cluster fails with a bind error naming the port; change the variable in group_vars and re-run.
• kubernetes imported from the wrong Python. The symptom is Failed to import the required Python library (kubernetes). The cause is almost always an ansible-playbook that is not the virtualenv’s. Run through ./scripts/lab.sh, or check .venv/bin/python -c 'import kubernetes'.
• A previous lab’s cluster is still running. kind get clusters lists it; docker ps shows the RAM it is using.

Next

The cluster is empty. Part 2 installs the Percona operator, stands MinIO up as the backup target, and goes through the PerconaServerMySQL custom resource field by field — including the flags that tell the operator you know your topology is unsafe, and the backup block that every later post depends on.