MySQL backup lab, part 2 – deploying MySQL by hand

This is part 2 of nine. The whole series builds one MySQL backup verification lab and explains every playbook, template and script in it.

---

Part 1 left a three-node kind cluster with nothing in it. This part fills it: the Percona Operator for MySQL, an in-cluster MinIO to back up into, and a PerconaServerMySQL custom resource that is explained field by field — because almost every later post in this series is about the consequences of one of those fields.

Every Ansible task below is paired with the kubectl or helm command it is standing in for, so the whole deployment can be done by hand.

The shape of it

kind cluster mysql-lab
└── namespace mysql
    ├── ps-operator                     (Helm release, the controller)
    ├── minio-0                         (StatefulSet, S3-compatible storage)
    ├── lab-mysql-0                     (source,  read_only=0)
    ├── lab-mysql-1                     (replica, read_only=1)
    ├── lab-haproxy-0                   (routes writes to whichever pod is source)
    ├── lab-orc-0                       (Orchestrator, failover management — present, not the subject)
    └── lab-binlog-server-0             (streams binlogs to MinIO, part 7)
└── namespace monitoring
    ├── kube-prometheus-stack
    └── Grafana

Two namespaces: mysql for everything the database needs, monitoring for the observability stack. They are separate because the monitoring stack is replaced and upgraded on a completely different cadence from the database, and because kubectl delete ns mysql should not take Prometheus with it.

Installing the operator

- name: Ensure MySQL namespace
  kubernetes.core.k8s:
    kubeconfig: "{{ kubeconfig_path }}"
    state: present
    definition:
      apiVersion: v1
      kind: Namespace
      metadata:
        name: "{{ mysql_namespace }}"

- name: Add Percona Helm repo
  kubernetes.core.helm_repository:
    name: percona
    repo_url: https://percona.github.io/percona-helm-charts/
    force_update: true

- name: Install Percona Server for MySQL operator
  kubernetes.core.helm:
    kubeconfig: "{{ kubeconfig_path }}"
    name: ps-operator
    chart_ref: percona/ps-operator
    chart_version: "{{ ps_operator_chart_version }}"     # 1.2.0
    release_namespace: "{{ mysql_namespace }}"
    create_namespace: true
    wait: true
    wait_timeout: "5m0s"

ansible/roles/mysql_operator/tasks/main.yml

Why an operator at all

You can run MySQL replication on Kubernetes with a StatefulSet and a pile of init containers. What you cannot easily do is make backup and restore first-class: an operator gives you PerconaServerMySQLBackup and PerconaServerMySQLRestore as API objects with a status you can poll, which is the entire reason this lab can be driven declaratively and its results recorded automatically. Parts 4 through 8 are, mechanically, “create an object, wait for its status, assert something about the database”.

Why the version is pinned

ps_operator_chart_version: "1.2.0" and ps_cr_version: "1.2.0" are two separate variables that happen to hold the same string, and they mean different things. The first selects the Helm chart — the controller image. The second goes into spec.crVersion on the custom resource, telling that controller which schema generation the object is written against.

Pinning matters more than usual here because this series documents behaviour, not just configuration: the timestamp format PITR accepts (part 7), the storage metadata a cross-cluster restore needs (part 8), and the way retention prunes (part 5) are all properties of this build. The recorded run captured the exact image:

docker.io/percona/percona-server-mysql-operator:1.2.0
  @sha256:28bfc38c1d4b642e860fda46aa7a964fc2de11a93a6dd05fcd7de4fd782982c8

force_update: true on the repository refreshes the index so a pinned chart version that was published after your last helm repo add is still findable.

Waiting for the operator, twice

- name: Wait for operator deployment
  ansible.builtin.command: >-
    kubectl -n {{ mysql_namespace }} rollout status
    deploy/ps-operator --timeout=180s --kubeconfig {{ kubeconfig_path }}
  changed_when: false
  register: operator_rollout
  failed_when: false

- name: Wait for any percona mysql operator pod
  kubernetes.core.k8s_info:
    kubeconfig: "{{ kubeconfig_path }}"
    kind: Pod
    namespace: "{{ mysql_namespace }}"
    label_selectors:
      - app.kubernetes.io/name=ps-operator
  register: op_pods
  until: op_pods.resources | selectattr('status.phase', 'equalto', 'Running') | list | length >= 1
  retries: 30
  delay: 5
  when: operator_rollout.rc | default(1) != 0

This looks redundant and is not. kubectl rollout status deploy/ps-operator is the fast, precise check — but it hard-codes the Deployment’s name, which is a chart detail that can change between chart versions. So the first task is allowed to fail (failed_when: false), and the second task runs only if it did, falling back to a label selector that survives a rename. Fast path first, resilient path as a fallback, and a re-run of deploy against a healthy lab skips the slow one entirely.

By hand

export KUBECONFIG="$PWD/.kube/config"
kubectl create namespace mysql
helm repo add percona https://percona.github.io/percona-helm-charts/
helm repo update
helm install ps-operator percona/ps-operator --version 1.2.0 -n mysql --wait --timeout 5m
kubectl -n mysql rollout status deploy/ps-operator --timeout=180s
kubectl get crd | grep ps.percona.com

That last line should list perconaservermysqls, perconaservermysqlbackups and perconaservermysqlrestores. If it does not, nothing later in this series will work.

MinIO: the backup target, deployed before the thing that backs up

Backups need somewhere to go, and “somewhere” should be object storage rather than a PVC, because the whole point of a backup is that it survives the loss of the thing that made it. In a laptop lab, real S3 is a cost, a credential and an internet dependency; MinIO is S3’s API in a single pod.

apiVersion: v1
kind: Secret
metadata:
  name: {{ minio_credentials_secret }}      # minio-backup-credentials
  namespace: {{ mysql_namespace }}
type: Opaque
stringData:
  AWS_ACCESS_KEY_ID: {{ minio_root_user }}
  AWS_SECRET_ACCESS_KEY: {{ minio_root_password }}
---
apiVersion: v1
kind: Secret
metadata:
  name: minio-root
  namespace: {{ mysql_namespace }}
type: Opaque
stringData:
  MINIO_ROOT_USER: {{ minio_root_user }}
  MINIO_ROOT_PASSWORD: {{ minio_root_password }}

ansible/roles/minio/templates/minio.yaml.j2 (first half)

Two secrets holding the same two values, and that is not an oversight. The key names are an interface:

• AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY are what the operator reads when a storage definition says credentialsSecret: minio-backup-credentials. Those names come from the AWS SDK, not from MinIO.
• MINIO_ROOT_USER / MINIO_ROOT_PASSWORD are what the MinIO server process reads from its environment at startup.

Merging them into one secret would work today and would couple two unrelated consumers to one object. They are separate because they are read by different software for different reasons.

stringData rather than data keeps the template readable — Kubernetes base64-encodes it on admission. These are lab credentials (minio / minio12345), printed in the README on purpose. Nothing here is a pattern for production secret management.

The server itself

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: minio
spec:
  serviceName: minio
  replicas: 1
  template:
    spec:
      containers:
        - name: minio
          image: {{ minio_image }}
          args: [server, /data, --console-address, ":9001"]
          envFrom:
            - secretRef:
                name: minio-root
          ports:
            - { name: api, containerPort: 9000 }
          volumeMounts:
            - { name: data, mountPath: /data }
          readinessProbe:
            httpGet: { path: /minio/health/ready, port: 9000 }
            initialDelaySeconds: 5
            periodSeconds: 5
  volumeClaimTemplates:
    - metadata: { name: data }
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 5Gi

A StatefulSet, not a Deployment, because the object store must keep its data across a pod restart and needs a stable identity bound to a stable volume. A Deployment with a PVC would mostly work with one replica and would be the wrong shape to explain.

The readiness probe on /minio/health/ready is what the Ansible wait actually depends on; without it a pod reports Running the instant the process starts and the bucket-creation Job that follows can race it.

5Gi is sized for the workload: the dataset is about 500 MB, and the recorded validation run accumulated 10GiB 2353 objects under the lab prefix across a whole suite of full and incremental backups plus scheduled ones. On a laptop that is a deliberate ceiling — when retention is broken, you find out by running out of space, which is exactly the failure mode part 5 is about.

Creating the buckets

- name: Delete previous bucket job if present
  kubernetes.core.k8s:
    state: absent
    api_version: batch/v1
    kind: Job
    name: minio-make-bucket
    wait: true
    wait_timeout: 60

- name: Create backup bucket
  kubernetes.core.k8s:
    state: present
    definition:
      apiVersion: batch/v1
      kind: Job
      metadata: { name: minio-make-bucket, namespace: "{{ mysql_namespace }}" }
      spec:
        backoffLimit: 6
        template:
          spec:
            restartPolicy: OnFailure
            containers:
              - name: mc
                image: "{{ minio_mc_image }}"
                command:
                  - /bin/sh
                  - -c
                  - |
                    mc alias set local http://minio:9000 "{{ minio_root_user }}" "{{ minio_root_password }}"
                    mc mb --ignore-existing local/{{ backup_bucket }}
                    mc mb --ignore-existing local/{{ binlog_bucket }}
                    mc ls local/

Four decisions in twenty lines.

The Job is deleted before it is created. Kubernetes Jobs are largely immutable — you cannot re-apply one with a changed pod template. Without the delete, a second deploy fails on an unchangeable field. Deleting first makes the role idempotent in the only way Jobs allow.

mc mb --ignore-existing makes the content idempotent too, so a re-run against existing buckets succeeds rather than erroring on “bucket already owned”.

backoffLimit: 6 with restartPolicy: OnFailure tolerates MinIO not being quite ready. Six retries with Kubernetes’ backoff is a couple of minutes of patience.

Two buckets, not one bucket with two prefixes. mysql-lab-backups holds XtraBackup output; mysql-lab-binlogs holds the binary log archive. They could have shared a bucket. Keeping them apart means mc ls shows the two streams independently, so “are binlogs actually being archived?” is answerable at a glance — a question part 7 leans on hard, and one that scenario t04 turns into an assertion about byte growth.

The wait afterwards is unusually explicit:

- name: Wait for bucket job
  kubernetes.core.k8s_info:
    api_version: batch/v1
    kind: Job
    name: minio-make-bucket
  register: bucket_job
  until:
    - bucket_job.resources | length == 1
    - bucket_job.resources[0].status.succeeded | default(0) | int == 1
      or (bucket_job.resources[0].status.failed | default(0) | int) >= 6
  retries: 36
  delay: 5

- name: Require bucket job success
  ansible.builtin.assert:
    that:
      - bucket_job.resources[0].status.succeeded | default(0) | int == 1
    fail_msg: "MinIO bucket job failed"

The until: waits for the job to reach either outcome — succeeded, or failed past its backoff limit. Then a separate assert decides the verdict. Folding both into one condition would make a genuine failure look like a timeout, and “timed out after three minutes” sends you looking at the wrong thing. This pattern — wait for terminal state, then assert which one — repeats throughout the lab.

By hand

kubectl -n mysql apply -f k8s/mysql/minio.yaml
kubectl -n mysql rollout status sts/minio --timeout=180s

kubectl -n mysql run mc --rm -i --restart=Never \
  --image=quay.io/minio/mc:RELEASE.2024-11-21T17-21-54Z --command -- sh -c '
    mc alias set local http://minio:9000 minio minio12345
    mc mb --ignore-existing local/mysql-lab-backups
    mc mb --ignore-existing local/mysql-lab-binlogs
    mc ls local/'

The MySQL user secret

apiVersion: v1
kind: Secret
metadata:
  name: {{ mysql_secrets_name }}          # lab-secrets
  namespace: {{ mysql_namespace }}
type: Opaque
stringData:
  root: {{ mysql_root_password }}
  xtrabackup: {{ mysql_root_password }}
  monitor: {{ mysql_monitor_password }}
  operator: {{ mysql_root_password }}
  replication: {{ mysql_root_password }}
  orchestrator: {{ mysql_root_password }}
  heartbeat: {{ mysql_root_password }}

ansible/roles/mysql_cluster/templates/secrets.yaml.j2

Seven keys, and the set is not free-form — the operator creates exactly these accounts and looks for exactly these key names. Supplying the secret ourselves rather than letting the operator generate one is what makes the lab reproducible: mysql -uroot -pmysql works the same on every rebuild, and the scenarios can hard-code it instead of reading a generated value.

There is a shipped offline test that guards the key set (test_user_secret_only_lists_operator_known_users in tests/test_lab_config.py), because a typo here produces a cluster that starts and then fails somewhere downstream with a permission error.

What each account is for:

Keep lab-secrets in mind. There is a second secret, internal-lab, generated by the operator, and part 8 is largely a story about what happens when you forget it exists.

The cluster: PerconaServerMySQL, field by field

This is the centre of the lab. It is presented in pieces.

Metadata and update policy

apiVersion: ps.percona.com/v1
kind: PerconaServerMySQL
metadata:
  name: {{ mysql_cluster_name }}          # lab
  namespace: {{ mysql_namespace }}
  finalizers:
    - percona.com/delete-mysql-pods-in-order
spec:
  crVersion: "{{ ps_cr_version }}"
  secretsName: {{ mysql_secrets_name }}
  updateStrategy: SmartUpdate
  upgradeOptions:
    apply: disabled
    versionServiceEndpoint: https://check.percona.com

The finalizer makes deletion orderly: replicas stop before the source, rather than the whole StatefulSet being torn down at once. It matters in this lab because part 8 creates and deletes a second cluster inside a test, and a messy teardown there would leave stranded PVCs that the next scenario trips over.

upgradeOptions.apply: disabled is the important one. Left at its default, the operator asks Percona’s version service what it should be running and can upgrade the cluster underneath you. For a lab whose entire output is “here is what this version did”, a silent version change would invalidate every recorded result. The endpoint stays configured but nothing is applied.

The MySQL section

  mysql:
    clusterType: {{ mysql_cluster_type }}     # async
    autoRecovery: true
    size: {{ mysql_size }}                    # 2
    image: {{ mysql_image }}                  # percona/percona-server:8.4.10-10.1
    imagePullPolicy: IfNotPresent
    gracePeriod: 30
    podDisruptionBudget:
      maxUnavailable: 1
    resources:
      requests: { cpu: 200m, memory: 512Mi }
      limits:   { memory: 1536Mi }
    affinity:
      antiAffinityTopologyKey: "none"
    exposePrimary:
      enabled: true
      type: ClusterIP
    volumeSpec:
      persistentVolumeClaim:
        resources:
          requests:
            storage: 5Gi

clusterType: async — classic source/replica asynchronous replication, not group replication. Chosen because it is what most people actually run, because its failure modes are the ones worth practising, and because asynchronous replication is the mechanism part 8 uses to prove a backup is usable: a restored copy attaches to the live source with SOURCE_AUTO_POSITION=1 and catches up. Group replication would not let a freshly restored outsider join that way.

size: 2 — one source, one replica. Two is the minimum that makes replication real. It is also the number that makes the health gate in part 9 able to say “exactly one writable primary and exactly one applying replica”.

imagePullPolicy: IfNotPresent — with pinned tags there is nothing to re-pull, and on a laptop that turns a pod restart from a network operation into a local one.

antiAffinityTopologyKey: "none" — all three kind nodes are containers on one host, so spreading pods across them provides no real fault isolation. Leaving the default anti-affinity in place would simply make pods unschedulable on a small cluster. This is a lab-only waiver and one of several places where the config says out loud “this is not production”.

exposePrimary: { enabled: true, type: ClusterIP } creates a stable in-cluster name for whichever pod is currently the source. Part 8 does not use it — it deliberately points replication at lab-mysql-0.lab-mysql.…, the pod’s own stable DNS name, because the test needs to attach to a specific known server rather than to whatever the service currently resolves to.

5Gi per MySQL pod against a ~500 MB dataset leaves room for the binary logs, the relay logs and the doublewrite files that restores and PITR generate.

The unsafe flags, said out loud

  unsafeFlags:
    mysqlSize: true
    orchestratorSize: true
    proxySize: true

The operator refuses, by default, to create topologies that cannot survive the failures they claim to handle: fewer MySQL nodes than its minimum, a single Orchestrator (which cannot form a quorum and therefore cannot safely arbitrate a failover), a single proxy (a single point of failure in front of the database).

Every one of those objections is correct, and every one is accepted here deliberately, because the subject is backup verification on one laptop. The value of unsafeFlags as a design is that the waiver is explicit and greppable. It is in the manifest, not in a footnote.

The MySQL configuration block

    configuration: |
      [mysqld]
      gtid_mode=ON
      enforce_gtid_consistency=ON
      log_bin=ON
      binlog_format=ROW
      binlog_expire_logs_seconds=86400
      innodb_buffer_pool_size=256M
      max_connections=80
      skip_name_resolve

Line by line, because each one is load-bearing later:

• gtid_mode=ON and enforce_gtid_consistency=ON — global transaction identifiers. Without them there is no SOURCE_AUTO_POSITION=1 (part 8), no gtid_purged restored alongside the data, and no pitr.type: gtid (part 7). GTIDs are what let a restored server say precisely which transactions it already contains instead of guessing from a file name and an offset.
• log_bin=ON — binary logging. Required for replication, and required for point-in-time recovery, which is nothing more than “replay the binary log on top of a restored backup, and stop at the right moment”.
• binlog_format=ROW — row-based replication. Statement-based would make replay non-deterministic for anything involving NOW(), RAND() or auto-increment races. When the test is “did the replayed copy end up identical to the source”, non-determinism is not acceptable.
• binlog_expire_logs_seconds=86400 — local binary logs are kept one day. This is short on purpose. Durable retention is the job of the binlog server archiving into MinIO (part 7), and a short local expiry keeps the argument honest: the recovery window comes from the archive, not from whatever happens to still be on the pod’s disk.
• innodb_buffer_pool_size=256M — matched to the 512Mi request and 1536Mi limit. The default would be sized against the node’s memory, which on a kind worker is the laptop’s memory, and the pod would be OOM-killed.
• max_connections=80 — small, matching the memory budget.
• skip_name_resolve — no reverse DNS on connect. In Kubernetes, reverse lookups for pod IPs are slow or absent, and this removes a class of multi-second connection stalls.

The exporter sidecar

    sidecars:
      - name: exporter
        image: {{ mysqld_exporter_image }}       # prom/mysqld-exporter:v0.16.0
        args:
          - --mysqld.address=127.0.0.1:3306
          - --mysqld.username=monitor
          - --collect.info_schema.innodb_metrics
          - --collect.info_schema.tables
          - --collect.global_status
          - --collect.global_variables
          - --collect.slave_status
        env:
          - name: MYSQLD_EXPORTER_PASSWORD
            valueFrom:
              secretKeyRef: { name: "{{ mysql_secrets_name }}", key: monitor }
        ports:
          - { name: metrics, containerPort: 9104 }
        resources:
          requests: { cpu: 20m, memory: 32Mi }
          limits:   { memory: 64Mi }

A sidecar, not a separate Deployment, so the exporter reaches MySQL over 127.0.0.1 inside the pod’s network namespace. No service, no network hop, and the metrics for a pod die with that pod instead of reporting a stale “up” for a server that no longer exists.

--collect.slave_status is the one that matters for this lab: it produces mysql_slave_status_slave_io_running and mysql_slave_status_seconds_behind_master, which is how replication health becomes a graph rather than a SHOW REPLICA STATUS someone has to remember to run.

The password comes from a secretKeyRef, so it is never a literal in the manifest even though it is a literal in group_vars. That is the right habit to keep even in a lab where the password is monitor.

Proxy and Orchestrator

  proxy:
    haproxy:
      enabled: true
      size: 1
      image: {{ haproxy_image }}              # percona/haproxy:2.8.18-1
      expose: { type: ClusterIP }
    router:
      enabled: false
  orchestrator:
    enabled: {{ 'true' if orchestrator_enabled else 'false' }}
    size: 1
    image: {{ orchestrator_image }}           # percona/percona-orchestrator:3.2.6-22
    configuration: '{"InstancePollSeconds": 2, "RecoveryPeriodBlockSeconds": 15}'

HAProxy on, Router off. MySQL Router is for InnoDB Cluster / group replication; this is an async topology, so HAProxy is the appropriate proxy. It routes writes to whichever pod is currently the source, which is what makes the host-port endpoint on 127.0.0.1:3307 usable without knowing which pod is primary — and what sysbench connects to in part 3.

Orchestrator is enabled but is not the subject. It watches the topology and would perform a failover if the source died. It stays because a backup lab without it would be pretending that production topologies are simpler than they are — and because the health gate that every scenario runs needs to observe a topology that something is actively managing. The tuned InstancePollSeconds: 2 simply makes it notice things quickly on a small cluster.

The backup block

  backup:
    enabled: true
    image: {{ xtrabackup_image }}            # percona/percona-xtrabackup:8.4.0-6.1
    schedule:
      - name: hourly-full
        schedule: "{{ backup_schedule }}"    # 15 * * * *
        keep: {{ backup_keep }}              # 3
        storageName: {{ backup_storage_name }}
        type: full
    storages:
      minio:
        type: s3
        verifyTLS: false
        s3:
          bucket: {{ backup_bucket }}                     # mysql-lab-backups
          credentialsSecret: {{ minio_credentials_secret }}
          endpointUrl: http://minio.{{ mysql_namespace }}.svc.cluster.local:9000
          region: us-east-1
          prefix: {{ mysql_cluster_name }}                # lab

This block is the foundation of parts 4, 5, 6 and 8.

storages is a named map. Every backup request names one with storageName: minio, which is why adding a second destination later is a configuration change and not a rewrite.

type: s3 against MinIO — MinIO speaks the S3 API, so the operator needs no special support. verifyTLS: false and a plain http:// endpoint are lab-only: in-cluster traffic to a pod with no certificate. region: us-east-1 is required by the SDK and meaningless here.

prefix: lab deserves the emphasis it gets in part 8. Every object for this cluster lands under s3://mysql-lab-backups/lab/…. When a different cluster restores one of these backups, it must be told the prefix as well as the bucket — a restore given only the bucket searches the wrong path and fails with backup not found in storage, which reads exactly like a corrupt backup.

The schedule creates a full backup at fifteen minutes past every hour and keeps three. It exists so that the lab has scheduled backup history without anybody doing anything, which parts 5 and 6 then manipulate — and, importantly, both of those scenarios save and restore this exact schedule around themselves rather than assuming what it was.

PITR, rendered conditionally

{% if pitr_enabled %}
    pitr:
      enabled: true
      binlogServer:
        size: 1
        image: {{ binlog_server_image }}
        serverId: {{ binlog_server_id }}                  # 1001
        checkpointInterval: {{ binlog_checkpoint_interval }}   # 30s
        checkpointSize: {{ binlog_checkpoint_size }}           # 16M
        logLevel: info
        storage:
          s3:
            bucket: {{ binlog_bucket }}                   # mysql-lab-binlogs
            credentialsSecret: {{ minio_credentials_secret }}
            endpointUrl: http://minio.{{ mysql_namespace }}.svc.cluster.local:9000
            region: us-east-1
            prefix: {{ binlog_prefix }}                   # lab
        resources:
          requests: { cpu: 50m, memory: 128Mi }
          limits:   { memory: 256Mi }
{% endif %}

The {% if %} is there so pitr_enabled: false produces a cluster with no pitr key at all, rather than one with enabled: false. Two shipped offline tests check both renderings, because “the block is absent” and “the block is present and disabled” are different objects and the operator may treat them differently.

The mechanics — what a binlog server is, why serverId must not collide, and what checkpointInterval and checkpointSize really define — are part 7’s subject. For now: this deploys lab-binlog-server-0, which registers with the source as though it were an ordinary replica and streams the binary log into the second bucket.

NodePort and metrics services

apiVersion: v1
kind: Service
metadata:
  name: mysql-primary-nodeport
spec:
  type: NodePort
  ports:
    - { name: mysql, port: 3306, targetPort: 3306, nodePort: {{ kind_nodeport_mysql }} }
  selector:
    app.kubernetes.io/instance: {{ mysql_cluster_name }}
    app.kubernetes.io/component: proxy
    app.kubernetes.io/managed-by: percona-server-mysql-operator
---
apiVersion: v1
kind: Service
metadata:
  name: mysql-metrics
  labels: { app: mysql-metrics }
spec:
  clusterIP: None
  ports:
    - { name: metrics, port: 9104, targetPort: 9104 }
  selector:
    app.kubernetes.io/instance: {{ mysql_cluster_name }}
    app.kubernetes.io/component: database
    app.kubernetes.io/managed-by: percona-server-mysql-operator

ansible/roles/mysql_cluster/templates/nodeports.yaml.j2

These are ours, not the operator’s, which is why they select on the operator’s labels. Two differences between them are worth noticing:

• component: proxy for the NodePort — it targets HAProxy, so a client on 127.0.0.1:3307 reaches the current source without caring which pod that is.
• component: database and clusterIP: None for metrics — a headless service. Prometheus must scrape every MySQL pod individually; a normal ClusterIP would load-balance across them and you would get one pod’s metrics at random per scrape, with the instance label lying about it.

A shipped test (test_nodeport_selectors_match_operator_labels) pins these selectors, because a label drift here produces a service with no endpoints and a connection refused that looks like a database problem.

Waiting for “ready” to mean ready

- name: Wait for MySQL pods to be Running
  kubernetes.core.k8s_info:
    kind: Pod
    label_selectors:
      - app.kubernetes.io/instance={{ mysql_cluster_name }}
      - app.kubernetes.io/component=database
  register: mysql_pods
  until:
    - mysql_pods.resources | length >= mysql_size | int
    - mysql_pods.resources | rejectattr('status.phase', 'equalto', 'Running') | list | length == 0
  retries: 90
  delay: 10

- name: Wait for PerconaServerMySQL to become ready
  kubernetes.core.k8s_info:
    api_version: ps.percona.com/v1
    kind: PerconaServerMySQL
    name: "{{ mysql_cluster_name }}"
  register: ps
  until:
    - ps.resources | length == 1
    - (ps.resources[0].status.state | default('')) in ['ready', 'Ready']
      or (ps.resources[0].status.mysql.state | default('')) in ['ready', 'Ready']
  retries: 90
  delay: 10

Two waits, fifteen minutes of patience each, and again not redundant. Pods Running is a Kubernetes fact. status.state: ready on the custom resource is the operator’s judgement that replication is configured and the topology is what was asked for. The second is the one that matters, and the first exists so that a stuck image pull fails while complaining about pods rather than about an opaque CR status.

The | default('') on every status lookup is defensive for a real reason: a freshly created CR has no status at all for the first few seconds, and without the default the whole until: expression raises an undefined-variable error instead of simply evaluating false and retrying.

The dual state / mysql.state check tolerates the status shape differing between operator versions.

Doing the whole thing by hand

export KUBECONFIG="$PWD/.kube/config"

# 1. operator
kubectl create namespace mysql
helm repo add percona https://percona.github.io/percona-helm-charts/ && helm repo update
helm install ps-operator percona/ps-operator --version 1.2.0 -n mysql --wait --timeout 5m

# 2. object storage
kubectl -n mysql apply -f k8s/mysql/minio.yaml
kubectl -n mysql rollout status sts/minio --timeout=180s
kubectl -n mysql run mc --rm -i --restart=Never \
  --image=quay.io/minio/mc:RELEASE.2024-11-21T17-21-54Z --command -- sh -c '
    mc alias set local http://minio:9000 minio minio12345
    mc mb --ignore-existing local/mysql-lab-backups
    mc mb --ignore-existing local/mysql-lab-binlogs'

# 3. credentials, cluster, services
kubectl -n mysql apply -f k8s/mysql/secrets.yaml
kubectl -n mysql apply -f k8s/mysql/cluster.yaml
kubectl -n mysql apply -f k8s/mysql/nodeports.yaml

# 4. wait for the operator's own verdict
kubectl -n mysql wait --for=jsonpath='{.status.state}'=ready ps/lab --timeout=15m
kubectl -n mysql get ps,pods

Because the templates are rendered to k8s/ on deploy, those file paths are real after one ./scripts/lab.sh deploy. To go entirely from scratch, render them yourself by substituting the group_vars values shown throughout this post.

A healthy result looks like this, from the recorded validation run’s environment snapshot:

"source_cluster_status": {
  "binlogServer": { "ready": 1, "size": 1, "state": "ready" },
  "haproxy":      { "ready": 1, "size": 1, "state": "ready", "version": "2.8.18" },
  "mysql":        { "ready": 2, "size": 2, "state": "ready", "version": "8.4.10-10" },
  "orchestrator": { "ready": 1, "size": 1, "state": "ready" },
  "host": "lab-haproxy.mysql",
  "state": "ready"
}

And the pods, with their sidecars:

lab-mysql-0          mysql, exporter, pt-heartbeat, xtrabackup
lab-mysql-1          mysql, exporter, pt-heartbeat, xtrabackup
lab-haproxy-0        haproxy, mysql-monit
lab-orc-0            orchestrator, mysql-monit
lab-binlog-server-0  binlog-server
minio-0              minio
ps-operator-…        manager

The xtrabackup container in each MySQL pod is not idle decoration — it is how the operator gets a backup process next to the data directory without mounting that volume anywhere else. Part 4 is about what happens inside it.

What can go wrong here

• unsafeFlags missing. The CR is rejected with a message about the minimum size for the cluster type. The flags are the acknowledgement, not a workaround.
• crVersion not matching the installed chart. The operator either refuses the object or ignores fields it does not recognise in that generation, and the latter is worse because it looks like it worked.
• The credentials secret has the wrong key names. The backup job fails with an S3 authentication error. AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY, exactly.
• MinIO created after the cluster. The cluster comes up fine and every backup fails until the bucket exists. The role order in site.yml prevents this.
• Cluster stuck initializing. Nearly always memory. kubectl -n mysql describe pod lab-mysql-0 showing OOMKilled means Docker Desktop needs a larger allocation.

Next

There is a database, and it is empty. Part 3 loads half a gigabyte of sysbench data through HAProxy, then builds the verification playbook that decides whether the lab is in a state worth testing — the same check that later gates every one of the twelve scenarios.