Quite recently we have seen the release of a new Vitess version (v12.0.0) and this may lead to some questions – how to upgrade your Vitess installation when you use the Kubernetes and Vitess operator? In this blog post we would like to show the upgrade process of a simple Vitess cluster.
There are several elements that we have to keep in mind. First, Vitess operator itself has to be upgraded. This can be done quite easily if you use the GitHub repository with the operator. Single git pull should update the repository:
git branch remotes/origin/release-12.0 root@k8smaster:~/vitess# git pull remote: Enumerating objects: 3256, done. remote: Counting objects: 100% (3210/3210), done. remote: Compressing objects: 100% (1013/1013), done. remote: Total 3256 (delta 2206), reused 2995 (delta 2122), pack-reused 46 Receiving objects: 100% (3256/3256), 8.84 MiB | 1.94 MiB/s, done. Resolving deltas: 100% (2207/2207), completed with 243 local objects. From https://github.com/vitessio/vitess f085f12b1c..90852014be main -> origin/main * [new branch] frances/plotly-replace -> origin/frances/plotly-replace * [new branch] partitions -> origin/partitions 62b8e97d68..7bc70f1c6a release-12.0 -> origin/release-12.0 * [new branch] sarabee-tailwind-font-size -> origin/sarabee-tailwind-font-size Updating f085f12b1c..90852014be
From this point we have to upgrade the operator within Kubernetes. We will just reapply the operator.yaml file:
root@k8smaster:~/vitess# kubectl apply -f examples/operator/operator.yaml Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition customresourcedefinition.apiextensions.k8s.io/etcdlockservers.planetscale.com configured customresourcedefinition.apiextensions.k8s.io/vitessbackups.planetscale.com configured customresourcedefinition.apiextensions.k8s.io/vitessbackupstorages.planetscale.com configured customresourcedefinition.apiextensions.k8s.io/vitesscells.planetscale.com configured customresourcedefinition.apiextensions.k8s.io/vitessclusters.planetscale.com configured customresourcedefinition.apiextensions.k8s.io/vitesskeyspaces.planetscale.com configured customresourcedefinition.apiextensions.k8s.io/vitessshards.planetscale.com configured serviceaccount/vitess-operator unchanged role.rbac.authorization.k8s.io/vitess-operator unchanged rolebinding.rbac.authorization.k8s.io/vitess-operator unchanged Warning: scheduling.k8s.io/v1beta1 PriorityClass is deprecated in v1.14+, unavailable in v1.22+; use scheduling.k8s.io/v1 PriorityClass priorityclass.scheduling.k8s.io/vitess configured priorityclass.scheduling.k8s.io/vitess-operator-control-plane configured deployment.apps/vitess-operator configured
From then we need to start working on the upgrade of the pods. Let’s double-check which version do we have running.
VTTablet:
root@k8snode1:~# docker exec -it 663fdeb6d7af /bin/bash vitess@vitesstpcc-vttablet-zone1-3896337564-86d89914:/$ /vt/ bin/ config/ dist/ secrets/ socket/ vtdataroot/ web/ vitess@vitesstpcc-vttablet-zone1-3896337564-86d89914:/$ /vt/bin/ mysqlctl mysqlctld vtbackup vtctlclient vtctld vtctldclient vtgate vtorc vttablet vtworker vitess@vitesstpcc-vttablet-zone1-3896337564-86d89914:/$ /vt/bin/vttablet --version ERROR: logging before flag.Parse: E1104 22:11:45.304104 36 syslogger.go:149] can't connect to syslog Version: 12.0.0-SNAPSHOT (Git revision 912fb2b85a branch 'main') built on Tue Oct 12 04:17:46 UTC 2021 by vitess@buildkitsandbox using go1.17 linux/amd64
VTCtl:
root@k8snode2:~# docker exec -it 447ce05d13db /bin/bash vitess@vitesstpcc-zone1-vtctld-f38ee748-7b75487874-flfql:/$ /vt/bin/vtctl vtctlclient vtctld vtctldclient vitess@vitesstpcc-zone1-vtctld-f38ee748-7b75487874-flfql:/$ /vt/bin/vtctld --version ERROR: logging before flag.Parse: E1104 22:15:38.690147 27 syslogger.go:149] can't connect to syslog Version: 12.0.0-SNAPSHOT (Git revision 912fb2b85a branch 'main') built on Tue Oct 12 04:17:46 UTC 2021 by vitess@buildkitsandbox using go1.17 linux/amd64
VTGate:
root@k8snode3:~# docker exec -it 856c1bf5f846 /bin/bash vitess@vitesstpcc-zone1-vtgate-67d1e711-7b8556d549-b8jsw:/$ /vt/bin/vtgate --version ERROR: logging before flag.Parse: E1104 22:14:41.725924 52 syslogger.go:149] can't connect to syslog Version: 12.0.0-SNAPSHOT (Git revision 912fb2b85a branch 'main') built on Tue Oct 12 04:17:46 UTC 2021 by vitess@buildkitsandbox using go1.17 linux/amd64
The Vitess documentation says that the upgrade order should be as follows: vttablets then vtctld and finally vtgate.
The upgrade is quite simple. We can edit the VitessCluster definition and change the image for every element of the cluster:
apiVersion: planetscale.com/v2
kind: VitessCluster
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"planetscale.com/v2","kind":"VitessCluster","metadata":{"annotations":{},"name":"vitesstpcc","namespace":"default"},"spec":{"cells":[{"gateway":{"authentication":{"static":{"secret":{"key":"users.json","name":"example-cluster-config"}}},"replicas":1,"resources":{"limits":{"memory":"1Gi"},"requests":{"cpu":1,"memory":"1Gi"}}},"name":"zone1"}],"images":{"mysqld":{"mysql80Compatible":"vitess/lite:v12.0.0"},"mysqldExporter":"prom/mysqld-exporter:v0.11.0","vtbackup":"vitess/lite:mysql80","vtctld":"vitess/lite:mysql80","vtgate":"vitess/lite:mysql80","vttablet":"vitess/lite:mysql80"},"keyspaces":[{"name":"newsbtest","partitionings":[{"equal":{"parts":1,"shardTemplate":{"databaseInitScriptSecret":{"key":"init_db.sql","name":"example-cluster-config"},"replication":{"enforceSemiSync":false},"tabletPools":[{"cell":"zone1","dataVolumeClaimTemplate":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"100Gi"}}},"extraVolumeMounts":[{"mountPath":"/mnt","name":"backupvol"}],"extraVolumes":[{"name":"backupvol","persistentVolumeClaim":{"accessModes":["ReadWriteMany"],"claimName":"backupvol","resources":{"requests":{"storage":"100Gi"}},"volumeName":"backup"}}],"mysqld":{"configOverrides":"innodb_flush_log_at_trx_commit=2\ninnodb_buffer_pool_size=512M\n","resources":{"limits":{"cpu":1,"memory":"2Gi"},"requests":{"cpu":1,"memory":"2Gi"}}},"replicas":3,"type":"replica","vttablet":{"extraFlags":{"backup_engine_implementation":"xtrabackup","backup_storage_implementation":"file","db_charset":"utf8mb4","file_backup_storage_root":"/mnt/backup","restore_from_backup":"true","xtrabackup_root_path":"/usr/bin","xtrabackup_stream_mode":"xbstream","xtrabackup_stripes":"8","xtrabackup_user":"root"},"resources":{"limits":{"cpu":1,"memory":"2Gi"},"requests":{"cpu":1,"memory":"2Gi"}}}}]}}}],"turndownPolicy":"Immediate"}],"updateStrategy":{"type":"Immediate"},"vitessDashboard":{"cells":["zone1"],"extraFlags":{"security_policy":"read-only"},"replicas":1,"resources":{"limits":{"memory":"128Mi"},"requests":{"cpu":"100m","memory":"128Mi"}}}}}
creationTimestamp: "2021-11-16T21:39:24Z"
generation: 3
name: vitesstpcc
namespace: default
resourceVersion: "8348046"
uid: deb87a7f-b3d4-4638-a110-739f615a2b6a
spec:
cells:
- gateway:
authentication:
static:
secret:
key: users.json
name: example-cluster-config
replicas: 1
resources:
limits:
memory: 1Gi
requests:
cpu: 1
memory: 1Gi
name: zone1
images:
mysqld:
mysql80Compatible: vitess/lite:mysql80
mysqldExporter: prom/mysqld-exporter:v0.11.0
vtbackup: vitess/lite:mysql80
vtctld: vitess/lite:mysql80
vtgate: vitess/lite:mysql80
vttablet: vitess/lite:mysql80
.
.
.
What we need is to change the image to v12.0.0 and watch. As mentioned in the documentation, we will start with vttablet and use vitess/lite:v12.0.0-mysql80 image:
root@k8smaster:~/vitesstests# kubectl edit VitessCluster
vitesscluster.planetscale.com/vitesstpcc edited
In the meantime, our application is issuing queries to vtgate:
.
.
.
[ 538s ] thds: 2 tps: 1.06 qps: 24.47 (r/w/o: 11.70/10.64/2.13) lat (ms,95%): 3706.08 err/s 0.00 reconn/s: 0.00
[ 539s ] thds: 2 tps: 1.00 qps: 3.99 (r/w/o: 0.00/2.00/2.00) lat (ms,95%): 2880.27 err/s 0.00 reconn/s: 0.00
[ 540s ] thds: 2 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s 0.00 reconn/s: 0.00
[ 541s ] thds: 2 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s 0.00 reconn/s: 0.00
[ 542s ] thds: 2 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s 0.00 reconn/s: 0.00
[ 543s ] thds: 2 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s 0.00 reconn/s: 0.00
[ 544s ] thds: 2 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s 0.00 reconn/s: 0.00
[ 545s ] thds: 2 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s 0.00 reconn/s: 0.00
[ 546s ] thds: 2 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s 0.00 reconn/s: 0.00
[ 547s ] thds: 2 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s 0.00 reconn/s: 0.00
FATAL: mysql_drv_query() returned error 1105 (target: newsbtest.-.primary: no healthy tablet available for 'keyspace:"newsbtest" shard:"-" tablet_type:PRIMARY') for query 'UPDATE warehouse4
SET w_ytd = w_ytd + 3353
WHERE w_id = 9'
FATAL: `thread_run' function failed: ./tpcc_run.lua:258: SQL error, errno = 1105, state = 'HY000': target: newsbtest.-.primary: no healthy tablet available for 'keyspace:"newsbtest" shard:"-" tablet_type:PRIMARY'
FATAL: mysql_drv_query() returned error 1105 (target: newsbtest.-.primary: no healthy tablet available for 'keyspace:"newsbtest" shard:"-" tablet_type:PRIMARY') for query 'UPDATE warehouse3
SET w_ytd = w_ytd + 992
WHERE w_id = 8'
FATAL: `thread_run' function failed: ./tpcc_run.lua:258: SQL error, errno = 1105, state = 'HY000': target: newsbtest.-.primary: no healthy tablet available for 'keyspace:"newsbtest" shard:"-" tablet_type:PRIMARY'
root@vagrant:~/sysbench-tpcc# ./tpcc.lua --mysql-host=10.20.0.10 --mysql-port=15336 --mysql-user=sbtest --mysql-password=sbtest --mysql-db=newsbtest --time=30000 --threads=2 --report-interval=1 --tables=10 --scale=10 --db-driver=mysql run
This is because Vitess Operator performs a rolling restart and, to make sure downtime is minimized, it moves PRIMARY tablet from one vttablet to another:
root@k8smaster:~/vitesstests# vtctlclient listalltablets zone1-1817996704 newsbtest - primary 10.244.3.235:15000 10.244.3.235:3306 [] 2021-11-18T12:27:19Z zone1-3154448388 newsbtest - replica 10.244.1.6:15000 10.244.1.6:3306 [] <null> zone1-3896337564 newsbtest - replica 10.244.2.195:15000 10.244.2.195:3306 [] <null> root@k8smaster:~/vitesstests# vtctlclient listalltablets zone1-1817996704 newsbtest - replica 10.244.3.236:15000 10.244.3.236:3306 [] <null> zone1-3154448388 newsbtest - replica 10.244.1.7:15000 10.244.1.7:3306 [] <null> zone1-3896337564 newsbtest - primary 10.244.2.195:15000 10.244.2.195:3306 [] 2021-11-18T13:03:00Z root@k8smaster:~/vitesstests# vtctlclient listalltablets zone1-1817996704 newsbtest - replica 10.244.3.236:15000 10.244.3.236:3306 [] <null> zone1-3154448388 newsbtest - primary 10.244.1.7:15000 10.244.1.7:3306 [] 2021-11-18T13:06:18Z zone1-3896337564 newsbtest - replica 10.244.2.195:15000 10.244.2.195:3306 [] <null>
In our case the downtime caused by failovers was 40 seconds in total – two breaks, 20 seconds each. This is not ideal but as for an automated, fire-and-forget approach, it is quite good. Rolling upgrade is not the only method you can use. You can use, for example, OnDelete update strategy where you have more control on when and in what order pods will be upgraded. Alternatively, if you have to minimize the downtime, you can go for two cluster approach with two Vitess clusters replicating off each other. In that case the switch will have to be done on the application or loadbalancer level, giving you even more control on how it should be done. What you have to keep in mind is that Vitess runs MySQL replication under the hood therefore whatever strategy you can apply to a typical MySQL replication setup (one source + several replicas) most likely can be used (albeit with some small changes) for Vitess.
While upgrading, Vitess is doing the rolling restart of the pods:
root@k8smaster:~/vitesstests# kubectl get pods NAME READY STATUS RESTARTS AGE recycler-for-pv14 0/1 Completed 0 39h vitess-operator-7ccd86b994-ctj7c 1/1 Running 1 13d vitesstpcc-etcd-8f68363b-1 1/1 Running 0 39h vitesstpcc-etcd-8f68363b-2 1/1 Running 0 39h vitesstpcc-etcd-8f68363b-3 1/1 Running 0 39h vitesstpcc-vttablet-zone1-1817996704-58f13a5a 3/3 Running 0 3m43s vitesstpcc-vttablet-zone1-3154448388-e058f0f6 3/3 Running 0 2m23s vitesstpcc-vttablet-zone1-3896337564-86d89914 0/3 Init:0/2 0 3s vitesstpcc-zone1-vtctld-f38ee748-7b75487874-nktdf 1/1 Running 2 39h vitesstpcc-zone1-vtgate-67d1e711-7b8556d549-xgpcp 1/1 Running 2 39h
Once all is done, we can check the vttablet pods:
root@k8snode1:~# docker ps | grep zone1-1817996704 01fe04d4a6e7 e80442e91b90 "/bin/mysqld_exporte…" 15 minutes ago Up 15 minutes k8s_mysqld-exporter_vitesstpcc-vttablet-zone1-1817996704-58f13a5a_default_af92ae68-a2d3-4fcc-87b8-d3bc25a48440_0 7af9af334e66 37c0bee9615c "/vt/bin/mysqlctld -…" 15 minutes ago Up 15 minutes k8s_mysqld_vitesstpcc-vttablet-zone1-1817996704-58f13a5a_default_af92ae68-a2d3-4fcc-87b8-d3bc25a48440_0 a4a630142a45 6099999075af "/vt/bin/vttablet --…" 15 minutes ago Up 15 minutes k8s_vttablet_vitesstpcc-vttablet-zone1-1817996704-58f13a5a_default_af92ae68-a2d3-4fcc-87b8-d3bc25a48440_0 050fde47f88a k8s.gcr.io/pause:3.2 "/pause" 15 minutes ago Up 15 minutes k8s_POD_vitesstpcc-vttablet-zone1-1817996704-58f13a5a_default_af92ae68-a2d3-4fcc-87b8-d3bc25a48440_0 root@k8snode1:~# docker exec -it a4a630142a45 /bin/bash vitess@vitesstpcc-vttablet-zone1-1817996704-58f13a5a:/$ /vt/bin/vttablet --version ERROR: logging before flag.Parse: E1118 13:19:13.483469 21 syslogger.go:149] can't connect to syslog Version: 12.0.0 (Git revision de06fcaa92 branch 'heads/v12.0.0') built on Tue Oct 26 10:59:18 UTC 2021 by vitess@buildkitsandbox using go1.17 linux/amd64
As you can see, we have upgraded to the Version: 12.0.0.
The rest of the process involves performing exactly the same steps for every other element of the cluster: vtctld, vtgate and vtbackup pods. In the end we will end up with upgraded pods across the whole cluster. You may want to edit the cluster yaml file to reflect the changes made by hand on the live cluster.
As you can see, the upgrade process is not very complex but we would always recommend testing it before performing it on the production setup. There are always potential gotchas that may show up due to particular configuration of an environment.
