ClickHouse – a replicated cluster on your laptop with kind and Ansible

Most ClickHouse tutorials either give you a single server in a Docker container or assume you already have a cloud Kubernetes account. The single server teaches you nothing about replication, and the cloud account is a poor place to practise breaking things. What I wanted was something in between: a cluster with the same control-plane and data-plane shape as production, small enough to fit on a laptop, and cheap enough to destroy and rebuild several times in an afternoon.

This post walks through that lab. The result is a two-replica ClickHouse cluster, a three-node ClickHouse Keeper ensemble, about a million rows of NYC taxi data, and a full Prometheus / Alertmanager / Grafana stack that can page a Slack channel. Ansible is the installer; Kubernetes is the source of truth at runtime. Every number below was measured on the running lab, not estimated.

What the stack looks like

The whole thing runs inside kind – Kubernetes nodes as Docker containers – with one control-plane and two workers.

Local Kubernetes	kind, 1 control-plane + 2 workers
ClickHouse	1 shard × 2 replicas, clickhouse/clickhouse-server:25.8
Coordination	3-node ClickHouse Keeper
Operator	Altinity ClickHouse operator, Helm chart 0.27.3
Data	NYC taxi trips_0.gz (~1M rows) from ClickHouse’s public bucket
Monitoring	kube-prometheus-stack 88.5.2
Alerts	PrometheusRules → Alertmanager → Slack Incoming Webhook

Why kind rather than minikube or k3d? Because kind runs the nodes as Docker containers on a host that almost certainly already has Docker, and because extra port mappings are a first-class feature – which matters a lot when you want to reach ClickHouse, Grafana, Prometheus, and Alertmanager from the host shell. k3d would work equally well; only one Ansible role would change.

Why an operator? Because a ClickHouse cluster is not one Deployment. You need StatefulSets, headless services, user and profile configuration, Keeper connection strings, and a way to roll schema changes. The Altinity operator encodes all of that in two custom resources – ClickHouseInstallation and ClickHouseKeeperInstallation – and that is far more honest than a wall of hand-written YAML.

The whole lab in four commands

Everything below — the Ansible roles, the custom resources, the load SQL, the alert rules — lives in a public repository: github.com/ghkrzysztof-ksiazek/database-labs. This lab is the 20260909-clickhouse-setup directory, and every command in this post is run from inside it:

git clone https://github.com/ghkrzysztof-ksiazek/database-labs.git
cd database-labs/20260909-clickhouse-setup

./scripts/lab.sh bootstrap
./scripts/lab.sh deploy
./scripts/lab.sh load-data
./scripts/lab.sh verify

bootstrap makes sure Docker is running and installs kind, kubectl, and helm. deploy creates the kind cluster, installs the operator, applies the Keeper and ClickHouse custom resources, and installs the monitoring stack. load-data runs a Kubernetes Job that pulls the taxi file straight from S3. verify asserts that everything is where it should be.

Give Docker about 8 GB of RAM and four CPUs. If pods sit in Pending, kubectl describe pod will almost always point at insufficient CPU or ephemeral storage.

Ansible is the outer installer on purpose. A blog post that is a pile of copy-pasted kubectl and helm invocations is not reproducible – you inevitably skip a line. Roles make the ordering explicit:

prereqs	Docker is up; kind / kubectl / helm installed
kind_cluster	Create clickhouse-lab, write .kube/config
clickhouse_operator	Helm install the operator into the clickhouse namespace
clickhouse_cluster	Keeper CR, CHI CR, NodePort services
monitoring	kube-prometheus-stack, ServiceMonitors, PrometheusRules
load_dataset	Job that runs load.sql

The topology inside ClickHouse

The cluster is one shard with two replicas, which means every row of nyc.trips is stored twice. That is the smallest layout where you can kill a pod and still get an answer out of a SELECT. Keeper runs three nodes, because a single-node Keeper is fine for a toy but teaches exactly the wrong failure story – with three you keep a quorum when one dies.

SELECT cluster, shard_num, replica_num, host_name
FROM system.clusters
WHERE cluster = 'lab';

shard_num	replica_num	host_name
1	1	chi-lab-lab-0-0
1	2	chi-lab-lab-0-1

The interesting parts of the ClickHouseInstallation are short. The Prometheus endpoint, the Keeper address, and the cluster layout:

spec:
  configuration:
    settings:
      prometheus/endpoint: /metrics
      prometheus/port: "9363"
      prometheus/metrics: "true"
      prometheus/events: "true"
      prometheus/asynchronous_metrics: "true"
      prometheus/status_info: "true"
    zookeeper:
      nodes:
        - host: keeper-lab
          port: 2181
    clusters:
      - name: lab
        secret:
          auto: "True"
        layout:
          shardsCount: 1
          replicasCount: 2

Note secret: auto: "True". That generates the cluster secret the replicas use to authenticate to each other. Leave it out and every Distributed query dies with AUTHENTICATION_FAILED – a failure mode that is genuinely confusing the first time, because single-replica queries keep working fine.

The data: a million taxi trips

The dataset is the public trips_0.gz file from ClickHouse’s documentation bucket – tab-separated, gzipped, roughly a million trips. The lab loads one file so that ingest on a laptop takes minutes rather than hours; there are more files (trips_1.gz and so on) if you want to scale the story up.

The load SQL does four things: create the database ON CLUSTER 'lab', create a ReplicatedMergeTree table, create a Distributed table in front of it, and insert straight from S3.

CREATE TABLE nyc.trips
(
    trip_id UInt32,
    vendor_id String,
    pickup_date Date,
    pickup_datetime DateTime,
    ...
    fare_amount Float32,
    total_amount Float32,
    pickup_ntaname String,
    dropoff_ntaname String
)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{uuid}/{shard}', '{replica}')
PARTITION BY toYYYYMM(pickup_date)
ORDER BY (pickup_datetime, trip_id);

CREATE TABLE nyc.trips_dist
AS nyc.trips
ENGINE = Distributed('lab', 'nyc', 'trips', cityHash64(trip_id));

The macros {shard}, {replica}, and {uuid} are filled in by the operator inside each pod’s configuration, which is precisely the work you would otherwise be doing by hand.

The insert uses the s3 table function with NOSIGN, because the bucket is public and there are no AWS credentials in play. A few fare columns come back from the TSV inferred as strings, so they get coerced with coalesce and toFloat32OrZero on the way in.

With one shard, trips and trips_dist return the same counts. The Distributed table is still the right pattern for anything application-shaped, because the client should not be pinning itself to a particular replica.

What the verification actually showed

SELECT count() FROM nyc.trips	1,000,660 on both replicas
SELECT count() FROM nyc.trips_dist	1,000,660
Pickup years in the sample	2015 only, average fare 13.20
Prometheus up{job="clickhouse-http"}	1 on both replicas
Prometheus up{job="keeper-metrics"}	1 on all three Keepers
Alertmanager	Ready; accepts ClickHouseLabTestAlert

Identical counts on both replicas is the point of the whole exercise: the insert went to one replica and Keeper carried the replication log to the other.

A couple of queries worth running once the data is in:

SELECT
    toYear(pickup_date) AS year,
    round(avg(fare_amount), 2) AS avg_fare,
    count() AS trips
FROM nyc.trips_dist
GROUP BY year
ORDER BY year;

SELECT
    pickup_ntaname,
    count() AS trips
FROM nyc.trips_dist
WHERE pickup_ntaname != ''
GROUP BY pickup_ntaname
ORDER BY trips DESC
LIMIT 10;

SELECT database, table, is_leader, total_replicas, active_replicas
FROM system.replicas
WHERE table = 'trips';

kind publishes the NodePorts on localhost, so all of this works from the host shell without any port-forwarding:

curl 'http://127.0.0.1:8123/?user=default&password=clickhouse' \
  --data-binary 'SELECT count() FROM nyc.trips_dist'

Host port	NodePort	Service
8123	30081	ClickHouse HTTP
9000	30090	ClickHouse native
3000	30300	Grafana
9090	30091	Prometheus
9093	30093	Alertmanager

Monitoring and paging

ClickHouse 25.x exposes Prometheus metrics on port 9363; Keeper exposes 7000. ServiceMonitor objects tell the Prometheus Operator to scrape those services, and serviceMonitorSelectorNilUsesHelmValues: false in the Helm values is what makes monitors living in the clickhouse namespace visible to a Prometheus installed in monitoring.

The alert catalogue is deliberately small and boring:

Alert	Fires when	Severity
ClickHouseDown	up{job="clickhouse-http"} == 0 for 2m	critical
ClickHouseKeeperDown	Keeper scrape fails for 2m	critical
ClickHouseReplicaReadonly	ReadonlyReplica > 0 for 5m	critical
ClickHouseDelayedReplicas	Absolute replica delay > 30s	warning
ClickHouseLowDisk	Default disk below 15% free	warning
ClickHouseRejectedInserts	Rejected inserts in the last 5m	warning

Slack is wired through an Incoming Webhook set as global.slack_api_url in the Alertmanager Helm values, read from a gitignored secrets file. Without a webhook the default receiver is a no-op called noop and alerts simply stay visible inside the Alertmanager UI. With a webhook, anything matching severity =~ "warning|critical" goes to the slack receiver. The end-to-end test posts a synthetic alert to the Alertmanager API rather than breaking ClickHouse:

./scripts/lab.sh test-alert

Give it about a minute – group_wait is 30 seconds. To exercise a real ClickHouseDown, scale a replica’s StatefulSet to zero and wait two minutes.

The things that cost me time

Every lab has a handful of traps, and these are the ones worth writing down.

Prometheus job names come from Services, not ServiceMonitors. The Prometheus Operator named the scrape jobs clickhouse-http and keeper-metrics after the underlying Services, not after the ServiceMonitor objects. If your alert expressions reference the ServiceMonitor names they will simply never fire, and nothing will tell you why.

Pin Keeper to a version that understands what the operator writes. Keeper 24.8 crash-looped under operator 0.27.x because the operator injects a use_xid_64 setting that the older build does not know. Matching Keeper to ClickHouse 25.8 fixed it.

Ansible group_vars must sit next to the inventory. A directory called ansible/group_vars/ that is not adjacent to the inventory file (or the playbook) is silently ignored. Your variables are just… not there.

The kubernetes.core collection needs the virtualenv Python. It imports the kubernetes package, so pointing Ansible at Homebrew’s bare python3 gives you an import error that looks like a collection problem.

YAML receiver: null is a real null. Alertmanager wants a receiver name. Name the silent one noop and move on.

What this is not

It is worth being explicit. This is not a production security baseline – there are plain-text passwords, a 0.0.0.0/0 network for the default user, no TLS, and no network policies. It is not a high-availability story for the control plane, because kind is a single Docker host. It is not a substitute for a tuned on-prem cluster or ClickHouse Cloud; storage is local persistent volumes inside kind.

What it is is a faithful control-plane and data-plane shape that you can smash and rebuild in an afternoon. When you want to know what happens to a Distributed query while one replica is down, or whether your alert expression actually matches the labels Prometheus is producing, that is worth considerably more than a single-node container.

./scripts/lab.sh teardown

The full lab — Ansible roles, manifests, load SQL, alert rules, and the Grafana dashboard — is at github.com/ghkrzysztof-ksiazek/database-labs, along with the sibling MySQL lab.