ClickHouse – a replicated cluster on your laptop with kind and Ansible

Most ClickHouse tutorials either give you a single server in a Docker container or assume you already have a cloud Kubernetes account. The single server teaches you nothing about replication, and the cloud account is a poor place to practise breaking things. What I wanted was something in between: a cluster with the same control-plane and data-plane shape as production, small enough to fit on a laptop, and cheap enough to destroy and rebuild several times in an afternoon.

This post walks through that lab. The result is a two-replica ClickHouse cluster, a three-node ClickHouse Keeper ensemble, about a million rows of NYC taxi data, and a full Prometheus / Alertmanager / Grafana stack that can page a Slack channel. Ansible is the installer; Kubernetes is the source of truth at runtime. Every number below was measured on the running lab, not estimated.

What the stack looks like

The whole thing runs inside kind – Kubernetes nodes as Docker containers – with one control-plane and two workers.

Local Kubernetes	kind, 1 control-plane + 2 workers
ClickHouse	1 shard × 2 replicas, clickhouse/clickhouse-server:25.8
Coordination	3-node ClickHouse Keeper
Operator	Altinity ClickHouse operator, Helm chart 0.27.3
Data	NYC taxi trips_0.gz (~1M rows) from ClickHouse’s public bucket
Monitoring	kube-prometheus-stack 88.5.2
Alerts	PrometheusRules → Alertmanager → Slack Incoming Webhook

Why kind rather than minikube or k3d? Because kind runs the nodes as Docker containers on a host that almost certainly already has Docker, and because extra port mappings are a first-class feature – which matters a lot when you want to reach ClickHouse, Grafana, Prometheus, and Alertmanager from the host shell. k3d would work equally well; only one Ansible role would change.

Why an operator? Because a ClickHouse cluster is not one Deployment. You need StatefulSets, headless services, user and profile configuration, Keeper connection strings, and a way to roll schema changes. The Altinity operator encodes all of that in two custom resources – ClickHouseInstallation and ClickHouseKeeperInstallation – and that is far more honest than a wall of hand-written YAML.

The whole lab in four commands

Everything below — the Ansible roles, the custom resources, the load SQL, the alert rules — lives in a public repository: github.com/ghkrzysztof-ksiazek/database-labs. This lab is the 20260909-clickhouse-setup directory, and every command in this post is run from inside it:

git clone https://github.com/ghkrzysztof-ksiazek/database-labs.git
cd database-labs/20260909-clickhouse-setup

./scripts/lab.sh bootstrap
./scripts/lab.sh deploy
./scripts/lab.sh load-data
./scripts/lab.sh verify

bootstrap makes sure Docker is running and installs kind, kubectl, and helm. deploy creates the kind cluster, installs the operator, applies the Keeper and ClickHouse custom resources, and installs the monitoring stack. load-data runs a Kubernetes Job that pulls the taxi file straight from S3. verify asserts that everything is where it should be.

Give Docker about 8 GB of RAM and four CPUs. If pods sit in Pending, kubectl describe pod will almost always point at insufficient CPU or ephemeral storage.

Ansible is the outer installer on purpose. A blog post that is a pile of copy-pasted kubectl and helm invocations is not reproducible – you inevitably skip a line. Roles make the ordering explicit:

prereqs	Docker is up; kind / kubectl / helm installed
kind_cluster	Create clickhouse-lab, write .kube/config
clickhouse_operator	Helm install the operator into the clickhouse namespace
clickhouse_cluster	Keeper CR, CHI CR, NodePort services
monitoring	kube-prometheus-stack, ServiceMonitors, PrometheusRules
load_dataset	Job that runs load.sql

The topology inside ClickHouse

The cluster is one shard with two replicas, which means every row of nyc.trips is stored twice. That is the smallest layout where you can kill a pod and still get an answer out of a SELECT. Keeper runs three nodes, because a single-node Keeper is fine for a toy but teaches exactly the wrong failure story – with three you keep a quorum when one dies.

SELECT cluster, shard_num, replica_num, host_name
FROM system.clusters
WHERE cluster = 'lab';

shard_num	replica_num	host_name
1	1	chi-lab-lab-0-0
1	2	chi-lab-lab-0-1

The interesting parts of the ClickHouseInstallation are short. The Prometheus endpoint, the Keeper address, and the cluster layout:

spec:
  configuration:
    settings:
      prometheus/endpoint: /metrics
      prometheus/port: "9363"
      prometheus/metrics: "true"
      prometheus/events: "true"
      prometheus/asynchronous_metrics: "true"
      prometheus/status_info: "true"
    zookeeper:
      nodes:
        - host: keeper-lab
          port: 2181
    clusters:
      - name: lab
        secret:
          auto: "True"
        layout:
          shardsCount: 1
          replicasCount: 2

Note secret: auto: "True". That generates the cluster secret the replicas use to authenticate to each other. Leave it out and every Distributed query dies with AUTHENTICATION_FAILED – a failure mode that is genuinely confusing the first time, because single-replica queries keep working fine.

The data: a million taxi trips

The dataset is the public trips_0.gz file from ClickHouse’s documentation bucket – tab-separated, gzipped, roughly a million trips. The lab loads one file so that ingest on a laptop takes minutes rather than hours; there are more files (trips_1.gz and so on) if you want to scale the story up.

The load SQL does four things: create the database ON CLUSTER 'lab', create a ReplicatedMergeTree table, create a Distributed table in front of it, and insert straight from S3.

CREATE TABLE nyc.trips
(
    trip_id UInt32,
    vendor_id String,
    pickup_date Date,
    pickup_datetime DateTime,
    ...
    fare_amount Float32,
    total_amount Float32,
    pickup_ntaname String,
    dropoff_ntaname String
)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{uuid}/{shard}', '{replica}')
PARTITION BY toYYYYMM(pickup_date)
ORDER BY (pickup_datetime, trip_id);

CREATE TABLE nyc.trips_dist
AS nyc.trips
ENGINE = Distributed('lab', 'nyc', 'trips', cityHash64(trip_id));

The macros {shard}, {replica}, and {uuid} are filled in by the operator inside each pod’s configuration, which is precisely the work you would otherwise be doing by hand.

The insert uses the s3 table function with NOSIGN, because the bucket is public and there are no AWS credentials in play. A few fare columns come back from the TSV inferred as strings, so they get coerced with coalesce and toFloat32OrZero on the way in.

With one shard, trips and trips_dist return the same counts. The Distributed table is still the right pattern for anything application-shaped, because the client should not be pinning itself to a particular replica.

What the verification actually showed

SELECT count() FROM nyc.trips	1,000,660 on both replicas
SELECT count() FROM nyc.trips_dist	1,000,660
Pickup years in the sample	2015 only, average fare 13.20
Prometheus up{job="clickhouse-http"}	1 on both replicas
Prometheus up{job="keeper-metrics"}	1 on all three Keepers
Alertmanager	Ready; accepts ClickHouseLabTestAlert

Identical counts on both replicas is the point of the whole exercise: the insert went to one replica and Keeper carried the replication log to the other.

A couple of queries worth running once the data is in:

SELECT
    toYear(pickup_date) AS year,
    round(avg(fare_amount), 2) AS avg_fare,
    count() AS trips
FROM nyc.trips_dist
GROUP BY year
ORDER BY year;

SELECT
    pickup_ntaname,
    count() AS trips
FROM nyc.trips_dist
WHERE pickup_ntaname != ''
GROUP BY pickup_ntaname
ORDER BY trips DESC
LIMIT 10;

SELECT database, table, is_leader, total_replicas, active_replicas
FROM system.replicas
WHERE table = 'trips';

kind publishes the NodePorts on localhost, so all of this works from the host shell without any port-forwarding:

curl 'http://127.0.0.1:8123/?user=default&password=clickhouse' \
  --data-binary 'SELECT count() FROM nyc.trips_dist'

Host port	NodePort	Service
8123	30081	ClickHouse HTTP
9000	30090	ClickHouse native
3000	30300	Grafana
9090	30091	Prometheus
9093	30093	Alertmanager

Opening Grafana, Prometheus, and Alertmanager

After ./scripts/lab.sh deploy, those host ports are already published on localhost. Open them in a browser. No kubectl port-forward is required.

UI	URL	Login
Grafana	http://127.0.0.1:3000	user admin, password admin
ClickHouse server dashboard	http://127.0.0.1:3000/d/clickhouse-server	the same Grafana login
ClickHouse Keeper dashboard	http://127.0.0.1:3000/d/clickhouse-keeper	the same Grafana login
Altinity operator dashboard	http://127.0.0.1:3000/d/clickhouse-operator	the same Grafana login
Prometheus	http://127.0.0.1:9090	none
Alertmanager	http://127.0.0.1:9093	none

Grafana is the only one with a password. Ansible sets grafana_admin_password: admin, and the Helm values pass that through as grafana.adminPassword. Prometheus and Alertmanager are installed without authentication, so those two URLs open straight into the UI.

Pages worth opening once the stack is up:

• Prometheus targets, http://127.0.0.1:9090/targets. The ClickHouse job clickhouse-http and the Keeper job keeper-metrics should be up.
• Prometheus alert rules, http://127.0.0.1:9090/alerts.
• Alertmanager, http://127.0.0.1:9093/#/alerts. This is where an alert sits when no Slack webhook is configured.

These are laptop defaults: plain text, bound to the host ports above. Do not reuse the Grafana password anywhere else.

Monitoring and paging

ClickHouse 25.x exposes Prometheus metrics on port 9363; Keeper exposes 7000. ServiceMonitor objects tell the Prometheus Operator to scrape those services, and serviceMonitorSelectorNilUsesHelmValues: false in the Helm values is what makes monitors living in the clickhouse namespace visible to a Prometheus installed in monitoring.

Three Grafana dashboards sit on top of those metrics, loaded by the Grafana sidecar from ConfigMaps. ClickHouse server is built from the server's own endpoint and covers query rate, failures and latency, inserts, merges and parts, replication delay and queue, the server's Keeper sessions, resources, and system.errors by code. ClickHouse Keeper shows which node leads, how many followers are in sync, request rate and latency, and the Raft log. The third is Altinity's own operator dashboard, which reads the operator's chi_* exporter and is included unchanged.

The alert catalogue is deliberately small and boring:

Alert	Fires when	Severity
ClickHouseDown	up{job="clickhouse-http"} == 0 for 2m	critical
ClickHouseKeeperDown	Keeper scrape fails for 2m	critical
ClickHouseReplicaReadonly	ReadonlyReplica > 0 for 5m	critical
ClickHouseDelayedReplicas	Absolute replica delay > 30s	warning
ClickHouseLowDisk	Default disk below 15% free	warning
ClickHouseRejectedInserts	Rejected inserts in the last 5m	warning

Alertmanager is installed either way. Until a webhook is configured, the default receiver is a no-op called noop, and alerts stay visible in the Alertmanager UI. With a webhook, anything matching severity =~ "warning|critical" goes to a receiver named slack.

Setting up Slack notifications

Alertmanager talks to Slack with an Incoming Webhook. That is a URL, not a bot token. Anyone who has the URL can post into the channel you selected when you created it, so treat it like a password and keep it out of git. The lab reads it from ansible/inventory/group_vars/secrets.yml, which is gitignored.

Do this from database-labs/20260909-clickhouse-setup, after deploy has finished.

1. Choose the channel. In Slack, create a channel or pick an existing one. #clickhouse-lab-alerts matches the lab default. You need permission to add an app to that channel.

2. Create the Slack app. Open https://api.slack.com/apps while logged into the workspace that should receive the alerts. Choose Create New App, then From scratch. Name it clickhouse-lab (any name is fine), pick that workspace, and create the app. On Enterprise Grid, an admin may have to approve the app before the next steps succeed.

3. Turn on Incoming Webhooks. In the app sidebar, open Incoming Webhooks and switch Activate Incoming Webhooks to On.

4. Bind a webhook to the channel. Click Add New Webhook to Workspace, select the channel from step 1, and click Allow. Slack shows a Webhook URL. Copy it. It always starts with https://hooks.slack.com/services/. That URL posts to the channel you just selected. Changing a channel name later in a config file does not move this webhook.

5. Put the URL in the lab secrets file.

cp ansible/inventory/group_vars/secrets.yml.example ansible/inventory/group_vars/secrets.yml

Edit secrets.yml. slack_channel should be the channel you selected in step 4, including the leading #:

slack_webhook_url: "https://hooks.slack.com/services/T…/B…/…"
slack_channel: "#clickhouse-lab-alerts"

6. Apply it to Alertmanager.

./scripts/lab.sh configure-slack

configure-slack loads secrets.yml, refuses a URL that does not start with https://hooks.slack.com/services/, and Helm-upgrades kube-prometheus-stack. The upgrade sets global.slack_api_url and adds the slack receiver. Resolved alerts are sent too (send_resolved: true). If this command fails the assert, the secrets file is missing or the URL was not pasted in full.

7. Send a synthetic alert. This does not stop ClickHouse. It posts ClickHouseLabTestAlert, severity warning, to the Alertmanager API:

./scripts/lab.sh test-alert

group_wait is 30 seconds, so the Slack message usually appears within about a minute. The title looks like [FIRING:1] ClickHouseLabTestAlert, and the body is Synthetic lab alert. The playbook also says whether slack_api_url is present in the Alertmanager config. If it says the webhook is not configured, step 6 did not apply secrets.yml.

If the message does not arrive:

• Open http://127.0.0.1:9093/#/alerts and confirm ClickHouseLabTestAlert is firing. If it is not there, Alertmanager never accepted the post.
• Confirm the webhook still posts to the channel chosen in the Slack UI. slack_channel is the channel field Alertmanager sends with the payload. An Incoming Webhook stays bound to the channel selected when the URL was created. To use a different channel, add a new webhook for that channel, update secrets.yml, and run configure-slack again.
• A Slack invalid_token or HTTP 404 means the URL was revoked. Create a new webhook and repeat steps 5 and 6.

To exercise a real ClickHouseDown, scale one replica’s StatefulSet to zero and wait two minutes. List the StatefulSets first; the name depends on the installation:

export KUBECONFIG="$PWD/.kube/config"
kubectl get sts -n clickhouse
kubectl -n clickhouse scale sts chi-lab-lab-0-0 --replicas=0

The alert fires after two minutes of a failed scrape. Scale the StatefulSet back to 1 when you are done. Alertmanager sends the resolution because send_resolved is on.

If the webhook URL leaks, delete it in the Slack app under Incoming Webhooks, create a new one, update secrets.yml, and run ./scripts/lab.sh configure-slack again. Do not commit secrets.yml.

The things that cost me time

Every lab has a handful of traps, and these are the ones worth writing down.

Prometheus job names come from Services, not ServiceMonitors. The Prometheus Operator named the scrape jobs clickhouse-http and keeper-metrics after the underlying Services, not after the ServiceMonitor objects. If your alert expressions reference the ServiceMonitor names they will simply never fire, and nothing will tell you why.

Pin Keeper to a version that understands what the operator writes. Keeper 24.8 crash-looped under operator 0.27.x because the operator injects a use_xid_64 setting that the older build does not know. Matching Keeper to ClickHouse 25.8 fixed it.

Ansible group_vars must sit next to the inventory. A directory called ansible/group_vars/ that is not adjacent to the inventory file (or the playbook) is silently ignored. Your variables are just… not there.

The kubernetes.core collection needs the virtualenv Python. It imports the kubernetes package, so pointing Ansible at Homebrew’s bare python3 gives you an import error that looks like a collection problem.

YAML receiver: null is a real null. Alertmanager wants a receiver name. Name the silent one noop and move on.

What this is not

It is worth being explicit. This is not a production security baseline – there are plain-text passwords, a 0.0.0.0/0 network for the default user, no TLS, and no network policies. It is not a high-availability story for the control plane, because kind is a single Docker host. It is not a substitute for a tuned on-prem cluster or ClickHouse Cloud; storage is local persistent volumes inside kind.

What it is is a faithful control-plane and data-plane shape that you can smash and rebuild in an afternoon. When you want to know what happens to a Distributed query while one replica is down, or whether your alert expression actually matches the labels Prometheus is producing, that is worth considerably more than a single-node container.

./scripts/lab.sh teardown

The full lab — Ansible roles, manifests, load SQL, alert rules, and the Grafana dashboards — is at github.com/ghkrzysztof-ksiazek/database-labs, along with the sibling MySQL lab.