Architecture · ClickHouse
ClickHouse cluster design: shards, replicas and Keeper
ClickHouse is quick to start and easy to outgrow. The decisions that hurt later are made early: the shard count, where Keeper runs, the ordering key. I design or review the cluster with you, and I can build it with your team.
When this helps
It's usually one of these:
- We're moving from a single server to a replicated cluster.
- How many shards and replicas do we need, and where does Keeper run?
- Inserts were lost or duplicated when a replica went down.
- Queries keep hitting memory limits.
- The table design made sense for last year's queries, not this year's.
What the design covers
The written design explains each choice and the failure it's meant to handle.
Shards and replicas
How many of each, and how data is spread across shards. The design also says what a Distributed table does when one replica of a shard is gone.
Keeper quorum and placement
Where the Keeper ensemble runs, and what inserts do when it loses its majority. In my lab, every insert hung while selects and the proxy health check carried on as normal.
Inserts and consistency
Which insert_quorum and deduplication settings make an acknowledged insert safe on more than one replica, and what the application has to do after a timeout.
Memory and query limits
How ClickHouse counts memory against the server and per-query caps, and which queries need a different plan rather than a bigger limit.
Table engines, partitions and ordering keys
The ReplicatedMergeTree variant, the partition key and the ORDER BY, chosen for the queries you run today and the ones you expect next.
What you receive
- The cluster layout and why it looks the way it does.
- Failure scenarios for replicas and Keeper, and the behaviour to expect in each.
- Insert settings and client retry rules that keep acknowledged data safe.
- Table design notes for your main queries.
- If you want it built: the cluster in your environment, tested by breaking it, plus runbooks for replica and Keeper failures.
How it runs
- We agree which data and queries matter, and an effort limit.
- I review the schemas and query examples your team shares.
- I write up the design and go through it with whoever owns the cluster.
- If a build is agreed, I set it up with your team, and we kill replicas and Keeper nodes on purpose before handover.
Scope, deliverables, effort limit, fee and schedule are agreed in writing before work begins.
Fixed fee, agreed after a short scope review.
Not included
- Running the system after handover: managed operations, on-call cover or emergency response.
- Guarantees about query speed or cost.
- Application code changes.
Related reading
- ClickHouse HA on a Mac, part 1. A hands-on lab I ran on a laptop: the build and the healthy state.
- part 2, when nodes die. The same lab: killing a replica, the Keeper leader and the Keeper majority under load.
- part 3 on insert_quorum and production. The same lab: proving no acknowledged insert was lost, and what I'd change before production.
- part 4, memory limits. The same lab: a memory limit error on a small query, and why it happened.
- a replicated cluster with kind and Ansible. An earlier lab: a replicated cluster with alerting, all on one Docker host.
Questions
Self-hosted or managed ClickHouse?
Either. On a managed service the platform decides some of this for you, such as where Keeper runs, so the work is mostly about table design and inserts.
Can you review an existing cluster?
Yes. I start from the configuration and the query log, then write down what to keep and what to change.
Can you build it for us?
Yes. Once the design is settled, we scope the build separately. I work with the access you grant and through your change approval, and the cluster isn't handed over until the failure tests pass.
Do you migrate the data?
The migration is planned as part of the design. I carry it out only if it's in the agreed build scope; otherwise your team runs it from the written plan.
Planning or outgrowing a ClickHouse cluster?
Describe the data, the queries and the layout you have or plan. Table names and rough volumes are plenty.