Many designs quietly assume that some decision is made exactly once: which database node is the primary, which worker owns a shard, who holds a lock, whether a config change happened. On one machine that's trivial. Across several machines that can crash and lose messages, getting a group to agree on one answer, and never later disagree about it, is the consensus problem.
\nYou will rarely implement consensus yourself. You will use systems built on it: etcd, ZooKeeper, Consul, and the replication inside databases like CockroachDB and Spanner. Interviewers want to see that you know when you need it, what it costs, and how to use a coordination service safely.
","headings":[]},{"id":"why-consensus-is-needed-split-brain","title":"Why consensus is needed: split brain","html":"The primer's fail-over section describes active-passive pairs: the passive node watches the active one's heartbeat and takes over when it stops. Now suppose the network between them fails, but both nodes are fine and clients can still reach both. The passive node sees no heartbeat and promotes itself. The old active node never crashed, so it keeps accepting writes. You now have two primaries. This is split brain.
\nBoth sides now accept conflicting writes: the last concert seat is sold on each side. When the network heals, there is no clean way to merge the two histories, and some writes get thrown away or need manual repair.
\nThe root problem is that a node can't tell \"the other node is dead\" from \"I can't reach the other node\". Timeouts can't distinguish them. So a node should never decide alone that it is the leader. It needs agreement from a group, arranged so that two competing nodes can't both get it.
","headings":[{"id":"why-consensus-is-needed-split-brain","title":"Why consensus is needed: split brain","depth":3}]},{"id":"quorums-and-majorities","title":"Quorums and majorities","html":"The arrangement that makes this work is the majority quorum. With N voting nodes, a decision needs floor(N/2) + 1 of them. Any two majorities of the same group share at least one node, so two candidates can't both collect a majority for the same term, and a new leader always finds at least one node that knows what the old majority decided.
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n| Cluster size | Majority | Failures tolerated |
|---|---|---|
| 1 | 1 | 0 |
| 2 | 2 | 0 |
| 3 | 2 | 1 |
| 4 | 3 | 1 |
| 5 | 3 | 2 |
| 7 | 4 | 3 |
Two lessons come from the table. A cluster of 2f + 1 nodes survives f failures. And even sizes don't help: 4 nodes tolerate the same single failure as 3, but need one more vote per write. That's why consensus clusters are almost always 3 or 5 nodes. More nodes add fault tolerance but make every write wait for more acknowledgments.
\nIn a partition, the side holding a majority keeps working and the minority side stops accepting writes. That is the CP choice from the CAP theorem: the minority side gives up availability so the system never has two histories.
\nQuorums also set latency. A write commits when the leader hears back from enough followers to form a majority, so commit time is roughly the round trip to the follower that completes the majority. Assume a 5-node cluster spread across three regions as 2 + 2 + 1, with the leader in region A. Assume round trips of 1 ms to its peer in region A, 30 ms to both nodes in region B, and 70 ms to region C. The leader needs 2 follower acknowledgments (3 of 5 including itself): the region A peer at 1 ms and a region B node at 30 ms. Commits take about 30 ms plus disk writes, and losing all of region C doesn't change that.
","headings":[{"id":"quorums-and-majorities","title":"Quorums and majorities","depth":3}]},{"id":"raft-at-the-intuition-level","title":"Raft, at the intuition level","html":"Raft is the consensus algorithm behind etcd, Consul, CockroachDB and Kafka's KRaft mode, among others. Its authors designed it to be easier to understand than Paxos. It solves consensus by keeping a replicated log: every node holds the same sequence of commands, applies them in order to its own copy of a state machine (say, a key-value map), and so ends up in the same state.
\nTime is divided into numbered terms. Each term starts with an election and has at most one leader. Every message carries the sender's term. A node that sees a higher term than its own updates its term and, if it was leader or candidate, steps down to follower.
\nFollowers expect regular heartbeats from the leader. If a follower's election timeout passes without one, it becomes a candidate: it increments the term, votes for itself, and asks the others for votes. Each node votes for at most one candidate per term, and only for a candidate whose log is at least as up to date as its own. A candidate that collects a majority becomes leader. The timeouts are randomized (the Raft paper suggests something like 150 to 300 ms), so usually one node times out first and wins before others start competing.
\nThe \"up to date\" rule is what keeps committed data safe: a committed entry is on a majority, a winner needs votes from a majority, so some voter has the entry and refuses any candidate missing it.
\nClients send writes to the leader. The leader appends the command to its log and sends it to followers. Once a majority of nodes (counting the leader) have stored the entry, it is committed: it will never be lost or replaced. The leader tracks the highest committed position as its commit index, applies committed entries to its state machine, replies to the client, and includes the commit index in later messages so followers apply the entries too.
\nOne subtlety, in case an interviewer probes: a new leader only counts replicas this way for entries from its own term. Entries left over from earlier terms become committed indirectly, once an entry from the current term commits after them (section 5.4.2 of the Raft paper explains why).
\nEach append message also names the position and term of the entry just before the new ones. A follower whose log doesn't match there rejects the message, and the leader steps back until they agree, then overwrites the follower's conflicting entries. That is how lagging followers, and leftovers from a failed leader, get repaired.
\nAn old leader cut off in a minority partition can't commit anything. When it reconnects and sees a higher term, it steps down and its uncommitted entries are overwritten. Reads need care too, since a deposed leader answering from local state could return stale data. etcd's default reads confirm leadership with a majority first, and it offers cheaper \"serializable\" reads that may be stale.
\nPaxos, from Leslie Lamport, came first and underlies Google's Chubby and Spanner; Multi-Paxos gives the same guarantees as Raft, and ZooKeeper uses a related protocol called Zab. In an interview, \"a Raft or Paxos group\" is enough detail. The FLP result (Fischer, Lynch and Paterson, 1985) proves no deterministic algorithm can guarantee consensus terminates in a fully asynchronous network if even one node may crash, which is why real systems stay safe always but rely on timeouts to make progress.
","headings":[{"id":"raft-at-the-intuition-level","title":"Raft, at the intuition level","depth":3},{"id":"terms-and-leader-election","title":"Terms and leader election","depth":4},{"id":"log-replication-and-the-commit-index","title":"Log replication and the commit index","depth":4},{"id":"paxos-and-flp","title":"Paxos and FLP","depth":4}]},{"id":"leases-and-fencing-tokens","title":"Leases and fencing tokens","html":"A lease is a lock with an expiry. A node that holds a lease may act as leader (or owner of a resource) until the lease runs out, and must renew it to keep going. If the holder crashes, the lease expires and someone else can take over, without anyone needing to know whether the holder is dead or just unreachable.
\nLeases rely on time, and the holder can't fully control it. Suppose worker A holds a 30-second lease on shard 12 and sends a write to storage, but the write gets stuck: the host is overloaded, or the packet sits in a retry queue somewhere on the network. Forty seconds pass. A's lease expires, worker B takes over shard 12 and starts writing. Then A's delayed write finally arrives, and storage has no way to tell it's stale. Process pauses (garbage collection, a VM being migrated) cause the same problem from the other direction: the holder wakes up and acts on a lease it no longer has.
\nThe fix is a fencing token: a number that increases every time the lock is granted. The client includes the token with every write, and the storage service remembers the highest token it has seen and rejects anything lower.
\nFencing needs the resource to cooperate. A database can check a token in a conditional write (UPDATE ... SET fence_token = 118, ... WHERE id = ? AND fence_token <= 118). An external API that knows nothing about your tokens can't, and then the lock alone can't prevent overlapping writes. Coordination services provide suitable numbers: ZooKeeper's transaction IDs and znode versions, and etcd's revision numbers, both of which only increase.
ZooKeeper and etcd are small, strongly consistent stores replicated with consensus. Their job is to hold small amounts of critical metadata, typically kilobytes per key, not application data. A few features make them useful for coordination:
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n| Feature | ZooKeeper | etcd |
|---|---|---|
| Data model | Tree of znodes, like a file system | Flat key space with range queries over prefixes |
| Tied to client liveness | Ephemeral znodes vanish when the client's session ends | Keys attached to a lease vanish when the lease expires |
| Change notification | Watches on znodes | Watch on keys or prefixes, from a given revision |
| Atomic updates | Conditional writes on znode version, multi-op | Transactions: compare, then put or delete |
| Ordering | Sequential znodes get increasing suffixes | Every change gets a global revision number |
These primitives combine into standard recipes:
\n/election. The lowest number is leader. Every other candidate watches only the node just below its own, so a leader's death wakes one successor instead of the whole group. etcd's client library offers an election API built on leases and revisions.Real systems use these heavily. Kubernetes stores all cluster state in etcd. HBase and HDFS high-availability rely on ZooKeeper for master election. Kafka relied on ZooKeeper for cluster metadata for years, then replaced it with its own Raft-based controller (KRaft), and Kafka 4.0 dropped ZooKeeper support entirely.
\nTwo cautions. Sessions and leases take seconds to expire, so failover isn't instant and an old leader may not learn right away that it lost; fencing covers that gap. And ZooKeeper serves reads from whichever server a client is connected to, which may lag; call sync first when you need the latest value.
Consensus costs extra round trips on every write, a cluster you must operate, and unavailability on the minority side of a partition. Many problems don't need it:
\nWhen you do need it, use etcd, ZooKeeper, Consul or your database's built-in replication. Say in the interview which component provides consensus and what happens to your system when that component's minority side is cut off.
","headings":[{"id":"when-you-dont-need-consensus","title":"When you don't need consensus","depth":3}]},{"id":"check-yourself","title":"Check yourself","html":"A 4-node cluster needs 3 votes for a majority, so it tolerates only 1 failure, the same as 3 nodes, while every write waits for one more acknowledgment. Use 3 nodes, or 5 if you need to survive 2 failures. Placing the nodes in different availability zones matters more than adding a fourth node.
\nThe old leader can't reach a majority, so its new writes never commit and clients get errors or timeouts. In the group of 3, a follower times out, wins an election in a higher term, and keeps serving writes. When the network heals, the old leader sees the higher term, steps down, and its uncommitted entries are overwritten by the new leader's log.
\nAn instance probably paused or ran long past its lease, another instance took the lock, and the first one kept writing after waking up. Have the lock service issue a fencing token (an etcd revision or ZooKeeper version), store the highest token per billing run in Postgres, and make each write conditional on its token being the highest seen. Making invoice creation idempotent, for example with a unique key per customer and billing period, adds a second layer of protection.
\nIf every candidate watched the leader's node, its deletion would wake all of them at once, and they would all hit ZooKeeper to re-check the list. With N candidates that is N requests for one change, known as the herd effect. Watching only the predecessor means each change wakes exactly one candidate, which checks whether it is now the lowest.
\nProbably not. Every like would pay a consensus round trip, and during a partition the minority side couldn't accept likes at all. Likes are additions that can be merged later, so sharded counters or per-replica counts summed on read give an accurate total eventually, with much lower cost. Exactness to the second rarely matters for a like count.
\n