📌 Saturday Deep Dive: Raft Consensus Algorithm. See the deep dive message above for the full 18-min read.
Quick Quiz: Days 46–49
Q1: [System Design] What algorithm does Distributed Consensus (Raft/Paxos) use to ensure only one leader at a time?
A1: Leader election via randomized election timeouts. A node starts an election when it hasn't heard from the leader within its timeout window. Majority quorum (⌊N/2⌋+1) vote required to win.
Q2: [Algorithms] What technique does "Find the Duplicate Number" use to detect a cycle in O(1) space?
A2: Floyd's Cycle Detection (tortoise and hare). Treat array values as "next pointers" — a duplicate creates a cycle. The meeting point in Phase 1 leads to the cycle entrance in Phase 2, which is the duplicate.
Q3: [Python Craft] What's the key difference between asyncio's gather and a Semaphore?
A3: gather runs all coroutines concurrently with no limit; a Semaphore caps the concurrency to N at a time. Use Semaphore when calling rate-limited APIs or managing resource pools (e.g., DB connections).
🔬 Saturday Deep Dive: Raft Consensus Algorithm (18 min read)
Raft 是 2014 年由 Diego Ongaro 和 John Ousterhout 在斯坦福设计的共识算法,目标是比 Paxos 更易理解。它被用于 etcd(Kubernetes 的大脑)、CockroachDB、TiKV 等生产系统中。
English:
Distributed consensus is the problem of getting multiple machines to agree on a single value — even when some machines crash or the network partitions. Raft was designed as a more understandable alternative to Paxos, and it underpins etcd (the backbone of Kubernetes), CockroachDB, TiKV, and more. If you're interviewing for Staff/Principal roles or distributed systems positions, this is non-negotiable.
Part 1: Theory / 理论基础 (5 min)
核心思想 / Core Idea
中文:
Raft 将共识问题分解为三个相对独立的子问题:
Leader 选举 — 在任何时刻,集群中最多有一个 Leader
日志复制 — Leader 接收客户端请求,将操作追加到日志,再复制到所有 Follower
安全性 — 一旦某个操作被提交(Committed),它永远不会被丢失
English:
Raft decomposes consensus into three sub-problems:
Leader Election — at most one leader at any time
Log Replication — leader receives requests, appends to its log, replicates to followers
Safety — once an entry is committed, it's never lost
关键概念 / Key Concepts
中文:
Term(任期):Raft 用单调递增的整数 term 来标记时间。每次选举开始时 term 加一。Term 是逻辑时钟,帮助节点识别过时的信息。
日志(Log):每个节点维护一个操作日志。日志条目格式:(term, index, command)。如果两个日志在相同 index 有相同 term,则它们之后的所有条目都相同(Log Matching Property)。
Quorum(多数派):一个 N 节点集群需要 ⌊N/2⌋ + 1 个节点同意才能提交。3 节点需要 2,5 节点需要 3。这保证了任意两个 Quorum 必然有交集。
English:
Term: A monotonically increasing integer. Every election starts a new term. Terms are logical clocks — nodes reject messages from stale terms.
Log: Each node maintains an ordered log. Entry format: (term, index, command). The Log Matching Property: if two logs agree at (term, index), they agree on all prior entries.
Quorum: ⌊N/2⌋ + 1 nodes must acknowledge before a log entry is committed. This ensures any two quorums share at least one member — the safety overlap.
节点状态机 / Node State Machine
timeout (no heartbeat) wins election
┌─────────────────────────┐ ┌───────────────────────┐
│ ▼ │ ▼
Follower ──────────────► Candidate ──────────────► Leader
▲ sees higher term │ discovers higher term │
└────────────────────┘ └───────────────────────┘
(also: any node that sees a higher term reverts to Follower)
中文:
Follower:被动响应,接受 Leader 的心跳和日志
Candidate:选举中的候选者,向其他节点发送投票请求
Leader:唯一处理客户端请求的节点,定期发送心跳防止选举
English:
Follower: Passive — responds to RPCs from Leader and Candidates
Candidate: Starts an election when its heartbeat timer expires
Leader: The sole entry point for client requests; sends periodic heartbeats (empty AppendEntries RPCs) to reset follower timers
Part 2: Step-by-Step Implementation / 一步一步实现 (8 min)
完整 Raft 实现 / Full Raft Implementation (Python)
import threading
import time
import random
import logging
from enum import Enum
from dataclasses import dataclass, field
from typing import Optional
logging.basicConfig(level=logging.INFO, format='%(asctime)s [%(name)s] %(message)s')
class NodeState(Enum):
FOLLOWER = "follower"
CANDIDATE = "candidate"
LEADER = "leader"
@dataclass
class LogEntry:
term: int # term when this entry was received by leader
index: int # 1-based position in the log
command: str # the state machine command (e.g., "SET x=5")class RaftNode:
"""
Simplified Raft implementation for educational purposes.
In production, RPCs would be network calls; here they're in-process method calls.
"""def __init__(self, node_id: int, peers: list['RaftNode']):
self.node_id = node_id
self.peers = peers # references to other nodes
self.logger = logging.getLogger(f"Node-{node_id}")
# --- Persistent state (must survive restarts in real systems) ---
self.current_term = 0# latest term seen
self.voted_for: Optional[int] = None# candidate voted for in current term
self.log: list[LogEntry] = [] # log entries; index 0 = first entry# --- Volatile state (all nodes) ---
self.commit_index = 0# highest log entry known to be committed
self.last_applied = 0# highest log entry applied to state machine# --- Volatile state (leaders only, reinitialized after election) ---
self.next_index: dict[int, int] = {} # for each peer, next log index to send
self.match_index: dict[int, int] = {} # for each peer, highest replicated index# --- Node behavior ---
self.state = NodeState.FOLLOWER
self.leader_id: Optional[int] = None
self.lock = threading.Lock()
self.election_timeout = self._new_election_timeout()
self.last_heartbeat = time.time()
# State machine (just a dict for demo)
self.state_machine: dict[str, str] = {}
def _new_election_timeout(self) -> float:
"""
Randomized timeout between 150-300ms prevents split votes.
All followers use different timeouts so one becomes candidate first.
"""return random.uniform(0.15, 0.30)
# =========================================================# PART A: Leader Election# =========================================================def start_election(self):
"""
Called when a follower's election timer fires.
Transition: Follower → Candidate
"""with self.lock:
self.state = NodeState.CANDIDATE
self.current_term += 1# increment term before starting election
self.voted_for = self.node_id # vote for self
term = self.current_term
last_log_index = len(self.log)
last_log_term = self.log[-1].term if self.log else0
self.logger.info(f"Starting election for term {term}")
votes_received = 1# self-vote
votes_needed = (len(self.peers) + 1) // 2 + 1# majority of total nodesfor peer in self.peers:
granted = peer.request_vote(
term=term,
candidate_id=self.node_id,
last_log_index=last_log_index,
last_log_term=last_log_term,
)
if granted:
votes_received += 1if votes_received >= votes_needed:
self._become_leader()
return# Didn't win; revert to follower and wait for next timeoutwith self.lock:
if self.state == NodeState.CANDIDATE:
self.state = NodeState.FOLLOWER
def request_vote(self, term: int, candidate_id: int,
last_log_index: int, last_log_term: int) -> bool:
"""
RPC handler: another node is asking for our vote.
Grant vote only if:
1. candidate's term >= our current term
2. we haven't voted for someone else this term
3. candidate's log is at least as up-to-date as ours (Election Restriction)
"""with self.lock:
# Step down if we see a higher termif term > self.current_term:
self.current_term = term
self.voted_for = None
self.state = NodeState.FOLLOWER
if term < self.current_term:
returnFalse# stale term; reject# Check if we already voted this termif self.voted_for isnotNoneand self.voted_for != candidate_id:
returnFalse# Election Restriction: candidate must be at least as up-to-date# "Up-to-date" = higher last term, OR same last term + longer log
my_last_log_term = self.log[-1].term if self.log else0
my_last_log_index = len(self.log)
candidate_is_up_to_date = (
last_log_term > my_last_log_term or
(last_log_term == my_last_log_term and last_log_index >= my_last_log_index)
)
if candidate_is_up_to_date:
self.voted_for = candidate_id
self.last_heartbeat = time.time() # reset our timer
self.logger.info(f"Voted for node {candidate_id} in term {term}")
returnTruereturnFalsedef _become_leader(self):
"""Transition to leader; initialize replication tracking."""with self.lock:
self.state = NodeState.LEADER
self.leader_id = self.node_id
# nextIndex = last log index + 1 for all peers (optimistic)
next_idx = len(self.log) + 1for peer in self.peers:
self.next_index[peer.node_id] = next_idx
self.match_index[peer.node_id] = 0
self.logger.info(f"Became LEADER for term {self.current_term}")
# =========================================================# PART B: Log Replication# =========================================================def client_request(self, command: str) -> bool:
"""
Entry point for client commands.
Only the leader should accept these; followers redirect.
"""with self.lock:
if self.state != NodeState.LEADER:
self.logger.warning(f"Not leader; redirecting to node {self.leader_id}")
returnFalse# Append to local log first
entry = LogEntry(
term=self.current_term,
index=len(self.log) + 1,
command=command,
)
self.log.append(entry)
self.logger.info(f"Appended to log: {entry}")
# Replicate to all followers; wait for quorumreturn self._replicate_log(entry.index)
def _replicate_log(self, target_index: int) -> bool:
"""
Send AppendEntries to all peers and wait for quorum acknowledgment.
Returns True when the entry is safely committed.
"""
acks = 1# leader counts itself
needed = (len(self.peers) + 1) // 2 + 1for peer in self.peers:
success = self._send_append_entries(peer)
if success:
acks += 1if acks >= needed:
# Quorum reached — commit!
self._advance_commit_index(target_index)
returnTruereturn acks >= needed # might still have quorumdef _send_append_entries(self, peer: 'RaftNode') -> bool:
"""
Send AppendEntries RPC to a single peer.
Includes a consistency check: prevLogIndex/prevLogTerm must match.
On failure, we decrement nextIndex and retry (back-tracking).
"""with self.lock:
next_idx = self.next_index.get(peer.node_id, 1)
prev_log_index = next_idx - 1
prev_log_term = (
self.log[prev_log_index - 1].term if prev_log_index > 0else0
)
entries = self.log[next_idx - 1:] # entries to send
term = self.current_term
leader_commit = self.commit_index
success, peer_term = peer.append_entries(
term=term,
leader_id=self.node_id,
prev_log_index=prev_log_index,
prev_log_term=prev_log_term,
entries=entries,
leader_commit=leader_commit,
)
with self.lock:
if peer_term > self.current_term:
# Discovered higher term; step down
self.current_term = peer_term
self.state = NodeState.FOLLOWER
self.voted_for = NonereturnFalseif success:
# Update our tracking of this peer's log
new_match = prev_log_index + len(entries)
self.match_index[peer.node_id] = new_match
self.next_index[peer.node_id] = new_match + 1else:
# Consistency check failed; back off by 1 and retry next time
self.next_index[peer.node_id] = max(1, self.next_index[peer.node_id] - 1)
return success
def append_entries(self, term: int, leader_id: int,
prev_log_index: int, prev_log_term: int,
entries: list[LogEntry], leader_commit: int):
"""
RPC handler: leader is sending log entries (or heartbeat if entries=[]).
Returns (success, my_current_term).
"""with self.lock:
if term > self.current_term:
self.current_term = term
self.voted_for = None
self.state = NodeState.FOLLOWER
if term < self.current_term:
returnFalse, self.current_term # stale leader; reject# Reset heartbeat timer — we heard from a valid leader
self.last_heartbeat = time.time()
self.leader_id = leader_id
self.state = NodeState.FOLLOWER
# Consistency check: does our log match up to prevLogIndex?if prev_log_index > 0:
if len(self.log) < prev_log_index:
returnFalse, self.current_term # we're missing entriesif self.log[prev_log_index - 1].term != prev_log_term:
# Conflict: delete from this index onward and retry
self.log = self.log[:prev_log_index - 1]
returnFalse, self.current_term
# Append new entries, overwriting any conflictsfor i, entry in enumerate(entries):
log_index = prev_log_index + i + 1if log_index <= len(self.log):
if self.log[log_index - 1].term != entry.term:
# Conflicting entry; truncate and append
self.log = self.log[:log_index - 1]
self.log.append(entry)
else:
self.log.append(entry)
# Advance commit index if leader says we canif leader_commit > self.commit_index:
self.commit_index = min(leader_commit, len(self.log))
self._apply_committed_entries()
returnTrue, self.current_term
def _advance_commit_index(self, new_index: int):
"""
Leader advances commit_index once a quorum has replicated the entry.
Critical safety rule: only commit entries from the CURRENT term.
(Prevents the 'Figure 8' bug from the Raft paper.)
"""with self.lock:
if (new_index > self.commit_index and
self.log[new_index - 1].term == self.current_term):
self.commit_index = new_index
self._apply_committed_entries()
def _apply_committed_entries(self):
"""Apply all committed but not-yet-applied entries to the state machine."""while self.last_applied < self.commit_index:
self.last_applied += 1
entry = self.log[self.last_applied - 1]
# Parse "SET key=value" commandsif entry.command.startswith("SET "):
parts = entry.command[4:].split("=", 1)
if len(parts) == 2:
self.state_machine[parts[0].strip()] = parts[1].strip()
self.logger.info(
f"Applied entry {self.last_applied}: {entry.command} "
f"→ state={self.state_machine}"
)
# =========================================================# DEMO: 3-node Raft cluster# =========================================================def demo_raft_cluster():
"""
Create a 3-node cluster, elect a leader, and replicate some commands.
This is in-process (no real network), so we manually trigger election.
"""# Create nodes (no peers yet)
nodes = [RaftNode(i, []) for i in range(3)]
# Wire up peers (each node knows about the other two)for i, node in enumerate(nodes):
node.peers = [n for n in nodes if n.node_id != i]
# Manually start election on node 0
print("\n=== Starting election on Node 0 ===")
nodes[0].start_election()
# Find who became leader
leader = next((n for n in nodes if n.state == NodeState.LEADER), None)
if leader:
print(f"\n=== Leader elected: Node {leader.node_id} ===")
# Submit commands
leader.client_request("SET name=Alice")
leader.client_request("SET age=30")
# Check state machine on all nodes
print("\n=== State machines after replication ===")
for node in nodes:
print(f" Node {node.node_id}: {node.state_machine}")
else:
print("No leader elected (split vote?)")
if __name__ == "__main__":
demo_raft_cluster()
关键数据流追踪 / Critical Flow Trace
中文:
让我们追踪一个写请求从客户端到提交的完整路径。
English:
Trace a single SET name=Alice request through the system:
Step 4 maps to _advance_commit_index() → _apply_committed_entries() in the Leader.
Step 5 maps to append_entries() handler where followers check if leader_commit > self.commit_index.
The diagram and code are consistent. ✓
Part 3: Edge Cases & Gotchas / 边界情况 (2 min)
中文:
1. Figure 8 问题(最容易踩的坑)
Raft 中有一个微妙规则:Leader 只能提交当前 term 的日志条目,不能直接提交旧 term 的条目。为什么?因为即使某个旧条目被多数节点复制了,也可能被新 Leader 用更新的条目覆盖。Raft 通过"间接提交"解决:当前 term 的新条目提交时,顺带提交之前所有未提交的旧条目。
English:
1. The Figure 8 Bug (Most Subtle)
A leader cannot directly commit log entries from previous terms, even if a majority has replicated them. An older entry could be overwritten by a new leader. The fix: commit a new entry from the current term — this implicitly commits all prior entries. This is _advance_commit_index's check: self.log[new_index-1].term == self.current_term.
2. 脑裂(Brain Split)/ Network Partition
If a network splits a 5-node cluster into [2] and [3]:
The partition with 3 nodes can elect a new leader and commit (quorum = 3)
The partition with 2 nodes cannot commit (no quorum)
When the partition heals, the 2-node side's leader steps down (sees higher term)
3. 选举超时必须随机化 / Randomized Election Timeout
If all nodes had the same timeout, they'd all start elections simultaneously — split vote forever. Randomization (150–300ms typical) ensures one node fires first with high probability.
4. nextIndex 回退 / nextIndex Back-tracking
When a follower rejects AppendEntries (consistency check fails), the leader decrements nextIndex by 1 and retries. In pathological cases (follower is very far behind), this is O(N) RPCs. Optimization: follower returns the first conflicting term to skip entire terms at once.
A 3-node cluster on a single datacenter: ~1-5ms per commit
Cross-datacenter: ~50-100ms (this is why geo-distributed systems use Multi-Paxos or alternatives like Spanner's TrueTime)
Throughput: etcd handles ~10K writes/sec; CockroachDB uses pipelining to batch
Part 5: Interview Simulation / 面试模拟 (3 min)
中文/English: 典型面试追问 + 简答 / Follow-up Questions
Q1: Why does Raft need randomized election timeouts? Why not just use a fixed timeout?
A: A fixed timeout causes all followers to start elections simultaneously when the leader dies, leading to perpetual split votes where no candidate gets majority. Randomization ensures one follower fires first (~150ms) and collects votes before others start. The probability of a split vote decreases exponentially with the range of randomization.
Q2: What happens if the leader crashes right after appending to its local log but before any follower replicates?
(Leader 在本地写入后、复制前崩溃怎么办?)
A: The entry was never committed (no quorum). When a new leader is elected, it will overwrite this uncommitted entry if needed. The client's request times out and must retry. This is safe — the entry never reached a committed state, so durability is preserved.
Q3: Why can't Raft elect a leader with a stale log? How is this prevented?
(为什么日志旧的节点不能成为 Leader?)
A: The Election Restriction in request_vote: a voter rejects a candidate whose log is "less up-to-date" (lower last-entry term, or same term but shorter log). Since a committed entry was replicated to a quorum, at least one node in any quorum has that entry. That node will reject a candidate with a stale log — the candidate can't get majority votes.
Q4: How does Raft handle a 5-node cluster where 2 nodes are permanently down?
A: The remaining 3 nodes form a quorum (3 > 5/2). The cluster continues normally. However, if a 3rd node goes down, the cluster becomes unavailable (only 2 nodes, can't reach majority of 3). This is the trade-off: CAP theorem — Raft chooses CP (Consistency + Partition tolerance, sacrifices Availability when below quorum).
Q5: What's the difference between Raft and Paxos? When would you use Multi-Raft?
(Raft 和 Paxos 的区别?Multi-Raft 是什么?)
A: Both solve consensus but Paxos is notoriously hard to implement correctly (it's underspecified; "Multi-Paxos" is informal). Raft adds strong leader semantics, explicit log structure, and clear membership change protocol — making it easier to implement and reason about.
Multi-Raft (used by CockroachDB, TiKV) runs thousands of Raft groups simultaneously — one per data shard. This gives horizontal scalability: instead of one Raft group bottlenecking all writes, each shard has independent consensus. The challenge is coordinating cross-shard transactions (solved with 2PC over Multi-Raft).