Today is Saturday — content delivered in the deep dive issue.
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Today is Saturday — content delivered in the deep dive issue.
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Today is Saturday — content delivered in the deep dive issue.
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Today is Saturday — content delivered in the deep dive issue.
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Today is Saturday — content delivered in the deep dive issue.
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
📊 Day 13/150 · NeetCode: 12/150 · SysDesign: 12/40 · Behavioral: 12/40 · Frontend: 12/50 · AI: 5/30
🔥 Keep the streak alive!
中文：
消息队列是现代分布式系统的血管。当两个服务需要通信，但你不希望它们紧密耦合在一起时，消息队列就登场了。从 WhatsApp 的消息投递，到 Uber 的行程分配，再到你下单后收到的确认邮件——几乎每个大型系统背后都有消息队列在默默支撑。
English:
Message queues are the circulatory system of modern distributed architecture. They allow services to communicate asynchronously — decoupling producers from consumers so each can scale, fail, and recover independently. If you've ever placed an Amazon order and received a confirmation email seconds later without the checkout page hanging, you've experienced event-driven architecture in action.
Why it matters in interviews: This topic appears in ~80% of senior-level system design interviews. "Design Twitter", "Design Uber", "Design a notification system" — all roads lead to message queues.
中文：
想象一个在线零售系统。用户下单时，系统需要：
如果全部同步完成，任何一步失败都会导致整个请求失败。支付服务宕机了？用户收到 500 错误。邮件服务慢了？用户等 10 秒。这就是同步耦合的代价。
English:
Message queues break all three couplings.
中文 → English glossary:
| 概念 | 英文 | 说明 |
|---|---|---|
| 生产者 | Producer | 发消息的服务 |
| 消费者 | Consumer | 收消息的服务 |
| 消息 | Message / Event | 传递的数据单元 |
| 队列 | Queue | 点对点：一条消息只被一个消费者消费 |
| 主题 | Topic | 发布/订阅：一条消息被多个消费者消费 |
| 消费者组 | Consumer Group | 多个消费者实例共享消费同一个 topic |
| 偏移量 | Offset (Kafka) | 消息在分区中的位置，消费者自己维护 |
| 确认 | Acknowledgment (ACK) | 消费者告诉队列"我已成功处理" |
| 死信队列 | Dead Letter Queue (DLQ) | 处理失败的消息的"最终归宿" |
Model 1: Point-to-Point (Queue)
Model 2: Publish-Subscribe (Topic)
中文： 面试中最常被问到的是 Kafka vs RabbitMQ。记住核心区别：
| 特性 | Apache Kafka | RabbitMQ / AWS SQS |
|---|---|---|
| 模型 | Log-based (追加写日志) | Queue-based (消费后删除) |
| 消息保留 | 时间/大小限制（可重放） | 消费后删除（默认） |
| 吞吐量 | 极高 (百万级/秒) | 高 (万-十万级/秒) |
| 顺序保证 | 分区内有序 | 队列内有序 |
| 适合场景 | 流处理、日志、事件溯源 | 任务队列、RPC、复杂路由 |
| 消费模型 | Pull (消费者拉取) | Push (队列推送) |
English:
The key insight: Kafka is a distributed log, not a traditional queue. Messages stay on disk until a retention policy removes them. Consumers maintain their own offsets, enabling:
中文： 我们设计一个类似于电商平台的通知系统，支持邮件、短信、推送通知。
English: Design a notification system that can send email, SMS, and push notifications after key events (order placed, payment failed, shipment update).
# events.py — Define event contracts clearly
from dataclasses import dataclass
from typing import Optional
import json
import time
@dataclass
class NotificationEvent:
event_id: str # Unique ID for deduplication
event_type: str # "order.placed", "payment.failed", "shipment.updated"
user_id: str # Kafka partition key — ensures order for same user
timestamp: float # Unix timestamp
payload: dict # Event-specific data
metadata: Optional[dict] = None # Tracing info, retry count, etc.
def to_json(self) -> bytes:
return json.dumps({
"event_id": self.event_id,
"event_type": self.event_type,
"user_id": self.user_id,
"timestamp": self.timestamp,
"payload": self.payload,
"metadata": self.metadata or {}
}).encode("utf-8")
@classmethod
def from_json(cls, data: bytes) -> "NotificationEvent":
d = json.loads(data)
return cls(**d)# producer.py
from kafka import KafkaProducer
from kafka.errors import KafkaError
import logging
logger = logging.getLogger(__name__)
class NotificationProducer:
def __init__(self, bootstrap_servers: list[str]):
self.producer = KafkaProducer(
bootstrap_servers=bootstrap_servers,
# Durability: wait for all in-sync replicas to acknowledge
acks="all",
# Enable idempotent producer — prevents duplicate messages on retry
enable_idempotence=True,
# Retry up to 5 times on transient failures
retries=5,
# Batch messages for throughput (linger up to 10ms)
linger_ms=10,
batch_size=16384, # 16KB batches
)
self.topic = "notification-events"
def publish(self, event: NotificationEvent) -> bool:
"""Publish event; use user_id as partition key for ordering."""
try:
future = self.producer.send(
self.topic,
key=event.user_id.encode("utf-8"), # Same user → same partition
value=event.to_json(),
)
# Block until broker confirms receipt (with timeout)
record_metadata = future.get(timeout=10)
logger.info(
f"Published {event.event_type} to partition "
f"{record_metadata.partition} offset {record_metadata.offset}"
)
return True
except KafkaError as e:
logger.error(f"Failed to publish event {event.event_id}: {e}")
# In production: send to a fallback DB for retry
return False
def close(self):
# Flush remaining buffered messages before shutdown
self.producer.flush()
self.producer.close()# consumer.py
from kafka import KafkaConsumer
from kafka.structs import OffsetAndMetadata, TopicPartition
import json, logging, time
logger = logging.getLogger(__name__)
class NotificationConsumer:
def __init__(self, bootstrap_servers: list[str], group_id: str):
self.consumer = KafkaConsumer(
"notification-events",
bootstrap_servers=bootstrap_servers,
group_id=group_id,
# Disable auto-commit — we commit AFTER successful processing
# This guarantees at-least-once delivery
enable_auto_commit=False,
# If no committed offset, start from the earliest message
auto_offset_reset="earliest",
# Deserialize from JSON bytes
value_deserializer=lambda b: json.loads(b.decode("utf-8")),
key_deserializer=lambda b: b.decode("utf-8") if b else None,
)
self.dlq_producer = DLQProducer() # Dead letter queue
def process(self):
"""Main consume loop with manual offset commit."""
for message in self.consumer:
event_data = message.value
event = NotificationEvent(**event_data)
success = self._handle_with_retry(event, max_retries=3)
if success:
# Only commit offset after successful processing
# This is the key to at-least-once semantics
tp = TopicPartition(message.topic, message.partition)
self.consumer.commit({tp: OffsetAndMetadata(message.offset + 1, None)})
else:
# Send to Dead Letter Queue for manual inspection/replay
self.dlq_producer.send(event)
# Still commit — we don't want to block the partition forever
tp = TopicPartition(message.topic, message.partition)
self.consumer.commit({tp: OffsetAndMetadata(message.offset + 1, None)})
def _handle_with_retry(self, event: NotificationEvent, max_retries: int) -> bool:
"""Route event to appropriate handler with exponential backoff retry."""
handler_map = {
"order.placed": self._send_order_confirmation,
"payment.failed": self._send_payment_alert,
"shipment.updated": self._send_shipment_update,
}
handler = handler_map.get(event.event_type)
if not handler:
logger.warning(f"No handler for event type: {event.event_type}")
return True # Don't retry unknown events
for attempt in range(max_retries):
try:
handler(event)
return True
except Exception as e:
wait = (2 ** attempt) # Exponential backoff: 1s, 2s, 4s
logger.warning(f"Attempt {attempt+1} failed for {event.event_id}: {e}. Retrying in {wait}s")
time.sleep(wait)
return False # All retries exhausted
def _send_order_confirmation(self, event: NotificationEvent):
# Call email service, SMS gateway, push notification service
user_id = event.user_id
order_id = event.payload["order_id"]
# ... actual notification logic
logger.info(f"Sent order confirmation to user {user_id} for order {order_id}")中文： 消息队列保证 at-least-once 投递，这意味着同一条消息可能被处理两次。如果你的系统没有幂等性保证，用户可能收到两封确认邮件，或被扣款两次。
English: At-least-once delivery means the same message can arrive twice (e.g., consumer crashed after processing but before committing offset). Always design consumers to be idempotent.
# idempotency.py — Using Redis to track processed events
import redis
class IdempotentNotificationHandler:
def __init__(self):
self.redis = redis.Redis(host="localhost", port=6379, db=0)
self.TTL = 86400 # 24 hours — enough to catch duplicates
def process_event(self, event: NotificationEvent) -> bool:
idempotency_key = f"processed:{event.event_id}"
# SET NX = "set if not exists" — atomic check-and-set
was_new = self.redis.set(
idempotency_key,
"1",
ex=self.TTL,
nx=True # Only set if key doesn't exist
)
if not was_new:
# This event was already processed — skip it
logger.info(f"Duplicate event {event.event_id} — skipping")
return True
# First time seeing this event — process it
return self._do_send_notification(event)中文 + English:
1. 消费者重平衡 / Consumer Rebalancing
当消费者加入或离开消费者组时，Kafka 会触发重平衡，暂停所有消费。设计时要确保处理过程的原子性。
When consumers join/leave a group, Kafka triggers rebalancing — all consumption pauses. Use cooperative rebalancing (partition.assignment.strategy=CooperativeStickyAssignor) to minimize disruption.
2. 消息顺序保证 / Ordering Guarantees
Kafka 只在分区内保证顺序。跨分区无顺序。设计时用有意义的 key（如 user_id）分区，确保同一用户的事件落在同一分区。
Kafka only guarantees order within a partition. Use a meaningful partition key (user_id, order_id) so related events land on the same partition.
3. 消费者落后 / Consumer Lag
如果消费者处理速度跟不上生产速度，lag 会越来越大。监控 consumer lag 是运维必备。
Monitor consumer_lag metric. If lag grows unbounded, add more consumer instances (up to the number of partitions) or optimize the handler.
4. 大消息问题 / Large Message Problem
Kafka 默认最大消息 1MB。发大文件？把内容存 S3，消息里只传引用。
Default max message size is 1MB. For large payloads: store in S3/GCS, put the reference URL in the message. Never put binary blobs in queues.
5. 时钟偏移 / Clock Skew
不要用消息里的 timestamp 做业务逻辑。生产者时钟可能偏移。
Don't use event timestamps for business logic ordering — clocks drift. Use Kafka's broker-assigned offset for true ordering.
中文： 真实系统里的消息队列：
English: How this looks in production systems:
LinkedIn (Kafka 的诞生地 / Kafka's birthplace)
Uber — 行程派单 / Ride Dispatch
Stripe — 支付事件 / Payment Events
Netflix — 实时数据管道 / Real-Time Pipeline
面试提示 / Interview Pattern:
每当你的设计里有一个服务需要触发多个下游服务，或者需要异步处理、削峰填谷、解耦，就应该加消息队列。
Whenever your design has one service that needs to trigger multiple downstream services, or you need async processing, buffering, or decoupling — reach for a message queue.
中文： 以下是面试中最常见的追问，以及简洁回答思路。
English: The 5 most common follow-up questions:
Q1: 如何保证消息不丢失？/ How do you guarantee no message loss?
Producer side: Setacks=all(wait for all in-sync replicas) +enable_idempotence=True.
Broker side: Set min.insync.replicas=2 to require at least 2 replicas to acknowledge.
Consumer side: Disable auto-commit; manually commit only after successful processing.
Result: At-least-once delivery. Accept that duplicates can happen, design consumers to be idempotent.
Q2: Kafka vs SQS，你会怎么选？/ Kafka vs SQS, how do you choose?
- Choose Kafka when: you need message replay, multiple independent consumers, very high throughput (>100K/s), stream processing, or event sourcing.
- Choose SQS when: you want fully managed simplicity, visibility timeout semantics, built-in DLQ, and don't need replay or complex routing. Great for task queues.
- Rule of thumb: SQS for task queues, Kafka for data pipelines and event streams.
Q3: 如何处理消费者崩溃？/ What happens when a consumer crashes?
With manual offset commit disabled (enable_auto_commit=False):
1. Consumer crashes after processing but before committing → same message redelivered to next consumer in group
2. Consumer crashes mid-batch → entire batch redelivered
This is why idempotency is non-negotiable. Use Redis or DB to track processed_event_ids.
Q4: 消息队列如何帮助削峰填谷？/ How does a queue help with traffic spikes?
Queue acts as a buffer. During Black Friday, order service produces 100K orders/minute. Payment service can only process 10K/minute. Without a queue, payment service crashes under load. With a queue, orders buffer up; payment service consumes at its own pace. Users see "order received" immediately, payment confirms asynchronously. The queue absorbs the spike.
Q5: 如果消费者处理一直失败怎么办？/ What if a message keeps failing?
Dead Letter Queue (DLQ) pattern:
1. Set max_delivery_attempts = 3 (or retry in consumer code)
2. After N failures, move message to DLQ topic
3. DLQ messages trigger alerts to on-call engineer
4. Engineer investigates, fixes the bug, then replays DLQ messages back to the original topic
Never silently drop messages. Always have a DLQ.
下周六继续深挖！Next Saturday: we'll go deeper on Kafka internals — partitions, replication, and the ISR mechanism. 🚀
References / 参考资料: