Rate Limiter: The Invisible Bouncer Protecting Every API You've Ever Used
Every time you scroll Twitter during the Super Bowl, smash the like button on Instagram, or fire off API calls to Stripe, there's an invisible system deciding whether you're allowed to continue. That system is a rate limiter — and designing one that's fast, fair, and fault-tolerant is one of the most practical system design challenges you'll face.
Why Rate Limiting Matters
Rate limiting isn't just about stopping malicious actors. It's about protecting shared resources from legitimate overuse. A single user with a buggy retry loop can take down your service for everyone. A sudden mobile app reconnection storm can saturate your database. Rate limiting is traffic control for your infrastructure.
The core tension: you want to block abuse without punishing legitimate bursts. A user sending 50 cached requests after reconnecting? Probably fine. That same user sustaining 50 requests per second for an hour? That's a problem.
The Four Algorithms You Need to Know
| Algorithm | Burst Handling | Memory | Precision | Best For |
|---|---|---|---|---|
| Token Bucket | Excellent | Low | Good | APIs with burst tolerance |
| Leaky Bucket | Smooths output | Low | Good | Steady-rate processing |
| Fixed Window | Poor (boundary exploit) | Very Low | Low | Simple use cases |
| Sliding Window | Good | Higher | High | Fair, accurate limiting |
Token Bucket is the most popular choice. Tokens accumulate at a steady rate (e.g., 10/second) up to a maximum bucket size. Each request costs a token. Bursts are naturally supported — if you've been idle, your bucket is full.
Fixed Window is simple but has a well-known exploit: send 1000 requests at 11:59:59 and another 1000 at 12:00:01 — technically both are within their respective windows, but you just doubled your effective rate.
Sliding Window fixes this by tracking requests across a rolling time period, but requires more memory per user — you need timestamps, not just counters.
Leaky Bucket processes requests at a fixed rate regardless of input burst, acting like a queue with constant drain. Great for smoothing traffic to downstream services.
The Interview Framework
When you get this in an interview, start here:
- Clarify requirements — Per-user? Per-IP? Per-endpoint? What's the scale (1K QPS or 100K)?
- Non-functional requirements — Latency budget (adding 50ms per request is unacceptable), availability requirements, fail-open vs fail-closed policy
- Back-of-envelope math — 10M users × 8 bytes per counter = 80MB. Fits easily in Redis.
- High-level architecture — API Gateway → Rate Limit Middleware → Redis counters → Your service
- Deep dive — Distributed coordination, algorithm choice, failure modes
Key insight: The rate limiter sits on the hot path of every request. Anything you add here multiplies across your entire QPS.
The Distributed Problem
Here's where "just use Redis" stops being a complete answer. With multiple API servers, you need coordination:
Centralized (Redis): All servers check/increment the same counter. Atomic operations (INCR + EXPIRE, or Lua scripts) prevent race conditions. Trade-off: every request adds a network hop to Redis.
Local + Periodic Sync: Each server counts locally and syncs to a central store periodically. Lower latency, but you lose precision — a user could exceed limits by hitting different servers before sync occurs.
Batched Updates: Buffer several requests worth of counter increments and flush them in batches. Reduces Redis load but introduces a small accuracy window.
The thundering herd is real: 100 servers simultaneously incrementing the same Redis key during a traffic spike. Solutions include request coalescing, Redis clustering, and client-side jitter.
Fail-Open vs Fail-Closed
When your rate limiting infrastructure goes down:
- Fail-open: Allow all requests through. Preserves availability but removes protection. Most companies choose this — a brief period without rate limiting is better than dropping all traffic.
- Fail-closed: Reject all requests. Safer against attacks but causes an outage for legitimate users.
The pragmatic approach: fail-open with local fallback counters and circuit breakers. If Redis is unreachable, apply conservative in-memory limits until connectivity recovers.
How the Big Players Built It
Stripe uses Redis with sliding window counters and Lua scripts for atomicity. They batch operations and handle billions of API requests with sub-millisecond latency overhead. Their multi-tier approach applies different limits at the API key, IP, and account levels simultaneously.
Cloudflare does rate limiting at the edge network level — blocking bad traffic before it reaches your origin servers. It's like having bouncers at every entrance to the city, not just your building.
GitHub evolved from simple counters to sophisticated abuse detection. They use a sharded, replicated rate limiter in Redis that differentiates between authenticated and unauthenticated requests, with different limits per endpoint category.
Key Takeaways
- Algorithm choice depends on your burst tolerance — Token bucket for most APIs, sliding window when fairness is critical
- Distributed coordination is the hard part — Not the algorithm itself
- Monitor your false positive rate — If legitimate users are getting rate limited, your thresholds are wrong
- Design for failure — Your rate limiter will go down; decide what happens when it does
- Layer your defenses — Edge-level, application-level, and database-level limits each catch different problems
- Don't add latency to the hot path — Async updates, batching, and local caches are your friends
Rate Limiter：保护你用过的每个 API 的隐形保安
每当你在超级碗期间刷 Twitter，在 Instagram 上疯狂点赞，或者向 Stripe 发起 API 调用时，都有一个隐形系统在决定是否允许你继续操作。这个系统就是 rate limiter——设计一个快速、公平且容错的 rate limiter 是你会遇到的最实用的系统设计挑战之一。
为什么 Rate Limiting 很重要
Rate limiting 不仅仅是为了阻止恶意行为者，它是为了保护共享资源免受合法的过度使用。一个有 bug 的重试循环就能让你的服务对所有人都不可用。移动应用的突然重连风暴可能会让你的数据库饱和。Rate limiting 就是你基础设施的交通管制。
核心矛盾在于：你想要阻止滥用而不惩罚合法的突发流量。用户重连后发送 50 个缓存请求？可能没问题。同一个用户持续一小时每秒发送 50 个请求？那就有问题了。
你需要了解的四种算法
| 算法 | 突发处理 | 内存占用 | 精度 | 最适合场景 |
|---|---|---|---|---|
| Token Bucket | 优秀 | 低 | 好 | 能容忍突发的 API |
| Leaky Bucket | 平滑输出 | 低 | 好 | 稳定速率处理 |
| Fixed Window | 差（边界利用） | 很低 | 低 | 简单用例 |
| Sliding Window | 好 | 较高 | 高 | 公平、准确的限制 |
Token Bucket 是最受欢迎的选择。Token 以稳定速率累积（比如每秒 10 个）直到达到最大桶容量。每个请求消耗一个 token。天然支持突发——如果你一直空闲，你的桶就是满的。
Fixed Window 简单但有个众所周知的漏洞：在 11:59:59 发送 1000 个请求，在 12:00:01 再发送 1000 个——技术上都在各自的窗口内，但你实际上把有效速率翻了一倍。
Sliding Window 通过跟踪滚动时间段内的请求来修复这个问题，但每个用户需要更多内存——你需要时间戳，而不仅仅是计数器。
Leaky Bucket 无论输入突发如何，都以固定速率处理请求，就像一个恒定排水的队列。非常适合平滑到下游服务的流量。
面试框架
当你在面试中遇到这个问题时，从这里开始：
- 明确需求 — 按用户？按 IP？按端点？规模是多少（1K QPS 还是 100K）？
- 非功能性需求 — 延迟预算（每个请求增加 50ms 是不可接受的）、可用性要求、fail-open vs fail-closed 策略
- 粗略计算 — 1000 万用户 × 每个计数器 8 字节 = 80MB。轻松放入 Redis。
- 高层架构 — API Gateway → Rate Limit Middleware → Redis 计数器 → 你的服务
- 深入探讨 — 分布式协调、算法选择、故障模式
关键洞察： Rate limiter 位于每个请求的热路径上。你在这里添加的任何东西都会在整个 QPS 上成倍放大。
分布式问题
这里"直接用 Redis"不再是完整答案。有了多个 API 服务器，你需要协调：
中心化（Redis）： 所有服务器检查/递增同一个计数器。原子操作（INCR + EXPIRE，或 Lua 脚本）防止竞态条件。权衡：每个请求都要增加到 Redis 的网络跳数。
本地 + 定期同步： 每个服务器本地计数并定期同步到中央存储。延迟更低，但失去精度——用户可能在同步发生前通过访问不同服务器超出限制。
批量更新： 缓冲几个请求的计数器递增并批量刷新。减少 Redis 负载但引入小的准确性窗口。
Thundering herd 是真实存在的：100 个服务器在流量高峰期同时递增同一个 Redis key。解决方案包括请求合并、Redis 集群和客户端抖动。
Fail-Open vs Fail-Closed
当你的 rate limiting 基础设施宕机时：
- Fail-open： 允许所有请求通过。保持可用性但移除保护。大多数公司选择这个——短暂没有 rate limiting 比丢弃所有流量要好。
- Fail-closed： 拒绝所有请求。对攻击更安全但对合法用户造成中断。
实用方法：fail-open 配合本地回退计数器和熔断器。如果 Redis 不可达，应用保守的内存限制直到连接恢复。
大公司是怎么构建的
Stripe 使用 Redis 配合 sliding window 计数器和 Lua 脚本保证原子性。他们批量操作并处理数十亿 API 请求，延迟开销在亚毫秒级。他们的多层方法同时在 API key、IP 和账户级别应用不同限制。
Cloudflare 在边缘网络级别做 rate limiting——在坏流量到达你的源服务器之前就阻止它。就像在城市的每个入口都有保安，而不仅仅是你的建筑。
GitHub 从简单计数器演进到复杂的滥用检测。他们使用分片、复制的 Redis rate limiter，区分认证和未认证请求，每个端点类别有不同限制。
关键要点
- 算法选择取决于你的突发容忍度 — 大多数 API 用 Token bucket，公平性关键时用 sliding window
- 分布式协调是难点 — 不是算法本身
- 监控你的误报率 — 如果合法用户被 rate limit，你的阈值就错了
- 为故障而设计 — 你的 rate limiter 会宕机；决定它宕机时会发生什么
- 分层防御 — 边缘级、应用级和数据库级限制各自捕获不同问题
- 不要给热路径增加延迟 — 异步更新、批处理和本地缓存是你的朋友