URL Shortener: The "Simple" System That's Anything But
Everyone thinks a URL shortener is the "Hello World" of system design interviews. Hash a URL, store the mapping, redirect on lookup — done, right? Not even close. Bitly handles 600 million clicks per month. TinyURL processes millions of URLs daily. When you dig in, you're designing a globally distributed, read-heavy, analytics-powered system that needs to respond in single-digit milliseconds.
Why It's Harder Than It Looks
The core challenge is the traffic pattern. For every URL you shorten, it might get clicked 100 times. That means you're building a system that's 99% reads — but still needs to handle millions of writes per day. Add analytics (clicks by country, device, time), abuse prevention, and global latency requirements, and your "simple" project is suddenly a real engineering challenge.
Quick math at scale: 100M URLs shortened daily × 100 clicks each = 10 billion redirects/day. That's ~115,000 reads per second and 1,200 writes per second. Storage for 5 years? About 90TB just for URL mappings.
The Encoding Problem: How to Generate Short Codes
This is where most candidates stumble. Here are your options:
| Approach | Pros | Cons | Verdict |
|---|---|---|---|
| MD5/SHA + Truncate | Simple | Unpredictable collisions, no fix without retry | ❌ Avoid |
| Auto-increment Counter | No collisions, simple | Predictable (scraping risk), single point of failure | ⚠️ Risky alone |
| Base62 + Counter | Compact, no collisions | Still predictable unless shuffled | ✅ With randomization |
| Snowflake-style ID | Unique, distributed, not predictable | More complex, longer codes | ✅ Best at scale |
The Instagram approach is elegant: combine a timestamp, machine ID, and sequence number, then base62 encode the whole thing. You get uniqueness without a single point of coordination, and the codes aren't predictable enough to scrape.
Key insight: Hashing is a trap. You can't efficiently detect or resolve collisions at scale. Counter-based approaches with distributed coordination (like Snowflake IDs) are what production systems actually use.
The Interview Framework
When you get this question, follow this path:
- Functional requirements — Shorten URLs, redirect (301 vs 302?), custom aliases, expiration, analytics
- Scale estimation — Write QPS, read QPS, storage over N years, bandwidth
- High-level architecture — Load balancer → Web servers → Encoding service → Database + Cache
- Deep dive on encoding — How to generate short codes without collisions
- Caching strategy — The 100:1 read ratio demands aggressive caching
- Extensions — Analytics, rate limiting, geographic distribution
The 301 vs 302 redirect choice matters more than you'd think: 301 (permanent) is better for SEO but makes analytics harder because browsers cache it. 302 (temporary) ensures every click hits your server — critical for accurate tracking.
Caching: Your Most Important Decision
With a 100:1 read-to-write ratio, your caching layer is arguably more important than your database choice. The pattern is straightforward:
Read path: Check cache → hit? redirect immediately. Miss? query DB → populate cache → redirect.
What to cache: The top 20% of URLs (by popularity) will cover 80%+ of your traffic. Use an LRU eviction policy with TTLs. A few hundred GB of Redis handles billions of active mappings.
Cache invalidation: Mostly a non-issue for URL shorteners — mappings rarely change. Expired URLs get evicted naturally. This is one of the rare systems where caching is simple because the data is essentially immutable.
Database Design: SQL vs NoSQL
For the URL mapping itself (short_code → long_url), you need:
- Fast point lookups by short_code (the redirect hot path)
- Fast writes when creating new URLs
- Horizontal scalability to billions of rows
NoSQL (DynamoDB, Cassandra) is the natural fit: key-value lookups are the primary access pattern, and sharding by short_code distributes load evenly. No joins needed.
SQL (MySQL, PostgreSQL) works too with proper indexing and sharding, especially if you want ACID guarantees for deduplication or custom alias reservation.
Sharding strategy: Hash the short_code to determine the shard. Since codes are already pseudo-random (if using Snowflake-style IDs), you get even distribution for free.
Real-World Implementations
Bitly evolved from a single MySQL instance to a distributed architecture. Their key insight: separate the write path (URL creation) from the read path (redirects) entirely. Writes go to a primary database; reads are served from replicas and cache.
TinyURL kept it simple — a focused architecture that proves you don't always need microservices. Their success shows that a well-tuned monolith with good caching can handle enormous scale.
Instagram's ID generation solved the distributed uniqueness problem without coordination: each shard generates its own IDs using (timestamp, shard_id, sequence). This pattern is directly applicable to URL shortener code generation.
Key Takeaways
- It's a read-heavy system — Design for the 100:1 ratio from day one. Cache aggressively.
- Don't hash for short codes — Use counter-based or Snowflake-style distributed ID generation
- 301 vs 302 is a product decision — Analytics needs vs SEO benefits
- Separate read and write paths — They have completely different scale characteristics
- Caching is your biggest lever — A few hundred GB of Redis eliminates most database load
- Shard by short_code — Even distribution, fast lookups, scales horizontally