๐๏ธ ็ณป็ป่ฎพ่ฎก / System Design โ Day 61
่ฎพ่ฎก AI ่ๅคฉๅนณๅฐ๏ผChatGPT ่งๆจก๏ผ
Design an AI Chatbot Platform (ChatGPT-scale)
โฑ๏ธ ้ข่ฎก้ ่ฏปๆถ้ด / Estimated read time: 3 min
๐ฏ ็ๅฎๅบๆฏ / Real-World Scenario
ๆณ่ฑกไฝ ๅจ OpenAI ๅทฅไฝ๏ผ้่ฆ่ฎพ่ฎกไธไธชๆฏๅคฉๅค็ 1 ไบฟๆฌกๅฏน่ฏ่ฏทๆฑ็ AI ่ๅคฉ็ณป็ปใ็จๆทๅไธๆกๆถๆฏ๏ผๅ ็งๅ ๅฐฑๆถๅฐๆตๅผๅๅคโโ่ๅ็ฉถ็ซๅ็ไบไปไน๏ผ
Imagine you're at OpenAI, designing a system handling 100 million conversation requests per day. A user sends a message and gets a streamed response within seconds โ what's happening under the hood?
๐๏ธ ๆถๆๅพ / Architecture Diagram
โ๏ธ ๆ ธๅฟๆ่กก / Key Tradeoffs
ไธบไปไน็จ SSE๏ผServer-Sent Events๏ผ่้ WebSocket๏ผ
- SSE ๆฏๅๅๆต โ ๆๅก็ซฏๆจ้ token ๆด็ฎๅ
- WebSocket ้่ฆๅๅ้ไฟก โ ่ๅคฉๅบๆฏไธๅฟ ่ฆ
- SSE ๅคฉ็ถๆฏๆ HTTP/2 ๅค่ทฏๅค็จ
Why SSE over WebSocket? SSE is one-directional โ perfect for streaming tokens. WebSocket adds bidirectional complexity you don't need here. SSE also works seamlessly with HTTP/2 multiplexing.
ไธบไปไน็จ Redis ๅญๅฏน่ฏไธไธๆ๏ผ
- LLM ๆฏๆ ็ถๆ็๏ผๆฏๆฌก้ฝ่ฆไผ ๅฎๆด history
- ็จๆทๅนณๅๅฏน่ฏ 10-20 ่ฝฎ๏ผtoken ้ข็ฎๆ้
- Redis TTL = 30min ่ชๅจ่ฟๆ๏ผไธๅ ๅๅพ
Why Redis for conversation context? LLMs are stateless โ you must send the full history every request. Redis with TTL handles the limited context window budget and auto-expires stale sessions.
ๆจ็้็พคๅฆไฝๆฉๅฎน๏ผ
- vLLM ็ PagedAttention๏ผGPU KV Cache ๅฉ็จ็ +3x
- ่ฏทๆฑ่ทฏ็ฑๆ session affinity๏ผๅ session โ ๅ GPU ่็น๏ผๅๅฐ KV cache miss๏ผ
How do you scale inference? vLLM's PagedAttention gives 3x better GPU memory utilization. Route by session affinity โ same session to same GPU node minimizes KV cache misses.
โ ๅธธ่ง่ฏฏๅบ / Common Mistakes
ๅ 1๏ผๅฏน่ฏ Context ๆฒกๆๆชๆญ็ญ็ฅ
- LLM ๆๆๅคง context window๏ผๅฆ 128K tokens๏ผ
- ่ถ ้็ดๆฅๆฅ้๏ผ่ฆๅฎ็ฐ sliding window ๆ summary ๅ็ผฉ
Trap 1: No context truncation strategy. LLMs have max context windows. You need a sliding window or summary compression before you hit the limit.
ๅ 2๏ผๆฒกๆ่ฏทๆฑๆ้ๅฏผ่ด GPU OOM
- ๅนถๅๅคง้ๆจ็่ฏทๆฑ โ GPU ๅ ๅญ็็ธ
- ๆญฃ็กฎๅๆณ๏ผinference queue + backpressure
Trap 2: No request queuing leads to GPU OOM. Concurrent requests overwhelm GPU memory. Add an inference queue with backpressure.
ๅ 3๏ผๆฏๆฌก้ฝ้ๆฐๅ ่ฝฝๆจกๅๆ้
- ๆ้ไธๆฌกๅ ่ฝฝ๏ผๅธธ้ฉป GPU
- ็จ Model Registry๏ผS3๏ผ็ฎก็็ๆฌ๏ผ็ญๆดๆฐ
Trap 3: Reloading model weights per request. Weights should be loaded once, resident on GPU. Use a Model Registry for version management with hot-reload.
๐ ๅ่่ตๆ / References
- vLLM: PagedAttention for Efficient LLM Serving
- OpenAI API Streaming Docs
- Designing ML Systems โ Chip Huyen
๐ง ELI5
ๅฐฑๅไธไธช่ถ ็บงๅฟซ็ๆๅญๅ๏ผ
่ๅๆๅพๅค็ต่๏ผGPU๏ผ่ฝฎๆตๅธฎๅฟๆๅญ๏ผ
่ฟๆไธไธช"่ฎฐๅฟๆฌ"่ฎฐไฝไฝ ไปฌ่่ฟไปไนใ