🏗️ 系统设计 / System Design — Day 61

Design an AI Chatbot Platform (ChatGPT-scale)

Difficulty: Expert | Read time: ~3 min

想象你在设计...

你是 OpenAI 的首席工程师。ChatGPT 突然爆红，用户从零增长到一亿。你需要设计一个支撑大规模 AI 对话的平台。每秒处理数万请求，每次请求都要调用一个巨大的语言模型——如何不崩？

You're OpenAI's principal engineer. ChatGPT goes viral overnight. You need a platform that handles hundreds of millions of users, each having multi-turn conversations with a giant language model. How do you architect this?

📐 架构图 / Architecture Diagram

🔑 关键权衡 / Key Tradeoffs

1. 流式 vs 批量 / Streaming vs Batching

• 流式 (SSE/WebSocket): 用户体验好，首 token 快；但连接资源消耗大
• 连续批处理: 把多个用户请求合并一次 GPU forward pass，吞吐量提升 10x
• 选择: 对外用流式，对内用连续批处理

Streaming (SSE/WebSocket): great UX, fast first token — but resource-heavy. Continuous batching: merge multiple user requests into one GPU forward pass = 10x throughput. Use streaming externally, continuous batching internally.

2. 上下文存储 / Context Storage

• 对话历史不能放在服务器内存（无状态横向扩展）
• 用 Redis 存 session → context，TTL 24h 节省成本
• KV Cache offloading: 把 attention KV cache 从 GPU 换到 CPU/SSD 减少显存压力

Conversation history can't live in server RAM (you need stateless horizontal scaling). Store sessions in Redis with 24h TTL. Offload KV caches to CPU/SSD to reduce GPU memory pressure.

3. 模型并行 / Model Parallelism

• 单张 A100 (80GB) 放不下 GPT-4 级别模型
• 张量并行 (Tensor Parallelism): 把权重矩阵切分到多张 GPU
• 流水线并行 (Pipeline Parallelism): 不同层跑在不同 GPU 组上
• 生产中通常 TP×PP 组合

A single A100 (80GB) can't hold a GPT-4-class model. Tensor parallelism splits weight matrices across GPUs. Pipeline parallelism puts different layers on different GPU groups. Production uses both (TP × PP).

4. 多租户隔离 / Multi-tenant Isolation

• 不同 tier（免费/Plus/API）用不同优先级队列
• 免费用户请求在高峰期可被降级（graceful degradation）

Different tiers (free/Plus/API) use different priority queues. Free user requests can be deprioritized during peak load.

🚫 别踩这个坑 / Common Mistakes

1. 用 HTTP 长轮询而不是 SSE — SSE 更简单，断线自动重连，不需要 WebSocket 的复杂握手
2. 不限制上下文长度 — 无限上下文 = 显存爆炸，必须有截断策略（sliding window 或 summarization）
3. 忽略 time-to-first-token (TTFT) — 用户感知的延迟，不是总生成时间，要单独优化
4. 单点 GPU 集群 — GPU 比 CPU 贵 100x，故障成本极高，必须做热备 + 快速重路由

Don't use HTTP long-polling instead of SSE. Don't allow unbounded context (it blows GPU memory). Track TTFT separately from total latency. Never have a single GPU cluster without hot standby.

📚 References

• vLLM: Efficient Memory Management for LLM Serving
• How ChatGPT Actually Works (High Scalability)
• OpenAI Triton & GPU Kernel Optimization

🧒 ELI5

想象一个超级繁忙的餐厅，厨房只有几个特别贵的大厨（GPU）。侍者（Chat Service）把所有订单（请求）汇总，让大厨一次做多道菜（批处理），这样比一道一道做快多了。Redis 是记事本，记住每桌客人之前点了什么（对话历史）。

Imagine a super-busy restaurant where the kitchen has a few very expensive master chefs (GPUs). Waiters collect all orders and let chefs cook multiple dishes at once (batching) — much faster than one at a time. Redis is the notepad that remembers what each table ordered before (conversation history).

💻 算法 / Algorithms — Day 62

🧩 前缀树模式 (2/3) — building on the Trie template from Day 61

#211 Design Add and Search Words Data Structure 🟡 Medium

🔗 LeetCode | 📹 NeetCode

🧩 Trie 模式回顾 (2/3)

这是 Tries 模式块的第 2 题（共 3 题）。上节课 #208 我们实现了标准 Trie（精确匹配）。今天的变化：支持 通配符 .，它可以匹配任意一个字符。

This is problem 2/3 in the Tries block. Day 61's #208 was exact-match Trie. Today's twist: support wildcard . that matches any single character.

模板延伸 / Template extension:

• Insert: 完全相同，直接复用
• Search: 遇到 . 时，需要对当前节点的所有子节点做递归搜索（DFS）

🌍 真实场景 / Real-world Analogy

你有一个联系人应用，用户可以搜索 J.hn（忘记是 John 还是 John-ish？）。. 代表"我不确定这个字母是什么"。Trie 需要在那个位置尝试所有可能。

You have a contacts app. User types J.hn — they forgot if it's John or Jahn. . means "I'm not sure about this character." The Trie needs to try all possibilities at that position.

💡 解题思路 / Solution Walkthrough

class TrieNode:
    def __init__(self):
        self.children = {}  # char -> TrieNode
        self.is_end = False

class WordDictionary:

    def __init__(self):
        self.root = TrieNode()

    def addWord(self, word: str) -> None:
        # Standard Trie insert — identical to template
        node = self.root
        for ch in word:
            if ch not in node.children:
                node.children[ch] = TrieNode()
            node = node.children[ch]
        node.is_end = True

    def search(self, word: str) -> bool:
        # NEW: handle '.' wildcard via DFS
        def dfs(node, i):
            if i == len(word):
                return node.is_end
            
            ch = word[i]
            if ch == '.':
                # Try ALL children — this is the key difference from #208
                for child in node.children.values():
                    if dfs(child, i + 1):
                        return True
                return False
            else:
                # Normal character — standard Trie traversal
                if ch not in node.children:
                    return False
                return dfs(node.children[ch], i + 1)
        
        return dfs(self.root, 0)

执行跟踪 / Execution Trace

⏱️ 复杂度 / Complexity

• addWord: O(L) time, O(L) space — 与模板相同
• search (无通配符): O(L) time — 与模板相同
• search (含 .): O(N × L) worst case — 每个 . 都要尝试所有子节点

- N = 节点总数, L = 单词长度

🔗 举一反三 / Pattern Connections

下节课 #212 是这个模式的终极 Boss：在二维字符网格里找所有存在于词典中的单词——把 Trie 和回溯结合起来。

Next problem #212 is the pattern boss: find all dictionary words in a 2D character grid — combine Trie with backtracking.

📊 Quiz

问题: WordDictionary 搜索 "..." (三个点) 时，最坏情况下会访问多少节点？

• A) 3 个
• B) 整棵树所有节点
• C) 只有叶子节点
• D) 与单词长度无关

<details><summary>显示答案 / Show Answer</summary>

B — 整棵树所有节点 / All nodes in the tree

三个 . 意味着每一层都要尝试所有子节点 → 完整 DFS 遍历整棵树。这是 Trie 通配符搜索的最坏情况。实际系统中通常限制通配符数量或深度。

Three . characters mean every level tries all children → full DFS of the entire tree. This is the worst case. Real systems typically limit the number of wildcards.

</details>

📚 References

• LeetCode #211 Discussion
• NeetCode Trie Playlist
• Trie Data Structure (Visualgo)

🧒 ELI5

普通的 Trie 像查字典——你一个字母一个字母往下找。今天的题加了个 . 通配符，就像"百搭牌"——遇到它就得把所有可能的字母都试一遍，用的是深度优先搜索（DFS）。

A normal Trie is like looking up a word in a dictionary — one letter at a time. Today's problem adds . as a wildcard "wild card" — when you hit it, try every possible letter using DFS.