🏗️ 系统设计 / System Design — Day 23

主题 / Topic: Design a Web Crawler

难度 / Difficulty: Intermediate | 阶段 / Phase: Growth

🌐 真实场景 / Real-World Scenario

想象你在谷歌工程团队——你需要构建一个爬虫，每天抓取数十亿个网页，为搜索引擎建立索引。如何在不压垮服务器、不陷入无限循环的情况下完成这个任务？

Imagine you're on the Google engineering team — you need to build a crawler that visits billions of pages every day to build a search index. How do you do this without overwhelming servers or getting stuck in infinite loops?

🗺️ 架构图 / Architecture Diagram

⚖️ 关键权衡 / Key Tradeoffs

为什么这样设计？ / Why design it this way?

1. URL 去重 — Bloom Filter vs. Hash Set

对于搜索引擎，Bloom Filter 是赢家——偶尔漏爬一个 URL 可接受。

For a search engine, Bloom Filter wins — occasionally missing a URL is acceptable.

2. URL 优先级 / URL Prioritization

• PageRank 高的页面 → 优先级高
• 更新频繁的网站（新闻站）→ 更高频率重爬
• 使用优先级队列（多个队列，按 tier 分）

3. Politeness（礼貌性）

同一个域名的请求间隔 ≥ 1 秒，防止对目标网站造成 DDoS。

Enforce ≥1s delay per domain to avoid accidentally DDoS-ing sites.

❌ 别踩这个坑 / Common Mistakes

坑 1: 忘记处理 Spider Trap（蜘蛛陷阱）

example.com/page?color=red&sort=asc&page=1
example.com/page?color=red&sort=asc&page=2
... (infinite)

修复：URL 规范化 + 最大深度限制（e.g., depth ≤ 10）

Fix: URL normalization + max depth limit

坑 2: 爬取重复内容

不同 URL 可能指向相同内容（www vs non-www, http vs https）。

Use content hash (MD5/SHA1) to detect duplicate content even at different URLs.

坑 3: 忽略 robots.txt

合法爬虫必须遵守 robots.txt 协议，否则可能面临法律风险和 IP 封禁。

Legitimate crawlers MUST respect robots.txt — legal liability + IP bans otherwise.

坑 4: 单点故障

URL Frontier 和存储层都需要分布式、高可用设计（Kafka + distributed DB）。

📊 规模估算 / Scale Estimation

• 目标：15 亿网页 in 4 weeks
• 每秒需爬取：~6,200 pages/sec
• 存储：平均 100KB/page → 150 TB total
• 带宽：6,200 × 100KB = 620 MB/s

📚 References

• https://web.stanford.edu/class/cs276/handouts/lecture22-crawling.pdf
• https://www.cs.cornell.edu/courses/cs6125/2015sp/slides/crawling.pdf
• https://www.youtube.com/watch?v=BKZxZwUgL3Y (System Design Interview — Web Crawler)

🧒 ELI5

爬虫就像一个图书馆员，从一本书开始，读完后抄下所有书里提到的其他书名，然后去找那些书来读。为了不重复，他有一个小本子记下所有读过的书。为了礼貌，他每次只借一本，不一次性抢光所有书。

A web crawler is like a librarian who starts with one book, notes down all the other books it references, then goes to read those — and so on. To avoid repeats, she keeps a notebook of everything she's already read. To be polite, she only borrows one book at a time and doesn't rush the library.

💻 算法 / Algorithms — Day 23

题目 / Problem: #20 Valid Parentheses (Easy)

模式 / Pattern: Stack — 第 1/7 题

🧩 新模式 / New Pattern: 单调栈 (Stack)

📍 本组共 7 题 / This block: 7 problems

什么时候用 / When to use:

括号匹配、下一个更大/更小元素、表达式求值、需要"记住历史"的线性扫描

Parentheses matching, next greater/smaller element, expression evaluation, any linear scan where you need to "remember history"

识别信号 / Signals:

keywords: next greater element, valid parentheses, evaluate expression, monotonic, "最近的"/"nearest"

通用模版 / Template:

stack = []
for i, num in enumerate(arr):
    while stack and arr[stack[-1]] < num:
        idx = stack.pop()
        result[idx] = num  # process: num is the "next greater"
    stack.append(i)
# remaining in stack have no next greater element

核心洞察 / Key Insight:

栈维护单调递减序列——遇到更大的就弹出结算。这是"懒处理"的精髓：把问题推迟到你有足够信息时再解决。

The stack maintains a monotonically decreasing sequence — pop and "settle" when you find something bigger. This is the essence of lazy evaluation: defer processing until you have enough information.

今日问题 / Today's Problem

🔗 LeetCode #20 🟢 Easy

📹 NeetCode Solution

🌎 真实类比 / Real-World Analogy

想象你在审查代码，看到了 {[(。你脑子里是不是会"压栈"，等待对应的 )]} 才放心？这就是栈的直觉！

Imagine reviewing code and seeing {[(. Your brain automatically "pushes" these onto a mental stack, waiting for the matching }]) before relaxing. That's exactly what a stack does!

📋 问题 / Problem

给定一个字符串，仅包含 (, ), {, }, [, ]，判断括号是否合法匹配。

Given a string of brackets, determine if it's valid (every open bracket has a matching close in the right order).

🗺️ 映射到模版 / Map to Template

这题是栈模式最纯粹的应用——不需要"单调"，只需要"匹配"：

This is the purest stack application — not monotonic, just matching:

1. 遇到开括号 (, [, { → 压栈 push
2. 遇到闭括号 → 检查栈顶是否是对应的开括号
3. 最后栈必须为空

✅ Python 解法 / Python Solution

def isValid(s: str) -> bool:
    stack = []
    # map each closing bracket to its opening counterpart
    close_to_open = {")": "(", "]": "[", "}": "{"}

    for char in s:
        if char in close_to_open:
            # closing bracket: check stack top
            if not stack or stack[-1] != close_to_open[char]:
                return False
            stack.pop()
        else:
            # opening bracket: push onto stack
            stack.append(char)

    return len(stack) == 0  # valid only if nothing left unmatched

执行轨迹 / Trace: s = "{[]}"

执行轨迹 / Trace: s = "([)]"

⏱️ 复杂度 / Complexity

• Time: O(n) — 每个字符只处理一次 / each character processed once
• Space: O(n) — 最坏情况全是开括号 / worst case all opening brackets

🔗 举一反三 / Pattern Connections

本组 7 题都用栈，但复杂度递增 / All 7 problems in this block use stacks, increasing in complexity:

📚 References

• https://leetcode.com/problems/valid-parentheses/editorial/
• https://neetcode.io/problems/validate-parentheses
• https://www.geeksforgeeks.org/check-for-balanced-parentheses-in-an-expression/

🧒 ELI5

就像检查你的鞋带是否对称打结：每次遇到一个"开始"就记在本子上，遇到"结束"就检查本子上最新的是不是对应的。如果对，就划掉；如果不对或本子空了，就乱了！

It's like checking if your shoelaces are tied symmetrically. Every time you see a "start" mark, write it in your notebook. When you see an "end", check if the latest note matches. If yes, cross it off. If not — it's all tangled!

📝 Quiz

{"question":"What does the stack contain after processing '([{' in Valid Parentheses?","options":["['(','[','{']","['{','[','(']","Empty — all balanced","[')','[','{']"],"correct_index":0}

🗣️ 软技能 / Soft Skills — Day 23

题目 / Question: How do you approach designing systems for scale?

类别 / Category: Technical Vision | 级别 / Level: Staff

阶段 / Phase: Growth

🎯 为什么重要 / Why This Matters

这道题不是在考你背了多少系统设计八股文——面试官真正想知道的是：你是否有工程判断力？你是否知道什么时候过度设计比没设计还危险？

This question isn't a test of how many system design patterns you've memorized. Interviewers want to know: do you have engineering judgment? Do you know when over-engineering is more dangerous than under-engineering?

对于 Staff 级别，这个问题还在考查你能否影响团队的技术方向，而不只是单独做出好决策。

At Staff level, this also tests whether you can influence your team's technical direction, not just make good decisions yourself.

⭐ STAR 框架 / STAR Breakdown

Situation（背景）

"在我的上一个职位，我们的用户数从 10 万增长到 1000 万，历时 18 个月。早期的架构（单体 + 单数据库）开始出现问题——每次大促前工程师都要彻夜待命。"

"At my previous company, we grew from 100K to 10M users over 18 months. Our early architecture — a monolith with a single database — started showing cracks. Engineers were on-call every time there was a major traffic spike."

Task（任务）

"我被要求制定一个技术路线图，在不停止功能开发的前提下，让系统支撑 10x 的增长。"

"I was asked to create a technical roadmap to support 10x growth without halting feature development."

Action（行动）

1. 先测量，后优化 / Measure first, optimize later

- 用 profiling 工具找到真正的瓶颈，而不是凭直觉猜

- "我们以为数据库是瓶颈，但实际上是缓存未命中率高达 80%"

- Found the real bottleneck with profiling — cache miss rate was 80%, not DB itself

1. 分阶段演进 / Phased evolution

- Phase 1 (1个月): 加缓存层 → 立竿见影降低 DB 负载

- Phase 2 (3个月): 读写分离 → 解放读压力

- Phase 3 (6个月): 拆分最热的服务（用户服务、支付服务）为微服务

- 不一步到位，每个 Phase 都有可量化的目标

1. 设计原则先行 / Establish design principles

- "任何新功能必须考虑 100x 流量下的行为"

- 写进 ADR（Architecture Decision Records），让整个团队对齐

Result（结果）

"系统平稳支撑了大促（峰值 50k RPS），工程师不再需要彻夜待命，且未延误任何计划功能的发布。"

"The system handled peak traffic of 50k RPS during major events without engineers being paged, while delivering every planned feature on schedule."

❌ Bad vs ✅ Good

❌ "我会用微服务架构，然后加 Kafka、Redis、Kubernetes..."

这是在背八股文，没有展示判断力。面试官会追问："为什么？你们当时有多少工程师？"

✅ "我首先会问：我们现在的瓶颈在哪？距离下个量级的增长还有多久？我们团队的规模能支撑微服务的运维成本吗？"

展示了系统性思维和对成本的清醒认识。

🌟 Senior / Staff 加分点

1. 技术债务的主动管理

"我会把可扩展性改进拆成小任务，混进每个 sprint，而不是等到出了大问题再救火。"

2. 影响力而非权威

"我写 RFC（Request for Comments）文档，让团队评审并达成共识，而不是单方面宣布决定。"

3. 权衡的明确性

"我们选择最终一致性而不是强一致性，因为我们的业务场景允许几秒的延迟——但我们把这个决定记录下来，以便未来的人理解为什么。"

📌 关键要点 / Key Takeaways

1. 先诊断再开药 — 用数据找瓶颈，不要凭直觉
2. 渐进式演进 — 分 Phase，每 Phase 有 metrics
3. 成本意识 — 过度设计和欠设计一样有害
4. 文档化决策 — ADR/RFC 让团队与未来的自己对话

📚 References

• https://martinfowler.com/articles/architect-resurgence.html
• https://www.oreilly.com/library/view/designing-distributed-systems/9781491983638/
• https://www.youtube.com/watch?v=Y-Gl4HEyeUQ (Martin Fowler on software architecture)

🧒 ELI5

就像建房子——你不会一开始就建一栋摩天大楼，因为你不确定是否会有那么多人住。但你会让地基足够结实，方便以后加层。先搞清楚现在哪里最拥挤，再一步一步扩建，每次扩建都要能量化效果。

It's like building a house — you don't start by building a skyscraper because you don't know yet if you'll need it. But you make sure the foundation is solid enough to add floors later. First figure out where things are most crowded, then expand step by step — and measure every step.

🎨 前端 / Frontend — Day 23

主题 / Topic: TypeScript with React — Props, Events, Refs

类别 / Category: TypeScript | Week: 4

阶段 / Phase: Growth

🎬 真实场景 / Real Scenario

你在做一个 dashboard 组件库，需要严格约束 props 类型、正确处理 DOM 事件，并且用 ref 访问 DOM 元素（比如自动聚焦一个输入框）。没有 TypeScript？你会在运行时才发现传错了 props。有了 TypeScript？编译时就报错了。

You're building a component library for a dashboard. You need to strictly type props, handle DOM events correctly, and use refs to access DOM elements (like auto-focusing an input). Without TypeScript? You'd find out about wrong props at runtime. With TypeScript? It fails at compile time.

💡 核心概念 / Core Concepts

1. 类型化 Props / Typed Props

// ❌ Bad — no typing, anything goes
const Button = ({ label, onClick, disabled }) => { ... }

// ✅ Good — explicit interface
interface ButtonProps {
  label: string;
  onClick: () => void;         // no arguments, no return
  disabled?: boolean;          // optional with ?
  variant?: "primary" | "danger" | "ghost";  // union type
}

const Button: React.FC<ButtonProps> = ({
  label,
  onClick,
  disabled = false,
  variant = "primary"
}) => (
  <button
    onClick={onClick}
    disabled={disabled}
    className={`btn btn-${variant}`}
  >
    {label}
  </button>
);

2. 事件类型 / Event Types

// The most common event types you'll use daily
const handleInputChange = (e: React.ChangeEvent<HTMLInputElement>) => {
  console.log(e.target.value);    // string
};

const handleFormSubmit = (e: React.FormEvent<HTMLFormElement>) => {
  e.preventDefault();
  // ...
};

const handleButtonClick = (e: React.MouseEvent<HTMLButtonElement>) => {
  console.log(e.currentTarget);  // typed as HTMLButtonElement
};

// Complete form component
interface SearchFormProps {
  onSearch: (query: string) => void;
}

const SearchForm: React.FC<SearchFormProps> = ({ onSearch }) => {
  const [query, setQuery] = React.useState("");

  return (
    <form onSubmit={(e: React.FormEvent<HTMLFormElement>) => {
      e.preventDefault();
      onSearch(query);
    }}>
      <input
        value={query}
        onChange={(e: React.ChangeEvent<HTMLInputElement>) =>
          setQuery(e.target.value)
        }
        aria-label="search"
      />
      <button type="submit">Search</button>
    </form>
  );
};

3. 类型化 Refs / Typed Refs

// useRef needs a type parameter
const inputRef = React.useRef<HTMLInputElement>(null);
//                                   ↑ DOM element type
//                                              ↑ initial value

// Usage: always check for null (could be before mount)
const focusInput = () => {
  if (inputRef.current) {
    inputRef.current.focus();   // TypeScript knows this is HTMLInputElement!
    inputRef.current.select();  // .select() also valid
  }
};

// Full example: Auto-focus on mount
const AutoFocusInput: React.FC = () => {
  const inputRef = React.useRef<HTMLInputElement>(null);

  React.useEffect(() => {
    inputRef.current?.focus();  // optional chaining: safe if null
  }, []);

  return <input ref={inputRef} aria-label="auto-focus-input" />;
};

🧠 猜猜输出 / What's the Output?

interface CardProps {
  title: string;
  count?: number;
}

const Card: React.FC<CardProps> = ({ title, count = 0 }) => {
  return <div>{title}: {count + 1}</div>;
};

// Usage:
<Card title="Users" />
<Card title="Posts" count={5} />

输出 / Output is:

• A: Users: 0 | Posts: 5
• B: Users: 1 | Posts: 6
• C: Users: NaN | Posts: 6
• D: TypeScript error — count is required

答案 / Answer: B — count defaults to 0, so 0 + 1 = 1. Second renders 5 + 1 = 6.

❌ vs ✅ Common Mistakes

坑 1: onClick 类型写错

// ❌ Wrong — MouseEvent is from the DOM API, not React
onClick: (e: MouseEvent) => void;

// ✅ Correct — React's synthetic event
onClick: (e: React.MouseEvent<HTMLButtonElement>) => void;

坑 2: useRef 初始值

// ❌ Wrong — initial value undefined causes issues
const ref = useRef<HTMLInputElement>();

// ✅ Correct — use null for DOM refs
const ref = useRef<HTMLInputElement>(null);

// ✅ Also correct — for mutable values (not DOM)
const countRef = useRef<number>(0);

坑 3: children 类型

// ❌ Old pattern — React.FC injects children automatically (deprecated in React 18)
const Wrapper: React.FC = ({ children }) => <div>{children}</div>;  // TS error!

// ✅ Correct — explicitly type children
interface WrapperProps {
  children: React.ReactNode;
}
const Wrapper: React.FC<WrapperProps> = ({ children }) => <div>{children}</div>;

🧭 何时用 / When to Use

📚 References

• https://react.dev/learn/typescript
• https://www.totaltypescript.com/tutorials/react-with-typescript
• https://react-typescript-cheatsheet.netlify.app/docs/basic/getting-started/basic_type_example/

🧒 ELI5

TypeScript 就像给积木零件贴标签——方块只能插方孔，圆球只能插圆孔。如果你试着把圆球塞进方孔，还没搭好你就知道出错了，不用等整个积木城堡倒塌。

TypeScript is like labeling LEGO pieces — square pegs only fit square holes, round pegs only fit round holes. If you try to force the wrong shape, you find out immediately — not after your entire LEGO castle collapses.

📝 Quiz

{"question":"Which is the correct TypeScript type for an input onChange handler in React?","options":["(e: Event) => void","(e: React.ChangeEvent<HTMLInputElement>) => void","(e: InputEvent) => void","(e: React.MouseEvent) => void"],"correct_index":1}

🤖 AI — Day 23 (NEWS)

日期 / Date: 2026-04-14

阶段 / Phase: Growth

1) 🧑‍⚖️ 联合国 AI 科学小组启动全球影响研究

中文

联合国宣布其“独立国际 AI 科学小组”开始开展一项全球影响研究，核心立场是：AI 发展不能只看算法能力，更要确保“人仍然处在决策中心”。对做 AI 产品/平台的人来说，这意味着两件事会越来越重要：

1) 人类在环（Human-in-the-loop） 的机制是否清晰（谁能覆盖模型、何时必须人工复核）；

English

Source: https://news.un.org/en/story/2026/04/1167263

2) 🧭 美国州级 AI 立法继续推进：聊天机器人披露、价格、健康/治疗等

中文

一篇隐私法律观察更新梳理了 2026 年 4 月中旬美国多州的 AI 法案进展，覆盖“聊天机器人与未成年人互动披露”“对外表明是否为 AI”“以及与健康/治疗服务相关的限制”等方向。工程上最直接的影响是：

• 你可能需要在 UI/对话入口就做“这不是人类”的披露；
• 对未成年人/敏感场景，需要更强的 guardrails（内容限制、引导到真人/专业人士、日志审计）。

English

A state-law tracker summarizes progress across multiple US states, including bills focused on conversational AI disclosures (especially for minors), representing whether a system is human, and restrictions in health/therapy contexts. The near-term engineering implications are: (a) disclosure at the UI / conversation entry point, and (b) stronger guardrails for minors and sensitive domains (content constraints, escalation paths, audit logs).

Source: https://www.troutmanprivacy.com/2026/04/proposed-state-ai-law-update-april-13-2026/

3) 🏢 PwC：AI 经济收益高度集中，领先者更关注“增长与重塑”，并强化治理

中文

PwC 的一份新闻稿强调：AI 的经济收益正在向少数“领先公司”集中，这些公司不仅是“加 AI 工具”，而是更倾向于重塑工作流并把 AI 用作“业务增长/重塑”的杠杆，同时在治理与信任机制上投入更多。对技术负责人而言，一个可操作的 takeaway 是：

• 别只做 feature-level 的“加一个 Copilot”；要做 workflow-level 的重新设计（例如审批链、数据回填、自动化决策边界）；
• 把 Responsible AI / governance 当作“可规模化的前提”，而不是合规尾巴。

English

PwC’s press release argues that AI’s economic gains are concentrated among a small set of leaders. These companies don’t just add AI tools; they redesign workflows and use AI for growth and business reinvention, while investing more in governance and trust mechanisms. A practical takeaway for tech leads: optimize at the workflow level (decision boundaries, approvals, data feedback loops), and treat governance as the prerequisite for scale, not an afterthought.

Source: https://www.pwc.com/gx/en/news-room/press-releases/2026/pwc-2026-ai-performance-study.html

📚 References

• UN News: https://news.un.org/en/story/2026/04/1167263
• Troutman Pepper Locke (privacy law update): https://www.troutmanprivacy.com/2026/04/proposed-state-ai-law-update-april-13-2026/
• PwC press release: https://www.pwc.com/gx/en/news-room/press-releases/2026/pwc-2026-ai-performance-study.html

🧒 ELI5

中文

这三条新闻合在一起就像在说：AI 变得更常见、更能干了，所以大家会更在意两件事：

1) 机器说了算还是人说了算？

2) 出了问题能不能查清楚发生了什么？

English

Together, these stories say: as AI becomes more common and more capable, people care more about (1) who’s actually in charge  the human or the machine, and (2) whether we can audit what happened when something goes wrong.