Start with breadth-first search, compare threads with asyncio, then practice timeout, retry, semaphores, and cancellation through focused examples.
A deterministic website with cycles, duplicates, and filtered links.
Run the core examples first, then open each follow-up lesson. Every lesson has a visible experiment and a self-check before moving on.
Understand queues, URL deduplication, fetched pages, and failed pages before introducing concurrency.
Start this module →Use blocking fetch code safely by handing a bounded amount of work to worker threads.
Study thread pools →See how coroutines yield at await so one event loop can coordinate many waits.
Practice timeout, retry, semaphores, and cancellation with tiny runnable files.
Practice controls →Each lesson gives you an idea to understand, code to run, and a question to answer before moving on.
A crawler treats pages as nodes and links as edges. Start sequentially so the queue order is visible and deterministic.
urls_to_crawl holds discovered work waiting to run.discovered_urls prevents cycles and duplicates.fetched_urls records completed pages.Run the baseline and predict the fetch order before reading the output.
python3 synchronous_crawlers.py
python3 -m unittest -v test_crawlersCheck yourself: Why must a URL be marked discovered when it is queued, rather than only after fetching it?
A thread pool runs blocking fetch functions in worker threads. submit() returns a Future, which is a handle to work that may still be running.
The crawler stores pending futures so it knows what is in flight, waits for the first one to finish, calls result() to observe failures, then submits more URLs.
python3 compare_examples.py
# Read:
ThreadPoolWebCrawler.run()
wait(..., FIRST_COMPLETED)Check yourself: Why does the crawler submit only up to the available worker slots instead of submitting every discovered URL immediately?
Asyncio schedules Task objects on one event loop thread. A task pauses at await, leaving the loop free to advance other tasks while a simulated request waits.
The async crawler owns shared state in one coordinator method, so completed tasks return links and the coordinator updates the queue without a thread lock.
python3 async_crawler.py
python3 compare_examples.py
# Read:
AsyncWebCrawler.run()
asyncio.wait(..., FIRST_COMPLETED)Check yourself: Why can completion order differ from sequential BFS order, even if every reachable URL is still found?
Non-blocking waiting still consumes a request slot. A timeout prevents one slow page from holding capacity indefinitely.
python3 -m lessons.lesson_timeout
return await asyncio.wait_for(
get_html_content_async(url),
timeout=timeout_seconds,
)Experiment: Lower the timeout below the simulated page delay. What result should fetch_with_timeout() return?
Retry only temporary failures. Exponential backoff spaces later attempts so a struggling service has time to recover.
python3 -m lessons.lesson_retry
delay_seconds = base_delay_seconds * (
2 ** (attempt_number - 1)
)Check yourself: Why should a malformed URL or permission failure generally not be retried?
A semaphore caps work inside the protected fetch region. It protects a service from a burst, but does not limit how many URLs have been discovered.
python3 -m lessons.lesson_semaphore
async with semaphore:
await fetch_page()Check yourself: What separate limit would stop a huge website from growing the crawl frontier?
Cancelling signals that work should stop. Awaiting cancelled tasks lets their cleanup complete and observes their terminal state.
python3 -m lessons.lesson_cancellation
task.cancel()
await asyncio.gather(
task, return_exceptions=True
)Check yourself: Why is creating tasks without later awaiting or cancelling them risky?
Start with the annotated excerpts, then open the full runnable source directly on this page. GitHub is optional, not the next required step.
def get_html_content(url: str) -> str:
time.sleep(PAGE_DOWNLOAD_DELAYS.get(url, 0.02))
return _read_example_page(url)
async def get_html_content_async(url: str) -> str:
await asyncio.sleep(PAGE_DOWNLOAD_DELAYS.get(url, 0.02))
return _read_example_page(url)submit() returns a Future; wait() pauses coordination until at least one fetch is done.future = thread_pool.submit(self.crawl_page, url)
self.pending_futures.add(future)
done_futures, pending_futures = wait(
self.pending_futures,
return_when=FIRST_COMPLETED,
)
for future in done_futures:
future.result()task = asyncio.create_task(self.fetch_links(url))
pending_tasks.add(task)
done_tasks, pending_tasks = await asyncio.wait(
pending_tasks,
return_when=asyncio.FIRST_COMPLETED,
)
for task in done_tasks:
url, discovered_links = task.result()These small simulations focus on the mental models behind the runnable Python examples.
Requests are already known here. Raising the limit overlaps waits; it does not make one page faster.
Total simulated time:
Using two request slots, advance one event and watch discovered, queued, running, and fetched URLs change.
Post-lesson design lab: combine the two separate runnable lessons. Set latency and temporary failures to predict an eventual request outcome before implementing it in a crawler.
Model: up to 3 attempts with backoff delays of 1 then 2 ticks. Read lesson_timeout.py and lesson_retry.py, then sketch how you would compose them.
Answer before opening the code again. Correct explanations matter more than memorizing syntax.
Why not call requests.get() directly inside async def?
Asyncio needs each task to yield at await. A blocking call keeps the event loop thread busy, so unrelated tasks cannot make progress.
What does max_concurrency=2 limit?
It limits active fetches, not total discovered pages. A crawler may still discover many URLs and hold them in the waiting queue.
What does pool.submit(crawl_page, url) return immediately?
A Future lets the coordinator remember scheduled work, wait for completion, and call result() to surface unexpected worker exceptions.
After cancelling pending tasks, why await them?
Cancellation is delivered into the coroutine. Awaiting gives cancelled tasks a chance to run cleanup and lets the caller observe their final state.