Respectful Web Crawling and Downloads with Courlan

A simple URL list is useful for prototyping but rapidly fails as the number of sources grows. It has no concept of domains, so one busy site absorbs all requests while others wait. From the perspective of the affected site this is also indistinguishable from a bot attack. How can you stay polite and optimize your download speed at the same time?

Courlan is a Python library for URL wrangling. It is also the backbone of Trafilatura, a software package to gather text on the Web. While Trafilatura is now downloaded several times per second on average, I want to shed light on its lesser-known underlying component.

Courlan uses a domain-aware internal representation called the URL Store which holds URLs along with relevant information and entails crawling helpers. It is designed for small to medium-scale crawls where correctness and politeness both matter. In its standard form it is best operated from a single machine, which is ideal for academic research with limited resources.

UrlStore: Helpers for Smarter Crawls

The robots.txt file is generally available at the root of a website. It describes what a crawler should or should not crawl according to the Robots Exclusion Standard.

Every website in the store tracks its robots.txt rules, along with information about the last access and the total number of pages visited per site. Politeness constraints follow naturally from the data model.

from courlan import UrlStore

store = UrlStore()

store.add_urls([
    "https://example.com/article-1",
    "https://example.com/article-2",
    "https://other-site.org/post",
])

print(store.total_url_number())          # 3
print(store.unvisited_websites_number()) # 2

Collecting URLs and handling robots.txt

A web crawler scouts the Web and constantly stores new URLs to visit. The add_from_html method discovers links and applies stored robots.txt rules before anything enters the queue, and routes navigation links (sitemaps, pagination) to the front. The incoming URLs are also automatically deduplicated to ensure the count is reliable.

from urllib.robotparser import RobotFileParser

parser = RobotFileParser()
parser.set_url("https://example.com/robots.txt")
parser.read()
store.store_rules("https://example.com", parser)

# Disallowed paths dropped, nav links prioritised — automatically
# Assuming html_content has already been downloaded
store.add_from_html(html_content, "https://example.com/article-1")

delay = store.get_crawl_delay("https://example.com", default=5.0)

Respectful Parallel Downloads

The get_download_urls() method returns URLs which are ready to fetch after time_limit seconds have passed. Here is the code Trafilatura uses under the hood:

from concurrent.futures import ThreadPoolExecutor, as_completed
from time import sleep
from trafilatura import fetch_url

def load_buffer(store, time_limit=5):
    while True:
        urls = store.get_download_urls(time_limit=time_limit)
        if urls or store.done:
            return urls
        sleep(time_limit)

with ThreadPoolExecutor(max_workers=4) as pool:
    while not store.done:
        futures = {pool.submit(fetch_url, url): url for url in load_buffer(store)}
        for future in as_completed(futures):
            store.add_from_html(future.result(), futures[future])

Each domain gets one URL per batch, spaced time_limit seconds from its last request. Requests to example.com and other-site.org run in parallel while requests to the same domain stay separated.

For a pre-computed download list, see establish_download_schedule() which returns (delay_seconds, url) pairs that distribute load across domains.

Persistence Across Sessions

The information can be preserved using a native Python format called pickle which doesn't require additional dependencies. This relatively light format can then be loaded into memory to continue.

store.write("my_crawl.pkl")

# Resume with all information later on
from courlan import load_store
store = load_store("my_crawl.pkl")

The Bottom Line

The UrlStore class steers the crawl and handles domain classification, per-host scheduling, robots.txt enforcement, priority ordering, and persistence. The original code is also easily extendable to adapt to specific cases, for example to affect the way certain websites are prioritized or to refine the politeness rules.

For more information, refer to the Courlan repository.