Scraping thousands of websites with Playwright, without running out of memory

At Data Department we visit Dutch business websites with Playwright and headless Chromium, to read what a normal visitor would see. Getting started with Playwright takes ten minutes. Keeping 32 pages loading at the same time, all day, on one box, took a lot longer. Everything below comes from our own production setup, and every number was measured on it.

1. Don’t download what you won’t read

A scraper needs the HTML and the JavaScript that builds it. It doesn’t need photos, web fonts or videos. Images are the expensive part, and not because of the download: the browser decodes every image into a bitmap in memory. Blocking images roughly halved the peak memory of our workers.

Playwright lets you intercept every request and abort the ones you don’t want:

from urllib.parse import urlsplit

BLOCKED_TYPES = {"image", "font", "media"}

# Blocked by host, because an embedded YouTube player is an iframe and a
# script bundle, not "media", until someone presses play.
BLOCKED_HOSTS = (
    "youtube.com", "ytimg.com", "googlevideo.com", "vimeocdn.com",
    "google-analytics.com", "doubleclick.net", "hotjar.com", "clarity.ms",
)

def blocked_host(url):
    host = (urlsplit(url).hostname or "").lower()
    return any(host == h or host.endswith("." + h) for h in BLOCKED_HOSTS)

async def block(route):
    request = route.request
    if request.resource_type in BLOCKED_TYPES or blocked_host(request.url):
        await route.abort()
    else:
        await route.continue_()

await context.route("**/*", block)

Three things we learned the hard way:

• Block by host as well as by type. Video players and trackers arrive as ordinary scripts and frames. A short list of hosts catches what the resource type misses. Build that list from your own traffic logs, not from guesses.
• Be careful with JavaScript. Analytics and ad scripts can go. The site’s own code can’t: it is what renders the page. The same goes for CAPTCHA and bot-check scripts such as Cloudflare Turnstile, which are what let you through on the harder sites.
• Never abort the document itself, or navigation fails. Stylesheets are a judgement call: for single-page apps where we only want the text, we block them too.

A blocked request still fires Playwright’s request event, so if you record which trackers a site loads, you still see them.

Two catches with routing. Requests made by a site’s service worker bypass your routes completely, so a site with a service worker can still load every image you thought you blocked. Create contexts with service_workers="block". And routing turns off the HTTP cache. With a fresh context for every page there is little cache to lose, but if you reuse a context for many pages on one site, you’ll download its shared files again for every page.

2. One browser, many contexts

The obvious way to run pages in parallel is to start more browsers. Don’t. A browser context is a separate, clean session inside one browser: its own cookies, storage and cache, but none of the startup cost. We measured both on the same server, the same list of websites and the same code:

Setup	Pages per minute	Peak memory
1 browser × 16 contexts	128	1.1 GB
2 browsers × 8 contexts	81	0.9 GB
4 browsers × 8 contexts	80	1.0 GB

Same number of pages at once, 59% faster with one browser. Four browsers with twice the slots were still slower. On our hardware the curve flattens at about 32 contexts per browser (180 pages a minute); going to 48 bought 5% more speed for 38% more memory.

The pattern is one context per page, closed no matter what happens:

import asyncio

slots = asyncio.Semaphore(32)

async def scrape(browser, url):
    async with slots:
        context = await browser.new_context(service_workers="block")
        try:
            await context.route("**/*", block)
            page = await context.new_page()
            await load(page, url)  # goto, plus crash detection (step 3)
            return await page.content()
        finally:
            await context.close()

One warning: this is a Chromium result. Camoufox, the hardened Firefox we use for difficult sites, behaves the opposite way. With one browser and four contexts, three out of four pages timed out; four browsers with one context each all succeeded. The failures looked like slow websites, not like a configuration mistake, which cost us a while to find. Measure each engine on its own.

3. Cap the memory, and contain the crashes

Some websites have JavaScript that never stops allocating memory. Chromium accepts flags for its JavaScript engine, so you can put a ceiling on each page’s heap:

browser = await playwright.chromium.launch(args=[
    "--js-flags=--max-old-space-size=512",   # MB of JavaScript heap
    "--disable-gpu",
    "--disable-extensions",
    "--disable-background-networking",
    "--disable-sync",
    "--no-first-run",
])

The limit applies to each renderer process, the part of Chromium that runs a page. A page that runs past it crashes its own renderer instead of eating the machine, and Playwright reports that as the page’s crash event. Pages in different contexts don’t share renderers, so the other pages keep loading. Chromium only starts reusing processes past a soft limit it sets from the machine’s memory; open chrome://process-internals on a running worker to see where you stand. Flags like --single-process or a low --renderer-process-limit save memory, but they put pages back in shared processes, and one crash then takes several pages with it.

Keep in mind what the flag doesn’t cover. It limits JavaScript objects only. The page itself, its layout and decoded images live outside that heap, so a heavy page can still use far more than 512 MB. The heap cap stops runaway scripts; the container limit in step 5 is what protects the machine.

Memory is only one way things go wrong. Contain the rest at three levels:

• The page. Give every URL a hard time limit on top of Playwright’s own timeouts. A page that hangs inside a wait should cost one slot for a minute and a half, not forever. Listen for the page’s crash event too: a crashed renderer otherwise shows up as a timeout, and you want to count heap-limit crashes separately from slow websites.
• The browser. Before creating a context, check that the browser is still connected, and start a new one if it isn’t. With 32 tasks running, they all notice at the same moment, so put the restart behind a lock and only replace the browser that died. Otherwise you start 32 replacements.
• The Playwright driver. Playwright talks to the browser through a separate driver process. When that process dies, browser.is_connected() keeps saying yes, and every new context fails. The only signal is the failed call itself: catch it, restart Playwright, and try once more. Without this, our workers kept taking work from the queue while failing every single page.

async def load(page, url):
    crashed = asyncio.get_running_loop().create_future()
    page.on("crash", lambda _: crashed.done() or crashed.set_result(True))
    goto = asyncio.ensure_future(page.goto(url, timeout=30_000))
    done, _ = await asyncio.wait({goto, crashed}, return_when=asyncio.FIRST_COMPLETED)
    if crashed in done:
        goto.cancel()
        raise RuntimeError("renderer_crash")  # its own error, not a timeout
    return goto.result()

async def scrape_with_limit(url):
    try:
        return await asyncio.wait_for(scrape(browser, url), timeout=90)
    except asyncio.TimeoutError:
        return None  # record it and move on; the slot is free again

async def new_context():
    global browser
    if not browser.is_connected():
        browser = await relaunch(browser)
    try:
        return await browser.new_context()
    except Exception as error:
        if "closed" not in str(error).lower():
            raise
        browser = await relaunch(browser)  # restarts the driver if needed
        return await browser.new_context()

relaunch_lock = asyncio.Lock()

async def relaunch(dead):
    global browser
    async with relaunch_lock:
        if browser is dead:  # someone else may have replaced it already
            browser = await launch_browser()
    return browser

4. Keep Chromium off the disk

This one took longest to find. Our server had plenty of processor time left, yet pages slowed down as we added more. The disk was 88% busy, writing about 400 MB per second, on a scraper that hardly stores anything.

Chromium’s processes share memory through /dev/shm. Docker gives containers only 64 MB there, which isn’t enough, so the usual advice is to add --disable-dev-shm-usage. Chromium then puts that shared memory in /tmp instead, which is on disk. Playwright adds the flag by default, so every renderer was writing its memory to disk while our 4 GB of /dev/shm sat unused at 0%.

Leaving the flag out of your own arguments doesn’t help, because Playwright always adds it. You have to remove it from Playwright’s defaults, and give the container a real /dev/shm. Playwright’s own Docker advice is --ipc=host, which shares the host’s memory instead; a large --shm-size does the same job while keeping the container separate from the host.

browser = await playwright.chromium.launch(
    args=[...],
    # Only do this when the container has a large /dev/shm (see below)
    ignore_default_args=["--disable-dev-shm-usage"],
)

Even with that fixed, the disk is the next wall: at our best setting the processors still spent about 30% of their time waiting for it. Watch the iowait figure when you scale, not only CPU and memory.

5. Put the browsers in a box

We run each worker as a container: one browser, 32 contexts, a fixed amount of memory and a fixed set of processor cores. When something goes wrong, the container is what fails, not the server. In plain Docker:

docker run \
  --shm-size=4g \
  --memory-reservation=4g --memory=10g \
  --cpuset-cpus=12-95 \
  --init \
  --stop-timeout=75 \
  my-browser-worker

• --shm-size: the real shared memory from step 4. We use 4 GB for 32 contexts with images on, and 512 MB for 3 contexts with images blocked.
• A hard cap, and a lower reservation. --memory is where the kernel kills the container. --memory-reservation is a soft limit Docker only enforces when the host runs short; in a scheduler like Nomad, the reservation is what it uses to decide where a container fits. Running out of memory is a cliff, not a slope: one kill loses all 32 pages in flight. Our workers use around 3 to 4.5 GB, but spikes went past 8 GB, so the cap is 10 GB. Note that Docker lets a container use as much swap again as its --memory if the host has swap; set --memory-swap to the same value if you’d rather fail fast than slow down.
• Pin the cores. Our browser box also runs Lighthouse audits on cores 0 to 11. Before we pinned the scrapers to the other cores, pages that ran out of time went from 0.2% to 18.5% within an hour, only because some of our threads landed on busy cores.
• --init reaps the browser’s child processes when they exit, so dead renderers don’t pile up as zombies.
• Don’t run as root. A scraper visits websites you don’t control. Chromium’s sandbox doesn’t work as root, so many Docker examples switch it off with --no-sandbox. Playwright advises a separate user inside the container for scraping, which keeps the sandbox on.
• Stop slowly. Give the container longer to stop than your per-page time limit, so a deploy lets pages finish and hands unfinished work back to the queue.

Then find out how many containers the machine really holds, by adding them one at a time and counting good pages, not pages. On our 96-core box, 12 containers looked busy but only 54% of pages succeeded. Six containers gave 83% success and about 474 pages a minute, more good pages than twelve.

The short version

1. Block images, fonts, media, video players and trackers; keep the site’s own scripts, and block service workers so nothing slips past.
2. One browser with many contexts, one context per page, always closed.
3. Cap the JavaScript heap, and plan for crashes at page, browser and driver level; listen for crash.
4. Give Chromium a real /dev/shm and remove Playwright’s default that sends it to disk.
5. Run each browser in a container with fixed memory and cores, as a normal user with the sandbox on, and measure how many fit.