The 2026 Playwright guide for production scraping

Playwright will happily open a headless Chromium instance and load any page you point it at. What it won’t do for free is keep loading that page a thousand times a day without getting fingerprinted, rate limited, or fed a CAPTCHA. Most tutorials stop at page.goto() and a screenshot; production scraping starts after that, when you have to keep it running unattended for weeks.

This guide is for people who already know Playwright’s basic API and now need to run it as an actual pipeline: proxy pools, retry logic, fingerprint hygiene, and enough monitoring that you find out about a broken selector before your data goes stale. I run scraping infrastructure for proxy testing and lead-gen projects out of Singapore, and everything below is built from scripts I currently run, not from reading the docs once. By the end you’ll have a scraper wired to a rotating proxy pool, running realistic fingerprints, retrying without hammering the target, and packaged in Docker.

what you need

• Node.js 18+ or Python 3.9+ — Playwright ships official bindings for both; I use Python (playwright on PyPI) below since most of my scraping stack is already Python
• Playwright itself, installed via pip install playwright or npm install playwright, plus the browser binaries it downloads on first setup
• A proxy pool — residential or mobile proxies for anything behind aggressive bot protection, datacenter proxies for lower-stakes targets. Providers like Bright Data, Oxylabs, or IPRoyal bill by the gigabyte, commonly $3-15/GB depending on volume and proxy type, so budget accordingly
• Compute — a VM with at least 2 vCPUs and 4GB RAM per handful of concurrent browser contexts. A Hetzner CPX21 or an AWS t3.medium is enough to start
• A CAPTCHA solving service if your targets run Cloudflare Turnstile or hCaptcha — 2Captcha and CapSolver both work, priced per thousand solves
• Somewhere to store output and Docker for packaging — even a Postgres table or JSONL files work, you just need something that survives a crashed run

step by step

1. Install Playwright and pull the browser binaries

Follow Playwright’s official installation guide for the full list of supported runtimes; the CLI stays the same either way.

pip install playwright
playwright install chromium --with-deps

Expected output: pip installs the package, then playwright install downloads a pinned build of Chromium (roughly 150-200MB) and prints a completion line for each browser it fetches. --with-deps also installs the system libraries Chromium needs on a fresh Linux box, avoiding a wall of missing .so errors later.

If it breaks: on a locked-down corporate network the download can hang or time out. Set PLAYWRIGHT_DOWNLOAD_HOST to a mirror, or run the install once on a machine with open internet and copy the ~/.cache/ms-playwright directory over.

2. Write a minimal scraper with proper context isolation

import asyncio
from playwright.async_api import async_playwright

async def scrape(url: str) -> str:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        context = await browser.new_context()
        page = await context.new_page()
        await page.goto(url, wait_until="domcontentloaded", timeout=30000)
        title = await page.title()
        await browser.close()
        return title

print(asyncio.run(scrape("https://example.com")))

Expected output: prints Example Domain. The habit to build here is new_context() per logical session rather than reusing one context for everything, since each context gets its own cookies, storage, and (once you add one) its own proxy.

If it breaks: a TimeoutError on page.goto almost always means the target is slow, blocking you outright, or waiting on a network request that never resolves. Switch wait_until from "load" to "domcontentloaded" first, since "load" waits on every resource including trackers some sites never finish loading.

3. Attach a rotating proxy pool

context = await browser.new_context(
    proxy={
        "server": "http://gate.proxypool.example:8000",
        "username": "user-session-abc123",
        "password": "pass",
    }
)

Expected output: requests from this context now originate from the proxy’s exit IP, not your server’s. Most providers give you one gateway endpoint and let you control rotation through the username — rotating per request, or holding a “sticky” session for a set duration so you don’t get logged out mid-flow.

If it breaks: if every request suddenly returns a login wall or a CAPTCHA, check whether your provider rotated you onto a burned IP. Residential pools cycle IPs constantly, so build a retry that swaps the session ID rather than assuming the target itself changed.

4. Match your fingerprint to your proxy’s geography

context = await browser.new_context(
    proxy={"server": "...", "username": "...", "password": "..."},
    locale="en-SG",
    timezone_id="Asia/Singapore",
    viewport={"width": 1366, "height": 768},
    user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
)

Expected output: the page now reports a timezone and locale consistent with the proxy’s exit country — one of the cheapest signals anti-bot systems check. A Singapore residential IP paired with a America/New_York timezone is an instant tell. Keep the User-Agent string current too — a Chrome 126 UA served in late 2026, when Chrome is several versions ahead, is its own tell.

If it breaks: if you’re still getting blocked after matching locale and timezone, the target is likely checking navigator.webdriver, canvas fingerprinting, or WebGL parameters that stock Playwright doesn’t spoof. That’s the point where people reach for patchright or a dedicated antidetect browser instead of patching Playwright by hand. I keep a running comparison of what holds up against Cloudflare and DataDome at antidetectreview.org/blog/ if you want to skip the trial and error.

5. Add retries with backoff instead of failing the whole run

import random

async def fetch_with_retry(context, url, attempts=3):
    for attempt in range(attempts):
        try:
            page = await context.new_page()
            await page.goto(url, timeout=30000)
            content = await page.content()
            await page.close()
            return content
        except Exception as e:
            wait = (2 ** attempt) + random.uniform(0, 1)
            print(f"attempt {attempt + 1} failed: {e}, retrying in {wait:.1f}s")
            await asyncio.sleep(wait)
    raise RuntimeError(f"failed after {attempts} attempts: {url}")

Expected output: transient failures (a proxy timeout, a slow page) get retried automatically with increasing delay instead of killing a batch over one bad request.

If it breaks: if every URL is hitting the retry ceiling, stop retrying and inspect one failure directly with headless=False and page.screenshot(). Retrying a request that’s actually being blocked just burns proxy bandwidth for nothing.

6. Cap concurrency so you don’t overrun your proxy pool

sem = asyncio.Semaphore(10)

async def bounded_fetch(context, url):
    async with sem:
        return await fetch_with_retry(context, url)

results = await asyncio.gather(*(bounded_fetch(context, u) for u in urls))

Expected output: at most 10 requests in flight at once, regardless of how many URLs you queue. This is the single biggest lever for staying under a target’s rate limit and under your proxy pool’s concurrent connection cap.

If it breaks: if throughput barely improves as you raise the semaphore limit, the bottleneck is probably your proxy provider’s concurrent connection limit, not your code. Check your plan details — most providers cap concurrent sessions separately from bandwidth.

7. Handle CAPTCHAs and bot challenges explicitly

Detect the challenge page rather than letting it silently corrupt your data:

if await page.locator("iframe[src*='challenges.cloudflare.com']").count() > 0:
    token = await solve_turnstile(page, api_key=CAPSOLVER_KEY)
    await page.evaluate(f"document.querySelector('[name=cf-turnstile-response]').value = '{token}'")

Expected output: the challenge iframe is detected, a solving service returns a token, and you inject it before submitting. Exact injection differs by challenge type — check your solver’s docs, since providers update their integration as Cloudflare and Google change their widgets.

If it breaks: if solve rates are low or slow, that’s usually a proxy quality problem, not a solver problem. Turnstile scores the IP reputation before it even shows a widget, so a flagged datacenter IP will fail challenges a clean residential IP sails through.

8. Log and persist output so a crash doesn’t lose the run

import json
from datetime import datetime, timezone

def save_result(data, path="output.jsonl"):
    with open(path, "a") as f:
        f.write(json.dumps({**data, "scraped_at": datetime.now(timezone.utc).isoformat()}) + "\n")

Expected output: every successful scrape appends one line to a JSONL file immediately, so a crash on record 4,000 of 5,000 still leaves you 4,000 usable records instead of zero.

If it breaks: if the file grows but downstream processing chokes, check for partial writes from concurrent tasks. Open the file once per worker process, not once per request, or wrap writes in a lock.

9. Package it in Docker

FROM mcr.microsoft.com/playwright/python:v1.48.0-jammy
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
CMD ["python", "run_scraper.py"]

Expected output: docker build produces an image with Chromium and its system dependencies preinstalled, so you skip the --with-deps dance on every new host. Microsoft publishes these images pinned to a specific Playwright version, which matters because a browser binary mismatch between dev machine and container is a common source of “works on my machine” bugs, as covered in Playwright’s own Docker documentation.

If it breaks: if the container exits immediately with a shared memory error, add --shm-size=1gb to your docker run command or set it in your compose file. Chromium needs more /dev/shm than Docker’s 64MB default.

common pitfalls

• Rotating IPs without rotating fingerprints. A fresh residential IP with the same canvas hash, viewport, and cookie jar as your last hundred requests is still a linkable pattern. Rotate the whole context, not just the proxy.
• Running max concurrency from hour one. I’ve watched a proxy pool get flagged and half-banned within twenty minutes because a script launched with 200 concurrent contexts against a single domain. Start at 5-10 and scale up while watching your success rate.
• Ignoring robots.txt and a target’s terms of service. The Robots Exclusion Protocol isn’t legally binding on its own, but it signals what a site operator considers acceptable, and breaching terms of service can carry legal weight depending on jurisdiction. Not legal advice: if you’re touching personal data or a login wall, talk to a lawyer first.
• Leaking browser contexts. Forgetting to await context.close() on every code path, including error paths, means memory climbs until the process gets OOM-killed. Wrap context lifecycle in a try/finally.
• Treating every block the same way. A 403, a CAPTCHA, and a silently truncated page all look identical if you only check the HTTP status code. Log the page content on failure so you can tell a rate limit from a broken selector.

scaling this

At 10 concurrent sessions, a single VM and a semaphore is all you need. Your proxy pool doesn’t notice you, and a Python script with asyncio.gather handles the whole workload.

At 100 concurrent sessions, you need a job queue (Redis with RQ, or Celery) distributing work across multiple worker processes instead of one giant event loop, plus per-domain rate limiting instead of a single global cap. Session management stops being an afterthought here too: a leaking context correlates scraper sessions the same way it would linked accounts, a problem the multiaccountops.com/blog/ team covers in depth.

At 1000+ concurrent sessions you’re generally not running raw Playwright per worker anymore. Most operators move to a browser farm pattern: self-hosted with a Playwright server (npx playwright launch-server) that workers connect to remotely, or a managed service like Browserless. Proxy strategy usually shifts too: pure residential proxies get expensive fast at this volume, so it’s common to mix in datacenter proxies for low-defense targets and reserve residential and mobile IPs for the sites that actually need them.

where to go next

Once the scraper itself is solid, the proxy layer is usually the next thing worth tuning. I go deeper on picking a pool for this exact use case in residential vs datacenter vs ISP proxies for scraping, and on when to hold a session versus rotate every request in rotating vs sticky sessions. If you’d rather drive a full framework than raw Playwright, the 2026 Scrapy guide covers the same production concerns. For the full archive, check the blog index.

Written by Xavier Fok

disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-07-11.