The 2026 Scrapy guide for production scraping

Most people who ask me about Scrapy already tried requests plus BeautifulSoup for a few hundred pages, watched it fall over on pagination, retries, and concurrency, and are now looking for something that survives contact with a real target site. Scrapy has been the answer to that problem since 2008 and it’s still the answer in 2026. It’s not the newest framework and it’s not the easiest to learn, but it’s the one that runs unattended for months without you rewriting the retry logic every time a site changes its anti-bot rules.

This guide is for operators, not hobbyists. I’m assuming you’re building something that needs to run on a schedule, survive bans, and produce clean data without a human babysitting it every day. I’m not covering “how do I parse HTML” here, I’m covering the parts that actually break in production: proxy rotation, throttling, deployment, and monitoring. By the end you’ll have a spider that runs on a schedule, rotates through proxies, backs off automatically when a target starts blocking you, and logs enough for you to know when something’s wrong before your data goes stale.

One thing I won’t do is tell you what to scrape or pretend scraping is legally uncomplicated everywhere. This is not legal advice. Scraping public data is generally treated differently from scraping behind a login wall or ignoring a site’s terms of service, and the rules vary by jurisdiction and by target. Check the target’s robots.txt and terms before you point a crawler at it, and if you’re unsure, talk to a lawyer, not a blog post.

what you need

• Python 3.10+ and pip (Scrapy 2.13, the current stable line as of mid-2026, supports 3.9 through 3.13)
• A code editor and basic comfort with the command line
• A VPS or a home server you can leave running (a $6/month Hetzner CX22 or DigitalOcean droplet is plenty for a single spider)
• A proxy provider if you’re scraping anything that rate-limits or geo-fences by IP. Residential or mobile proxies cost more per GB than datacenter ones but survive longer against fingerprinting; datacenter proxies are fine for sites that don’t fingerprint aggressively
• Scrapyd or a cron-based scheduler for running spiders unattended
• Somewhere to store output: a Postgres database, S3 bucket, or even flat JSON lines files to start
• Optional but recommended past a handful of spiders: a lightweight monitoring hook (a Slack webhook or a simple health-check endpoint) so you know when a spider silently stops producing items

step by step

1. Install Scrapy and scaffold a project

python -m venv venv
source venv/bin/activate   # venv\Scripts\activate on Windows
pip install scrapy
scrapy startproject pricewatch
cd pricewatch

Expected output: a pricewatch/ directory with settings.py, items.py, middlewares.py, pipelines.py, and a spiders/ folder. Run scrapy version to confirm the install.

If it breaks: on Windows, Scrapy depends on Twisted, which sometimes fails to build from source if you’re missing a C compiler. Install the prebuilt wheel with pip install Twisted before pip install scrapy, or use WSL2 instead of native Windows Python.

2. Generate and write your spider

scrapy genspider prices example.com

Open spiders/prices.py and write a minimal parse method:

import scrapy

class PricesSpider(scrapy.Spider):
    name = "prices"
    start_urls = ["https://example.com/catalog"]

    def parse(self, response):
        for product in response.css("div.product"):
            yield {
                "name": product.css("h2::text").get(),
                "price": product.css("span.price::text").get(),
                "url": response.urljoin(product.css("a::attr(href)").get()),
            }
        next_page = response.css("a.next::attr(href)").get()
        if next_page:
            yield response.follow(next_page, self.parse)

Expected output: a spider that yields one dict per product and follows pagination automatically via response.follow.

If it breaks: if response.css() returns nothing, the site is likely rendering with JavaScript. Check the raw HTML with curl -s https://example.com/catalog | less before assuming your selector is wrong.

3. Test interactively with the Scrapy shell

scrapy shell "https://example.com/catalog"

This drops you into an IPython session with response already loaded, so you can try selectors live: response.css("div.product h2::text").getall().

Expected output: a list of matched strings, confirming your CSS or XPath selector is correct before you run the full crawl.

If it breaks: if the shell returns a 403 or an empty response, the site is checking headers or blocking the default Scrapy user agent. Set a real browser User-Agent in settings.py and retry.

4. Add proxy rotation

Scrapy doesn’t rotate proxies out of the box. The simplest reliable approach is a downloader middleware. In middlewares.py:

import random

class ProxyRotationMiddleware:
    def __init__(self, proxies):
        self.proxies = proxies

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings.getlist("PROXY_LIST"))

    def process_request(self, request, spider):
        request.meta["proxy"] = random.choice(self.proxies)

In settings.py:

PROXY_LIST = [
    "http://user:pass@proxy1.example.com:8000",
    "http://user:pass@proxy2.example.com:8000",
]
DOWNLOADER_MIDDLEWARES = {
    "pricewatch.middlewares.ProxyRotationMiddleware": 350,
}

For anything beyond a handful of proxies, don’t hand-roll a pool manager. pip install scrapy-rotating-proxies or point Scrapy at a gateway endpoint from a proxy vendor (most residential proxy providers give you one sticky gateway IP that handles rotation server-side, which is less code to maintain).

Expected output: requests going out through different IPs, visible if you log request.meta["proxy"] in the middleware.

If it breaks: a TunnelError or repeated connection timeouts usually means the proxy is dead or the port is wrong, not that Scrapy is misconfigured. Test the proxy directly with curl -x http://user:pass@proxy1.example.com:8000 https://icanhazip.com first.

5. Configure AutoThrottle and concurrency

In settings.py:

AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1
AUTOTHROTTLE_MAX_DELAY = 30
AUTOTHROTTLE_TARGET_CONCURRENCY = 2.0
CONCURRENT_REQUESTS_PER_DOMAIN = 4
ROBOTSTXT_OBEY = True

AutoThrottle adjusts delay based on the target’s actual response latency instead of a fixed sleep, which is documented in detail in the official AutoThrottle docs. ROBOTSTXT_OBEY = True makes Scrapy respect the site’s crawl rules as defined in the Robots Exclusion Protocol (RFC 9309), which is the right default even if you plan to override it selectively later.

Expected output: request rate that climbs when the target responds fast and backs off automatically when it slows down or starts erroring.

If it breaks: if you’re still getting banned with AutoThrottle on, the target is probably fingerprinting beyond rate, things like TLS fingerprint, header order, or JS challenge cookies. That’s a different problem than throttling and usually means you need a headless browser layer (Playwright via scrapy-playwright) or a purpose-built unblocking API.

6. Handle retries, bans, and dead proxies

RETRY_ENABLED = True
RETRY_TIMES = 3
RETRY_HTTP_CODES = [429, 500, 502, 503, 504, 522, 524, 408]

Add a custom check in your spider’s parse method for soft bans, sites that return HTTP 200 with a captcha page instead of your data:

def parse(self, response):
    if "captcha" in response.text.lower() or response.css("div.product") == []:
        self.logger.warning(f"Possible block at {response.url}")
        return
    # normal parsing continues

Expected output: failed requests retried automatically up to 3 times, and soft-banned pages logged instead of silently yielding empty data.

If it breaks: if retries pile up and never succeed, rotate the proxy on retry instead of reusing the same one, by removing request.meta["proxy"] before Scrapy’s retry middleware reissues the request, so your rotation middleware assigns a fresh one.

7. Deploy with Scrapyd

For anything running on a schedule, don’t rely on a screen session. Install Scrapyd on your server:

pip install scrapyd scrapyd-client
scrapyd &
scrapyd-deploy default
scrapyd-deploy default -p pricewatch

Then schedule a run:

curl http://localhost:6800/schedule.json -d project=pricewatch -d spider=prices

Wire that curl call into cron for a daily run at, say, 3am server time.

Expected output: Scrapyd’s web UI at localhost:6800 shows the job queue, and you can check job status without SSH-ing in and grepping logs.

If it breaks: if scrapyd-deploy fails with an egg-building error, check that scrapy.cfg in your project root has the right [deploy] target URL, that’s the most common misconfiguration.

8. Monitor output, not just uptime

A spider that runs successfully but returns zero items is a silent failure. Add an item count check to your pipeline or a post-run script that alerts if output drops below a threshold compared to the prior run. A simple Slack webhook call after each Scrapyd job finishes is enough for most operators; you don’t need a full observability stack for a handful of spiders.

Expected output: a Slack message (or equivalent) after every run stating item count and any warnings logged.

If it breaks: if counts drop to zero without errors in the log, the target likely changed its HTML structure. This is the single most common cause of “working” spiders that quietly stop producing data.

common pitfalls

• Ignoring robots.txt without a reason. ROBOTSTXT_OBEY = True is the default for a reason. If you’re overriding it, know why, and know the terms of service implications for that specific target.
• Running full concurrency from day one. Sites that would tolerate 2 requests/second forever will ban you in minutes at 50. Start conservative, watch your ban rate, then increase.
• Treating proxy IP quality as uniform. Datacenter IPs get flagged fast on e-commerce and social targets. If you’re getting banned constantly despite reasonable throttling, the proxy pool, not your spider code, is usually the problem.
• No schema validation on output. A site redesign can quietly return None for every price field for weeks before anyone notices, if nothing checks the shape of what’s coming out.
• Storing everything in memory or a single JSON file. Past a few thousand items per run, write directly to a database or JSON lines file with FEED_EXPORT_BATCH_ITEM_COUNT set, so a crash mid-run doesn’t lose the whole batch.

scaling this

At 10 targets, a single VPS running Scrapyd with cron-scheduled jobs is enough. One proxy pool, one settings.py, minimal monitoring beyond a Slack ping.

At 100 targets, you need per-target concurrency and delay settings (not every site tolerates the same rate), a proxy budget that scales with GB used, and a real job queue instead of cron, something like Scrapyd’s built-in queue or a lightweight task runner like Celery if you need retries across job failures, not just request failures. This is also where per-domain logging starts mattering, because debugging 100 spiders from one combined log file doesn’t work.

At 1000+ targets, you’re running a distributed crawl, typically Scrapy behind a message queue (Redis via scrapy-redis is the common pattern for distributing URLs across multiple worker machines), with proxy management handled by a dedicated gateway rather than a hardcoded list, and monitoring that tracks per-domain success rate, not just whether the overall job finished. At this scale the bottleneck usually isn’t Scrapy, it’s proxy cost and target-specific anti-bot handling, which is closer to a full-time operations job than a weekend script.

where to go next

If you’re setting up the proxy side of this, I go deeper into provider selection and pricing in our guide to picking proxies for scraping in 2026. For the pure-Python approach without Scrapy’s overhead, see how to rotate proxies in Python requests. And if your crawl needs to look like distinct real users rather than one bot hitting rotating IPs, that’s session and fingerprint management territory, multiaccountops.com’s blog covers the account and session infrastructure side of that problem in more depth than I can here.

More tutorials like this one are indexed on our blog.

Written by Xavier Fok

disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-07-11.