How to scrape Amazon at scale in 2026 with proxies that work
Amazon is probably the single most scraped site in ecommerce, and also one of the most hostile to being scraped. product pages, search results and review pages carry pricing intelligence, keyword data and demand signals that thousands of sellers, dropshippers and market researchers want. the problem is that Amazon has spent over a decade building detection systems specifically to stop you from getting it: fingerprinting, rate limiting by ASIN, CAPTCHA challenges, and IP-range blocks that hit entire datacenter subnets at once.
I’ve run scraping pipelines against Amazon for price monitoring and competitor research since around 2019, first on a single VPS with a handful of datacenter proxies, then later on rotating residential pools once the datacenter approach stopped working around 2021. this tutorial is for operators who need product data (price, stock status, rating, review count, BSR) across hundreds or thousands of ASINs on a recurring schedule, not for someone doing a one-off pull of twenty pages. if that’s you, by the end of this you’ll have a working pipeline that survives Amazon’s anti-bot layer at reasonable volume, plus a clear idea of what changes as you scale from 10x to 1000x.
one caveat before we start: Amazon’s Conditions of Use prohibit automated data collection without permission, and its robots.txt disallows crawling most product and search paths. this isn’t legal advice, and I’m not a lawyer, but you should read those terms yourself and understand that scraping in violation of a site’s terms can carry contract and, in some jurisdictions, computer-fraud exposure under statutes like the Computer Fraud and Abuse Act. the safer, sanctioned path if you’re an actual seller is Amazon’s Selling Partner API, which gives you pricing and catalog data directly. what follows assumes you’ve made your own informed decision about scope and risk.
what you need
- Python 3.11+ with
pipand a virtual environment manager (venv oruv) - Scrapy or Playwright for the crawler itself. Scrapy for pure HTTP scraping, Playwright when you need a real rendered browser for JS-heavy category pages
- a rotating residential or ISP proxy pool. datacenter IPs get flagged fast on Amazon. providers commonly used for this include Bright Data, Oxylabs, Decodo (formerly Smartproxy) and IPRoyal. budget roughly $4-15 per GB depending on provider and volume tier, residential is more expensive than datacenter but it’s the difference between working and not working
- a CAPTCHA solving service as a fallback, 2Captcha or CapSolver are the two most commonly integrated
- somewhere to land the data. Postgres if you want to query it later, or plain CSV/Parquet on S3 if you just need snapshots
- a scheduler, cron on a VPS is enough at small scale, something like Airflow once you’re running dozens of jobs
- realistic budget expectations. for a few hundred ASINs checked daily, expect proxy costs in the tens of dollars a month; costs scale close to linearly with request volume once you’re past the free tiers most proxy providers offer
step by step
1. define your scope
Decide exactly what you’re pulling: specific ASINs, category search results, or both. Amazon renders product pages differently by marketplace (amazon.com vs amazon.co.uk vs amazon.sg), so lock in which marketplace(s) you’re targeting before writing any code, since HTML structure and price formatting differ.
expected output: a list of target URLs or ASINs saved to a seed file, plus a note on which marketplace domain each belongs to.
if it breaks: if you’re not sure whether you need product pages or search results, start with product pages. they’re more stable across Amazon’s frequent front-end changes than search result grids.
2. set up your project
Create a virtual environment and install Scrapy.
python -m venv venv
source venv/bin/activate # or venv\Scripts\activate on Windows
pip install scrapy
scrapy startproject amazon_scraper
cd amazon_scraper
scrapy genspider products amazon.com
expected output: a working Scrapy project skeleton with settings.py, items.py, pipelines.py and a products.py spider file.
if it breaks: if scrapy startproject fails with a permissions error on Windows, run PowerShell as your normal user (not admin) from a directory you own, this is almost always a path permissions issue, not a Scrapy bug.
3. wire up your proxy pool
Most rotating residential providers give you a single gateway endpoint that handles rotation server-side, rather than a list of IPs you manage yourself. Add it as a downloader middleware in settings.py:
# settings.py
DOWNLOADER_MIDDLEWARES = {
"amazon_scraper.middlewares.ProxyMiddleware": 350,
}
PROXY_URL = "http://username-session-rand123:password@gate.proxyprovider.com:7000"
# middlewares.py
class ProxyMiddleware:
def process_request(self, request, spider):
request.meta["proxy"] = spider.settings.get("PROXY_URL")
Rotating the session ID in the username (the rand123 part above) on each request forces the gateway to hand you a fresh residential IP.
expected output: requests going out through different IPs, verifiable by hitting https://httpbin.org/ip a few times through the same spider and seeing the IP change.
if it breaks: if every request returns the same IP, check whether your provider requires a “sticky session” duration setting, some gateways default to holding one IP for 10 minutes unless you explicitly randomize the session string per request.
4. build realistic request headers
Amazon fingerprints more than your IP, it looks at header order, Accept-Language, and whether your User-Agent matches a real, current browser build. Use a maintained list of current desktop User-Agent strings and rotate them alongside your proxy.
HEADERS = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
"Accept-Language": "en-US,en;q=0.9",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
}
expected output: a lower rate of immediate 503s on the first few dozen requests compared to using Scrapy’s default user agent.
if it breaks: if you’re still getting flagged with rotating headers and proxies, check whether your headers are internally consistent, a desktop Chrome user agent paired with a mobile Accept header is itself a fingerprint mismatch.
5. write the parser
Target stable selectors. Amazon changes class names periodically, so anchor on id attributes and semantic structure where you can.
def parse(self, response):
yield {
"asin": response.url.split("/dp/")[1][:10],
"title": response.css("#productTitle::text").get("").strip(),
"price": response.css(".a-price .a-offscreen::text").get(),
"rating": response.css("span.a-icon-alt::text").get(),
"review_count": response.css("#acrCustomerReviewText::text").get(),
}
expected output: structured dicts flowing into your item pipeline, one per product page.
if it breaks: if fields come back empty but the page loads fine in a browser, the content may be rendered client-side. Switch that specific page type to Playwright instead of raw Scrapy requests.
6. add retry and backoff logic
Set Scrapy to retry on the status codes Amazon uses to signal a block, and add jitter so you’re not hammering at a fixed interval.
RETRY_HTTP_CODES = [500, 502, 503, 429]
RETRY_TIMES = 3
DOWNLOAD_DELAY = 2
RANDOMIZE_DOWNLOAD_DELAY = True
expected output: requests that fail transiently succeed on retry with a new IP and header set, instead of the spider dying on the first 503.
if it breaks: if retries keep failing on the same ASIN, that specific product page may be serving a CAPTCHA wall rather than a generic error, check the response body for a captcha form before assuming it’s a rate limit.
7. handle CAPTCHA challenges
At volume, you will hit Amazon’s CAPTCHA page. Detect it by checking the response for known markers ("Enter the characters you see below" or a captcha form field), then route flagged requests to a solving service like 2Captcha before retrying.
expected output: a measurable CAPTCHA rate you can track over time (flagged requests / total requests), ideally kept under a few percent.
if it breaks: if your CAPTCHA rate climbs above 10-15% of requests, that’s usually a signal your proxy pool has gotten “burned,” it’s cheaper to rotate to a fresh IP subnet or a different provider than to keep paying for solves on an IP range Amazon has already fingerprinted.
8. store and monitor
Push parsed items into Postgres or Parquet, and log block/CAPTCHA rate alongside your data on every run so you can see degradation before it becomes total failure.
expected output: a dashboard or simple daily log showing success rate, average response time, and CAPTCHA rate per run.
if it breaks: if your success rate drops sharply overnight with no code changes, check Amazon’s front-end first (they ship layout changes without notice), then check your proxy provider’s status page before assuming your own pipeline broke.
common pitfalls
- using datacenter IPs to save money. they’re cheaper per GB but Amazon blocks known datacenter ASN ranges aggressively, you’ll burn more in wasted requests and retries than you save on proxy cost.
- scraping every ASIN at the same fixed interval. identical timing patterns across thousands of requests are themselves a fingerprint. add randomized jitter to your schedule, not just your delay.
- ignoring marketplace differences. amazon.com, amazon.co.uk and amazon.sg all have different HTML structures and currency formats. a parser tuned for one will silently return
Noneon another. - not separating session identity from request identity. reusing the same proxy session across an entire crawl of one ASIN’s variant pages links all those requests together as one visitor, defeating the purpose of rotation.
- treating scraped data as permanent. prices and stock status on Amazon change by the hour on popular ASINs. build your pipeline to run on a schedule and version your snapshots, not as a one-time pull.
scaling this
at 10x (a few hundred to a couple thousand requests a day), a single VPS running Scrapy with a rotating residential gateway is enough. your bottleneck is proxy cost, not infrastructure.
at 100x (tens of thousands of requests a day), you need a distributed crawl, multiple worker processes pulling from a shared queue (Redis or RabbitMQ works fine), and a bigger proxy pool with sticky-session controls so you’re not accidentally reusing burned IPs across workers. this is also the point where paying for a CAPTCHA solving service stops being optional, manual handling doesn’t scale.
at 1000x (hundreds of thousands to millions of requests), you’re running a real crawling infrastructure: containerized workers on Kubernetes or a managed queue service, a proxy contract with dedicated account management rather than a self-serve dashboard, and active monitoring for block-rate spikes by ASIN category and marketplace. at this volume it’s worth seriously reconsidering whether a scraping pipeline is even the right tool versus a commercial ecommerce data provider or, if you’re a seller, the SP-API. the marginal cost of maintaining a scraper against a target that actively fights you tends to grow faster than linear once you’re past this scale.
where to go next
If you’re building out the proxy side of this pipeline, read our breakdown of residential proxy providers for ecommerce scraping before committing to a plan, pricing tiers vary a lot between providers at this volume. For the CAPTCHA and fingerprinting side specifically, our guide on reducing CAPTCHA rates when scraping at scale goes deeper into header and TLS fingerprint consistency than we covered here. If you’re also managing multiple scraping identities or seller accounts across marketplaces, multiaccountops.com’s blog covers profile isolation in more depth than fits in this piece. And for the full archive of tutorials like this one, browse the blog index.
Written by Xavier Fok
disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-07-11.