How to scrape Google SERP at scale in 2026 with proxies that work
If you’ve tried pulling Google search results programmatically in 2026, you’ve probably already hit a wall: a CAPTCHA after 15 requests, a 429 after 40, or a full IP ban after your third session. Google does not want you scraping its search results pages, and its own robots.txt disallows crawling /search directly. This guide is for people who need SERP data on a recurring basis, rank trackers, SEO agencies watching featured snippets, growth teams monitoring competitor ad copy, and who need a pipeline that survives more than a weekend.
I run proxy infrastructure for a living, that’s literally what this site is about, and the honest answer is that scraping Google SERPs at any real volume is mostly a proxy problem, not a code problem. The scraping logic is maybe 20% of the work. Getting IPs that Google’s systems don’t immediately flag, and making your traffic pattern look like a person instead of a bot farm, is the other 80%. By the end of this you’ll have a working pipeline: a proxy pool that survives days instead of hours, request logic that mimics real browsing, and a parser that doesn’t break the first time Google changes its HTML.
One thing up front: scraping Google’s SERPs isn’t illegal in most jurisdictions I’m aware of, but it does violate Google’s Terms of Service, which prohibit automated querying without permission. This isn’t legal advice, check your own jurisdiction and use case, but go in knowing you’re operating outside Google’s rules, not inside some gray zone they’ve quietly blessed.
what you need
- A scraping framework. Python with
httpxorrequestsfor raw HTTP, or Playwright if you need JS-rendered SERP features (People Also Ask expansion, local packs, shopping carousels) - Residential or mobile proxies. Datacenter IPs (AWS, DigitalOcean, OVH ranges) get flagged by Google within a handful of requests. Budget $3 to $15 per GB for residential, $5 to $30 per GB for mobile, depending on the vendor and target country
- A proxy vendor account. Bright Data, Oxylabs, and Smartproxy are the three most commonly used for this specifically, pick based on the countries you need
- An HTML parser:
BeautifulSouporlxmlfor static requests, or Playwright’s own selector API if you’re rendering - Somewhere to store output. Postgres or even flat JSON files work fine under 50k rows/day, past that you want a real database with an index on query plus date
- A place to run scheduled jobs: cron on a VPS is enough to start, a queue (Celery, RQ) once you’re running concurrent workers
- Budget headroom for CAPTCHA solving if you go past a few thousand queries a day, expect to pay a solving service (2Captcha, CapSolver) roughly $1-3 per 1000 solves
- Patience for the first week: your ban rate will be high until you tune request pacing and headers
step by step
1. decide your query set and cadence
Write out the exact list of queries you’re tracking, or the logic that generates them (keyword lists x locations x devices). Decide how often each needs refreshing, daily rank tracking doesn’t need hourly polling.
Expected output: a CSV or table of query, location, device, and refresh frequency.
If it breaks: if your query list is dynamically generated and balloons past what you budgeted for, cap it explicitly before you build anything else. Uncontrolled query growth is the single biggest reason proxy bills spiral.
2. pick your proxy pool and validate it
Buy a residential proxy plan sized to your expected data usage (a single SERP page is roughly 150-400KB of uncompressed HTML). Before wiring it into your scraper, hit a test query through 20-30 different exit IPs from the pool and confirm you’re getting real result pages back, not immediate CAPTCHAs.
import httpx
proxy_pool = [
"http://user:pass@residential-gw.example.com:22225",
]
for p in proxy_pool:
r = httpx.get("https://www.google.com/search?q=test", proxy=p, timeout=15)
print(p, r.status_code, "captcha" in r.text.lower())
Expected output: most requests return 200 with real result markup, a small percentage show a captcha or “sorry” page.
If it breaks: if more than roughly 20% of fresh IPs are getting captcha’d immediately, the pool has bad-reputation IPs (common with cheaper providers or IPs recently burned by other customers on the same plan). Switch vendor or request a fresher subnet before building anything on top of it.
3. build the request layer with realistic headers
Set a full, consistent header set per session, not just User-Agent. Google’s bot detection looks at header ordering and completeness, a request with only User-Agent and no Accept-Language, Accept-Encoding, sec-ch-ua and so on reads as automated.
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
"Accept-Language": "en-US,en;q=0.9",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"Accept-Encoding": "gzip, deflate, br",
"Sec-Ch-Ua": '"Chromium";v="126", "Not:A-Brand";v="8"',
"Sec-Fetch-Mode": "navigate",
}
Expected output: requests that look, header for header, like a real Chrome session on the OS your User-Agent claims.
If it breaks: if you’re still getting flagged with clean headers, the issue is likely TLS or JA3 fingerprinting rather than HTTP headers, httpx and requests have a different TLS handshake signature than a real browser. Switch to Playwright with a real Chromium binary for the queries that keep getting flagged.
4. match proxy geography to the results you want
Google localizes results by IP geolocation as well as any gl/hl query params you pass. If you’re tracking US rankings, use US exit IPs and set gl=us&hl=en. Mismatched proxy country and locale params is one of the fastest ways to get flagged as suspicious, and you’ll also just get the wrong SERP for your use case.
Expected output: results matching the actual local Google SERP a person in that country would see.
If it breaks: if your rankings look wrong compared to a manual check, suspect proxy IP country before you suspect your parser.
5. add rotation, delay, and jitter
Rotate exit IP per request, or per small batch of requests from the same session, and add randomized delay between requests, 3-8 seconds is a reasonable starting range for a single worker thread. Don’t hammer requests back to back even across different IPs, Google’s rate limiting also looks at request timing patterns.
import random
import time
def fetch(query, proxy_pool, headers):
proxy = random.choice(proxy_pool)
time.sleep(random.uniform(3, 8))
return httpx.get(
"https://www.google.com/search",
params={"q": query, "gl": "us", "hl": "en"},
proxy=proxy, headers=headers, timeout=20,
)
Expected output: a steady, sustainable request rate with a low block rate over hours, not minutes.
If it breaks: if your ban rate climbs over the course of a session even with rotation, your concurrency is probably too high relative to pool size, thin out concurrent workers relative to the number of distinct IPs available.
6. parse the SERP into structured data
Write selectors against the actual result containers, not brittle absolute paths. Google changes class names periodically, structure results by semantic position (organic block, ad block, featured snippet) rather than exact CSS classes wherever you can.
from bs4 import BeautifulSoup
def parse_serp(html):
soup = BeautifulSoup(html, "lxml")
results = []
for block in soup.select("div.g"):
title = block.select_one("h3")
link = block.select_one("a")
if title and link:
results.append({"title": title.text, "url": link.get("href")})
return results
Expected output: a clean list of dicts per query with title, URL, and position.
If it breaks: if your parser suddenly returns empty lists across the board, Google has changed its markup, this happens a few times a year. Pull a fresh sample page and re-check your selectors before assuming the proxy is the problem.
7. handle blocks and CAPTCHAs gracefully
Detect CAPTCHA pages explicitly (check for /sorry/ in the response URL, or known CAPTCHA page markers) and route them to a retry queue with a different IP, rather than letting them silently pollute your dataset as empty results.
Expected output: blocked requests are flagged and retried, not recorded as “zero results” for that query.
If it breaks: if your CAPTCHA rate stays high after switching IPs, the whole pool may already be reputation-damaged from prior use by other customers, a real risk on budget shared-proxy plans.
8. store, monitor, and alert
Log status code, block or CAPTCHA rate, and latency per batch, not just the parsed results. Set a threshold alert (block rate over roughly 15% for an hour, say) so you find out about a burned pool before a week of bad data quietly accumulates.
Expected output: a dashboard or simple daily log showing success-rate trend, not just raw output files.
If it breaks: if you only find out about a burned proxy pool because your rank data looks wrong a week later, you’re missing this step, add it before you scale further.
common pitfalls
Using datacenter IPs to save money. They’re 5-10x cheaper per GB than residential, and they get blocked on Google almost immediately at any real volume. It’s a false economy, you’ll burn more in wasted requests and CAPTCHA solves than you saved on the proxy bill.
Rotating IP but not the rest of the fingerprint. Same User-Agent, same header order, same TLS client across a thousand “different” sessions is still one fingerprint to Google’s systems. IP is one signal among several.
No geographic consistency. Proxy exit in Germany, hl=en&gl=us in the query, and a browser locale set to French. Inconsistent signals across IP, headers, and params get flagged faster than any single bad signal alone would.
Scraping at a constant rate. Real users don’t send a request every 4.000 seconds exactly. Fixed intervals are trivially easy to fingerprint statistically even when IPs rotate.
Treating your query list as static forever. SERP HTML structure and even the URL parameters Google accepts shift over time, a scraper that worked fine in January can silently degrade by July if nobody’s watching the block rate.
scaling this
10x, a few hundred to low thousands of queries a day. Cron plus a single worker, a modest residential proxy plan, manual spot-checks of output quality. This is the “does it work” stage, keep it simple.
100x, tens of thousands of queries a day. You need real concurrency management, a queue system instead of cron, a proxy pool large enough that no single IP gets reused too often per hour, and automated block-rate monitoring instead of manual checks. Cost becomes a real line item here, budget for it and track cost per successful query, not just total spend.
1000x, hundreds of thousands or more a day. Proxy cost usually dominates your unit economics at this point. Most operators running at this scale blend their own scraping with a commercial SERP API (SerpApi, DataForSEO SERP API, or similar) for burst capacity or high-priority queries, and reserve in-house scraping for cost-sensitive bulk volume. You also need real infrastructure discipline: distributed workers, per-vendor proxy pool segmentation so one burned pool doesn’t take down the whole pipeline, and on-call alerting, because a silent failure at this scale means a lot of bad data very fast.
If you’re managing this across multiple browser profiles or accounts rather than raw HTTP calls, a lot of the fingerprinting logic overlaps with antidetect browser setups, we go deeper on that on our sister site at antidetectreview.org/blog.
where to go next
This tutorial covers the scraping pipeline itself. A few things worth reading next, all on our article index:
- /blog/best-residential-proxies-for-serp-scraping-2026, a direct rundown of which providers actually survive Google’s blocking at scale
- /blog/how-to-manage-proxy-pools-across-multiple-scraping-jobs, once you’re running more than one scraper against the same pool
If your workflow also touches multiple accounts or sessions running in parallel, that’s a related but distinct problem, our sister site multiaccountops.com/blog covers the session-management side of that in more depth.
Written by Xavier Fok
disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-07-11.