Proxies for Web Scraping That Keep Your Crawlers Running

Quick answer

For most web scraping you need rotating residential proxies. They give each request a different real-user IP, so per-IP rate limits and reputation filters stop being the bottleneck. Switch to sticky residential sessions when the target ties state to an IP, use ISP proxies when you need a fixed address for days, and use datacenter proxies for targets that do not screen traffic.

Why web scraping breaks without the right proxies

A single IP making hundreds of requests per minute looks nothing like a human. The first response is usually HTTP 429 Too Many Requests, then a CAPTCHA interstitial, then a silent shadow-limit where the server returns HTTP 200 with empty or stale content. By the time your parser notices, the crawl has already collected garbage.

IP reputation is the second problem. Cloud provider ranges are published, and many sites score them as non-human before the first request lands. A datacenter IP can be served a lighter HTML variant, a login wall, or a different price than a home connection sees.

The third problem is geography. Search results, retail prices, shipping options and streaming catalogues change by country and often by city. If your scraper exits from the wrong region, the data is technically valid and practically useless.

Which Geonode proxy type fits

For general web scraping, start with Residential Proxies. The pool has 2.5M IPs online at any time and 30 to 50M unique IPs rotating monthly across 195+ countries, so the same target rarely sees the same address twice unless you ask for it. You can target by country, state, city, ASN or OS, and choose rotating or sticky sessions up to 24 hours. There is no fixed thread limit, so you scale by adding workers, not by negotiating a bigger plan.

ISP proxies are static residential IPs from real ISPs with unlimited bandwidth per IP, for scrapers that must keep one address for days. Datacenter proxies are the budget option for public targets that do not score IP reputation, such as open data or sitemaps.

If you would rather not manage proxies, headless browsers and retries at all, the Scraper API handles that layer and returns HTML, Markdown or JSON. For this page we assume you run your own scraper.

Residential — Best for: Protected targets, geo-specific content, large rotation; IP source: Real user devices, 195+ countries; Session: Rotating or sticky up to 24 h; Price from: $0.27/GB at volume
ISP (static residential) — Best for: Long-lived sessions, fixed-IP workflows; IP source: Static IPs from real ISPs; Session: Unlimited, static; Price from: $1.25/IP
Datacenter — Best for: Public, unprotected targets at scale; IP source: Rotating datacenter ranges; Session: Rotating; Price from: $0.14/GB

How to set it up

Create an account at app.geonode.com, open the Residential product and copy the proxy username and password from the dashboard. The 3-day trial costs $5 and unlocks the full pool.
Decide on rotation. For stateless page fetches leave the username without a session suffix. For multi-step flows append -session-<id> with any string you control.
Add targeting to the username when the content depends on location: -country-us for country, -country-us-state-texas for state, or -country-us-city-austin for city.
Point your HTTP client at premium-residential.geonode.com:9000 using the username and password, then send a test request to a page that echoes your IP.
Wrap the fetch in a retry with exponential back-off on 429 and 5xx, log the status code and exit IP per request, and only then scale the worker count.

# verify endpoint against docs.geonode.com before publishing
import requests

USER = "USERNAME-country-us"          # add -session-abc123 for a sticky IP
PASS = "PASSWORD"
PROXY = f"http://{USER}:{PASS}@premium-residential.geonode.com:9000"

r = requests.get(
    "https://example.com/products?page=1",
    proxies={"http": PROXY, "https": PROXY},
    timeout=30,
)
print(r.status_code, len(r.text))

Get the exact snippet for your language from the API Code Generator in the dashboard.

Web scraping-specific settings that matter

Rotating versus sticky sessions

Use rotating for product listings, articles and anything you can fetch in one request. Use sticky for add-to-cart flows, paginated results that store a cursor in a cookie, and any sequence where the server compares your IP across steps. A sticky session holds the IP for up to 24 hours, though it can end earlier if the device goes offline, so handle that case.

Country and city targeting

Set the country whenever prices, availability or language vary by region. Add state or city only when the page genuinely changes at that level, because narrower targeting draws from a smaller pool. State and city cannot be combined in one username, so pick the one that matters for the target.

Concurrency and pacing

Geonode sets no fixed thread limit, which means the target site is the constraint. Start with a modest number of workers per domain, measure the 429 and CAPTCHA rate, then raise concurrency until errors begin to climb. Spread requests across IPs rather than hammering one sticky session.

Retries and back-off

Treat 429, 403 and 5xx as retryable, but rotate the IP before retrying rather than repeating from the same address. Use exponential back-off with jitter, cap the attempts per URL, and send failed URLs to a dead-letter queue so a single hostile page does not stall the crawl.

What Geonode gives you for web scraping

2.5M residential IPs online and 30 to 50M unique IPs rotating monthly.
Targeting by country, state, city, ASN and OS across 195+ countries.
Rotating or sticky sessions up to 24 hours, set from the username.
HTTP, HTTPS, SOCKS5 and UDP support with no fixed concurrency limit.
Pay-as-you-go billing, 24/7 support, card and crypto payment.

Pricing for web scraping

Residential proxies are pay-as-you-go with no contract. Entry is 10 GB for $7.92 per month, 50 GB is $34.65, and the per-GB rate drops to $0.27 at volume. A 3-day trial costs $5. Teams above 500 GB per month qualify for the "1TB on us" program, and the unlimited residential plan is $1,800 per month. ISP proxies start at $1.25 per IP and datacenter at $0.14/GB. Compare every tier on the pricing page.

FAQ

Which proxies for web scraping should I start with?

Start with rotating residential proxies. They solve the two most common failures, per-IP rate limits and datacenter reputation scoring, in one step. Move to sticky sessions, ISP or datacenter only when a specific target or budget constraint calls for it.

How much bandwidth does a typical scraping job use?

It depends entirely on page weight. HTML-only fetches are usually small, while pages with embedded images or JavaScript bundles cost more per request. Log response size during a test run of a few hundred pages and extrapolate before you buy a larger package.

Can I use proxies for web scraping with headless browsers?

Yes. Playwright, Puppeteer and Selenium all accept an HTTP or SOCKS5 proxy with credentials, and the same username format for targeting and sessions applies. Use sticky sessions for browser work so cookies, local storage and the IP stay aligned across the page lifecycle.

Do you offer mobile proxies for scraping?

No. Geonode does not sell mobile proxies. Residential proxies are the substitute: the IPs belong to real consumer connections and carry the same ISP-level reputation that most sites use to distinguish home users from cloud traffic, which covers nearly every scraping scenario.

Is web scraping with proxies legal and compliant?

Scrape only data you are permitted to access, respect each site's terms, robots policies and rate limits, and use accounts only when you own them or have authorisation to manage them. Geonode's acceptable use policy prohibits fraud, spam and credential abuse, and we enforce it.

What happens when a sticky session drops before 24 hours?

The IP may change if the underlying device goes offline. Your client receives a new exit IP under the same session id, so design flows to detect the change, re-establish cookies if the target cares, and continue rather than fail the job.

Start

Load the 3-day trial and run the snippet above against your first target. Get started in minutes, or Contact sales if you need volume pricing or a dedicated setup.