High Volume Web Scraping Proxies Priced for Terabyte Crawls

Quick answer

At terabyte scale, cost per GB decides the architecture. Route the bulk of your crawl through rotating datacenter proxies at $0.14/GB for targets that do not score IP reputation, and send only the protected or geo-sensitive share through residential at $0.27/GB at volume. Neither network imposes a fixed thread limit, so concurrency is bounded by your workers and the target, not the proxy.

Why high volume web scraping breaks without the right proxies

Volume changes which failures matter. A crawl of ten thousand pages can absorb a few CAPTCHAs; a crawl of a hundred million cannot, because each one costs a retry, a rotated IP and bandwidth you pay for twice. HTTP 429 responses are the cheap failure. Shadow-limits, where the server returns HTTP 200 with a stripped page, are the expensive one, since you store bad data and discover it after the budget is spent.

IP exhaustion is the second problem. If your proxy pool is small relative to your request rate, the same addresses hit the same target repeatedly and reputation decays across the whole pool at once. The crawl then slows down in the middle of the run.

The third problem is cost drift. Residential bandwidth is priced per GB, and routing pages through residential IPs when a datacenter range would have been accepted raises the bill for no gain.

Which Geonode proxy type fits

For high volume web scraping the recommended default is Rotating Datacenter Proxies at $0.14/GB. Most of any large crawl is unprotected: sitemaps, public listings, open data portals, documentation, news archives and your own sources. Datacenter IPs fetch those pages at roughly half the per-GB cost of residential, with no fixed thread limit.

Residential proxies carry the remainder. When a target scores IP reputation, serves region-specific content, or triggers CAPTCHAs on cloud ranges, switch that domain to the residential pool of 2.5M IPs online across 195+ countries. Pricing falls to $0.27/GB at volume, accounts above 500 GB per month qualify for the "1TB on us" program, and the unlimited residential plan at $1,800 per month removes the per-GB meter entirely for crawls that are residential-heavy.

Residential — Best for: Protected or geo-specific share of the crawl; IP source: Real user devices, 195+ countries; Session: Rotating or sticky up to 24 h; Price from: $0.27/GB at volume
ISP (static residential) — Best for: Small fixed-IP subsets inside a large job; IP source: Static IPs from real ISPs; Session: Unlimited, static; Price from: $1.25/IP
Datacenter — Best for: Bulk of unprotected targets at lowest cost per GB; IP source: Rotating datacenter ranges; Session: Rotating; Price from: $0.14/GB

How to set it up

Create an account and enable both the datacenter and residential products so one set of credentials covers both. Datacenter is selected by adding -type-datacenter to the username.
Build a per-domain routing table with a default of datacenter. Record the 429, CAPTCHA and empty-body rate per domain during a pilot of a few thousand pages each.
Promote any domain whose failure rate stays high to residential by dropping the -type-datacenter suffix and adding -country-xx when the content is region-dependent.
Set worker counts per domain, not globally. Spread requests across IPs with rotating sessions, and use sticky sessions only where a target requires cookie continuity.
Meter bandwidth per domain and per proxy type daily. Compare against the per-GB rates, and strip images, fonts and tracking scripts from fetches to cut bytes.

# verify endpoint against docs.geonode.com before publishing
import requests

PASS = "PASSWORD"
HOST = "premium-residential.geonode.com:9000"

ROUTES = {
    "datacenter":  f"http://USERNAME-type-datacenter:{PASS}@{HOST}",
    "residential": f"http://USERNAME-country-us:{PASS}@{HOST}",
}

def fetch(url, tier="datacenter"):
    proxy = ROUTES[tier]
    return requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=30)

r = fetch("https://example.com/sitemap.xml")          # cheap tier
r2 = fetch("https://example.com/item/42", "residential")  # protected tier
print(r.status_code, r2.status_code)

Get the exact snippet for your language from the API Code Generator in the dashboard.

High volume scraping-specific settings that matter

Tiered routing by domain

Keep a routing table that maps each domain to a proxy tier and review it weekly. Start every new domain on datacenter, promote it on evidence, and demote it again when a target relaxes. Version the table and log every change with the failure numbers that justified it.

Concurrency without a thread cap

Neither Geonode network limits threads, so the ceiling is the target's tolerance and your own infrastructure. Scale workers per domain until the error rate starts rising, then hold. Spread load over time where you can; a crawl that runs evenly for twelve hours puts far less strain on the target than one that bursts for one.

Bandwidth control

Bytes are the bill. Send Accept-Encoding for compression, block image, font and media requests in any headless browser, and prefer API or JSON endpoints over rendered HTML when the site offers them. Measure average bytes per page per domain and treat growth as a regression to investigate.

Retry budget

Define a maximum retry count per URL and a maximum retry share per domain. Rotate the IP before each retry, back off exponentially with jitter, and route a persistently failing URL to a dead-letter queue instead of burning more bandwidth. Retries through residential are the most expensive bytes you buy.

What Geonode gives you for high volume web scraping

Rotating datacenter proxies from $0.14/GB for the unprotected majority of a crawl.
Residential proxies from $0.27/GB at volume, 2.5M IPs online, 195+ countries.
No fixed thread or concurrency limit on either network.
"1TB on us" above 500 GB per month; unlimited residential at $1,800 per month.
One credential set, tier selected per request from the username.

Pricing for high volume web scraping

Datacenter proxies start at $0.14/GB. Residential is pay-as-you-go with no contract: 10 GB for $7.92, 50 GB for $34.65, and $0.27/GB at volume, with a 3-day trial for $5. Accounts above 500 GB per month get the "1TB on us" program, and crawls that are mostly residential can move to the unlimited residential plan at $1,800 per month. ISP proxies start at $1.25 per IP. Model your split on the pricing page.

FAQ

Which proxies are best for high volume web scraping?

Datacenter for the bulk and residential for the protected share. Classify domains by measured failure rate, route the easy ones through datacenter at $0.14/GB, and reserve residential at $0.27/GB at volume for targets that score IP reputation or serve region-specific content.

Is there a concurrency or thread limit?

No. Geonode sets no fixed thread limit on residential or datacenter proxies. The practical ceiling is how many requests each target accepts before raising its error rate, and how many workers your own infrastructure can run and monitor.

How do I estimate cost per GB for a high volume web scraping job?

Run a pilot of a few thousand pages per domain, log bytes per page and the share that needs residential, then multiply by the planned page count. Compare the result against the 500 GB threshold for "1TB on us" and the unlimited residential plan.

When does the unlimited residential plan make sense?

When your residential share alone exceeds what pay-as-you-go would cost at $1,800 per month, or when you need predictable spend for a long-running crawl. For mixed crawls, pay-as-you-go residential plus datacenter is usually cheaper and more flexible.

Is large-scale scraping compliant with platform terms?

Scrape only data you are permitted to access, honour robots policies, terms of service and rate limits, and use accounts only when you own them or have authorisation to manage them. Geonode's acceptable use policy prohibits fraud, spam and credential abuse, and we act on reports.

Can I mix datacenter and residential in one job?

Yes. Both tiers use the same host, port and password; the tier is chosen by the username suffix. A routing layer in your scraper can decide per request, so one job moves domains between tiers without redeploying or changing credentials.

Start

Run a pilot on datacenter, measure what needs residential, and size the plan from real numbers. Get started with a 3-day trial, or Contact sales for volume terms above 500 GB a month.