Proxies for AI training data: crawl public web text at scale

Quick answer

For AI training data you need two layers: a crawler that returns clean text, and an IP pool wide enough that no single site sees a flood from one address. The Geonode Scraper API crawl endpoint handles rendering and returns Markdown or JSON per page. For custom crawlers, residential proxies with 2.5M IPs online spread requests across real ISP addresses so sites see normal traffic.

Why AI training data collection breaks without the right proxies

A training corpus spans millions of pages across thousands of domains. Hitting each domain from one datacenter range triggers HTTP 429 within minutes, and many sites answer repeated requests from cloud ASNs with CAPTCHAs or empty shells.

JavaScript-heavy sites return almost no text to a plain HTTP client. Forums, documentation portals and news archives render the article body client-side, so your dataset fills with navigation chrome and cookie banners instead of prose.

Region matters too. A news site serves a different edition, or a consent wall, depending on where the request originates. For non-English text you need IPs in those countries to see the local version rather than a redirect.

Finally, licensing and robots policies are part of the pipeline, not an afterthought. A crawler that ignores robots.txt or fetches content behind a login creates legal exposure no model quality gain is worth.

Which Geonode proxy type fits

For most AI training data projects, the recommended product is the Scraper API. Its crawl endpoint takes a seed URL, follows links within the scope you define, renders JavaScript and returns each page as HTML, Markdown or JSON. Markdown output is what you want for tokenisation because it keeps headings and paragraph structure while dropping scripts and styling.

The free tier gives 1.5K requests per month and a 48-hour trial with 5 threads, enough to validate extraction quality on a few hundred pages. Paid usage starts at $3.50 per 10K requests, or monthly unlimited plans from $47 (2 threads) to $1,799 (100 threads) for sustained corpus builds.

If you already run your own crawler (Scrapy, a custom Go fetcher), pair it with residential proxies. The pool has 30–50M unique IPs rotating monthly across 195+ countries, with no fixed concurrency limit, so a crawl of a hundred thousand domains never concentrates on one address.

Residential — Best for: Broad-domain crawls, geo-specific editions; IP source: Real ISP devices, 2.5M online; Session: Rotating or sticky up to 24h; Price from: $0.27/GB at volume
ISP (static residential) — Best for: Long polling of a fixed site list; IP source: Static IPs from real ISPs; Session: Unlimited sessions; Price from: $1.25/IP
Datacenter — Best for: Bulk re-fetch of known-friendly sources; IP source: Datacenter ranges; Session: Rotating; Price from: $0.14/GB

How to set it up

Define the corpus scope: seed domains, URL patterns to include or exclude, maximum depth and a page budget per domain. Write it down first; it becomes your licensing record.
Start on the Scraper API free tier. Submit a crawl job for one seed domain with Markdown output and inspect a sample of pages for boilerplate leakage.
Add a robots.txt and licence check in front of each seed. Skip domains whose terms forbid automated collection or whose content carries a non-permissive licence.
Scale out by batching seeds, storing raw Markdown with URL, fetch timestamp and HTTP status for every page so dedup and audits are possible later.
Run near-duplicate detection (MinHash or SimHash) over stored text before training, and keep the hash index so incremental crawls only add new documents.

# verify endpoint against docs.geonode.com before publishing
curl -X POST "https://scraper.geonode.com/v1/crawl" \
  -H "X-Api-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example-docs-site.org/",
    "max_depth": 3,
    "limit": 500,
    "format": "markdown",
    "respect_robots": true
  }'

Get the exact snippet for your language from the API Code Generator in the dashboard.

AI training data-specific settings that matter

Rotating sessions, not sticky

A corpus crawl has no login state to preserve, so use rotating IPs for every request. Drop the -session- suffix from the proxy username. Each page fetch then leaves from a different residential address, and per-domain request counts stay well under the thresholds that trigger rate limiting.

Per-domain concurrency, not global

Geonode sets no fixed thread limit on residential traffic, but target sites do. Cap concurrency per domain and spread total throughput across many domains at once. Global parallelism can be high as long as per-host pressure stays low.

Country targeting for multilingual corpora

Append -country-de or -country-jp to the username when you want the local edition of a site. Keep a mapping of target language to country code in your crawl config so the same seed list can be re-run per locale without code changes.

Back-off on 429 and 503

Treat HTTP 429 and 503 as signals to slow a specific domain, not as failures to retry immediately. Exponential back-off with a per-domain cooldown keeps the crawl polite and stops you wasting bandwidth on error pages.

What to log

Store URL, final URL after redirects, status code, content hash, fetch timestamp, robots decision and the licence note for the domain. This metadata lets you prove provenance later and remove a source cleanly if a rights holder asks.

What Geonode gives you for AI training data

Scraper API crawl endpoint with JavaScript rendering and Markdown, JSON or HTML output.
Free tier of 1.5K requests per month plus a 48-hour trial with 5 threads.
Residential pool of 2.5M IPs online, 30–50M unique IPs rotating monthly, 195+ countries.
No fixed concurrency limit on residential traffic; HTTP, HTTPS, SOCKS5 and UDP.
Country, state, city and ASN targeting for locale-specific editions of the same site.

Pricing for AI training data

The Scraper API starts free at 1.5K requests per month, then $3.50 per 10K requests, with monthly unlimited plans from $47 for 2 threads up to $1,799 for 100 threads. Residential proxies are pay-as-you-go from $0.27/GB at volume; the entry pack is 10 GB for $7.92, and a 3-day trial costs $5 with no contract. ISP proxies start at $1.25 per IP and rotating datacenter at $0.14/GB. Accounts above 500 GB per month qualify for the "1TB on us" program. Full details on the pricing page.

FAQ

Do I need proxies for AI training data if I use the Scraper API?

No. The Scraper API manages IPs, rendering and retries, so you only send URLs and receive text. Proxies become relevant when you run your own crawler and want to control fetch logic, scheduling and storage yourself.

Which proxies for AI training data work best for multilingual corpora?

Residential proxies with country targeting. Many sites serve a different language edition or consent flow depending on the visitor's country, so fetching from an IP inside that country returns the content a local reader actually sees.

How do I handle deduplication at corpus scale?

Hash each page's Markdown with MinHash or SimHash and keep the index between crawls. Store the content hash with every fetch record so incremental runs only process new or changed documents and near-duplicates can be dropped before tokenisation.

Is collecting training data from the public web legal?

Collect only publicly accessible pages, honour robots.txt and site terms, respect rate limits, and never fetch content behind logins. Geonode's acceptable use policy prohibits fraud, spam and credential abuse. Record licence terms per source so you can remove a domain on request.

Can the Scraper API follow links across an entire site?

Yes. The crawl endpoint takes a seed URL, a depth limit and a page limit, then follows internal links within that scope. Combine it with the map endpoint to list a site's URLs first and filter them before crawling.

How much bandwidth does a training corpus crawl consume?

Rendered pages are heavier than raw HTML. The Scraper API bills per request rather than per gigabyte, which is predictable for page-counted corpora. With residential proxies, store Markdown rather than full HTML to keep storage smaller.

Start

Validate extraction on a few hundred pages with the free Scraper API tier, then scale once the Markdown looks clean. Get started or Contact sales for corpus-scale thread counts and residential volume pricing.