Proxies for academic research: reproducible web corpora at scale

Quick answer

Proxies for academic research need to do two things: return the same page a user in the study's target country would see, and leave an audit trail that makes the crawl reproducible. Geonode residential proxies cover 195+ countries with country, state and city targeting, and the Scraper API returns rendered HTML, Markdown or JSON with 1.5K free requests per month, which is enough to pilot a corpus before applying for budget.

Why academic web collection breaks without the right proxies

A university crawler runs from a campus network with a handful of public IPs. Any site that counts requests per address starts returning HTTP 429 or CAPTCHAs well before a corpus of tens of thousands of pages is complete, and the whole department shares the penalty.

Localisation is the second problem. News sites, search results, social feeds and e-commerce pages change content, language and even availability by the visitor's country. A study of health misinformation in Brazilian Portuguese media collected from a US campus IP is measuring the wrong population.

Reproducibility is the third. Reviewers increasingly ask how the corpus was gathered, from where, and whether it could be regathered.

Finally, modern pages are rendered in JavaScript. A plain HTTP fetch returns a shell with no article body or comments, and the team spends weeks maintaining a headless-browser cluster instead of doing research.

Which Geonode proxy type fits

The recommended product is Residential Proxies, combined with the Scraper API for rendering. Residential IPs come from real consumer devices in 195+ countries and support country, state, city and ASN targeting (state and city are not combined in one request). That lets a research team define the study population geographically and collect exactly that view of the web.

Sticky sessions up to 24 hours matter for studies that follow a user journey, such as personalisation or recommendation research, where the same session must persist across pages. Rotating mode suits broad corpus collection where every page is independent.

The Scraper API handles JavaScript rendering and returns HTML, Markdown or JSON, so article text and comment threads arrive ready for analysis. The extract, batch, map and crawl endpoints cover single pages, URL lists, site maps and whole-site crawls.

Datacenter proxies are appropriate for open repositories, preprint servers and government statistics sites that do not geo-resolve. ISP proxies fit a long-running panel study that needs a stable identity.

Residential — Best for: Country-specific corpora, personalisation studies, media and platform research; IP source: Real consumer devices, 195+ countries, 2.5M IPs online; Session: Rotating or sticky up to 24 h; Price from: $0.27/GB at volume; 10 GB = $7.92
ISP (static residential) — Best for: Longitudinal panels from a fixed identity; IP source: Static IPs from real ISPs; Session: Unlimited sessions, unlimited bandwidth; Price from: $1.25/IP
Datacenter — Best for: Open repositories, preprint servers, statistics portals; IP source: Rotating datacenter ranges; Session: Rotating; Price from: $0.14/GB

How to set it up

Write the collection protocol first: target sites, countries, time window, crawl rate, and what will be stored. This document is what your ethics board and reviewers will read.
Create a Geonode account and define one username variant per country in the study so every request carries its geography in the log.
Seed the crawl with the Scraper API map endpoint to enumerate URLs, then fetch pages with extract and request Markdown for text analysis.
Store the raw response, the proxy username used, the timestamp and the response headers for every page in an append-only log.
Pilot on a few hundred pages, compare against manual browser checks from the same country, then run the full corpus.

# verify endpoint against docs.geonode.com before publishing
import requests, json, datetime

USER = "USERNAME-country-br"          # rotating, country-targeted
PASS = "PASSWORD"
proxy = f"http://{USER}:{PASS}@premium-residential.geonode.com:9000"

url = "https://example-news.com.br/saude/artigo-123"
r = requests.get(url, proxies={"http": proxy, "https": proxy},
                 headers={"Accept-Language": "pt-BR"}, timeout=30)

with open("crawl_log.jsonl", "a") as f:
    f.write(json.dumps({
        "url": url, "proxy_user": USER, "status": r.status_code,
        "bytes": len(r.content), "ts": datetime.datetime.utcnow().isoformat()
    }) + "\n")

Get the exact snippet for your language from the API Code Generator in the dashboard.

Research-specific settings that matter

Country targeting as a study variable

Treat the proxy username as part of the experimental design. One username per country or state, never mixed within a sample, and record it with every page. When a reviewer asks whether the Brazilian corpus was collected from Brazil, the log answers.

Rotating for breadth, sticky for journeys

Corpus collection of independent pages should rotate so no IP carries a large share of requests to one site. Personalisation and recommender studies need a sticky session id so the platform sees one consistent visitor for up to 24 hours.

Rate limits as an ethics requirement

Set per-domain concurrency low and add delays. Overloading a small newsroom or forum is a harm, and ethics boards treat it that way. Geonode has no fixed thread limit; your protocol sets the ceiling. Honour robots.txt and stop on sustained 429 responses.

Render once, store the source

Use the Scraper API to get rendered Markdown for analysis, but keep the raw HTML as well. Future researchers may need fields you did not extract, and a stored source is the only way to re-derive them without re-crawling.

What Geonode gives you for academic research

Residential IPs in 195+ countries with country, state, city and ASN targeting.
Scraper API with JavaScript rendering, Markdown output and 1.5K free requests per month.
Map and crawl endpoints to enumerate and collect whole sites.
Pay-as-you-go billing with no contract, suitable for grant-funded budgets.
24/7 support and documentation at docs.geonode.com.

Pricing for academic research

Residential traffic is pay-as-you-go from $0.27/GB at volume. The entry plan is 10 GB for $7.92 per month and a 3-day trial costs $5, with no contract, which suits a pilot before a grant is approved. The Scraper API is free for 1.5K requests per month with a 48-hour trial at 5 threads, then from $3.50 per 10K requests or unlimited monthly plans from $47. Datacenter traffic starts at $0.14/GB. See the pricing page.

FAQ

Why do researchers use proxies for academic research instead of a campus network?

A campus has few public IPs, so per-IP rate limits halt a crawl early and affect everyone on the network. Proxies distribute requests, let you choose the country the study targets, and give every request a logged origin for reproducibility.

How do proxies for academic research help with reproducibility?

Every request carries a username that encodes country, state or city and an optional session id. Logged alongside the timestamp and response, this lets you state exactly where and when each page was collected and lets another team repeat it.

Is web scraping for research compliant with ethics and data-protection rules?

Collect only publicly available content, follow your IRB or ethics board protocol, honour GDPR and CCPA when pages contain personal data, respect site terms, robots policies and rate limits, and never collect credentials or private content. Use accounts you own or are authorised to manage. Geonode's AUP prohibits fraud, spam and credential abuse.

Can I collect JavaScript-rendered pages without running browsers?

Yes. The Geonode Scraper API renders JavaScript and returns HTML, Markdown or JSON, so article bodies, comments and dynamically loaded lists arrive ready for text analysis without maintaining a headless-browser cluster.

How is this different from the NGO use case?

The NGO page covers documenting censorship and human-rights content under hostile conditions. Academic research is about corpus collection and reproducible measurement: defining a population geographically, crawling politely, and keeping an auditable log for peer review.

How much data does a typical corpus pilot use?

Text pages are small, usually well under a megabyte with media blocked, so a few thousand pages fit inside the 10 GB entry plan. The Scraper API's free 1.5K monthly requests cover an initial pilot before any budget is spent.

Start

Create an account, run the $5 three-day trial or the free Scraper API tier on a few hundred pages, and compare with manual checks from the target country before scaling the corpus. Get started or Contact sales for institutional invoicing.