Web data for RAG: fresh, clean context for LLM grounding

Quick answer

A RAG pipeline needs web data that is current, structurally clean and fetched reliably on a schedule. The Geonode Scraper API extract and batch endpoints render JavaScript and return each page as Markdown or JSON, which chunks and embeds without a cleaning step. For sources that serve different content by region, residential proxies with country targeting fetch the version your users would see.

Why LLM grounding breaks without the right proxies

Retrieval quality depends on what you put in the index. A fetcher that gets an HTTP 429 or a CAPTCHA page stores that error page as a document, and the model later cites a "verify you are human" message as fact. Datacenter IPs on a fixed schedule are the fastest way there.

Raw HTML is the second problem. Navigation menus, cookie notices and footer links become chunks, get embedded and surface as retrieved context. Your model then answers questions about a privacy policy instead of the article the user asked about.

Freshness is the third. Documentation, pricing pages and news change daily. A pipeline refetching hourly from one IP looks like a monitoring bot and gets stale cached pages or shadow-limited responses that silently omit content.

Region-locked sources add a fourth failure: a page that differs by country returns the wrong variant unless the request originates where your users are.

Which Geonode proxy type fits

The recommended product for web data for RAG is the Scraper API. The extract endpoint takes a URL and returns Markdown, JSON or HTML with JavaScript rendering built in. The batch endpoint accepts a list of URLs in one call, which maps directly onto a scheduled refresh of your source list. Markdown output preserves headings, so you can chunk by section rather than by arbitrary character count.

The free tier of 1.5K requests per month covers a small knowledge base. Paid usage is $3.50 per 10K requests, and monthly unlimited plans run from $47 (2 threads) to $1,799 (100 threads) for large source lists refreshed often.

Residential proxies are the secondary option for region-specific content or when you already run a LangChain or LlamaIndex loader. The pool has 2.5M IPs online across 195+ countries with country, state and city targeting, and sticky sessions up to 24 hours for multi-page fetches that depend on cookies.

Residential — Best for: Region-specific sources, custom loaders; IP source: Real ISP devices, 2.5M online; Session: Rotating or sticky up to 24h; Price from: $0.27/GB at volume
ISP (static residential) — Best for: Fixed source list polled on a schedule; IP source: Static IPs from real ISPs; Session: Unlimited sessions; Price from: $1.25/IP
Datacenter — Best for: High-volume refresh of permissive sources; IP source: Datacenter ranges; Session: Rotating; Price from: $0.14/GB

How to set it up

List your sources with a refresh interval per URL: hourly for pricing and status pages, daily for docs, weekly for evergreen references. Group by interval for batch calls.
Call the extract endpoint on a few representative URLs with Markdown output and check that the main content, not the chrome, is what comes back.
Move the grouped lists to the batch endpoint. Store each result with URL, fetch time, content hash and HTTP status beside the Markdown text.
Chunk by Markdown heading, embed, and upsert into your vector store keyed by URL plus chunk index so a refetch replaces old chunks instead of duplicating them.
Skip embedding when the content hash is unchanged. This keeps embedding costs tied to real changes rather than to the refresh schedule.

# verify endpoint against docs.geonode.com before publishing
import requests

API_KEY = "YOUR_API_KEY"
urls = [
    "https://example.com/docs/getting-started",
    "https://example.com/pricing",
]

resp = requests.post(
    "https://scraper.geonode.com/v1/batch",
    headers={"X-Api-Key": API_KEY},
    json={"urls": urls, "format": "markdown", "render_js": True},
    timeout=120,
)
for page in resp.json()["results"]:
    print(page["url"], len(page["markdown"]), "chars")

Get the exact snippet for your language from the API Code Generator in the dashboard.

RAG-specific settings that matter

Markdown over HTML

Request Markdown output and chunk on headings. A section boundary is a semantic boundary, so chunks carry their own context. HTML chunked by character count splits tables and code blocks mid-way, which hurts answer quality more than embedding model choice.

Refresh cadence per source

Do not refetch everything at the highest frequency. Match the interval to how often each source changes, and use the content hash to skip embedding when nothing moved, so request counts and embedding spend track real change.

Country targeting for localised sources

When a source serves different content by region, append -country-us or the relevant code to the residential username, or set the country parameter on the Scraper API request. Store the country with the document so retrieval can filter by the user's locale.

Sticky sessions for multi-page sources

Some documentation portals set a cookie on first visit and serve a shorter page without it. Use a sticky session (-session-<id>) for one site's fetch, then rotate. Keep sessions short; 24 hours is the maximum, not the target.

What to log

Record URL, final URL, status, content hash, fetch time, output format and byte size for each fetch. When a user reports a wrong answer, this log shows whether the index held stale text, an error page or correct content at the time.

What Geonode gives you for RAG pipelines

Extract and batch endpoints returning Markdown, JSON or HTML with JavaScript rendering built in.
Free 1.5K requests per month and a 48-hour trial with 5 threads.
Map and crawl endpoints to discover and ingest whole documentation sites.
Residential IPs in 195+ countries with country, state, city and ASN targeting.
MCP server with keyless access for agents that fetch context at query time.

Pricing for RAG pipelines

Start on the free Scraper API tier at 1.5K requests per month. Beyond that, pay $3.50 per 10K requests or choose a monthly unlimited plan from $47 for 2 threads to $1,799 for 100 threads. Residential proxies for custom loaders are pay-as-you-go from $0.27/GB at volume, with an entry pack of 10 GB at $7.92 and a 3-day trial for $5. No contract on any plan. ISP proxies start at $1.25 per IP and datacenter at $0.14/GB. See the pricing page for all tiers.

FAQ

What format should web data for RAG be in?

Markdown. It keeps headings, lists, tables and code blocks while discarding scripts and styling, so chunks split on natural section boundaries. JSON output is useful when you extract specific fields such as price or date rather than full page text.

How fresh can web data for RAG be with the Scraper API?

As fresh as your schedule. The batch endpoint handles a list of URLs per call, so hourly refreshes of a few hundred pages are routine. Use content hashing to skip re-embedding pages that have not changed since the last fetch.

Do I need proxies if I already use the Scraper API?

Usually not. The Scraper API handles IP rotation and rendering internally. Residential proxies matter when you run your own loader, need a specific country's view of a page, or want sticky sessions across several pages of one site.

Can I ground a model on an entire documentation site?

Yes. Use the map endpoint to list every URL on the site, filter to the sections you need, then pass the list to the batch endpoint or run a crawl job with depth and page limits.

What are the compliance limits for RAG data collection?

Collect only publicly accessible pages, follow robots.txt and site terms, and respect rate limits. Do not fetch content behind logins or paywalls. Geonode's acceptable use policy prohibits fraud, spam and credential abuse. Keep source attribution so answers can cite their origin.

How do I avoid indexing error pages or consent walls?

Check HTTP status and minimum content length before embedding. Discard pages under a sensible size or containing consent-wall phrases. Country targeting fetches the correct regional version so consent interstitials for other regions never reach the index.

Start

Run a few sources through the extract endpoint on the free tier and compare the Markdown with what you index today. Get started or Contact sales for high-thread plans and residential volume.