Incremental web crawling is the practice of refetching only the pages that changed, not the whole site. By the end of this guide you will have a Python module that discovers URLs from a sitemap or the Geonode /v1/map endpoint, normalises them, keeps per-URL state in SQLite, uses ETag and content hashes to decide what changed, adapts revisit intervals, detects deletions, and pulls the changed set through /v1/extract or /v1/batch. Geonode fetches the pages; your script owns the state and the comparison.
Why full recrawls waste budget
A site with 50,000 pages where 2% change per week needs about 1,000 fetches a week to stay current. A weekly full recrawl spends 50,000. That is a 50x difference in API requests, bandwidth, and the time your pipeline blocks on I/O.
The waste compounds on JavaScript-rendered pages, where each fetch is slower and more expensive, and on sites that rate-limit aggressively. Incremental web crawling turns the problem around: you keep a small amount of state per URL and let that state tell you where to spend the next request.
The module below is about 120 lines. Each section adds one piece. Install the dependency with pip install requests (httpx works identically if you prefer it).
Discovering URLs
Two sources, in order of preference. A sitemap gives you URLs plus an optional lastmod, which is the cheapest change signal there is. When a site has no usable sitemap, POST /v1/map discovers URLs by combining sitemap parsing with HTML link extraction and returns each URL with its source.
import os
import xml.etree.ElementTree as ET
import requests
API = "https://scraper.geonode.io"
HEADERS = {"X-Api-Key": os.environ["GEONODE_API_KEY"]}
SM_NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
def urls_from_sitemap(sitemap_url: str, session: requests.Session) -> list[tuple[str, str | None]]:
"""Return (loc, lastmod) pairs. Follows sitemap index files one level."""
root = ET.fromstring(session.get(sitemap_url, timeout=30).content)
out: list[tuple[str, str | None]] = []
for sm in root.findall("sm:sitemap", SM_NS): # sitemap index
out.extend(urls_from_sitemap(sm.findtext("sm:loc", namespaces=SM_NS), session))
for u in root.findall("sm:url", SM_NS):
out.append((u.findtext("sm:loc", namespaces=SM_NS),
u.findtext("sm:lastmod", namespaces=SM_NS)))
return out
def urls_from_map(site_url: str, session: requests.Session) -> list[tuple[str, None]]:
body = {"url": site_url, "include_subdomains": False, "ignore_query_parameters": True}
r = session.post(f"{API}/v1/map", json=body, headers=HEADERS, timeout=120)
r.raise_for_status()
return [(link["url"], None) for link in r.json()["links"]]
lastmod from a sitemap is a hint, not a guarantee. Some CMSs stamp every URL with the deploy time; others never update it. Treat it as a reason to check sooner, never as a reason to skip a check entirely.
URL normalisation
Without normalisation, https://Example.com/docs/?utm_source=x#top and https://example.com/docs become two rows and two fetches. Normalise before anything touches the state store.
from urllib.parse import urlsplit, urlunsplit, parse_qsl, urlencode
TRACKING = {"utm_source", "utm_medium", "utm_campaign", "utm_term", "utm_content",
"gclid", "fbclid", "ref", "mc_cid", "mc_eid"}
def normalize(url: str) -> str:
parts = urlsplit(url.strip())
scheme = parts.scheme.lower()
host = parts.netloc.lower()
if (scheme, host.rsplit(":", 1)[-1]) in {("http", "80"), ("https", "443")}:
host = host.rsplit(":", 1)[0]
query = [(k, v) for k, v in parse_qsl(parts.query, keep_blank_values=True)
if k.lower() not in TRACKING]
query.sort()
path = parts.path or "/"
if len(path) > 1 and path.endswith("/"):
path = path[:-1] # pick one convention and stick to it
return urlunsplit((scheme, host, path, urlencode(query), "")) # fragment dropped
The trailing-slash rule is a policy choice. Some frameworks serve different content at /docs and /docs/. If the target site is one of them, remove that line and let the two URLs live as separate rows.
The state store
One SQLite table carries everything the crawler needs to decide. The columns map to the signals discussed in the rest of this guide.
import sqlite3
import time
SCHEMA = """
CREATE TABLE IF NOT EXISTS pages (
normalized_url TEXT PRIMARY KEY,
url TEXT NOT NULL,
last_seen REAL,
last_fetched REAL,
next_check REAL DEFAULT 0,
interval_s INTEGER DEFAULT 86400,
content_hash TEXT,
etag TEXT,
last_modified TEXT,
sitemap_lastmod TEXT,
missing_runs INTEGER DEFAULT 0,
gone INTEGER DEFAULT 0
);
"""
def open_db(path: str = "crawl_state.sqlite") -> sqlite3.Connection:
db = sqlite3.connect(path)
db.executescript(SCHEMA)
return db
def upsert_discovered(db: sqlite3.Connection, found: list[tuple[str, str | None]]) -> None:
now = time.time()
for raw, lastmod in found:
n = normalize(raw)
row = db.execute("SELECT sitemap_lastmod FROM pages WHERE normalized_url=?", (n,)).fetchone()
if row is None:
db.execute("INSERT INTO pages (normalized_url,url,last_seen,sitemap_lastmod) VALUES (?,?,?,?)",
(n, raw, now, lastmod))
else:
bump = lastmod is not None and lastmod != row[0]
db.execute("UPDATE pages SET last_seen=?, missing_runs=0, sitemap_lastmod=?, "
"next_check=CASE WHEN ? THEN 0 ELSE next_check END WHERE normalized_url=?",
(now, lastmod, bump, n))
db.commit()
last_seen records the last discovery run that listed the URL, which is separate from last_fetched. A changed sitemap_lastmod resets next_check to zero so the page is picked up in the next change queue.
Conditional requests: ETag and If-Modified-Since
Where a site sends ETag or Last-Modified, a conditional request lets the origin answer 304 with no body. You send those headers through the headers field of /v1/extract and read the origin's status from metadata.http_status. The request still counts against your plan, but it is faster than a full render.
Not every site honours validators, and CDNs sometimes strip them. The code below sends them when present and falls through to hashing when the answer is 200.
Content hashing after stripping volatile blocks
A raw hash of the Markdown changes every run on pages that print the current time, a CSRF token or a "related articles" block. Strip those before hashing. Keep the stripping rules in one place so you can tune them per site.
import hashlib
import re
VOLATILE = [
re.compile(r"\b\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}(:\d{2})?\S*"), # ISO timestamps
re.compile(r"(?i)(csrf|nonce|session)[_-]?token[^\s]*"),
re.compile(r"(?i)^#+\s*related (posts|articles).*?(?=^#|\Z)", re.M | re.S),
re.compile(r"[ \t]+"),
]
def content_fingerprint(markdown: str) -> str:
text = markdown
for rx in VOLATILE:
text = rx.sub(" ", text)
text = "\n".join(line.strip() for line in text.splitlines() if line.strip())
return hashlib.sha256(text.encode("utf-8")).hexdigest()
def fetch_one(session: requests.Session, row: sqlite3.Row | tuple) -> dict:
"""Sync extract with conditional headers. Returns a dict the caller can store."""
normalized_url, url, etag, last_modified = row
cond = {}
if etag:
cond["If-None-Match"] = etag
if last_modified:
cond["If-Modified-Since"] = last_modified
body = {"url": url, "formats": ["markdown"], "render_js": False,
"processing_mode": "sync", "headers": cond or None, "timeout": 60000}
r = session.post(f"{API}/v1/extract", json=body, headers=HEADERS, timeout=90)
if r.status_code == 429:
raise RuntimeError(f"rate limited: {r.json().get('error', r.json())}")
r.raise_for_status()
payload = r.json()
status = payload["metadata"].get("http_status")
md = (payload.get("data") or {}).get("markdown") or ""
return {"http_status": status, "markdown": md,
"hash": content_fingerprint(md) if status == 200 else None}
Whether the origin's ETag and Last-Modified response headers are exposed in metadata depends on the API version you are on; check the response for your account before relying on them. If they are not exposed, the content hash still gives you a reliable changed/unchanged answer, you just lose the 304 shortcut.
The change queue and adaptive revisit
The queue is the heart of incremental web crawling, and it is a query, not a data structure: every row whose next_check is in the past and which is not marked gone. After each fetch the interval moves. Change found: halve it, floor one hour. No change: multiply by 1.5, ceiling 30 days. Over a few weeks the schedule converges on each page's observed change frequency.
MIN_S, MAX_S = 3600, 30 * 86400
def change_queue(db: sqlite3.Connection, limit: int = 500) -> list[tuple]:
return db.execute(
"SELECT normalized_url, url, etag, last_modified FROM pages "
"WHERE gone=0 AND next_check<=? ORDER BY next_check LIMIT ?",
(time.time(), limit)).fetchall()
def record_result(db: sqlite3.Connection, normalized_url: str, res: dict) -> str:
now = time.time()
old_hash, interval = db.execute(
"SELECT content_hash, interval_s FROM pages WHERE normalized_url=?",
(normalized_url,)).fetchone()
if res["http_status"] in (404, 410):
db.execute("UPDATE pages SET gone=1, last_fetched=? WHERE normalized_url=?", (now, normalized_url))
db.commit()
return "gone"
changed = res["http_status"] == 200 and res["hash"] != old_hash
interval = max(MIN_S, interval // 2) if changed else min(MAX_S, int(interval * 1.5))
db.execute("UPDATE pages SET last_fetched=?, next_check=?, interval_s=?, "
"content_hash=COALESCE(?, content_hash) WHERE normalized_url=?",
(now, now + interval, interval, res["hash"], normalized_url))
db.commit()
return "changed" if changed else "unchanged"
A 304 arrives with http_status == 304 and no hash, so it falls into the unchanged branch and the interval grows. Any 5xx or network error should leave the row untouched so it is retried on the next run; wrap fetch_one in a try/except in your runner for that.
Detecting deleted URLs
Two signals, used together. A direct 404 or 410 flips gone immediately in record_result. A URL that disappears from the sitemap is weaker evidence, because sitemaps lag and sometimes omit sections. Count consecutive discovery runs where it was absent and only force a confirming fetch after N of them.
def mark_missing(db: sqlite3.Connection, run_started: float, n_runs: int = 3) -> None:
"""Call after upsert_discovered. Rows not seen this run get missing_runs+1."""
db.execute("UPDATE pages SET missing_runs=missing_runs+1 WHERE last_seen<? AND gone=0", (run_started,))
db.execute("UPDATE pages SET next_check=0 WHERE missing_runs>=? AND gone=0", (n_runs,))
db.commit()
Rows marked gone stay in the table. That keeps a stale inbound link from reintroducing the URL as new. If the page comes back, a later discovery run will list it again; reset gone to 0 in upsert_discovered if you want that behaviour.
Fetching the changed set in bulk
For a few hundred URLs the sync loop above is fine, and on a mature incremental web crawling setup the changed set is usually that small. For thousands, submit the queue as one /v1/batch job and poll it. The batch body accepts the same formats, render_js, proxy, headers and wait_config fields as a single extract, applied to every URL, so conditional headers cannot vary per URL in a batch. Use batch for the plain content-hash path and keep the sync path for URLs where you hold an ETag.
def batch_fetch(session: requests.Session, urls: list[str], poll_s: int = 10) -> dict[str, dict]:
body = {"urls": urls, "formats": ["markdown"], "render_js": False, "ignore_invalid_urls": True}
job = session.post(f"{API}/v1/batch", json=body, headers=HEADERS, timeout=60).json()
while True:
st = session.get(f"{API}/v1/batch/{job['job_id']}", headers=HEADERS, timeout=60).json()
if st["status"] in ("completed", "failed", "cancelled"):
break
time.sleep(poll_s)
out = {}
for res in st.get("results", []):
if res["status"] != "completed":
continue
md = (res.get("data") or {}).get("markdown") or ""
out[res["url"]] = {"http_status": res["metadata"].get("http_status"),
"markdown": md, "hash": content_fingerprint(md)}
return out
The alternative is processing_mode: "async" on individual /v1/extract calls, which returns 202 with a job_id you poll at GET /v1/extract/{job_id}. That keeps per-URL headers but costs one request per URL. Either way, feed the results through record_result exactly as the sync path does.
Putting the run together
A single run is: discover, upsert, mark missing, drain the queue, record. Run it from cron or a scheduler at whatever cadence your freshness target needs; the adaptive intervals keep the fetch count low regardless of how often the runner fires.
def run(sitemap_url: str | None, site_url: str) -> None:
db = open_db()
session = requests.Session()
started = time.time()
found = urls_from_sitemap(sitemap_url, session) if sitemap_url else urls_from_map(site_url, session)
upsert_discovered(db, found)
mark_missing(db, started)
queue = change_queue(db)
with_etag = [r for r in queue if r[2] or r[3]]
plain = [r for r in queue if not (r[2] or r[3])]
for row in with_etag:
try:
record_result(db, row[0], fetch_one(session, row))
except Exception as exc: # leave row for next run
print("retry later:", row[1], exc)
if plain:
results = batch_fetch(session, [r[1] for r in plain])
for row in plain:
if row[1] in results:
record_result(db, row[0], results[row[1]])
This is the entire incremental web crawling module. The table below summarises which signal does what, so you can drop pieces you do not need.
| Signal | Cost | Reliability | What it does in this module |
|---|---|---|---|
Sitemap lastmod | one sitemap fetch | low to medium | resets next_check to 0 |
| ETag / Last-Modified | one request, no render | high where supported | 304 means unchanged |
| Content hash | one full fetch | high | the ground truth comparison |
| Adaptive interval | none | n/a | decides when to re-check |
| 404 / 410 | one request | high | marks gone |
| Absent from sitemap N runs | none | low | forces a confirming fetch |
Common mistakes
Hashing before stripping volatile content
A page that prints "Generated at 14:02:11" changes every fetch, so your crawler halves its interval until it hits the floor and refetches hourly. Look at a diff of two consecutive fetches for any page that reports change on every run and add the pattern to VOLATILE.
Normalising after the first insert
If the first run stores raw URLs and normalisation is added later, every row is a duplicate waiting to happen. Normalise from day one and make normalized_url the primary key, as the schema does.
Trusting sitemap absence alone
Sitemaps lag deploys, split across index files, and sometimes drop whole sections after a CMS change. Marking a page gone because it vanished from one sitemap fetch will delete live content from your index. Require a confirming 404 or 410.
Running batch with per-URL conditional headers
/v1/batch applies one headers object to every URL in the job. Sending an If-None-Match meant for one page to a thousand others produces meaningless results. Route ETag-bearing rows through the sync path and the rest through batch, as run() does.
Next step
The module above keeps state, compares content and schedules revisits, and it needs a fetch layer that returns clean Markdown with an honest http_status for every URL, including the 304s and 404s it relies on. The Geonode Scraper API provides that layer: /v1/extract for single pages with per-request headers, /v1/batch for the bulk changed set, and /v1/map when there is no sitemap. The free tier covers 1.5K requests a month, enough to prove the incremental loop on a real site before you size a plan. See the Scraper API page for plans, and pricing for the request tiers.
FAQ
What is incremental web crawling?
Incremental web crawling means revisiting only the pages that are likely to have changed since your last run, instead of refetching an entire site every time. The crawler keeps a state store with a fingerprint per URL, discovers candidates cheaply (sitemap lastmod, ETag, Last-Modified), and spends fetch budget on the subset that actually needs it.
How do I know which pages changed without downloading them?
You usually cannot know for certain without a request, but you can rank candidates. Sitemap lastmod is the cheapest signal. A conditional request with If-None-Match or If-Modified-Since returns 304 on sites that support validators. For everything else, fetch and compare a content hash computed after stripping volatile blocks like timestamps and session tokens.
Does the Geonode Scraper API handle state and change detection for me?
No. The Scraper API fetches pages and returns Markdown or HTML plus metadata such as http_status. Your script owns the state store, the URL normalisation, the hashing and the scheduling. That split is deliberate: the API stays stateless and cheap to call, while your incremental web crawling logic lives in code you control and can test.
How should I schedule revisits for incremental web crawling?
Start with a default interval, then adapt per URL. If a fetch finds a change, halve the interval down to a floor; if it finds no change, multiply it up to a ceiling. Pages that churn get checked often, static pages drift toward the ceiling. Store next_check in the state table and query for rows whose next_check has passed.
How do I detect pages that were deleted?
Use two signals together. A 404 or 410 on fetch is a strong signal. Absence from the sitemap on N consecutive runs is a weaker one, because sitemaps are often incomplete. Mark the row as gone only when a direct fetch confirms it, and keep the row so you do not rediscover and refetch the URL from stale links.
What are the limits and compliance rules I should respect?
Respect robots.txt and the site's terms of service, keep request rates modest, and never bypass authentication. On the Scraper API side, the free tier covers 1.5K requests per month and paid plans run from 2 to 100 threads, so size your changed set to your plan. Conditional requests still count as requests.