Website change detection is how you find out that a vendor rewrote a terms-of-service clause, deprecated an endpoint in their docs, or quietly edited a privacy policy before anyone announces it. By the end of this guide you will have a Python script that snapshots a list of pages as Markdown through the Geonode Scraper API, strips the noise, stores every version in SQLite, produces a readable diff, and posts to Slack only when the content really changed. It runs from cron or GitHub Actions and survives flaky fetches.
This is deliberately not a price monitor. Prices are numbers in a known slot; documents are prose where the location and shape of a change are unknown in advance. The design choices below follow from that.
What you are monitoring and why Markdown
The pages worth watching share a profile: public, text-heavy, infrequently updated, and consequential when they do update. API reference pages, authentication guides, rate-limit tables, terms and conditions, data-processing agreements, acceptable-use policies, changelogs and status-page incident histories all qualify.
Comparing raw HTML of these pages is painful. Build hashes in asset URLs, CSRF tokens, A/B test class names and injected analytics change on every load. Converting to Markdown first discards nearly all of that and leaves the sentences. The /v1/extract endpoint returns Markdown directly, so the script never touches an HTML parser.
Install the one dependency with pip install requests. Everything else is standard library.
Taking a snapshot
The fetcher asks for Markdown, keeps JavaScript rendering off by default, and turns it on per page for docs sites that render client-side. The wait_config block is only needed when a page populates its content after load; it auto-enables rendering when present.
import os
import requests
API = "https://scraper.geonode.io"
HEADERS = {"X-Api-Key": os.environ["GEONODE_API_KEY"]}
def snapshot(session: requests.Session, url: str, render_js: bool = False,
wait_for: str | None = None) -> tuple[int, str]:
"""Return (origin_http_status, markdown). Raises on API-level errors."""
body = {"url": url, "formats": ["markdown"], "render_js": render_js,
"processing_mode": "sync", "timeout": 60000}
if wait_for:
body["wait_config"] = {"wait_until": "domcontentloaded", "wait_for": wait_for,
"wait_timeout": 2000}
body.pop("render_js") # wait_config implies rendering; sending false is rejected
r = session.post(f"{API}/v1/extract", json=body, headers=HEADERS, timeout=90)
if r.status_code == 429:
err = r.json().get("error", r.json())
raise RuntimeError(f"rate limited ({err.get('code')}): {err.get('message')}")
r.raise_for_status()
payload = r.json()
return payload["metadata"]["http_status"], (payload.get("data") or {}).get("markdown") or ""
The origin status matters as much as the body. A 503 maintenance page converts to perfectly valid Markdown; without checking http_status you would store it as a new version and alert on it.
Stripping noise
Two mechanisms, applied in order. A regex pass removes lines that are known to churn: dates, "last updated" banners, cookie notices, copyright footers, "edit this page" links. Then a section allowlist keeps only the headings you care about, which drops navigation, sidebars and marketing blocks that the Markdown converter kept because they were real text.
import re
NOISE = [
re.compile(r"(?im)^.*\b(last (updated|modified|reviewed)|effective date|updated on)\b.*$"),
re.compile(r"(?im)^.*\b(cookie|cookies)\b.*\b(accept|consent|preferences)\b.*$"),
re.compile(r"(?im)^.*(©|copyright|all rights reserved).*$"),
re.compile(r"(?im)^.*\b(edit this page|was this page helpful|on this page)\b.*$"),
re.compile(r"\b(?:\d{1,2}\s)?(?:jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)[a-z]*\.?\s+\d{1,2},?\s+\d{4}\b", re.I),
re.compile(r"\b\d{4}-\d{2}-\d{2}(?:[T ]\d{2}:\d{2}(?::\d{2})?\S*)?\b"),
re.compile(r"!\[[^\]]*\]\([^)]*\)"), # images
]
def strip_noise(md: str, keep_sections: list[str] | None = None) -> str:
for rx in NOISE:
md = rx.sub("", md)
if keep_sections:
kept, keep = [], False
for line in md.splitlines():
if line.startswith("#"):
title = line.lstrip("#").strip().lower()
keep = any(s.lower() in title for s in keep_sections)
if keep:
kept.append(line)
md = "\n".join(kept)
return normalize_ws(md)
def normalize_ws(text: str) -> str:
lines = (re.sub(r"[ \t]+", " ", ln).strip() for ln in text.splitlines())
return "\n".join(ln for ln in lines if ln)
keep_sections is a list of substrings matched against headings, so ["rate limit", "authentication"] keeps those H2/H3 blocks and everything under them until the next heading. Leave it None for policy pages where any clause matters.
The date regexes are deliberately aggressive. A policy page's "Effective 1 March 2026" line is exactly the kind of change you want to see, so if that matters to you, remove the first NOISE entry and let the diff show it.
Hashing versus diffing
Hashing answers "did anything change" in constant time. Diffing answers "what changed" in a form a person can read in a Slack message. Most website change detection tools stop at the hash; use both, hash to gate, diff to report.
import difflib
import hashlib
def digest(text: str) -> str:
return hashlib.sha256(text.encode("utf-8")).hexdigest()
def unified(old: str, new: str, url: str, context: int = 2, max_lines: int = 120) -> str:
diff = difflib.unified_diff(old.splitlines(), new.splitlines(),
fromfile=f"{url} (previous)", tofile=f"{url} (current)",
lineterm="", n=context)
lines = list(diff)
if len(lines) > max_lines:
lines = lines[:max_lines] + [f"... {len(lines) - max_lines} more lines"]
return "\n".join(lines)
difflib.unified_diff is in the standard library and documented at docs.python.org. The n argument sets context lines; two is enough for prose. Cap the output so a full-page rewrite does not produce a 5,000-line alert.
Storing snapshots
SQLite keeps full history, not just the last version. That makes it possible to compare against the last two snapshots (next section) and to answer "when did this clause first appear" months later.
import sqlite3
import time
SCHEMA = """
CREATE TABLE IF NOT EXISTS snapshots (
id INTEGER PRIMARY KEY,
url TEXT NOT NULL,
taken_at REAL NOT NULL,
http_status INTEGER,
hash TEXT,
content TEXT
);
CREATE INDEX IF NOT EXISTS idx_url_time ON snapshots(url, taken_at DESC);
"""
def open_db(path: str = "changes.sqlite") -> sqlite3.Connection:
db = sqlite3.connect(path)
db.executescript(SCHEMA)
return db
def last_snapshots(db: sqlite3.Connection, url: str, n: int = 2) -> list[tuple[str, str]]:
"""Most recent first: [(hash, content), ...] for successful (200) fetches only."""
return db.execute("SELECT hash, content FROM snapshots WHERE url=? AND http_status=200 "
"ORDER BY taken_at DESC LIMIT ?", (url, n)).fetchall()
def save(db: sqlite3.Connection, url: str, status: int, h: str | None, content: str | None) -> None:
db.execute("INSERT INTO snapshots (url, taken_at, http_status, hash, content) VALUES (?,?,?,?,?)",
(url, time.time(), status, h, content))
db.commit()
Policy pages are small, so storing full content per run is cheap. If you monitor hundreds of long docs hourly, store content only when the hash changed and keep the hash row otherwise.
Handling flaky fetches
Flaky fetches are the main source of false positives in website change detection, so there are three defences. Retry the fetch with backoff before giving up. Treat anything but a 200 as "no data", store the status for the record, and do not compare. And compare the new hash against the last two good snapshots, not just the last one: if the page flipped A, B, A, the B was most likely a transient variant (an error page that still returned 200, a cache serving a stale build) and does not deserve an alert.
def fetch_with_retry(session: requests.Session, url: str, attempts: int = 3, **kw) -> tuple[int, str]:
delay = 5
for i in range(attempts):
try:
status, md = snapshot(session, url, **kw)
if status < 500 and md.strip():
return status, md
except (requests.RequestException, RuntimeError) as exc:
if i == attempts - 1:
raise
time.sleep(delay)
delay *= 3
return status, md
def check(db: sqlite3.Connection, session: requests.Session, url: str,
keep_sections: list[str] | None = None, **kw) -> str | None:
"""Returns a unified diff if a real change was detected, else None."""
status, raw = fetch_with_retry(session, url, **kw)
if status != 200:
save(db, url, status, None, None)
return None
clean = strip_noise(raw, keep_sections)
h = digest(clean)
previous = last_snapshots(db, url, n=2)
save(db, url, status, h, clean)
if not previous:
return None # first run, nothing to compare
if any(h == old_hash for old_hash, _ in previous):
return None # unchanged, or a revert of a blip
return unified(previous[0][1], clean, url)
The second-snapshot rule has one cost: a genuine change that is reverted within one run is not reported. For documentation and policy monitoring that is almost always the right trade.
Notifying
Slack incoming webhooks accept a JSON body with a text field; wrap the diff in a code block so it stays monospaced. Email through smtplib is a few more lines and useful where Slack is not available.
import json
import smtplib
from email.message import EmailMessage
SLACK_WEBHOOK = os.environ.get("SLACK_WEBHOOK_URL")
def notify_slack(url: str, diff: str) -> None:
if not SLACK_WEBHOOK:
return
fence = "`" * 3 # literal ``` would close the Markdown block
text = f"*Change detected:* {url}\n{fence}\n{diff[:3500]}\n{fence}"
requests.post(SLACK_WEBHOOK, data=json.dumps({"text": text}),
headers={"Content-Type": "application/json"}, timeout=15).raise_for_status()
def notify_email(url: str, diff: str, to: str) -> None:
msg = EmailMessage()
msg["Subject"] = f"Change detected: {url}"
msg["From"] = os.environ["SMTP_FROM"]
msg["To"] = to
msg.set_content(diff)
with smtplib.SMTP(os.environ["SMTP_HOST"], int(os.environ.get("SMTP_PORT", "587"))) as s:
s.starttls()
s.login(os.environ["SMTP_USER"], os.environ["SMTP_PASS"])
s.send_message(msg)
Slack truncates long messages, hence the 3,500-character cap on the diff. For long diffs, post the first screen and attach the full text elsewhere, or point the message at a stored snapshot ID.
The watchlist and the runner
Configuration lives in a list so you can keep it in the repo and review changes to it like code. Each entry carries the URL, whether rendering is needed, an optional selector to wait for, and an optional section allowlist.
WATCHLIST = [
{"url": "https://example.com/legal/terms"},
{"url": "https://example.com/legal/privacy"},
{"url": "https://example.com/docs/api/authentication",
"keep_sections": ["authentication", "rate limit"]},
{"url": "https://example.com/changelog", "render_js": True, "wait_for": "main article"},
]
def main() -> None:
db = open_db()
session = requests.Session()
for item in WATCHLIST:
url = item["url"]
try:
diff = check(db, session, url, keep_sections=item.get("keep_sections"),
render_js=item.get("render_js", False), wait_for=item.get("wait_for"))
except Exception as exc:
print(f"[skip] {url}: {exc}")
continue
if diff:
print(f"[changed] {url}")
notify_slack(url, diff)
else:
print(f"[ok] {url}")
if __name__ == "__main__":
main()
The first run seeds the database and reports nothing. From the second run on, every real change produces one Slack message carrying the diff.
Scheduling website change detection
Two options that need no infrastructure. A cron line on any machine that has Python and the environment variables:
# daily at 06:10, log to a file
10 6 * * * cd /opt/change-monitor && /usr/bin/python3 monitor.py >> monitor.log 2>&1
Or a GitHub Actions workflow, which is convenient because the SQLite file can be committed back to the repo as the state store and the secrets live in the repository settings:
name: change-monitor
on:
schedule:
- cron: "10 6 * * *"
workflow_dispatch:
jobs:
run:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: pip install requests
- run: python monitor.py
env:
GEONODE_API_KEY: ${{ secrets.GEONODE_API_KEY }}
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
- run: |
git config user.name "change-monitor"
git config user.email "bot@users.noreply.github.com"
git add changes.sqlite && git commit -m "snapshot $(date -u +%F)" || true
git push
Hourly schedules are fine for changelogs; for legal pages, daily is already more than most teams need. The table below gives a reasonable starting point.
| Page type | Suggested cadence | render_js | keep_sections |
|---|
| Terms, privacy, AUP | daily | no | none (whole page) |
| API reference | every 6 h | depends on docs engine | yes, the endpoints you use |
| Changelog / release notes | hourly | often | none |
| Status page incident history | hourly | usually | none |
A note on terms of service and robots
Reading a public documentation or policy page once a day is ordinary use of the web, and it is the same thing a browser does. Still, check the site's terms and robots.txt before adding it to the watchlist, keep the cadence proportionate, and never monitor pages behind a login you were not granted. If a site publishes an RSS feed or a changelog API, prefer it: it is cheaper for both sides and it will not break when the page layout changes.
Common mistakes
Comparing raw HTML
Every build of a modern docs site rewrites asset hashes and class names. Diffs become unreadable and every deploy fires an alert. Convert to Markdown before anything else, as snapshot() does.
Alerting on the first non-200
A single 503 or a bot-challenge page returning 200 with "Please verify you are human" should not become a stored version or an alert. Check http_status, require non-empty content, and compare against the last two good snapshots.
Stripping too little or too much
If your diff shows only a date, add it to NOISE. If a clause change was missed because keep_sections dropped the heading it lived under, widen the allowlist. Run the monitor for a week and tune the two lists against the diffs it produces before trusting it.
Letting the watchlist drift from reality
A URL that redirects to a new path, or a docs page that moved under a versioned prefix, silently turns into a 404 or a different page. Log non-200 statuses loudly and review the "skip" lines in the output every so often.
Next step
Everything above is standard-library Python plus one HTTP call per page, which is all website change detection for documents needs. The part that is hard to self-host is the fetch itself: a clean Markdown conversion that handles JavaScript-rendered docs sites, returns the origin's real http_status, and does not get blocked when you check forty vendor pages every morning. The Geonode Scraper API provides exactly that call, with render_js and wait_config for client-rendered pages and a free tier of 1.5K requests a month that comfortably covers a daily watchlist. See the Scraper API page for plans and the web scraping use case for related patterns.
FAQ
What is website change detection and when is it useful?
Website change detection is the practice of fetching a page on a schedule, comparing it to the previous snapshot, and alerting when the meaningful content differs. It is most useful for pages you depend on but do not control: API documentation, terms of service, privacy policies, pricing terms and vendor changelogs, where a silent edit can break an integration or a compliance assumption.
Why compare Markdown instead of raw HTML?
HTML carries layout, scripts, tracking parameters and build hashes that change without the text changing. Converting to Markdown first collapses the page to its content, so a diff shows the sentence that was edited rather than a hundred lines of attribute churn. The Geonode Scraper API returns Markdown directly, which keeps the pipeline short.
How do I avoid false alerts from dates, cookie banners and navigation?
Strip known noise with a small set of regular expressions (dates, 'last updated' lines, cookie notices, footers), then keep only the sections you care about using a heading allowlist. Normalise whitespace before hashing. Finally, compare against the last two snapshots so a transient error page does not fire an alert on its own.
Should I hash the page or diff it?
Both. A hash is a fast equality check that decides whether to do anything at all. A unified diff from difflib is what a human reads in the alert. Hash first, diff only when the hash differs, and send the diff with the notification so the reader can judge the change without opening the page.
How often should website change detection run?
Match the cadence to the page's stakes and churn. Legal and policy pages rarely change more than a few times a year, so daily is plenty. Active API docs and changelogs justify hourly checks. Run it from cron or a GitHub Actions schedule, and keep the interval long enough that a flaky fetch is retried before it is reported.
What are the compliance and limit considerations?
Monitoring public documentation and policy pages at a modest cadence is a normal use of the web, but read the site's terms of service and robots.txt and respect them, and do not monitor pages behind authentication you are not entitled to. On the API side, the Scraper API free tier covers 1.5K requests per month; a daily check of 40 pages fits inside that.
