Scraper API Webhooks vs Polling: Building a Reliable Results Pipeline

Async scraping jobs finish on their own schedule, and the question is how your code finds out. This guide compares web scraping webhooks with status polling, then builds the pieces you need for either: a polling loop with backoff and a hard cap, a FastAPI receiver that verifies signatures and enqueues work, an idempotency table, replay from the delivery log and a fallback poller that catches anything a webhook missed. By the end you will have a results pipeline that does not lose jobs or process them twice.

Sync vs async extraction

POST /v1/extract on the Geonode Scraper API runs in one of two modes. With processing_mode: "sync" the request blocks and returns 200 with data.markdown or data.html, a metadata object (url, http_status, duration_ms) and tokens_charged. With processing_mode: "async" it returns 202 with job_id, status: "queued", status_url and estimated_tokens, and the job runs in the background.

Sync is right when you need the answer inline: an agent tool call, a request handler, a one-off script. Async is right when the page needs JavaScript rendering and a long wait, when you are submitting many URLs, or when the caller should not hold a connection open. Batch (POST /v1/batch) and crawl (POST /v1/crawl) are always jobs. The moment you have a job_id, you have to choose between polling and web scraping webhooks.

import os
import httpx

BASE = "https://scraper.geonode.io"
HEADERS = {"X-Api-Key": os.environ["GEONODE_SCRAPER_API_KEY"]}

def submit(url: str) -> str:
    r = httpx.post(
        f"{BASE}/v1/extract",
        headers=HEADERS,
        json={
            "url": url,
            "formats": ["markdown"],
            "render_js": True,
            "processing_mode": "async",
        },
        timeout=30.0,
    )
    r.raise_for_status()  # 202 on success
    return r.json()["job_id"]

Polling: fine for a few jobs and expensive for many

Polling means calling GET /v1/extract/{job_id} until status is completed or failed. The job status guide lists four states: queued, processing, completed, failed. The mistake most people make is a fixed one-second sleep. It hammers the API while the job is queued, and with a 429 rate limit (error.code "RATE_LIMITED") the poll loop starts failing before the job does.

Use exponential backoff with full jitter and a hard cap on total wait. Jitter spreads concurrent pollers out so they stop lining up on the same second.

import os
import random
import time
import httpx

BASE = "https://scraper.geonode.io"
HEADERS = {"X-Api-Key": os.environ["GEONODE_SCRAPER_API_KEY"]}

class JobTimeout(Exception):
    pass

def poll_job(job_id: str, max_wait: float = 300.0) -> dict:
    deadline = time.monotonic() + max_wait
    delay = 1.0
    while True:
        r = httpx.get(f"{BASE}/v1/extract/{job_id}", headers=HEADERS, timeout=15.0)
        if r.status_code == 429:
            retry_after = float(r.headers.get("Retry-After", delay))
            time.sleep(retry_after)
            continue
        r.raise_for_status()
        job = r.json()
        if job["status"] in ("completed", "failed"):
            return job
        if time.monotonic() >= deadline:
            raise JobTimeout(job_id)
        # full jitter: sleep somewhere in [0, delay], then grow delay up to 30 s
        time.sleep(random.uniform(0, delay))
        delay = min(delay * 2, 30.0)

This is fine for a script that submits ten URLs and wrong for a nightly batch of twenty thousand. Two hundred jobs in flight polling every few seconds is thousands of empty status calls per minute, and latency is still bounded by the poll interval.

When polling wins and when webhooks win

Webhooks move complexity from the client loop into your infrastructure. If you do not already run a public endpoint and a queue, polling is the correct first version. If you do, web scraping webhooks are cheaper, faster and scale with completions instead of seconds.

Registering a webhook

The webhook endpoints are listed in the API reference. POST /v1/webhooks takes url (required, a URI), event_type (required, one of extract_completed, batch_completed, crawl_completed) and an optional description. The 201 response returns the webhook id, url, description, is_active, event_type, timestamps and a secret. The secret is returned on creation and by POST /v1/webhooks/{webhook_id}/rotate-secret; store it in a secrets manager at that moment, because it is what you will use to verify every delivery.

Note that at the time of writing the docs mark the webhook endpoints as documented but not yet available in production. The contract below follows the published reference; verify it before you deploy.

import os
import httpx

BASE = "https://scraper.geonode.io"
HEADERS = {"X-Api-Key": os.environ["GEONODE_SCRAPER_API_KEY"]}

def register_webhook(receiver_url: str, event_type: str = "extract_completed") -> dict:
    r = httpx.post(
        f"{BASE}/v1/webhooks",
        headers=HEADERS,
        json={
            "url": receiver_url,
            "event_type": event_type,
            "description": "results pipeline, prod",
        },
        timeout=15.0,
    )
    r.raise_for_status()  # 201
    hook = r.json()
    # hook["secret"] is only shown here and after rotate-secret: persist it now
    return hook

One webhook subscribes to one event_type, so a pipeline that uses extraction, batch and crawl registers three.

A minimal receiver that verifies and enqueues

A webhook receiver has exactly three jobs: verify the signature, return 2xx as fast as possible, and hand the event to a queue. Anything slower than a few hundred milliseconds in the request handler is a design error, because the platform's delivery attempt may time out and you will see retries for work you already did.

The reference does not yet publish the signature header name, the hashing scheme or the payload schema, so the receiver below keeps those configurable and verifies an HMAC-SHA256 over the raw body, which is the common pattern. Confirm the exact header and algorithm in the docs and set them via environment variables.

import hashlib
import hmac
import json
import os
import sqlite3

from fastapi import FastAPI, HTTPException, Request

app = FastAPI()

WEBHOOK_SECRET = os.environ["GEONODE_WEBHOOK_SECRET"].encode()
SIGNATURE_HEADER = os.environ.get("GEONODE_SIGNATURE_HEADER", "X-Webhook-Signature")  # confirm header name in docs
DB = sqlite3.connect("events.db", check_same_thread=False)
DB.execute(
    "CREATE TABLE IF NOT EXISTS processed_events ("
    "  job_id TEXT PRIMARY KEY,"
    "  event_type TEXT NOT NULL,"
    "  delivery_id TEXT,"
    "  received_at TEXT DEFAULT CURRENT_TIMESTAMP"
    ")"
)

def verify_signature(raw_body: bytes, header_value: str | None) -> bool:
    if not header_value:
        return False
    expected = hmac.new(WEBHOOK_SECRET, raw_body, hashlib.sha256).hexdigest()  # confirm algorithm in docs
    return hmac.compare_digest(expected, header_value.strip())

def enqueue(event: dict) -> None:
    # Replace with your queue client (Redis, SQS, Celery, etc.).
    print("enqueued", event.get("job_id"))

@app.post("/webhooks/geonode")
async def receive(request: Request):
    raw = await request.body()
    if not verify_signature(raw, request.headers.get(SIGNATURE_HEADER)):
        raise HTTPException(status_code=401, detail="bad signature")

    event = json.loads(raw)
    job_id = event.get("job_id")  # confirm field name in docs
    event_type = event.get("event_type")  # confirm field name in docs
    delivery_id = event.get("id")  # confirm field name in docs
    if not job_id:
        raise HTTPException(status_code=400, detail="missing job_id")

    try:
        DB.execute(
            "INSERT INTO processed_events (job_id, event_type, delivery_id) VALUES (?, ?, ?)",
            (job_id, event_type, delivery_id),
        )
        DB.commit()
    except sqlite3.IntegrityError:
        # Duplicate delivery: already recorded, acknowledge and do nothing.
        return {"status": "duplicate"}

    enqueue({"job_id": job_id, "event_type": event_type})
    return {"status": "accepted"}

Three details matter. The signature is computed over the raw bytes, not a re-serialised JSON object, because any whitespace change breaks it. The insert into processed_events happens before the enqueue, so a crash between the two leaves a row with no work, which the fallback poller fixes later. And the handler returns 2xx on a duplicate rather than an error, because an error tells the platform to retry what it has already delivered.

Idempotency and out-of-order deliveries

Any webhook system is at-least-once. Network blips, a slow worker, a rolling deploy that drops a response: all of these produce a second delivery for the same job. The processed_events table with job_id as primary key is the whole fix. A unique constraint makes the duplicate check atomic, which a SELECT followed by an INSERT does not.

Out-of-order is a different problem. A batch_completed event may arrive before you have finished recording the batch submission, or two events for related jobs may land in reverse order. Do not make the worker depend on order. The worker's only input should be a job_id; it fetches the job's current state from the API and acts on that, which is always the truth regardless of when the event arrived.

import os
import httpx

BASE = "https://scraper.geonode.io"
HEADERS = {"X-Api-Key": os.environ["GEONODE_SCRAPER_API_KEY"]}

def store_result(job_id: str, job: dict) -> None: ...   # write to your store
def resubmit(job_id: str) -> None: ...                  # re-POST the original URL
def dead_letter(job_id: str, err: dict) -> None: ...    # park for manual review

def handle_job(job_id: str) -> None:
    r = httpx.get(f"{BASE}/v1/extract/{job_id}", headers=HEADERS, timeout=15.0)
    r.raise_for_status()
    job = r.json()
    if job["status"] == "completed":
        store_result(job_id, job)
    elif job["status"] == "failed":
        err = job.get("error", {})  # confirm field name in docs
        if err.get("retryable"):
            resubmit(job_id)
        else:
            dead_letter(job_id, err)
    else:
        # Event arrived before the job settled; let the fallback poller retry.
        pass

Replay from the delivery log

GET /v1/webhooks/{webhook_id}/deliveries lists every attempt the platform made. Each item carries id, event_type, job_id, attempt_number, status, http_status, delivered_at and created_at, with page, page_size (max 100) and an optional status filter as query parameters. The possible status values are not enumerated in the reference, so filter on http_status instead when you want to find deliveries your endpoint rejected.

import os
import httpx

BASE = "https://scraper.geonode.io"
HEADERS = {"X-Api-Key": os.environ["GEONODE_SCRAPER_API_KEY"]}

def replay_failed(webhook_id: str) -> list[str]:
    missed: list[str] = []
    page = 1
    while True:
        r = httpx.get(
            f"{BASE}/v1/webhooks/{webhook_id}/deliveries",
            headers=HEADERS,
            params={"page": page, "page_size": 100},
            timeout=15.0,
        )
        r.raise_for_status()
        body = r.json()
        for d in body["items"]:
            ok = d.get("http_status") is not None and 200 <= d["http_status"] < 300
            if not ok and d["job_id"] not in missed:
                missed.append(d["job_id"])
        if page >= body["page_count"]:
            break
        page += 1
    for job_id in missed:
        handle_job(job_id)
    return missed

Because handle_job goes through the same idempotency table, replaying a job that did in fact succeed is harmless.

The fallback poller

The delivery log tells you about deliveries the platform attempted. It cannot tell you about events that were never generated, or a webhook that was deactivated. The last line of defence is a slow reconciliation loop: every few minutes, list jobs with GET /v1/extract/jobs, and for any job older than your threshold that has no row in processed_events, process it directly.

import os
import httpx
from datetime import datetime, timedelta, timezone

BASE = "https://scraper.geonode.io"
HEADERS = {"X-Api-Key": os.environ["GEONODE_SCRAPER_API_KEY"]}

def reconcile(older_than: timedelta = timedelta(minutes=10)) -> None:
    r = httpx.get(f"{BASE}/v1/extract/jobs", headers=HEADERS, timeout=30.0)
    r.raise_for_status()
    cutoff = datetime.now(timezone.utc) - older_than
    for job in r.json().get("items", []):  # confirm field name in docs
        created = datetime.fromisoformat(job["created_at"].replace("Z", "+00:00"))  # confirm field name in docs
        if created > cutoff or job["status"] not in ("completed", "failed"):
            continue
        seen = DB.execute(
            "SELECT 1 FROM processed_events WHERE job_id = ?", (job["job_id"],)
        ).fetchone()
        if seen is None:
            DB.execute(
                "INSERT INTO processed_events (job_id, event_type) VALUES (?, ?)",
                (job["job_id"], "reconciled"),
            )
            DB.commit()
            handle_job(job["job_id"])

Run this at a cadence that is slow enough to be cheap and fast enough that a lost webhook does not become a data gap your users notice. Ten minutes is a sensible starting point.

Secret rotation

Rotate the secret on a schedule and immediately if it leaks. POST /v1/webhooks/{webhook_id}/rotate-secret returns a new secret and, per the docs, invalidates the previous one. That means there is a window between the API switching and your receiver reloading its config in which deliveries will fail verification. Keep the window short: have the receiver read the secret from a source it can refresh without a restart, rotate, update the source, then check the delivery log for anything that landed with a 401 and replay it.

Common mistakes

Doing the work inside the request handler

Fetching the result, parsing it and writing to the database inside the webhook handler is the most common failure. The platform's delivery times out, it retries, and now you have duplicates and a slow endpoint. Acknowledge and enqueue, nothing more.

Checking for duplicates without a unique constraint

A SELECT then INSERT has a race between two concurrent deliveries of the same job. Make job_id the primary key and let the database reject the second insert.

Trusting the event payload over the job status

Treat the webhook as a notification, not as data. The worker should always re-fetch GET /v1/extract/{job_id} and act on the real status, which also makes replay and reconciliation trivial.

Running webhooks with no reconciliation

Web scraping webhooks are at-least-once, not exactly-once, and they are not guaranteed to be at-least-once across every outage you will have. Without a fallback poller you will eventually discover a gap with no way to know how big it is.

Next step

Everything above runs against the Geonode Scraper API: async extraction with processing_mode: "async", batch and crawl jobs, and webhook subscriptions with rotatable secrets and a queryable delivery log. There is a free tier of 1.5K requests per month to build and test the receiver against, and pay-as-you-go from $3.50 per 10K requests after that. Start with polling, add webhooks when the job count justifies it, and keep the reconciler running either way.

FAQ

What are web scraping webhooks and when should I use them?

Web scraping webhooks are HTTP callbacks the scraper platform sends to your server when an asynchronous job finishes, so you do not have to keep asking for its status. Use them when you run batch or crawl jobs, or more than a few dozen async extractions at once. For a handful of short jobs, polling is simpler and good enough.

Is polling a scraper API bad practice?

No. Polling is the right choice for small volumes, scripts, notebooks and anything where a few seconds of latency does not matter. It becomes a problem when you have hundreds of in-flight jobs, because each one costs a request per poll interval and you still cannot react faster than that interval. That is where web scraping webhooks pay off.

How do I verify that a webhook really came from the scraper platform?

Each Geonode webhook returns a signing secret when it is created and the secret can be rotated. The receiver recomputes a signature over the raw request body with that secret and compares it in constant time to the header the platform sends. Reject anything that fails the check before parsing the payload, and never log the secret.

What happens if my webhook endpoint is down when a job completes?

The platform records a delivery attempt that you can inspect through GET /v1/webhooks/{webhook_id}/deliveries, which lists attempt_number, status and http_status per attempt. Your pipeline should not depend on redelivery alone. Run a fallback poller that reconciles any job older than a threshold with no processed event, and fetch its result directly.

Are Geonode Scraper API webhooks generally available?

At the time of writing, the webhook endpoints are documented on docs.geonode.com but marked as not yet available in production, and the signature header, payload schema and retry schedule are not published. Build the receiver against the documented contract, keep header and field names configurable, and verify them against the docs before you ship.

Do I need both web scraping webhooks and polling?

In production, yes. Webhooks give you low latency and low request volume. A slow reconciliation poller catches jobs whose delivery was lost, your endpoint rejected, or your queue dropped. Together they give you at-least-once processing, and an idempotency table keyed on job_id turns that into effectively-once.