Image dataset scraping proxies: public media for computer vision

Quick answer

A computer-vision dataset has two phases with different economics. Discovery (crawling galleries, listings and search results to find media URLs and metadata) is low-bandwidth and needs residential IPs so pages render fully. Downloading the files is high-bandwidth and should run through rotating datacenter proxies at $0.14/GB. Splitting the phases is what keeps a multi-terabyte collection affordable.

Why image and video dataset collection breaks without the right proxies

Media hosting is where bandwidth bills explode. A single video can be hundreds of megabytes, and a dataset of a million images runs into terabytes. Pulling all of that through residential IPs priced per gigabyte makes the proxy cost larger than the storage cost.

Discovery pages are the opposite problem. Image galleries, stock libraries and product listings lazy-load thumbnails and paginate with infinite scroll. Fetched from a datacenter range, they serve HTTP 429, CAPTCHAs or a shortened grid that hides most of the items. Without JavaScript rendering, you capture placeholder URLs instead of real media links.

Metadata lives in a third place. Alt text, captions, EXIF-derived fields, licence tags and uploader location are what make a dataset usable for training, and they are usually on the page rather than in the file. Collecting files without the page context gives you a pile of unlabelled bytes.

Region also matters: some galleries filter visible content by country, and licensing displayed to a visitor in one jurisdiction may differ elsewhere.

Which Geonode proxy type fits

The recommended product for bulk media downloads is rotating datacenter proxies at $0.14/GB. CDN-hosted media files rarely apply the defences that discovery pages do, and datacenter bandwidth is roughly half the cost of residential at volume. Route every file download through the datacenter endpoint by appending -type-datacenter to the username.

Use residential proxies for discovery. The pool of 2.5M IPs online across 195+ countries renders galleries and listings as a consumer would see them, with country and city targeting for region-filtered sources. Discovery traffic is small (HTML and JSON, not media), so residential pricing from $0.27/GB at volume is a minor line item.

The Scraper API with JavaScript rendering and JSON output fits discovery for sites that build their grids client-side. It returns the rendered page with real image URLs and captions, which you then hand to the datacenter download queue.

Residential — Best for: Discovery, galleries, region-filtered sources; IP source: Real ISP devices, 2.5M online; Session: Rotating or sticky up to 24h; Price from: $0.27/GB at volume
ISP (static residential) — Best for: Repeated polling of a fixed source list; IP source: Static IPs from real ISPs; Session: Unlimited sessions; Price from: $1.25/IP
Datacenter — Best for: Bulk image and video file downloads; IP source: Datacenter ranges; Session: Rotating; Price from: $0.14/GB

How to set it up

Define the dataset: classes or categories, target count, licence constraints, and minimum resolution. Record which sources permit automated collection under their terms.
Crawl discovery pages through residential proxies or the Scraper API with JavaScript rendering. Extract media URL, caption, alt text, licence tag and page URL into a manifest.
Deduplicate the manifest on media URL and on a perceptual hash of thumbnails before downloading, so repeated or mirrored items are not fetched twice.
Download files from the manifest through the datacenter endpoint with a bounded worker pool, verifying content type and byte size, and writing a checksum per file.
Keep manifest and files linked by checksum. Store licence and source URL with each item so you can filter or remove sources later.

# verify endpoint against docs.geonode.com before publishing
import requests, hashlib

DC_USER = "USERNAME-type-datacenter"
RES_USER = "USERNAME-country-us"
PASS = "PASSWORD"
HOST = "premium-residential.geonode.com:9000"

discovery = {"https": f"http://{RES_USER}:{PASS}@{HOST}"}
download = {"https": f"http://{DC_USER}:{PASS}@{HOST}"}

page = requests.get("https://example-gallery.com/category/birds?page=1",
                    proxies=discovery, timeout=30)
# ... parse page.text for media URLs into `media_urls` ...

for url in media_urls:
    r = requests.get(url, proxies=download, timeout=60, stream=True)
    data = r.content
    print(url, r.headers.get("content-type"), hashlib.sha256(data).hexdigest()[:12])

Get the exact snippet for your language from the API Code Generator in the dashboard.

Dataset-specific settings that matter

Two proxy users, two queues

Create one residential user for discovery and one datacenter user for downloads, and never mix them. The discovery queue is small and latency-tolerant; the download queue is large and bandwidth-bound. Separate credentials make the cost split visible in the dashboard.

Rotating for downloads, sticky for galleries

Media files have no session state, so rotate on every download. Gallery pages often paginate with a cursor tied to cookies; use a sticky session id for the duration of one gallery crawl, then drop it.

Concurrency and chunk size

Geonode sets no fixed thread limit, but a CDN will throttle one client pulling hundreds of parallel streams. Keep download workers modest per host, use streamed responses, and verify byte length against Content-Length so truncated files are retried rather than stored.

Verify before you store

Check the Content-Type header, decode the image or probe the video container, and compute a checksum. A surprising share of "images" from discovery crawls are HTML error pages or tracking pixels; filtering them at download time keeps the dataset clean.

What to log

Log media URL, source page, proxy type used, status, bytes, checksum, licence tag and timestamp. When a rights holder asks for removal, this log identifies every file from that source in seconds.

What Geonode gives you for image and video datasets

Rotating datacenter proxies from $0.14/GB for bulk media file downloads.
Residential IPs in 195+ countries with country, state and city targeting for discovery.
Scraper API with JavaScript rendering and JSON output for lazy-loaded galleries.
No fixed concurrency limit; HTTP, HTTPS, SOCKS5 and UDP protocols.
"1TB on us" program for residential accounts above 500 GB per month.

Pricing for image and video datasets

Datacenter proxies for downloads start at $0.14/GB. Residential proxies for discovery are pay-as-you-go from $0.27/GB at volume, with an entry pack of 10 GB for $7.92, 50 GB for $34.65 and a 3-day trial for $5. No contract on any plan. The Scraper API is free for 1.5K requests per month, then $3.50 per 10K requests. ISP proxies start at $1.25 per IP for fixed source polling. Accounts above 500 GB per month on residential qualify for the "1TB on us" program. See the pricing page.

FAQ

Why use datacenter image dataset scraping proxies instead of residential for everything?

Bandwidth. Media files are large and CDN hosts rarely apply the defences that gallery pages do, so datacenter at $0.14/GB handles downloads at lower cost, while residential IPs are reserved for the small discovery traffic that needs a consumer-looking address.

Which image dataset scraping proxies should I use for discovery?

Residential proxies with country targeting. Galleries and listings render client-side and often filter by region, so a residential IP in the right country returns the full grid with real media URLs, captions and licence tags.

Can I collect video as well as images?

Yes. The same split applies: discover video pages through residential IPs or the Scraper API, extract the media URL and metadata, then download through the datacenter endpoint with streamed responses and a byte-length check against Content-Length.

How do I keep licensing clean?

Record the licence tag and source URL with every item in the manifest, collect only from sources whose terms permit automated access, and skip anything behind a login. Keep the manifest so a source can be removed from the dataset on request.

What are the compliance limits for media collection?

Collect only publicly accessible files, respect robots.txt, site terms and rate limits, and never download content behind authentication. Geonode's acceptable use policy prohibits fraud, spam and credential abuse. Watch for personal data in images and apply your own review process.

How much bandwidth will a dataset need?

Estimate from a sample: download a few hundred items, take the median file size and multiply by your target count. Datacenter pricing at $0.14/GB makes the proxy line item predictable once you know that number.

Start

Run discovery for one category through the residential trial, build a manifest, and download a sample through the datacenter endpoint to measure cost per item. Get started or Contact sales for terabyte-scale pricing.