aiohttp is the client you pick when a scraper needs hundreds of in-flight requests and you want full control over the event loop. Routing it through a proxy is a few keyword arguments, but the defaults around authentication, connection limits and timeouts are different from requests and from httpx, and that is where most aiohttp proxy setups go wrong. By the end of this guide you will have a ClientSession that authenticates against a rotating residential gateway, runs bounded parallel requests, times out predictably and reports proxy errors by type.
Examples target aiohttp 3.14 on Python 3.11 or newer. Where behaviour changed in a recent release, the version is called out.
Configure an aiohttp proxy on the session
The proxy keyword is accepted both per request and on the session. Per request is the original API and still works; session-level is the one you want for a scraper, because it removes the chance of a single call escaping the proxy.
import asyncio
import aiohttp
PROXY = "http://USERNAME:PASSWORD@premium-residential.geonode.com:9000"
async def main() -> None:
async with aiohttp.ClientSession(proxy=PROXY) as session:
async with session.get("https://api.ipify.org?format=json") as resp:
print(resp.status, await resp.json())
asyncio.run(main())
Port 9000 is Geonode's HTTP(S) rotating gateway, so each new connection exits from a different residential IP. An individual call can still override the default with its own proxy= argument, which is handy when one request must go direct, for example to an internal metrics endpoint.
Per-request form
async with aiohttp.ClientSession() as session:
async with session.get("https://api.ipify.org?format=json", proxy=PROXY) as resp:
print(await resp.json())
Use this when the proxy varies from call to call, such as when a pool manager hands out a different sticky session per task.
Authentication: URL credentials or proxy_headers
There are two supported ways to send proxy credentials, and one deprecated way that you will still see in older tutorials.
| Method | Example | Status |
|---|
| Credentials in the proxy URL | proxy="http://user:pass@host:9000" | Supported |
proxy_headers with encode_basic_auth | proxy_headers={"Proxy-Authorization": aiohttp.encode_basic_auth("user", "pass")} | Supported, recommended |
proxy_auth=aiohttp.BasicAuth("user", "pass") | | Deprecated since 3.14, removed in 4.0 |
The URL form is shortest. The header form keeps the secret out of the URL, which matters if proxy URLs end up in logs or exception messages. aiohttp.encode_basic_auth returns the ready-made Basic ... string:
import asyncio
import os
import aiohttp
PROXY = "http://premium-residential.geonode.com:9000"
async def main() -> None:
proxy_headers = {
"Proxy-Authorization": aiohttp.encode_basic_auth(
os.environ["GEONODE_USER"], os.environ["GEONODE_PASS"]
)
}
async with aiohttp.ClientSession(proxy=PROXY, proxy_headers=proxy_headers) as session:
async with session.get("https://api.ipify.org?format=json") as resp:
print(await resp.json())
asyncio.run(main())
If your codebase still uses proxy_auth=aiohttp.BasicAuth(...), it works on 3.14 with a DeprecationWarning. Migrate now: the parameter and the class are removed in aiohttp 4.0, and the replacement is a one-line change.
Parallel requests with asyncio.gather and a Semaphore
The reason to use aiohttp is concurrency, and the reason concurrency breaks proxies is that nothing bounds it by default. asyncio.gather over a thousand coroutines will try to open a thousand connections; the connector caps that at 100, and the rest queue inside aiohttp, which is better than nothing but gives you no control over pacing. A Semaphore makes the bound explicit and lets you tune it independently of the connector.
import asyncio
import aiohttp
PROXY = "http://USERNAME:PASSWORD@premium-residential.geonode.com:9000"
CONCURRENCY = 20
async def fetch(session: aiohttp.ClientSession, sem: asyncio.Semaphore, url: str) -> tuple[str, int]:
async with sem:
async with session.get(url) as resp:
await resp.read()
return url, resp.status
async def main() -> None:
urls = [f"https://httpbin.org/get?i={i}" for i in range(100)]
sem = asyncio.Semaphore(CONCURRENCY)
connector = aiohttp.TCPConnector(limit=CONCURRENCY, ttl_dns_cache=300)
timeout = aiohttp.ClientTimeout(total=30, connect=10, sock_read=20)
async with aiohttp.ClientSession(
proxy=PROXY, connector=connector, timeout=timeout
) as session:
results = await asyncio.gather(
*(fetch(session, sem, u) for u in urls), return_exceptions=True
)
ok = sum(1 for r in results if not isinstance(r, Exception))
print(f"{ok}/{len(urls)} succeeded")
asyncio.run(main())
return_exceptions=True keeps one failed request from cancelling the batch. Match the semaphore to the connector limit so neither side silently queues work the other cannot drain.
TCPConnector: limit and limit_per_host
The connector owns the socket pool. Two parameters matter for aiohttp proxy traffic.
limit is the total number of simultaneous connections, default 100. limit_per_host caps connections to one (host, port, is_ssl) triple, default 0 meaning unlimited. Through a proxy, every request to every target shares the proxy's host and port, so the per-host cap applies to the gateway, not the destination sites. For a pure proxy workload, set limit to your intended concurrency and leave limit_per_host at 0; set limit_per_host only when the same session also talks to origins directly and you want to protect them.
connector = aiohttp.TCPConnector(limit=30, limit_per_host=10, ttl_dns_cache=300)
ttl_dns_cache=300 keeps the gateway's DNS answer for five minutes instead of the default ten seconds, which removes a resolver round trip from most new connections. The session takes ownership of the connector and closes it on exit; pass connector_owner=False only if you deliberately share one connector across sessions.
ClientTimeout instead of the 5-minute default
aiohttp's default total timeout is 300 seconds. That is generous for direct requests and far too long for proxy traffic, where a stalled residential exit should fail fast so you can retry on another IP.
timeout = aiohttp.ClientTimeout(total=30, connect=10, sock_read=20)
total bounds the whole request including redirects and body download. connect bounds acquiring a connection from the pool plus the TCP and proxy handshake. sock_read bounds the gap between consecutive bytes, which catches targets that accept the connection and then trickle data. Pass the object to the session for a default, or to a single request to override it. Expiry raises asyncio.TimeoutError, discussed below.
trust_env and environment proxies
Unlike requests, aiohttp ignores HTTP_PROXY, HTTPS_PROXY and NO_PROXY unless you opt in with trust_env=True. When enabled it uses urllib.request.getproxies(), honours no_proxy, and also reads proxy credentials from ~/.netrc if present.
async with aiohttp.ClientSession(trust_env=True) as session:
async with session.get("https://api.ipify.org?format=json") as resp:
print(await resp.json())
For a scraper that must always use a specific gateway, leave trust_env off and set proxy= explicitly. That way a corporate proxy on a CI runner cannot hijack routing, and no_proxy cannot quietly exempt a target.
SOCKS5 with aiohttp-socks
aiohttp has no built-in SOCKS support, so an aiohttp proxy over SOCKS5 needs a third-party connector. The aiohttp-socks package provides a ProxyConnector that replaces TCPConnector and handles the SOCKS handshake. Install it with pip install aiohttp-socks. Geonode's SOCKS5 rotating port is 11000:
import asyncio
import aiohttp
from aiohttp_socks import ProxyConnector
SOCKS_URL = "socks5://USERNAME:PASSWORD@premium-residential.geonode.com:11000"
async def main() -> None:
connector = ProxyConnector.from_url(SOCKS_URL)
async with aiohttp.ClientSession(connector=connector) as session:
async with session.get("https://api.ipify.org?format=json") as resp:
print(await resp.json())
asyncio.run(main())
Do not combine ProxyConnector with proxy= on the session; the connector already routes everything. ProxyConnector accepts the same limit and limit_per_host keywords as TCPConnector.
The TLS-in-TLS limitation
aiohttp speaks to HTTP proxies and tunnels https:// targets through them with CONNECT. What it does not reliably do is connect to the proxy itself over TLS, that is an https:// proxy URL, because TLS-in-TLS depends on asyncio transport support that has varied across Python releases. The documentation is explicit that this should not be assumed. In practice it does not matter for Geonode: the gateway is reached over plain HTTP on 9000/10000 and your https:// traffic is still encrypted end to end inside the CONNECT tunnel, so keep the proxy URL scheme as http://.
Sticky sessions for multi-step flows
The rotating port gives a new IP per connection. For login flows or paginated listings you need the same IP for a few minutes, which Geonode exposes on HTTP(S) sticky port 10000 through the username:
import secrets
import aiohttp
def sticky_proxy(country: str = "us", lifetime: int = 600) -> str:
session_id = secrets.token_hex(4) # 8 alphanumeric characters
user = f"USERNAME-country-{country}-session-{session_id}-lifetime-{lifetime}"
return f"http://{user}:PASSWORD@premium-residential.geonode.com:10000"
session = aiohttp.ClientSession(proxy=sticky_proxy())
Lifetime ranges from 180 to 86400 seconds. State and city targeting cannot be combined, and asn requires a country. Generating a new session id is how you rotate on demand.
Diagnostics: which exception means what
| Exception | Meaning | Typical fix |
|---|
aiohttp.ClientProxyConnectionError | Could not open a TCP connection to the proxy host | Check hostname, port and firewall; raise connect timeout |
aiohttp.ClientHttpProxyError with .status == 407 | Proxy rejected credentials during CONNECT | Fix username or password; check username option grammar |
aiohttp.ClientHttpProxyError with 502/503 | Gateway could not establish the tunnel | Retry with a new session id; confirm the port matches the protocol |
asyncio.TimeoutError | A ClientTimeout field expired | Retry on a fresh IP; tune sock_read |
aiohttp.ClientConnectorError | Generic connection failure (DNS, refused) | Inspect .os_error |
ClientProxyConnectionError is a subclass of ClientConnectorError, and ClientHttpProxyError is a subclass of ClientResponseError, so catch the specific ones first.
import asyncio
import aiohttp
PROXY = "http://USERNAME:PASSWORD@premium-residential.geonode.com:9000"
async def fetch_ip(session: aiohttp.ClientSession) -> str | None:
try:
async with session.get("https://api.ipify.org?format=json") as resp:
resp.raise_for_status()
return (await resp.json())["ip"]
except aiohttp.ClientHttpProxyError as e:
if e.status == 407:
print("proxy auth failed: check credentials")
else:
print("proxy returned", e.status)
except aiohttp.ClientProxyConnectionError as e:
print("cannot reach proxy:", e)
except asyncio.TimeoutError:
print("timed out through proxy")
return None
async def main() -> None:
timeout = aiohttp.ClientTimeout(total=30, connect=10, sock_read=20)
async with aiohttp.ClientSession(proxy=PROXY, timeout=timeout) as session:
print(await fetch_ip(session))
await asyncio.sleep(0.25) # let SSL transports close before the loop stops
asyncio.run(main())
Note that for a plain http:// target the proxy's 407 arrives as an ordinary response rather than an exception, so check resp.status there too.
Graceful shutdown
When a session closes, asyncio may still be tearing down the underlying SSL transports. If the event loop stops first you get ResourceWarning: unclosed transport. The documented fix is a short pause after the session's async with block: await asyncio.sleep(0) for plain HTTP sessions, and await asyncio.sleep(0.25) when the session made HTTPS requests, which through a CONNECT tunnel it almost always has. The example above includes it.
Common mistakes
Creating a session per request
ClientSession holds the connection pool and cookie jar. Opening one per request defeats keep-alive to the gateway, multiplies proxy handshakes and triggers the shutdown warning on every call. Create one session per worker and pass it around.
Forgetting that the proxy is the only "host"
With limit_per_host set to a low number, every proxied request counts against the gateway's single (host, port) entry, so a per-host cap of 5 throttles your entire scraper to 5 connections. Set limit instead, or leave limit_per_host at 0.
Relying on the default timeout
Five minutes is long enough that a stalled residential exit will hold a semaphore slot for the whole window. Always pass a ClientTimeout with connect and sock_read values.
Still passing proxy_auth
It works today with a warning and breaks on 4.0. Replace it with proxy_headers={"Proxy-Authorization": aiohttp.encode_basic_auth(user, pw)} or put the credentials in the proxy URL.
Next step
With the session above you have authenticated, bounded and observable aiohttp proxy traffic; what remains is the pool it exits through. Geonode residential proxies serve the endpoints used here: 2.5M IPs online, 30–50M unique IPs rotating monthly across 195+ countries, with country, state, city and ASN targeting via the username and sticky sessions up to 24 hours over HTTP, HTTPS and SOCKS5. Pricing starts at $0.27/GB at volume, the entry plan is 10 GB for $7.92/month, and a 3-day trial is $5 with no contract. The web scraping and price intelligence use cases show how teams combine this connector setup with retry and scheduling logic.
FAQ
How do I set an aiohttp proxy for a whole session?
Pass proxy= to the ClientSession constructor: aiohttp.ClientSession(proxy="http://user:pass@host:port"). Every request from that session uses it unless the call overrides proxy= with another value. Session-level proxy has been available since aiohttp 3.11; on older releases you must pass proxy= on each get() or post() call instead.
Why is proxy_auth deprecated in aiohttp and what replaces it?
Since aiohttp 3.14 the proxy_auth parameter and the aiohttp.BasicAuth class are deprecated and scheduled for removal in 4.0. The replacement is a plain header: proxy_headers={"Proxy-Authorization": aiohttp.encode_basic_auth("user", "pass")}. It does the same Base64 encoding without a helper class and works both per request and at session level.
Does aiohttp support HTTPS proxies or SOCKS5?
aiohttp natively speaks to HTTP proxies and uses CONNECT to tunnel https:// targets through them. Connecting to the proxy itself over TLS (TLS-in-TLS) depends on asyncio support in your Python build and should not be relied on. SOCKS4 and SOCKS5 need the third-party aiohttp-socks package and its ProxyConnector.
How many concurrent requests can I run through an aiohttp proxy?
The TCPConnector allows 100 simultaneous connections by default with no per-host cap. Through a proxy every request shares the same host and port, so set limit explicitly, add limit_per_host if you also make direct calls, and wrap request coroutines in an asyncio.Semaphore so queued tasks do not all open sockets at once.
Why does my aiohttp request through a proxy hang instead of failing?
ClientSession has a 5-minute total timeout by default, which looks like a hang when a residential exit stalls. Pass aiohttp.ClientTimeout(total=30, connect=10, sock_read=20) to the session or the request so stalls surface as asyncio.TimeoutError quickly, then retry with a fresh session id rather than waiting.
What are the practical limits of routing scrapers through residential proxies?
A proxy only changes the source IP. It does not grant permission to ignore robots.txt, site terms or rate limits, and it does not make collecting personal data lawful. Keep concurrency reasonable, back off on 429s, and remember that Geonode sticky sessions last at most 24 hours before the IP is released.
