How to Use a Proxy in Python's Requests Library

How to Use a Proxy in Python's Requests Library

Python's requests library is the most common way developers make HTTP calls, and it has built-in, straightforward proxy support.

The basic pattern

You define a dictionary that maps the 'http' and 'https' schemes to your proxy address, then pass it as the proxies argument on any get, post, or session call. Every request made with that dictionary routes through the specified proxy instead of your direct connection.

Using authenticated proxies

If the proxy requires a username and password, you embed the credentials directly in the proxy URL before the host, in the standard scheme://user:pass@host:port format. requests parses this automatically without any extra configuration.

Reusing a session for rotation

For scraping across many pages, create a requests.Session object and swap out its proxies dictionary between batches of requests to rotate IPs, rather than reconnecting from scratch each time, which is both faster and closer to how production scrapers are built.