How to properly use proxies in Python requests?

Question

I have a list of proxies. When I test one of them using curl :

curl -proxy https://x.x.x:port https://www.google.com/

I get the expected result, but when I run:

proxies = {'http' : 'x.x.x:port', 'https' : 'x.x.x:port'}
requests.get('https://www.google.com/', proxies = proxies)

It is stuck for a while and then I get this error:

requests.exceptions.ProxyError: HTTPSConnectionPool(host='www.google.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', ConnectionResetError(104, 'Connection reset by peer')))

I tried different ways of proxy definition, like:

'http' : 'http://x.x.x:port', 'https' : 'https://x.x.x:port'

and nothing works, always get this error, how to fix that?

Answer 1

EDIT : Just saw your last note of trying different ways which aligns with my first suggestion. You may want to try setting the proxy with the os library in my later suggestion.

There's two things that stand out to me:

1. You used https://xxx:port in your curl command but used xxx:port in Python. You might have to include https:// .

2. You used both http and https. I don't think this one would cause an error, but still worth investigating

First, I would suggest including https:// like so:

proxies = {'https' : 'https://x.x.x:port'}  
requests.get('https://www.google.com/', proxies = proxies) 

Second, I would try just setting https (and if that fails, try just setting http):

import requests

proxies = {
  'https': 'https://x.x.x.x:port',
}

requests.get('https://example.org', proxies=proxies)

Also, you can set environmental proxies through Python like:

import os
os.environ["http_proxy"] = "http://x.x.x.x:port"
os.environ["https_proxy"] = "https://x.x.x.x:port"

or through the command line / command prompt

Windows

set http_proxy=http://x.x.x.x:port
set https_proxy=https://x.x.x.x:port

Linux/OS X:

export http_proxy=http://x.x.x.x:port
export https_proxy=https://x.x.x.x:port