在Python的Selenium webdriver中使用带有无头firefox的http代理

Question

I'm using Firefox headless like this:

from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
from selenium import webdriver
import os
import sys

# Set the MOZ_HEADLESS environment variable which casues Firefox to
# start in headless mode.
os.environ['MOZ_HEADLESS'] = '1'

# Select your Firefox binary.
binary = FirefoxBinary('/usr/bin/firefox', log_file=sys.stdout)

# Start selenium with the configured binary.
driver = webdriver.Firefox(firefox_binary=binary)

But now I want to add a http proxy that requires a user/password. After searching around, I tried the following:

from selenium.webdriver.common.proxy import Proxy, ProxyType

myProxy = "xx.xx.xx.xx:80"

proxy = Proxy({
    'proxyType': ProxyType.MANUAL,
    'httpProxy': myProxy,
    'ftpProxy': myProxy,
    'sslProxy': myProxy,
    'noProxy': '' # set this value as desired
    })

driver = webdriver.Firefox(firefox_binary=binary, proxy=proxy)

I also tried

profile = webdriver.FirefoxProfile() 
profile.set_preference("network.proxy.type", 1)
profile.set_preference("network.proxy.http", "xx.xx.xx.xx")
profile.set_preference("network.proxy.http_port", 80)
profile.update_preferences() 
driver=webdriver.Firefox(firefox_binary=binary,firefox_profile=profile)

Finally, I tried adding "socksUsername" and "socksPassword" with the creds to the proxy , more out of desperation than any real hope.

Needless to say none of these works, and testing shows requests are still using my usual IP, not the proxy.

Also system-wide proxy is not an option in this case.

Where should the http proxy credentials live? How can I use a proxy with headless firefox?

Testing

driver.get("https://www.ipinfo.io");
driver.find_element_by_xpath('//h4/following-sibling::p').text

Answer 1

If your proxy requires a username and a password, you have to write it like that :

from selenium.webdriver.common.proxy import Proxy, ProxyType

myProxy = "username:password@proxyDomain:proxyPort"

proxy = Proxy({
    'proxyType': ProxyType.MANUAL,
    'httpProxy': myProxy,
    'ftpProxy': myProxy,
    'sslProxy': myProxy,
    'noProxy': '' # set this value as desired
    })

driver = webdriver.Firefox(firefox_binary=binary, proxy=proxy)

Answer 2

Have you try setting up a profile manually with

./firefox --ProfileManager

manually setup the proxy and then load the profile you manually set

from selenium import webdriver

url = "https://mail.google.com"
fp = webdriver.FirefoxProfile('/Users/<username>/Library/Application Support/Firefox/Profiles/71v1uczn.default')

driver = webdriver.Firefox(fp)

Answer 3

You can try setting up an Environment variable " HTTP_PROXY " in following mnemonics:

http://<username>:<password>@<proxy_url>

Add your credentials separated by colon ':' before proxy url that is preceded by '@' for eg

http://username:password@proxy.com:8080/file.pac