如何在Python上使用selenium webdriver和browsermob代理捕獲網絡流量？

Question

我想通過在Python上使用Selenium Webdriver來捕獲網絡流量。 因此，我必須使用代理（如BrowserMobProxy）

當我使用webdriver.Chrome時：

from browsermobproxy import Server

server = Server("~/browsermob-proxy")

server.start()
proxy = server.create_proxy()

from selenium import webdriver
co = webdriver.ChromeOptions()
co.add_argument('--proxy-server={host}:{port}'.format(host='localhost', port=proxy.port))

driver = webdriver.Chrome(executable_path = "~/chromedriver", chrome_options=co)

proxy.new_har
driver.get(url)
proxy.har # returns a HAR 

for ent in proxy.har['log']['entries']:
    print ent['request']['url']

網頁已正確加載，所有請求都可在HAR文件中訪問和訪問。 但是當我使用webdriver.Firefox時：

# The same as above
# ...
from selenium import webdriver
profile  = webdriver.FirefoxProfile()
driver = webdriver.Firefox(firefox_profile=profile, proxy = proxy.selenium_proxy())

proxy.new_har
driver.get(url)
proxy.har # returns a HAR

for ent in proxy.har['log']['entries']:
    print ent['request']['url']

無法正確加載網頁，並且HAR文件中的請求數小於應該的請求數。

你知道第二個代碼中的代理設置有什么問題嗎？ 我應該如何修復它以正確使用webdriver.Firefox？

Answer 1

偶然發現了這個項目https://github.com/derekargueta/selenium-profiler 。 吐出URL的所有網絡數據。 不應該很難入侵並集成到您正在運行的任何測試中。

原始來源： https ： //www.openhub.net/p/selenium-profiler

Answer 2

對我來說，以下代碼組件工作正常。

profile = webdriver.FirefoxProfile()
profile.set_proxy(proxy.selenium_proxy())
driver = webdriver.Firefox(firefox_profile=profile)

Answer 3

I am sharing my solution, this would not write any logs to any file
but you can collect all sort of messages such as Errors, 
Warnings, Logs, Info, Debug , CSS, XHR as well as Requests(traffic)

1. We are going to create Firefox profile so that we can enable option of 
"Persist Logs" on Firefox (you can try it to enable on your default browser and see 
if it launches with "Persist Logs" without creating firefox profile )

2. we need to modify the Firefox initialize code 
where this line will do magic : options.AddArgument("--jsconsole");

so complete Selenium Firefox code would be,  this will open Browser Console
everytime you execute your automation :

 else if (browser.Equals(Constant.Firefox))
            {

                var profileManager = new FirefoxProfileManager();
                FirefoxProfile profile = profileManager.GetProfile("ConsoleLogs");
                FirefoxDriverService service = FirefoxDriverService.CreateDefaultService(DrivePath);
                service.FirefoxBinaryPath = DrivePath;                
                profile.SetPreference("security.sandbox.content.level", 5);
                profile.SetPreference("dom.webnotifications.enabled", false);
                profile.AcceptUntrustedCertificates = true;
                FirefoxOptions options = new FirefoxOptions();
                options.AddArgument("--jsconsole");
                options.AcceptInsecureCertificates = true;
                options.Profile = profile;
                options.SetPreference("browser.popups.showPopupBlocker", false);
                driver = new FirefoxDriver(service.FirefoxBinaryPath, options, TimeSpan.FromSeconds(100));
                driver.Manage().Timeouts().ImplicitWait = TimeSpan.FromSeconds(10);               

            }

3. Now you can write your logic since you have traffic/ logging window open so don't 
go to next execution if test fails. That way Browser Console will keep your errors 
messages and help you to troubleshoot further 

Browser : Firefox  v 61 

How can you launch Browser Console for firefox:
1. open firefox (and give any URL )
2. Press Ctrl+Shift+J (or Cmd+Shift+J on a Mac) 

Link : https://developer.mozilla.org/en-US/docs/Tools/Browser_Console