How to capture traffic in python

Question

I have a webdriver using selenium that opens a browser for me, points it to an ip Address, does a bunch of stuff and closes.

I want to know all of the urls accessed during this time. That is, any ads that are loaded, any css calls that were made out to any url and so on.

Here is the code im using

from selenium import webdriver

browser = webdriver.Firefox(profile) # Get local session of firefox
browser.get(url) # Open a url and wait for it to finish

Answer 1

I did it by loading the firefox plugins Firebug and Netexport. The first is a tool that allows you to see all the exchange of information, the second allows to write all of it in a file (.har extension). So basically selenium has to load the plugins, the website and wait the time you want, and when it closes, you get a file with the result.

Answer 2

Its not a python solution.. But you can add fiddler plug in to Firefox. We needed to do exact same thing about a year ago. We used selenium to open browser and all UI stuff and in background Fiddler captured all traffic (http and https) .. This also list all JS CSS src and you can debug later with inspector see what request is sent and what response is received