Watir::browser in ruby - Catch all the XHR requests with their responses, made by browser

Question

I'm using following gems to open the site in the browser

gem "watir", "~> 6.1"
gem 'watir-scroll'
gem "geckodriver-helper"
gem 'chromedriver-helper'

I'm making an instance for the browser to operate on loaded HTML like,

browser = Watir::Browser.new :firefox
browser.goto "https://www.mysitessss.net/testing"

I just want to catch all the XHR requests made by this page URL after loading using the browser object, so I can get their responses.

Please help!

Answer 1

The primary purpose of webdriver and tools like WATIR and capybara that sit on top of webdriver is to drive the browser the way a user would. Deep inspection of the resulting network traffic is not really part of the tool.

To do this manually you can use the developer tools network tab to see all the requests that happen as the page is loaded. You could also use network sniffing or proxy tools such as fiddler-2, Charles, Wireshark, or Netmon.

To do it programmatically you might want to use nokogiri or a similar gem that specializes in parsing html perhaps in combination with something like rest-client if you want to actually make requests of the various api endpoints and work with the XML/JSON returned by the services being called. This blog posting might be helpful https://readysteadycode.com/howto-extract-data-from-html-with-ruby

Answer 2

You can achieve this with:

script = 'var performance = window.performance || window.mozPerformance || window.msPerformance || window.webkitPerformance || {}; return performance.getEntries().filter(e=>e.initiatorType=="xmlhttprequest")'
browser.execute_script(script)