How can I programmatically see what requests (e.g. API or resource requests) are being made by a website

Question

In Chrome, you can go to developer tools >.network tab to see all the requests the website is making.

What would be a good way to get the list of these requests programmatically? I guess I can grab the content of the site, grab all the URLs that are in the page and parse them, but that seems a bit tedious, especially if the requests are being made from a JS file.

Is there an easier way?

Answer 1

On the server side, there's the access log that will have request information for all requests made against it. It might take some configuration to save all the fields you want to use (method, POST data, cookies, time, remote client address) but this is more efficient than trying to do things from a browser. Each web server has a different way of configuring its log files.