刮下XHR请求

Question

I want to scrape the second page of this user reviews .

However the next button executes a XHR request, and while I can see it using Chrome developer tools, I cannot replicate it.

Answer 1

It's not so easy task. First of all you should install this extension . It helps you to test own requests based on captured data, ie catch and simulate requests with captured data.

As I see they send a token in this XHR request, so you need to get it in from html page body(stores in source code, js variable "taSecureToken" ).

Next you need to do four steps:

1. Catch POST request with plugin
2. Change token to saved before
3. Set limit and offset variables in POST request data
4. Generate request with resulted body

Note: on this request server returns json data(not the html with next page) containing info about loaded objects on next page.