如何在 python scrapy 中获取请求标头

Question

How to get request headers in python scrapy for a dynamic table data?

The request is made when a button is clicked.

Is there is a way to get the data without using click simulation?

Answer 1

You can extract the request from the website using the browser network tab. Observe the network tab to check which request is made when the button is clicked.

After that you can make the same request to the API in scrapy using the Request class.

Something like this:

yield scrapy.Request("http://www.example.com/some_page.html",
                     headers={'header', 'value'},
                     callback=self.parse_table
                    )

More info on Request here