使用 scrapy 抓取网站时，如何动态生成请求？

Question

I am trying to scrape data from website using scrapy. But Page is generated using Request which means url + other data(I found that using developer tools of browser). I copy the request as cURL and translated it to scrapy using curl to scrapy link. I can get data from these page when I create request manually and then fetch. So I need to create request manually. My Question is how can I create request dynamically in spider.

class PowerGenerationSpider(scrapy.Spider):
    name = "power_generation"
    # make request
    url = 'http://my/website'

headers = {
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0",
    "Accept": "application/json, text/plain, */*",
    "Accept-Language": "en-US,en;q=0.5",
    "Authorization": "authorization_string",
    "Content-Type": "application/json",
    "Origin": "http://my/website",
    "DNT": "1",
    "Connection": "keep-alive",
    "Referer": "mywebsite"
}
# need to run this for month 01 to 10 or how can i change this programatically
body = '{"month":"01","year":"2020","user_id":"vishvajeet"}'

request = scrapy.Request(
    url=url,
    method='POST',
    dont_filter=True,
    headers=headers,
    body=body,
)

def start_requests(self):
    yield self.request #scrapy.Request(self.request, self.parse)

def parse(self, response):
    data = json.loads(response.text)
    solar_energy_datalist = data["resultObject"]
    item = PowerGenerationItem()
    for solar_energy_data in solar_energy_datalist:
        date = solar_energy_data['date']
        power_generation = solar_energy_data['power_generation']

        item['date'] = date
        item['power_generation'] = power_generation
        yield item

how can i generate request with diffrent parameter inside body of request and pass it to the crawl for next request to crawl. Note: I found other resource on web which tell how to generate url dynamically. This is not about url, I want to generate request becaues URL is same.

edit 1: cURL which i converted to scrapy request is below. i might delete authentication related information later.

curl 'http://3.6.0.2/inject-solar-angular/inject_solar_server/graph/Graph/cumulative_month_graph' -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0' -H 'Accept: application/json, text/plain, */*' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'Authorization: eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpZCI6IjkwIiwicm9sZSI6IjQiLCJ0aW1lc3RhbXAiOjE1OTU1NjcxMjEsInN0YXR1cyI6MX0.-RFFUc69PxDsAk_zmn3VI8OqUh-mYkYioFyTSBU17_s' -H 'Content-Type: application/json' -H 'Origin: http://www.injectsolar.com' -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Referer: http://www.injectsolar.com/portal/' --data-raw '{"month":"01","year":"2020","user_id":"triose"}'

edit: 2

I am trying to fetch jan-20 to june-20 data on daily basis of power generation. for that i need to change the parameter inside body. If inside body directory month=01 it give jan-20 data. If i change it to month=02 it give me feb data. But I want to do this automatically through crawler. As if crawler crawl from one page to other page. It should give me data like that.

Answer 1

A few things in terms of the code provided. I would like you to provide abit more code of an attempt at trying to generate the request with different body parameters dynamically. This is something I don't see in your code yet, so there's only so much I can help. But there's enough in this text here to show you how to transfer ANY information you want from one function to another so you should be able to make an attempt.

As another disclaimer, because I don't have the URL, I can't run this code myself, so again it's hard to unpick if there's problems with it.

Corrections

1. Just put the scrapy Request within start_requests function
2. Refering to variables outside the function requires you to add self.VARIABLE. In this case self.headers
3. To input the parameters you need to use the meta argument. So cb_kwargs = self.body . The cb_kwargs argument is used to transfer any dictionary of information you want from one function to another. Headers is a defined argument in scrapy.Request , so it doesn't apply to that. But the body dictionary you created is user created information and the cb_kwargs argument applies to this.
4. You need to define a callback for the response to be processed. Here we define the callback as self.parse which is the default callback for the start_requests function.
5. Unless the url is different to URL in the start_urls . Refer to the URL argument in the scrapy.Request as self.start_urls[0] . In this case you've defined the variable url, so in your scrapy.Request you need to refer to it as self.url

Code Example

   class PowerGenerationSpider(scrapy.Spider):
        name = "power_generation"
        # make request
        url = 'http://my/website'

        headers = {
           "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0",
           "Accept": "application/json, text/plain, */*",
           "Accept-Language": "en-US,en;q=0.5",
           "Authorization": "authorization_string",
           "Content-Type": "application/json",
           "Origin": "http://my/website",
           "DNT": "1",
           "Connection": "keep-alive",
           "Referer": "mywebsite"
              }
   # need to run this for month 01 to 10 or how can i change this programatically
        body = {"month":"01","year":"2020","user_id":"vishvajeet"}



        def start_requests(self):
             yield scrapy.Request(
                              url=self.url,
                              method='POST',
                              dont_filter=True,
                              headers=self.headers,
                              cb_kwargs=self.body,
                              callback=self.parse
                             )

        def parse(self, response):
            data = json.loads(response.text)
            solar_energy_datalist = data["resultObject"]
            item = PowerGenerationItem()
            for solar_energy_data in solar_energy_datalist:
                date = solar_energy_data['date']
                power_generation = solar_energy_data['power_generation']

                item['date'] = date
                item['power_generation'] = power_generation
                yield item

Additional Information

1. Remember that the start_requests function is used to generate requests for the start_urls.

2. The self.VARIABLE comes from knowing about classes. Understanding what this means is essential to following along this example. The beauty of self.VARIABLE means that if you define a variable in a class that is NOT within a funtion in the class, it can be used in ANY function within the class but you have to remember to refer to as self.VARIABLE in each function. This type of variable is called a class variable.

3. Consider ItemLoaders for extracting data if you need to modify the responses scrapy gives you or want to apply a small function to the extracted data to clean it up. ItemLoaders gives you a lot more freedom to change the outputs of the scrapy responses.

Resources for additional learning

1. For more information about class variables please see here
2. Read the scrapy docs for scrapy.Request here if you read carefully it will explain about cb_kwargs here but also you should have a better understanding of the arguments you can use.