如何按特定顺序存储scrapy中抓取的数据？

Question

I have to crawl data from a web page in a specfic order as liked i declared fields in my item class and then have to put them in csv file.problem now occuring is there its stores data not in specfic order as like its scrapping data of any field and putting in csv file but i want it should store data as i declared in my item class. I am newbie in python. can you tell me how to do this

For ex: my item class is class DmozItem(Item): title = Field() link = Field() desc = Field()

Now when its storing data in csv file its storing first desc ,link and then title "desc": [], "link": ["/Computers/Programming/"], "title": ["Programming"]}

Answer 1

The reason that the order of data in csv file is not what you declared is because item is a dict data type. The order of keys in dict is decided by their alphabet order. The logic of export items to csv file is implemented in

scrapy\\contrib\\exporter__init__.py

You can rewrite the _get_serialized_fields method of BaseItemExporter to let it yield key-value pair in the order of your declaration. Here is an example code

field_iter = ['title', 'link', 'desc']
for field_name in field_iter:
    if field_name in item:
        field = item.fields[field_name]
        value = self.serialize_field(field, field_name, item[field_name])
    else:
        value = default_value
    yield field_name, value

But do remember, it is not an universal solution.

Answer 2

为此，我们必须在BaseItemExporter类field_iter = ['Offer'，'SKU'，'VendorCode']中的fields_to_export中创建一个列表，然后在字段中传递此列表