简体   繁体   English

Scrapy CSV 格式不正确

[英]Scrapy CSV incorrectly formatted

I'm new to scrapy package, and here's my problem:我是scrapy包的新手,这是我的问题:

import scrapy

class simpleSpider(scrapy.Spider):
    name = "simple_spider"

    start_urls = ['http://quotes.toscrape.com/login']


    def parse(self, response):
        token = response.css("input[name=csrf_token] ::attr(value)").extract_first()
        formdata = {
            'csrf_token' : token,
            'username' : 'rseiji',
            'password' : 'seiji1234'
        }

        yield scrapy.FormRequest(response.url, formdata=formdata, callback=self.parse_logged)

    def parse_logged(self, response):

        yield {
            'text' : response.css('span.text::Text').extract(),
            'author' : response.css('small.author::Text').extract(),
            'tags' : response.css('div.tags a.tag::Text').extract()
        }

This is my spider.这是我的蜘蛛。 And it does work.它确实有效。 But when I try to:但是当我尝试:

scrapy crawl simple_spider -o mySpider.csv

the .csv file doesn't seen to be correctly formated. .csv 文件的格式不正确。 It extracts only the "text" column.它仅提取“文本”列。

What's wrong?怎么了?

Thank you!谢谢!

Editted: This is my .csv file:编辑:这是我的 .csv 文件:

text,author,tags
"“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”,“It is our choices, Harry, that show what we truly are, far more than our abilities.”,“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”,“The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.”,“Imperfection is beauty, madness is genius and it's better to be absolutely ridiculous than absolutely boring.”,“Try not to become a man of success. Rather become a man of value.”,“It is better to be hated for what you are than to be loved for what you are not.”,“I have not failed. I've just found 10,000 ways that won't work.”,“A woman is like a tea bag; you never know how strong it is until it's in hot water.”,“A day without sunshine is like, you know, night.”","Albert Einstein,J.K. Rowling,Albert Einstein,Jane Austen,Marilyn Monroe,Albert Einstein,André Gide,Thomas A. Edison,Eleanor Roosevelt,Steve Martin","change,deep-thoughts,thinking,world,abilities,choices,inspirational,life,live,miracle,miracles,aliteracy,books,classic,humor,be-yourself,inspirational,adulthood,success,value,life,love,edison,failure,inspirational,paraphrased,misattributed-eleanor-roosevelt,humor,obvious,simile"
...

Figure out now that it is not that there are empty columns.现在弄清楚并不是有空列。 The .csv file format is not well defined. .csv 文件格式没有明确定义。 Everything came up in just one row!一切都出现在一行中!

Solved!解决了!

import scrapy

class simpleSpider(scrapy.Spider):
    name = "simple_spider"

    start_urls = ['http://quotes.toscrape.com/login']


    def parse(self, response):
        formdata = {
            'username' : 'rseiji',
            'password' : 'seiji1234'
        }

        yield scrapy.FormRequest.from_response(response, formdata=formdata, callback=self.parse_logged,)


    def parse_logged(self, response):
        # Get list of Selector objects and loop through them
        for quote in response.css('div.quote'):
            # yield each item individually
            yield {
                'text' : quote.css('span.text::Text').extract_first(),
                'author' : quote.css('small.author::Text').extract_first(),
                'author_goodreads_url' : quote.css('span a[href*="goodreads.com"]::attr(href)').extract_first(),
                'tags' : quote.css('div.tags a.tag::Text').extract()
            }

The problem was I was using extract().问题是我正在使用extract()。 What I wanted to do was get a list of Selector objects.我想要做的是获取Selector对象的列表。

Using extract() will always produce a list output.使用 extract() 总是会产生一个列表输出。 When you use extract you get a list of strings of the html you requested with the selector, or when using extract_first() a single string.当您使用 extract 时,您将获得使用选择器请求的 html 的字符串列表,或者在使用 extract_first() 时获得单个字符串。 By not using extract() nor extract_first() you create a list of selectors which you can then iterate through and chain a new selector on it allowing you to pickout each individual item.通过不使用extract() 和extract_first(),您可以创建一个选择器列表,然后您可以遍历该列表并在其上链接一个新的选择器,从而可以挑选出每个单独的项目。

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM