刮擦空的csv文件

Question

我正在嘗試運行我的小蜘蛛，它沒有返回錯誤，但輸出了一個空的csv文件

我正在通過命令行抓取爬行啟動蜘蛛AnimeReviews -o AnimeReviews.csv -t csv

這是我用過的圖書館

import scrapy
import json
from functools import reduce
from scrapy.selector import Selector
from AnimeReviews.items import AnimereviewsItem
last_page = 1789

這是我的蜘蛛

class AnimeReviewsSpider(scrapy.Spider):
    name = 'AnimeReviews_spider'
    allowed_urls =['myanimelist.net']
    start_urls = ['https://myanimelist.net/reviews.php?t=anime']

def parse(self, response):
    page_urls = [response.url + "&p=" + str(pageNumber) for pageNumber in range(1, last_page+1)]
    for page_url in page_urls:
        yield scrapy.Request(page_url,
            callback = self.parse_reviews_page)

def parse_reviews_page(self, response):
    item = AnimereviewsItem()
    reviews = response.xpath('//*[@class="borderDark pt4 pb8 pl4 pr4 mb8"]').extract()       #each page displays 50 reviews

    for review in reviews:
        anime_title = Selector(text = review).xpath('//div[1]/a[1]/strong/text()').extract()
        anime_url = Selector(text = review).xpath('//a[@class="hoverinfo_trigger"]/@href').extract()
        anime_url = map(lambda x: 'https://myanimelist.net'+ x ,anime_url)
        review_time = Selector(text = review).xpath('//*[@style="float: right;"]/text()').extract()[0]
        reviewer_name = Selector(text = review).xpath('//div[2]/table/tr/td[2]/a/text()').extract()
        rating = Selector(text = review).xpath('//div[2]/table/tr/td[3]/div[2]/text()').extract()
        for i in range(len(rating)):
            rating_temp = rating[i]
            rating[i] = rating_temp.split(" ")[1]
        review_text = Selector(text = review).xpath('//*[@class="spaceit textReadability word-break"]').extract()
        for i in range(len(review_text)):
            text = Selector(text = review_text[i]).xpath('//text()').extract()
        pic_url = Selector(text = review).xpath('//div[3]/div[1]/div[1]/a/img/@data-src').extract()
        item['anime_title'] = anime_title
        item['anime_url'] = anime_url
        item['review_time'] = review_time
        item['reviewer'] = reviewer_name
        item['rating'] = rating
        item['review_text'] = review_text
        item['pic_url'] = pic_url
        yield item

這是爬網后的日志

2018-06-22 13:37:14 [scrapy.core.engine] INFO: Closing spider (finished)
2018-06-22 13:37:14 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 698849,
 'downloader/request_count': 1791,
 'downloader/request_method_count/GET': 1791,
 'downloader/response_bytes': 148209070,
 'downloader/response_count': 1791,
 'downloader/response_status_count/200': 1791,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2018, 6, 22, 11, 37, 14, 546133),
 'log_count/DEBUG': 1792,
 'log_count/INFO': 13,
 'request_depth_max': 1,
 'response_received_count': 1791,
 'scheduler/dequeued': 1790,
 'scheduler/dequeued/memory': 1790,
 'scheduler/enqueued': 1790,
 'scheduler/enqueued/memory': 1790,
 'start_time': datetime.datetime(2018, 6, 22, 11, 30, 38, 403920)}
2018-06-22 13:37:14 [scrapy.core.engine] INFO: Spider closed (finished)

如果您需要更多信息，請告訴我。

Answer 1

這里最大的問題是您的xpath表達式。
它們看起來像是自動生成的，而且過於具體。

例如， reviews的xpath都不匹配任何東西。
像//div[@class="borderDark"]這樣簡單的事情就可以匹配頁面上的所有50條評論，css表達式.borderDark 。

我建議您熟悉xpath和/或CSS選擇器，並手動編寫選擇器。

另外，您正在將選擇器轉換為文本（使用.extract ），然后又轉換回選擇器（使用Selector ）。 無需這樣做，您只需使用.xpath返回的選擇器.xpath 。