尝试解析列表数据时，Scrapy抛出Traceback

Question

I am running Scrapy.org version 2.7 64 bit on Windows Vista 64 bit. I have some Scrapy code that is trying parse data contained within a table at the URL contained within the following code:

from scrapy.spider import Spider
from scrapy.selector import Selector
from scrapy.utils.markup import remove_tags
from scrapy.cmdline import execute
import re


class MySpider(Spider):
    name = "wiki"
    allowed_domains = ["whoscored.com"]
    start_urls = ["http://www.whoscored.com/Players/3859/Fixtures/Wayne-Rooney"]

def parse(self, response):

    for row in response.selector.xpath('//table[@id="player-fixture"]//tr[td[@class="tournament"]]'):
    # Is this row contains goal symbols?
        list_of_goals = row.xpath('//span[@title="Goal"')
        if list_of_goals:
            print remove_tags(list_of_goals).encode('utf-8')     

execute(['scrapy','crawl','wiki'])

However, it is throwing up the following error:

Traceback (most recent call last):
  File "c:\Python27\lib\site-packages\twisted\internet\base.py", line 1201, in mainLoop
    self.runUntilCurrent()
  File "c:\Python27\lib\site-packages\twisted\internet\base.py", line 824, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File "c:\Python27\lib\site-packages\twisted\internet\defer.py", line 383, in callback
    self._startRunCallbacks(result)
  File "c:\Python27\lib\site-packages\twisted\internet\defer.py", line 491, in _startRunCallbacks
    self._runCallbacks()
--- <exception caught here> ---
  File "c:\Python27\lib\site-packages\twisted\internet\defer.py", line 578, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "c:\Python27\lib\site-packages\scrapy\spider.py", line 56, in parse
    raise NotImplementedError
exceptions.NotImplementedError:

Can anyone tell me what the issue is here? I am trying to get a screen print of all items in the table, including the data in the goals and assists column.

Thanks

Answer 1

Your indentation is wrong:

class MySpider(Spider):
    name = "wiki"
    allowed_domains = ["whoscored.com"]
    start_urls = ["http://www.whoscored.com/Players/3859/Fixtures/Wayne-Rooney"]

    def parse(self, response):

        for row in response.selector.xpath('//table[@id="player-fixture"]//tr[td[@class="tournament"]]'):
        # Is this row contains goal symbols?
            list_of_goals = row.xpath('//span[@title="Goal"')
            if list_of_goals:
                print remove_tags(list_of_goals).encode('utf-8')

Implementing a parse method is a requirement when you use the Spider class , this is what the method is like in the source code:

def parse(self, response):
        raise NotImplementedError

Your indentation was wrong so parse was not part of the class and therefore you had not implemented the required method.

The raise NotImplementedError is there to ensure you write the required parse method when inheriting from the Spider base class.

You now just have to find the correct xpath ;)