在Scrapy中使用规则和请求会引发异常TypeError：__init __（）为关键字参数'callback'获得了多个值

Question

I am a beginner in scrapy, and I have been trying to do the following workflow: Start from page A , which is a result search page containing links for full articles whose url end is a digit. My intention is to grab each link of each result search page, access the links and scrap the full article.

I iterate over each page collecting links with the following rule:

   rules = (Rule(SgmlLinkExtractor(allow=(r'\d+',)), callback='parse_short_story',follow=True),)

Each ensures that the last digit of the search page iterates to the next one after I am done collecting the links and scrapping the full articles of the current page.

The parse_short_story method merely uses a select to filter the portion of the html page which, and afterwards loops over the remaining portion to acquire the links of the full stories and pass it on for the request:

for short_story in short_stories:
        item = DmozItem()

        full_story_link = short_story.select(".//h2/a/@href").extract()

        if full_story_link:
            yield Request(full_story_link, self.parse_full_story, callback='self.parse_full_story', errback=lambda _: item, meta=dict(item=item),)            

        items.append(item)
    return items     

On my understanding from the tutorial of scrapy, I need to return the items by the end of the parser methods, so that the rule properly append in a final list of items which I can throw in a json file or something else on running by the console. Notice this portion below of Response and return calls which crashes. I can't figure out how to use both the Request and the return items.

The method parse_full_story gets the response parameter like the parse_short_story does, and recover the item I send as parameter with

    item = response.meta.get('item')

After properly setting the information I desired on my item item I use return item .

In summary,

My expectation were that the rule would take care of moving along the search pages containing the links of the full article using the callback of parse_short_story , while for each link of each page, the parse_full_story would access the full articles of those links, scrap what I wanted, add to the item item, and exit, hopefully scanning all full articles in the end.

Apparently my understanding is wrong and I get the error: yield Request(full_story_link, self.parse_full_story, callback='self.parse_full_story', errback=lambda _: item, meta=dict(item=item),) exceptions.TypeError: __init__() got multiple values for keyword argument 'callback' You can find the full runable code here . As it runs, you will see that it keeps throwing the exception. If it is feasible to perform a direct fix and give/or a short explanation of what is wrong on this I would appreciate it, since similar problems lead me to Django associated questions on the web.

Answer 1

Put only one callback parameter, use the self.parse_full_story ( Request() expects a callable; see here )

The "callback name string" version is only for Rules ( see here )

Use

yield Request(full_story_link,
    self.parse_full_story,
    errback=lambda _: item,
    meta=dict(item=item),
)

instead of

yield Request(full_story_link,
    self.parse_full_story, callback='self.parse_full_story',
    errback=lambda _: item,
    meta=dict(item=item),
)