如何获取 scrapy.http.request.Request？

Question

Note: I've gone through the scrapy tutorial , I'd just like to know how fetch works.

With scrapy shell, this code works well.

>>> import scrapy
>>> url = 'http://quotes.toscrape.com/page/1/'
>>> def parse(response):
...     print('parse %s' % response)
... 
>>> req = scrapy.Request(url=url, callback=parse)
>>> fetch(req)

Which gets me

2020-07-03 05:21:04 [scrapy.core.engine] DEBUG: Crawled (404) <GET http://quotes.toscrape.com/robots.txt> (referer: None)
2020-07-03 05:21:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/1/> (referer: None)
parse <200 http://quotes.toscrape.com/page/1/>

How do I run it in a.py file?

I put the code in fetch_req.py file and run the file with this command

python fetch_req.py

then I got

NameError: name 'fetch' is not defined

I understood this, as fetch is a method of scrapy.shell.Shell instance , so I added this to fetch_req.py .

from scrapy import shell
shell.Shell.fetch(req)

then I got

TypeError                                 Traceback (most recent call last)
<ipython-input-34-914d5e1bbfe3> in <module>()
----> 1 shell.Shell.fetch(req)

TypeError: fetch() missing 1 required positional argument: 'request_or_url'

I googled the error but got no hit. How do I fix this?

Answer 1

You will have to yield the request so that scrapy engine puts into it's queue and executes the request.

To do understand this better you should follow @Gallaecio suggestion and follow scrapy's tutorial . It's pretty straightforward.

EDIT

I understand now what you mean, however I don't understand why you want to use scrapy in this way. Surely fetch method wasn't desgined for this.

Anyhow the problem is that you are calling fetch as if it was a staticmethod.

from scrapy import shell
shell.Shell.fetch(req)

You should instantiate the object and then call the method from the object. The missing required argument is raised because it expects self and request_or_url as arguments.

You can also try passing the class as the first argument. (As if it was a classmethod)

Those will likely lead to new exceptions though.