Unable to run scrapy for Python

Question

I am new to this platform as well as to Python scraping. I hope that my question will still be understandable and somebody can help me. Sorry, in case I make something unclear...

I have already checked other posts on a similar topic but could not manage to overcome my problem. I am currently getting into web-scraping and wanted to try Scrapy. Therefore, I followed the installation instructions on the website. http://doc.scrapy.org/en/0.16/intro/install.html#intro-install After I had figured out how it works I decided to run in a virtual environment.

I installed virtualenv and pip. Then I installed Scrapy.

When I now want to start with the tutorial

scrapy startproject tutorial

I get the following error message:

    File "/Users/XXX/environment_trial/bin/scrapy", line 3, in <module>
    from scrapy.cmdline import execute
  File "/Users/XXX/environment_trial/lib/python2.7/site-packages/scrapy/cmdline.py", line 7, in <module>
    from scrapy.crawler import CrawlerProcess
  File "/Users/XXX/environment_trial/lib/python2.7/site-packages/scrapy/crawler.py", line 3, in <module>
    from twisted.internet import reactor, defer
ImportError: No module named twisted.internet
(environment_trial)XXX-iMac:~ XXX$ 

I could not find a Twisted.py on my Mac as suggested by other posts.

Can somebody please tell me what to do?

Answer 1

Simply put, you need to install twisted. You can get it from the download page . It looks like you'll need to install from source on a newer Mac, but that's just a case of extracting the tarball and running python setup.py install in the extracted folder.

edit: Since you already have pip installed, you can also grab twisted with it.

pip -E twisted_env install -U twisted

Answer 2

Please be sure that all the binaries that you are installing correspond to the exact same version of python that you have installed (For eg python 2.7 ).

I did this mistake of installing pyopenSSL for python3.6 and it took me a lomng time to realize that versions did not match.