Errors and Warnings when installing Scrapy with pip on OSX

Question

I want to install Scrapy on my macbook OSX 10.9

I followed all of the prerequisites listed here: http://doc.scrapy.org/en/latest/intro/install.html

And then typed the following into the terminal: pip install Scrapy

At the very bottom of the terminal it says:

2 warnings and 6 errors generated.

error: command '/usr/bin/clang' failed with exit status 1

And then below that it says:

----------------------------------------
Cleaning up...
Command /Library/Frameworks/Python.framework/Versions/3.3/bin/python3 -c "import setuptools, tokenize;__file__='/private/var/folders/sj/fs5lzbyd7p37_jdkqw03_q4c0000gn/T/pip_build_johnmosely/Twisted/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/sj/fs5lzbyd7p37_jdkqw03_q4c0000gn/T/pip-lkdl1g-record/install-record.txt --single-version-externally-managed --compile failed with error code 1 in /private/var/folders/sj/fs5lzbyd7p37_jdkqw03_q4c0000gn/T/pip_build_johnmosely/Twisted
Storing debug log for failure in /Users/johnmosely/.pip/pip.log

I went and looked and there is now a folder on my macbook called "Scrapy" and it contains a zip file called "scrapy-scrapy-0.22.0-7-gcde9a8c.zip" but that's it so I'm assuming that it was not installed successfully?

Or should I just ignore the errors and warnings and unzip the zip file?

Thanks for the help.

EDIT: I know for a fact it's not installed because when I try to start a new scrapy project the terminal gives me this output:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.3/bin/scrapy", line 3, in <module>
    from scrapy.cmdline import execute
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/scrapy/__init__.py", line 43, in <module>
    from twisted import version as _txv
ImportError: No module named 'twisted'

EDIT 2 - Here are the 6 errors that were printed:

^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
twisted/python/sendmsg.c:129:9: error: non-void function 'initsendmsg' should return a value [-Wreturn-type]
        return;
        ^
twisted/python/sendmsg.c:138:9: error: non-void function 'initsendmsg' should return a value [-Wreturn-type]
        return;
        ^
twisted/python/sendmsg.c:145:9: error: non-void function 'initsendmsg' should return a value [-Wreturn-type]
        return;
        ^
twisted/python/sendmsg.c:159:9: error: non-void function 'initsendmsg' should return a value [-Wreturn-type]
        return;
        ^
twisted/python/sendmsg.c:165:9: error: non-void function 'initsendmsg' should return a value [-Wreturn-type]
        return;
        ^
twisted/python/sendmsg.c:170:9: error: non-void function 'initsendmsg' should return a value [-Wreturn-type]
        return;

Answer 1

Scrapy needs twisted to work and it is trying to compile it but runnning into errors. I have found bug tracker for this here . That bug is still open as far as I can see, so on Python 3.3 it is not possible to install twisted right now. You might like to try installing it with Python 2.7.

Answer 2

Unfortunately, Scrapy doesn't support Python 3 yet - porting is not started because of twisted issues. Scrapy 0.22 only works with Python 2.7.

Answer 3

To successfully install Scrapy, you need to make sure that you have already installed the tool Twisted.

I met the same problem,"error: command '/usr/bin/clang' failed with exit status 1" , when I tried to install Scrapy and Twisted in my mac.

This problem was fixed after I installed the Xcode. I'm not sure if you use MacOS system like me. If yes, this may help you.

command line:

xcode-select --install