unable to run scrapy in ec2

Question

I'm trying to run a code on an ec2 server. It is a python scrapy project which is executed fine on my own pc. when trying to run it in the ec2 i get this:

Traceback (most recent call last):
File "/usr/local/bin/scrapy", line 4, in <module>
execute()
File "/usr/local/lib/python2.7/site-packages/scrapy/cmdline.py", line 122, in execute
cmds = _get_commands_dict(settings, inproject)
File "/usr/local/lib/python2.7/site-packages/scrapy/cmdline.py", line 46, in _get_commands_dict
cmds = _get_commands_from_module('scrapy.commands', inproject)
File "/usr/local/lib/python2.7/site-packages/scrapy/cmdline.py", line 29, in _get_commands_from_module
for cmd in _iter_command_classes(module):
File "/usr/local/lib/python2.7/site-packages/scrapy/cmdline.py", line 20, in _iter_command_classes
for module in walk_modules(module_name):
File "/usr/local/lib/python2.7/site-packages/scrapy/utils/misc.py", line 68, in walk_modules
submod = import_module(fullpath)
File "/usr/local/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/usr/local/lib/python2.7/site-packages/scrapy/commands/bench.py", line 3, in <module>
from scrapy.tests.mockserver import MockServer
File "/usr/local/lib/python2.7/site-packages/scrapy/tests/mockserver.py", line 6, in <module>
from twisted.internet import reactor, defer, ssl
File "/usr/local/lib/python2.7/site-packages/twisted/internet/ssl.py", line 59, in <module>
from OpenSSL import SSL
File "/usr/local/lib/python2.7/site-packages/OpenSSL/__init__.py", line 8, in <module>
from OpenSSL import rand, crypto, SSL
File "/usr/local/lib/python2.7/site-packages/OpenSSL/rand.py", line 11, in <module>
from OpenSSL._util import (
File "/usr/local/lib/python2.7/site-packages/OpenSSL/_util.py", line 4, in <module>
binding = Binding()
File "/usr/local/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 87, in __init__
self._ensure_ffi_initialized()
File "/usr/local/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 106, in _ensure_ffi_initialized
libraries=libraries,
File "/usr/local/lib/python2.7/site-packages/cryptography/hazmat/bindings/utils.py", line 39, in build_ffi
ffi = cffi.FFI()
File "/usr/local/lib/python2.7/site-packages/cffi/api.py", line 56, in __init__
**import _cffi_backend as backend
ImportError: libffi.so.5: cannot open shared object file: No such file or directory**

I've tried to update && reinstall all the mentioned libraries.

it always says it is most updated version.

this is my linux distro and default python is 2.7:

#  cat /etc/*-release
NAME="Amazon Linux AMI"
VERSION="2015.03"
ID="amzn"
ID_LIKE="rhel fedora"
VERSION_ID="2015.03"
PRETTY_NAME="Amazon Linux AMI 2015.03"
ANSI_COLOR="0;33"
CPE_NAME="cpe:/o:amazon:linux:2015.03:ga"
HOME_URL="http://aws.amazon.com/amazon-linux-ami/"
Amazon Linux AMI release 2015.03

UPDATE: it looks like some sort of problem in scrapy dependencies. trying

 pip install scrapy 

gave me the following error:

creating /usr/lib/python2.6/dist-packages/cssselect
error: could not create '/usr/lib/python2.6/dist-packages/cssselect':       Permission denied

----------------------------------------
Command "/usr/bin/python2.6 -c "import setuptools,   
tokenize;__file__='/tmp/pip-build- aoghBl/cssselect/setup.py'
;exec(compile(getattr(tokenize, 'open', open)
(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))"
install --record /tmp/pip-1Qpx2a-record/install-record.txt
--single-version-externally-managed --compile" 
failed with error code 1 in /tmp/pip-build-aoghBl/cssselect

Answer 1

Your updated question is a permissions error. you likely need to run

sudo pip install cssselect

May need to do that for other dependancies. It's strange though that it's trying to install under python2.6...

Edit: Given that it's trying to run under 2.6, something is wrong with your instance. I'd suggest terminating and recreating the instance. The following commands are what I ran to get scrapy dependancies and my project up and running

sudo su
yum groupinstall 'Development Tools'
yum install -y libffi-devel libxslt-devel libxml2-devel openssl-devel python-devel mysql-devel

Answer 2

I also had problems getting scrapy installed. I found the following pages helpful:

• How to Install Scrapy in Linux with out Any Errors
• Installing Scrapy on an Amazon CentOS AMI

In the end, I had to launch a "fresh" Amazon Linux instance, which successfully installed with instructions from those pages. My older instance had too many issues to resolve.