Celery + SQS - pycurl error

Question

Today I have been trying to setup Celery using AWS SQS as the broker, however upon excututing the following:

test.py

from celery import Celery

access_key_id = '********************'
secret_access_key = '****************************************'

broker_url = 'sqs://%s:%s@' %(access_key_id, secret_access_key)

app = Celery('test', backend=None, broker=broker_url)

@app.task
def add(x, y):
    return x + y

Attempting to run the Celery worker using the command

celery -A test worker --loglevel=info

With this I get the following error:

Unrecoverable error: ImportError('The curl client requires the pycurl library.',)

I have the following packages installed:

amqp (2.2.2)
billiard (3.5.0.3)
boto (2.48.0)
boto3 (1.4.7)
botocore (1.7.48)
celery (4.1.0)
docutils (0.14)
jmespath (0.9.3)
kombu (4.1.0)
pip (9.0.1)
pycurl (7.43.0)
python-dateutil (2.6.1)
pytz (2017.3)
s3transfer (0.1.11)
schedule (0.5.0)
setuptools (37.0.0)
six (1.11.0)
vine (1.1.4)
wheel (0.30.0)

Can anyone see what I am doing wrong? Thanks in advance!

Answer 1

Looks like it could be an issue with the version of Celery (4.1.0) that you have. If I execute the same code after downgrading to version 3.1.25 it works fine.

Answer 2

I had the same error and these steps worked for me:

mac:

brew upgrade openssl

export LDFLAGS="-L/usr/local/opt/openssl/lib"
export CPPFLAGS="-I/usr/local/opt/openssl/include"   

export PYCURL_SSL_LIBRARY=openssl

python -m pip install celery[sqs]

centos:

pip uninstall pycurl
export PYCURL_SSL_LIBRARY=nss
pip install --compile pycurl

Answer 3

If you are on MAC, you need to compile and install pycurl. You might need to install openssl if not already installed.

brew install openssl


PYCURL_SSL_LIBRARY=openssl LDFLAGS="-L/usr/local/opt/openssl/lib" CPPFLAGS="-I/usr/local/opt/openssl/include" pip install --no-cache-dir pycurl

Similar for linux. Check this out on http://pycurl.io/docs/latest/install.html#ssl

Answer 4

This works on Debian

sudo apt install libcurl4-openssl-dev libssl-dev
pip uninstall pycurl
pip install pycurl --compile --global-option="--with-openssl" --no-cache-dir