Issues installing Python library with binary dependency in virtualenv without sudo

Question

In my Python 2.7 project I need to install the ta-lib library. On the target environment I do not have root or sudo permissions, therfore the Python application runs in a virtual environment.

For unknown reasons the installation of the ta-lib library with pip fails, when libta_lib.so resides in the user directory structure (instead of the system's /usr folder).

What I did:

I'm using Python 2.7 to create a fresh virtual environment:

ec2-user:~/environment $ python -V
Python 2.7.14

ec2-user:~/environment $ virtualenv -p /usr/bin/python27 my_env
Running virtualenv with interpreter /usr/bin/python27
New python executable in /home/ec2-user/environment/my_env/bin/python27
Also creating executable in /home/ec2-user/environment/my_env/bin/python
Installing setuptools, pip, wheel...done.

ec2-user:~/environment $ source my_env/bin/activate
(my_env) ec2-user:~/environment $ 

Within my_env I download and untar the latest ta-lib . I use the prefix-flag to configure the installation path to be within the virtual env.

(my_env) ec2-user:~/environment/my_env $ ./configure --prefix=/home/ec2-user/environment/my_env
(my_env) ec2-user:~/environment/my_env $ make
(my_env) ec2-user:~/environment/my_env $ make install

Configure, make and install work just fine. At the end the system tells me someting like:

Libraries have been installed in:
   /home/ec2-user/environment/my_env/lib

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the `-LLIBDIR'
flag during linking and do at least one of the following:
   - add LIBDIR to the `LD_LIBRARY_PATH' environment variable
     during execution
   - add LIBDIR to the `LD_RUN_PATH' environment variable
     during linking
[...]

So I added /home/ec2-user/environment/my_env/lib to $LD_LIBRARY_PATH and $LD_RUN_PATH. So far everyting looks fine, but when I

(my_env) ec2-user:~/environment/my_env $ pip install ta-lib
Collecting ta-lib
  Using cached https://files.pythonhosted.org/packages/[...]/
TA-Lib-0.4.17.tar.gz
Requirement already satisfied: numpy in ./lib/python2.7/dist-packages 
(from ta-lib) (1.15.1)
Building wheels for collected packages: ta-lib
  Running setup.py bdist_wheel for ta-lib ... error
  Complete output from command /home/ec2-user/environment/my_env/bin/python27 
-u -c "import setuptools, tokenize;__file__='/tmp/pip-install-MD3Ds7/ta- 
lib/setup.py';f=getattr(tokenize, 'open', open) 
(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, 
 __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-dtdhyb --python-tag cp27:
  /tmp/pip-install-MD3Ds7/ta-lib/setup.py:79: UserWarning: Cannot find ta-lib 
 library, installation may fail.
    warnings.warn('Cannot find ta-lib library, installation may fail.')

and ...

creating build/temp.linux-x86_64-2.7/talib
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/home/ec2-user/environment/my_env/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/include -I/usr/local/include -I/opt/include -I/opt/local/include -I/usr/include/python2.7 -c talib/_ta_lib.c -o build/temp.linux-x86_64-2.7/talib/_ta_lib.o
talib/_ta_lib.c:526:28: fatal error: ta-lib/ta_defs.h: No such file or directory
 #include "ta-lib/ta_defs.h"
                            ^
compilation terminated.
error: command 'gcc' failed with exit status 1

----------------------------------------
Command "/home/ec2-user/environment/my_env/bin/python27 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-MD3Ds7/ta-lib/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-YqZ5hO/install-record.txt --single-version-externally-managed --compile --install-headers /home/ec2-user/environment/my_env/include/site/python2.7/ta-lib" failed with error code 1 in /tmp/pip-install-MD3Ds7/ta-lib/
[...]

I also tried adding "/home/ec2-user/environment/my_env/lib" to the $PATH, but the result is the same.

Why won't pip find the librar which resides in the lib folder inside the virtual environment?

When I install the ta-lib binary into the default /usr/lib folder (using sudo of course), pip install ta-lib will find it and install just fine. Unfortuantely this is not an option for the target system.

Am I doing someting wrong, or is pip ignoring folders and path variables inside the virtual envrionemt?

Regards, Ale

Answer 1

You need to pass custom location of installed headers & shared objects when pip install ing the python bindings:

$ CPPFLAGS='-I/home/ec2-user/environment/my_env/include' \
  LDFLAGS='-L/home/ec2-user/environment/my_env/lib' pip install ta-lib