Can't get Pandas to install with OpenShift

Question

I am ssh-ing a Flask application on OpenShift and one of the Python dependencies is Pandas 0.16.1. Looking through the OpenShift documentation , I created my setup.py file as follows:

from setuptools import setup

setup(name='MyApp',
    version='0.1a',
    description='some description',
    author='me',
    author_email='me@gmail.com',
    url='http://myapp.com/',
    install_requires=['Flask>=0.10.1','numpy>=1.9.2','pandas>=0.16.1'],
    )

When creating the app using the requires, the process fails. The following error is produced... help!

The initial build for the application failed: Shell command
'/sbin/runuser -s /bin/sh 557ba9e9e0b8cd360b000131 -c "exec
/usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c1,c69' /bin/sh
-c \"gear postreceive --init >> /tmp/initial-build.log 2>&1\""' returned an error. rc=137 .Last 10 kB of build output: Stopping Python
2.7 cartridge Repairing links for 1 deployments Building git ref 'master', commit 239ba5f Activating virtenv Running setup.py script..
running develop running egg_info creating GTFS_Viewer.egg-info writing
requirements to GTFS_Viewer.egg-info/requires.txt writing
GTFS_Viewer.egg-info/PKG-INFO writing top-level names to
GTFS_Viewer.egg-info/top_level.txt writing dependency_links to
GTFS_Viewer.egg-info/dependency_links.txt writing manifest file
'GTFS_Viewer.egg-info/SOURCES.txt' reading manifest file
'GTFS_Viewer.egg-info/SOURCES.txt' writing manifest file
'GTFS_Viewer.egg-info/SOURCES.txt' running build_ext Creating
/var/lib/openshift/557ba9e9e0b8cd360b000131/app-root/runtime/dependencies/python/virtenv/lib/python2.7/site-packages/GTFS-Viewer.egg-link
(link to .) Adding GTFS-Viewer 0.1a to easy-install.pth file Installed
/var/lib/openshift/557ba9e9e0b8cd360b000131/app-root/runtime/repo
Processing dependencies for GTFS-Viewer==0.1a Searching for
pandas>=0.16.1 Reading
http://mirror1.ops.rhcloud.com/mirror/python/web/simple/pandas/ Best
match: pandas 0.16.1 Downloading
mirror1.ops.rhcloud.com/mirror/python/web/packages/source/p/pandas/pandas-0.16.1.zip#md5=d465643d588c4f886b8e796ae56673ad
Processing pandas-0.16.1.zip Writing
/tmp/easy_install-97qgiA/pandas-0.16.1/setup.cfg Running
pandas-0.16.1/setup.py -q bdist_egg --dist-dir
/tmp/easy_install-97qgiA/pandas-0.16.1/egg-dist-tmp-VpNuur warning: no
files found matching 'README.rst' no previously-included directories
found matching 'doc/build' warning: no directories found matching
'examples' warning: no previously-included files matching '*.so' found
anywhere in distribution warning: no previously-included files
matching '*.pyd' found anywhere in distribution warning: no
previously-included files matching '*.pyc' found anywhere in
distribution warning: no previously-included files matching '*~' found
anywhere in distribution warning: no previously-included files
matching '#*' found anywhere in distribution warning: no
previously-included files matching '.git*' found anywhere in
distribution warning: no previously-included files matching
'.DS_Store' found anywhere in distribution warning: no
previously-included files matching '*.png' found anywhere in
distribution In file included from
/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1728,
from
/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:17,
from
/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:15,
from pandas/index.c:250:
/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/npy_deprecated_api.h:11:2:
warning: #warning "Using deprecated NumPy API, disable it by #defining
NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" pandas/index.c: In function
'__pyx_f_6pandas_5index_11IndexEngine__maybe_get_bool_indexer':
pandas/index.c:4088: warning: '__pyx_v_last_true' may be used
uninitialized in this function pandas/index.c: In function
'__pyx_f_6pandas_5index_13Float64Engine__maybe_get_bool_indexer':
pandas/index.c:7981: warning: '__pyx_v_last_true' may be used
uninitialized in this function pandas/index.c: In function
'__pyx_f_6pandas_5index_11Int64Engine__maybe_get_bool_indexer':
pandas/index.c:7356: warning: '__pyx_v_last_true' may be used
uninitialized in this function pandas/index.c: In function
'__pyx_f_6pandas_5index__bin_search': pandas/index.c:8769: warning:
'__pyx_v_mid' may be used uninitialized in this function In file
included from
/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1728,
from
/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:17,
from
/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:15,
from pandas/src/datetime/np_datetime.c:18:
/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/npy_deprecated_api.h:11:2:
warning: #warning "Using deprecated NumPy API, disable it by #defining
NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION"
/opt/rh/python27/root/usr/include/python2.7/datetime.h:188: warning:
'PyDateTimeAPI' defined but not used In file included from
/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1728,
from
/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:17,
from
/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:15,
from pandas/src/datetime/np_datetime_strings.c:17:
/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/npy_deprecated_api.h:11:2:
warning: #warning "Using deprecated NumPy API, disable it by #defining
NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION"
pandas/src/datetime/np_datetime_strings.c: In function
'make_iso_8601_datetime':
pandas/src/datetime/np_datetime_strings.c:1147: warning: format
'%04ld' expects type 'long int', but argument 4 has type 'long long
int' pandas/src/datetime/np_datetime_strings.c:1147: warning: format
'%04ld' expects type 'long int', but argument 4 has type 'long long
int' pandas/src/datetime/np_datetime_strings.c: At top level:
pandas/src/datetime/np_datetime_strings.c:127: warning: 'get_gmtime'
defined but not used In file included from
/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1728,
from
/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:17,
from
/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:15,
from pandas/src/period.c:251:
/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/npy_deprecated_api.h:11:2:
warning: #warning "Using deprecated NumPy API, disable it by #defining
NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION"
/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/__multiarray_api.h:1594:
warning: '_import_array' defined but not used
/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/__ufunc_api.h:236:
warning: '_import_umath' defined but not used In file included from
/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1728,
from
/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:17,
from
/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:15,
from pandas/src/datetime/np_datetime.c:18:
/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/npy_deprecated_api.h:11:2:
warning: #warning "Using deprecated NumPy API, disable it by #defining
NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION"
/opt/rh/python27/root/usr/include/python2.7/datetime.h:188: warning:
'PyDateTimeAPI' defined but not used In file included from
/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1728,
from
/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:17,
from
/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:15,
from pandas/src/datetime/np_datetime_strings.c:17:
/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/npy_deprecated_api.h:11:2:
warning: #warning "Using deprecated NumPy API, disable it by #defining
NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION"
pandas/src/datetime/np_datetime_strings.c: In function
'make_iso_8601_datetime':
pandas/src/datetime/np_datetime_strings.c:1147: warning: format
'%04ld' expects type 'long int', but argument 4 has type 'long long
int' pandas/src/datetime/np_datetime_strings.c:1147: warning: format
'%04ld' expects type 'long int', but argument 4 has type 'long long
int' pandas/src/datetime/np_datetime_strings.c: At top level:
pandas/src/datetime/np_datetime_strings.c:127: warning: 'get_gmtime'
defined but not used In file included from
/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1728,
from pandas/src/period_helper.h:12, from pandas/src/period_helper.c:1:
/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/npy_deprecated_api.h:11:2:
warning: #warning "Using deprecated NumPy API, disable it by #defining
NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION"
pandas/src/period_helper.c:33: warning: 'NULL_AF_INFO' defined but not
used In file included from
/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1728,
from
/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:17,
from
/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:15,
from pandas/algos.c:250:
/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/npy_deprecated_api.h:11:2:
warning: #warning "Using deprecated NumPy API, disable it by #defining
NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION"

Answer 1

I have had the same problem: Installing pandas through requirements.txt or using pip did not work.

Openshift stops and kills the process when trying to compile algos.c . Here is the manual install workaround that did the trick for me (python3.3 cartridge). It is the result of an somewhat tedious try-and-error process, and I cannot really explain why it works. But you might want to give it a try.

1. Connect to openshift: rhc ssh <appname> .

2. Download latest pandas source into the /tmp directory to install manually

    cd /tmp mkdir pandas cd pandas wget https://pypi.python.org/packages/source/p/pandas/pandas-0.16.2.zip#md5=860a6c7e5e1a24bb0aa549b115830252 unzip pandas-0.16.2.zip cd pandas-0.16.2 

3. Edit the setup.py file using your favourite (available) editor, eg emacs and search for "algos". The second time it appears comment out the algos package in the variable ext_data :

    ... index={'pyxfile': 'index', 'sources': ['pandas/src/datetime/np_datetime.c', 'pandas/src/datetime/np_datetime_strings.c']}, # algos={'pyxfile': 'algos', # 'depends': [srcpath('generated', suffix='.pyx'), # srcpath('join', suffix='.pyx')]}, parser={'pyxfile': 'parser', 'depends': ['pandas/src/parser/tokenizer.h', 'pandas/src/parser/io.h', 'pandas/src/numpy_helper.h'], ... 

4. Run the first part of the installation procedure:

    ./setup.py install 

5. Now, let us compile the missing algos piece by hand. To do so, remove the comments you made in step 2, and comment out all other entries (except the algos entry) in ext_data .

6. Run setup in "dry-run" mode to see the two gcc commands you need to compile algos

    ./setup.py -n build 

7. Copy the first gcc command to run it from shell. When I tried this, this command made openshift shut down the connection. So, I adjusted it by removing doublicate options and remove the -O2 flag (ie, the optimization). This worked for me:

    gcc -pthread -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -I/opt/rh/python33/root/usr/include -Ipandas/src/klib -Ipandas/src -I/opt/rh/python33/root/usr/lib64/python3.3/site-packages/numpy/core/include -I$OPENSHIFT_HOMEDIR/python/virtenv/venv/include -I/opt/rh/python33/root/usr/include/python3.3m -c pandas/algos.c -o build/temp.linux-x86_64-3.3/pandas/algos.o 

8. The second command can be run unchanged. For me it was:

    gcc -pthread -shared -L/opt/rh/python33/root/usr/lib64 -L/usr/lib6464 build/temp.linux-x86_64-3.3/pandas/algos.o -L/opt/rh/python33/root/usr/lib64 -lpython3.3m -o build/lib.linux-x86_64-3.3/pandas/algos.cpython-33m.so 

9. Now, the hand-made files need to be installed:

    ./setup.py install 

10. Quick check if it worked for you. First cd to your home directory, then try python:

     cd python Python 3.3.2 (default, Mar 20 2014, 20:25:51) [GCC 4.4.6 20120305 (Red Hat 4.4.6-4)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import pandas 

11. Delete the /tmp/pandas directory.

     rm -rf /tmp/pandas 

Hope this works for you, too!

Answer 2

Can you do that manually ? ie :

ssh 12hh3456789hh01011@nameofapp-username.rhcloud.com

And activate the environment :

source app-root/runtime/dependencies/python/virtenv/bin/activate

And then install the dependencies. First Numpy

pip install -U numpy

And then Pandas:

pip install pandas

Answer 3

Restart Openshift Application from web console and commit the changes and deploy. it worked for me!