如何在Travis CI下安装Vowpal Wabbit Python绑定？

Question

I'm trying to use Vowpal Wabbit from my Python projects. I can install the Python VW bindings locally (in Ubuntu 18.04) using pip install vowpalwabbit , as long as I've first installed the build dependencies ( apt install libboost-program-options-dev libboost-python-dev zlib1g-dev ).

I want to run unit tests for Python programs under Travis CI, so I need to be able to install dependencies within the Travis environment as well. However, I cannot get the Python VW bindings to install. I created a minimal project demonstrating the problem. I'm using the most recent xenial (ie Ubuntu 16.04 based) image of Travis and Python versions 3.5, 3.6 and 3.7. Here's the Travis CI output .

The Python 3.5 issue seems to be caused by linking to the wrong (Python 2.7) version of libboost_python.so (see eg this issue ) when the python extension is built. That issue had a workaround which involved changing the symlink libboost_python.so to point to the Python 3.5 version (instead for the 2.7 version). I tried that on the boost-py2-py3-workaround branch, but it didn't seem to help .

Here is the .travis.yml config:

dist: xenial
language: python
python:
- '3.5'
- '3.6'
- '3.7'
cache: pip
install:
- sudo apt-get install libboost-program-options-dev libboost-python-dev zlib1g-dev
- pip install vowpalwabbit
script:
- python test-vw.py

The test-vw.py script just imports the pyvw module and runs some basic commands (taken from the Python VW documentation):

from vowpalwabbit import pyvw

vw = pyvw.vw(quiet=True)
ex = vw.example('1 | a b c')
vw.learn(ex)
print(vw.predict(ex))

I would expect the pip install to succeed and then the test-vw.py should run and produce a little bit of output as the model is created and its prediction is printed.

The actual result depends on the Python version. For Python 3.5, the vowpalwabbit library seems to be successfully installed but running the Python script fails with this error ( full build log ):

$ python test-vw.py Traceback (most recent call last): File "test-vw.py", line 6, in <module> from vowpalwabbit import pyvw File "/home/travis/virtualenv/python3.5.6/lib/python3.5/site-packages/vowpalwabbit/pyvw.py", line 2, in <module> import pylibvw ImportError: /usr/lib/x86_64-linux-gnu/libboost_python-py27.so.1.58.0: undefined symbol: PyClass_Type The command "python test-vw.py" exited with 1.

For Python 3.6 and 3.7, the pip install fails ( full build log ):

  Using Python 3.6
  pyenv: python3.6-config: command not found

  The `python3.6-config' command exists in these Python versions:
    3.6
    3.6.7

  /usr/bin/g++ -std=c++0x  -I /usr/local/include/boost -I /usr/include -I ../rapidjson/include -fPIC -c pylibvw.cc -o pylibvw.o
  In file included from /usr/include/boost/python/detail/prefix.hpp:13:0,
                   from /usr/include/boost/python/args.hpp:8,
                   from /usr/include/boost/python.hpp:11,
                   from pylibvw.cc:14:
  /usr/include/boost/python/detail/wrap_python.hpp:50:23: fatal error: pyconfig.h: No such file or directory

So there seems to be two separate issues, depending on which Python version is being tried. I'm happy for help with either issue.

Answer 1

I got it working for the Python 3.5 case. The crucial part was changing the symlinks to point to the Python 3.5 version of libboost_python.so, like this in .travis.yml:

install:
- sudo apt-get install libboost-program-options-dev libboost-python-dev zlib1g-dev
- sudo ln -sf /usr/lib/x86_64-linux-gnu/libboost_python-py35.a /usr/lib/x86_64-linux-gnu/libboost_python.a
- sudo ln -sf /usr/lib/x86_64-linux-gnu/libboost_python-py35.so /usr/lib/x86_64-linux-gnu/libboost_python.so
- pip install vowpalwabbit

Here is the working code on the boost-py2-py3-workaround branch, and the Travis build output .

This is enough to have tests running at least for one Python version, which is OK for now. For the other Python versions (3.6 and 3.7) the problem is that there is no easily available precompiled version of libboost-python that could just be installed into the Travis CI virtual environment. The libboost-python libraries available via apt-get are only available for Python 3.5, which is the default Python 3 version available in Ubuntu 16.04 Xenial.