Mutalyzer py.test fails with python-magic error

Question

I am trying to install mutalyzer 2.0.14 on Centos 6.7 using Anaconda Python 2.7 as per these instructions.

My install goes fine, but I get a series of fails when I attempt to run the py.test suite.

My suspicion is that there is a version clash between the OS' version of libmagic and the python2.7 package, python-magic .

I'd be very interested if anyone could explain what is going on here or better still suggest a fix for this problem.

My libmagic installs:

$ rpm -qa | grep 'file-'
file-libs-5.04-21.el6.x86_64
file-5.04-21.el6.x86_64
file-devel-5.04-21.el6.x86_64

Corresponding Python packages:

$ source activate  py27
discarding /opt/anaconda2/bin from PATH
prepending /opt/anaconda2/envs/py27/bin to PATH
(py27) $ pip list | grep magic 
Magic-file-extensions (0.2, /home/chris.guest/temp/mutalyzer/src/magic-file-extensions)
python-magic (0.4.10)

py.test begins like this:

# py.test
======================================================= test session starts =======================================================
platform linux2 -- Python 2.7.11, pytest-2.8.2, py-1.4.30, pluggy-0.3.1
rootdir: /home/chris.guest/temp/mutalyzer, inifile: 
collected 610 items / 3 errors 

tests/test_backtranslator.py ......
tests/test_base.py .
tests/test_crossmap.py .......................
tests/test_grammar.py ........................................................................................
tests/test_mapping.py ..............................
tests/test_migrations.py .
tests/test_mutator.py ................................................................................................................................................................
tests/test_ncbi.py ............................................................................................................
tests/test_parsers_genbank.py ..........

And finishes with a lot of errors. Many tests failing because of undefined symbol: magic_list :

_________________________________________ ERROR at setup of test_batch_unicode[sqlite://] _________________________________________

    @pytest.fixture
    def website():
>       return create_app().test_client()

tests/test_website.py:29: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
mutalyzer/website/__init__.py:35: in create_app
    from mutalyzer.website.views import website
mutalyzer/website/views.py:26: in <module>
    from mutalyzer import (announce, backtranslator, File, Retriever, Scheduler,
mutalyzer/File.py:23: in <module>
    import magic           # open(), MAGIC_MIME, MAGIC_NONE
src/magic-file-extensions/magic.py:94: in <module>
    _list = _libraries['magic'].magic_list
/opt/anaconda2/envs/py27/lib/python2.7/ctypes/__init__.py:378: in __getattr__
    func = self.__getitem__(name)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <CDLL 'libmagic.so.1', handle 160cc50 at 7f23b97340d0>, name_or_ordinal = 'magic_list'

    def __getitem__(self, name_or_ordinal):
>       func = self._FuncPtr((name_or_ordinal, self))
E       AttributeError: /usr/lib64/libmagic.so.1: undefined symbol: magic_list

/opt/anaconda2/envs/py27/lib/python2.7/ctypes/__init__.py:383: AttributeError
______________________________________ ERROR at setup of test_batch_unicode_email[sqlite://] ______________________________________

    @pytest.fixture
    def website():
>       return create_app().test_client()

tests/test_website.py:29: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
mutalyzer/website/__init__.py:35: in create_app
    from mutalyzer.website.views import website
mutalyzer/website/views.py:26: in <module>
    from mutalyzer import (announce, backtranslator, File, Retriever, Scheduler,
mutalyzer/File.py:23: in <module>
    import magic           # open(), MAGIC_MIME, MAGIC_NONE
src/magic-file-extensions/magic.py:94: in <module>
    _list = _libraries['magic'].magic_list
/opt/anaconda2/envs/py27/lib/python2.7/ctypes/__init__.py:378: in __getattr__
    func = self.__getitem__(name)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <CDLL 'libmagic.so.1', handle 160cc50 at 7f23b984ef50>, name_or_ordinal = 'magic_list'

    def __getitem__(self, name_or_ordinal):
>       func = self._FuncPtr((name_or_ordinal, self))
E       AttributeError: /usr/lib64/libmagic.so.1: undefined symbol: magic_list

/opt/anaconda2/envs/py27/lib/python2.7/ctypes/__init__.py:383: AttributeError
============================================== 548 passed, 65 error in 41.42 seconds ==============================================
(py27)[root@localhost mutalyzer]# 

Answer 1

We have one earlier report of a user trying to install Mutalyzer on CentOS 6 and they too ran into troubles with libmagic. I don't think this was resolved, other than them switching to CentOS 7.

Unfortunately we have very limited resources and Debian being the only OS we use ourselves that's all we can realistically support. Of course we would be happy to accept patches or documentation updates :)

Thanks for using Mutalyzer!

Answer 2

The version of magiclib that is packaged with Centos 6.7 is missing the magic_list function.

I was able to make this patch to src/magic-file-extensions/magic.py so that py.test runs without error.

$ diff -u src/magic-file-extensions/magic.old.py src/magic-file-extensions/magic.py
--- src/magic-file-extensions/magic.old.py  2015-12-31 14:04:50.769635453 +1100
+++ src/magic-file-extensions/magic.py  2016-01-11 15:17:49.987270845 +1100
@@ -91,9 +91,13 @@
 _check.restype = c_int
 _check.argtypes = [magic_t, c_char_p]

-_list = _libraries['magic'].magic_list
-_list.restype = c_int
-_list.argtypes = [magic_t, c_char_p]
+try:
+    _list = _libraries['magic'].magic_list
+    _list.restype = c_int
+    _list.argtypes = [magic_t, c_char_p]
+except AttributeError, e:
+    _list = None
+    

 _errno = _libraries['magic'].magic_errno
 _errno.restype = c_int

While the immediate AttributeError is handled with this patch, the Magic.list method in src/magic-file-extensions/magic.py will still fail if it is called. This is not an immediate issue as it doesn't appear to be called within the mutalyzer codebase.