What does CompileError/LinkerError: "command 'gcc' failed with exit status 1" mean, when running %%cython-magic cell in IPython

Question

Sometimes, when I run a %%cython-cell in an IPython-notebook I get a long backtrace with ends with a quite short error message:

CompileError: command 'gcc' failed with exit status 1

or

LinkError: command 'gcc' failed with exit status 1

On Windows the corresponding messages are:

CompileError: command 'C:.\\Microsoft Visual Studio\\..\\cl.exe' failed with exit status X

and

LinkError: command 'C:..\\Microsoft Visual Studio\\..\\link.exe' failed with exit status YYYY

Is it possible to get more precise information about the underlying error?

An example of such a %%cython-cell follows:

[1] %load_ext Cython
[2] %%cython
    from sklearn.tree._tree cimport Node
    print("loaded")

Answer 1

The %%cython magic uses distutils to build the Cython-extension under the hood and IPython doesn't not capture the output gcc or other compilers/linkers log to standard error/output.

In order to see the errors and warnings logged by the compiler/linker, one must go to the place where the errors are logged to by the compiler, which depends on the way the IPython was started.

On Linux there is another possibility: to install wurlitzer package and activate it via %load_ext wurlitzer , which would also capture the output from gcc and show it in the notebook, see also this answer .

Sadly, wurlitzer works only on Linux, while the options bellow work for any OS.

IPython/Jupiter notebook:

When the notebook was started from the terminal, eg via ipython notebook or similar, then the compiler output can be seen in this terminal - we see that the problem with the above cell is:

/home/ed/.cache/ipython/cython/_cython_magic_5f6d267a6f541c572b4517a74d5c9aad.c:607:31: fatal error : numpy/arrayobject.h : No such file or directory compilation terminated.

missing numpy-headers, which can be found in numpy.get_include() .

IPython-console

If IPython is started from a terminal, the errors are logged directly to the IPython console . But be aware: the compiler/linker outputs will come directly at the beginning of the error-trace:

 >>> ipython
Python 3.7.3 (default, Mar 27 2019, 22:11:17) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.4.0 -- An enhanced Interactive Python. Type '?' for help.                                                                       

In [1]: %load_ext Cython                                                        

In [2]: %%cython 
   ...: from sklearn.tree._tree cimport Node 
   ...: print("loaded") 
   ...:  
   ...: 
/home/ed/.cache/ipython/cython/_cython_magic_1182f410e5c0a56b03b28dd88700704d.c:607:31: fatal error: numpy/arrayobject.h: No such file or directory
compilation terminated.
---------------------------------------------------------------------------
DistutilsExecError                        Traceback (most recent call last)
....

CompileError: command 'gcc' failed with exit status 1

The first lines tell us everything we need to know!

Spyder:

At least since Spyder 3.3.3, the output of the compiler/linker is seen in the IPython console (the same as in a standalone IPython-console).

---

The example %%cython-cell can be fixed as follows:

%%cython -I <path from numpy.get_include()>
from sklearn.tree._tree cimport Node
print("loaded")