Undefined Symbol Error with embedded Python interpreter

Question

Originally I had this error within a larger project with pybind11 to embed an anaconda Python interpreter. I was able to boil it down and reproduce the error with just a simple minimal example.

When I run my executable (which embeds python), I obtain this error:

Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "/app/Python-3.8.2-build/lib/python3.8/struct.py", line 13, in <module>
    from _struct import *
ImportError: /app/Python-3.8.2-build/lib/python3.8/lib-dynload/_struct.cpython-38-x86_64-linux-gnu.so: undefined symbol: PyByteArray_Type

At first, I built Python-3.8.2 from source. Then I compiled an executable from the following C code:

#include <Python.h>

int main(int argc, char *argv[])
{
    Py_Initialize();
    PyRun_SimpleString("import struct");
    if (Py_FinalizeEx() < 0) {
        exit(120);
    }
    return 0;
}

using this command:

gcc -o execpy execpy.c \
-I/app/Python-3.8.2-build/include/python3.8 \
-Wno-unused-result -Wsign-compare  -DNDEBUG -g -fwrapv -O3 \
-L/app/Python-3.8.2-build/lib  -lcrypt -lpthread -ldl  -lutil -lm \
/app/Python-3.8.2/libpython3.8.a

Then simply execute ./execpy gives the error from above... Any ideas?

EDIT : In this example I want to link libpython statically just like the python interpreter does not depend on any libpython.so.

EDIT : _struct.*.so does not seem to have dependencies to libpython linked in (this is the same for my standard anaconda python interpreter):

$ ldd /app/Python-3.8.2-build/lib/python3.8/lib-dynload/_struct.cpython-38-x86_64-linux-gnu.so
    linux-vdso.so.1 =>  (0x00007fff32bf0000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f71a5634000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f71a5266000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f71a5a5c000)

I also checked the _struct.*.so of my system python interpreter on a different machine and it has it:

    linux-vdso.so.1 =>  (0x00007ffe2b3d9000)
    libpython3.6m.so.1.0 => /lib64/libpython3.6m.so.1.0 (0x00007febe24fd000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007febe22e1000)
    libc.so.6 => /lib64/libc.so.6 (0x00007febe1f13000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007febe1d0f000)
    libutil.so.1 => /lib64/libutil.so.1 (0x00007febe1b0c000)
    libm.so.6 => /lib64/libm.so.6 (0x00007febe180a000)
    /lib64/ld-linux-x86-64.so.2 (0x00007febe2c30000)

Answer 1

Linking libpython statically

Short answer: Adding -rdynamic to the flags makes it work for me.

Documentation for -rdynamic flag:

-rdynamic
    Pass the flag -export-dynamic to the ELF linker, on targets that support it. This 
    instructs the linker to add all symbols, not only used ones, to the dynamic symbol 
    table. This option is needed for some uses of dlopen or to allow obtaining 
    backtraces from within a program.

Linking libpython dynamically

I also found out: If you want to embed the Python 3.8 interpreter dynamically ( libpython3.8.so ), there are some changes since version 3.8 :

On Unix, C extensions are no longer linked to libpython except on Android and Cygwin. When Python is embedded, libpython must not be loaded with RTLD_LOCAL, but RTLD_GLOBAL instead. Previously, using RTLD_LOCAL, it was already not possible to load C extensions which were not linked to libpython, like C extensions of the standard library built by the shared section of Modules/Setup. (Contributed by Victor Stinner in bpo-21536.)

Note also (see here ):

To embed Python into an application, a new --embed option must be passed to python3-config --libs --embed to get -lpython3.8 (link the application to libpython). To support both 3.8 and older, try python3-config --libs --embed first and fallback to python3-config --libs (without --embed) if the previous command fails.

Add a pkg-config python-3.8-embed module to embed Python into an application: pkg-config python-3.8-embed --libs includes -lpython3.8. To support both 3.8 and older, try pkg-config python-XY-embed --libs first and fallback to pkg-config python-XY --libs (without --embed) if the previous command fails (replace XY with the Python version).

So compiling and linking dynamically like this also works for me now:

gcc -o execpy execpy.c -I/app/Python-3.8.2-build/include/python3.8 \
    -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 \
    -lcrypt -lpthread -ldl  -lutil -lm -lpython3.8\
    -L/app/Python-3.8.2-build/lib/ -Wl,-rpath,/app/Python-3.8.2-build/lib/

Answer 2

I think this happens because PyByteArray_Type is not referenced in execpy.c , so the linker uses its default --gc-sections logic to drop the unused symbol. Try adding one of the following options:

-Wl,--no-gc-sections :

Enable garbage collection of unused input sections. It is ignored on targets that do not support this option. The default behaviour (of not performing this garbage collection) can be restored by specifying --no-gc-sections on the command line. Note that garbage collection for COFF and PE format targets is supported, but the implementation is currently considered to be experimental.

-Wl,--gc-keep-exported :

When --gc-sections is enabled, this option prevents garbage collection of unused input sections that contain global symbols having default or protected visibility. This option is intended to be used for executables where unreferenced sections would otherwise be garbage collected regardless of the external visibility of contained symbols. Note that this option has no effect when linking shared objects since it is already the default behaviour. This option is only supported for ELF format targets.