Undefined references when linking boost python

Question

I am fairly new to using Linux. But I believe I have done the correct steps to get set up: wget for Python 3.8.5, did the make for it. wget for boost 1.74, bootstrap, ./b2

And boost installed and setup everything except for Python which it did not auto-detect. After many hours of frustration, I can get it to find python with:

./bootstrap.sh --with-python=python3.8

running ./b2 --debug-configuration --with-python

shows that it has found it here:

notice: [python-cfg] Details of this Python configuration: notice:
[python-cfg]   interpreter command: "/usr/local/bin/python3.8" notice:
[python-cfg]   include path: "/usr/local/include/python3.8" notice:
[python-cfg]   library path: "/usr/local/lib/python3.8/config"
"/usr/local/lib"

So to this end I am compiling my program successfully with the boost include and python include folder mentioned from b2's debug output above. I am linking similarly with -lpython3.8 -lboost_python38 -lboost_system

All 3 are being found evidently as it does not complain about missing libraries.

Parser.o: In function `_Py_DECREF':
Parser.cpp:(.text+0x5b): undefined reference to `_Py_Dealloc'
Parser.o: In function `sections_to_html(std::__cxx11::list<DivSection, std::allocator<DivSection> > const&)':
Parser.cpp:(.text+0x1eb3): undefined reference to `PyUnicode_FromString'
Parser.cpp:(.text+0x1edf): undefined reference to `PyUnicode_FromString'
Parser.o: In function `PyInit_Parser':
Parser.cpp:(.text+0x245b): undefined reference to `boost::python::detail::init_module(PyModuleDef&, void (*)())'
Parser.o: In function `boost::python::type_info::name() const':
Parser.cpp:(.text._ZNK5boost6python9type_info4nameEv[_ZNK5boost6python9type_info4nameEv]+0x1f): undefined reference to `boost::python::detail::gcc_demangle(char const*)'
Parser.o: In function `boost::python::api::object::object()':
Parser.cpp:(.text._ZN5boost6python3api6objectC2Ev[_ZN5boost6python3api6objectC5Ev]+0x14): undefined reference to `_Py_NoneStruct'
Parser.o: In function `_object* boost::python::expect_non_null<_object>(_object*)':
Parser.cpp:(.text._ZN5boost6python15expect_non_nullI7_objectEEPT_S4_[_ZN5boost6python15expect_non_nullI7_objectEEPT_S4_]+0x14): undefined reference to `boost::python::throw_error_already_set()'
Parser.o: In function `boost::python::tuple boost::python::make_tuple<boost::python::handle<_object>, boost::python::handle<_object> >(boost::python::handle<_object> const&, boost::python::handle<_object> const&)':
Parser.cpp:(.text._ZN5boost6python10make_tupleINS0_6handleI7_objectEES4_EENS0_5tupleERKT_RKT0_[_ZN5boost6python10make_tupleINS0_6handleI7_objectEES4_EENS0_5tupleERKT_RKT0_]+0x2c): undefined reference to `PyTuple_New'
Parser.cpp:(.text._ZN5boost6python10make_tupleINS0_6handleI7_objectEES4_EENS0_5tupleERKT_RKT0_[_ZN5boost6python10make_tupleINS0_6handleI7_objectEES4_EENS0_5tupleERKT_RKT0_]+0x51): undefined reference to `boost::python::api::object::object(boost::python::handle<_object> const&)'
Parser.cpp:(.text._ZN5boost6python10make_tupleINS0_6handleI7_objectEES4_EENS0_5tupleERKT_RKT0_[_ZN5boost6python10make_tupleINS0_6handleI7_objectEES4_EENS0_5tupleERKT_RKT0_]+0xda): undefined reference to `boost::python::api::object::object(boost::python::handle<_object> const&)'
Parser.o: In function `void boost::python::def<boost::python::tuple (*)(char const*)>(char const*, boost::python::tuple (*)(char const*))':
Parser.cpp:(.text._ZN5boost6python3defIPFNS0_5tupleEPKcEEEvS4_T_[_ZN5boost6python3defIPFNS0_5tupleEPKcEEEvS4_T_]+0x50): undefined reference to `boost::python::detail::scope_setattr_doc(char const*, boost::python::api::object const&, char const*)'
Parser.o: In function `boost::python::api::object boost::python::detail::make_function_aux<boost::python::tuple (*)(char const*), boost::python::default_call_policies, boost::mpl::vector2<boost::python::tuple, char const*> >(boost::python::tuple (*)(char const*), boost::python::default_call_policies     const&, boost::mpl::vector2<boost::python::tuple, char const*> const&)':
Parser.cpp:(.text._ZN5boost6python6detail17make_function_auxIPFNS0_5tupleEPKcENS0_21default_call_policiesENS_3mpl7vector2IS3_S5_EEEENS0_3api6objectET_RKT0_RKT1_    [_ZN5boost6python6detail17make_function_auxIPFNS0_5tupleEPKcENS0_21default_call_policiesENS_3mpl7vector2IS3_S5_EEEENS0_3api6objectET_RKT0_RKT1_]+0x67): undefined reference to `boost::python::objects::function_object(boost::python::objects::py_function const&)'
Parser.o: In function `boost::python::objects::py_function_impl_base::py_function_impl_base()':
Parser.cpp:(.text._ZN5boost6python7objects21py_function_impl_baseC2Ev[_ZN5boost6python7objects21py_function_impl_baseC5Ev]+0xb): undefined reference to `vtable for boost::python::objects::py_function_impl_base'
Parser.o:(.data.rel.ro._ZTVN5boost6python7objects23caller_py_function_implINS0_6detail6callerIPFNS0_5tupleEPKcENS0_21default_call_policiesENS_3mpl7vector2IS5_S7_EEEEEE    [_ZTVN5boost6python7objects23caller_py_function_implINS0_6detail6callerIPFNS0_5tupleEPKcENS0_21default_call_policiesENS_3mpl7vector2IS5_S7_EEEEEE]+0x30): undefined reference to `boost::python::objects::py_function_impl_base::max_arity() const'
Parser.o: In function `boost::python::objects::caller_py_function_impl<boost::python::detail::caller<boost::python::tuple (*)(char const*), boost::python::default_call_policies, boost::mpl::vector2<boost::python::tuple, char const*> > >::~caller_py_function_impl()':
Parser.cpp:(.text._ZN5boost6python7objects23caller_py_function_implINS0_6detail6callerIPFNS0_5tupleEPKcENS0_21default_call_policiesENS_3mpl7vector2IS5_S7_EEEEED2Ev    [_ZN5boost6python7objects23caller_py_function_implINS0_6detail6callerIPFNS0_5tupleEPKcENS0_21default_call_policiesENS_3mpl7vector2IS5_S7_EEEEED5Ev]+0x22): undefined reference to `boost::python::objects::py_function_impl_base::~py_function_impl_base()'
Parser.o:(.data.rel.ro._ZTIN5boost6python7objects23caller_py_function_implINS0_6detail6callerIPFNS0_5tupleEPKcENS0_21default_call_policiesENS_3mpl7vector2IS5_S7_EEEEEE    [_ZTIN5boost6python7objects23caller_py_function_implINS0_6detail6callerIPFNS0_5tupleEPKcENS0_21default_call_policiesENS_3mpl7vector2IS5_S7_EEEEEE]+0x10): undefined reference to `typeinfo for boost::python::objects::py_function_impl_base'
Parser.o: In function `boost::python::converter::pointer_arg_from_python<char const*>::pointer_arg_from_python(_object*)':
Parser.cpp:(.text._ZN5boost6python9converter23pointer_arg_from_pythonIPKcEC2EP7_object[_ZN5boost6python9converter23pointer_arg_from_pythonIPKcEC5EP7_object]+0x18): undefined reference to `_Py_NoneStruct'
Parser.cpp:(.text._ZN5boost6python9converter23pointer_arg_from_pythonIPKcEC2EP7_object[_ZN5boost6python9converter23pointer_arg_from_pythonIPKcEC5EP7_object]+0x34): undefined reference to `boost::python::converter::get_lvalue_from_python(_object*, boost::python::converter::registration const&)'
Parser.o: In function `boost::python::converter::pointer_arg_from_python<char const*>::operator()() const':
Parser.cpp:(.text._ZNK5boost6python9converter23pointer_arg_from_pythonIPKcEclEv[_ZNK5boost6python9converter23pointer_arg_from_pythonIPKcEclEv]+0x1e): undefined reference to `_Py_NoneStruct'
Parser.o: In function `boost::python::converter::expected_pytype_for_arg<boost::python::tuple>::get_pytype()':
Parser.cpp:(.text._ZN5boost6python9converter23expected_pytype_for_argINS0_5tupleEE10get_pytypeEv[_ZN5boost6python9converter23expected_pytype_for_argINS0_5tupleEE10get_pytypeEv]+0x1b): undefined reference to `boost::python::converter::registry::query(boost::python::type_info)'
Parser.cpp:(.text._ZN5boost6python9converter23expected_pytype_for_argINS0_5tupleEE10get_pytypeEv[_ZN5boost6python9converter23expected_pytype_for_argINS0_5tupleEE10get_pytypeEv]+0x32): undefined reference to `boost::python::converter::registration::expected_from_python_type() const'
Parser.o: In function `boost::python::converter::expected_pytype_for_arg<char const*>::get_pytype()':
Parser.cpp:(.text._ZN5boost6python9converter23expected_pytype_for_argIPKcE10get_pytypeEv[_ZN5boost6python9converter23expected_pytype_for_argIPKcE10get_pytypeEv]+0x1b): undefined reference to `boost::python::converter::registry::query(boost::python::type_info)'
Parser.cpp:(.text._ZN5boost6python9converter23expected_pytype_for_argIPKcE10get_pytypeEv[_ZN5boost6python9converter23expected_pytype_for_argIPKcE10get_pytypeEv]+0x32): undefined reference to `boost::python::converter::registration::expected_from_python_type() const'
Parser.o: In function `boost::python::converter::registration const& boost::python::converter::detail::registry_lookup2<char const volatile>(char const volatile& (*)())':
Parser.cpp:(.text._ZN5boost6python9converter6detail16registry_lookup2IVKcEERKNS1_12registrationEPFRT_vE[_ZN5boost6python9converter6detail16registry_lookup2IVKcEERKNS1_12registrationEPFRT_vE]+0x1f): undefined reference to `boost::python::converter::registry::lookup(boost::python::type_info)'
Parser.o: In function `boost::python::converter::pyobject_type<boost::python::tuple, &PyTuple_Type>::get_pytype()':
Parser.cpp:(.text._ZN5boost6python9converter13pyobject_typeINS0_5tupleEXadL_Z12PyTuple_TypeEEE10get_pytypeEv[_ZN5boost6python9converter13pyobject_typeINS0_5tupleEXadL_Z12PyTuple_TypeEEE10get_pytypeEv]+0x7): undefined reference to `PyTuple_Type'
collect2: error: ld returned 1 exit status
makefile:8: recipe for target 'Parser' failed
make: *** [Parser] Error 1

I see a bunch of boost namespace stuff coming up.

So this leads me to believe that either there is a change in the name mangling between what I've compiled and what is being linked in, or there is a difference with defines which is causing either namespace or other differences.

But I am unsure how to find out the cause, assuming I am correct with my previous assumptions.

EDIT: I had cut the dump short and said that 'most' of it seems to be from boost, but JaMiT called me on it. So theres the full dump.

EDIT 2: I checked my python config and the results are:

user@vm:/usr/local/lib/python3.8/config-3.8-x86_64-linux-gnu$ python python-config.py --cflags

-I/usr/local/include/python3.8 -I/usr/local/include/python3.8 -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall

user@vm:/usr/local/lib/python3.8/config-3.8-x86_64-linux-gnu$ python python-config.py --ldflags

-L/usr/local/lib/python3.8/config-3.8-x86_64-linux-gnu -lcrypt -lpthread -ldl -lutil -lm -lm

EDIT 3:

I used nm on libpython3.8.a from my /usr/local/lib/python3.8/config-3.8-x86_64-linux-gnu and found that PyUnicode_FromString and some others do appear correctly as type 'T' function-with-param.

So they are there.

Answer 1

Thanks to a comment by 'sehe' mentioning ordering of libs, I did some Google searching and found the problem - my linking step in the makefile was:

$(CC) $(COMMONFLAGS) $(LDFLAGS) $(LIBS) -o Parser Parser.o grammar.o words.o

The additional libraries I use should come at the end eg:

$(CC) $(COMMONFLAGS) $(LDFLAGS) -o Parser Parser.o grammar.o words.o $(LIBS)

But after fixing the aforementioned ordering of arguments on the linker invocation, I had additional undefined references crop up from within Python which had not been reported before.

This turned out not to be an issue as building an executable was just step one - making sure the code compiles and I can link.

In the end to get a makefile building a module I can import into Python, I needed to compile all cpp files with -fPIC and then link with -shared -export-dynamic

Python's lib was not needed and so my final libs are -lboost_python38 -lboost_system

After setting my LD_LIBRARY_PATH to point to the boost /stage/lib folder, I was able to import my Parser.so module and it worked correctly.