How can i include 3rd party libraries into a python3 manylinux build?

Question

I'm trying to package a C extension for cpython and I'm not sure how to proceed. From what I understand I need to first generate the wheel file with python3 -m build and then do auditwheel repair dist/my_wheel_file.whl -w dist/ or something to that effect. when I build the package locally and do auditwheel show dist/my_wheel_file.whl it says this

winlin-1.0.0-cp310-cp310-linux_x86_64.whl is consistent with the
following platform tag: "linux_x86_64".

The wheel references external versioned symbols in these
system-provided shared libraries: libc.so.6 with versions
{'GLIBC_2.8', 'GLIBC_2.4', 'GLIBC_2.3', 'GLIBC_2.3.4', 'GLIBC_2.2.5',
'GLIBC_2.33', 'GLIBC_2.14', 'GLIBC_2.17'}, libxkbcommon.so.0 with
versions {'V_0.5.0'}

This constrains the platform tag to "manylinux_2_34_x86_64". In order
to achieve a more compatible tag, you would need to recompile a new
wheel from source on a system with earlier versions of these
libraries, such as a recent manylinux image.

when I run the same command in a manylinux container on a freshly built wheel file I get this output.

winlin-1.0.0-cp310-cp310-linux_x86_64.whl is consistent with the
following platform tag: "linux_x86_64".

The wheel references external versioned symbols in these
system-provided shared libraries: libc.so.6 with versions
{'GLIBC_2.3.4', 'GLIBC_2.4', 'GLIBC_2.3', 'GLIBC_2.2.5'},
libxkbcommon.so.0 with versions {'V_0.5.0'}

The following external shared libraries are required by the wheel:
{
    "libX11.so.6": "/lib64/libX11.so.6.3.0",
    "libXau.so.6": "/lib64/libXau.so.6.0.0",
    "libXinerama.so.1": null,
    "libXtst.so.6": null,
    "libc.so.6": "/lib64/libc-2.17.so",
    "libdl.so.2": "/lib64/libdl-2.17.so",
    "libpthread.so.0": "/lib64/libpthread-2.17.so",
    "libxcb.so.1": "/lib64/libxcb.so.1.1.0",
    "libxdo.so.3": "/usr/local/lib/libxdo.so.3",
    "libxkbcommon.so.0": null
}

podman mount command

podman run --rm -ti \
       -v "$PWD/winlin:/winlin:rw" \
       -v "/usr/local/include/xdo:/usr/local/include/xdo:ro" \
       -v "/usr/local/lib/xdo:/usr/local/lib/xdo:ro" \
       quay.io/pypa/manylinux2014_x86_64

this is my setup.py file

from setuptools import setup, Extension
#from distutils.core import setup, Extension
module1 = Extension(
    'winlin',
    define_macros = [('MAJOR_VERSION', '1'),('MINOR_VERSION', '0')],
    include_dirs = ['/usr/local/include/xdo/'],
    libraries = ['xdo'],
    library_dirs = ['/usr/local/lib/xdo/'],
    sources = ['src/winlin.c']
)

setup(
  name = 'winlin',
  version = '1.0',
  description = 'A tool kit for manipulating windows in linux',
  author = 'Kurt Godel',
  author_email = 'corndog@corn.dog',
  url = 'https://google.com',
  long_description = '#TODO',
  ext_modules = [module1]
)
``` C
and my c extension file winlin.c looks like this
```#include <Python.h>
#include <xdo.h>


static PyObject* resize(PyObject* self, PyObject *args){
    int wid, w, h;
    if (!PyArg_ParseTuple(args, "iii", &wid, &w, &h))
        return NULL;

    xdo_t *xdo_inst = xdo_new(NULL);
    xdo_set_window_size(xdo_inst, wid, w, h, 0);
    xdo_free(xdo_inst);
    Py_RETURN_NONE;
    //return Py_BuildValue("s", "it worked, maybe?");
}
static char resize_docs[] = "\
change the size of a given window given its id and a new height and width\n\
";
/*----------------------------------------------------------------------------*/

static PyObject* move(PyObject* self, PyObject *args){
    int wid, x, y;
    if (!PyArg_ParseTuple(args, "iii", &wid, &x, &y))
        return NULL;

    xdo_t *xdo_inst = xdo_new(NULL);
    xdo_move_window(xdo_inst, wid, x, y);
    xdo_free(xdo_inst);
    Py_RETURN_NONE;
}
static char move_docs[] = "\
change the position of a window given its id and a new x and y\n\
";

/*----------------------------------------------------------------------------------*/
static PyMethodDef winlin_funcs[] = {
    {"resize", (PyCFunction)resize, METH_VARARGS, resize_docs},
    {"move", (PyCFunction)move, METH_VARARGS, move_docs},
    {NULL}
};

static char winlin_module_docs[] = "Module used to manipulate windows in linux";

static struct PyModuleDef winlin_module = {
    PyModuleDef_HEAD_INIT,
    "winlin",
    winlin_module_docs,
    -1,
    winlin_funcs
};

PyMODINIT_FUNC
PyInit_winlin(void){
    PyObject* m = PyModule_Create(&winlin_module);
    return m;
}

I was able to hack in the libxdo-dev headers by including them into the podman command to mount the container. But even if I was to mount the rest of the so files into the containers would that even work? or do I need to compile the so files from source in the container?

So how do I include the functionality and dependencies of the libxdo-dev package into my python3 C extension? I know this is a bit rambly and incoherent at the moment but so is my brain trying to figure all this out. Any help is greatly appreciated!

Answer 1

I have solved my issue by running the following command in the container

yum install libxdo-devel

I've always had apt the whole time I've been using linux so not having it threw me off balance I guess. occams razor prevails again.

Answer 2

You can package the third-party libraries to the wheel file via build_ext .

Check out my setup.py file.

I created a custom class to copy all dependent library files to the packaging folder:

class CustomBuildExt(build_ext.build_ext):
        def run(self):
                build_ext.build_ext.run(self)
                dst =  os.path.join(self.build_lib, "barcodeQrSDK")
                copylibs(dbr_lib_dir, dst)
                filelist = os.listdir(self.build_lib)
                for file in filelist:
                    filePath = os.path.join(self.build_lib, file)
                    if not os.path.isdir(file):
                        copylibs(filePath, dst)
                        # delete file for wheel package
                        os.remove(filePath)