What's the correct way to compile maturin packages?

Question

I'm developing a Rust library that has Python bindings using PyO3 which are build using maturin.

The problem

If I build the bindings on my os (Arch, with Glibc 2.31) then the generated.whl don't work on older systems because the Glibc is not compatible and raises an error such as:

ImportError: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found 
(required by /home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/my_package.cpython-37m-x86_64-linux-gnu.so)

The current workaround

In order to solve this issue we are building our bindings on a centos 6.9 which has Glibc 2.12, therefore "should be compatible" with nearly all Linux distro.

The actual question

This is clearly just a workaround and there should be a better solution.

What's the correct way to compile maturin libraries in such a way that they works on most of Linux distributions?

From my understanding this is what the manylinux standard is for, then why doesn't it works? What am I missing?

My hypothesis & Tests

I guess that a possible solution would be to statically link the Glibc into the library, and therefore targeting musl which is meant for static compilation.

Is it possible to do this? I've found this blog which suggests that it can be done but I see that inside of the.whl files (which are just python installable archives) there is a .so , therefore a shared library.

Would it be even possible to build a shared library that has statically linked the libc using musl?

I found a possible working example of this: Inside of the tensorflow who tf_nightly-2.4.0.dev20200710-cp36-cp36m-manylinux2010_x86_64 , there is the file /tensorflow/python/_pywrap_utils.so that:

file _pywrap_utils.so                                                                                       [2]
_pywrap_utils.so: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), statically linked, BuildID[md5/uuid]=aee3c5fa4ae1243775857b2643b80ea0, not stripped

But it seems to dynamically link anyway:

ldd _pywrap_utils.so
        linux-vdso.so.1 (0x00007fffb3782000)
        libtensorflow_framework.so.2 => not found
        _pywrap_tensorflow_internal.so => not found
        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007f5c95672000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f5c95658000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f5c95491000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007f5c9534a000)
        /usr/lib64/ld-linux-x86-64.so.2 (0x00007f5c95a9f000)```

So is this actually possible?

Some people , strongly advise against doing this, why is that? I get that different implementation of threads and allocators in the same executable could lead to compatibility issues.

I've tried to change in the Cargo.toml from

[lib]
crate-type = ["cdylib"]

to this ( as seen here )

[lib]
crate-type = ["staticlib"]

but maturin don't seem to be happy about that.

💥 maturin failed
  Caused by: Cargo didn't build a cdylib. Did you miss crate-type = ["cdylib"] in the lib section of your Cargo.toml?

Maybe this could be done with the compilation flag crt-static ?

Answer 1

Apparently, the supposed way to do this is to compile the library inside of the official manylinux1 docker.

Which can be done with:

sudo docker run --rm -v $(pwd):/io konstin2/maturin build --release