A 32bit version of an application, built on CentOS 6 x64, crashes during exceptions in `dl_iterate_phdr`, when started on newer Linux

Question

TL;DR; My research shows that if I build an application (and its dependencies) on older Linux like CentOS 6 (with GLIBC 2.12 ), it is supposed to work perfectly fine on any other Linux distro, which has newer GLIBC . Isn't this assumption correct?

---

Sorry, it's gonna be a long post, but it's not a trivial question.

Here's the build machine:

$ rpm -q centos-release
  centos-release-6-10.el6.centos.12.3.x86_64
$ ldd --version
  ldd (GNU libc) 2.12
$ ld -v
  GNU ld version 2.30-54.el6
$ gcc --version
  gcc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3)
$ g++ --version
  g++ (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3)

The application links with almost everything statically, so:

$ ldd ./app 
linux-gate.so.1 (0xf7f73000)
libdl.so.2 => /lib32/libdl.so.2 (0xf7f4c000)
librt.so.1 => /lib32/librt.so.1 (0xf7f41000)
libstdc++.so.6 => ./libstdc++.so.6 (0xf7e22000)
libm.so.6 => /lib32/libm.so.6 (0xf7d53000)
libgcc_s.so.1 => ./libgcc_s.so.1 (0xf7d35000)
libpthread.so.0 => /lib32/libpthread.so.0 (0xf7d14000)
libc.so.6 => /lib32/libc.so.6 (0xf7b36000)
/lib/ld-linux.so.2 (0xf7f75000)

So:

• linux-gate and ld-linux aren't exactly libs, so they can't be deployed;
• libdl , librt , libm , libpthread and libc are a part of the GLIBC , so they shouldn't be deployed as well (especially given the backward compatibility of the GNU C library);
• libstdc++ and libgcc_s are "deployed" together with the binary (taken from the build machine)

The test machine:

$ lsb_release -a
  Description:  Debian GNU/Linux 10 (buster)
$ ldd --version
  ldd (Debian GLIBC 2.28-10) 2.28

The 64bit build (with its libstdc++ and libgcc_s ) seems to work perfectly fine.

The issue is with the 32bit build - it crashes , when an exception is thrown (not for all exceptions, but seems to be consistent). Here's how interesting the stack trace is:

 SIGSEGV at  0# __kernel_sigreturn in linux-gate.so.1
 1# dl_iterate_phdr in /lib32/libc.so.6
 2# _Unwind_Find_FDE in ./libgcc_s.so.1
 3# 0xF7D88AAE in ./libgcc_s.so.1
 4# 0xF7D89227 in ./libgcc_s.so.1
 5# _Unwind_RaiseException in ./libgcc_s.so.1
 6# __cxa_throw in ./libstdc++.so.6
 7# <some funct> in ./app
 7# <some funct> in ./app
 7# <some funct> in ./app
 7# <some funct> in ./app
 7# <some funct> in ./app
 7# <some funct> in ./app
13# make_fcontext in ./app

I can see two possible options here:

1. there's something broken with the 32bit build (although it works without exceptions);
2. I do have some big misunderstanding on the matter.

Any ideas?

Answer 1

I suspect that if you "deploy" GCC libs (g++/gcc) from build machine to target one with different OS or OS version, then your app (or one of its dependencies) can actually use GCC libs from the target system, and other dependencies use libs that you deployed, leading to potential incompatibilities.

It's much safer to build your application on target box, or use cross-compilation specifying exactly target box OS and version.

Also it's safer to use dynamic linking with system libs because otherwise you may end up with the situation of mixing static and dynamic lib in one app, and this may lead to problems (eg two instances of some global var in gcc lib).

You'll need to check whole tree of lib dependencies to ensure that only one instance of every lib is loaded from the same location. For example your app depends on ./libgcc_s.so.1 and ./libstdc++.so.6 but ./libstdc++.so.6 may depend on system's libgcc_s.so .

You may search on Internet about these and other pitfalls with building your app on non-target box. Eg here :

So just compiling your application on an older distribution is not enough.