Why is cmake finding 32-bit libraries instead of 64-bit libs on a 64-bit system?

Question

Problem Description

I'm working on porting several codebases over from Red Hat 5 to 6, and I've run into a cmake issue that I'm completely stumped by.

Cmake is consistently finding the 32-bit versions of libraries under /usr/lib instead of the 64-bit versions under /usr/lib64 on an RHEL6 system, while it correctly detects the lib64 versions on the an RHEL5 system.

Minimal Example

For example, I have a very minimal CMakeLists.txt file:

cmake_minimum_required(VERSION 2.8)

find_library(XTEST X11)
message("Found X11 at ${XTEST}")

On an RHEL6 system, running cmake on this results in:

$ cmake ..
-- The C compiler identification is GNU 4.4.7
-- The CXX compiler identification is GNU 4.4.7
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
Found X11 at /usr/lib/libX11.so
-- Configuring done
-- Generating done
-- Build files have been written to: $HOME/software/64bit_problem/build

(The key part here is the Found X11 at /usr/lib/libX11.so line)

However, if I do the same thing on an RHEL5 system, it correctly detects the /usr/lib64/ version: (Note that I'm clearing out the CMakeCache.txt and other temporary cmake-files in between runs.)

$ cmake ..
-- The C compiler identification is GNU 4.1.2
-- The CXX compiler identification is GNU 4.1.2
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
Found X11 at /usr/lib64/libX11.so
-- Configuring done
-- Generating done
-- Build files have been written to: $HOME/software/64bit_problem/build

Troubleshooting Info

The /usr/lib64 versions of the libraries exist on both systems. Here's the listing on the RHEL6 system:

$ ls /usr/lib64/libX11.so*
/usr/lib64/libX11.so.6  /usr/lib64/libX11.so.6.3.0

And on the RHEL5 system:

$ ls /usr/lib64/libX11.so*
/usr/lib64/libX11.so.6  /usr/lib64/libX11.so.6.3.0

Just to confirm that /usr/lib is indeed the 32-bit versions (also, it's not a symlink to another location):

$ file /usr/lib/libX11.so*
/usr/lib/libX11.so:       symbolic link to `libX11.so.6.3.0'
/usr/lib/libX11.so.6:     symbolic link to `libX11.so.6.3.0'
/usr/lib/libX11.so.6.3.0: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, stripped

$ file /usr/lib64/libX11.so*
/usr/lib64/libX11.so.6:     symbolic link to `libX11.so.6.3.0'
/usr/lib64/libX11.so.6.3.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, stripped

This is probably due to an environment setting somewhere, but I'm a bit stumped as to where. I don't have LD_LIBRARY_PATH , LD_RUN_PATH , or LDFLAGS set. Same for CFLAGS and CXXFLAGS . My user environment should be identical, as my $HOME is an NFS share that's the same on both machines.

/usr/lib isn't in my $PATH , and regardless, limiting my path to a minimal subset doesn't seem to help:

$ export PATH=/bin:/usr/bin:$HOME/local/bin

$ cmake ..
<snip>
Found X11 at /usr/lib/libX11.so
-- Configuring done
<snip>

@SergyA had the excellent suggestion of checking which environment variables were being accessed by using ltrace . strace didn't turn up anything diagnostic from what I could see, but ltrace shows the environment variable access nicely. Here's a quick summary:

$ ltrace -o ltrace_output cmake ..

$ grep getenv ltrace_output
getenv("PWD")                                                    = "$HOME/software/64bit_problem"
getenv("PATH")                                                   = "/bin:/usr/bin:$HOME/local/bin"
getenv("CMAKE_ROOT")                                             = NULL
getenv("MAKEFLAGS")                                              = NULL
getenv("VERBOSE")                                                = NULL
getenv("CFLAGS")                                                 = NULL
getenv("LDFLAGS")                                                = NULL
getenv("LDFLAGS")                                                = NULL
getenv("LDFLAGS")                                                = NULL
getenv("CXXFLAGS")                                               = NULL
getenv("LDFLAGS")                                                = NULL
getenv("LDFLAGS")                                                = NULL
getenv("LDFLAGS")                                                = NULL

cmake -specific Trobleshooting

The cmake version is identical on both machines (in fact, it's the same executable for reasons I'll omit for brevity):

$ cmake --version
cmake version 2.8.11.2

$ which cmake
$HOME/local/bin/cmake

I've tried explicitly enabling FIND_LIBRARY_USE_LIB64_PATHS , but this doesn't seem to make a difference:

cmake_minimum_required(VERSION 2.8)

set_property(GLOBAL PROPERTY FIND_LIBRARY_USE_LIB64_PATHS ON)

find_library(XTEST X11)
message("Found X11 at ${XTEST}")

As @Ravi mentioned, this is more likely due to some issue with CMAKE_LIBRARY_PATH , but it's not set, and changing it as either an environment variable or as a cmake variable doesn't seem to help. However, I fully admit I am largely ignorant of the various cmake configuration variables, so it's very likely that I'm missing something obvious here.

One key thing I've recently realized is that it's not all libraries... For example:

cmake_minimum_required(VERSION 2.8)

find_library(PNGTEST png)
message("Found libpng at ${PNGTEST}")

Finds /usr/lib64/libpng.so instead of /usr/lib/libpng.so .

This leads me to think it's something cmake-specific, at any rate.

find_package instead of find_library

Given the library-specific issues I mentioned above, I thought to try finding the entire X11 package instead of a single library (which is really what the codebase I was working with should have done anyway).

However, I'm getting even more confusing results... It seems to be detecting a mix of 64-bit and 32-bit libraries? For example:

cmake_minimum_required(VERSION 2.8)

FIND_PACKAGE(X11 REQUIRED)

message("X11_LIBRARIES: ${X11_LIBRARIES}")

We'll get:

$ cmake ..
<snip>
-- Looking for XOpenDisplay in /usr/lib/libX11.so;/usr/lib/libXext.so
-- Looking for XOpenDisplay in /usr/lib/libX11.so;/usr/lib/libXext.so - not found
<snip>
-- Found X11: /usr/lib/libX11.so
X11_LIBRARIES: /usr/lib64/libSM.so;/usr/lib64/libICE.so;/usr/lib/libX11.so;/usr/lib/libXext.so
-- Configuring done
<snip>

However, if we have a look at these specific libraries in X11_LIBRARIES , they're a mix of 32-bit and 64-bit versions!

$ file /usr/lib64/libSM.so.6.0.1
/usr/lib64/libSM.so.6.0.1: ELF 64-bit LSB shared object, x86-64, <snip>

$ file /usr/lib64/libICE.so.6.3.0
/usr/lib64/libICE.so.6.3.0: ELF 64-bit LSB shared object, x86-64, <snip>

$ file /usr/lib/libX11.so.6.3.0
/usr/lib/libX11.so.6.3.0: ELF 32-bit LSB shared object, Intel 80386, <snip>

$ file /usr/lib/libXext.so.6.4.0
/usr/lib/libXext.so.6.4.0: ELF 32-bit LSB shared object, Intel 80386, <snip>

In summary, what else should I try?

Am I missing a cmake-specific or possibly X11-specific configuration option?

Answer 1

You could try :

list(INSERT 0 CMAKE_SYSTEM_LIBRARY_PATH /usr/lib64)

The full list of variables used by find_library is available in the documentation: https://cmake.org/cmake/help/v3.5/command/find_library.html

Answer 2

It turns out that the root problem is the version suffix and the lack of a symlink to specifically /usr/lib64/libX11.so on the RHEL6 side.

cmake looks specifically for libX11.so and not libX11.so.6 or some other version-specific variant.

In this case, making the symlink wasn't an option, but I was able to prefer the specific version by listing the filename first:

cmake_minimum_required(VERSION 2.8)

find_library(XTEST NAMES libX11.so.6 X11)
message("Found X11 at ${XTEST}")

However, there are undoubtedly a variety of better ways of handling this, and I'd be very interested to hear them if anyone has any better approaches.