Linker library path in makefile confusion

Question

I've been programming a while but I still don't fully understand how the linker behaves.

For example, today I downloaded and installed a library that I want to use in my application in Linux. (It was Xerces - for parsing XML files).

I created a makefile and gave it the path to the .so and .a files in my command : -L/usr/local/lib and also told it the name of the library to include : -lxerces-c-3.1.

My application compiled fine but failed during runtime with "cannot open shared object file libxerces-c-3.1.so". Why would this be the case, when I properly give it the path and name in the makefile?

I then added the library path to the LD_LIBRARY_PATH variable in my .bashrc file and then it worked. That's fine , but if I now remove the path to the library in my makefile and don't even include the name of the library , it still works.

I'm confused as to what is going on here. How can it still find the correct library just by assigning the path to the LD_LIBRARY_PATH variable and will only work if I do so? I have read elsewhere to not even use LD_LIBRARY_PATH.

I appreciate any answer for this. The question is a bit long and hopefully not off-topic but I hope others can learn from it too. Thanks

Answer 1

compilation and running are different things. :)

1) A make file contains rules on how to build your application. So when you write a rule like:

    -L/usr/local/lib -lxerces-c-3.1

You are passing options to the linker. The -L option tells the linker to add additional libraries (in this case '/usr/local/lib') to the linker's search path. The -l option names the library that should be linked in.

2) When you go to run an executable, the loader needs to find all the required libraries. For example, on a Linux system you can use the ldd command to see what shared libraries are used. For example on my system:

ldd FEParser
    linux-vdso.so.1 =>  (0x00007ffcdc7c9000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f835b143000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f835ae3d000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f835ac27000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f835a862000)
/lib64/ld-linux-x86-64.so.2 (0x00007f835b447000)

From this, you can see the name of the library that is linked to the left of the '=>' token, and the path to the library to the right. If a library is not found it will show up as missing.

Now in your case, you were able to successfully compile and link your program due to giving the path and library name to use. You were not able to run the program due to the loader not being able to find the library at run time.

You have several options here:

1) you can move the library in a directory that is in the loaders search path.

2) you can modify LD_LIBRARY_PATH which adds additional directories to the loaders search path. Additionally, the directories specified in LD_LIBRARY_PATH will be passed to the linker (it will be appended after all -L flags). You do need to be careful with this, as if you set it globally (say in .bashrc ), then it will effect all compilations that you do. You may or may not want this behavior.

3) As others have specified you can use -Wl,rpath=.... , but it is deprecated, and I rarely use it.

4) If you need the library installed in an unusual location, you can add a create a file under /etc/ld.so.conf.d that contains, for example (file is yaml.conf):

# yaml default configuration
/opt/yaml/lib

At system boot, the loader reads all the files in this directory and appends the paths to the loader path. If you make a modification to this directory you can use the ldconfig command to cause the loader to reprocess /etc/ld.so.con.d . NB I know this works on centOS and Ubuntu flavors of GNU/Linux - can't speak authoritatively on other flavors.

Answer 2

Compile your program with -Wl,-rpath=/usr/local/lib . This way you'll add /usr/local/lib to runtime library search patch of your program and you won't need the LD_LIBRARY_PATH
Warning : since modern dynamic linkers consider rpath as deprecated you can set also runpath (which supersedes it) by specifying -Wl,-rpath=/usr/local/lib,--enable-new-dtags

Answer 3

There is no mystery here. Default library paths for linker (one you call to make your executable file, for example, ld ) and runtime linker (one which is responsible for loading shared libraries when you execute your program, for example, ld.so ) are different. Runtime linker uses LD_LIBRARY_PATH, while linker uses whatever was configured when ld was build.

In your case, looks like /usr/local/lib is part of one, but not another.

Answer 4

If you're using static linking, all you have to do is tell the linker where your library is at compile/link time. The library (or as much of it as is necessary) gets copied into your executable, and your executable is standalone.

But for various reasons, these days we generally use dynamic linking, not static linking. With dynamic linking, you have to tell the linker where to find the library at compile/link time, and the dynamic linker ( ld.so ) has to be able to find the library at run time.

If the library was in one of the standard places ( /lib , /usr/lib , etc.) there's no problem. But if you linked against a library in a nonstandard place, in general, you have to add that nonstandard place to your LD_LIBRARY_PATH, so that the runtime linker will always be able to find it.