如何找到使用ldd和rpath无法显示的隐藏库？

Question

I'm building High Performance Linpack, and I am trying to share the binary to my colleagues. Before I build it, the following libraries are also build by myself, and are installed in /opt . These libraries are: /opt/blis , /opt/knem , and /opt/openmpi .

I tried to transfer these binary and only necessary libraries to my colleagues. However I found that the library I build located in /opt/openmpi/lib must be referenced in runtime.

I already check all binaries and libraries about rpath/runpath, and the usage of share libraries by using ldd, chrpath, and objdump. No other library in /opt/openmpi/lib should be referenced.

Is there any suggested method to know which library will be referenced, or how to trace runtime file access for me, to find out what are the minimal set of files that I need to transfer to my colleague?

Using ldd and found that following libraries and binaries will be used:

Binaries:

• xhpl
• mpirun

Libraries:

• libblis.so.2
• libjemalloc.so.2
• libmpi.so.40
• libopen-pal.so.40
• libopen-rte.so.40

I used chrpath to let all the binaries and libraries point to the running directory, but it doesn't work:

• chrpath -c [binaries and libraries]
• chrpath -r ./ [binaries and libraries]

I expected I can only transfer the minimal number of files to my colleagues and run it without modifying the system config.

Answer 1

You have luck that Linux is very strong in such things. The likely reason, why ldd doesn't show them, is that ldd shows only the libraries linked by the dynamic linker. It does not show the libs which were linked by C api calls ( dlopen(...) et al).

How to trace a program in runtime for the opened sharedlibs

You can check with the strace tool, what a program tries to do while it runs. Mostly, in loads its libraries in the first seconds:

The command

strace -s 4096 -f -o sux ./yourbinary

will start your ./yourbinary in tracing mode, and writes the output in a file named sux . For example, the first some lines of a strace of a /bin/echo looks so:

execve("/bin/echo", ["/bin/echo", "arg"], 0x7ffd2542fc50 /* 39 vars */) = 0
brk(NULL)                               = 0x5631fce2c000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (file not found)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=275366, ...}) = 0
mmap(NULL, 275366, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7edeb8de6000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 <--- here you can see, that it opened /lib/x86_64-linux-gnu/libc.so.6 !

It traces the kernel calls, what the binary executed, what results a lot of unneeded (and, for unexperienced eyes, often hardly comprehensible) information, too. But, any time if it uses an open -like call to some library (in my example, the /lib/<arch>/libc.so.6 file), it will be clearly visible.

You can also use the grep tool to filter the output, for example grep openat.*\\.so sux will filter for the library openings in the output file ( sux in our example).

The important thing is what we use here, that opening a shared lib is being done with a simple openat() call on the kernel level (what is a kernel-call variant of the open() in the C libraries).

How to check a running process, which sharedlibs has it mapped

It is being done by the lsof tool. The linked in shared libraries are mmap() -ped files in the address space of the running process. MMap means that the file is not read into the heap of the process, instead it is mapped into its address space. The difference is not important in our case, but lsof shows them differently. Essentially, the content of an mmap() -ped file will be available in the memory of the process on a pointer: reading it will read the file, and writing to it will write the file (on the disk). It can be even executed (called by call on the assembly level), and this is what we with the linked shared libs mostly do!

To get the list of the opened (mmapped) files of a running process, you can use the command

lsof -n -P -p <pid>

-n -P is only to disable some here unneeded network resolutions, -p is the important thing, because it shows the opened files of <pid> . For example, the lsof output of a running bash shell is this:

COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
bash    9072 root  cwd    DIR    8,7     4096  721255 /root
bash    9072 root  rtd    DIR    8,7     4096       2 /
bash    9072 root  txt    REG    8,7  1168776 6422594 /bin/bash
bash    9072 root  mem    REG    8,7    55792 2105586 /lib/x86_64-linux-gnu/libnss_files-2.28.so
bash    9072 root  mem    REG    8,7   164738 3844637 /usr/share/locale/de/LC_MESSAGES/libc.mo
bash    9072 root  mem    REG    8,7   337184 3051462 /usr/lib/locale/de_AT.utf8/LC_CTYPE
bash    9072 root  mem    REG    8,7  2586242 2753498 /usr/lib/locale/aa_DJ.utf8/LC_COLLATE
bash    9072 root  mem    REG    8,7  1824496 2105577 /lib/x86_64-linux-gnu/libc-2.28.so
bash    9072 root  mem    REG    8,7    14592 2105579 /lib/x86_64-linux-gnu/libdl-2.28.so
bash    9072 root  mem    REG    8,7   183528 2097224 /lib/x86_64-linux-gnu/libtinfo.so.6.1
bash    9072 root  mem    REG    8,7    89016 2105583 /lib/x86_64-linux-gnu/libnsl-2.28.so
bash    9072 root  mem    REG    8,7    47608 2105588 /lib/x86_64-linux-gnu/libnss_nis-2.28.so
bash    9072 root  mem    REG    8,7    39736 2105584 /lib/x86_64-linux-gnu/libnss_compat-2.28.so
bash    9072 root  mem    REG    8,7       54 3020617 /usr/lib/locale/agr_PE/LC_NUMERIC
bash    9072 root  mem    REG    8,7   165632 2105569 /lib/x86_64-linux-gnu/ld-2.28.so
<...many unneeded thingy...>
bash    9072 root    0u   CHR  136,2      0t0       5 /dev/pts/2
bash    9072 root    1u   CHR  136,2      0t0       5 /dev/pts/2
bash    9072 root    2u   CHR  136,2      0t0       5 /dev/pts/2
bash    9072 root  255u   CHR  136,2      0t0       5 /dev/pts/2

In the fd column, where you can see mem , it is an mmap() -ed file. If a file with a .so extension is mapped, then it is a runtime-linked shared lib.

Again you can use the grep tool to filter the output.