Debugging functions in __libc_start_main

Question

I'm writing a library that hooks some CUDA functions to add some functionality. The "constructor" hooks the CUDA functions and set up message queue and shared memory to communicate with other hooked CUDA binaries. When launching several hooked CUDA binaries (by python subprocess.Popen('<path-to-binary>', shell=True) ) some processes hangs. So I used gdb -p <pid> to attach one suspended process, hoping to figure out what's going wrong. Here's the result:

Attaching to process 7445
Reading symbols from /bin/dash...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libc-2.27.so...done.
done.
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/ld-2.27.so...done.
done.
0x00007f9cefe8b76a in wait4 () at ../sysdeps/unix/syscall-template.S:78
78      ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) bt
#0  0x00007f9cefe8b76a in wait4 () at ../sysdeps/unix/syscall-template.S:78
#1  0x000055fff93be8a0 in ?? ()
#2  0x000055fff93c009d in ?? ()
#3  0x000055fff93ba6d8 in ?? ()
#4  0x000055fff93b949e in ?? ()
#5  0x000055fff93b9eda in ?? ()
#6  0x000055fff93b7944 in ?? ()
#7  0x00007f9cefdc8b97 in __libc_start_main (main=0x55fff93b7850, argc=3, argv=0x7ffca7c7beb8, init=<optimized out>,
    fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffca7c7bea8) at ../csu/libc-start.c:310
#8  0x000055fff93b7a4a in ?? ()

I've added -g flag but it seems that the program hangs on wait4 before entering main .

Thanks for any insights on:

• How can I load these debug symbols to get rid of ??
• Where is ../csu/libc-start.c:310 located?
• What else can I do to locate the bug?

System Info: gcc 6.5.0 , Ubuntu 18.04 with 4.15.0-54-generic .

Answer 1

How can I load these debug symbols to get rid of ??

You appear to need the debug symbols for /bin/dash , which are probably going to be in a package called dash-dbg or dash-dbgsym or something like that.

Also, I suspect your stack trace would make more sense if you compiled your library with -fno-optimize-sibling-calls .

Where is ../csu/libc-start.c:310 located?

See this answer .

What else can I do to locate the bug?

You said that you are writing a library that uses __attribute__((constructor)) , but you showed a stack trace for /bin/dash (which I presume is DASH and not a program you wrote) that does not appear to involve symbols from your library. I infer from this, that your library is loaded with LD_PRELOAD into programs that are not expecting it to be there.

Both of those things -- LD_PRELOAD and __attribute__((constructor)) -- break the normal expectations of both whatever unsuspecting program is involved, and the C library. You should only do those things if you have no other choice , and you should try to do as little as possible within the injected code. (In particular, I do not think any design that involves spawning processes from a constructor function will be workable, period.) If you tell us about your larger goals we may be able to suggest alternative means that are less troublesome.

---

EDIT:

 subprocess.Popen('<path-to-binary>', shell=True) 

With shell=True , Python doesn't invoke the program directly, it runs a command of the form /bin/sh -c 'string passed to Popen' . In many cases this will naturally produce a /bin/dash process sleeping (not hung) in a wait syscall for the entire lifetime of the actual binary. Unless you actually need to evaluate some shell code before running the program, try the default shell=False instead and see if that makes your problem go away. (If you do need to evaluate shell code, try Popen('<shell code>; exec <binary>', shell=True) .)