在Linux中，不使用GDB进行实时调试，如何在挂起C程序时查找其状态？

Question

We have an ancient network application (a client program running on one machine and a server program on another). It usually works fine, but in rare cases it hangs. We are yet to identify what exactly is causing the hang. It's plain old socket programming written in C.

The problem is, we can't reproduce this, it just happens sometimes.

What can I do to find out the status of both server and client programs(code blocks where it's hung) when it hangs? Attaching to debugger is not possible for various reasons(the program immediately needs to be killed to release the blocked port).

I just read about gcore, I will give it a try.

Any better ideas?

Update: gcore(or other ways to generate core dumps) worked perfectly fine.

Answer 1

Few I am familiar with

Tip Number 1

pstack

usage

pstack <process Id>

for example

below is the sample out for bash process with pid 3258

     PID TTY          TIME CMD
 3258 pts/6    00:00:00 bash
 3326 pts/6    00:00:00 ps


    pstack 3258
#0  0x0000003ab5eaa77e in waitpid () from /lib64/libc.so.6
#1  0x000000000043ebf2 in ?? ()
#2  0x000000000043fe4f in wait_for ()
#3  0x0000000000430a59 in execute_command_internal ()
#4  0x0000000000430c4e in execute_command ()
#5  0x000000000041d536 in reader_loop ()
#6  0x000000000041ccf9 in main ()

Tip Number 2

gdb attach <process id>

and then

(gdb) info threads 

Quite helpful

there are few others mentioned earlier like gcore , netstat also helps in determining the state

Answer 2

Sending abort signal to the process will also generate core dump which can be later analysed using gdb. Similar to gcore.