C / C ++程序中的损坏堆栈问题

Question

I am running a C/C++ program in linux servers to serve videos. The program's(say named Plugin) core functionality is to convert videos and we fork a separate Plugin process for each video request. But I am having a weird problem for which sometimes server load average gets unexpectedly high. What I see from top command at this stage is that there are some processes which are running for long time and taking some huge CPU's.

When I debug this running program with gdb and backtrace stack,what I found is the corrupt stack: "Previous frame inner to this frame (corrupt stack?)". I have searched the net and found that this occurs if the program gets segmentation fault.

But what I know if the program gets segmentation fault, the program should crash and exit at that point. But surprisingly the program still running after segmentation fault.

What can be the causes of this? I know there must be some big problems in the program but I just can't understand from where to start fixing the problem...It would be great if any of you can show me some lights...

Thanks in advance

Answer 1

Attaching the debugger changes the behavior of the process so you won't get reliable investigation results most probably. Corrupted stack message from the debugger can mean that the particular debugger does not understand text info from the binary.

I would recommend running pstack several time subsequently on the problematic (this is known as "Monte Carlo performance profiling") and also attach strace or truss to the problematic and check what system calls is the process doing when consuming CPU.

Answer 2

在Valgrind下运行程序并修复它找到的任何无效内存写入。

Answer 3

某些优化（例如帧指针遗漏）会使调试器更难理解堆栈。

Answer 4

If you have the code, compile the program in debug and run Valgrind on it.

If you don't have the code, contact the author/provider of the program.

The corrupt stack message simply means the code is doing something weird with the memory. It does not mean the program has a segmentation fault. Also, the program can still run if it choose to handle the SIGSEGV signal.

If by forking you mean that you have some process which spawn and run other smaller processes, just monitor for such spikes and restart the process. This assumes that you have no access to the fix the program.

Answer 5

通过汇编代码操作可能会对堆栈进行一些有趣的操作，例如真正的尾递归优化，自修改代码，非返回函数等，这些操作可能导致调试器无法正确回溯跟踪堆栈并导致它触发损坏的堆栈错误，但这并不一定意味着内存已损坏......但绝对是非传统的东西发生在引擎盖下。