How to debug a multi-threaded program that only deadlocks as a standalone application

Question

I have a C++ program that uses 2 threads. The program runs fine in Debug and Release mode when run through Visual Studio. However, when run as a stand-alone application it always stalls. I tried to add in printf messages so that I could output the information to the screen, but that just fixed my problem. I take the messages back out and I'm back to the original problem again.

Anyone have any wise words of wisdom to share for an issue like this?

Answer 1

You can simply run the program stand-alone, wait until it deadlocks and then attach the Visual Studio using "Tools"->"Attach To Process...".

Then select your process and when you are attached, press Pause.

You can then use all debugger features and watch all your threads, stacks and variables...

Answer 2

If you have access to Visual Studio 2010 Professional (or better), you may want to try the profiler, through the Analyze->Launch Performance Wizard->Concurrency menu (enable the Visualize the behavior of a multithreaded application option).

Your program will be instrumented and run, and then you should be able to see which thread blocks which thread, with the relevant call stacks.

Answer 3

To find your bug, you need to collect some trace information. Your tool needs to meet the following requirements:

1. not add any delay
2. not use any locking
3. be multithreading safe
4. trace what happened in the correct sequence.

Most tools use locking and add too much delay, which will change the behavior of your program, which you found out already. But actually, you can write your own trace into memory. In C#, it looks something like this:

public const int MaxMessages = 0x100;
string[] messages = new string[MaxMessages];
int messagesIndex = -1;

public void Trace(string message) {
  int thisIndex = Interlocked.Increment(ref messagesIndex);
  messages[thisIndex] = message;
}

The method Trace() is multithreading safe, non blocking and can be called from any thread. On my PC, it takes about 2 microseconds to execute, which should be fast enough.

Add Trace() instructions wherever you think something might go wrong, let the program run, wait until the error happens, stop the trace and then investigate the trace for any errors.

A more detailed description for this approach which also collects thread and timing information, recycles the buffer and outputs the trace nicely you can find at: CodeProject: Debugging multithreaded code in real time 1

Answer 4

I haven't (to my memory) experienced the joy of debugging a Heisenbug in a multithreaded application, so take my advice with a grain of salt.

Debugging, in general, involves three key activities:

1. Observation: Run the program and see what happens, run the program under a debugger, add log messages, add asserts, etc.

2. Analysis: Look over the code you've already written. Document functions. Hopefully, you'll encounter a flaw in your logic mid-sentence.

3. Conjecture: Use your imagination to come up with things that could possibly go wrong.

Since (1) isn't working out in your situation, try (2) and (3). Look over your code to make sure you didn't make an obvious mistake. Imagine situations that could cause your program to deadlock. Skim the Wikipedia article on Deadlock .

Also, look for other ways to observe the behavior of the program. For example, see where you can add print statements without making the bug disappear, as that may provide a clue.

Answer 5

While I don't know your toolchain, an easy way to fix a deadlock with gcc is to build the program with debug information, but fully optimized, run the program until the deadlock occurs, and then kill it with a core-dump-producing signal (eg SIGSEGV ). The core dump gives you invaluable information about the deadlock.

I'm aware you are working with the Microsoft toolchain, but maybe you are able to translate this into their terms.