调试多线程服务器

Question

I was asked this on an interview and now I'm curious because I don't think interviewer was satisfied with my answer. Here's the question:

A Multi-threaded server application stops working and the last log message from the application is:

"Some Server Related Message..."

Code looks like:

CalledFunc ()
{
    Code ...

    Acquiring Thread lock
    Line printing "Some Server Related Message..."
    Func();
    Releasing Thread Lock
}

1. What should the programmer in charge do to debug this?
2. What has happened wrong in the Func() ?
3. If an exception is thrown in the Func() what should be done to fix problem ?

Answer 1

Reason #1: it's a database problem. This may sound strange, but the main reason an application server hangs is not directly related to the application server itself. The location of the symptom is rarely the location of the root cause. The following scenario is quite common:

The database is bottlenecked, causing queries to run slower than usual. Requests that used to take 1 second, now take 5 seconds to complete. The average number of concurrent requests slowly increases (due to backlog). The server runs out of threads and the application server hangs. If you manage to get a thread dump, you'll just see a bunch of threads waiting and another group that's actually running. Another possibility is that the number of waiting threads (or queued threads) will gobble up all available memory and, eventually, lead to an OutOfMemory error.

Reason #2: deadlocks. If it seems that the application server is doing nothing, look for deadlocks. These can be database deadlocks that cause your SQL queries to hang, or seek the update statements. For example, a transaction log that is written to the database for each request may easily hang the entire application if the log table is locked. Also check for shared objects—an operating system file that is written to from multiple threads at once.

Reason #3: run-away thread. In cases where the application server is indeed to blame, you should look for a run-away thread. These are hard to detect because they hardly show up on logs since they are usually only written when the request has completed. A run-away thread will probably not return until it has already affected the entire application. Therefore, the hanging request will not be written to the log. These 'runaway' threads typically include infinite loops or code that results in consuming too much heap memory resulting in out of memory. For example, a query that should show results that does not include the option of paging between result pages suddenly needs to display a large number of results. The page takes forever to render and clobbers the application server, eventually causing it to hang.

Answer 2

It is likely that either:

• Func() is trying to get the lock once again (easy to check), or
• Func() has thrown an exception with the lock locked (more likely and subtle)

So:

1. Check the code of Func() to check if all possible paths (exceptions included) release the lock
2. One of the two options above
3. Release the lock before throwing the exception or catch the exception in CalledFunc() and release the lock

Answer 3

To overcome the problem with the exception in Func() you can use a scoped lock. RAII is a good way to ensure exception safety and avoid leaks in general. That link also happens to have a mutex as an example.

Also, seeing that line in the log doesn't mean that the problem comes from this part of the code.

Answer 4

I think they where looking for this:

What should the programmer in charge do to debug this?

Get a hang dump of the process and then use windbg to figure out the cause, ie if its a dead lock then it will be obvious from the dump.

What has happened wrong in the Func() ?

From what the next question asks we can assume it must have thrown an exception as some point, causing the lock to never be released, or it attempted to get the lock again causing a dead lock.

If an exception is thrown in the Func() what should be done to fix problem ?

Use RAII to be exception safe and for better/cleaner code.