GDB/DDD: Debug shared library with multi-process application C/C++

Question

I am trying to debug a server application but I am running into some difficulties breaking where I need to. The application is broken up into two parts:

• A server application, which spawns worker processes (not threads) to handle incoming requests. The server basically spawns off processes which will process incoming requests first-come first-served.
• The server also loads plugins in the form of shared libraries. The shared library defines most of the services the server is able to process, so most of the actual processing is done here.

As an added nugget of joy, the worker processes "respawn" (ie exit and a new worker process is spawned) so the PIDs of the children change periodically. -_-'

Basically I need to debug a service that's called within the shared library but I don't know which process to attach to ahead of time since they grab requests ad-hoc. Attaching to the main process and setting a breakpoint hasn't seemed to work so far.

Is there a way to debug this shared library code without having to attach to a process in advance? Basically I'd want to debug the first process that called the function in question.

For the time being I'll probably try limiting the number of worker processes to 1 with no respawn, but it'd be good to know how to handle a scenario like this in the future, especially if I'd like to make sure it still works in the "release" configuration.

I'm running on a Linux platform attempting to debug this with DDD and GDB.

Edit: To help illustrate what I'm trying to accomplish, let me provide a brief proof on concept.

#include <iostream>
#include <stdlib.h>
#include <unistd.h>

using namespace std;

int important_function( const int child_id )
{
    cout << "IMPORTANT(" << child_id << ")" << endl;
}

void child_task( const int child_id )
{
    const int delay = 10 - child_id;
    cout << "Child " << child_id << " started. Waiting " << delay << " seconds..." << endl;
    sleep(delay);
    important_function(child_id);
    exit(0);
}

int main( void )
{
    const int children = 10;
    for (int i = 0; i < 10; ++i)
    {
        pid_t pid = fork();
        if (pid < 0) cout << "Fork " << i << "failed." << endl;
        else if (pid == 0) child_task(i);
    }

    sleep(10);
    return 0;
}

This program will fork off 10 processes which will all sleep 10 - id seconds before calling important_function , the function in which I want to debug in the first calling child process (which should, here, be the last one I fork).

Setting the follow-fork-mode to child will let me follow through to the first child forked, which is not what I'm looking for. I'm looking for the first child that calls the important function.

Setting detach-on-fork off doesn't help, because it halts the parent process until the child process forked exits before continuing to fork the other processes (one at a time, after the last has exited).

In the real scenario, it is also important that I be able to attach on to an already running server application who's already spawned threads, and halt on the first of those that call the function.

I'm not sure if any of this is possible since I've not seen much documentation on it. Basically I want to debug the first application to call this line of code, no matter what process it's coming from. (While it's only my application processes that'll call the code, it seems like my problem may be more general: attaching to the first process that calls the code, no matter what its origin).

Answer 1

You can set a breakpoint at fork(), and then issue "continue" commands until the main process's next step is to spawn the child process you want to debug. At that point, set a breakpoint at the function you want to debug, and then issue a "set follow-fork-mode child" command to gdb. When you continue, gdb should hook you into the child process at the function where the breakpoint is.

If you issue the command "set detach-on-fork off", gdb will continue debugging the child processes. The process that hits the breakpoint in the library should halt when it reaches that breakpoint. The problem is that when detach-on-fork is off, gdb halts all the child processes that are forked when they start. I don't know of a way to tell it to keep executing these processes after forking.

A solution to this I believe would be to write a gdb script to switch to each process and issue a continue command. The process that hits the function with the breakpoint should stop.

A colleague offered another solution to the problem of getting each child to continue. You can leave "detach-on-fork" on, insert a print statement in each child process's entry point that prints out its process id, and then give it a statement telling it to wait for the change in a variable, like so:

{
    volatile int foo = 1;
    printf("execute \"gdb -p %u\" in a new terminal\n", (unsigned)getpid());
    printf("once GDB is loaded, give it the following commands:\n");
    printf("    set variable foo = 0\n");
    printf("    c\n");
    while (foo == 1) __asm__ __volatile__ ("":::"memory");
}

Then, start up gdb, start the main process, and pipe the output to a file. With a bash script, you can read in the process IDs of the children, start up multiple instances of gdb, attach each instance to one of the different child processes, and signal each to continue by clearing the variable "foo".