Child processes gives seg fault on exit function

Question

My environment:

• FreeBSD 9.1 and I got a pre-forked multi-process environment.
• A main process, a poller to manage workers (to decide busy or avaiable) and workers.

If I send a SIGTERM to main process it is all right, if count of workers is not high. If high (x > 20) I get seg faults.

I got two truss outputs, one of them is normal exit and the other has seg fault.

gdb output of core file :

Die: DW_TAG_unspecified_type (abbrev = 62, offset = 1535499)
    has children: FALSE
    attributes:
        DW_AT_name (DW_FORM_strp) string: "decltype(nullptr)"
Dwarf Error: Cannot find type of die [in module /usr/local/lib/gcc46/libstdc++.so.6]
Die: DW_TAG_unspecified_type (abbrev = 62, offset = 1535499)
    has children: FALSE
    attributes:
        DW_AT_name (DW_FORM_strp) string: "decltype(nullptr)"
Dwarf Error: Cannot find type of die [in module /usr/local/lib/gcc46/libstdc++.so.6]

truss output of normal exited process:

> close(0)                                         = 0 (0x0) close(7)   
> = 0 (0x0) gettimeofday({1360957175.665250 },0x0)           = 0 (0x0) write(5,"2013-02-15 21:39:35.665 [64591] "...,64) = 64 (0x40) close(5)
> = 0 (0x0)
> 
> sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x
> 0) = 0 (0x0)
> 
> sigprocmask(SIG_SETMASK,0x0,0x0)                 = 0 (0x0)
> 
> .............. .............. so many same sigprocmask call
> 
> 
> sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x
> 0) = 0 (0x0)
> sigprocmask(SIG_SETMASK,0x0,0x0)                 = 0 (0x0)
> process exit, rval = 0

process, that has seg fault :

> gettimeofday({1360954514.547064 },0x0)           = 0 (0x0)
> write(5,"2013-02-15 20:55:14.547 [7580] I"...,63) = 63 (0x3f)
> close(5)                                         = 0 (0x0)
> SIGNAL 11 (SIGSEGV)
> process exit, rval = 0

It gives seg fault when the process tries to call sigprocmask. I didn't call sigprocmask explicitly and suspect race condition for call to sigprocmask. Did you encounter race condition, that is related with sigprocmask ?

Answer 1

It gives seg fault when the process tries to call sigprocmask

I don't see that. It seems, the crash follows the closing of the 5th file-descriptor. It further appears, the descriptor 5 gets a log-message (with a timestamp) written right before getting closed.

Your "master" daemon tries to clean-up, and dies trying. I suggest, attaching gdb to it at run time -- just before sending the SIGTERM. When the SIGTERM is sent, tell gdb to continue.

When the SIGSEGV is sent, investigate.

I'd also highly recommend running the daemon under valgrind. You are likely to catch a lot of nits that way...