gdb how to break in new thread when debugging multi threaded daemon program on linux

Question

Before asking this question - I referred to How does gdb multi-thread debugging coordinate with Linux thread scheduling? for added context

I have a c++ daemonized multithreaded program running on linux. It launches a thread every time It processes a transaction . The code to be debugged is part of the transaction processing code invoked in new thread.

This is the process I follow to debug.

I start gdb, turn off pagination, turn on synchronous command execution and set non-stop mode.

Then I attach gdb to running daemon program and set breakpoints on "all" threads (See illustration below) and send a new transaction.

The problem is that gdb seeems to inform me clearly that breakpoint is in thread 1. I don't get a break on the breakpoint in the new thread. A new thread is created when I send a new transaction and exits without breaking at breakpoint.

Please help me understand what is the possible reason (what am I missing)

(gdb) set pagination off
(gdb) set target-async on
(gdb) set non-stop on
(gdb) attach 11067 # <daemon pid>  
(gdb) thread apply all b foo 
Thread 1 (Thread 0x7f94bb9f3740 (LWP 11067)): Breakpoint 1 at <filename> , 
line <line#>
(gdb) c
 Continuing.
 [New Thread 0x7f94b725e700 (LWP 15750)]
 [Thread 0x7f94b725e700 (LWP 15750) exited]

The problem is - it did not break in function "foo" ( my breakpoint )
What is it that I am missing ? How do I instruct gdb to follow new child threads. Isn't "thread apply all" supposed to apply to "all" threads ?

Answer 1

i don't have enough reputation to comment, but in any event, as a brain-cramp check your gdb session doesn't show you actually creating a break point. from gdb, use info threads to see if the thread you expect to see is actually running. can you just start from the debugger instead of attaching? by starting then attaching you may be missing the interesting part. try setting a breakpoint on something like pthread_create() to see if your thread is getting created. there's my $0.02

have to respond here... if you have access to the source i've set breakpoints on dummy functions not called anywhere else as a convenience. other things i can think to check is did you compile with symbols. is it statically linked (can't break on libc stuff without symbols when statically linked). kinda running out of gas here, maybe someone else will have a better ideas?

Answer 2

I found a workaround that works :-)

While I still can NOT get gdb to break at break-point of my interest (occurs in a short lived thread spawed by main thread), I found a way to switch to that thread explicitly soon enough before that thread ended.

The problem is that gdb needs you to explicitly switch to the thread you are interested in, to stop at break-point.

since the thread that I was trying to debug was rather short lived I can not switch to it soon enough (before it finished) .

In addition to break-point in function of my interest I placed another break-point in a logger function that occurred often in all threads and kept continuing ( annoying for sure) till the thread of my interest was spawned and then switched to that explicitly when new thread was spawned and by listing threads.

(gdb) info threads
(gdb) Thread <thread id>

I am posting this so if you have similar problem you may try this workaround.

Your comments, better answers :-) more responses welcome