Python crashed; how to decode segfault in dmesg log?

Question

I have a Python daemon running on a 64-bit Linux box. It is crashing. Not a friendly, straightforward to debug, Python exception stack trace sort of crash, either-- this is a segmentation fault. Linux's dmesg log has a succinct post-mortem:

python2.7[27509]: segfault at 7fe500000008 ip 00007fe56644a891 sp 00007fe54e1fa230 error 4 in libpython2.7.so.1.0[7fe566359000+193000]
python2.7[23517]: segfault at 7f5600000008 ip 00007f568bb45891 sp 00007f5678e55230 error 4 in libpython2.7.so.1.0[7f568ba54000+193000]

libpython2.7.so.1.0 on this system has symbols and I can run objdump -d to get an assembly language dump. So I'm curious to know which function is causing the segfault.

How can I decode one of these dmesg segfault notices and find the errant function? One line says "7fe566359000+193000" and the next says "7f568ba54000+193000". I'm guessing this means both segfaults come from the same location. 193000 = 0x2f1e8. I thought that 0x2f1e8 would lead to an instruction in the Python library assembly dump, but it didn't; 0x2f1e8 is well out of range of the disassembly.

Answer 1

That is the address from the base of the library load, so you should compare it with the load address of .text as returned by (for example) eu-readelf :

flame@saladin ~ % eu-readelf -S /usr/lib/libpython2.7.so
There are 25 section headers, starting at offset 0x1b1a80:

Section Headers:
[Nr] Name                 Type         Addr             Off      Size     ES Flags Lk Inf Al
[ 0]                      NULL         0000000000000000 00000000 00000000  0        0   0  0
[….]
[10] .text                PROGBITS     000000000003f220 0003f220 000e02a0  0 AX     0   0 16
[….]

What you should be able to do is to use the address you got with the addr2line tool:

addr2line -e /usr/lib/libpython2.7.so 0x6e408

In this case I can't get the data because my copy of the library differs from yours so the address makes no sense.

Of course you still won't get a full backtrace unless you had a core file.