How to debug the Linux kernel with QEMU and KGDB?

Question

I have been able to boot a powerpc based system (MPC8544DS to be specific) using the following way to invoke qemu (v1.7.0)

qemu-system-ppc -M mpc8544ds -m 512 -kernel zImage -s -nographic -initrd busyboxfs.img -append "root=/dev/ram rdinit=/bin/sh kgdboc=ttyS0,115200 kgdbwait"

where zImage is a custom cross compiled Linux Kernel (v2.6.32) which has KGDB enabled and compiled in (for startupcode debugging) and busyboxfs.img is the busybox based rootfs.

Since I'm using the -s flag to Qemu, I can break-in to the kernel using cross gdb like so:

(gdb) target remote localhost:1234
Remote debugging using localhost:1234
mem_serial_in (p=<value optimized out>, offset=5) at drivers/serial/8250.c:405
405  }

However if I remove the -s flag and try to break in to the kernel over /dev/ttyS0 it gives me a permission denied error:

(gdb) set remotebaud 115200
(gdb) target remote /dev/ttyS0
permission denied 

Is it because it has been held over by Qemu? Additionally in example across the internet, kgdboc has been set to ttyAMA0 which I've come to understand stands for the AMBA bus which is specific to ARM based systems. Do we have something similar for PowerPC? Am I doing something wrong here?

Answer 1

KGDB + QEMU step-by-step

First, QEMU's -gdb option is strictly more powerful than KGDB, so you might want to use that instead: How to debug the Linux kernel with GDB and QEMU? QEMU is however an easy way to play around with KGDB in preparation for real hardware. I have posted some Raspberry Pi KGDB pointers at: Linux kernel live debugging, how it's done and what tools are used?

If you want to get started quickly from scratch, I've made a minimal fully automated Buildroot example at: https://github.com/cirosantilli/linux-kernel-module-cheat/tree/d424380fe62351358d21406280bc7588d795209c#kgdb

The main steps are:

1. Compile the kernel with:

    CONFIG_DEBUG_KERNEL=y CONFIG_DEBUG_INFO=y CONFIG_CONSOLE_POLL=y CONFIG_KDB_CONTINUE_CATASTROPHIC=0 CONFIG_KDB_DEFAULT_ENABLE=0x1 CONFIG_KDB_KEYBOARD=y CONFIG_KGDB=y CONFIG_KGDB_KDB=y CONFIG_KGDB_LOW_LEVEL_TRAP=y CONFIG_KGDB_SERIAL_CONSOLE=y CONFIG_KGDB_TESTS=y CONFIG_KGDB_TESTS_ON_BOOT=n CONFIG_MAGIC_SYSRQ=y CONFIG_MAGIC_SYSRQ_DEFAULT_ENABLE=0x1 CONFIG_SERIAL_KGDB_NMI=n 

   Most of those are not mandatory, but this is what I've tested.

2. Add to your QEMU command:

    -append 'kgdbwait kgdboc=ttyS0,115200' \\ -serial tcp::1234,server,nowait 

3. Run GDB with from the root of the Linux kernel source tree with:

    gdb -ex 'file vmlinux' -ex 'target remote localhost:1234' 

4. In GDB:

    (gdb) c 

   and the boot should finish.

5. In QEMU:

    echo g > /proc/sysrq-trigger 

   And GDB should break.

6. Now we are done, you can use GDB as usual:

    b sys_write c 

Tested in Ubuntu 14.04.

ARM

Can't get it work. Possibly related to: How to use kgdb on ARM??

Answer 2

You appear to be confusing the host serial device /dev/ttyS0 for the guest one, and QEMU's own gdbserver for KGDB in the guest kernel.

There is normally no reason for QEMU to touch the host's serial port. Really the only reason for doing that would be if you wanted to have one physical machine host QEMU, and effectively give its physical serial port to the guest, so that you could then use a different physical machine connected by an actual serial cable to debug the guest.

When you use the -s flag, you tell QEMU to run its own GDB server (by default listening on host loopback TCP port 1234) allowing you to break into whatever program is running on the guest, be that a kernel or bootloader or something else. This is not the same as having the guest kernel itself cooperate with debugging via KGDB.

If you want to use KGDB, you are going to need to so something like configure KGDB in the kernel build to use the guest side of an emulated serial port, and then tell GDB on the host to use the host end of that emulated port. The QEMU command line documenation covers this in detail:

Debug/Expert options:

'-serial dev' Redirect the virtual serial port to host character device dev. The default device is vc in graphical mode and stdio in non graphical mode.

This option can be used several times to simulate up to 4 serial ports.

An abbreviated list of some of your more interesting options:

'pty' [Linux only] Pseudo TTY (a new PTY is automatically allocated)

'/dev/XXX' [Linux only] Use host tty, eg '/dev/ttyS0'. The host serial port parameters are set according to the emulated ones.

This is what you don't want - unless you want to use a serial cable to a different physical machine that will run GDB.

'tcp:[host]:port[,server][,nowait][,nodelay]' The TCP Net Console has two modes of operation. It can send the serial I/O to a location or wait for a connection from a location. By default the TCP Net Console is sent to host at the port. If you use the server option QEMU will wait for a client socket application to connect to the port before continuing, unless the nowait option was specified. The nodelay option disables the Nagle buffering algorithm. If host is omitted, 0.0.0.0 is assumed. Only one TCP connection at a time is accepted. You can use telnet to connect to the corresponding character device.

Example to send tcp console to 192.168.0.2 port 4444 -serial tcp:192.168.0.2:4444

Example to listen and wait on port 4444 for connection -serial tcp::4444,server

Example to not wait and listen on ip 192.168.0.100 port 4444 -serial tcp:192.168.0.100:4444,server,nowait

This is a good and common choice. You can use basically the same GDB syntax, for example if you specify the loopback interface address 127.0.0.1 and the port 1234 you can just use exactly the same GDB command as before.

'unix:path[,server][,nowait]' A unix domain socket is used instead of a tcp socket. The option works the same as if you >had specified -serial tcp except the unix domain socket path is used for connections.

This is a good choice too, assuming your GDB supports it.

You may need to configure one of these options first, run without KGDB and get a shell up and figure out what the guest end of the emulated device is called, then reboot with KGDB configured to use that.