Debugging the Linux kernel with JTAG

As with many Linux-related topics, the issue of using debuggers to troubleshoot the Linux kernel is not only technical--it's political. Linux is being mostly developed on the x86 platform, which does not have JTAG debugging capabilities, and software-only kernel debuggers are complex and unreliable. Because of this and other reasons, Linus Torvalds objected for a long time to inclusion of the KGDB (Linux kernel debugger) patch in the main Linux tree until Ingo Molnar managed to get a rather slimmed down KGDB variant into the 2.6.26 version. Putting politics aside, I believe that most developers would agree with me, that at least in the embedded world, a kernel debugger is a must-have tool for many tasks, BSP (board support package) development being probably the most obvious example.

Fortunately, compared with the world of x86, embedded platforms introduce not only additional challenges but also a good tool to help us tackle the problem--a JTAG debugger. It's easy to use, reliable, provides some nice features that are not available in software-only debuggers and is free from any controversy that is always associated with invasive Linux kernel patches, such as the original KGDB.

I assume that the topic of JTAG debugging is not new to you--as an experienced embedded systems developer, you have probably used something like Wind River's On-Chip Debugging in the past. I will go very briefly over general JTAG debugging capabilities and show you the peculiarities of Linux kernel debugging using JTAG.

JTAG is your friend
JTAG (Joint Test Action Group) was initially developed as a way to test circuit boards after manufacture, however today it's more commonly used for debugging embedded systems. A JTAG adaptor, sometimes referred to as in-circuit emulator (ICE), is used to access on-chip debug modules inside the target CPU.

It allows, among other things, to halt the CPU, inspect its registers and memory, single step through the code, and define breakpoints. In addition to the hard, you will also need a software debugger that supports it. Some JTAG adapter vendors provide software tools while others rely on open source packages.

Even though most JTAG debuggers nowadays support Linux, if you're shopping for a low-cost device, I suggest you ask whether it supports Linux, which boils down to memory management units (MMU), Linux binary formats, and loadable modules support. If it supports remote GDB (GNU debugger) protocol as well, you can be sure that it's going to work.

Cast of characters
I will use my setup of a FemtoLinux project throughout this article to demonstrate how to debug the Linux kernel on ARM, so it's beneficial to describe the setup first. The aim of this project is to reduce the Linux system's call latency and overhead in order to make it possible to port VxWorks 5.5 monolithic applications to Linux without redesign. As you would imagine, this kind of low-level Linux kernel hacking would be nearly impossible without JTAG.

For development, we use a LPC3250 board by Embedded Artists. It's a very convenient platform based on NXP CPU with ARM926 core. The hardware specs being pretty impressive are of no particular interest in the scope of this article, but what I really like about this board is that it's well thought out and specifically designed for developer's needs. Features such as power over USB combined with USB-to-serial bridge clearly shows that.

Naturally, the board has a 20-pin JTAG interface for debugging and a full Linux support. As USB JTAG adapter, we use the Flyswatter by Tin Can Tool. This is rather basic device, but it too surprised me very positively--before I found it, I had no idea that it was possible to get a fully functional JTAG debugger for less than $100.

I guess that one of the reasons Tin Can Tool were able to introduce this low-cost device is that they didn't invest in the debugging software at all, relying instead on open source software packages OpenOCD and GDB, which leads us to the most important part of the article--software tools for JTAG debugging.

Software tools for JTAG
In the scope of this article, OpenOCD--the on-chip debugger (OCD)--can be thought of as a software middleware running between the debugger, for instance GDB, and the JTAG adapter. It does more than that, but as far as JTAG debugging is concerned, OpenOCD translates GDB commands to USB commands, which it sends to the JTAG adapter.

OpenOCD implements a remote gdbserver protocol, so for a GDB client it appears as an instance of gdbserver, which just happens to debug the Linux kernel itself instead of one of the applications running under Linux.

OpenOCD supports ARM7, ARM9, XScale, and Cortex-M3 cores, which means that if you are using MIPS or PowerPC, you'll have to look for a different and usually more expensive JTAG debugger, such as Abatron BDI3000. OpenOCD works with a fairly large number of JTAG adapters, in particular FT2232-based ones.

FT2232 is a very popular chip manufactured by Future Technologies Devices International and used in many low-end JTAG adapters, such as Tin Can Tools' Flyswatter. In addition to GDB support, OpenOCD has an extensive list of commands available through telnet command line interface that include target state command, memory access, flash programming, ARM specific commands, JTAG low-level commands, and much more. OpenOCD comes with exhaustive documentation that explains in detail every command, configuration, devices support, and more; I encourage you to read it.

GDB is in the house
The last part of the setup puzzle is GDB, or the GNU debugger. It's a standard and widely used debugger for UNIX systems, including Linux, and it hardly needs an introduction. It offers extensive facilities for program tracing, altering, and debugging. A full GDB manual is beyond the scope of this article, so I'll cover only what is relevant for JTAG and embedded debugging.

Most commonly GDB is used as a single application; however it also has a client server mode. In this mode, GDB client, in other words the front end, can be run on one machine (usually the host PC) and GDB server, the back end, on another (usually the embedded target system). GDB server is used to control the application, and GDB client is the user interface: they both communicate using GDB's remote TCP-based protocol.

This configuration is what you would normally use when debugging embedded Linux applications. In the JTAG debugging case, the OpenOCD daemon and the JTAG adapter would simulate the gdbserver, while the GDB client will remain the same.

One important drawback of GDB is a lack of graphical user interface (GUI). Most people use Data Display Debugger (DDD), a GUI wrapper for GDB. Personally, I'm not a big fond of this program as it has a very slow response time (by modern standards) and clunky interface.

Fortunately, there is a better alternative--Insight. Unlike DDD, Insight is not a wrapper but a version of GDB with integrated GUI. (See http://sourceware.org/insight/faq.php for more about Insight.) The upside is that the GUI is much nicer and easier to use, but there is a downside--as it has GDB built in, you'll probably have to compile it for your target CPU architecture, while DDD would allow you to use an external GDB version from your cross compiler toolchain.

Configuring OpenOCD
Assuming you've installed all the software I've mentioned and compiled your kernel, the last step you need to take before starting debugging is to configure OpenOCD. If you're lucky, OpenOCD may already include configuration files for your JTAG adapter and your board, so it may "just work." Actually, this was almost the case with Flyswatter and LPC3250 (I only had to modify the CPUTAPID parameter), but your mileage may vary.

Refer to OpenOCD documentation for a full explanation of the configuration. I'll explain only the basic commands using in my configuration as an example. Omitting default value boilerplate code, my configuration file looks like that in Listing 1.

Click on image to enlarge.

The first part is the interface configuration, which defines the type of JTAG adapter you are using. The "interface" command identifies the driver that should be used, FT2232 in my case, and the ft2232_vid_pid commands specify vendor Id and product Id of the FT2232 device.

The rest are the target board configuration commands. The "jtag newtap" command defines a JTAG TAP in the chain. You will need at least one for your CPU, along with the set_TARGETNAME and the "target create" commands.

Here is what an actual JTAG debugging session looks like. As you can see in Figure 1, LPC3250 board is connected to Flyswatter via 20-pin JTAG cable and both devices are connected via USB to a Linux PC. Linux is running OpenOCD daemon and Insight GDB client.

Click on image to enlarge.

Debug session:
1. Power up the JTAG adaptor.
2. Power up the board.
3. Mount the usbfs (if it is not mounted already): sudo mount -t usbfs none /proc/bus/usb
4. Run the OpenOCD daemon: sudo bin/openocd -f flyswatter.cfg -f lpc3250.cfg
5. Run the Insight (or any other GDB compatible) debugger: arm-unknown-linux-gnueabi-insight vmlinux
6. Set the target settings (post 3333 is the default used by OpenOCD, you can change it in the configuration file):

    Target: Remote/TCP
    Hostname: localhost
    Port:3333

7. Connect to the target.

You will need an uncompressed kernel image vmlinux, preferably compiled with debugging symbols, in other words, CONFIG_DEBUG and CONFIG_DEBUG_INFO kernel build parameters should be enabled.

If you connected successfully, you can now use the graphical debugger interface to debug the Linux kernel. You can single step in both assembly and source code (I especially like the mixed assembly/C mode for low-level debugging), set breakpoints, check/modify memory and registers, and generally debug the kernel as you normally would debug an application. The screenshot in Figure 2 shows main Insight windows.

Click on image to enlarge.

Of course, if you prefer a command line interface, a console window is available as well. In addition to all standard GDB commands, you can send JTAG commands directly to OpenOCD using a monitor command. For instance, you can check all banked registers using arm reg command as is shown in Listing 2, or you can print information about the caches using arm926ejs cache_info command shown in Listing 3. (For a full list of supported commands refer to the OpenOCD documentation.)

Click on image to enlarge.

Click on image to enlarge.

The instructions that I've described so far assume that you're interested in debugging a statically linked kernel image. This is true most of the time--after all, JTAG is most useful for an early initialization debugging, and this code is of course statically linked.

Sometimes, however, you may wish to debug a driver or some other loadable kernel module. The problem here is that after you load the module using insmod or modprobe Linux commands, the GDB/Insight is not aware of this module's symbols. It's your responsibility to tell the debugger what the symbols names are and to where they've been relocated.

You can find the base address of each module by reading the /proc/modules file. For instance, you can see that on my system, the nfs module has been relocated to the address 0xf8716000:

cat /proc/modules
    nfs 271912 1 - Live 0xf8716000
    ...

The offset is printed in the last column. When you know the offset, you can tell GDB to load this module's symbol table using the following command:

add-symbol-file nsf.ko 0xf8716000

When you are finished, you can continue debugging the kernel and your module.

A good start
This article is by no means a definitive guide to JTAG debugging Linux, but I hope that it gives you enough information to get started. As usually the case with Linux, there are alternatives to almost everything I've described here.

For instance, if you're unwilling to spend the time assembling the system by yourself and money is not an issue, Wind River Workbench On-Chip Debugging is a good option. It supports pretty much all processors currently used in embedded systems and comes with a nice integrated development environment. Needless to say, it supports both VxWorks and Linux.

Another slightly less expensive option is Abatron BDI3000. Abatron also supports all relevant processors and can be used with any client that supports remote GDB protocol, in much the same way as I described in this article. The major difference is that it has an Ethernet rather than USB interface and runs a GDB server emulation software--the equivalent of OpenOCD, inside the device itself, which is very convenient.

If you prefer an Eclipse-based integrated environment but cannot afford Wind River OCD, you can purchase an add-on software product LinuxScope-JTD (JTAG target debugger), which is a debugger for the Eclipse IDE that has been optimized for use with Abatron's probes.

Sasha Sirotkin currently works on the FemtoLinux project improving Linux system call overhead for embedded ARM, MIPS, and PowerPC systems. For more information visit FemtoLinux's web site at www.femtolinux.com. Sasha can be reached at demiurg@femtolinux.com.

Resources:
1. FemtoLinux, www.femtolinux.com
2. EmbeddedArtists, www.embeddedartists.com
3. Tin Can Tools, www.tincantools.com
4. OpenOCD, http://openocd.berlios.de/web/
5. Insight, http://sourceware.org/insight/
6. LinuxScope-JTD, www.ultsol.com/mfgs_comp_linuxscope.htm, or www.linuxscope.com/
7. Wind River OCD, www.windriver.com/products/OCD/
8. Abatron BDI3000, www.abatron.ch/products/bdi-family/bdi3000.html