How to build a Debian Package for GDB

I've resisted titling this post as 'building an Ubuntu package' even though I'm building it for Ubuntu - technically it's more proper to call it a Debian package given its lineage. Nevertheless the mechanism behind building your own packages is pretty much the same for the two.

I'll use GDB as an example of how to build your own package - for a good reason, firstly because the stock version of GDB that is shipped with Ubuntu is terribly broken. Here's what I mean:

% gdb --args java
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-linux-gnu"...
(no debugging symbols found)
(gdb) r
Starting program: /usr/bin/java Test
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
[Thread debugging using libthread_db enabled]
[New process 16487]
Executing new program: /usr/lib/jvm/java-6-openjdk/jre/bin/java
warning: Cannot initialize thread debugging library: generic error
warning: Cannot initialize thread debugging library: generic error
warning: Cannot initialize thread debugging library: generic error
warning: Cannot initialize thread debugging library: generic error
[New process 16487]
[Thread debugging using libthread_db enabled]
Segmentation fault

The stock build of GDB doesn't handle multi-threaded applications properly, among other minor issue like not setting the path to point to the correct debug library paths, which makes it unusable for serious debugging tasks.

Secondly, GDB 7.0 has reversible debugging, which makes it doubly tempting to roll my own. Finally, GDB has minimal external and library dependencies, which is an easy example to build a package without going into the complexity of having to generate a chrooted environment.

But why not just do the typical 'configure && make install' combination?

The drawback in doing so, is that the process is one-way: once you've installed it like that, there is no easy way of uninstalling it, short of remembering the list of what's being installed and removing them manually. Do-able? Sure, but certainly cumbersome. The neater way to do this is to create a package and have the package manager deal with installation/uninstallation for you.

Building GDB

We need to perform the usual compiling and installing steps like we normally do; the only difference is that we want the installer to place all the resultant files into a separate directory for generating a package. Doing this is straightforward using the prefix flag provided by configure. The steps are commented and reproduced below:

# Assuming that you're in the current source directory /home/user/gdb-sources
% mkdir -p custom-gdb-7.0-amd64/usr/
% configure --prefix=/home/user/gdb-sources/custom-gdb-7.0-amd64/
% make && make install

There are dependencies that GDB will need in order to compile properly (things like bison, lex as far as I remember), but I'll assume that you know how to resolve these dependencies yourself. Otherwise, the source should finish compiling and installing to the destination /home/user/gdb-sources/custom-gdb-7.0-amd64/.

Generating the Control file

In order to generate a package, a Debian control file is required, which contains the information that the 'dpkg-deb' package generator will need. Here's how we write one:

% mkdir custom-gdb-7.0-amd64/DEBIAN
% cat > custom-gdb-7.0-amd64/DEBIAN/control
Package: customgdb
Version: 7.0
Section: base
Priority: optional
Architecture: amd64
Depends: lex, bison
Maintainer: Vincent Liu <blog@vinceliu.com>
Description: Custom build of GDB
This version of GDB provides cutting edge
capabilities that the stock package does not provide.
^D
%
# The control-D symbol above is to indicate the
# file termination character

There are plenty of details I've omitted here, and you will have to read more to understand and tune your own control file configuration. Here's the tutorial I referenced, and the Debian manual to help you figure out the details of each control field.

Generating and Installing the Package

Once you've got the control file generated, building the package is just a single dpkg-deb away:

% fakeroot dpkg-deb --build custom-gdb-7.0-amd64

You will get a resulting custom-gdb-7.0-amd64.deb package generated for installation. To install it, you'll have to remove the existing GDB package, as it conflicts with your new installation. Do the following:

# remove the original gdb
% dpkg -r gdb

# install the new gdb
% dpkg -i custom-gdb-7.0-amd64.deb

If you ever needed to revert back to the stock versions of GDB, you can now easily remove your custom version by dpkg -r customgdb, and reinstalling it using apt-get or your own favourite package manager.

GDB 7.0 Is Out!

Read the announcement here. It's actually the pre-release version, nevertheless it's quite stable.

What makes this version interesting? It's got reversible debugging, which means that it's the first time you can actually make your code go back in time to find out what it did before it crashed.

I've tried the new commands, but I'm haven't been successful in stepping backwards still - well more experimenting ahead!

Update: The tutorial is available at
http://www.sourceware.org/gdb/wiki/ProcessRecord/Tutorial. (Thanks to Michael!)

GDB: Relaying Trapped OS Signals

By chance, I have managed to land myself in a situation where the bug occurs only at a signal handler. In my situation, this means that it happens only when I try to kill the program using a SIGINT, or more commonly known as the 'ctrl-c' keystroke.

GDB usually traps this signal, and other signals such as SIGSEGV (Segmentation Fault), so that you can trace buggy behaviour that is causing your application to fail. But once in a while, the error may occur after the signal is sent, when the code failure resides within the signal handler.

However the default behaviour of GDB is to trap these signals and subsequently consume them, in effect preventing the bug from occurring. To prevent unwanted behaviour in rare cases such as mine, you need to issue the following command:

(gdb) handle SIGINT pass

Given that GDB utilises this signal internally, it will ask you for a confirmation to change it. Say 'y' to it, and GDB will correspondingly pass the signal to the application after trapping, which will give you a chance to debug the handler code that is causing the bug.

GDB's Conditional Breakpoints

Conditional Breakpoints for Scalar Types

Let's assume that you, the brilliant hacker, has coded up some really uber-cool stuffs, like this piece of code below:

1: for ( int i = 0; i < gazillion; i++ ) {
2:   doSlightlyBuggyButUberCoolStuffs(i)
3: }
4:
5: void doSlightlyBuggyButUberCoolStuffs(int i) {
6:   // your code here that needs some
7:   // fixing before it becomes uber-cool
8: }

It is doing all the cool stuffs as intended, but somehow something always goes wrong when the code executes up to 2147483648, which is kind of puzzling.

So what to do?

You may be tempted to breakpoint at line 5, at the start of the doSlightlyBuggyButUberCoolStuffs():

(gdb) br doSlightlyBuggyButUberCoolStuffs

And gdb dutifully does what it's told; every single time doSlightlyBuggyButUberCoolStuffs() gets executed, it stops and waits for you to act on it:

Breakpoint 1, doBuggyButUberCoolStuffs (i=1) at test.cpp:6
6:        // start of your uber-cool code
(gdb) c

Breakpoint 1, doBuggyButUberCoolStuffs (i=2) at test.cpp:6
6:        // start of your uber-cool code
(gdb) c

.....

Breakpoint 1, doBuggyButUberCoolStuffs (i=100) at test.cpp:6
6:        // start of your uber-cool code
(gdb) c

After 100 iterations, you think you've had enough! So it's time to do it the smart way, by setting a conditional:

(gdb) br test.cpp:2
Breakpoint 1 at 0x1234: file test.cpp, line 2.
(gdb) cond 1 i==2147483648
(gdb) run

After the breakpoint is set, gdb only notifies you when the loop is at its 2147483648th iteration:

Breakpoint 1 at 0x5678: file test.cpp:2
2:   doBuggyButUberCoolStuffs(i)
(gdb) s
6:         // start of your uber-cool code
(gdb) p i
$1 = 2147483648

Jackpot! You're now at the 2147483648th iteration! And very soon after, you found the offending piece of code, caused by a numerical overflow of a signed integer. Another bug trampled, and peace returns to your realm once more.


Conditional Breakpoints for char* Strings
But very soon after, you run into another irritating problem which is happening within another section of your uber-cool code. This time, the conditional depends on parsing a huge portion of text that comes from, um..., /dev/random :P

1: while ( true ) {
2:   char* c = getStringFromDevRandom();
3:   launchNuclearMissileIfCodeMatch(c);
4:}

Somehow, you are absolutely convinced that /dev/random will eventually provide correct codes to launch the nuclear missile, but given that launchNuclearMissileIfCodeMatch() is a really top-secret and highly obfuscated code residing in an external library called libtopsecret.so, it isn't such a good idea to debug into the call unless you want the NSA bursting through your front doors...

But since you do know the launch code (it's one of those things that you'll have to kill your friends if you ever told them), you can perform a conditional check on the string, and breakpoint at it to tell you if the secret code is ever generated by /dev/random to find out if launchNuclearMissileIfCodeMatch() is really a hoax:

(gdb) br test.cpp:3
Breakpoint 1 at 0xdeadbabe: file test.cpp, line 3.
(gdb) set $secret_code = "MyUberSecretivePassword"
(gdb) cond 1 strcmp ( $secret_code, c ) == 0
(gdb) run

And then, you let your code run... (!)

Well, unfortunately, you get sick of sitting around and waiting for it to happen after a whole day. It seems like /dev/random doesn't really generate your uber-secret nuclear launch codes as frequently as you would like to think. In the meantime, the world thanks their lucky stars that you haven't caused a nuclear winter to materialise just yet... :)