Using the command line interface of debug

Some notes on multithread debugging

The following notes discuss some of the intricacies of using debug on a multithreaded application like traverse. At this point, you know what the problem is with traverse. The focus here is performing the debugging session from a different perspective, that is, creating the process within the debugger and taking advantage of its features to learn more about the multithreaded application.

Threads off the LWP

You cannot apply the run or step commands on threads Off the LWP. Try running any of those threads and see what happens.

   debug> run
   Error: Invalid operation on idle thread p1.1
   debug> run -p p1.2
   Error: Invalid operation on idle thread p1.2
   debug>

You can, however, examine the state of such threads, print or set the contents of their registers or local variables, list their source among others.

NOTE: There is a small window when one thread is giving up its LWP to another thread during which the register contents of one or the other of the threads is inconsistent. debug will give a warning if you attempt to do an operation that depends on the register contents during this interval.

Since the program is in a deadlock state, traverse will never continue. Kill the program by using the kill command.

   debug> kill
   p1 killed
   No more processes
   debug>

Creating traverse within debug

In the current session, you grabbed traverse while it was executing in the background. In this new session, try executing the program from within the debugger so that you get a feel of how debug operates on a multithreaded operation. Create traverse from within debug and see how it progresses.

   debug> create traverse /var/tmp /var/cron
   New program traverse (process p1) created
   Created 1 thread(s) for process p1
   HALTED p2.1 [main in traverse.c]
   135:            int             exit_code = 0;
   debug>

At this point, there is only one thread, p2.1.

Spawning the threads

Run p2.1 and observe what happens.

   debug> run
   Thread p2.1 has created new thread p2.2
   debug> run
   Thread p2.1 has created new thread p2.3

As expected, traverse creates two threads (in addition to the primary thread, p2.1) since you have given it two directories to traverse.

NOTE: You can change the behavior of debug on thread state changes such as thread creation and exit by using the debugger variable %thread_change. If you had set %thread_change=ignore, you would not have seen any of the previous messages and traverse would have executed up to the point where it just stays hung. See ``Releasing or ignoring processes and threads'' and ``Debugger variable description'' for descriptions of this variable.

Examining the thread states

Enter the command ps to see the states of all three threads.

   debug> ps
   Program      ID     Pid Thread State     Function   Location       Command
   *traverse   p2.1   17903   1 Stopped    _thr_debug 0xbff8d97c    traverse /var/tmp /var/cron
    traverse   p2.2   17903   2 Off LWP    _thr_start 0xbff8b5d0    traverse /var/tmp /var/cron
    traverse   p2.3   17903   3 Off LWP    _thr_start 0xbff8b5d0    traverse /var/tmp /var/cron

The two created threads are off the LWP while the main thread is currently stopped. At this point, you can only issue the run command on p2.1 and not on the other two threads.

Setting a breakpoint for a specific thread

Although threads may have common code to execute, you can set breakpoints within that code that apply to specific threads only. You can set a breakpoint at the add_to_list call for thread p2.3 only.

   debug> stop -p p2.3 traverse.c@add_to_list
   EVENT [1] assigned
   debug> run
   Multiplexed thread p2.1 has given up its LWP
   debug>
   HALTED p2.2 [traverse in traverse.c]
   79:             const char      *path = ((INFO *)info)->path;

p2.1 has given up its LWP. The thread p2.2 stops by default at the location specified by the call to thr_create (traverse()). Do a ps to determine the states of the other two threads.

   debug> ps
   Program      ID      Pid Thread State     Function   Location         Command
   *traverse   p2.1    17903   1 Off LWP    _thr_debug 0xbff8c4f2      traverse /var/tmp /var/cron
    traverse   p2.2    17903   2 Stopped    traverse   traverse.c@79   traverse /var/tmp /var/cron
    traverse   p2.3    17903   3 Off LWP    _thr_start 0xbff8b5d0      traverse /var/tmp /var/cron
   debug>

This time, p2.2 is in a Stopped state and the others are off the LWP. Run p2.2.

   debug> run -p p2.2
   Multiplexed thread p2.2 has given up its LWP
   debug>
   HALTED p2.3 [traverse in traverse.c]
   79:             const char      *path = ((INFO *)info)->path;

p2.2 has given up its LWP. Enter the ps command again.

   debug> ps
   Program       ID     Pid Thread State      Function   Location        Command
   *traverse    p2.1   17903   1 Off LWP    _thr_debug 0xbff8c4f2      traverse /var/tmp /var/cron
    traverse    p2.2   17903   2 Off LWP    _thr_debug 0xbff8c4f2      traverse /var/tmp /var/cron
    traverse    p2.3   17903   3 Stopped    traverse   traverse.c@79   traverse /var/tmp /var/cron

Now it's p2.3 that's in a Stopped state. Run the thread and then enter the ps command.

   debug> run -p p2.3
   STOP EVENT TRIGGERED: traverse.c@add_to_list  in p2.3 [add_to_list in traverse.c]
   61:             mutex_lock(&mutex);

p2.3 stopped at the call to add_to_list as you had specified in the stop command.

   debug> ps
   Program      ID     Pid Thread State     Function   Location         Command
   *traverse   p2.1   17903   1 Off LWP    _thr_debug 0xbff8c4f2      traverse /var/tmp /var/cron
    traverse   p2.2   17903   2 Off LWP    _thr_debug 0xbff8c4f2      traverse /var/tmp /var/cron
    traverse   p2.3   17903   3 Stopped    add_to_lis traverse.c@61   traverse /var/tmp /var/cron

Run p2.3 and see what happens.

   debug> run -p p2.3

Nothing happens; it appears that you have reached the point where the process is hung. You will have to interrupt execution (by hitting the DELETE key) to regain control and get another prompt from the debugger. After interrupting, enter the stack command for all three threads and you should get the same stack traces as in the earlier session.

   debug>
   HALTED p2.3 [_block]
           0xbffdf556 (_block+12:)       jb        +0x1 <bffd559> [ _block ]
   debug> stack -p p2.1,p2.2,p2.3
   Stack Trace for p2.1, Program traverse
   *[0] _thr_debug_notify(0xbff77cc4, 0x3) [0xbff8d97c]
    [1] _thr_resume(0xbff77cc4, 0xbff90d30, 0, 0x1, 0x1)   [0xbff8cdb6]
    [2] _thr_swtch(presumed: 0x1, 0xbff90d30, 0)   [0xbff8c4ed]
    [3] _thr_cond_wait(presumed: 0xbff914a4, 0, 0xbff91a98        [0xbff872be]
    [4] thr_join(0, 0, 0x80477bc)  [0xbff83d17]
    [5] main(argc=3, argv=0x80477ec, 0x80477fc)    [traverse.c@162]
    [6] _start()   [0x80486f4]
   
   Stack Trace for p2.2, Program traverse
   *[0] _thr_debug_notify(0xbff72cc4, 0x3) [0xbff8d97c]
    [1] _thr_resume(0xbff72cc4, 0xbff77cc4, 0, 0x1, 0x1)   [0xbff8cdb6]
    [2] _thr_swtch(0x1, 0xbff77cc4, 0x8049c74)     [0xbff8c4ed]
    [3] mutex_lock(0x8049c6c)      [0xbff8dc02]
    [4] add_to_list(name="/var/tmp/pa268.data")    [traverse.c@61]
    [5] traverse(info=0x804a058, presumed: 0, 0)   [traverse.c@111]
    [6] _thr_start()       [0xbff8b6d2]
   

   Stack Trace for p2.3, Program traverse
   *[0] _block(0)  [0xbffd259a]
    [1] __lwp_cond_wait(0xbff72f1c, 0xbff91ad0, 0) [0xbffea5b9]
    [2] _lwp_cond_wait(presumed: 0xbff72f1c, 0xbff91ad0, 0xbff95b14)  [0xbffea63a]
    [3] _thr_disp(0xbff72cc4)      [0xbff8bfe7]
    [4] _thr_swtch(0x1, 0xbff72cc4, 0x8049c74)     [0xbff8c3ee]
    [5] mutex_lock(0x8049c6c)      [0xbff8dc02]
    [6] add_to_list(name="/var/cron/log")  [traverse.c@61]
    [7] traverse(info=0x804a068, presumed: 0, 0)   [traverse.c@111]
    [8] _thr_start()       [0xbff8b6d2]

You can then draw the same conclusion as before.

Simplifying your command input

If you plan on applying debugger commands on a specific thread, you can simplify your entry by setting the variable %thread to the thread id (e.g., set %thread=p2.3). By doing this, you do not have to specify the -p p2.3 option to the appropriate commands. For example:

   debug> stack -c2
   Stack Trace for p2.1, Program traverse
   *[0] _thr_debug_notify(0xbff77cc4, 0x3) [0xbff8d97c]
    [1] _thr_resume(0xbff77cc4, 0xbff90d30, 0, 0x1, 0x1) [0xbff8cdb6]
   debug> set %thread p2.3
   Current thread is now p2.3, program traverse
   debug> stack -c2
   Stack Trace for p2.3, Program traverse
   *[0] _block(0)  [0xbffd259a]
    [1] __lwp_cond_wait(0xbff72f1c, 0xbff91ad0, 0) [0xbffea5b9]

There are other ways you can simplify your debugging session. These are discussed in ``Technical tips''.