Linux uprobes [1] allows to dynamically instrument user applications, injecting programmable breakpoints at arbitrary instructions. It is remarkably handy in ad-hoc profiling, debugging and tracing, especially when combined with eBPF [2]. My last post on tracing was aimed to introduce the technology. This time we will have a look at an application traced with uprobes in gdb: to see the “traces” of a tracer.

eBPF and bcc: A Quick Introduction

First, let’s get acquainted with a bleeding edge of Linux tracing tools - bcc or BPF Compiler Collection [3]. bcc is a toolkit that allows convenient creation of fast, lightweight and specialized tracing tools. It is based on an eBPF (Extended Berkley Packet Filter), introduced in Linux Kernel 3.5.

In short, eBPF is a tiny assembly language for a virtual machine that can be executed inside Linux Kernel. eBPF instructions are relatively close to their x86 counterparts which makes it possible to JIT-compile them into native code. eBPF was originally conceived to power tools like tcpdump and implement programmable network packed dispatch and tracing. Since Linux 4.1, eBPF programs can be attached to kprobes and later - uprobes, enabling efficient programmable tracing.

eBPF Internals
Linux eBPF Internals. Source: http://www.brendangregg.com/ebpf.html Copyright 2017 Brendan Gregg

One of the biggest benefits of eBPF programs is their safety. Due to quite strong restrictions on applied by the Kernel, it is possible to validate them before execution ensuring lack of side effects or other sorts of misconduct, intentional or otherwise. This makes eBPF safe to use in production environments and gives more confidence during everyday debugging use.

As mentioned earlier, since Linux 4.1 eBPF programs can be attached to the kprobes and uprobes. That means we can attach programmable breakpoints to (almost) any code running in kernel- or user-space, while still writing probe handler in safe language which would gather and aggregate us information in kernel space, providing only necessary output to the user space. Cool? Yes. Now, the downside of this is - eBPF programs are quite hard to write. This is where bcc comes in: combining power of LLVM toolchain, eBPF and Python, it simplifies creation of eBPF-based tools. In addition to that, bcc provides huge collection of ready-made tools and examples, making it possible to use all the power of eBPF without even knowing it.

Setting up bcc

bcc is an actively developed project with wonderful documentation, so please go through Installing BCC on the instructions for your system. This page also lists kernel options that you should have enabled on your system.

If you are running Ubuntu 16.04, first consider installing LTS Enablement Stack. Following command will switch you to the 4.10 kernel series, which ensures that all bcc functionality will work:

sudo apt-get install --install-recommends linux-generic-hwe-16.04 xserver-xorg-hwe-16.04

Next, feel free to pick your favorite way to install bcc. I use the nightly release repository and am happy so far:

echo "deb [trusted=yes] https://repo.iovisor.org/apt/xenial xenial-nightly main" | sudo tee /etc/apt/sources.list.d/iovisor.list
sudo apt-get update
sudo apt-get install bcc-tools libbcc-examples

That’s it! After installation is done, you will find the tools in /usr/share/bcc/tools.

Tracing bash:readline

Let’s take a look at the quite famous example of using bcc/trace to sniff all bash commands executed in the interactive mode on the running system:

$ trace 'r:bash:readline "%s" retval'
PID     TID     COMM            FUNC             -
19409   19409   bash            readline         hello
19409   19409   bash            readline         world
...

I won’t go here into details of the tool syntax as you can easily look it up yourself [3]. However, a quick recap of what happens internally might be necessary:

  • First, we assume we know that bash implementation has a function readline that returns a null-terminated char* string that contains a command that was just read from the user input.
  • We attach a return uprobe to the function, instructing to record the return value and output it as a string.
  • We wait until the readline function gets executed, i.e. user presses “enter”.
  • uprobes intercepts execution and runs the probe handler providing us with a desired output.
Instruction Probe Execution

Now, let’s fire up gdb and have a closer look.

Does the Tracer Leave any Traces?

First, let’s attach to our bash command with a debugger and get the disassembly of the readline function.

$ gdb --pid <target bash PID>
(gdb) disassemble /r readline
Dump of assembler code for function readline:
   0x000000000049a520 <+0>:  83 3d 81 5e 26 00 ff     cmpl   $0xffffffff,0x265e81(%rip)
   0x000000000049a527 <+7>:  53                       push   %rbx
   0x000000000049a528 <+8>:  74 6e                    je     0x49a598 <readline+120>
   0x000000000049a52a <+10>: e8 21 ee ff ff           callq  0x499350 <rl_set_prompt>
   0x000000000049a52f <+15>: e8 3c fd ff ff           callq  0x49a270 <rl_initialize>
...

Now, if we execute $ trace 'r:bash:readline "%s" retval' in a different terminal, continue bash execution (gdb) c, interrupt again (^C) and repeat the disassembly we will get a very different picture:

(gdb) disassemble /r readline
Dump of assembler code for function readline:
   0x000000000049a520 <+0>:  cc                       int3
   0x000000000049a521 <+1>:  3d 81 5e 26 00           cmp    $0x265e81,%eax
   0x000000000049a526 <+6>:  ff 53 74                 callq  *0x74(%rbx)
   0x000000000049a529 <+9>:  6e                       outsb  %ds:(%rsi),(%dx)
   0x000000000049a52a <+10>: e8 21 ee ff ff           callq  0x499350 <rl_set_prompt>
   0x000000000049a52f <+15>: e8 3c fd ff ff           callq  0x49a270 <rl_initialize>
...

Here is a diff of two blocks for clarity:

-0x000000000049a520 <+0>:  83 3d 81 5e 26 00 ff     cmpl   $0xffffffff,0x265e81(%rip)
-0x000000000049a527 <+7>:  53                       push   %rbx
-0x000000000049a528 <+8>:  74 6e                    je     0x49a598 <readline+120>
+0x000000000049a520 <+0>:  cc                       int3
+0x000000000049a521 <+1>:  3d 81 5e 26 00           cmp    $0x265e81,%eax
+0x000000000049a526 <+6>:  ff 53 74                 callq  *0x74(%rbx)
+0x000000000049a529 <+9>:  6e                       outsb  %ds:(%rsi),(%dx)

We can clearly see two things:

  • uprobes injecting int3 (cc) instruction at the first byte of the readline function (at 0x49a520).
  • gdb being completely confused as per what follows the instruction all the way until address 0x49a52a - a call to rl_set_prompt. My guess is this happens because gdb has no clue of what we are doing - the breakpoint does not belong to it, and as far as it is concerned, it is as valid an instruction as any other.

During execution uprobes patches everything up and ensures that instructions are executed as they are in the original binary, with the exception of a breakpoint of course.

Return Probes: It’s All About The Stack

Now, as mentioned in the other post, return probes have to do some extra work to break into function returns, which includes replacing return address on the stack. Let’s try to inspect this behavior with gdb too.

First, we execute bash without tracing enabled and insert a breakpoint somewhere in the middle of the function (far enough from the entry address as not to interfere with uprobes):

(gdb) disassemble readline
Dump of assembler code for function readline:
...
   0x000000000049a54d <+45>:	callq  0x4993d0 <readline_internal_setup>
...

(gdb) br *0x000000000049a54d

Now, when we hit the breakpoint, let’s examine the current stack frame:

Breakpoint 1, 0x000000000049a54d in readline ()
(gdb) info frame
Stack level 0, frame at 0x7ffccd6fca30:
 rip = 0x49a54d in readline; saved rip = 0x42176e
 called by frame at 0x7ffccd6fca50
 Arglist at 0x7ffccd6fca18, args: 
 Locals at 0x7ffccd6fca18, Previous frame's sp is 0x7ffccd6fca30
 Saved registers:
  rbx at 0x7ffccd6fca20, rip at 0x7ffccd6fca28

After running $ trace 'r:bash:readline "%s" retval', next time we hit a breakpoint we will notice a few changes:

  • First, the breakpoint we inserted has transformed into a SIGTRAP - a hint of gdb being slightly confused.
  • Second, breakpoint address is different by 1 byte - 0x49a54e. Let’s examine the stack here:
Program received signal SIGTRAP, Trace/breakpoint trap.
0x000000000049a54e in readline ()
(gdb) info frame
Stack level 0, frame at 0x7ffccd6fca30:
 rip = 0x49a54e in readline; saved rip = 0x7fffffffe000
 called by frame at 0x7ffccd6fca38
 Arglist at 0x7ffccd6fca18, args: 
 Locals at 0x7ffccd6fca18, Previous frame's sp is 0x7ffccd6fca30
 Saved registers:
  rbx at 0x7ffccd6fca20, rip at 0x7ffccd6fca28

Again here is the diff of last two blocks:

 Stack level 0, frame at 0x7ffccd6fca30:
- rip = 0x49a54d in readline; saved rip = 0x42176e
- called by frame at 0x7ffccd6fca50
+ rip = 0x49a54e in readline; saved rip = 0x7fffffffe000
+ called by frame at 0x7ffccd6fca38

Here is the expected bit: return address is replaced with 0x7fffffffe000 which I would expect to be uprobes trampoline. Inspecting the memory at this address, however, fails:

(gdb) x 0x7fffffffe000
0x7fffffffe000:	Cannot access memory at address 0x7fffffffe000

Moreover, continuing execution leads to a grinding halt:

(gdb) c
Continuing.

Program received signal SIGILL, Illegal instruction.
0x000000000049a550 in readline ()

This does not make much sense, so I suspect gdb, being confused and unaware of the presence of another debugger which is patching things here and there, goes a little crazy during debugging and corrupts the text segment. Thus a SIGTRAP instead of a recognized breakpoint. Thus the unreachable frame pointer. Thus the ultimate SIGILL.

Summarizing

Okay, that was fun wasn’t it? It was for me for sure. What did we learn?

  • First, one can inspect uprobes in action if they really want to. At least, see the interrupt instruction.
  • Second, one should not use uprobes and gdb at the same time (or maybe at least in the same routine).

Knowing more details about PTrace and gdb would help to understand what is going on, but I would leave it out for now, surely somebody wrote a post about it :wink:

This is it for now. Happy tracing, Happy New Year, and stay tuned for the next post!

References

Leave a Comment