Linux uprobes  allows to dynamically instrument user applications, injecting programmable breakpoints at arbitrary instructions. It is remarkably handy in ad-hoc profiling, debugging and tracing, especially when combined with eBPF . My last post on tracing was aimed to introduce the technology. This time we will have a look at an application traced with uprobes in gdb: to see the “traces” of a tracer.
eBPF and bcc: A Quick Introduction
First, let’s get acquainted with a bleeding edge of Linux tracing tools - bcc or BPF Compiler Collection . bcc is a toolkit that allows convenient creation of fast, lightweight and specialized tracing tools. It is based on an eBPF (Extended Berkley Packet Filter), introduced in Linux Kernel 3.5.
In short, eBPF is a tiny assembly language for a virtual machine that can be executed inside Linux Kernel. eBPF instructions are relatively close to their x86 counterparts which makes it possible to JIT-compile them into native code. eBPF was originally conceived to power tools like tcpdump and implement programmable network packed dispatch and tracing. Since Linux 4.1, eBPF programs can be attached to kprobes and later - uprobes, enabling efficient programmable tracing.
One of the biggest benefits of eBPF programs is their safety. Due to quite strong restrictions on applied by the Kernel, it is possible to validate them before execution ensuring lack of side effects or other sorts of misconduct, intentional or otherwise. This makes eBPF safe to use in production environments and gives more confidence during everyday debugging use.
As mentioned earlier, since Linux 4.1 eBPF programs can be attached to the kprobes and uprobes. That means we can attach programmable breakpoints to (almost) any code running in kernel- or user-space, while still writing probe handler in safe language which would gather and aggregate us information in kernel space, providing only necessary output to the user space. Cool? Yes. Now, the downside of this is - eBPF programs are quite hard to write. This is where bcc comes in: combining power of LLVM toolchain, eBPF and Python, it simplifies creation of eBPF-based tools. In addition to that, bcc provides huge collection of ready-made tools and examples, making it possible to use all the power of eBPF without even knowing it.
Setting up bcc
bcc is an actively developed project with wonderful documentation, so please go through Installing BCC on the instructions for your system. This page also lists kernel options that you should have enabled on your system.
If you are running Ubuntu 16.04, first consider installing LTS Enablement Stack. Following command will switch you to the 4.10 kernel series, which ensures that all bcc functionality will work:
sudo apt-get install --install-recommends linux-generic-hwe-16.04 xserver-xorg-hwe-16.04
Next, feel free to pick your favorite way to install bcc. I use the nightly release repository and am happy so far:
echo "deb [trusted=yes] https://repo.iovisor.org/apt/xenial xenial-nightly main" | sudo tee /etc/apt/sources.list.d/iovisor.list sudo apt-get update sudo apt-get install bcc-tools libbcc-examples
That’s it! After installation is done, you will find the tools in
Let’s take a look at the quite famous example of using bcc/trace to sniff all bash commands executed in the interactive mode on the running system:
$ trace 'r:bash:readline "%s" retval' PID TID COMM FUNC - 19409 19409 bash readline hello 19409 19409 bash readline world ...
I won’t go here into details of the tool syntax as you can easily look it up yourself . However, a quick recap of what happens internally might be necessary:
- First, we assume we know that bash implementation has a function
readlinethat returns a null-terminated
char*string that contains a command that was just read from the user input.
- We attach a return uprobe to the function, instructing to record the return value and output it as a string.
- We wait until the
readlinefunction gets executed, i.e. user presses “enter”.
- uprobes intercepts execution and runs the probe handler providing us with a desired output.
Now, let’s fire up gdb and have a closer look.
Does the Tracer Leave any Traces?
First, let’s attach to our bash command with a debugger and get the disassembly of the
$ gdb --pid <target bash PID> (gdb) disassemble /r readline Dump of assembler code for function readline: 0x000000000049a520 <+0>: 83 3d 81 5e 26 00 ff cmpl $0xffffffff,0x265e81(%rip) 0x000000000049a527 <+7>: 53 push %rbx 0x000000000049a528 <+8>: 74 6e je 0x49a598 <readline+120> 0x000000000049a52a <+10>: e8 21 ee ff ff callq 0x499350 <rl_set_prompt> 0x000000000049a52f <+15>: e8 3c fd ff ff callq 0x49a270 <rl_initialize> ...
Now, if we execute
$ trace 'r:bash:readline "%s" retval' in a different terminal, continue bash execution
(gdb) c, interrupt again (^C) and repeat the disassembly we will get a very different picture:
(gdb) disassemble /r readline Dump of assembler code for function readline: 0x000000000049a520 <+0>: cc int3 0x000000000049a521 <+1>: 3d 81 5e 26 00 cmp $0x265e81,%eax 0x000000000049a526 <+6>: ff 53 74 callq *0x74(%rbx) 0x000000000049a529 <+9>: 6e outsb %ds:(%rsi),(%dx) 0x000000000049a52a <+10>: e8 21 ee ff ff callq 0x499350 <rl_set_prompt> 0x000000000049a52f <+15>: e8 3c fd ff ff callq 0x49a270 <rl_initialize> ...
Here is a diff of two blocks for clarity:
-0x000000000049a520 <+0>: 83 3d 81 5e 26 00 ff cmpl $0xffffffff,0x265e81(%rip) -0x000000000049a527 <+7>: 53 push %rbx -0x000000000049a528 <+8>: 74 6e je 0x49a598 <readline+120> +0x000000000049a520 <+0>: cc int3 +0x000000000049a521 <+1>: 3d 81 5e 26 00 cmp $0x265e81,%eax +0x000000000049a526 <+6>: ff 53 74 callq *0x74(%rbx) +0x000000000049a529 <+9>: 6e outsb %ds:(%rsi),(%dx)
We can clearly see two things:
cc) instruction at the first byte of the
gdbbeing completely confused as per what follows the instruction all the way until address
0x49a52a- a call to
rl_set_prompt. My guess is this happens because
gdbhas no clue of what we are doing - the breakpoint does not belong to it, and as far as it is concerned, it is as valid an instruction as any other.
During execution uprobes patches everything up and ensures that instructions are executed as they are in the original binary, with the exception of a breakpoint of course.
Return Probes: It’s All About The Stack
Now, as mentioned in the other post, return probes have to do some extra work to break into function returns, which includes replacing return address on the stack. Let’s try to inspect this behavior with gdb too.
First, we execute
bash without tracing enabled and insert a breakpoint somewhere in the middle of the function (far enough from the entry address as not to interfere with uprobes):
(gdb) disassemble readline Dump of assembler code for function readline: ... 0x000000000049a54d <+45>: callq 0x4993d0 <readline_internal_setup> ... (gdb) br *0x000000000049a54d
Now, when we hit the breakpoint, let’s examine the current stack frame:
Breakpoint 1, 0x000000000049a54d in readline () (gdb) info frame Stack level 0, frame at 0x7ffccd6fca30: rip = 0x49a54d in readline; saved rip = 0x42176e called by frame at 0x7ffccd6fca50 Arglist at 0x7ffccd6fca18, args: Locals at 0x7ffccd6fca18, Previous frame's sp is 0x7ffccd6fca30 Saved registers: rbx at 0x7ffccd6fca20, rip at 0x7ffccd6fca28
$ trace 'r:bash:readline "%s" retval', next time we hit a breakpoint we will notice a few changes:
- First, the breakpoint we inserted has transformed into a
SIGTRAP- a hint of gdb being slightly confused.
- Second, breakpoint address is different by 1 byte -
0x49a54e. Let’s examine the stack here:
Program received signal SIGTRAP, Trace/breakpoint trap. 0x000000000049a54e in readline () (gdb) info frame Stack level 0, frame at 0x7ffccd6fca30: rip = 0x49a54e in readline; saved rip = 0x7fffffffe000 called by frame at 0x7ffccd6fca38 Arglist at 0x7ffccd6fca18, args: Locals at 0x7ffccd6fca18, Previous frame's sp is 0x7ffccd6fca30 Saved registers: rbx at 0x7ffccd6fca20, rip at 0x7ffccd6fca28
Again here is the diff of last two blocks:
Stack level 0, frame at 0x7ffccd6fca30: - rip = 0x49a54d in readline; saved rip = 0x42176e - called by frame at 0x7ffccd6fca50 + rip = 0x49a54e in readline; saved rip = 0x7fffffffe000 + called by frame at 0x7ffccd6fca38
Here is the expected bit: return address is replaced with
0x7fffffffe000 which I would expect to be uprobes trampoline. Inspecting the memory at this address, however, fails:
(gdb) x 0x7fffffffe000 0x7fffffffe000: Cannot access memory at address 0x7fffffffe000
Moreover, continuing execution leads to a grinding halt:
(gdb) c Continuing. Program received signal SIGILL, Illegal instruction. 0x000000000049a550 in readline ()
This does not make much sense, so I suspect gdb, being confused and unaware of the presence of another debugger which is patching things here and there, goes a little crazy during debugging and corrupts the text segment. Thus a
SIGTRAP instead of a recognized breakpoint. Thus the unreachable frame pointer. Thus the ultimate
Okay, that was fun wasn’t it? It was for me for sure. What did we learn?
- First, one can inspect uprobes in action if they really want to. At least, see the interrupt instruction.
- Second, one should not use uprobes and gdb at the same time (or maybe at least in the same routine).
Knowing more details about PTrace and gdb would help to understand what is going on, but I would leave it out for now, surely somebody wrote a post about it
This is it for now. Happy tracing, Happy New Year, and stay tuned for the next post!