Part 7 · 1 chapters · ~8 min

eBPF and bpftrace

What eBPF is, the verifier and JIT, attach points (kprobes, tracepoints, uprobes, USDT, perf events), maps and histograms, bpftrace one-liners, the BCC tool collection, CO-RE and libbpf, overhead, and eBPF beyond tracing (Cilium networking, security with Tetragon and Falco).

8

One-liners that answer real questions

code
# which processes are calling which syscalls most?
bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'
# read() size distribution for one process
bpftrace -e 'tracepoint:syscalls:sys_exit_read /comm == "node"/ { @bytes = hist(args.ret); }'
# new processes with arguments (short-lived processes that top never shows)
bpftrace -e 'tracepoint:syscalls:sys_enter_execve { printf("%s -> %s\n", comm, str(args.filename)); }'
# block I/O latency histogram
bpftrace -e 'kprobe:blk_account_io_start { @s[arg0] = nsecs; } kprobe:blk_account_io_done /@s[arg0]/ { @us = hist((nsecs - @s[arg0]) / 1000); delete(@s[arg0]); }'
# Postgres query latency via uprobes on the binary
bpftrace -e 'uprobe:/usr/lib/postgresql/16/bin/postgres:exec_simple_query { @start[tid] = nsecs; } uretprobe:/usr/lib/postgresql/16/bin/postgres:exec_simple_query /@start[tid]/ { @ms = hist((nsecs - @start[tid]) / 1e6); delete(@start[tid]); }'

Kernel function names used by kprobes change between versions (the block probe above differs on some kernels), which is why stable tracepoints are preferred where they exist. Beyond tracing: Cilium implements Kubernetes networking and policy in eBPF; Tetragon and Falco use it for runtime security.

HOW eBPF WORKS
small verified programs attached to kernel events
bpftrace / BCC / libbpfyour programverifierproves it is safeJITnative codeattach pointkprobe, tracepoint, uprobe, perf eventmapscounts, histogramsuser spacereads and prints
swipe the figure sideways, or tap expand for full screen
1/4
write and load
Tools compile a small program to eBPF bytecode and load it into the kernel.
bytecode into the kernelbpftrace, BCC, libbpf