Profiling eBPF Code

Naveen Srinivasan wrote up how he measured the overhead of an eBPF LSM hook on openat. The post is a method, not a result: he says outright that the numbers depend on what your hook does and deliberately leaves the before/after latency figures out. What is left is a recipe that isolates a kernel hook's cost from everything else on the machine.

The harness

The benchmark is a single C file with almost nothing in it, because everything it does inside the timing loop shows up in the measurement.

for (uint64_t i = 0; i < n; i++) {
    uint64_t t0 = now_ns();
    long fd = syscall(SYS_openat, AT_FDCWD, path, O_RDONLY);   /* raw, no libc wrapper */
    uint64_t t1 = now_ns();
    if (fd >= 0) close(fd);
    d[i] = (uint32_t)(t1 - t0);
}

Four decisions carry the design. now_ns() calls clock_gettime(CLOCK_MONOTONIC, …), which resolves through the VDSO and costs no syscall, so the clock reads do not themselves enter the kernel. The open goes through syscall(SYS_openat, …) rather than libc's openat() wrapper, cutting one layer of userspace out of the path. The result array is mmaped with MAP_POPULATE, mlocked, and then written once to fault everything in, so no page fault happens mid-loop. And nothing is printed until the loop is over — the printf pass runs afterwards over the stored samples.

The same file gets reopened n times under a warm cache, which is the point: this measures the syscall and hook path, not the filesystem or the disk. The first 10% of samples are discarded as warmup, so 100,000 iterations leave 90,000 usable ones for p50/p99.

Run conditions matter as much as the code:

sudo taskset -c 3 chrt -f 99 ./bench /etc/hostname 100000 > /tmp/samples.txt

taskset -c 3 pins the process to one CPU so migrations do not add noise, and chrt -f 99 puts it at SCHED_FIFO priority 99, where it runs ahead of almost everything else until it finishes, blocks, or is interrupted. You run this once with the eBPF program detached to get a baseline, then again with it attached.

Making BPF programs show up in perf

A JIT-compiled BPF program is anonymous kernel memory by default, so a profile of it is a raw address and nothing else. Two sysctls fix that:

sudo sysctl -w net.core.bpf_jit_enable=1
sudo sysctl -w net.core.bpf_jit_kallsyms=1

With bpf_jit_kallsyms on, each loaded program gets a bpf_prog_<hash>_<name> entry in /proc/kallsyms, and perf resolves stack frames to program names. Srinivasan verifies the symbols exist before recording, with bpftool prog show filtered for lsm and a grep over /proc/kallsyms for bpf_prog_[0-9a-f]+_ narrowed to security|path|file|open. He also runs a custom kernel, so the matching perf binary is not on $PATH and gets invoked explicitly from /usr/lib/linux-tools/6.8.0-134-generic/perf — perf and kernel have to be version-matched for this to work.

The recording step:

sudo $PERF record -g --call-graph fp -e cycles:k -F 997 -o ~/perf.data \
  -- taskset -c 3 chrt -f 99 ./bench /etc/hostname 200000 > /tmp/samples.txt

-e cycles:k samples cycles in kernel mode only, which keeps the userspace benchmark loop out of the profile and leaves syscall entry, VFS, LSM, and BPF execution. --call-graph fp unwinds through frame pointers. The 997 Hz sampling rate is deliberately not a round number, so it does not lock step with anything periodic on the box. Reporting is perf report --stdio --sort comm,dso,symbol; the post also shows the same perf.data rendered as a flamegraph with Inferno.

Reading the output

The call graph from the run is what makes the case:

|–90.52%–do_dentry_open
   --89.78%–bpf_lsm_file_open
      --89.30%–0xffffffffc0288c18
         |–87.40%–bpf_prog_b06f413955402a4b_tail_call_security_check
         |  |–77.57%–bpf_prog_934361d723613c1c_enforce_access_policy
         |  |  |–57.87%–bpf_prog_a0f18f4b0b140d77_path_check_callback
         |  |  |  |–29.94%–bpf_probe_read_kernel
         |  |  |  |  |–18.88%–copy_from_kernel_nofault
         |  |  |  |   --4.99%–copy_from_kernel_nofault_allowed
         |  |  |  |–4.88%–htab_map_hash
         |  |  |   --2.29%–copy_from_kernel_nofault

Nearly all of do_dentry_open is bpf_lsm_file_open and what it tail-calls, so the hook is not a rounding error on the open path — it is the open path. The tail-call chain resolves into named programs because of the kallsyms setting, and the cost concentrates in bpf_probe_read_kernel (29.94% of the sampled kernel cycles), most of which is copy_from_kernel_nofault, the fault-safe copy that every probe read pays for. A smaller slice is htab_map_hash, the hash-map lookup. That shape points at fewer probe reads or a cache in front of the map, not at micro-tuning the verifier-visible instruction count. One frame stays unresolved (0xffffffffc0288c18), which is the trampoline between the LSM hook and the program.

This is the measurement half of everything the vault has on eBPF policy hooks. ebpf-sock-ops describes the sibling attachment point on the TCP socket lifecycle, where the same question applies to connection setup rather than file open, and bypassing-dpi-with-ebpf is a case where a program sits in that path on every outgoing connection. little-snitch-linux is the kind of tool whose eBPF layer would live exactly where bpf_lsm_file_open does, and whose known failure mode under load — in-kernel cache tables overflowing — is the sort of thing a p99 from this harness would surface before users report it.