# Profiling eBPF Code

Naveen Srinivasan wrote up how he measured the overhead of an eBPF LSM hook on `openat`. The post is a method, not a result: he says outright that the numbers depend on what your hook does and deliberately leaves the before/after latency figures out. What is left is a recipe that isolates a kernel hook's cost from everything else on the machine.

## The harness

The benchmark is a single C file with almost nothing in it, because everything it does inside the timing loop shows up in the measurement.

```c
for (uint64_t i = 0; i < n; i++) {
    uint64_t t0 = now_ns();
    long fd = syscall(SYS_openat, AT_FDCWD, path, O_RDONLY);   /* raw, no libc wrapper */
    uint64_t t1 = now_ns();
    if (fd >= 0) close(fd);
    d[i] = (uint32_t)(t1 - t0);
}
```

Four decisions carry the design. `now_ns()` calls `clock_gettime(CLOCK_MONOTONIC, …)`, which resolves through the VDSO and costs no syscall, so the clock reads do not themselves enter the kernel. The open goes through `syscall(SYS_openat, …)` rather than libc's `openat()` wrapper, cutting one layer of userspace out of the path. The result array is `mmap`ed with `MAP_POPULATE`, `mlock`ed, and then written once to fault everything in, so no page fault happens mid-loop. And nothing is printed until the loop is over — the `printf` pass runs afterwards over the stored samples.

The same file gets reopened `n` times under a warm cache, which is the point: this measures the syscall and hook path, not the filesystem or the disk. The first 10% of samples are discarded as warmup, so 100,000 iterations leave 90,000 usable ones for p50/p99.

Run conditions matter as much as the code:

```
sudo taskset -c 3 chrt -f 99 ./bench /etc/hostname 100000 > /tmp/samples.txt
```

`taskset -c 3` pins the process to one CPU so migrations do not add noise, and `chrt -f 99` puts it at SCHED_FIFO priority 99, where it runs ahead of almost everything else until it finishes, blocks, or is interrupted. You run this once with the eBPF program detached to get a baseline, then again with it attached.

## Making BPF programs show up in perf

A JIT-compiled BPF program is anonymous kernel memory by default, so a profile of it is a raw address and nothing else. Two sysctls fix that:

```
sudo sysctl -w net.core.bpf_jit_enable=1
sudo sysctl -w net.core.bpf_jit_kallsyms=1
```

With `bpf_jit_kallsyms` on, each loaded program gets a `bpf_prog_<hash>_<name>` entry in `/proc/kallsyms`, and perf resolves stack frames to program names. Srinivasan verifies the symbols exist before recording, with `bpftool prog show` filtered for `lsm` and a grep over `/proc/kallsyms` for `bpf_prog_[0-9a-f]+_` narrowed to `security|path|file|open`. He also runs a custom kernel, so the matching perf binary is not on `$PATH` and gets invoked explicitly from `/usr/lib/linux-tools/6.8.0-134-generic/perf` — perf and kernel have to be version-matched for this to work.

The recording step:

```
sudo $PERF record -g --call-graph fp -e cycles:k -F 997 -o ~/perf.data \
  -- taskset -c 3 chrt -f 99 ./bench /etc/hostname 200000 > /tmp/samples.txt
```

`-e cycles:k` samples cycles in kernel mode only, which keeps the userspace benchmark loop out of the profile and leaves syscall entry, VFS, LSM, and BPF execution. `--call-graph fp` unwinds through frame pointers. The 997 Hz sampling rate is deliberately not a round number, so it does not lock step with anything periodic on the box. Reporting is `perf report --stdio --sort comm,dso,symbol`; the post also shows the same `perf.data` rendered as a flamegraph with Inferno.

## Reading the output

The call graph from the run is what makes the case:

```
|–90.52%–do_dentry_open
   --89.78%–bpf_lsm_file_open
      --89.30%–0xffffffffc0288c18
         |–87.40%–bpf_prog_b06f413955402a4b_tail_call_security_check
         |  |–77.57%–bpf_prog_934361d723613c1c_enforce_access_policy
         |  |  |–57.87%–bpf_prog_a0f18f4b0b140d77_path_check_callback
         |  |  |  |–29.94%–bpf_probe_read_kernel
         |  |  |  |  |–18.88%–copy_from_kernel_nofault
         |  |  |  |   --4.99%–copy_from_kernel_nofault_allowed
         |  |  |  |–4.88%–htab_map_hash
         |  |  |   --2.29%–copy_from_kernel_nofault
```

Nearly all of `do_dentry_open` is `bpf_lsm_file_open` and what it tail-calls, so the hook is not a rounding error on the open path — it is the open path. The tail-call chain resolves into named programs because of the kallsyms setting, and the cost concentrates in `bpf_probe_read_kernel` (29.94% of the sampled kernel cycles), most of which is `copy_from_kernel_nofault`, the fault-safe copy that every probe read pays for. A smaller slice is `htab_map_hash`, the hash-map lookup. That shape points at fewer probe reads or a cache in front of the map, not at micro-tuning the verifier-visible instruction count. One frame stays unresolved (`0xffffffffc0288c18`), which is the trampoline between the LSM hook and the program.

This is the measurement half of everything the vault has on eBPF policy hooks. [[ebpf-sock-ops]] describes the sibling attachment point on the TCP socket lifecycle, where the same question applies to connection setup rather than file open, and [[bypassing-dpi-with-ebpf]] is a case where a program sits in that path on every outgoing connection. [[little-snitch-linux]] is the kind of tool whose eBPF layer would live exactly where `bpf_lsm_file_open` does, and whose known failure mode under load — in-kernel cache tables overflowing — is the sort of thing a p99 from this harness would surface before users report it.