Rewriting Every Syscall in a Linux Binary at Load Time

A deep technical post on intercepting every syscall a Linux process makes by rewriting the binary at load time. Part 1 of a 7-part series on building a minimal VM runtime for AI agent execution.

The problem

Containers share the host kernel. A typical single-process workload uses ~40 syscalls, but the kernel exposes ~450. That's 410 syscalls worth of attack surface the process doesn't need but can probe, exploit, or use in unintended combinations. For untrusted code — third-party libraries, generated code, AI agents — this is a security problem.

Why existing approaches fail

ptrace — Two context switches per syscall, 10-20µs overhead. Designed for debugging, not enforcement.

seccomp-bpf — Fast (runs in-kernel), but the BPF filter only sees register values, not memory. Can't inspect the filename being open()ed or the buffer being write()n. Actions are coarse: allow, kill, return error, or trap (which brings back ptrace overhead).

eBPF — Can observe and deny, but deliberately can't modify process state. You can deny a connect(), but can't change the destination or return a custom result. Allow/deny only.

Compiler/libc interposition — Too many paths to a syscall instruction. Go makes syscalls directly. So does musl in some paths. JIT compilers emit raw syscall. Miss one path and the process escapes your control.

The insight

Every path — compiler output, libc wrappers, JIT, hand-written assembly — converges on the same two bytes: 0F 05, the syscall opcode. Work at that level and you only have one thing to catch.

The technique

Scan the .text section of the ELF binary at load time. But you can't just search for 0F 05 — those bytes might appear as part of a larger instruction (immediate operand, displacement). Naive replacement would corrupt unrelated instructions.

Instead, walk instruction-by-instruction using an Instruction Length Decoder (ILD). The ILD doesn't fully disassemble — it just computes each instruction's length. That's enough to advance to the next instruction boundary and distinguish opcodes from operands.

When you find 0F 05 at an opcode position, replace it:

// Patch: SYSCALL (0F 05) → INT3 (CC) + NOP (90)
code[opc] = 0xCC;     // INT3
code[opc + 1] = 0x90; // NOP

Both sequences are 2 bytes. No instruction boundaries shift, no relocation needed.

On a statically-linked Python 3.12 binary (19MB, 8.7MB of executable code): 363 syscalls patched in 48ms.

The shim

The rewritten binary runs inside a KVM VM with no operating system. A small shim (a few KB of Rust) is the only thing between the process and the hardware.

Before the guest runs, the hypervisor sets up an IDT with vector 3 pointing to the shim's handler. When INT3 fires:

1. CPU pushes RIP/CS/RFLAGS, jumps to handler
2. Handler reads rax (syscall number), rdi-r9 (arguments)
3. Dispatch: check policy → deny returns -EPERM, emulate locally, or escalate to hypervisor
4. Write result to rax
5. IRETQ back to guest

The entire path runs in under 1µs for the common case. The process can't tell the difference — it gets standard syscall return values.

Self-healing for JIT

The binary rewriter runs once at load time. But JIT compilers (V8, LuaJIT, Python regex) emit fresh code containing syscall instructions that don't exist when the rewrite runs.

Solution: set the LSTAR MSR as a safety net. On x86-64, the syscall instruction transfers control to the LSTAR address. Point it to a self-healing handler:

1. JIT'd syscall executes → jumps to LSTAR handler
2. Handler records the address, patches it in place (0F 05 → CC 90)
3. Constructs interrupt frame, falls through to INT3 handler

First execution takes the slow LSTAR path. Every subsequent execution hits the patched INT3. The system self-heals — every new syscall gets caught and patched on first encounter.

What this enables

Once every syscall routes through your handler:

• open("/etc/shadow") → deny, return -ENOENT
• connect("pastebin.com:443") → deny, return -EPERM
• write(socket_fd, buffer_containing_pii) → block before a byte hits the network

The process can't detect the interception (the syscall instruction was replaced), can't bypass it (every path goes through the shim), and can't tell it's being sandboxed.

Edge cases

LLVM optimizations — A static initialized to zeros gets constant-folded away. Policy table reads must use read_volatile to prevent the optimizer from assuming the value.

Linker bloat — lld emits .dynamic, .dynsym, .gnu.hash even in no_std binaries. Explicit /DISCARD/ in the linker script keeps the shim minimal.

Related work

This is a different approach than hazmat, which uses macOS user isolation + Seatbelt + pf firewall. The binary rewriting approach is Linux-specific but gives finer-grained control — you can inspect and modify syscall arguments, not just allow/deny.

The INT3 patching, the interrupt frame, and the "why is ptrace so slow" question all come from debugger territory. Sy Brand's building-a-debugger builds the other user of these mechanisms from scratch — software breakpoints, single-stepping, DWARF unwinding — and is the place to go if you want the debugging side of the same machinery.

The series continues with how a ~40-syscall "library kernel" handles the intercepted calls.