# Why cache padding uses 128 bytes on a 64-byte cache line

Two atomics on the same cache line make the cores that own them play ping-pong: coherence protocols work per cache line, not per address, so every write to either variable dirties the line for both. The standard fix is to pad them apart, and the standard question is how far. `x86_64` cache lines are 64 bytes, and both *Rust Atomics and Locks* and Bakhvalov's *Performance Analysis and Tuning on Modern CPUs* say to use the cache line size. Yet `crossbeam-utils` and Facebook's `folly` both pad to 128. Ivan Boldyrev set out to find out whether that extra 64 bytes buys anything measurable.

The stated reason is the spatial prefetcher: since Sandy Bridge, Intel's L2 prefetcher can pull cache lines in adjacent pairs. That does not make the effective line size 128 bytes, but it does mean a write to line N can drag line N+1 into another core's cache and produce contention that a 64-byte pad doesn't prevent. The behavior is controlled by a per-core MSR variously named "Adjacent Cache Line Prefetcher Disable" or "L2 Adjacent Cache Line Prefetcher Disable", which is worth checking before benchmarking anything in this area — assuming the machine lets you read it at all.

## Getting the effect to show up

Two threads each hammering one atomic turns out not to reproduce anything: once each line settles into its owner's cache, there's no further MESI traffic to observe. What worked was a vector of `SIZE` records, each holding an `add` and a `sub` atomic separated by a `CachePadded<T>` wrapper aligned to either 64 or 128 bytes, with one thread walking the vector incrementing every `add` and the other walking it incrementing every `sub`, `AcqRel` ordering, a billion operations total.

Two details make the numbers trustworthy. The inner loop is kept dead simple — the total operation count is `SIZE * floor(N/SIZE)` rather than exactly `N`, which doesn't matter at `N = 1e9` and small `SIZE`. And both threads are pinned, to core 0 and core **2**: core 1 is usually the sibling hyperthread of core 0, sharing caches, which would erase the effect being measured.

## Results

On an AWS `c5d.4xlarge` (Xeon Platinum 8124M, Skylake), the difference is real but modest in the mean and large in the variance:

| SIZE | align 64 | align 128 |
|---|---|---|
| 1 | 7.426 s ± 0.001 | 7.427 s ± 0.001 |
| 2 | 5.464 s ± 0.002 | 5.349 s ± 0.001 |
| 3 | 5.475 s ± 0.093 | 5.249 s ± 0.002 |
| 5 | 5.696 s ± 0.247 | 5.363 s ± 0.002 |
| 7 | 5.684 s ± 0.201 | 5.137 s ± 0.001 |
| 16 | 6.943 s ± 0.281 | 6.646 s ± 0.001 |

The mean gap is a few percent. The standard deviation gap is two orders of magnitude — 0.2 s versus 0.001 s at `SIZE=7`. That is the more useful finding: prefetcher-induced coherence traffic arrives in unpredictable order and takes unpredictable time, so the 64-byte version has a long tail the 128-byte version simply doesn't have. For latency-sensitive work (the post names HFT) the tail is what you were padding for in the first place.

`SIZE=1` shows no difference at all, consistent with the "two threads, two atomics" case that failed to reproduce anything.

## Where it doesn't reproduce

A Digital Ocean VPS produced nothing reliable in either configuration, with large deviations throughout — the author suspects the adjacent-line prefetcher is simply disabled there.

An AWS `c6i.4xlarge` (Xeon Platinum 8375C, Ice Lake) shows essentially no difference between 64 and 128, and small deviations in both. Something in the newer prefetcher defeats this particular benchmark; the post doesn't claim to know what.

The surprise is Apple Silicon. The M1 genuinely has 128-byte cache lines, so it should show the effect more sharply than any Intel part — and it doesn't. `SIZE=3` runs 2.391 s at align 64 and 2.425 s at align 128, i.e. slightly *worse* with the larger padding. The author couldn't explain it and links an independent confirmation of the same non-result.

So the honest summary is: the 128-byte convention is defensible on Skylake-era Intel, mostly for tail latency rather than throughput, and unverified everywhere else the author could test. Which is a decent argument for keeping it (the cost is memory, the benefit is occasionally real) and a poor argument for treating it as a law.

The post ends with a disclaimer that it was written by a human, with AI used only for brainstorming and proofreading.

## Related

Another instance of performance living below the source, like [[compiler-codegen-luck]] and [[huge-pages]] — nothing in the Rust here changes, only where the bytes land. [[golang-green-tea-gc]] runs into the same measurement problem from the other side: there the effect is invisible in L3 counters and only resolves once L1 MPKI is available, which takes a machine that exposes the PMU counters for L1 events rather than the VPS this benchmark could not get a reading out of. The atomics being contended are the same primitives underneath a [[spinlock]] or the reclamation counters in [[epoch-based-reclamation]], both of which pad for exactly this reason. The coherence protocol behind that ping-pong, and when other cores may see a write, is [[write-atomicity]]. For hot-path tuning at the instruction level instead of the memory level, see [[go-bounds-checks-unsafe]]. Padding also works in reverse: removing small fields can shrink a struct by more than their size, as in [[big-pineapple-dns-cache-layout]].