Shrinking 1.1.1.1's DNS cache entries

Big Pineapple is the Rust service behind Cloudflare's 1.1.1.1 resolver, Gateway DNS, DNS Firewall, AS112 and other DNS products. It holds over 250 billion cache entries at any moment, so one wasted byte per entry is more than 250 GB across the fleet. A Cloudflare post from August 2026 describes five successive changes to how an entry is laid out in memory cloudflare-dns-cache-memory. In Cloudflare's benchmark they took an entry from 953 to 420 bytes. In production they released roughly 100 TB of working-set memory, which the post equates to the RAM in 130 Gen 13 servers, and the cache got faster rather than slower.

What an entry looks like

A key is the query name, record type, an authenticated flag and a tag. A value holds timestamps, TTL, a hit counter and the parsed response split into sections:

pub struct CacheEntry {
    timestamp: UnixTimeStamp,
    pub inception: Instant,
    pub ttl: Ttl,
    pub hits: u32,
    pub answers: Vec<Record>,
    pub authority: Vec<Record>,
    pub additional: Vec<Record>,
    pub errors: Vec<ExtendedError>,
    // ...
}

Two properties of DNS make this layout wasteful. An entry is never modified after insertion, and most responses are small A or AAAA answers whose records are owned by the name that was asked about. Where the cache uses EDNS Client Subnet, the same query is cached once per client network, which multiplies both the count and the per-entry size, so ECS-heavy data centers benefit most.

The benchmark fills the cache with random entries shaped like production traffic: 56% A, 25% AAAA, 19% TXT, one to four records per entry, and TXT data of 64 to 224 bytes standing in for every variable-length type. A wrapper around Rust's System allocator counts allocations and bytes per entry, and insert throughput and lookup latency are measured over the full cache path.

1. Drop the capacity field

A Vec<T> is a pointer, a length and a capacity, and it usually has spare heap slots reserved for growth. A frozen entry needs neither. Box<[T]> is just pointer and length, sized exactly, and Box<str> does the same for String. An entry had eight Vec and String fields, so this saves 64 bytes of inline size per entry plus all the over-allocated heap tails. Across the fleet the post puts this at over 15 TB.

2. One record list with offsets

Instead of three separate lists for the answer, authority and additional sections, the entry keeps one list and two u16 offsets marking where the second and third sections start. A DNS section's record count fits in 16 bits. Two pointer-plus-length pairs (32 bytes) become two 2-byte offsets, saving 28 bytes per entry.

In the same step several bool fields were packed into one bitflags value. The post points out that such savings are not just the bytes removed: Rust rounds a struct's size up to its alignment and pads between fields, so removing a small field can remove padding with it, and the struct shrinks by more than the booleans' size. false-sharing-alignment-128 covers the same alignment rules from the other side, where padding is added on purpose.

3. Don't store the owner when it is the query name

Every record has an owner name. On the wire, RFC 1035 name compression replaces repeats with a 2-byte pointer, but chasing those pointers on every lookup is too slow for the hot path, so the cache stored a full owner name with each record. Most records' owner is the queried name itself, and the cache key already has it. The record now carries owner: Option<Box<Name>>: None means "same as the query", filled back in from the key when the response is built, and Some points to a heap-allocated name for cases like the A records behind a CNAME.

The cost is that a record is no longer self-contained; it can only be interpreted next to its key. Since the key is always present during a lookup, Cloudflare judged that acceptable, and the common case now has no owner allocation at all.

4. Box the large enum variants

Record data was a Rust enum with one variant per record type. An enum is as large as its largest variant plus a tag. The largest here was NAPTR at 136 bytes (three variable-length strings, a domain name, two integers), which made every RecordData 144 bytes. An A record needs 4 bytes and an AAAA 16, and those two are over 80% of traffic.

pub enum RecordData {
    A(Ipv4Addr),
    Aaaa(Ipv6Addr),
    Txt(Box<Txt>),
    Naptr(Box<Naptr>),
    Svcb(Box<Svcb>),
    // ...
}

Keeping the small, common variants inline and boxing the rest shrinks the enum to 24 bytes and saves 120 bytes on every A and AAAA record. Boxed types such as TXT and CNAME also gain, because their heap allocation is sized to their data. NAPTR gets slightly worse, paying for a pointer and an allocation, which is acceptable because it is rare.

Boxing brought two costs. The allocator rounds each box to a size class: jemalloc puts a 32-byte TXT into a 32-byte bin with no waste, but a 40-byte MX goes into a 48-byte bin. And each boxed value lives in its own heap region, so reading an entry's records means following pointers that can land on cold cache lines anywhere in the heap, which with millions of entries they do.

5. Store record data as wire bytes

The obvious endpoint, caching the whole wire-format response and patching the message ID per client, was rejected. DNSSEC records are only returned to clients that set the DO flag, so a full message would have to be cached twice or filtered after the fact, and parsing a whole message on each lookup costs more than reading already-parsed fields.

The chosen middle ground keeps the entry's metadata as structured fields but replaces the list of parsed records with a single Box<[u8]>: each record's raw bytes preceded by a 2-byte length. This removes the enum and every per-record box from step 4, and puts all record data in one contiguous allocation. Records can no longer be indexed at random and have to be walked in order, which complicates round-robin rotation of A/AAAA answers, but an entry holds few records, so the post calls the cost negligible.

Response building gets cheaper too. A, AAAA, TXT and all DNSSEC types are copied straight from the buffer into the outgoing message instead of being re-serialized field by field. Only types containing domain names (CNAME, NS, MX, SOA) are still parsed, so the resolver can compress names in the response. With the better locality, this step cut lookup latency by 5% in the benchmark.

Inserts use a reusable scratch buffer that persists across insertions and so rarely reallocates. Records are serialized into it, and once the final size is known, one exactly sized Box<[u8]> is allocated and the bytes are copied in. That replaces one allocation per boxed record with one per entry, and avoids shrinking a Vec<u8>, where the allocator may not be able to reclaim the freed tail. This change alone raised insert throughput 13%. It is a small version of the idea behind arena-allocation: build in a scratch region, then commit a single block.

Results

Benchmark, all five changes together:

The rollout ran from May 18 to July 6, 2026, one or more changes per release, so resident memory fell in steps. Each restart began with an empty cache that refilled, so the post reads the stable plateaus rather than the dips. Per-instance resident memory at p99 fell from 9.3 GB to 5.3 GB (43%), and at p90 from 6.5 GB to 3.8 GB (42%). The production reduction is smaller than the 56% per-entry figure because resident memory includes everything else in the process, and the post is explicit that its synthetic inputs approximate production rather than reproduce it. golang-maps-swiss-tables shows how wide that gap between a microbenchmark and a fleet can get; here it stayed modest.

Cloudflare plans to spend the freed memory on more cache capacity at the same footprint, for higher hit rates and fewer upstream queries.

The pattern

Each step trades a general-purpose representation for one that fits data which is written once and read many times. Growable containers become fixed slices, per-section lists become offsets into one list, a field that repeats information already in the key disappears, and a type-per-variant enum becomes bytes that the output format can use directly. The fourth step is instructive because it was not the end state: boxing fixed the enum's size but spread the data across the heap, and the fifth step removed both the padding and the pointers.