Marginalia: Unranked, systemd, crawls

Three operational changes to Marginalia Search, written up by its author in July 2026: production moved off docker compose onto bare systemd, the index gained a query path that skips ranking entirely, and the crawler was split so that blog farms stop holding up everything else. His summary of the effect is "removed a lot of operational headaches, reduced expensive queries leaving more computational power for the rest, and cut crawl times in half."

Why docker didn't fit

Three constraints in the deployment push against a standard container setup.

The production box has two CPUs, each with its own RAM bank. Reaching into the other CPU's memory works but costs, and index and database processes are bottlenecked on RAM bandwidth in the first place, so NUMA placement matters more here than it does for a typical service.

Process lifecycles vary wildly. A search engine is anything but stateless: some processes are heavy and run for weeks, some services take a long time to start, some need restarting often. That alone forces the engine to be cut into separate pieces — services, though the author is careful to say not necessarily microservices.

The crawler and its neighbours run off about a dozen public IP addresses on one host, done with ipvlan and network namespaces. Linux has plenty of machinery for giving a process its own network stack and wiring it up with virtual switches and patch cables; the author's complaint is that no tool for managing it is any good — each is either too low-level, or abstracts away the part you need, or keeps a second source of truth that drifts from the first.

Docker can handle all three, at the price of considerable jank. Its networking model supports ipvlan but doesn't map onto it well: with a finite subnet you need spare free IPs to restart services reliably. It won't let you set firewall rules on the ipvlan interface, so anything living there had better not bind the public IP. And there is no adequate way for a process inside the container to learn which interface is the public one, leaving you to guess from RFC 1918 ranges and hope.

The argument for systemd is that docker is cgroups and namespaces underneath anyway, and systemd is the version of that with less model mismatch and more of the configuration in your hands, while still supplying health checks and configurable automatic restarts. Writing a service unit per process is tedious, and drop-ins take the edge off it. The reported result: more stable than docker ever felt, faster deployments, less service-discovery jank because "containers" keep the same internal and external IP, and builds down to 2-3 seconds now that JIB is out of the build path. His hedge is that systemd on a desktop is overengineered and painful, but for this kind of deployment the design is well motivated.

Unranked queries

The normal query pipeline is expensive and does work that many queries don't need. A backlink lookup like site:foo.com links:bar.com is a pure intersection of two term lists — there is nothing to rank — yet threads were being allocated to ranking and term positions were being retrieved anyway.

The new path does dumb term intersections and close to nothing else. It runs single-threaded in around 5ms, at least an order of magnitude below a full query, and the fact that it is single-threaded is a large part of the win: it hands threads back to the execution pool for work that actually needs them. Up to half the query load can go to the endpoint, though the share swings around because much of that traffic is bots and scrapers walking the /site viewer.

Skipping ranking also made exhaustive retrieval possible. Unlike the ranked path, an unranked query can be paged through to the end, given a cursor that tracks position across index partitions:

28mbshfptkj.6ijoop7xty.43nivb3bn1.817ucldxy2x.90.12ria2fmp7a.32cydk0dei2t.73wrhi4igjr.53lb1o4iof7

Each period-delimited part starts with one character identifying a partition, followed by a base-36 document id to resume from on that partition. Short enough to survive in a query string or an API call, which the author says is all that matters.

Splitting the crawler

Crawl time had been creeping up for years. Each partition was taking almost two weeks, and with 8 main index partitions a full crawl was approaching four months — long enough that results go stale before the pass finishes.

The cause is that subdomains are pareto distributed and crawler politeness forbids hammering several hosts on the same top domain at once. Known subdomain counts per top domain, from the post: tumblr.com 8,450,330, blogspot.com 1,203,425, wordpress.com 941,330, then uptodown.com, bandcamp.com, livejournal.com, appstor.io, github.io, substack.com, dreamwidth.org, medium.com and wixsite.com trailing from 334,941 down to 112,578. Substack in particular is quick to return 429s, and burning an IP means losing the ability to index the domain at all, which is worse than being slow.

So the main crawl was finishing in a few days and then trickling for another week-plus through substack, medium, wordpress, github.io and neocities. The fix was a separate crawler partition holding the wide domains, crawled on a time budget rather than to completion: it takes as many sites as it can in a week, then stops so the updates can be indexed. It runs in parallel with the main crawler, which now finishes in about five days instead of two weeks.

The same author states the underlying principle three days later in your-harddrive-is-full: impose a real constraint deliberately smaller than the resources you actually have, and optimize against that rather than waiting for the pain point. A search engine on one tight box, with a crawler on a time budget instead of a completion target, is that argument run in production.