Pulpie: Pareto-Optimal Models for Cleaning the Web

Feyn released Pulpie, a family of three open-weight models that take raw HTML and return the main content. The claim in the announcement is a cost claim rather than a quality claim: the smallest model scores 0.862 ROUGE-5 F1 on WebMainBench against Dripper's 0.864, but does it at 210M parameters instead of 600M and at 13.7 pages/sec instead of 0.68 on the same NVIDIA L4. Cleaning a billion pages works out to roughly $7,900 with Pulpie Orange Small and $159,000 with Dripper.

Why extraction quality is worth money

Feyn's framing is that models eat the web twice, in pre-training and again at inference, and both times most of the input is junk. Their own measurement during data discovery: about 70% of the blocks on a typical HTML page are navigation, ads, sidebars and footers, leaving main content as a small fraction of the file.

The pre-training evidence comes from the AICC paper (Ma et al., 2025). Two corpora were built from the same Common Crawl snapshot with everything held constant except the extractor — heuristics on one side, a model-based parser on the other — and identical models trained on each. The model trained on the model-extracted corpus scored 1.08 percentage points higher averaged over 13 benchmarks, and beat models trained on FineWeb and RefinedWeb, both of which get their reputation from elaborate filtering and deduplication rather than from parsing.

Heuristic extractors also corrupt structured content rather than merely dropping it. Trafilatura preserves code blocks at 0.13 similarity and formulas at 0.61, against 0.91 and 0.94 for model-based extraction. A model trained on that corpus inherits the damage.

The inference half of the argument leans on Shi et al. (ICML 2023): a single irrelevant passage is enough to derail an answer. Which is the same reason clean extraction matters for context-engineering — a noisy page in context costs both tokens and accuracy.

Structure-based versus reading extractors

The post splits existing extractors by one question: does the method read the page, or inspect its structure?

Structure-based extractors judge a block by surface signals — tags, DOM shape, text density. Trafilatura, Readability and magic-html work this way, and Boilerpipe trains a classifier over the same signals. They are cheap to run and they confuse elements that are built alike. A navigation table and a data table are indistinguishable to something counting cells.

Reading extractors feed the page to a transformer and label each block by what it says. Dripper is a decoder doing this, and the decoder architecture is where the cost comes from: emitting one label at a time means reading the full model out of GPU memory for each step, so throughput is bound by memory bandwidth.

Pulpie keeps the reading approach and moves the bottleneck to compute. It is an encoder that labels every block in one forward pass, which is a dense matrix multiply limited by FLOPS rather than by bandwidth. That difference is why the speed gap is wider than the size ratio would suggest — 7.1x on an A100, 20x on an L4. The A100 has ~6.8x the memory bandwidth of the L4 but only ~2.6x the tensor-core throughput, so dropping to the cheaper card starves the decoder much harder than the encoder. Pulpie Orange Large matches Dripper on the A100 and pulls ahead on the L4.

The pipeline

Four stages. Simplify the HTML (strip scripts, styles and formatting noise, tag each block with an ID), chunk the blocks into packs of at most 8,192 tokens, run one forward pass to label each block content or boilerplate, return the kept blocks as HTML or convert to Markdown. About 80% of pages fit in a single chunk.

The chunking has a side benefit that shows up in the benchmark. Dripper returns nothing on 135 of the 6,647 WebMainBench English pages, and 130 of those are pages that overflow its 32k-token context window. Pulpie's chunk packing means page length can't cause a failure; Orange Small returns nothing on 45 pages, Orange Large on 21.

How it was trained

No public dataset had block-level labels, so Feyn built one. They sampled 16,670 English Common Crawl pages, one per domain, split each into blocks with MinerU-HTML, and labeled every block with DeepSeek V3.2. After removing empty and corrupted pages, 15,880 remained. They then ran Dripper 0.6B as a second labeler over all of them; block-level agreement with DeepSeek was 93.3%, and they kept only the 14,959 pages where the two labelers agreed on at least 70% of blocks.

The teacher is EuroBERT-2.1B fine-tuned on those pages — learning rate 2e-5, effective batch size 8, class-weighted cross-entropy with weights set inversely to the 28.6% content rate, on 4x A100. It scores 0.873.

The two students are distilled from it in the Hinton et al. style: KL divergence against the teacher's softened distribution weighted 0.7, hard-label cross-entropy 0.3, temperature 2.0, same training data. Orange Base (610M) lands at 0.863 and Orange Small (210M) at 0.862, so a tenfold size cut costs 1.1 F1 points.

Broken out by difficulty, the heuristics fall fastest: 14 to 20 F1 points from simple to hard pages, against about 9 for the encoders. Feyn puts frontier LLMs near 0.90 on this benchmark, which is the ceiling Pulpie is approaching rather than reaching.

Using it

from pulpie import Extractor

extractor = Extractor()               # defaults to Pulpie Orange Small
result = extractor.extract(html)

print(result.markdown)
print(result.n_main, result.n_other)  # blocks kept vs dropped

Extractor(model="large") trades speed for the teacher's quality, and a Pipeline class overlaps CPU preprocessing with GPU inference for bulk runs across one or more GPUs. All three models are on Hugging Face under feyninc/, built on EuroBERT with a shared tokenizer and the same <|sep|> block-marker architecture.

Feyn credits the MinerU-HTML and Dripper team throughout: their simplify_html preprocessing, their block-level annotation scheme, the WebMainBench benchmark, and Dripper 0.6B itself as a cross-validating labeler.

Where this touches the vault

This vault's ingest path runs on the heuristic side of the split. claude-defuddle uses the defuddle CLI, which is a structure-based extractor of the Readability lineage, and the 0.13 code-block similarity number is exactly the failure mode that matters when clipping technical articles. A 210M encoder is small enough to run locally, so the model-based route is no longer only available to people with pre-training budgets. On the retrieval side, kagi-cli's `extract` command solves the same problem by paying an API instead.