#crawling

2 pages search “crawling” ↳ all tags

Wiki 2

Consent in Crisis: The Rapid Decline of the AI Data Commons Longpre et al.'s audit of 14,000 domains behind C4, RefinedWeb and Dolma - AI crawl restrictions surged in 2023-2024, cutting off the best web data
Marginalia: Unranked, systemd, crawls Marginalia Search drops docker compose for systemd, adds an unranked query path, halves crawl time