Web Crawl Filtering & Quality Pipelines
Raw web dumps like Common Crawl contain up to 80% spam, machine-generated junk, duplicate boilerplates, and SEO text. Pre-training on unfiltered data degrades model reasoning and produces high loss floors.
Heuristic Quality Filters
Filters check token-to-symbol ratios, digit-to-char ratios, mean word length, line-ending punctuation, and toxic word lists to eliminate non-prose web garbage.
FastText Classifier Scoring
Training lightweight FastText or n-gram classifiers on high-quality target sets (Wikipedia, curated books) vs. raw web dumps assigns a probability score to every web document.
Language Identification
Enforcing clean multilingual proportions using FastText lang-id prevents low-quality untranslated web spam from corrupting domain allocations.
Linear classification over document n-gram embeddings x_doc provides microsecond quality filtering at petabyte scale.
- FineWeb (Hugging Face) filtered 96 Common Crawl dumps down to 15 trillion high-quality tokens using an open, multi-stage heuristic and classifier pipeline.