TRAINING AT SCALE

MODULE 2: DATA AT WEB SCALE · SCENE 06

From crawl to corpus

Heuristic filters — symbol ratios, language ID, text density — kill most of the web before a model sees it.

MIN SYMBOL RATIOPASSED FILTER: 8 / 10 DOCS
DOC #062%KEEPDOC #182%KEEPDOC #249%DISCARDDOC #353%KEEPDOC #459%KEEPDOC #564%DISCARDDOC #676%KEEPDOC #753%KEEPDOC #851%KEEPDOC #967%KEEP

Heuristic quality filters evaluate symbol density, punctuation, and FastText classification to eliminate web spam.

ARCHITECTURE & FORMULATIONModule 2: Data Engineering & Curation at Web-Scale

Web Crawl Filtering & Quality Pipelines

Raw web dumps like Common Crawl contain up to 80% spam, machine-generated junk, duplicate boilerplates, and SEO text. Pre-training on unfiltered data degrades model reasoning and produces high loss floors.

Heuristic Quality Filters

Filters check token-to-symbol ratios, digit-to-char ratios, mean word length, line-ending punctuation, and toxic word lists to eliminate non-prose web garbage.

FastText Classifier Scoring

Training lightweight FastText or n-gram classifiers on high-quality target sets (Wikipedia, curated books) vs. raw web dumps assigns a probability score to every web document.

Language Identification

Enforcing clean multilingual proportions using FastText lang-id prevents low-quality untranslated web spam from corrupting domain allocations.

MATHEMATICAL FORMULATION · FASTTEXT DOCUMENT QUALITY PROBABILITY
P(Quality | doc) = exp( w_quality · x_doc ) / ∑_c exp( w_c · x_doc )

Linear classification over document n-gram embeddings x_doc provides microsecond quality filtering at petabyte scale.

REAL-WORLD PRODUCTION ENGINEERING
  • FineWeb (Hugging Face) filtered 96 Common Crawl dumps down to 15 trillion high-quality tokens using an open, multi-stage heuristic and classifier pipeline.
← 05 · The square the model must pay for
07 · Delete first, train later →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]