TRAINING AT SCALE

MODULE 2: DATA AT WEB SCALE · SCENE 07

Delete first, train later

MinHash + LSH find near-duplicates so the model memorizes the world once, not a million copies of it.

RAW DOCUMENTS: 8DUPLICATE BUCKETS: 2UNIQUE DOCS: 5
#102KEEP#451KEEP#102DUP -> DELETE#889KEEP#451DUP -> DELETE#310KEEP#991KEEP#102DUP -> DELETE

MinHash + LSH maps similar document shingles into matching buckets, enabling 20-50% corpus deduplication.

ARCHITECTURE & FORMULATIONModule 2: Data Engineering & Curation at Web-Scale

Advanced Deduplication: MinHash, LSH, & Suffix Arrays

Web data contains millions of near-identical documents, news syndications, privacy disclaimers, and boilerplate footers. Deduplication prevents memorization, reduces loss spikes, and saves thousands of GPU hours.

Document-Level MinHash + LSH

Documents are split into n-gram sets (shingles). MinHash computes signature vectors. Locality-Sensitive Hashing (LSH) hashes signature bands into buckets to catch near-duplicates with Jaccard similarity > 0.8 without pairwise comparisons.

Line-Level & Sub-String Dedup

Common headers, footers, and cookie banners span across otherwise distinct documents. Suffix arrays locate repeated byte spans >= 50 tokens across multi-terabyte datasets.

Impact on Pre-Training

Deduplication reduces corpus size by 20%-50% while improving downstream benchmark accuracy, as models spend FLOPs learning concepts rather than repeating memorized text.

MATHEMATICAL FORMULATION · JACCARD SIMILARITY MINHASH PROPERTY
Prob( min_hash(A) == min_hash(B) ) = Jaccard(A, B) = |A ∩ B| / |A ∪ B|

The probability that two document shingle sets produce identical minimum hash values equals their exact Jaccard overlap.

REAL-WORLD PRODUCTION ENGINEERING
  • Llama 3 data processing pipelines execute multi-stage LSH dedup across Spark/Ray clusters spanning thousands of CPU cores.
← 06 · From crawl to corpus
08 · Building the vocabulary →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]